arno/bitcoinbook
1
0
Fork 0
mirror of https://github.com/bitcoinbook/bitcoinbook synced 2024-12-24 15:38:08 +00:00
Code Issues Releases Wiki Activity

Merge branch 'develop' and 'Chapter6-Reorg'

Browse Source 
Conflicts:
	ch05.asciidoc
This commit is contained in:
Andreas M. Antonopoulos 2014-07-21 20:30:23 -05:00
commit c235408ceb
24 changed files with 883 additions and 503 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

52
ch04.asciidoc
View File

@ -12,6 +12,52 @@ In the payment portion of a bitcoin transaction, the recipient's public key is r
In this chapter we will introduce wallets, which contain cryptographic keys. We will look at how keys are generated, stored and managed. We will review the various encoding formats used to represent private and public keys, addresses and script addresses. Finally we will look at special uses of keys: to sign messages, to prove ownership and to create vanity addresses and paper wallets. In this chapter we will introduce wallets, which contain cryptographic keys. We will look at how keys are generated, stored and managed. We will review the various encoding formats used to represent private and public keys, addresses and script addresses. Finally we will look at special uses of keys: to sign messages, to prove ownership and to create vanity addresses and paper wallets.
=== ECDSA Arithmetic
Just as RSA uses integer arithmetic operators over a prime modulus to provide cryptographic functions, ECDSA defines an arithmetic that has some features in common with traditional arithmetic.
Wikipedia has a good article that explains the details of how arithmetic operations work on an elliptic curve. See http://en.wikipedia.org/wiki/Elliptic_curve_cryptography for more information.
Glossing over details, here are the fundamental facts:
An elliptic curve field is a set of points (x, y) each of which that satisfies the equation
y^2^ = x^3^ + ax + b (mod P)
for some constants a, b and P (where P is prime). Bitcoin uses a standard curve known as secp256, where a=0, b=7, and P = 2^256^ - 2^32^ - 2^9^ - 2^8^ - 2^7^ - 2^6^ - 2^4^ - 1.
So for example, (55066263022277343669578718895168534326250603453777594175500187360389116729240, 32670510020758816978083085130507043184471273380659243275938904335757337482424) is a point on the secp256 curve. You can check this yourself using Python.
----
Python 3.4.0 (default, Mar 30 2014, 19:23:13)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> p = 115792089237316195423570985008687907853269984665640564039457584007908834671663
>>> x = 55066263022277343669578718895168534326250603453777594175500187360389116729240
>>> y = 32670510020758816978083085130507043184471273380659243275938904335757337482424
>>> (x ** 3 + 7 - y**2) % p
0
----
In addition, there is also a "point at infinity", which roughly corresponds to the role of 0 in addition. On computers, it's sometimes represented by x = y = 0 (which doesn't satisfy the elliptic curve equation -- but it's an easy separate case that can be checked).
There is also an operator "+", called "addition" which has some properties similar to the traditional addition of real numbers that grade school children learn. Given two points P~1~ and P~2~ on the elliptic curve, there is a third point P~3~ = P~1~ + P~2~, also on the elliptic curve.
Geometrically, this third point P~3~ is calculated by drawing a line between P~1~ and P~2~. This line will intersect the elliptic curve in exactly one additional place. Call this point P~3~' = (x, y). Then reflect in the X axis to get P~3~ = (x, -y).
There are a couple of special cases which explain the need for the "point at infinity".
If P~1~ and P~2~ are the same point, the line "between" P~1~ and P~2~ should extend to be the tangent on the curve at this point P~1~. This tangent will intersect the curve in exactly one new point. You can use techniques from calculus to determine the slope of the tangent line (techniques which curiously work even though we are restricting our interest to points on the curve with to integer coordinates!).
In some cases (ie. if P~1~ and P~2~ have the same x values but different y values), the tangent line will be exactly vertical, in which case P3 = "point at infinity".
If one of P~1~ is the "point at infinity", then the sum P~1~ + P~2~ = P~2~. Similary, if P~2~ is the point at infinity, then P~1~ + P~2~ = P~1~. This shows how the point at infinity plays the roll of 0.
It turns out that + is commutative, which means that (A+B)+C = A+(B+C). That means we can write A+B+C without parantheses without any ambiguity.
Now that we have defined addition, we can define multiplication in the standard way that extends addition. For a point P on the elliptic curve, if k is a whole number, then kP = P + P + P + ... + P (k times). Note that k is sometimes confusingly called an "exponent" in this case. (It would make a lot more sense to call it this if we used an operator that looked like multiplication rather than "+".)
=== Keys === Keys
==== Public key cryptography and crypto-currency ==== Public key cryptography and crypto-currency
@ -28,7 +74,7 @@ When spending bitcoins, the current bitcoin owner presents their public key and
[TIP] [TIP]
==== ====
In most implementations, the private and public keys are stored together as a _key pair_ for convenience. However, it is trivial to reproduce the public key if one has the private key, so storing only the private key is also possible. In most implementations, the private and public keys are stored together as a _key pair_ for convenience. However, the public key can be calculated from the private key, so storing only the private key is also possible.
==== ====
==== Private and Public Keys ==== Private and Public Keys
@ -70,7 +116,7 @@ Below is a randomly generated private key shown in hexadecimal format (256 binar
[TIP] [TIP]
==== ====
The size of bitcoin's private key, 2^256^ is an unfathomably large number. It is approximately 10^77^ in decimal. The visible universe is estimated to contain 10^80^ atoms. The size of bitcoin's private key space, 2^256^ is an unfathomably large number. It is approximately 10^77^ in decimal. The visible universe is estimated to contain 10^80^ atoms.
==== ====
To generate a new key with the Bitcoin Core Client (see <<ch03_bitcoin_client>>), use the +getnewaddress+ command. For security reasons it displays the public key only, not the private key. To ask bitcoind to expose the private key, use the +dumpprivkey+ command. The +dumpprivkey+ shows the private key in a base-58 checksum encoded format called the Wallet Import Format (WIF), which we will examine in more detail in <<priv_formats>>. Here's an example of generating and displaying a private key using these two commands: To generate a new key with the Bitcoin Core Client (see <<ch03_bitcoin_client>>), use the +getnewaddress+ command. For security reasons it displays the public key only, not the private key. To ask bitcoind to expose the private key, use the +dumpprivkey+ command. The +dumpprivkey+ shows the private key in a base-58 checksum encoded format called the Wallet Import Format (WIF), which we will examine in more detail in <<priv_formats>>. Here's an example of generating and displaying a private key using these two commands:
@ -100,7 +146,7 @@ A private key is just a number. A public key can be generated from any number, u
[[pubkey]] [[pubkey]]
==== Public Keys ==== Public Keys
The public key is calculated from the private key using elliptic curve multiplication, which is irreversible: latexmath:[\(K = k * G\)]+ where +k+ is the private key, +G+ is a constant point called the _Generator Point_ and +K+ is the resulting public key. The reverse operation, division -- calculating +k+ if you know +K+ -- is as difficult as trying all possible values of +k+, i.e. a brute-force search. Before we demonstrate how to generate a public key from a private key, let's look at Elliptic Curve Cryptography in a bit more detail. The public key is calculated from the private key using elliptic curve multiplication, which is irreversible: latexmath:[\(K = k * G\)]+ where +k+ is the private key, +G+ is a constant point called the _Generator Point_ and +K+ is the resulting public key. The reverse operation, known as "finding the discrete logarithm" -- calculating +k+ if you know +K+ -- is as difficult as trying all possible values of +k+, i.e. a brute-force search. Before we demonstrate how to generate a public key from a private key, let's look at Elliptic Curve Cryptography in a bit more detail.
[[elliptic_curve]] [[elliptic_curve]]
==== Elliptic Curve Cryptography Explained ==== Elliptic Curve Cryptography Explained

730
ch06.asciidoc
View File

@ -1,524 +1,278 @@
[[ch6]] [[ch6]]
== Chapter 6 - The Blockchain & Mining == Chapter 6 - The Bitcoin Network
*DRAFT - DO NOT SUBMIT ISSUES OR PULL REQUESTS YET PLEASE - CONSTANT CHANGES HAPPENING* *DRAFT - DO NOT SUBMIT ISSUES OR PULL REQUESTS YET PLEASE - CONSTANT CHANGES HAPPENING*
=== Introduction === Introduction
((("blockchain")))
Bitcoin's blockchain is the global public ledger (list) of all transactions, which everyone in the bitcoin network accepts as the authoritative record of ownership. === Peer-to-Peer Network Architecture
But how can everyone in the network agree on a single universal "truth" about who owns what, without having to trust anyone? All traditional payment systems depend on a trust model that has a central authority providing a clearinghouse service, basically verifying and clearing all transactions. Bitcoin has no central authority, yet somehow every node has a complete copy of a public ledger that it can trust as the authoritative record. The blockchain is not created by a central authority, but is assembled independently by every node in the network. Somehow, every node in the network, acting on information transmitted across insecure network connections can arrive at the same conclusion and assemble a copy of the same public ledger as everyone else. This chapter examines the process by which the bitcoin network achieves global consensus without central authority. Bitcoin is structured as a peer-to-peer network architecture on top of the Internet. The term peer-to-peer or P2P means that the computers that participate in the network are peers to each other, they are all equal there are no "special" nodes and all nodes share the burden of providing network services. The network nodes interconnect in a mesh network with a "flat" topology. There is no "server", no centralized service and no hierarchy within the network. Nodes in a peer-to-peer network both provide and consume services at the same time, with reciprocity acting as the incentive for participation. Peer-to-peer networks are inherently resilient, de-centralized and open. The pre-eminent example of a P2P network architecture was the early Internet itself, where nodes on the IP network were equal. Today's Internet architecture is more hierarchical, but the Internet Protocol still retains its flat-topology essence. Beyond bitcoin, the largest and most successful application of P2P technologies is file sharing, with Napster as the pioneer and bittorrent as the most recent evolution of the architecture.
Satoshi Nakamoto's main invention is the decentralized mechanism for emergent consensus. All the properties of bitcoin, including currency, transactions, payments and the security model that does not depend central authority or trust derive from this invention. Bitcoins P2P network architecture is much more than a topology choice. Bitcoin is a peer-to-peer digital cash system by design, and the network architecture is both a reflection and a foundation of that core characteristic. De-centralization of control is a core design principle and that can only be achieved and maintained by a flat, de-centralized P2P consensus network.
Bitcoin's consensus emerges from the interplay of three processes that occur independently on nodes across the network: The term "bitcoin network" refers to the collection of nodes running the bitcoin P2P protocol. In addition to the bitcoin P2P protocol, there are other protocols such as Stratum, that are used for mining and lightweight or mobile wallets. These additional protocols are provided by gateway routing servers that access the bitcoin network using the bitcoin P2P protocol and then extend that network to nodes running other protocols. For example, Stratum servers connect Stratum mining nodes via the Stratum protocol to the main bitcoin network and bridge the Stratum protocol to the bitcoin P2P protocol. We use the term "extended bitcoin network" to refer to the overall network that includes the bitcoin P2P protocol, pool mining protocols, the Stratum protocol and any other related protocols connecting the components of the bitcoin system.
* Independent verification of each transaction, by every full node, based on a comprehensive list of criteria === Nodes Types and Roles
* Independent aggregation of those transactions into new blocks by mining nodes, coupled with demonstrated computation through a Proof-of-Work algorithm
* Independent assembly of the new blocks by every full node into a chain and selection of the chain with the most cumulative computation demonstrated through Proof-of-Work
In the next few sections we will examine these processes and how they interact to create the emergent property of network-wide consensus that allows any bitcoin node to assemble its own copy of the authoritative, trusted, public, global ledger. While nodes in the bitcoin P2P network are equal, they may take on different "roles", depending on the functionality they are supporting. A bitcoin node is a collection of functions: routing, the blockchain database, mining, and wallet services. A full node with all four of these functions is shown below:
Each of these processes also aggregates smaller bitcoin units into larger data structures. First, unspent transaction outputs (UTXO) are aggregated into transactions. Next, many transactions are aggregated into a block. Finally, blocks are added to a chain of blocks, the blockchain. In the previous chapter we looked at transactions as a data structure. In this chapter we will also look at the larger data structures: blocks and the blockchain. [[full_node_reference]]
.A bitcoin network node with all four functions: Network routing, Blockchain database, Mining and Wallet
image::images/FullNodeReferenceClient_Small.png["FullNodeReferenceClient_Small"]
=== Independently Verifying Transactions All nodes include the routing function to participate in the network and may include other functionality. All nodes validate and propagate transactions and blocks, discover and maintain connections to peers. In the full node example above, the routing function is indicated by an orange circle named "Network Routing Node".
In the previous chapter we saw how wallet software creates transactions by collecting UTXO, providing the appropriate unlocking scripts and then constructing new outputs assigned to a new owner. The resulting transaction is then sent to the neighboring nodes in the bitcoin network so that it may be propagated across the entire bitcoin network. Some nodes, called full nodes, also maintain a complete and up-to-date copy of the blockchain. Full nodes can autonomously and authoritatively verify any transaction without external reference. Some nodes maintain only a subset of the blockchain and verify transactions using a method called _Simple Payment Verification_ or SPV. These nodes are known as SPV or Lightweight nodes. In the full node example above, the full node blockchain database function is indicated by a blue circle named "Blockchain Database". SPV nodes are drawn without the blue circle, showing that they do not have a full copy of the blockchain.
Every bitcoin node that receives a transaction will first verify the transaction before forwarding it to its neighbors. This ensures that only valid transactions are propagated across the network, while invalid transactions are discarded at the first node that encounters them. Mining nodes compete to create new blocks by running specialized hardware to solve the proof-of-work algorithm. Some mining nodes are also full nodes, maintaining a full copy of the blockchain while others are lightweight nodes participating in pool mining and depending on a pool server to maintain a full node. The mining function is shown in the full node above as a black circle named "Mining".
Each node verifies every transaction against a long checklist of criteria: User wallets may be part of a full node, as is usually the case with desktop bitcoin clients. Increasingly many user wallets, especially those running on resource constrained devices such as smart phones, are SPV nodes. The wallet function is shown above as a green circle named "Wallet".
* Check the syntactic correctness of the transaction's data structure In addition to the main node types on the bitcoin P2P protocol, there are servers and nodes running other protocols, such as specialized mining pool protocols and lightweight client access protocols.
* Make sure neither lists of inputs or outputs are empty
* The transaction size in bytes is less than MAX_BLOCK_SIZE
* Each output value, as well as the total, must be within the allowed range of values (less than 21m coins, more than 0)
* Check none of the inputs have hash=0, N=-1 (coinbase transactions should not be relayed)
* Check that nLockTime is less than or equal to INT_MAX
* Check that the transaction size in bytes is greater than or equal to 100
* Check the number of signature operations contained in the transaction is less than the signature operation limit
* Reject "nonstandard" transactions: unlocking script (scriptSig) doing anything other than pushing numbers on the stack, or the locking script (scriptPubkey) not matching isStandard forms
* Check for a matching transaction in the pool, or in a block in the main branch, if so reject this transaction
* For each input, if the referenced output exists in any other transaction in the pool, reject this transaction.
* For each input, look in the main branch and the transaction pool to find the referenced output transaction. If the output transaction is missing for any input, this will be an orphan transaction. Add to the orphan transactions, if a matching transaction is not already in the pool.
* For each input, if the referenced output transaction is a coinbase output, it must have at least COINBASE_MATURITY (100) confirmations; else reject this transaction
* For each input, if the referenced output does not exist (e.g. never existed or has already been spent), reject this transaction
* Using the referenced output transactions to get input values, check that each input value, as well as the sum, are in the allowed range of values (less than 21m coins, more than 0)
* Reject if the sum of input values < sum of output values
* Reject if transaction fee would be too low to get into an empty block
* Verify the unlocking scripts for each input against the corresponding output locking scripts
These conditions can be seen in detail in the functions AcceptToMemoryPool, CheckTransaction and CheckInputs in the bitcoin reference client. Note that the conditions change over time, to address new types of Denial-of-Service attacks or sometimes to relax the rules so as to include more types of transactions. Here are the most common node types on the extended bitcoin network:
By independently verifying each transaction as it is received and before propagating it, every node builds a pool of valid new transactions (the transaction pool), roughly in the same order. [[node_type_ledgend]]
.Different types of nodes on the extended bitcoin network
image::images/BitcoinNodeTypes.png["BitcoinNodeTypes"]
=== The Extended Bitcoin Network
The main bitcoin network, running the bitcoin P2P protocol consists of between 7,000 to 10,000 nodes running various versions of the bitcoin reference client (Bitcoin Core) and a few hundred nodes running various other implementations of the bitcoin P2P protocol, such as BitcoinJ, Libbitcoin and btcd. A small percentage of the nodes on the bitcoin P2P network are also mining nodes, competing in the mining process, validating transactions and creating new blocks. Various large companies interface with the bitcoin network by running full-node clients based on the Bitcoin Core client, with full copies of the blockchain and a network node, but without mining or wallet functions. These nodes act as network edge routers, allowing various other services (exchanges, wallets, block explorers, merchant payment processing) to be built on top.
The extended bitcoin network includes the network running the bitcoin P2P protocol, described above, as well as nodes running specialized protocols. Attached to the main bitcoin P2P network are a number of pool servers and protocol gateways that connect nodes running other protocols, mostly pool mining nodes (see <<mining>>) and lightweight wallet clients, which do not carry a full copy of the blockchain.
The diagram below shows the extended bitcoin network with the various types of nodes, gateway servers, edge routers and wallet clients and the various protocols they use to connect to each other.
[[bitcoin_network]]
.The extended bitcoin network showing various node types, gateways and protocols
image::images/BitcoinNetwork.png["BitcoinNetwork"]
=== Network Discovery
When a new node boots up, it must discover other bitcoin nodes on the network in order to participate. To start this process, a new node must discover at least one existing node on the network and connect to it. The geographic location of the other nodes is irrelevant, the bitcoin network topology is not geographically defined. Therefore, any existing bitcoin nodes can be selected at random.
To connect to a known peer, nodes establish a TCP connection, usually to port 8333 (the bitcoin "well known" port), or an alternative port if one is provided. Upon establishing a connection, the node will start a "handshake" by transmitting a +version+ message, which contains basic identifying information, including:
* PROTOCOL_VERSION, a constant that defines the bitcoin P2P protocol version the client "speaks". E.g. 70002
* nLocalServices, a list of local services supported by the node, currently just NODE_NETWORK
* nTime, the current time
* addrYou, the IP address of the remote node as seen from this node
* addrMe, the IP address of the local node, as discovered by the local node
* subver, a sub-version showing the type of software running on this node, e.g. "/Satoshi:0.9.2.1/"
* BestHeight, the block height of this node's blockchain
(See https://github.com/bitcoin/bitcoin/blob/d3cb2b8acfce36d359262b4afd7e7235eff106b0/src/net.cpp#L562 for an example of the +version+ network message)
The peer node responds with +verack+ to acknowledge and establish a connection, and optionally sends its own +version+ message if it wishes to reciprocate the connection and connect back as a peer.
[[network_handshake]]
.The initial handshake between peers
image::images/NetworkHandshake.png["NetworkHandshake"]
How does a new node find peers? While there are no special nodes in bitcoin, there are some long running stable nodes that are listed in the client as _seed nodes_. While a new node does not have to connect with the seed nodes, it can use them to quickly discover other nodes in the network. In the Bitcoin Core client, the option to use the seed nodes is controlled by the option switch +-dnsseed+, which is set to 1, to use the seed nodes, by default. Alternatively, a bootstrapping node that knows nothing of the network must be given the IP address of at least one bitcoin node after which it can establish connections through further introductions. The command line argument +-seednode+ can be used to connect to one node just for introductions, using it as a DNS seed. After the initial seed node is used to form introductions, the client will disconnect from it and use the newly discovered peers.
Once one or more connections is established, the new node will send an +addr+ message containing its own IP address, to its neighbors. The neighbors will in turn forward the +addr+ message to their neighbors, ensuring that the newly connected node becomes well known and better connected. Additionally, the newly connected node can send +getaddr+ to the neighbors asking them to return a list of IP addresses of other peers. That way, a node can find peers to connect to and advertise its existence on the network for other nodes to find it. On a node running the Bitcoin Core client, you can list the peer connections with the command +getpeerinfo+:
[[address_propagation]]
.Address Propagation and Discovery
image::images/AddressPropagation.png["AddressPropagation"]
----
$ bitcoin-cli getpeerinfo
[
{
"addr" : "85.213.199.39:8333",
"services" : "00000001",
"lastsend" : 1405634126,
"lastrecv" : 1405634127,
"bytessent" : 23487651,
"bytesrecv" : 138679099,
"conntime" : 1405021768,
"pingtime" : 0.00000000,
"version" : 70002,
"subver" : "/Satoshi:0.9.2.1/",
"inbound" : false,
"startingheight" : 310131,
"banscore" : 0,
"syncnode" : true
},
{
"addr" : "58.23.244.20:8333",
"services" : "00000001",
"lastsend" : 1405634127,
"lastrecv" : 1405634124,
"bytessent" : 4460918,
"bytesrecv" : 8903575,
"conntime" : 1405559628,
"pingtime" : 0.00000000,
"version" : 70001,
"subver" : "/Satoshi:0.8.6/",
"inbound" : false,
"startingheight" : 311074,
"banscore" : 0,
"syncnode" : false
}
]
----
A node must connect to a few different peers in order to establish diverse paths into the bitcoin network. These paths are not reliable, nodes come and go, and so the node must continue to discover new nodes as it loses old connections as well as assist other nodes when they bootstrap. Only one connection is needed to bootstrap, as the first node can offer introductions to its peer nodes and those peers can offer further introductions. Its also unnecessary and wasteful of network resources to connect to more than a handful of nodes. After bootstrapping a node will remember its most recent successful peer connections, so that if it is rebooted it can quickly reestablish connections with its former peer network. If none of the former peers respond to its connection request, the node can use the seed nodes to bootstrap again.
To override the automatic management of peers and to specify a list of IP addresses, users can provide the option +-connect=<IPAddress>+ and specify one or more IP addresses. If this option is used, the node will only connect to the selected IP addresses, instead of discovering and maintaining the peer connections automatically.
If there is no traffic on a connection, nodes will periodically send a message to maintain the connection. If a node has not communicated on a connection for more than 90 minutes it is assumed to be disconnected and a new peer will be sought. Thus the network dynamically adjusts to transient nodes, network problems, and can organically grow and shrink as needed without any central control.
=== Full Nodes
Full nodes are nodes that maintain a full blockchain, with all transactions. More accurately they probably should be called "full blockchain nodes". In the early years of bitcoin, all nodes were full nodes and currently the Bitcoin Core client is a full blockchain node. In the last two years however, new forms of bitcoin clients have been introduced, which do not maintain a full blockchain but run as lightweight clients. These are examined in more detail in the next section.
Full blockchain nodes maintain a complete and up-to-date copy of the bitcoin blockchain, with all the transactions, which they independently build and verify, starting with the very first block (genesis block) and up to the latest known block in the network. A full blockchain node can independently and authoritatively verify any transaction, without recourse or reliance on any other node or source of information. The full blockchain node relies on the network to receive updates about new blocks of transactions, which it then verifies and incorporates into its local copy of the blockchain.
Running a full blockchain node gives you the pure bitcoin experience: independent verification of all transactions without the need to rely on, or trust, any other systems. It's easy to tell if you're running a full node because it requires several gigabytes of persistent storage (disk space) to store the full blockchain. If you need a lot of disk and it takes 2-3 days to "sync" to the network you are running a full node. That is the price of complete independence and freedom from central authority.
There are a few alternative implementations of full-blockchain bitcoin clients, built using different programming languages and software architectures. However, the most common implementation is the reference client Bitcoin Core, also known as the Satoshi Client. More than 90% of the nodes on the bitcoin network run various versions of Bitcoin Core. It is identified as "Satoshi" in the sub-version string sent in the +version+ message and shown by the command +getpeerinfo+ as we saw above, for example +/Satoshi:0.8.6/+.
=== Exchanging "Inventory"
The first thing a full node will do once it connects to peers is try to construct a complete blockchain. If it is a brand-new node and has no blockchain at all, then it only knows one block (the genesis block), which is statically embedded in the client software. Starting with block #0, the genesis block, the new node will have to download hundreds of thousands of blocks to synchronize with the network and establish a full blockchain.
The process of "syncing" the blockchain starts with the +version+ message, as that contains +BestHeight+, a node's current blockchain height (number of blocks). A node will see the +version+ messages from its peers, know how many blocks they each have and be able to compare to how many blocks it has in its own blockchain. Peered nodes will exchange a +getblocks+ message that contains the hash (fingerprint) of the top block on their local blockchain. One of the peers will be able to identify the received hash as belonging to a block that is not at the top, but rather belongs to an older block, thus deducing that its own local blockchain is longer than its peer's.
The peer that has the longer blockchain has more blocks that the other node and can identify which blocks the other node needs to "catch up". It will identify the first 500 blocks to share and transmit their hashes using an +inv+ (inventory) message. The node missing these blocks will then retrieve them, by issuing a series of +getdata+ messages requesting the full block data and identifying the requested blocks using the hashes from the +inv+ message.
Let's assume for example that a node only has the genesis block. It will then receive an +inv+ message from its peers containing the hashes of the next 500 blocks in the chain. It will start requesting blocks from all its connected peers, spreading the load and ensuring that it doesn't overwhelm any peer with requests. The node keeps track of how many blocks are "in transit" per peer connection, meaning blocks that it has requested but not received, checking that it does not exceed a limit (MAX_BLOCKS_IN_TRANSIT_PER_PEER). This way, if it needs a lot of blocks, it will only request new ones as previous requests are fulfilled, allowing the peers to control the pace of updates and not overwhelming the network. As each block is received, it is added to the blockchain as we will see in the next chapter <<blockchain>>. The local blockchain is gradually built up, more blocks are requested and received and the process continues until the node catches up to the rest of the network.
This process of comparing the local blockchain with the peers and retrieving any missing blocks happens any time a node goes offline for any period of time. Whether a node has been offline for a few minutes and is missing a few blocks, or a month and is missing a few thousand blocks, it starts by sending +getblocks+, gets an +inv+ response and starts downloading the missing blocks.
[[inventory_synchronization]]
.Node synchronizing the blockchain by retrieving blocks from a peer
image::images/InventorySynchronization.png["InventorySynchronization"]
=== Simple Payment Verification (SPV) Nodes
Not all nodes have the ability to store the full blockchain. Many bitcoin clients are designed to run on space- and power-constrained devices, such as smartphones, tablets or embedded systems. For such devices, a _simple payment verification_ (SPV) method is used to allow them to operate without storing the full blockchain. These types of clients are called SPV clients or lightweight clients. As bitcoin adoption surges, the SPV node is becoming the most common form of bitcoin node, especially for bitcoin wallets.
SPV nodes download only the block headers and do not download the transactions included in each block. The resulting chain of blocks, without transactions, is 1,000 times smaller than the full blockchain. SPV nodes cannot construct a full picture of all the UTXO that is available for spending, as they do not know about all the transactions on the network. SPV nodes verify transactions using a slightly different methodology that relies on peers to provide partial views of relevant parts of the blockchain on-demand.
As an analogy, a full node is like a tourist in a strange city, equipped with a detailed map of every street and every address. By comparison, an SPV node is like a tourist in a strange city asking random strangers for turn-by-turn directions while knowing only one main avenue. While both tourists can verify the existence of a street by visiting it, the tourist without a map doesn't know what lies down any of the side streets and doesn't know what other streets exist. Positioned in front of 23 Church Street, the tourist without a map cannot know if there are a dozen other "23 Church Street" addresses in the city and whether this is the right one. The map-less tourist's best chance is to ask enough people and hope some of them are not trying to mug the tourist.
Simple Payment Verification verifies transactions by reference to their _depth_ in the blockchain instead of their _height_. Whereas a full-blockchain node will construct a fully verified chain of thousands of blocks and transactions reaching down the blockchain (back in time) all the way to the genesis block, an SPV node will verify the chain of all blocks and link that chain to the transaction of interest.
For example, when examining a transaction in block 300,000, a full node links all 300,000 blocks down to the genesis block and builds a full database of UTXO, establishing the validity of the transaction by confirming that the UTXO remains unspent. An SPV node cannot validate whether the UTXO is unspent. Instead, the SPV node will establish a link between the transaction and the block that contains it, using a Merkle Path (see <<merkle_trees>>). Then, the SPV node waits until is sees the six blocks 300,001 through 300,006 piled on top of the block containing the transaction and verifies it by establishing its depth under blocks 300,006 to 300,001. The fact that other nodes on the network accepted block 300,000 and then did the necessary work to produce 6 more blocks on top of it is proof, by proxy, that the transaction was not a double-spend.
An SPV node cannot be persuaded that a transaction exists in a block, when it does not in fact exist. The SPV node establishes the existence of a transaction in a block by requesting a merkle path proof and by validating the proof-of-work in the chain of blocks. However, a transaction's existence can be "hidden" from an SPV node. An SPV node can definitely prove that a transaction exists but cannot verify that a transaction, such as a double-spend of the same UTXO, doesn't exist because it doesn't have a record of all transactions. This type of attack can be used as a Denial-of-Service attack or as a double-spending attack against SPV nodes. To defend against this, an SPV node needs to connect randomly to several nodes, to increase the probability that it is in contact with at least one honest node. SPV nodes are therefore vulnerable to network partitioning attacks or Sybil attacks, where they are connected to fake nodes or fake networks and do not have access to honest nodes or the real bitcoin network.
For most practical purposes, well-connected SPV nodes are secure enough, striking the right balance between resource needs, practicality and security. For the truly security conscious, however, nothing beats running a full blockchain node.
[TIP]
====
A full blockchain node verifies a transaction by checking the chain of thousands of blocks below it and checks the UTXO is not spent, whereas an SPV node checks how deep the block is buried by a handful of blocks above it.
====
To get the block headers, SPV nodes use a +getheaders+ message instead of +getblocks+. The responding peer will send up to 2000 block headers using a single +headers+ message. The process is otherwise the same as that used by a full node to retrieve full blocks. SPV nodes also set a filter on the connection to peers, to filter the stream of future blocks and transactions sent by the peers. Any transactions of interest are retrieved using a +getdata+ request. The peer generates a +tx+ message containing the transactions, in response.
[[spv_synchronization]]
.SPV Node synchronizing the block headers
image::images/SPVSynchronization.png["SPVSynchronization"]
Because SPV nodes need to retrieve specific transactions in order to selectively verify them, they also create a privacy risk. Unlike full-blockchain nodes, which collect all transactions within each block, the SPV node's requests for specific data can inadvertently reveal the addresses in their wallet. For example, a third party monitoring a network could keep track of all the transactions requested by a wallet on an SPV node and use those to associate bitcoin addresses with the user of that wallet, destroying the user's privacy.
Shortly after the introduction of SPV/lightweight nodes, the bitcoin developers added a feature called _bloom filters_ to address the privacy risks of SPV nodes. Bloom filters allow SPV nodes to receive a subset of the transactions without revealing precisely which addresses they are interested in, through a filtering mechanism that uses probabilities rather than fixed patterns.
=== Bloom Filters
A bloom filter is a probabilistic search filter, a way to describe a desired pattern without specifying it exactly. Bloom filters offer an efficient way to express a search pattern while protecting privacy. They are used by SPV nodes to ask their peers for transactions matching a specific pattern, without revealing exactly which addresses they are searching for.
Using our previous analogy of a tourist without a map asking for directions to a specific address "23 Church St". If they ask strangers for directions to this street, they inadvertently reveal their destination. A bloom filter is like asking "Are there any streets in this neighborhood whose name ends in R-C-H". A question like that reveals slightly less about the desired destination, than asking for "23 Church St". Using this technique, a tourist could specify the desired address in more detail as "ending in U-R-C-H" or less detail as "ending in H". By varying the precision of the search, the tourist reveals more or less information, at the expense of getting more or less specific results. If they ask a less specific pattern, they get a lot more possible addresses and better privacy but many of the results are irrelevant. If they ask for a very specific pattern then they get fewer results but they lose privacy.
Bloom filters serve this function by allowing an SPV node to specify a search pattern for transactions that can be tuned towards precision or privacy. A more specific bloom filter will produce accurate results, but at the expense of revealing what addresses are used in the user's wallet. A less specific bloom filter will produce more data about more transactions, many irrelevant to the node, but will allow the node to maintain better privacy.
An SPV node will initialize a bloom filter as "empty" and in that state the bloom filter will not match any patterns. The SPV node will then make a list of all the addresses in its wallet and create a search pattern matching the transaction output that corresponds to each address. Usually, the search pattern is a Pay-to-Public-Key-Hash script that is the expected locking script that will be present in any transaction paying to the public-key-hash (address). If the SPV node is tracking the balance of a P2SH address, then the search pattern will be a Pay-to-Script-Hash script, instead. The SPV node then adds each of the search patterns to the bloom filter, so that the bloom filter can recognize the search pattern if it is present in a transaction. Finally, the bloom filter is sent to the peer and the peer uses it to match transactions for transmission to the SPV node.
Bloom filters are implemented as a variable-size array of N binary digits (a bit field) and a variable number of M hash functions. The hash functions are designed to always produce an output that is between 1 and N, corresponding to the array of binary digits. The hash functions are generated deterministically, so that any node implementing a bloom filter will always use the same hash functions and get the same results for a specific input. By choosing different length (N) bloom filters and a different number (M) of hash functions, the bloom filter can be tuned, varying the level of accuracy and therefore privacy. In the example below, we use a very small array of 16 bits and a set of 3 hash functions to demonstrate how bloom filters work.
[[bloom1]]
.An example of a simplistic bloom filter, with 16 bit field and 3 hash functions
image::images/Bloom1.png["Bloom1"]
The bloom filter is initialized so that the array of bits is all zeros. To add a pattern to the bloom filter, the pattern is hashed by each hash function in turn. Applying the first hash function to the input results in a number between 1 and N. The corresponding bit in the array (indexed from 1 to N) is found and set to +1+, thereby recording the output of the hash function. Then, the next hash function is used to set another bit and so on and so forth. Once all M hash functions have been applied, the search pattern will be "recorded" in the bloom filter as M bits have been changed from +0+ to +1+.
Here's an example of adding a pattern "A" to the simple bloom filter shown above:
[[bloom2]]
.Adding a pattern "A" to our simple bloom filter
image::images/Bloom2.png["Bloom2"]
Adding a second pattern is as simple as repeating this process. The pattern is hashed by each hash function in turn and the result is recorded by setting the bits to +1+. Note that as a bloom filter is filled with more patterns, a hash function result may coincide with a bit that is already set to +1+ in which case the bit is not changed. In essence, as more patterns record on overlapping bits, the bloom filter starts to become saturated with more bits set to +1+ and the accuracy of the filter decreases. This is why the filter is a probabilistic data structure - it gets less accurate as more patterns are added. The accuracy depends on the number of patterns added versus the size of the bit array (N) and number of hash functions (M). A larger bit array and more hash functions can record more patterns with higher accuracy. A smaller bit array or fewer hash functions will record fewer patterns and produce less accuracy.
Below is an example of adding a second pattern "B" to the simple bloom filter:
[[bloom3]]
.Adding a second pattern "B" to our simple bloom filter
image::images/Bloom3.png["Bloom3"]
To test if a pattern is part of a bloom filter, the pattern is hashed by each hash function and the resulting bit pattern is tested against the bit array. If all the bits indexed by the hash functions are set to +1+, then the patten is _probably_ recorded in the bloom filter. Since the bits may be set because of overlap from multiple patterns, the answer is not certain, but is rather probabilistic. In simple terms, a bloom filter positive match is a "Maybe, Yes".
Below is an example of testing the existence of pattern "X" in the simple bloom filter. The corresponding bits are set to +1+, so the pattern is probably a match:
[[bloom4]]
.Testing the existence of pattern "X" in the bloom filter. The result is probabilistic positive match, meaning "Maybe"
image::images/Bloom4.png["Bloom4"]
On the contrary, if a pattern is tested against the bloom filter and any one of the bits is set to +0+, then this proves that the pattern was not recorded in the bloom filter. A negative result is not a probability, it is a certainty. In simple terms, a negative match on a bloom filter is a "Definitely No".
Below is an example of testing the existence of pattern "Y" in the simple bloom filter. One of the corresponding bits is set to +0+, so the pattern is definitely not a match:
[[bloom5]]
.Testing the existence of pattern "Y" in the bloom filter. The result is a definitive negative match, meaning "Definitely No"
image::images/Bloom5.png["Bloom5"]
Bitcoin's implementation of bloom filters is described in Bitcoin Improvement Proposal 37 (BIP0037). See <<bip0037>> or visit:
https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki
=== Bloom Filters and Inventory Updates
Bloom filters are used to filter the transactions (and blocks containing them) that an SPV node receives from its peers. SPV nodes will create a filter that matches only the addresses held in the SPV node's wallet. The SPV node will then send a +filterload+ message to the peer, containing the bloom filter to use on the connection. After a filter is established, the peer will then test each transaction's outputs against the bloom filter. Only transactions which match the filter are sent to the node.
In response to a +getdata+ message from the node, peers will send a +merkleblock+ message that contains only block headers for blocks matching the filter and a merkle path (See <<merkle_trees>>) for each matching transaction. The peer will also then send +tx+ messages containing the transactions matched by the filter.
The node setting the bloom filter can interactively add patterns to the filter by sending a +filteradd+ message. To clear the bloom filter, the node can send a +filterclear+ message. Since it is not possible to remove a pattern from a bloom filter, a node has to clear and re-send a new bloom filter if a pattern is no longer desired.
[[transaction_pools]] [[transaction_pools]]
== Transaction Pools === Transaction Pools
Almost every node on the bitcoin network maintains a temporary list of unconfirmed transactions called the memory pool or transaction pool. Once a transaction is verified using the detailed checklist introduced in the section above, it is added to the transaction pool. Nodes use this pool to keep track of transactions that are known to the network but are not yet included in the blockchain. For example, a node that holds a users wallet will use the transaction pool to track incoming payments to the users wallet that have been received on the network but are not yet confirmed. Every node also maintains a separate pool of orphaned transactions as detailed in <<orphan_transactions>>. If a transactions inputs refer to a transaction that is not yet known, a missing parent, then it will be stored temporarily in the orphan pool until the parent transaction arrives. Both the transaction pool and orphan pool are stored in local memory and are not saved on persistent storage, rather they are dynamically populated from incoming network messages. When a node starts, both pools are empty and are gradually populated with new transactions received on the network. Almost every node on the bitcoin network maintains a temporary list of unconfirmed transactions called the memory pool or transaction pool. Nodes use this pool to keep track of transactions that are known to the network but are not yet included in the blockchain. For example, a node that holds a users wallet will use the transaction pool to track incoming payments to the users wallet that have been received on the network but are not yet confirmed.
As transactions are received and verified using the criteria in the previous section, they are added to the transaction pool and relayed to the neighboring nodes to propagate on the network. As transactions are received and verified, they are added to the transaction pool and relayed to the neighboring nodes to propagate on the network.
Some node implementations also maintain a separate pool of orphaned transactions as detailed in <<orphan_transactions>>. If a transaction's inputs refer to a transaction that is not yet known, a missing parent, then the orphan transaction will be stored temporarily in the orphan pool until the parent transaction arrives.
When a transaction is added to the transaction pool, the orphan pool is checked for any orphans that reference this transaction's outputs (its children). Any orphans found are pulled from the orphan pool and validated using the above checklist. If valid, they are also added to the transaction pool, completing the chain that started with the parent transaction. In light of the newly added transaction which is no longer an orphan, the process is repeated recursively looking for any further descendants, until no more descendants are found. Through this process, the arrival of a parent transaction triggers a cascade reconstruction of an entire chain of interdependent transactions by re-uniting the orphans with their parents all the way down the chain. When a transaction is added to the transaction pool, the orphan pool is checked for any orphans that reference this transaction's outputs (its children). Any orphans found are pulled from the orphan pool and validated using the above checklist. If valid, they are also added to the transaction pool, completing the chain that started with the parent transaction. In light of the newly added transaction which is no longer an orphan, the process is repeated recursively looking for any further descendants, until no more descendants are found. Through this process, the arrival of a parent transaction triggers a cascade reconstruction of an entire chain of interdependent transactions by re-uniting the orphans with their parents all the way down the chain.
Most nodes also create a UTXO pool which is the set of all unspent outputs on the blockchain, this may be housed in local memory or as an indexed database table on persistent storage. Unlike the transaction and orphan pools, the UTXO pool is not initialized empty but instead contains millions of entries of unspent transaction outputs including some dating back to 2009. Whereas the transaction and orphan pools represent a nodes local perspective and may vary significantly from node to node depending upon when the node was started or restarted, the UTXO pool represents the emergent consensus of the network and therefore will vary little between nodes. Furthermore the transaction and orphan pools only contain unconfirmed transactions, while the UTXO pool only contains confirmed outputs. Both the transaction pool and orphan pool (where implemented) are stored in local memory and are not saved on persistent storage, rather they are dynamically populated from incoming network messages. When a node starts, both pools are empty and are gradually populated with new transactions received on the network.
=== Aggregating Transactions into Blocks Some implementations of the bitcoin client also maintain a UTXO database or UTXO pool which is the set of all unspent outputs on the blockchain. While the name "UTXO pool" sounds similar to the transaction pool, it represents a different set of data. Unlike the transaction and orphan pools, the UTXO pool is not initialized empty but instead contains millions of entries of unspent transaction outputs including some dating back to 2009. The UTXO pool may be housed in local memory or as an indexed database table on persistent storage.
Some of the nodes on the bitcoin network are specialized nodes called _miners_. A miner will collect, validate and relay new transactions just like any other node. Unlike other nodes, a miner will then aggregate these transactions into a _block_. The block of transactions constructed by a miner is a candidate block and becomes valid only if the miner succeeds in winning the mining competition. Each miner competes by trying billions of possible solutions to an equation based on a cryptographic hash. If they find a solution, they broadcast the candidate block for everyone to record on the blockchain. The competition difficulty is calibrated to ensure that a new block solution is found by someone every 10 minutes on average. We will look at the mining process itself in more detail in <<mining>>. For now, let's look at how miners aggregate transactions into blocks. Whereas the transaction and orphan pools represent a single node's local perspective and may vary significantly from node to node depending upon when the node was started or restarted, the UTXO pool represents the emergent consensus of the network and therefore will vary little between nodes. Furthermore the transaction and orphan pools only contain unconfirmed transactions, while the UTXO pool only contains confirmed outputs.
In Chapter 1 we introduced Jing, a computer engineering student in Shanghai China, who is a bitcoin miner. Jing earns bitcoin by running a "mining rig" which is a specialized computer-hardware system designed to mine bitcoins. Jing started mining for bitcoin in 2010, when mining was not as competitive as it is today. At that time, Jing could mine bitcoin using a desktop computer. Today, he uses a much more powerful mining system based on Application Specific Integrated Circuits (ASICs), which are specialized silicon chips designed exclusively for one application - bitcoin mining. Over time, the way Jing participates in the mining process has changed slightly, but the fundamentals remain the same. We will start by looking at how Jing mined in 2010, when things were simpler and then look at how he mines today, as bitcoin mining has become a more complex and competitive activity. === Alert Messages
In 2010, Jing mined on a desktop computer. At the time, he would run a full bitcoin node, connected to the bitcoin network. A full bitcoin node keeps a full copy of the blockchain, the list of all transactions since the first ever transaction in 2009. Jing's bitcoin node would receive transactions propagated by other nodes, just like any other node on the bitcoin network. After validating those transactions, the bitcoin software would add them to the _memory pool_, or _transaction pool_, where transactions would await until they could be included (mined) into a block. Alert messages are a seldom used function, which is nevertheless implemented in most nodes. Alert messages are bitcoin's "emergency broadcast system", a means by which the core bitcoin developers can send an emergency text message to all bitcoin nodes. This feature is implemented to allow the core developer team to notify all bitcoin users of a serious problem in the bitcoin network, such as a critical bug that requires user action. The alert system has only been used a handful of times, most notably in the Spring of 2013 when a critical database bug caused a multi-block fork to occur in the bitcoin blockchain.
Jing's bitcoin node is also listening for new blocks, propagated on the bitcoin network. As soon a Jing's bitcoin node receives a valid new block, it immediately starts the next round of competition. Receiving a new block signifies that someone else has won the previous round, meaning that Jing's system did not win that round and should abandon its current efforts and shift its resources to try and win the next round. During the previous 10 minutes, while Jing's node was searching for a solution, it was also collecting transactions. By now it has collected a few hundred transactions in the memory pool. After removing any transactions that appear in the new block recently received, Jing's memory pool is left containing unconfirmed transactions that are waiting to be recorded in a new block. Alert messages are propagated by the +alert+ message. The alert message contains several fields, including:
Jing's node immediately constructs a new candidate block, to participate in the competition. * ID - An alert identified so that duplicate alerts can be detected
* Expiration - a time after which the alert expires
* RelayUntil - a time after which the alert should not be relayed
* MinVer, MaxVer - the range of bitcoin protocol versions that this alert applies to
* subVer - The client software version that this alert applies to
* Priority - An alert priority level, currently unused
=== Structure of a Block Alerts are cryptographically signed by a public key. The corresponding private key is held by a few select members of the core development team. The digital signature ensures that fake alerts will not be propagated on the network.
A block is a container data structure that aggregates transactions for inclusion in the public ledger, the blockchain. The block is made of a header, containing metadata, followed by a long list of transactions that make up the bulk of it's size. Each node receiving this alert message will verify it, check for expiration and propagate it to all its peers, thus ensuring rapid propagation across the entire network. In addition to propagating the alert, each node may implement a user interface function to present the alert to the user. In the Bitcoin Core client, the alert is configured with the command line option +-alertnotify+, which specifies a command to run when an alert is received. The alert message is passed as a parameter to the alertnotify command. Most commonly, the alertnotify command is set to generate an email message to the administrator of the node, containing the alert message. The alert is also displayed as a pop-up dialog in the graphical user interface (bitcoin-Qt) if it is running. Other implementations of the bitcoin protocol may handle the alert in different ways. Many hardware-embedded bitcoin mining systems do not implement the alert message function, as they have no user interface. It is strongly recommended that miners running such mining systems subscribe to alerts via a mining pool operator or by running a lightweight node just for alert purposes.
[[block_structure]]
.The structure of a block
[options="header"]
|=======
|Size| Field | Description
| 4 bytes | Magic Number | A constant (0xD9B4BEF9) used to easily recognize bitcoin blocks
| 4 bytes | Block Size | The size of the block, in bytes, following this field
| 80 bytes | Block Header | Several fields form the block header (see below)
| 1-9 bytes (VarInt) | Transaction Counter | How many transactions follow
| Variable | Transactions | The transactions recorded in this block
|=======
Jing's mining node creates a candidate block by building an empty data structure and then filling it with transactions and the appropriate metadata. We'll ignore the header for now, as it is the last thing filled in by a node and concentrate instead on the transactions and how they are added to the block. Jing's node uses a selection algorithm to pick transactions from the memory pool (and the orphan pool if the parent has arrived) and adds them after the block header. The selection algorithm is detailed in the next section.
=== Adding Transactions to a Candidate Block
To construct the candidate block Jing's bitcoin node selects transactions from the memory pool, by applying a priority metric to each transaction and adding the highest priority transactions first. Transactions are prioritized based on the "age" of the UTXO that is being spent in their inputs, allowing for old and high-value inputs to be prioritized over newer and smaller inputs. Prioritized transactions can be sent without any fees, if there is enough space in the block.
The priority of a transaction is calculated as the sum of the value and age of the inputs divided by the total size of the transaction:
----
Priority = Sum (Value of input * Input Age) / Transaction Size
----
In the equation above, the value of an input is measured in the base unit, satoshis (1/100m of a bitcoin). The age of a UTXO is the number of blocks that have elapsed since the UTXO was recorded on the blockchain, measuring how many blocks "deep" into the blockchain it is. The size of the transaction is measured in bytes.
For a transaction to be considered "high priority" its priority must be greater than 57,600,000, which corresponds to one bitcoin (100m satoshis), aged one day (144 blocks) in a transaction of 250 bytes total size.
----
High Priority = 100,000,000 satoshis * 144 blocks / 250 bytes = 57,600,000
----
The first 50 kilobytes of transaction space in a block are set aside for high priority transactions. Jing's node will fill the first 50 kilobytes, prioritizing the highest priority transactions first, regardless of fee. This allows high-priority transactions to be processed even if they carry zero fees.
Jing's mining node then fills the rest of the block up to the maximum block size (MAX_BLOCK_SIZE in the code), with transactions that carry at least the minimum fee, prioritizing those with the highest fee per kilobyte of transaction.
If there is any space remaining in the block, Jing's mining node may choose to fill it with no-fee transactions. Some miners choose to mine transactions without fees on a best-effort basis. Other miners may choose to ignore transactions without fees.
Any transactions left in the memory pool after the block is filled will remain in the pool for inclusion in the next block. As transactions remain in the memory pool, their inputs "age", as the UTXO they spend get deeper into the blockchain with new blocks added on top. Since a transactions priority depends on the age of its inputs, transactions remaining in the pool will age and therefore increase in priority. Eventually a transaction without fees may reach a high enough priority to be included in the block for free.
Bitcoin transactions do not have an expiration time-out. A transaction that is valid now will be valid in perpetuity. However, if a transaction is only propagated across the network once it will persist only as long as it is held in a mining node memory pool. When a mining node is restarted, its memory pool is wiped clear, as it is a transient non-persistent form of storage. While a valid transaction may have been propagated across the network, if it is not executed it may eventually not reside in the memory pool of any miner. Wallet software is expected to retransmit such transactions or reconstruct them with higher fees if they are not successfully executed within a reasonable amount of time.
=== Block Header
The block header consists of three sets of block metadata. First, there is a reference to a previous block hash, which connects this block to the previous block in the blockchain. We will examine this in more detail in <<blockchain>>. The second set of metadata, namely the difficulty, timestamp and nonce, relate to the mining competition, as detailed in <<mining>>. The third piece of metadata is the Merkle Tree root, a data structure used to efficiently summarize all the transactions in the block. Merkle Trees are discussed in the next section.
Jing's node will next assemble the metadata and fill in the candidate block's header.
[[block_structure]]
.The structure of the block header
[options="header"]
|=======
|Size| Field | Description
| 4 bytes | Version | A version number to track software/protocol upgrades
| 32 bytes | Previous Block Hash | A reference to the hash of the previous (parent) block in the chain
| 32 bytes | Merkle Root | A hash of the root of the Merkle-Tree of this block's transactions
| 4 bytes | Timestamp | The approximate creation time of this block (seconds from Unix Epoch)
| 4 bytes | Difficulty Target | The proof-of-work algorithm difficulty target for this block
| 4 bytes | Nonce | A counter used for the proof-of-work algorithm
|=======
=== Merkle Trees
A _Merkle Tree_, also known as a _Binary Hash Tree_ is a data structure created by Ralph Merkle used for efficiently summarizing and verifying the integrity of large sets of data. Merkle Trees are binary trees containing cryptographic hashes. When N data elements are hashed and summarized in a Merkle Tree, you can check to see if any one data element is included in the tree with at most +2*log~2~(N)+ calculations, making this a very efficient data structure. The term "tree" is used in computer science to describe a branching data structure, but trees are usually displayed upside down with the "root" at the top and the "leaves" at the bottom of a diagram, as you will see in the examples that follow.
Merkle trees are used in bitcoin to summarize all the transactions in a block, producing an overall digital fingerprint of the entire set of transactions, which can be used to prove that a transaction is included in the set. A merkle tree is constructed by recursively hashing pairs of nodes until there is only one hash, called the _root_, or _merkle root_. The cryptographic hash algorithm used in bitcoin's merkle trees is SHA256 applied twice, also known as double-SHA256.
The merkle tree is constructed bottom-up. In the example below, we start with four transactions A, B, C and D, which form the _leaves_ of the Merkle Tree, shown in the diagram at the bottom. The transactions are not stored in the merkle tree, rather their data is hashed and the resulting hash is stored in each leaf node as H~A~, H~B~, H~C~ and H~D~:
+H~A~ = SHA256(SHA256(Transaction A))+
Consecutive pairs of leaf nodes are then summarized in a parent node, by concatenating the two hashes and hashing them together. For example, to construct the parent node H~AB~, the two 32-byte hashes of the children are concatenated to create a 64-byte string. That string is then double-hashed to produce the parent node's hash:
+H~AB~ = SHA256(SHA256(H~A~ + H~B~))+
The process continues until there is only one node at the top, the node known as the Merkle Root. That 32-byte hash is stored in the block header and summarizes all the data in all four transactions.
[[simple_merkle]]
.Calculating the nodes in a Merkle Tree
image::images/MerkleTree.png["merkle_tree"]
Since the merkle tree is a binary tree, it needs an even number of leaf nodes. If there is an odd number of transactions to summarize, the last transaction hash is duplicated to create an even number of leaf nodes, also known as a _balanced tree_. This is shown in the example below, where transaction C is duplicated:
[[merkle_tree_odd]]
.An even number of data elements, by duplicating one data element
image::images/MerkleTreeOdd.png["merkle_tree_odd"]
The same method for constructing a tree from four transactions can be generalized to construct trees of any size. In bitcoin it is common to have several hundred to more than a thousand transactions in a single block, which are summarized in exactly the same way producing just 32-bytes of data from a single merkle root. In the diagram below, you will see a tree built from 16 transactions:
[[merkle_tree_large]]
.A Merkle Tree summarizing many data elements
image::images/MerkleTreeLarge.png["merkle_tree_large"]
To prove that a specific transaction is included in a block, a node need only produce +log~2~(N)+ 32-byte hashes, constituting an _authentication path_ or _merkle path_ connecting the specific transaction to the root of the tree. This is especially important as the number of transactions increases, because the base-2 logarithm of the number of transactions increases much more slowly. This allows bitcoin nodes to efficiently produce paths of ten or twelve hashes (320-384 bytes) which can provide proof of a single transaction out of more than a thousand transactions in a megabyte sized block. In the example below, a node can prove that a transaction K is included in the block by producing a merkle path that is only four 32-byte hashes long (128 bytes total). The path consists of the four hashes H~L~, H~IJ~, H~MNOP~ and H~ABCDEFGH~. With those four hashes provided as an authentication path, any node can prove that H~K~ is included in the merkle root by computing four additional pair-wise hashes H~KL~, H~IJKL~ and H~IJKLMNOP~ that lead to the merkle root.
[[merkle_tree_path]]
.A Merkle Path used to prove inclusion of a data element
image::images/MerkleTreePathToK.png["merkle_tree_path"]
The efficiency of merkle trees becomes obvious as the scale increases. For example, proving that a transaction is part of a block requires:
[[block_structure]]
.Merkle Tree Efficiency
[options="header"]
|=======
|Number of Transactions| Path Size (Hashes) | Path Size (Bytes)
| 16 transactions | 4 hashes | 128 bytes
| 512 transactions | 9 hashes | 288 bytes
| 2048 transactions | 11 hashes | 352 bytes
| 65,535 transactions | 16 hashes | 512 bytes
|=======
[[blockchain]]
=== The Blockchain
Bitcoin's core innovation is the _blockchain_, a distributed, timestamped ledger. The ledger consists of a cryptographically verified chain of _blocks_ in historical order, each of which contains transactions, new coins and a signature (hash) of the previous block. Each full bitcoin node in the network keeps a complete local replica of the blockchain, and independently verify all transactions and balances from that replica.
In this chapter we will look at the structure of the blockchain and the blocks within it. We will examine various mechanisms for querying the blockchain database to retrieve information about blocks and transactions. Finally, we will see how new blocks are added to the blockchain through the mechanism of distributed consensus based on a Proof-of-Work algorithm, known as "mining".
The blockchain data structure is an ordered linked list of blocks of transactions. The blockchain can be stored as a flat file, or in a simple database. The bitcoin core client stores the blockchain metadata using Google's LevelDB database.
The blockchain is made of blocks, which are generated {not are generated - maybe are usually computed / created / produced / found / mined} every ten minutes on average and hold several hundred transactions that occurred during the previous ten minutes. {is this true? is it always those that occurred during the previous 10 min?} Each block is identified by a hash, generated using the SHA256 cryptographic hash algorithm on the header of the block. {who generates this hash?} Each block also {references the previous block's hash - thereby linking the two together} contains a link to the previous block in the chain, by reference to the previous block's hash. Since a block contains a cryptographic hash of the previous block as a reference, the hash proves that the previous block was known when the current block was created. The hash of the previous block is also part of the data that creates the hash of the current block, making the ancestry of each block an immutable part of its identity. The chain of hashes guarantees that a block cannot be modified retrospectively without forcing the re-computation of all following blocks, because a retrospective change in any block would change the hash, thereby changing the reference in the next block whose hash also changes, and so on. As new blocks are added to the chain, they strengthen the immutability of the ledger by effectively incorporating all previous blocks by reference in their cryptographic hash.
Since each block can only reference one previous block, every chain of blocks can be traced back to the first block every created, the _genesis block_.
=== The Genesis Block
The first block in the chain is called the _genesis block_ and was created in 2009. It is the "common ancestor" of all the blocks in the blockchain, meaning that if you start at any block and follow the chain backwards in time you will eventually arrive at the _genesis block_.
The genesis block has the identifier hash +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+. You can search for that block hash in any block explorer website, such as blockchain.info, and you will find a page describing the contents of this block, with a URL containing that hash:
https://blockchain.info/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
https://blockexplorer.com/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
Using the Bitcoin Core reference client on the command-line:
----
$ bitcoind getblock 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
{
"hash" : "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
"confirmations" : 308321,
"size" : 285,
"height" : 0,
"version" : 1,
"merkleroot" : "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
"tx" : [
"4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b"
],
"time" : 1231006505,
"nonce" : 2083236893,
"bits" : "1d00ffff",
"difficulty" : 1.00000000,
"nextblockhash" : "00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048"
}
----
=== Linking Blocks in a Chain
[[chain_of_blocks]]
.Blocks linked in a chain, by reference to the previous block header hash
image::images/ChainOfBlocks.png["chain_of_blocks"]
[[mining]]
=== Proof-of-Work (Mining) and Consensus
((("Mining", "Proof of Work", "SHA256", "hashing power", "difficulty", "nonce")))
Mining is the process by which new bitcoin is added to the money supply. Mining also serves to secure the bitcoin system against fraudulent transactions or transactions spending the same amount of bitcoin more than once, known as a double-spend. Miners act as a decentralized clearinghouse, validating new transactions and recording them on the global ledger. A new block, containing transactions which occurred since the last block, is "mined" every 10 minutes thereby adding those transactions to the blockchain. Transactions that become part of a block and added to the blockchain are considered "confirmed", which allows the new owners of bitcoin to spend the bitcoin they received in those transactions. Miners receive two types of reward for mining: new coins created with each new block and transaction fees from all the transactions included in the block. To earn this reward, the miners compete to solve a difficult mathematical problem based on a cryptographic hash algorithm. The solution to the problem, called the Proof-of-Work, is included in the new block and acts as proof that the miner expended significant computing effort. The competition to solve the Proof-of-Work algorithm to earn reward and the right to record transactions on the blockchain is the basis for bitcoin's security model.
Bitcoin's security is underpinned by computation. New blocks are added to the blockchain through a consensus mechanism called the _Proof-of-Work_ (PoW) that requires a predictable computational effort, one that takes approximately 10 minutes to solve on average. Specialized bitcoin nodes called _miners_ validate transactions and collect them into blocks, then attempt to find the solution that satisfies the Proof-of-Work algorithm. The first miner to find such a solution, propagates the newly created block across the network. All other nodes on the network verify that the new block contains valid transactions and satisfies the Proof-of-Work algorithm, then they add it to the blockchain, thereby extending it by one block. The miners add a special coin generation transaction into the blocks they build, which creates new bitcoin from nothing and is payable to the miner's own bitcoin address. Once the block is accepted as valid by the entire network, that transaction is also recorded on the blockchain, thereby rewarding the miner for the computational effort it took to satisfy the Proof-of-Work. This de-centralized consensus mechanism, based on a global competition and requiring computation to create new blocks, is the basis for the security of the bitcoin transaction ledger and also for the issuance of new bitcoin. {move the last sentence to the beginning of the paragraph - to explain more about security} The equilibrium between the incentive of bitcoin reward and the immense computing effort required to win it force the participants to behave honestly, without the need for a centralized clearinghouse or currency issuer. The bitcoin consensus mechanism is a dynamic, self-regulating and completely decentralized security model that operates at very large scale.
The process of new coin generation is called mining, because the reward is designed to simulate diminishing returns, just like mining for precious metals. Bitcoin's money supply is created through mining, just like a central bank issues new money by printing bank notes. The amount of newly created bitcoin a miner can add to a block decreases approximately every four years (or precisely every 210,000 blocks). It started at 50 bitcoin per block in January of 2009 and halved to 25 bitcoin per block in November of 2012. It will halve again to 12.5 bitcoin per block sometime in 2016. Based on this formula, bitcoin mining rewards decrease exponentially until approximately the year 2140 when all 21 million bitcoin have been issued.
Bitcoin miners also earn fees from transactions. Every transaction may include a transaction fee, in the form of a surplus of bitcoin between the transaction's inputs and outputs. The bitcoin miner gets to "keep the change" on the transactions.
Today the fees represent 1% or less of a bitcoin miner's income, the vast majority coming from the newly minted bitcoins. However, as the reward decreases over time and the number of transactions per block increases, a greater proportion of bitcoin mining earnings will come from fees. After 2140 all bitcoin miner earnings will be in the form of transaction fees.
[[figure_sha256_logical]]
.The Secure Hash Algorithm (SHA-256)
image::images/sha256-logical.png["SHA256"]
With SHA-256, the output is always 256 bits long, regardless of the size of the input. In the example below, we will use the Python interpreter to calculate the SHA256 hash of the phrase "I am Satoshi Nakamoto".
[[sha256_example1]]
.SHA256 Example
----
$ *python*
Python 2.7.1
>>> import hashlib
>>> print hashlib.sha256("I am Satoshi Nakamoto").hexdigest()
5d7c7ba21cbbcd75d14800b100252d5b428e5b1213d27c385bc141ca6b47989e
----
The example shows that if we calculate the hash of the phrase +"I am Satoshi Nakamoto"+, it will produce +5d7c7ba21cbbcd75d14800b100252d5b428e5b1213d27c385bc141ca6b47989e+. This 256-bit number is the _hash_ or _digest_ of the phrase and depends on every part of the phrase. Adding a single letter, punctuation mark or any character will produce a different hash.
Now, if we vary the phrase, we will expect to see completely different hashes. Let's try that by adding a number to the end of our phrase, using this simple Python script
[[sha256_example_generator]]
.SHA256 A script for generating many hashes by iterating on a nonce
====
[source, python]
----
include::code/hash_example.py[]
----
====
Running this will produce the hashes of several phrases, made different by adding a unique number, called a _nonce_ at the end of the text. By incrementing the nonce, we can get different hashes.
((("nonce")))
[[sha256_example_generator_output]]
.SHA256 Output of a script for generating many hashes by iterating on a nonce
----
$ *python hash_example.py*
I am Satoshi Nakamoto0 => a80a81401765c8eddee25df36728d732...
I am Satoshi Nakamoto1 => f7bc9a6304a4647bb41241a677b5345f...
I am Satoshi Nakamoto2 => ea758a8134b115298a1583ffb80ae629...
I am Satoshi Nakamoto3 => bfa9779618ff072c903d773de30c99bd...
I am Satoshi Nakamoto4 => bce8564de9a83c18c31944a66bde992f...
I am Satoshi Nakamoto5 => eb362c3cf3479be0a97a20163589038e...
I am Satoshi Nakamoto6 => 4a2fd48e3be420d0d28e202360cfbaba...
I am Satoshi Nakamoto7 => 790b5a1349a5f2b909bf74d0d166b17a...
I am Satoshi Nakamoto8 => 702c45e5b15aa54b625d68dd947f1597...
I am Satoshi Nakamoto9 => 7007cf7dd40f5e933cd89fff5b791ff0...
I am Satoshi Nakamoto10 => c2f38c81992f4614206a21537bd634a...
I am Satoshi Nakamoto11 => 7045da6ed8a914690f087690e1e8d66...
I am Satoshi Nakamoto12 => 60f01db30c1a0d4cbce2b4b22e88b9b...
I am Satoshi Nakamoto13 => 0ebc56d59a34f5082aaef3d66b37a66...
I am Satoshi Nakamoto14 => 27ead1ca85da66981fd9da01a8c6816...
I am Satoshi Nakamoto15 => 394809fb809c5f83ce97ab554a2812c...
I am Satoshi Nakamoto16 => 8fa4992219df33f50834465d3047429...
I am Satoshi Nakamoto17 => dca9b8b4f8d8e1521fa4eaa46f4f0cd...
I am Satoshi Nakamoto18 => 9989a401b2a3a318b01e9ca9a22b0f3...
I am Satoshi Nakamoto19 => cda56022ecb5b67b2bc93a2d764e75f...
----
Each phrase produces a completely different hash result. They seem completely random, but you can re-produce the exact results in this example on any computer with Python and see the same exact hashes.
To make a challenge out of this algorithm, let's set an arbitrary target: find a phrase starting with "I am Satoshi Nakamoto" which produces a hash that starts with a zero. In numerical terms, that means finding a hash value that is less than +0x1000000000000000000000000000000000000000000000000000000000000000+. Fortunately, this isn't so difficult! If you notice above, we can see that the phrase "I am Satoshi Nakamoto13" produces the hash 0ebc56d59a34f5082aaef3d66b37a661696c2b618e62432727216ba9531041a5, which fits our criteria. It only took 13 attempts to find it.
==== Proof-of-Work Algorithm
Bitcoin's proof-of-work is very similar to the problem above. First, a miner will generate a new block, containing:
((("block")))
* Transactions waiting to be included in a block
* The hash from the previous block
* A _nonce_
The only part a miner can modify is the nonce. Now, the miner will calculate the hash of this block's header and see if it is smaller than the current _target difficulty_. The miner will likely have to try many nonces before finding one that results in a low enough hash.
A very simplified proof-of-work algorithm is implemented in Python here:
((("proof of work")))
[[pow_example1]]
.Simplified Proof-Of-Work Implementation
====
[source, python]
----
include::code/proof-of-work-example.py[]
----
====
Running the code above, you can set the desired difficulty (in bits, how many of the leading bits must be zero) and see how long it takes for your computer to find a solution. In the following examples, you can see how it works on an average laptop:
[[pow_example_outputs]]
.Running the proof-of-work example for various difficulties
----
$ *python proof-of-work-example.py*
Difficulty: 1 (0 bits)
[...]
Difficulty: 8 (3 bits)
Starting search...
Success with nonce 9
Hash is 1c1c105e65b47142f028a8f93ddf3dabb9260491bc64474738133ce5256cb3c1
Elapsed Time: 0.0004 seconds
Hashing Power: 25065 hashes per second
Difficulty: 16 (4 bits)
Starting search...
Success with nonce 25
Hash is 0f7becfd3bcd1a82e06663c97176add89e7cae0268de46f94e7e11bc3863e148
Elapsed Time: 0.0005 seconds
Hashing Power: 52507 hashes per second
Difficulty: 32 (5 bits)
Starting search...
Success with nonce 36
Hash is 029ae6e5004302a120630adcbb808452346ab1cf0b94c5189ba8bac1d47e7903
Elapsed Time: 0.0006 seconds
Hashing Power: 58164 hashes per second
[...]
Difficulty: 4194304 (22 bits)
Starting search...
Success with nonce 1759164
Hash is 0000008bb8f0e731f0496b8e530da984e85fb3cd2bd81882fe8ba3610b6cefc3
Elapsed Time: 13.3201 seconds
Hashing Power: 132068 hashes per second
Difficulty: 8388608 (23 bits)
Starting search...
Success with nonce 14214729
Hash is 000001408cf12dbd20fcba6372a223e098d58786c6ff93488a9f74f5df4df0a3
Elapsed Time: 110.1507 seconds
Hashing Power: 129048 hashes per second
Difficulty: 16777216 (24 bits)
Starting search...
Success with nonce 24586379
Hash is 0000002c3d6b370fccd699708d1b7cb4a94388595171366b944d68b2acce8b95
Elapsed Time: 195.2991 seconds
Hashing Power: 125890 hashes per second
[...]
Difficulty: 67108864 (26 bits)
Starting search...
Success with nonce 84561291
Hash is 0000001f0ea21e676b6dde5ad429b9d131a9f2b000802ab2f169cbca22b1e21a
Elapsed Time: 665.0949 seconds
Hashing Power: 127141 hashes per second
----
As you can see, increasing the difficulty by 1 bit causes an exponential increase in the time it takes to find a solution. If you think of the entire 256-bit number space, each time you constrain one more bit to zero, you decrease the search space by half. In the example above, it takes 84 million hash attempts to find a nonce that produces a hash with 26 leading bits as zero. Even at a speed of more than 120 thousand hashes per second, it still requires ten minutes on a consumer laptop to find this solution.
At the time of writing this, the network is attempting to find a block whose header hash is less than +000000000000004c296e6376db3a241271f43fd3f5de7ba18986e517a243baa7+. As you can see, there are a lot of zeroes at the beginning of that hash, meaning that the acceptable range of hashes is much smaller, hence more difficult to find a valid hash. It will take on average more 150 quadrillion hash calculations per second for the network to discover the next block. That seems like an impossible task, but fortunately the network is bringing 500 TH/sec of processing power to bear, which will be able to find a block in about 10 minutes on average.
==== Difficulty Target and Re-Targetting
Bitcoin is tuned to generate blocks approximately every 10 minutes. This is achieved by automatically adjusting the target difficulty to account for increases and decreases in the available computing power on the network. This process occurs automatically and on every full node independently. Each node recalculates the expected difficulty every 2106 blocks, based on the time it took to hash the previous 2106 blocks. In simple terms: If the network is finding blocks faster than every 10 minutes, the difficulty increases. If block discovery is slower than expected, the difficulty will decrease.
{miners that are on mining pools get the difficulty (do not calculate difficulty independently) they are given the difficulty from the mining pool so they don't have to calculate the difficulty themselves and they are actually given a lower difficulty target. There are essentially two classifications of miners today - pool miners and solo miners. Solo miners run a full node and compete on their own. Whereas pool miners collaborate with one another and compete against the network as a team, while sharing the reward. The reason miners join pools - solo miners need an enormous amount of hashing power in order to have even the slimmest chance of finding a solution to a block which will make their earnings erratic. By participating in a pool, miners get smaller shares but a more regular share of rewards, reducing uncertainty. Solo mining is becoming obsolete, as the difficulty increases the likelihood of a solo miner finding a solution is more like winning the lottery.}
{ASIC miners do not run full nodes. Full nodes independently calculate the difficulty using the same equation on the same block, arriving at the same result for the new difficulty. Retargeting the difficulty at block heights that are multiples of 2106 from the genesis block. The equation for retargeting difficulty measures the time it took to find the last 2106 blocks, compares that to the expected time of 21,060 minutes (based upon a desired 10 minute block time), the difference is calculated as a percentage and a corresponding percentage adjustment is made to the difficulty. To avoid extreme volatility in the difficulty, the retargeting adjustment cannot exceed {X%} per retargeting. The difficulty will only be retargeted up or down by maximum of {X%} per cycle. If the required difficulty adjustment is greater than the maximum it will be reflected in the next retargeting adjustment as the imbalance will persist through the next 2106 blocks. Large discrepancies between hashing power and difficulty may take several cycles to even out. This leads to a potential problem which has been observed in alt coins, where very large changes in difficulty can cause hashing power to collapse leading to excessively long block times. If the aggregate network hashing power collapses due to the departure of many miners simultaneously, the remaining hashing power may be insufficient to meet the difficulty target leading to excessively long block intervals. Since retargeting is not a function of time but rather block number, a large hashing deficit can mean the next cycle is very far in the future. Usually this is caused for two reasons - scenario one - entry for a brief period of a lot of hashing which temporarily increases the difficulty, followed by the departure of that hashing, resulting in a collapse of block solutions. Essentially a hashing pump and dump. Usually a deliberate attack. This is not a concern in bitcoin because new hashing power introduced into the network will not effect the average enough to cause a major change in difficulty. The other scenario in which hashing power can collapse is a crash in bitcoin price, making mining unprofitable. (If the miner cannot pay their electricity bill, the miner will leave the network.) This is a weakness of the protocol, as an insurmountable hashing deficit could occur with a precipitous collapse in price and corresponding reduction in available hashing power. The network would be unable to recover because ... }
[TIP]
====
The difficulty of finding a bitcoin block is approximately '10 minutes of processing' for the entire network, based on the time it took to find the previous 2106 blocks, adjusted every 2106 blocks.
====
Note that the target difficulty is independent of the number of transactions or the value of transactions. This means that the amount of hashing power and therefore electricity expended to secure bitcoin is also entirely independent of the number of transactions. Bitcoin can scale up, achieve broader adoption and remain secure without any increase in hashing power from today's level. The increase in hashing power represents market forces as new miners enter the market to compete for the reward. As long as enough hashing power is under the control of miners acting honestly in pursuit of the reward, it is enough to prevent "takeover" attacks and therefore it is enough to secure bitcoin.
The target difficulty is closely related to the cost of electricity and the exchange rate of bitcoin vis-a-vis the currency used to pay for electricity. High performance mining systems are about as efficient as possible with the current generation of silicon fabrication, converting electricity into hashing computation at the highest rate possible. The primary influence on the mining market is the price of one kilowatt-hour in bitcoin, as that determines the profitability of mining and therefore the incentives to enter or exit the mining market.
==== Mining New Bitcoins
Bitcoins are "minted" during the creation of each block at a fixed and diminishing rate. Each block, generated on average every 10 minutes, contains a _reward_ that consists of entirely new bitcoins. The reward was 50BTC for the first four years of operation of the network. Every four years the reward is decreased by 50%, resulting in a diminishing rate of issuance over time. In 2012, the reward was decreased to 25BTC and it will decrease again to 12.5BTC in 2016. By approximately 2140, the last fragments of a bitcoin will be mined, for a total of 21 million bitcoins. {Clarify coinbase transaction as first - includes the reward and transactions. Discuss how the coinbase transaction will change in 2140}
The finite and diminishing issuance creates a fixed monetary supply that resists inflation. Unlike a fiat currency which can be printed in infinite numbers by a central bank, bitcoin can never be inflated by printing.
===== Monetary supply
Bitcoin's monetary supply is defined as the number of coins in circulation (minted). Like any other currency, this measure of monetary supply is called M0, which represents the narrowest measure of the money supply. Just like any other currency, bitcoin can also have a _fractional reserve banking_ which means that an organization can trade bitcoins "off blockchain" which are not part of the M0 monetary measure, but of the broader monetary supply measures M1-M3. {have you explained M1-M3?}{also, i think you should explain fractional reserve banking a bit here}
While the total bitcoins in circulation will not exceed 21m, that monetary base can support a much broader economy through fractional reserve banking and expansion of the available credit.
===== Deflationary Money
The most important and debated consequence of a fixed and diminishing monetary issuance is that the currency will tend to be inherently _deflationary_. Deflation is the phenomenon of appreciation of value due to a mismatch in supply and demand that drives up the value (and exchange rate) of a currency. The opposite of inflation, price deflation means that your money has more purchasing power over time.
Many economists argue that a deflationary economy is a disaster that should be avoided at all costs. That is because in a period of rapid deflation, the incentives for regular people are to hoard the money and not spend it, hoping that prices will fall. Such a phenomenon unfolded during Japan's "Lost Decade", when a complete collapse of demand pushed the currency into a deflationary spiral.
Bitcoin experts argue that deflation is not bad *per se*. Rather, we associate deflation with a collapse in demand because that is the only example of deflation we have to study. In a fiat currency with the possibility of unlimited printing, it is very difficult to enter a deflationary spiral unless there is a complete collapse in demand and an unwillingness to print money. Deflation in bitcoin is not caused by a collapse in demand, but by predictably constrained supply.
In practice, it has become evident that the hoarding instinct caused by a deflationary currency can be overcome by discounting from vendors, until the discount overcomes the hoarding instinct of the buyer. Since the seller is also motivated to hoard, the discount becomes the equilibrium price at which the two hoarding instincts are matched. With discounts of 30% on the bitcoin price, most bitcoin retailers are not experiencing difficulty overcoming the hoarding instinct and generating revenue. It remains to be seen whether the deflationary aspect of the currency is really a problem when it is not driven by rapid economic retraction.
==== Blockchain Forks
{Discuss chain selection: As new blocks are found they are added to the chain. Each full node constructs a chain and calculates the cumulative difficulty of that chain. As blocks are constructed and propagated across the network,}
{create a graphic showing propagating transaction}
{Because the blockchain is a decentralized data structure, different copies of it are not always consistent. Blocks may arrive at different nodes at different times, causing them to have a different perspective o ft the blockchain. To resolve this, each node always selects and attempts to extend the chain of blocks that represents the most Proof-of-Work, also known as the longest chain or greatest cumulative difficulty chain, by adding the difficulty recorded in each block for a chain a node can calculate the total amount of PoW that has been expended to create that chain. As long as all nodes select the longest, i.e. the longest cumulative difficulty chain, the global bitcoin network eventually converges to a consistent state. Forks occur as temporary inconsistencies between versions of the blockchain, which are resolved by the eventual reconvergence.}
{Bitcoin's _consensus mechanism_, which creates the is comprised of the independent validation of transactions by every node, the cumulative work of the miners, and the network convergence upon the greatest difficulty chain. The interplay of these three processes manifests the emergent property of consensus that allows for a global decentralized public ledger without a central authority. which creates one global public ledger, emerges as a property of (1) the selection of the greatest difficulty chain. This chapter is about the emergent property of consensus. This consensus is created by the interplay of three processes - (1) ,2,3. The emergent property of network-wide consensus is what establishes a trusted decentralized global public ledger. Satohsi's invention was not proof of work, elliptic curve cryptography. Satoshi's invention was how the interplay of these processes creates emergent consensus in a decentralized network without the need for a centralized trusted authority.}
A "fork" occurs whenever there are two candidate blocks competing to form the longest blockchain. This occurs under normal conditions whenever two miners solve the Proof-of-Work algorithm within a short period of time from each other. As both miners discover a solution for their respective candidate blocks, they immediately broadcast their own "winning" block to their immediate neighbors who begin propagating the block across the network. Each node that receives a valid block will incorporate it into their blockchain, extending the blockchain by one block. If that node later sees another candidate block extending the same parent, they ignore the second candidate. As a result, some nodes will "see" one candidate block first, while other nodes will see the other candidate block and two competing versions of the blockchain will emerge.
{create a graphic with the globe, two miners each - bitcoin topology map}
Let's assume for example that a miner in Canada finds a proof-of-work solution for block "A" that extends the blockchain from height 315000 to height 315001, building on top of parent block "P". Almost simultaneously, an Australian miner who was also extending block "P", finds a solution for block "B", their candidate block. Now, there are two possible candidates for block height 315001, one we call "A", originating in Canada and one we call "B", originating in Australia. Both blocks are valid, both blocks contain a valid solution to the proof of work, both blocks extend the same parent. Both blocks likely contain most of the same transactions, with only perhaps a few differences in the order of transactions. From that moment, the bitcoin network nodes closest (topologically, not geographically) to the Canadian node will hear about block "A" first and will create a new greatest-cumulative-difficulty blockchain with height 315001 and "A" as the last block in the chain (e.g. P-A), ignoring the candidate block "B" that arrives a bit later. Meanwhile, nodes closer to the Australian node will take that block as the winner and extend the blockchain to height 315001 with "B" as the last block (e.g. P-B), ignoring "A" when it arrives a few seconds later. Any miners that saw "A" first will immediately build candidate blocks that reference "A" as the parent and start trying to solve the PoW for these candidate blocks. The miners that accepted "B" instead, will start extending that chain.
----
Block "A" extends the chain: P-A
Block "B" also extends the chain: P-B
----
Forks are almost always resolved within one block. As part of the network's hashing power is dedicated to building on top of "A" as the parent, another part of the hashing power is focused on building on top of "B". Even if the hashing power is almost evenly split, it is likely that one set of miners will find a solution and propagate it before the other set of miners have found any solutions. Let's say for example that the miners building on top of "B" find a new block "X" that extends the chain to height 315002 (e.g. P-B-X). They immediately propagate this new block and the entire network sees it as a valid solution. All nodes that had chosen "B" as the winner in the previous round will simply extend the chain one more block. The nodes that chose "A" as the winner, however, will now see a block extending an even longer chain (greater-cumulative difficulty), that does not include "A" in it. Any miners working on extending the chain P-A will now stop that work because their candidate block is an "orphan", as its parent "A" is no longer on the longest chain. The block "A" is removed from the blockchain by any nodes that had accepted it and any transactions within it are queued up again for processing in the next block. The entire network re-converges on a single blockchain P-B-X, with "X" as the last block in the chain. All miners immediately start working on candidate blocks that reference "X" as their parent to extend the P-B-X chain.
It is theoretically possible for a fork to extend to two blocks, if two blocks are found almost simultaneously by miners on opposite "sides" of a previous fork. However, the chance of that happening is very low. Whereas a one-block fork may occur every week, a two-block fork is exceedingly rare.
----
Block "X" extends the chain: P-B-X
Old chain is now "shorter": P-A
----
[TIP]
====
As of version 0.9, Bitcoin Core's +alertnotify+ option will send alerts whenever a 6-block or longer fork occurs
====
[[chainforks]]
.A blockchain showing two instances of forking
image::images/BlockChainWithForks.png["chainforks"]
==== Highest Difficulty Chain Selection
==== Competition and Coinbase
==== Mining Pools
===== Managed Pools
===== P2Pool
==== Mining Economics
==== Consensus Attacks
===== 51% Attack
===== Selfish Mining Attack
==== Normal Forks
==== Soft Forks
==== Hard Forks
==== Unusual Forks

204
ch07.asciidoc Normal file
View File

@ -0,0 +1,204 @@
[[ch7]]
== Chapter 7 - The Blockchain
*DRAFT - DO NOT SUBMIT ISSUES OR PULL REQUESTS YET PLEASE - CONSTANT CHANGES HAPPENING*
[[blockchain]]
=== The Blockchain
{Now that the Jing's node has built a candidate block and populated the transactions and merkle root, this block must be linked to the other blocks in the blockchain {must be ready to link to other blocks (if he wins the competition)}. In this section we will look at the blockchain data structure in more detail and see how Jing will extend this chain with the new block. MOVE TO 8?}
The blockchain data structure is an ordered linked list of blocks of transactions. The blockchain can be stored as a flat file, or in a simple database. The bitcoin core client stores the blockchain metadata using Google's LevelDB database.
Each block within the blockchain is identified by a hash, generated using the SHA256 cryptographic hash algorithm on the header of the block. Each block also references a previous block, known as the _parent_ block, through the "previous block hash" field in the block header. In other words, each block contains the hash of its parent inside its own header. The sequence of hashes linking each block to its parent, creates a chain going back all the way to the first block ever created, known as the _genesis block_. A block can have multiple children, each of which refers to it as its parent and contains the same hash in the "previous block hash" field. However, since a block only hash one single "previous block hash", that means each block can only have one parent. {one of which will win out while the others die...}
The hash of the parent is also part of the data (the block header) that creates the hash of the child, making the ancestry of each block an immutable part of its identity. The chain of hashes guarantees that a block cannot be modified retrospectively without forcing the re-computation of all following blocks (children), because a retrospective change in any block would change its hash, thereby changing the reference in any children whose hash also changes, and so on. As new blocks are added to the chain, they strengthen the immutability of the ledger by effectively incorporating all previous blocks by reference in their cryptographic hash.
=== Structure of a Block
A block is a container data structure that aggregates transactions for inclusion in the public ledger, the blockchain. The block is made of a header, containing metadata, followed by a long list of transactions that make up the bulk of it's size.
[[block_structure]]
.The structure of a block
[options="header"]
|=======
|Size| Field | Description
| 4 bytes | Magic Number | A constant (0xD9B4BEF9) used to easily recognize bitcoin blocks
| 4 bytes | Block Size | The size of the block, in bytes, following this field
| 80 bytes | Block Header | Several fields form the block header (see below)
| 1-9 bytes (VarInt) | Transaction Counter | How many transactions follow
| Variable | Transactions | The transactions recorded in this block
|=======
Jing's mining node creates a candidate block by building an empty data structure and then filling it with transactions and the appropriate metadata. We'll ignore the header for now, as it is the last thing filled in by a node and concentrate instead on the transactions and how they are added to the block. Jing's node uses a selection algorithm to pick transactions from the memory pool (and the orphan pool if the parent has arrived) and adds them after the block header. The selection algorithm is detailed in the next section.
[[block_header]]
=== Block Header
The block header consists of three sets of block metadata. First, there is a reference to a previous block hash, which connects this block to the previous block in the blockchain. We will examine this in more detail in <<blockchain>>. The second set of metadata, namely the difficulty, timestamp and nonce, relate to the mining competition, as detailed in <<mining>>. The third piece of metadata is the Merkle Tree root, a data structure used to efficiently summarize all the transactions in the block.
Jing's node will next assemble the metadata and fill in the candidate block's header.
[[block_structure]]
.The structure of the block header
[options="header"]
|=======
|Size| Field | Description
| 4 bytes | Version | A version number to track software/protocol upgrades
| 32 bytes | Previous Block Hash | A reference to the hash of the previous (parent) block in the chain
| 32 bytes | Merkle Root | A hash of the root of the Merkle-Tree of this block's transactions
| 4 bytes | Timestamp | The approximate creation time of this block (seconds from Unix Epoch)
| 4 bytes | Difficulty Target | The proof-of-work algorithm difficulty target for this block
| 4 bytes | Nonce | A counter used for the proof-of-work algorithm
|=======
[[block_hash]]
=== Block Identifiers - Block Header Hash and Block Height
The primary identifier of a block is its cryptographic hash, a digital fingerprint, made by hashing the block header twice through the SHA256 algorithm. The resulting 32-byte hash, is called the _block hash_, but is more accurately the _block *header* hash_, as only the block header is used to compute it. For example, the block hash of the first bitcoin block ever created is +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+. The block hash identifies a block uniquely and unambiguously and can be independently derived by any node by simply hashing the block header.
Note that the block hash is not actually included inside the block's data structure, neither when the block is transmitted on the network, nor when it is stored on a node's persistence storage as part of the blockchain. Instead, the block's hash is computed by each node as the block is received from the network. On full nodes, the block hash may be stored in a separate database table as part of the block's metadata, to facilitate indexing and faster retrieval of block from disk.
A second way to identify a block is by its position in the blockchain, called the _block height_. The first block ever created is at block height 0 (zero), and is the same block that was referenced by the block hash +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+ {above - CHECK TO SEE IF ABOVE OR BELOW}. A block can thus be identified two ways, either by referencing the block hash, or by referencing the block height. Each subsequent block added "on top" of that first block is one position "higher" in the blockchain, like boxes stacked one on top of the other. The block height on January 1st 2014 was approximately 278,000, meaning there were 278,000 blocks stacked on top of the first block created in January 2009.
Unlike the block hash, the block height is not a unique identifier. While a single block will always have a specific and invariant block height, the reverse is not true - the block height does not always identify a single block. Two or more blocks may have the same block height, competing for the same position in the blockchain. This scenario is discussed in detail in the section on <<forks>>. The block height is also not a part of the block's data structure, it is not stored within the block. Each node dynamically identifies a block's position (height) in the blockchain when it is received from the bitcoin network. The block height may also be stored as metadata in an indexed database table for faster retrieval.
[TIP]
====
A block's _block hash_ always identifies a single block uniquely. A block also always has a specific _block height_. However, it is not always the case that a specific block height can identify a single block, rather more than one blocks can compete for a single position in the blockchain.
====
=== The Genesis Block
The first block in the blockchain is called the _genesis block_ and was created in 2009. It is the "common ancestor" of all the blocks in the blockchain, meaning that if you start at any block and follow the chain backwards in time you will eventually arrive at the _genesis block_.
The genesis block has the identifier hash +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+. You can search for that block hash in any block explorer website, such as blockchain.info, and you will find a page describing the contents of this block, with a URL containing that hash:
https://blockchain.info/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
https://blockexplorer.com/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
Using the Bitcoin Core reference client on the command-line:
----
$ bitcoind getblock 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
{
"hash" : "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
"confirmations" : 308321,
"size" : 285,
"height" : 0,
"version" : 1,
"merkleroot" : "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
"tx" : [
"4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b"
],
"time" : 1231006505,
"nonce" : 2083236893,
"bits" : "1d00ffff",
"difficulty" : 1.00000000,
"nextblockhash" : "00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048"
}
----
=== Linking Blocks in the Blockchain
Jing's node maintains a complete copy of the blockchain, starting at the genesis block. The local copy of the blockchain is constantly updated as new blocks are found and used to extend the chain. As Jing's node receives incoming blocks from the network, it will validate these blocks and then link them to the existing blockchain. To establish a link, Jing's node will examine the incoming block header and look for the "previous block hash".
Let's assume for example that Jing's node has 277,314 blocks in the local copy of the blockchain. The last block Jing's node knows about is block 277314, with a block header hash of +00000000000000027e7ba6fe7bad39faf3b5a83daed765f05f7d1b71a1632249+.
Jing's node then receives a new block from the network, which it parses as follows:
----
{
"size" : 43560,
"version" : 2,
"previousblockhash" :
"00000000000000027e7ba6fe7bad39faf3b5a83daed765f05f7d1b71a1632249",
"merkleroot" :
"5e049f4030e0ab2debb92378f53c0a6e09548aea083f3ab25e1d94ea1155e29d",
"time" : 1388185038,
"difficulty" : 1180923195.25802612,
"nonce" : 4215469401,
"tx" : [
"257e7497fb8bc68421eb2c7b699dbab234831600e7352f0d9e6522c7cf3f6c77",
[... many more transactions omitted ...]
"05cfd38f6ae6aa83674cc99e4d75a1458c165b7ab84725eda41d018a09176634"
]
}
----
Looking at this new block, Jing's node sees the "previousblockhash" field contains the hash of its parent block. It is a hash known to Jing's node, that of the last block on the chain at height 277314. Therefore, this new block is a child of the last block on the chain and extends the existing blockchain. Jing's node adds this new block to the end of the chain, making its heigh 277315.
The moment Jing's node received this block (height 277315), this event signified that another node on the network had won this round of the mining competition. Jing's mining node then created the candidate block for the new round of the competition. After filling the candidate block with transactions, Jing's node needs to populate the header of the candidate block and therefore link it to the rest of the blockchain. The existing block at height 277315 will be the parent of Jing's new candidate block. Jing's node will therefore populate the "previous block hash" field in the candidate block with the hash of the block at height 277315.
[[chain_of_blocks]]
.Blocks linked in a chain, by reference to the previous block header hash
image::images/ChainOfBlocks.png["chain_of_blocks"]
[[merkle_trees]]
=== Merkle Trees
Each block in the bitcoin blockchain contains a summary of all the transactions in the block, using a _Merkle Tree_.
A _Merkle Tree_, also known as a _Binary Hash Tree_ is a data structure used for efficiently summarizing and verifying the integrity of large sets of data. Merkle Trees are binary trees containing cryptographic hashes. The term "tree" is used in computer science to describe a branching data structure, but these trees are usually displayed upside down with the "root" at the top and the "leaves" at the bottom of a diagram, as you will see in the examples that follow.
Merkle trees are used in bitcoin to summarize all the transactions in a block, producing an overall digital fingerprint of the entire set of transactions, which can be used to prove that a transaction is included in the set. A merkle tree is constructed by recursively hashing pairs of nodes until there is only one hash, called the _root_, or _merkle root_. The cryptographic hash algorithm used in bitcoin's merkle trees is SHA256 applied twice, also known as double-SHA256.
When N data elements are hashed and summarized in a Merkle Tree, you can check to see if any one data element is included in the tree with at most +2*log~2~(N)+ calculations, making this a very efficient data structure.
The merkle tree is constructed bottom-up. In the example below, we start with four transactions A, B, C and D, which form the _leaves_ of the Merkle Tree, shown in the diagram at the bottom. The transactions are not stored in the merkle tree, rather their data is hashed and the resulting hash is stored in each leaf node as H~A~, H~B~, H~C~ and H~D~:
+H~A~ = SHA256(SHA256(Transaction A))+
Consecutive pairs of leaf nodes are then summarized in a parent node, by concatenating the two hashes and hashing them together. For example, to construct the parent node H~AB~, the two 32-byte hashes of the children are concatenated to create a 64-byte string. That string is then double-hashed to produce the parent node's hash:
+H~AB~ = SHA256(SHA256(H~A~ + H~B~))+
The process continues until there is only one node at the top, the node known as the Merkle Root. That 32-byte hash is stored in the block header and summarizes all the data in all four transactions.
[[simple_merkle]]
.Calculating the nodes in a Merkle Tree
image::images/MerkleTree.png["merkle_tree"]
Since the merkle tree is a binary tree, it needs an even number of leaf nodes. If there is an odd number of transactions to summarize, the last transaction hash is duplicated to create an even number of leaf nodes, also known as a _balanced tree_. This is shown in the example below, where transaction C is duplicated:
[[merkle_tree_odd]]
.An even number of data elements, by duplicating one data element
image::images/MerkleTreeOdd.png["merkle_tree_odd"]
The same method for constructing a tree from four transactions can be generalized to construct trees of any size. In bitcoin it is common to have several hundred to more than a thousand transactions in a single block, which are summarized in exactly the same way producing just 32-bytes of data from a single merkle root. In the diagram below, you will see a tree built from 16 transactions:
[[merkle_tree_large]]
.A Merkle Tree summarizing many data elements
image::images/MerkleTreeLarge.png["merkle_tree_large"]
To prove that a specific transaction is included in a block, a node need only produce +log~2~(N)+ 32-byte hashes, constituting an _authentication path_ or _merkle path_ connecting the specific transaction to the root of the tree. This is especially important as the number of transactions increases, because the base-2 logarithm of the number of transactions increases much more slowly. This allows bitcoin nodes to efficiently produce paths of ten or twelve hashes (320-384 bytes) which can provide proof of a single transaction out of more than a thousand transactions in a megabyte sized block. In the example below, a node can prove that a transaction K is included in the block by producing a merkle path that is only four 32-byte hashes long (128 bytes total). The path consists of the four hashes H~L~, H~IJ~, H~MNOP~ and H~ABCDEFGH~. With those four hashes provided as an authentication path, any node can prove that H~K~ is included in the merkle root by computing four additional pair-wise hashes H~KL~, H~IJKL~ and H~IJKLMNOP~ that lead to the merkle root.
[[merkle_tree_path]]
.A Merkle Path used to prove inclusion of a data element
image::images/MerkleTreePathToK.png["merkle_tree_path"]
The efficiency of merkle trees becomes obvious as the scale increases. For example, proving that a transaction is part of a block requires:
[[block_structure]]
.Merkle Tree Efficiency
[options="header"]
|=======
|Number of Transactions| Approx. Size of Block | Path Size (Hashes) | Path Size (Bytes)
| 16 transactions | 4 kilobytes | 4 hashes | 128 bytes
| 512 transactions | 128 kilobytes | 9 hashes | 288 bytes
| 2048 transactions | 512 kilobytes | 11 hashes | 352 bytes
| 65,535 transactions | 16 megabytes | 16 hashes | 512 bytes
|=======
As you can see from the table above, while the block size increases rapidly, from 4KB with 16 transactions to a block size of 16 MB to fit 65,535 transactions, the merkle path required to prove the inclusion of a transaction increases much more slowly, from 128 bytes to only 512 bytes. With merkle trees, a node can download just the block headers (80 bytes per block) and still be able to identify a transaction's inclusion in a block by retrieving a small merkle path from a full node, without storing or transmitting the vast majority of the blockchain which may be several gigabytes in size. Nodes which do not maintain a full blockchain, called Simple Payment Verification or SPV nodes use merkle paths to verify transactions without downloading full blocks.
==== Highest Difficulty Chain Selection

376
ch08.asciidoc Normal file
View File

@ -0,0 +1,376 @@
[[ch8]]
== Chapter 8 - Consensus & Mining
*DRAFT - DO NOT SUBMIT ISSUES OR PULL REQUESTS YET PLEASE - CONSTANT CHANGES HAPPENING*
=== Introduction
((("blockchain")))
Bitcoin's blockchain is the global public ledger (list) of all transactions, which everyone in the bitcoin network accepts as the authoritative record of ownership.
But how can everyone in the network agree on a single universal "truth" about who owns what, without having to trust anyone? All traditional payment systems depend on a trust model that has a central authority providing a clearinghouse service, basically verifying and clearing all transactions. Bitcoin has no central authority, yet somehow every node has a complete copy of a public ledger that it can trust as the authoritative record. The blockchain is not created by a central authority, but is assembled independently by every node in the network. Somehow, every node in the network, acting on information transmitted across insecure network connections can arrive at the same conclusion and assemble a copy of the same public ledger as everyone else. This chapter examines the process by which the bitcoin network achieves global consensus without central authority.
Satoshi Nakamoto's main invention is the decentralized mechanism for emergent consensus. All the properties of bitcoin, including currency, transactions, payments and the security model that does not depend central authority or trust derive from this invention.
Bitcoin's consensus emerges from the interplay of three processes that occur independently on nodes across the network:
* Independent verification of each transaction, by every full node, based on a comprehensive list of criteria
* Independent aggregation of those transactions into new blocks by mining nodes, coupled with demonstrated computation through a Proof-of-Work algorithm
* Independent assembly of the new blocks by every full node into a chain and selection of the chain with the most cumulative computation demonstrated through Proof-of-Work
In the next few sections we will examine these processes and how they interact to create the emergent property of network-wide consensus that allows any bitcoin node to assemble its own copy of the authoritative, trusted, public, global ledger.
Each of these processes also aggregates smaller bitcoin units into larger data structures. First, unspent transaction outputs (UTXO) are aggregated into transactions. Next, many transactions are aggregated into a block. Finally, blocks are added to a chain of blocks, the blockchain. In the previous chapter we looked at transactions as a data structure. In this chapter we will also look at the larger data structures: blocks and the blockchain.
=== Independent Verification of Transactions
In the previous chapter we saw how wallet software creates transactions by collecting UTXO, providing the appropriate unlocking scripts and then constructing new outputs assigned to a new owner. The resulting transaction is then sent to the neighboring nodes in the bitcoin network so that it may be propagated across the entire bitcoin network.
Every bitcoin node that receives a transaction will first verify the transaction before forwarding it to its neighbors. This ensures that only valid transactions are propagated across the network, while invalid transactions are discarded at the first node that encounters them.
Each node verifies every transaction against a long checklist of criteria:
* Check the syntactic correctness of the transaction's data structure
* Make sure neither lists of inputs or outputs are empty
* The transaction size in bytes is less than MAX_BLOCK_SIZE
* Each output value, as well as the total, must be within the allowed range of values (less than 21m coins, more than 0)
* Check none of the inputs have hash=0, N=-1 (coinbase transactions should not be relayed)
* Check that nLockTime is less than or equal to INT_MAX
* Check that the transaction size in bytes is greater than or equal to 100
* Check the number of signature operations contained in the transaction is less than the signature operation limit
* Reject "nonstandard" transactions: unlocking script (scriptSig) doing anything other than pushing numbers on the stack, or the locking script (scriptPubkey) not matching isStandard forms
* Check for a matching transaction in the pool, or in a block in the main branch, if so reject this transaction
* For each input, if the referenced output exists in any other transaction in the pool, reject this transaction.
* For each input, look in the main branch and the transaction pool to find the referenced output transaction. If the output transaction is missing for any input, this will be an orphan transaction. Add to the orphan transactions, if a matching transaction is not already in the pool.
* For each input, if the referenced output transaction is a coinbase output, it must have at least COINBASE_MATURITY (100) confirmations; else reject this transaction
* For each input, if the referenced output does not exist (e.g. never existed or has already been spent), reject this transaction
* Using the referenced output transactions to get input values, check that each input value, as well as the sum, are in the allowed range of values (less than 21m coins, more than 0)
* Reject if the sum of input values < sum of output values
* Reject if transaction fee would be too low to get into an empty block
* Verify the unlocking scripts for each input against the corresponding output locking scripts
These conditions can be seen in detail in the functions AcceptToMemoryPool, CheckTransaction and CheckInputs in the bitcoin reference client. Note that the conditions change over time, to address new types of Denial-of-Service attacks or sometimes to relax the rules so as to include more types of transactions.
By independently verifying each transaction as it is received and before propagating it, every node builds a pool of valid new transactions (the transaction pool), roughly in the same order.
=== Aggregating Transactions into Blocks
Some of the nodes on the bitcoin network are specialized nodes called _miners_. A miner will collect, validate and relay new transactions just like any other node. Unlike other nodes, a miner will then aggregate these transactions into a _block_. The block of transactions constructed by a miner is a candidate block and becomes valid only if the miner succeeds in winning the mining competition. Each miner competes by trying billions of possible solutions to an equation based on a cryptographic hash. If they find a solution, they broadcast the candidate block for everyone to record on the blockchain. The competition difficulty is calibrated to ensure that a new block solution is found by someone every 10 minutes on average. We will look at the mining process itself in more detail in <<mining>>. For now, let's look at how miners aggregate transactions into blocks.
In Chapter 1 we introduced Jing, a computer engineering student in Shanghai China, who is a bitcoin miner. Jing earns bitcoin by running a "mining rig" which is a specialized computer-hardware system designed to mine bitcoins. Jing started mining for bitcoin in 2010, when mining was not as competitive as it is today. At that time, Jing could mine bitcoin using a desktop computer. Today, he uses a much more powerful mining system based on Application Specific Integrated Circuits (ASICs), which are specialized silicon chips designed exclusively for one application - bitcoin mining. Over time, the way Jing participates in the mining process has changed slightly, but the fundamentals remain the same. We will start by looking at how Jing mined in 2010, when things were simpler and then look at how he mines today, as bitcoin mining has become a more complex and competitive activity.
In 2010, Jing mined on a desktop computer. At the time, he would run a full bitcoin node, connected to the bitcoin network. A full bitcoin node keeps a full copy of the blockchain, the list of all transactions since the first ever transaction in 2009. Jing's bitcoin node receives transactions propagated by other nodes, just like any other node on the bitcoin network. After validating those transactions, the bitcoin software adds them to the _memory pool_, or _transaction pool_, where transactions would await until they could be included (mined) into a block.
Jing's bitcoin node is also listening for new blocks, propagated on the bitcoin network. Let's follow the blocks that were created during the time Alice bought a cup of coffee from Bob's Cafe (see <<cup_of_coffee>>). Alice's transaction was included in block 277316. For the purpose of demonstrating the concepts in this chapter let's assume that block was mined by Jing's mining system and follow Alice's transaction as it becomes part of this new block.
Jing's mining node maintains a local copy of the blockchain, the list of all blocks created since the beginning of the bitcoin system in 2009. By the time Alice buys the cup of coffee, Jing's node has assembled a chain of 277,314 blocks of transactions. Jing's node is listening for transactions, trying to mine a new block and also listening for blocks discovered by other nodes. Jing's node receives an incoming block, propagated by the bitcoin network, which fits into the existing blockchain at height 277,315.
As soon a Jing's bitcoin node receives a valid new block, it immediately starts the next round of competition. Receiving a new block signifies that someone else has won the previous round, meaning that Jing's system did not win that round and should abandon its current efforts and shift its resources to try and win the next round. During the previous 10 minutes, while Jing's node was searching for a solution, it was also collecting transactions. By now it has collected a few hundred transactions in the memory pool. After removing any transactions that appear in the new block recently received, Jing's memory pool is left containing unconfirmed transactions that are waiting to be recorded in a new block.
Jing's node immediately constructs a new candidate block, to participate in the competition.
=== Adding Transactions to a Candidate Block
To construct the candidate block Jing's bitcoin node selects transactions from the memory pool, by applying a priority metric to each transaction and adding the highest priority transactions first. Transactions are prioritized based on the "age" of the UTXO that is being spent in their inputs, allowing for old and high-value inputs to be prioritized over newer and smaller inputs. Prioritized transactions can be sent without any fees, if there is enough space in the block.
The priority of a transaction is calculated as the sum of the value and age of the inputs divided by the total size of the transaction:
----
Priority = Sum (Value of input * Input Age) / Transaction Size
----
In the equation above, the value of an input is measured in the base unit, satoshis (1/100m of a bitcoin). The age of a UTXO is the number of blocks that have elapsed since the UTXO was recorded on the blockchain, measuring how many blocks "deep" into the blockchain it is. The size of the transaction is measured in bytes.
For a transaction to be considered "high priority" its priority must be greater than 57,600,000, which corresponds to one bitcoin (100m satoshis), aged one day (144 blocks) in a transaction of 250 bytes total size.
----
High Priority = 100,000,000 satoshis * 144 blocks / 250 bytes = 57,600,000
----
The first 50 kilobytes of transaction space in a block are set aside for high priority transactions. Jing's node will fill the first 50 kilobytes, prioritizing the highest priority transactions first, regardless of fee. This allows high-priority transactions to be processed even if they carry zero fees.
Jing's mining node then fills the rest of the block up to the maximum block size (MAX_BLOCK_SIZE in the code), with transactions that carry at least the minimum fee, prioritizing those with the highest fee per kilobyte of transaction.
If there is any space remaining in the block, Jing's mining node may choose to fill it with no-fee transactions. Some miners choose to mine transactions without fees on a best-effort basis. Other miners may choose to ignore transactions without fees.
Any transactions left in the memory pool after the block is filled will remain in the pool for inclusion in the next block. As transactions remain in the memory pool, their inputs "age", as the UTXO they spend get deeper into the blockchain with new blocks added on top. Since a transactions priority depends on the age of its inputs, transactions remaining in the pool will age and therefore increase in priority. Eventually a transaction without fees may reach a high enough priority to be included in the block for free.
Bitcoin transactions do not have an expiration time-out. A transaction that is valid now will be valid in perpetuity. However, if a transaction is only propagated across the network once it will persist only as long as it is held in a mining node memory pool. When a mining node is restarted, its memory pool is wiped clear, as it is a transient non-persistent form of storage. While a valid transaction may have been propagated across the network, if it is not executed it may eventually not reside in the memory pool of any miner. Wallet software is expected to retransmit such transactions or reconstruct them with higher fees if they are not successfully executed within a reasonable amount of time.
[[mining]]
=== Proof-of-Work (Mining) and Consensus
((("Mining", "Proof of Work", "SHA256", "hashing power", "difficulty", "nonce")))
Mining is the process by which new bitcoin is added to the money supply. Mining also serves to secure the bitcoin system against fraudulent transactions or transactions spending the same amount of bitcoin more than once, known as a double-spend. Miners act as a decentralized clearinghouse, validating new transactions and recording them on the global ledger. A new block, containing transactions which occurred since the last block, is "mined" every 10 minutes thereby adding those transactions to the blockchain. Transactions that become part of a block and added to the blockchain are considered "confirmed", which allows the new owners of bitcoin to spend the bitcoin they received in those transactions. Miners receive two types of reward for mining: new coins created with each new block and transaction fees from all the transactions included in the block. To earn this reward, the miners compete to solve a difficult mathematical problem based on a cryptographic hash algorithm. The solution to the problem, called the Proof-of-Work, is included in the new block and acts as proof that the miner expended significant computing effort. The competition to solve the Proof-of-Work algorithm to earn reward and the right to record transactions on the blockchain is the basis for bitcoin's security model.
Bitcoin's security is underpinned by computation. New blocks are added to the blockchain through a consensus mechanism called the _Proof-of-Work_ (PoW) that requires a predictable computational effort, one that takes approximately 10 minutes to solve on average. Specialized bitcoin nodes called _miners_ validate transactions and collect them into blocks, then attempt to find the solution that satisfies the Proof-of-Work algorithm. The first miner to find such a solution, propagates the newly created block across the network. All other nodes on the network verify that the new block contains valid transactions and satisfies the Proof-of-Work algorithm, then they add it to the blockchain, thereby extending it by one block. The miners add a special coin generation transaction into the blocks they build, which creates new bitcoin from nothing and is payable to the miner's own bitcoin address. Once the block is accepted as valid by the entire network, that transaction is also recorded on the blockchain, thereby rewarding the miner for the computational effort it took to satisfy the Proof-of-Work. This de-centralized consensus mechanism, based on a global competition and requiring computation to create new blocks, is the basis for the security of the bitcoin transaction ledger and also for the issuance of new bitcoin. {move the last sentence to the beginning of the paragraph - to explain more about security} The equilibrium between the incentive of bitcoin reward and the immense computing effort required to win it force the participants to behave honestly, without the need for a centralized clearinghouse or currency issuer. The bitcoin consensus mechanism is a dynamic, self-regulating and completely decentralized security model that operates at very large scale.
The process of new coin generation is called mining, because the reward is designed to simulate diminishing returns, just like mining for precious metals. Bitcoin's money supply is created through mining, just like a central bank issues new money by printing bank notes. The amount of newly created bitcoin a miner can add to a block decreases approximately every four years (or precisely every 210,000 blocks). It started at 50 bitcoin per block in January of 2009 and halved to 25 bitcoin per block in November of 2012. It will halve again to 12.5 bitcoin per block sometime in 2016. Based on this formula, bitcoin mining rewards decrease exponentially until approximately the year 2140 when all 21 million bitcoin have been issued.
Bitcoin miners also earn fees from transactions. Every transaction may include a transaction fee, in the form of a surplus of bitcoin between the transaction's inputs and outputs. The bitcoin miner gets to "keep the change" on the transactions.
Today the fees represent 1% or less of a bitcoin miner's income, the vast majority coming from the newly minted bitcoins. However, as the reward decreases over time and the number of transactions per block increases, a greater proportion of bitcoin mining earnings will come from fees. After 2140 all bitcoin miner earnings will be in the form of transaction fees.
[[figure_sha256_logical]]
.The Secure Hash Algorithm (SHA-256)
image::images/sha256-logical.png["SHA256"]
With SHA-256, the output is always 256 bits long, regardless of the size of the input. In the example below, we will use the Python interpreter to calculate the SHA256 hash of the phrase "I am Satoshi Nakamoto".
[[sha256_example1]]
.SHA256 Example
----
$ *python*
Python 2.7.1
>>> import hashlib
>>> print hashlib.sha256("I am Satoshi Nakamoto").hexdigest()
5d7c7ba21cbbcd75d14800b100252d5b428e5b1213d27c385bc141ca6b47989e
----
The example shows that if we calculate the hash of the phrase +"I am Satoshi Nakamoto"+, it will produce +5d7c7ba21cbbcd75d14800b100252d5b428e5b1213d27c385bc141ca6b47989e+. This 256-bit number is the _hash_ or _digest_ of the phrase and depends on every part of the phrase. Adding a single letter, punctuation mark or any character will produce a different hash.
Now, if we vary the phrase, we will expect to see completely different hashes. Let's try that by adding a number to the end of our phrase, using this simple Python script
[[sha256_example_generator]]
.SHA256 A script for generating many hashes by iterating on a nonce
====
[source, python]
----
include::code/hash_example.py[]
----
====
Running this will produce the hashes of several phrases, made different by adding a unique number, called a _nonce_ at the end of the text. By incrementing the nonce, we can get different hashes.
((("nonce")))
[[sha256_example_generator_output]]
.SHA256 Output of a script for generating many hashes by iterating on a nonce
----
$ *python hash_example.py*
I am Satoshi Nakamoto0 => a80a81401765c8eddee25df36728d732...
I am Satoshi Nakamoto1 => f7bc9a6304a4647bb41241a677b5345f...
I am Satoshi Nakamoto2 => ea758a8134b115298a1583ffb80ae629...
I am Satoshi Nakamoto3 => bfa9779618ff072c903d773de30c99bd...
I am Satoshi Nakamoto4 => bce8564de9a83c18c31944a66bde992f...
I am Satoshi Nakamoto5 => eb362c3cf3479be0a97a20163589038e...
I am Satoshi Nakamoto6 => 4a2fd48e3be420d0d28e202360cfbaba...
I am Satoshi Nakamoto7 => 790b5a1349a5f2b909bf74d0d166b17a...
I am Satoshi Nakamoto8 => 702c45e5b15aa54b625d68dd947f1597...
I am Satoshi Nakamoto9 => 7007cf7dd40f5e933cd89fff5b791ff0...
I am Satoshi Nakamoto10 => c2f38c81992f4614206a21537bd634a...
I am Satoshi Nakamoto11 => 7045da6ed8a914690f087690e1e8d66...
I am Satoshi Nakamoto12 => 60f01db30c1a0d4cbce2b4b22e88b9b...
I am Satoshi Nakamoto13 => 0ebc56d59a34f5082aaef3d66b37a66...
I am Satoshi Nakamoto14 => 27ead1ca85da66981fd9da01a8c6816...
I am Satoshi Nakamoto15 => 394809fb809c5f83ce97ab554a2812c...
I am Satoshi Nakamoto16 => 8fa4992219df33f50834465d3047429...
I am Satoshi Nakamoto17 => dca9b8b4f8d8e1521fa4eaa46f4f0cd...
I am Satoshi Nakamoto18 => 9989a401b2a3a318b01e9ca9a22b0f3...
I am Satoshi Nakamoto19 => cda56022ecb5b67b2bc93a2d764e75f...
----
Each phrase produces a completely different hash result. They seem completely random, but you can re-produce the exact results in this example on any computer with Python and see the same exact hashes.
To make a challenge out of this algorithm, let's set an arbitrary target: find a phrase starting with "I am Satoshi Nakamoto" which produces a hash that starts with a zero. In numerical terms, that means finding a hash value that is less than +0x1000000000000000000000000000000000000000000000000000000000000000+. Fortunately, this isn't so difficult! If you notice above, we can see that the phrase "I am Satoshi Nakamoto13" produces the hash 0ebc56d59a34f5082aaef3d66b37a661696c2b618e62432727216ba9531041a5, which fits our criteria. It only took 13 attempts to find it.
==== Proof-of-Work Algorithm
Bitcoin's proof-of-work is very similar to the problem above. First, a miner will generate a new block, containing:
((("block")))
* Transactions waiting to be included in a block
* The hash from the previous block
* A _nonce_
The only part a miner can modify is the nonce. Now, the miner will calculate the hash of this block's header and see if it is smaller than the current _target difficulty_. The miner will likely have to try many nonces before finding one that results in a low enough hash.
A very simplified proof-of-work algorithm is implemented in Python here:
((("proof of work")))
[[pow_example1]]
.Simplified Proof-Of-Work Implementation
====
[source, python]
----
include::code/proof-of-work-example.py[]
----
====
Running the code above, you can set the desired difficulty (in bits, how many of the leading bits must be zero) and see how long it takes for your computer to find a solution. In the following examples, you can see how it works on an average laptop:
[[pow_example_outputs]]
.Running the proof-of-work example for various difficulties
----
$ *python proof-of-work-example.py*
Difficulty: 1 (0 bits)
[...]
Difficulty: 8 (3 bits)
Starting search...
Success with nonce 9
Hash is 1c1c105e65b47142f028a8f93ddf3dabb9260491bc64474738133ce5256cb3c1
Elapsed Time: 0.0004 seconds
Hashing Power: 25065 hashes per second
Difficulty: 16 (4 bits)
Starting search...
Success with nonce 25
Hash is 0f7becfd3bcd1a82e06663c97176add89e7cae0268de46f94e7e11bc3863e148
Elapsed Time: 0.0005 seconds
Hashing Power: 52507 hashes per second
Difficulty: 32 (5 bits)
Starting search...
Success with nonce 36
Hash is 029ae6e5004302a120630adcbb808452346ab1cf0b94c5189ba8bac1d47e7903
Elapsed Time: 0.0006 seconds
Hashing Power: 58164 hashes per second
[...]
Difficulty: 4194304 (22 bits)
Starting search...
Success with nonce 1759164
Hash is 0000008bb8f0e731f0496b8e530da984e85fb3cd2bd81882fe8ba3610b6cefc3
Elapsed Time: 13.3201 seconds
Hashing Power: 132068 hashes per second
Difficulty: 8388608 (23 bits)
Starting search...
Success with nonce 14214729
Hash is 000001408cf12dbd20fcba6372a223e098d58786c6ff93488a9f74f5df4df0a3
Elapsed Time: 110.1507 seconds
Hashing Power: 129048 hashes per second
Difficulty: 16777216 (24 bits)
Starting search...
Success with nonce 24586379
Hash is 0000002c3d6b370fccd699708d1b7cb4a94388595171366b944d68b2acce8b95
Elapsed Time: 195.2991 seconds
Hashing Power: 125890 hashes per second
[...]
Difficulty: 67108864 (26 bits)
Starting search...
Success with nonce 84561291
Hash is 0000001f0ea21e676b6dde5ad429b9d131a9f2b000802ab2f169cbca22b1e21a
Elapsed Time: 665.0949 seconds
Hashing Power: 127141 hashes per second
----
As you can see, increasing the difficulty by 1 bit causes an exponential increase in the time it takes to find a solution. If you think of the entire 256-bit number space, each time you constrain one more bit to zero, you decrease the search space by half. In the example above, it takes 84 million hash attempts to find a nonce that produces a hash with 26 leading bits as zero. Even at a speed of more than 120 thousand hashes per second, it still requires ten minutes on a consumer laptop to find this solution.
At the time of writing this, the network is attempting to find a block whose header hash is less than +000000000000004c296e6376db3a241271f43fd3f5de7ba18986e517a243baa7+. As you can see, there are a lot of zeroes at the beginning of that hash, meaning that the acceptable range of hashes is much smaller, hence more difficult to find a valid hash. It will take on average more 150 quadrillion hash calculations per second for the network to discover the next block. That seems like an impossible task, but fortunately the network is bringing 500 TH/sec of processing power to bear, which will be able to find a block in about 10 minutes on average.
==== Difficulty Target and Re-Targetting
Bitcoin is tuned to generate blocks approximately every 10 minutes. This is achieved by automatically adjusting the target difficulty to account for increases and decreases in the available computing power on the network. This process occurs automatically and on every full node independently. Each node recalculates the expected difficulty every 2106 blocks, based on the time it took to hash the previous 2106 blocks. In simple terms: If the network is finding blocks faster than every 10 minutes, the difficulty increases. If block discovery is slower than expected, the difficulty will decrease.
{miners that are on mining pools get the difficulty (do not calculate difficulty independently) they are given the difficulty from the mining pool so they don't have to calculate the difficulty themselves and they are actually given a lower difficulty target. There are essentially two classifications of miners today - pool miners and solo miners. Solo miners run a full node and compete on their own. Whereas pool miners collaborate with one another and compete against the network as a team, while sharing the reward. The reason miners join pools - solo miners need an enormous amount of hashing power in order to have even the slimmest chance of finding a solution to a block which will make their earnings erratic. By participating in a pool, miners get smaller shares but a more regular share of rewards, reducing uncertainty. Solo mining is becoming obsolete, as the difficulty increases the likelihood of a solo miner finding a solution is more like winning the lottery.}
{ASIC miners do not run full nodes. Full nodes independently calculate the difficulty using the same equation on the same block, arriving at the same result for the new difficulty. Retargeting the difficulty at block heights that are multiples of 2106 from the genesis block. The equation for retargeting difficulty measures the time it took to find the last 2106 blocks, compares that to the expected time of 21,060 minutes (based upon a desired 10 minute block time), the difference is calculated as a percentage and a corresponding percentage adjustment is made to the difficulty. To avoid extreme volatility in the difficulty, the retargeting adjustment cannot exceed {X%} per retargeting. The difficulty will only be retargeted up or down by maximum of {X%} per cycle. If the required difficulty adjustment is greater than the maximum it will be reflected in the next retargeting adjustment as the imbalance will persist through the next 2106 blocks. Large discrepancies between hashing power and difficulty may take several cycles to even out. This leads to a potential problem which has been observed in alt coins, where very large changes in difficulty can cause hashing power to collapse leading to excessively long block times. If the aggregate network hashing power collapses due to the departure of many miners simultaneously, the remaining hashing power may be insufficient to meet the difficulty target leading to excessively long block intervals. Since retargeting is not a function of time but rather block number, a large hashing deficit can mean the next cycle is very far in the future. Usually this is caused for two reasons - scenario one - entry for a brief period of a lot of hashing which temporarily increases the difficulty, followed by the departure of that hashing, resulting in a collapse of block solutions. Essentially a hashing pump and dump. Usually a deliberate attack. This is not a concern in bitcoin because new hashing power introduced into the network will not effect the average enough to cause a major change in difficulty. The other scenario in which hashing power can collapse is a crash in bitcoin price, making mining unprofitable. (If the miner cannot pay their electricity bill, the miner will leave the network.) This is a weakness of the protocol, as an insurmountable hashing deficit could occur with a precipitous collapse in price and corresponding reduction in available hashing power. The network would be unable to recover because ... }
[TIP]
====
The difficulty of finding a bitcoin block is approximately '10 minutes of processing' for the entire network, based on the time it took to find the previous 2106 blocks, adjusted every 2106 blocks.
====
Note that the target difficulty is independent of the number of transactions or the value of transactions. This means that the amount of hashing power and therefore electricity expended to secure bitcoin is also entirely independent of the number of transactions. Bitcoin can scale up, achieve broader adoption and remain secure without any increase in hashing power from today's level. The increase in hashing power represents market forces as new miners enter the market to compete for the reward. As long as enough hashing power is under the control of miners acting honestly in pursuit of the reward, it is enough to prevent "takeover" attacks and therefore it is enough to secure bitcoin.
The target difficulty is closely related to the cost of electricity and the exchange rate of bitcoin vis-a-vis the currency used to pay for electricity. High performance mining systems are about as efficient as possible with the current generation of silicon fabrication, converting electricity into hashing computation at the highest rate possible. The primary influence on the mining market is the price of one kilowatt-hour in bitcoin, as that determines the profitability of mining and therefore the incentives to enter or exit the mining market.
==== Mining New Bitcoins
Bitcoins are "minted" during the creation of each block at a fixed and diminishing rate. Each block, generated on average every 10 minutes, contains a _reward_ that consists of entirely new bitcoins. The reward was 50BTC for the first four years of operation of the network. Every four years the reward is decreased by 50%, resulting in a diminishing rate of issuance over time. In 2012, the reward was decreased to 25BTC and it will decrease again to 12.5BTC in 2016. By approximately 2140, the last fragments of a bitcoin will be mined, for a total of 21 million bitcoins. {Clarify coinbase transaction as first - includes the reward and transactions. Discuss how the coinbase transaction will change in 2140}
The finite and diminishing issuance creates a fixed monetary supply that resists inflation. Unlike a fiat currency which can be printed in infinite numbers by a central bank, bitcoin can never be inflated by printing.
===== Monetary supply
Bitcoin's monetary supply is defined as the number of coins in circulation (minted). Like any other currency, this measure of monetary supply is called M0, which represents the narrowest measure of the money supply. Just like any other currency, bitcoin can also have a _fractional reserve banking_ which means that an organization can trade bitcoins "off blockchain" which are not part of the M0 monetary measure, but of the broader monetary supply measures M1-M3. {have you explained M1-M3?}{also, i think you should explain fractional reserve banking a bit here}
While the total bitcoins in circulation will not exceed 21m, that monetary base can support a much broader economy through fractional reserve banking and expansion of the available credit.
===== Deflationary Money
The most important and debated consequence of a fixed and diminishing monetary issuance is that the currency will tend to be inherently _deflationary_. Deflation is the phenomenon of appreciation of value due to a mismatch in supply and demand that drives up the value (and exchange rate) of a currency. The opposite of inflation, price deflation means that your money has more purchasing power over time.
Many economists argue that a deflationary economy is a disaster that should be avoided at all costs. That is because in a period of rapid deflation, the incentives for regular people are to hoard the money and not spend it, hoping that prices will fall. Such a phenomenon unfolded during Japan's "Lost Decade", when a complete collapse of demand pushed the currency into a deflationary spiral.
Bitcoin experts argue that deflation is not bad *per se*. Rather, we associate deflation with a collapse in demand because that is the only example of deflation we have to study. In a fiat currency with the possibility of unlimited printing, it is very difficult to enter a deflationary spiral unless there is a complete collapse in demand and an unwillingness to print money. Deflation in bitcoin is not caused by a collapse in demand, but by predictably constrained supply.
In practice, it has become evident that the hoarding instinct caused by a deflationary currency can be overcome by discounting from vendors, until the discount overcomes the hoarding instinct of the buyer. Since the seller is also motivated to hoard, the discount becomes the equilibrium price at which the two hoarding instincts are matched. With discounts of 30% on the bitcoin price, most bitcoin retailers are not experiencing difficulty overcoming the hoarding instinct and generating revenue. It remains to be seen whether the deflationary aspect of the currency is really a problem when it is not driven by rapid economic retraction.
[[forks]]
==== Blockchain Forks
{Discuss chain selection: As new blocks are found they are added to the chain. Each full node constructs a chain and calculates the cumulative difficulty of that chain. As blocks are constructed and propagated across the network,}
{create a graphic showing propagating transaction}
{Because the blockchain is a decentralized data structure, different copies of it are not always consistent. Blocks may arrive at different nodes at different times, causing them to have a different perspective o ft the blockchain. To resolve this, each node always selects and attempts to extend the chain of blocks that represents the most Proof-of-Work, also known as the longest chain or greatest cumulative difficulty chain, by adding the difficulty recorded in each block for a chain a node can calculate the total amount of PoW that has been expended to create that chain. As long as all nodes select the longest, i.e. the longest cumulative difficulty chain, the global bitcoin network eventually converges to a consistent state. Forks occur as temporary inconsistencies between versions of the blockchain, which are resolved by the eventual reconvergence.}
{Bitcoin's _consensus mechanism_, which creates the is comprised of the independent validation of transactions by every node, the cumulative work of the miners, and the network convergence upon the greatest difficulty chain. The interplay of these three processes manifests the emergent property of consensus that allows for a global decentralized public ledger without a central authority. which creates one global public ledger, emerges as a property of (1) the selection of the greatest difficulty chain. This chapter is about the emergent property of consensus. This consensus is created by the interplay of three processes - (1) ,2,3. The emergent property of network-wide consensus is what establishes a trusted decentralized global public ledger. Satohsi's invention was not proof of work, elliptic curve cryptography. Satoshi's invention was how the interplay of these processes creates emergent consensus in a decentralized network without the need for a centralized trusted authority.}
[[fork1]]
.Visualization of a blockchain fork event - Before the Fork
image::images/GlobalFork1.png["globalfork1"]
A "fork" occurs whenever there are two candidate blocks competing to form the longest blockchain. This occurs under normal conditions whenever two miners solve the Proof-of-Work algorithm within a short period of time from each other. As both miners discover a solution for their respective candidate blocks, they immediately broadcast their own "winning" block to their immediate neighbors who begin propagating the block across the network. Each node that receives a valid block will incorporate it into their blockchain, extending the blockchain by one block. If that node later sees another candidate block extending the same parent, they ignore the second candidate. As a result, some nodes will "see" one candidate block first, while other nodes will see the other candidate block and two competing versions of the blockchain will emerge.
{create a graphic with the globe, two miners each - bitcoin topology map}
[[fork2]]
.Visualization of a blockchain fork event - Two blocks found simultaneously
image::images/GlobalFork2.png["globalfork2"]
Let's assume for example that a miner in Canada finds a proof-of-work solution for block "A" that extends the blockchain from height 315000 to height 315001, building on top of parent block "P". Almost simultaneously, an Australian miner who was also extending block "P", finds a solution for block "B", their candidate block. Now, there are two possible candidates for block height 315001, one we call "A", originating in Canada and one we call "B", originating in Australia. Both blocks are valid, both blocks contain a valid solution to the proof of work, both blocks extend the same parent. Both blocks likely contain most of the same transactions, with only perhaps a few differences in the order of transactions.
[[fork2]]
.Visualization of a blockchain fork event - Two blocks propagate, splitting the network
image::images/GlobalFork3.png["globalfork3"]
From that moment, the bitcoin network nodes closest (topologically, not geographically) to the Canadian node will hear about block "A" first and will create a new greatest-cumulative-difficulty blockchain with height 315001 and "A" as the last block in the chain (e.g. P-A), ignoring the candidate block "B" that arrives a bit later. Meanwhile, nodes closer to the Australian node will take that block as the winner and extend the blockchain to height 315001 with "B" as the last block (e.g. P-B), ignoring "A" when it arrives a few seconds later. Any miners that saw "A" first will immediately build candidate blocks that reference "A" as the parent and start trying to solve the PoW for these candidate blocks. The miners that accepted "B" instead, will start extending that chain.
----
Block "A" extends the chain: P-A
Block "B" also extends the chain: P-B
----
[[fork4]]
.Visualization of a blockchain fork event - A new block extends one fork
image::images/GlobalFork4.png["globalfork4"]
Forks are almost always resolved within one block. As part of the network's hashing power is dedicated to building on top of "A" as the parent, another part of the hashing power is focused on building on top of "B". Even if the hashing power is almost evenly split, it is likely that one set of miners will find a solution and propagate it before the other set of miners have found any solutions. Let's say for example that the miners building on top of "B" find a new block "X" that extends the chain to height 315002 (e.g. P-B-X). They immediately propagate this new block and the entire network sees it as a valid solution.
All nodes that had chosen "B" as the winner in the previous round will simply extend the chain one more block. The nodes that chose "A" as the winner, however, will now see a block extending an even longer chain (greater-cumulative difficulty), that does not include "A" in it. Any miners working on extending the chain P-A will now stop that work because their candidate block is an "orphan", as its parent "A" is no longer on the longest chain. The block "A" is removed from the blockchain by any nodes that had accepted it and any transactions within it are queued up again for processing in the next block. The entire network re-converges on a single blockchain P-B-X, with "X" as the last block in the chain. All miners immediately start working on candidate blocks that reference "X" as their parent to extend the P-B-X chain.
It is theoretically possible for a fork to extend to two blocks, if two blocks are found almost simultaneously by miners on opposite "sides" of a previous fork. However, the chance of that happening is very low. Whereas a one-block fork may occur every week, a two-block fork is exceedingly rare.
----
Block "X" extends the chain: P-B-X
Old chain is now "shorter": P-A
----
[TIP]
====
As of version 0.9, Bitcoin Core's +alertnotify+ option will send alerts whenever a 6-block or longer fork occurs
====
[[chainforks]]
.A blockchain showing two instances of forking
image::images/BlockChainWithForks.png["chainforks"]
==== Competition and Coinbase
==== Mining Pools
===== Managed Pools
===== P2Pool
==== Mining Economics
==== Consensus Attacks
===== 51% Attack
===== Selfish Mining Attack
==== Normal Forks
==== Soft Forks
==== Hard Forks
==== Unusual Forks

BIN
images/AddressPropagation.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.0 KiB

BIN
images/BitcoinNetwork.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 350 KiB

BIN
images/BitcoinNodeTypes.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 139 KiB

BIN
images/BlockRain2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 113 KiB

BIN
images/Bloom1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

BIN
images/Bloom2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

BIN
images/Bloom3.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

BIN
images/Bloom4.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 30 KiB

BIN
images/Bloom5.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 31 KiB

BIN
images/ChainOfBlocks.png Executable file → Normal file
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 64 KiB

After

Width:  |  Height:  |  Size: 92 KiB

BIN
images/FullNodeReferenceClient_Small.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

BIN
images/GlobalFork1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 130 KiB

BIN
images/GlobalFork2.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 130 KiB

After

Width:  |  Height:  |  Size: 126 KiB

BIN
images/GlobalFork3.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 136 KiB

BIN
images/GlobalFork4.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 132 KiB

BIN
images/GlobalFork5.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 133 KiB

BIN
images/InventorySynchronization.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

BIN
images/NetworkHandshake.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.3 KiB

BIN
images/SPVSynchronization.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

24
preface.asciidoc
View File

@ -19,7 +19,7 @@ _The best way to get the right answer on the Internet is not to ask a question,
I hope you can help me find and publish the "right answer" by the time this book is ready to print. I hope you can help me find and publish the "right answer" by the time this book is ready to print.
The early-realease source code is published on Github: https://github.com/aantonop/bitcoinbook The early-release source code is published on Github: https://github.com/aantonop/bitcoinbook
File an "Issue", or fork, modify and create a "Pull Request" with your changes. File an "Issue", or fork, modify and create a "Pull Request" with your changes.
@ -141,8 +141,8 @@ difficulty target::
((("target difficulty"))) ((("target difficulty")))
A difficulty at which all the computation in the network will find blocks approximately every 10 minutes. A difficulty at which all the computation in the network will find blocks approximately every 10 minutes.
difficulty re-targetting:: difficulty re-targeting::
((("difficulty re-targetting"))) ((("difficulty re-targeting")))
A network-wide re-calculation of the difficulty which occurs once every 2106 blocks and considers the hashing power of the previous 2106 blocks. A network-wide re-calculation of the difficulty which occurs once every 2106 blocks and considers the hashing power of the previous 2106 blocks.
fees:: fees::
@ -199,15 +199,15 @@ Many contributors offered comments, corrections and additions to the early-relea
===== (Name - Github ID) ===== (Name - Github ID)
* *Minh T. Nguyen - enderminh: Github contribution editor* * *Minh T. Nguyen - enderminh: Github contribution editor*
* Erik Wahlström - erikwam
* Eric Winchell - winchell * Eric Winchell - winchell
* Richard Kiss - richardkiss
* Sergej Kotliar - ziggamon * Sergej Kotliar - ziggamon
* Erik Wahlström - erikwam * Nagaraj Hubli - nagarajhubli
* ishotjr * Michalis Kargakis - kargakis
* Ish Ot Jr. - ishotjr
* Alex Waters - alexwaters
* Mihail Russu - MihailRussu * Mihail Russu - MihailRussu
* James Addison - jayaddison
* Joe Bauers - joebauers
* Stephan Oeste - Emzy