mirror of
https://github.com/bitcoinbook/bitcoinbook
synced 2024-11-22 16:18:11 +00:00
undo CR/LF line ending
This commit is contained in:
Andreas M. Antonopoulos
parent
908e9c2312
commit
51198bdd99
564
ch06.asciidoc
564
ch06.asciidoc
@ -1,282 +1,282 @@
|
||||
[[ch6]]
|
||||
== The Bitcoin Network
|
||||
|
||||
=== Introduction
|
||||
|
||||
=== Peer-to-Peer Network Architecture
|
||||
|
||||
Bitcoin is structured as a peer-to-peer network architecture on top of the Internet. The term peer-to-peer or P2P means that the computers that participate in the network are peers to each other, they are all equal, there are no "special" nodes and all nodes share the burden of providing network services. The network nodes interconnect in a mesh network with a "flat" topology. There is no "server", no centralized service, and no hierarchy within the network. Nodes in a peer-to-peer network both provide and consume services at the same time, with reciprocity acting as the incentive for participation. Peer-to-peer networks are inherently resilient, de-centralized, and open. The pre-eminent example of a P2P network architecture was the early Internet itself, where nodes on the IP network were equal. Today's Internet architecture is more hierarchical, but the Internet Protocol still retains its flat-topology essence. Beyond bitcoin, the largest and most successful application of P2P technologies is file sharing, with Napster as the pioneer and bittorrent as the most recent evolution of the architecture.
|
||||
|
||||
Bitcoin's P2P network architecture is much more than a topology choice. Bitcoin is a peer-to-peer digital cash system by design, and the network architecture is both a reflection and a foundation of that core characteristic. De-centralization of control is a core design principle and that can only be achieved and maintained by a flat, de-centralized P2P consensus network.
|
||||
|
||||
The term "bitcoin network" refers to the collection of nodes running the bitcoin P2P protocol. In addition to the bitcoin P2P protocol, there are other protocols such as Stratum, that are used for mining and lightweight or mobile wallets. These additional protocols are provided by gateway routing servers that access the bitcoin network using the bitcoin P2P protocol and then extend that network to nodes running other protocols. For example, Stratum servers connect Stratum mining nodes via the Stratum protocol to the main bitcoin network and bridge the Stratum protocol to the bitcoin P2P protocol. We use the term "extended bitcoin network" to refer to the overall network that includes the bitcoin P2P protocol, pool mining protocols, the Stratum protocol, and any other related protocols connecting the components of the bitcoin system.
|
||||
|
||||
=== Nodes Types and Roles
|
||||
|
||||
While nodes in the bitcoin P2P network are equal, they may take on different "roles", depending on the functionality they are supporting. A bitcoin node is a collection of functions: routing, the blockchain database, mining, and wallet services. A full node with all four of these functions is shown below:
|
||||
|
||||
[[full_node_reference]]
|
||||
.A bitcoin network node with all four functions: Network routing, Blockchain database, Mining and Wallet
|
||||
image::images/FullNodeReferenceClient_Small.png["FullNodeReferenceClient_Small"]
|
||||
|
||||
All nodes include the routing function to participate in the network and may include other functionality. All nodes validate and propagate transactions and blocks, discover and maintain connections to peers. In the full node example above, the routing function is indicated by an orange circle named "Network Routing Node".
|
||||
|
||||
Some nodes, called full nodes, also maintain a complete and up-to-date copy of the blockchain. Full nodes can autonomously and authoritatively verify any transaction without external reference. Some nodes maintain only a subset of the blockchain and verify transactions using a method called _Simple Payment Verification_ or SPV. These nodes are known as SPV or Lightweight nodes. In the full node example above, the full node blockchain database function is indicated by a blue circle named "Blockchain Database". SPV nodes are drawn without the blue circle, showing that they do not have a full copy of the blockchain.
|
||||
|
||||
Mining nodes compete to create new blocks by running specialized hardware to solve the proof-of-work algorithm. Some mining nodes are also full nodes, maintaining a full copy of the blockchain while others are lightweight nodes participating in pool mining and depending on a pool server to maintain a full node. The mining function is shown in the full node above as a black circle named "Mining".
|
||||
|
||||
User wallets may be part of a full node, as is usually the case with desktop bitcoin clients. Increasingly many user wallets, especially those running on resource constrained devices such as smart phones, are SPV nodes. The wallet function is shown above as a green circle named "Wallet".
|
||||
|
||||
In addition to the main node types on the bitcoin P2P protocol, there are servers and nodes running other protocols, such as specialized mining pool protocols and lightweight client access protocols.
|
||||
|
||||
Here are the most common node types on the extended bitcoin network:
|
||||
|
||||
[[node_type_ledgend]]
|
||||
.Different types of nodes on the extended bitcoin network
|
||||
image::images/BitcoinNodeTypes.png["BitcoinNodeTypes"]
|
||||
|
||||
=== The Extended Bitcoin Network
|
||||
|
||||
The main bitcoin network, running the bitcoin P2P protocol consists of between 7,000 to 10,000 nodes running various versions of the bitcoin reference client (Bitcoin Core) and a few hundred nodes running various other implementations of the bitcoin P2P protocol, such as BitcoinJ, Libbitcoin, and btcd. A small percentage of the nodes on the bitcoin P2P network are also mining nodes, competing in the mining process, validating transactions, and creating new blocks. Various large companies interface with the bitcoin network by running full-node clients based on the Bitcoin Core client, with full copies of the blockchain and a network node, but without mining or wallet functions. These nodes act as network edge routers, allowing various other services (exchanges, wallets, block explorers, merchant payment processing) to be built on top.
|
||||
|
||||
The extended bitcoin network includes the network running the bitcoin P2P protocol, described above, as well as nodes running specialized protocols. Attached to the main bitcoin P2P network are a number of pool servers and protocol gateways that connect nodes running other protocols, mostly pool mining nodes (see <<mining>>) and lightweight wallet clients, which do not carry a full copy of the blockchain.
|
||||
|
||||
The diagram below shows the extended bitcoin network with the various types of nodes, gateway servers, edge routers, and wallet clients and the various protocols they use to connect to each other.
|
||||
|
||||
[[bitcoin_network]]
|
||||
.The extended bitcoin network showing various node types, gateways and protocols
|
||||
image::images/BitcoinNetwork.png["BitcoinNetwork"]
|
||||
|
||||
=== Network Discovery
|
||||
|
||||
When a new node boots up, it must discover other bitcoin nodes on the network in order to participate. To start this process, a new node must discover at least one existing node on the network and connect to it. The geographic location of the other nodes is irrelevant, the bitcoin network topology is not geographically defined. Therefore, any existing bitcoin nodes can be selected at random.
|
||||
|
||||
To connect to a known peer, nodes establish a TCP connection, usually to port 8333 (the bitcoin "well known" port), or an alternative port if one is provided. Upon establishing a connection, the node will start a "handshake" by transmitting a +version+ message, which contains basic identifying information, including:
|
||||
|
||||
* PROTOCOL_VERSION, a constant that defines the bitcoin P2P protocol version the client "speaks". E.g. 70002
|
||||
* nLocalServices, a list of local services supported by the node, currently just NODE_NETWORK
|
||||
* nTime, the current time
|
||||
* addrYou, the IP address of the remote node as seen from this node
|
||||
* addrMe, the IP address of the local node, as discovered by the local node
|
||||
* subver, a sub-version showing the type of software running on this node, e.g. "/Satoshi:0.9.2.1/"
|
||||
* BestHeight, the block height of this node's blockchain
|
||||
|
||||
(See https://github.com/bitcoin/bitcoin/blob/d3cb2b8acfce36d359262b4afd7e7235eff106b0/src/net.cpp#L562 for an example of the +version+ network message)
|
||||
|
||||
The peer node responds with +verack+ to acknowledge and establish a connection, and optionally sends its own +version+ message if it wishes to reciprocate the connection and connect back as a peer.
|
||||
|
||||
[[network_handshake]]
|
||||
.The initial handshake between peers
|
||||
image::images/NetworkHandshake.png["NetworkHandshake"]
|
||||
|
||||
How does a new node find peers? While there are no special nodes in bitcoin, there are some long running stable nodes that are listed in the client as _seed nodes_. While a new node does not have to connect with the seed nodes, it can use them to quickly discover other nodes in the network. In the Bitcoin Core client, the option to use the seed nodes is controlled by the option switch +-dnsseed+, which is set to 1, to use the seed nodes, by default. Alternatively, a bootstrapping node that knows nothing of the network must be given the IP address of at least one bitcoin node after which it can establish connections through further introductions. The command line argument +-seednode+ can be used to connect to one node just for introductions, using it as a DNS seed. After the initial seed node is used to form introductions, the client will disconnect from it and use the newly discovered peers.
|
||||
|
||||
Once one or more connections is established, the new node will send an +addr+ message containing its own IP address, to its neighbors. The neighbors will in turn forward the +addr+ message to their neighbors, ensuring that the newly connected node becomes well known and better connected. Additionally, the newly connected node can send +getaddr+ to the neighbors asking them to return a list of IP addresses of other peers. That way, a node can find peers to connect to and advertise its existence on the network for other nodes to find it.
|
||||
|
||||
|
||||
[[address_propagation]]
|
||||
.Address Propagation and Discovery
|
||||
image::images/AddressPropagation.png["AddressPropagation"]
|
||||
|
||||
A node must connect to a few different peers in order to establish diverse paths into the bitcoin network. These paths are not reliable, nodes come and go, and so the node must continue to discover new nodes as it loses old connections as well as assist other nodes when they bootstrap. Only one connection is needed to bootstrap, as the first node can offer introductions to its peer nodes and those peers can offer further introductions. Its also unnecessary and wasteful of network resources to connect to more than a handful of nodes. After bootstrapping a node will remember its most recent successful peer connections, so that if it is rebooted it can quickly reestablish connections with its former peer network. If none of the former peers respond to its connection request, the node can use the seed nodes to bootstrap again.
|
||||
|
||||
On a node running the Bitcoin Core client, you can list the peer connections with the command +getpeerinfo+:
|
||||
----
|
||||
$ bitcoin-cli getpeerinfo
|
||||
[
|
||||
{
|
||||
"addr" : "85.213.199.39:8333",
|
||||
"services" : "00000001",
|
||||
"lastsend" : 1405634126,
|
||||
"lastrecv" : 1405634127,
|
||||
"bytessent" : 23487651,
|
||||
"bytesrecv" : 138679099,
|
||||
"conntime" : 1405021768,
|
||||
"pingtime" : 0.00000000,
|
||||
"version" : 70002,
|
||||
"subver" : "/Satoshi:0.9.2.1/",
|
||||
"inbound" : false,
|
||||
"startingheight" : 310131,
|
||||
"banscore" : 0,
|
||||
"syncnode" : true
|
||||
},
|
||||
{
|
||||
"addr" : "58.23.244.20:8333",
|
||||
"services" : "00000001",
|
||||
"lastsend" : 1405634127,
|
||||
"lastrecv" : 1405634124,
|
||||
"bytessent" : 4460918,
|
||||
"bytesrecv" : 8903575,
|
||||
"conntime" : 1405559628,
|
||||
"pingtime" : 0.00000000,
|
||||
"version" : 70001,
|
||||
"subver" : "/Satoshi:0.8.6/",
|
||||
"inbound" : false,
|
||||
"startingheight" : 311074,
|
||||
"banscore" : 0,
|
||||
"syncnode" : false
|
||||
}
|
||||
]
|
||||
----
|
||||
|
||||
To override the automatic management of peers and to specify a list of IP addresses, users can provide the option +-connect=<IPAddress>+ and specify one or more IP addresses. If this option is used, the node will only connect to the selected IP addresses, instead of discovering and maintaining the peer connections automatically.
|
||||
|
||||
If there is no traffic on a connection, nodes will periodically send a message to maintain the connection. If a node has not communicated on a connection for more than 90 minutes, it is assumed to be disconnected and a new peer will be sought. Thus the network dynamically adjusts to transient nodes, network problems, and can organically grow and shrink as needed without any central control.
|
||||
|
||||
=== Full Nodes
|
||||
|
||||
Full nodes are nodes that maintain a full blockchain, with all transactions. More accurately they probably should be called "full blockchain nodes". In the early years of bitcoin, all nodes were full nodes and currently the Bitcoin Core client is a full blockchain node. In the last two years however, new forms of bitcoin clients have been introduced, which do not maintain a full blockchain but run as lightweight clients. These are examined in more detail in the next section.
|
||||
|
||||
Full blockchain nodes maintain a complete and up-to-date copy of the bitcoin blockchain, with all the transactions, which they independently build and verify, starting with the very first block (genesis block) and up to the latest known block in the network. A full blockchain node can independently and authoritatively verify any transaction, without recourse or reliance on any other node or source of information. The full blockchain node relies on the network to receive updates about new blocks of transactions, which it then verifies and incorporates into its local copy of the blockchain.
|
||||
|
||||
Running a full blockchain node gives you the pure bitcoin experience: independent verification of all transactions without the need to rely on, or trust, any other systems. It's easy to tell if you're running a full node because it requires several gigabytes of persistent storage (disk space) to store the full blockchain. If you need a lot of disk and it takes 2-3 days to "sync" to the network, you are running a full node. That is the price of complete independence and freedom from central authority.
|
||||
|
||||
There are a few alternative implementations of full-blockchain bitcoin clients, built using different programming languages and software architectures. However, the most common implementation is the reference client Bitcoin Core, also known as the Satoshi Client. More than 90% of the nodes on the bitcoin network run various versions of Bitcoin Core. It is identified as "Satoshi" in the sub-version string sent in the +version+ message and shown by the command +getpeerinfo+ as we saw above, for example +/Satoshi:0.8.6/+.
|
||||
|
||||
=== Exchanging "Inventory"
|
||||
|
||||
The first thing a full node will do once it connects to peers is try to construct a complete blockchain. If it is a brand-new node and has no blockchain at all, then it only knows one block (the genesis block), which is statically embedded in the client software. Starting with block #0, the genesis block, the new node will have to download hundreds of thousands of blocks to synchronize with the network and establish a full blockchain.
|
||||
|
||||
The process of "syncing" the blockchain starts with the +version+ message, as that contains +BestHeight+, a node's current blockchain height (number of blocks). A node will see the +version+ messages from its peers, know how many blocks they each have and be able to compare to how many blocks it has in its own blockchain. Peered nodes will exchange a +getblocks+ message that contains the hash (fingerprint) of the top block on their local blockchain. One of the peers will be able to identify the received hash as belonging to a block that is not at the top, but rather belongs to an older block, thus deducing that its own local blockchain is longer than its peer's.
|
||||
|
||||
The peer that has the longer blockchain, has more blocks than the other node and can identify which blocks the other node needs to "catch up". It will identify the first 500 blocks to share and transmit their hashes using an +inv+ (inventory) message. The node missing these blocks will then retrieve them, by issuing a series of +getdata+ messages requesting the full block data and identifying the requested blocks using the hashes from the +inv+ message.
|
||||
|
||||
Let's assume for example that a node only has the genesis block. It will then receive an +inv+ message from its peers containing the hashes of the next 500 blocks in the chain. It will start requesting blocks from all of its connected peers, spreading the load and ensuring that it doesn't overwhelm any peer with requests. The node keeps track of how many blocks are "in transit" per peer connection, meaning blocks that it has requested but not received, checking that it does not exceed a limit (MAX_BLOCKS_IN_TRANSIT_PER_PEER). This way, if it needs a lot of blocks, it will only request new ones as previous requests are fulfilled, allowing the peers to control the pace of updates and not overwhelming the network. As each block is received, it is added to the blockchain as we will see in the next chapter <<blockchain>>. The local blockchain is gradually built up, more blocks are requested and received and the process continues until the node catches up to the rest of the network.
|
||||
|
||||
This process of comparing the local blockchain with the peers and retrieving any missing blocks happens any time a node goes offline for any period of time. Whether a node has been offline for a few minutes and is missing a few blocks, or a month and is missing a few thousand blocks, it starts by sending +getblocks+, gets an +inv+ response, and starts downloading the missing blocks.
|
||||
|
||||
[[inventory_synchronization]]
|
||||
.Node synchronizing the blockchain by retrieving blocks from a peer
|
||||
image::images/InventorySynchronization.png["InventorySynchronization"]
|
||||
|
||||
=== Simple Payment Verification (SPV) Nodes
|
||||
|
||||
Not all nodes have the ability to store the full blockchain. Many bitcoin clients are designed to run on space- and power-constrained devices, such as smartphones, tablets or embedded systems. For such devices, a _simple payment verification_ (SPV) method is used to allow them to operate without storing the full blockchain. These types of clients are called SPV clients or lightweight clients. As bitcoin adoption surges, the SPV node is becoming the most common form of bitcoin node, especially for bitcoin wallets.
|
||||
|
||||
SPV nodes download only the block headers and do not download the transactions included in each block. The resulting chain of blocks, without transactions, is 1,000 times smaller than the full blockchain. SPV nodes cannot construct a full picture of all the UTXOs that are available for spending, as they do not know about all the transactions on the network. SPV nodes verify transactions using a slightly different methodology that relies on peers to provide partial views of relevant parts of the blockchain on-demand.
|
||||
|
||||
As an analogy, a full node is like a tourist in a strange city, equipped with a detailed map of every street and every address. By comparison, an SPV node is like a tourist in a strange city asking random strangers for turn-by-turn directions while knowing only one main avenue. While both tourists can verify the existence of a street by visiting it, the tourist without a map doesn't know what lies down any of the side streets and doesn't know what other streets exist. Positioned in front of 23 Church Street, the tourist without a map cannot know if there are a dozen other "23 Church Street" addresses in the city and whether this is the right one. The map-less tourist's best chance is to ask enough people and hope some of them are not trying to mug the tourist.
|
||||
|
||||
Simple Payment Verification verifies transactions by reference to their _depth_ in the blockchain instead of their _height_. Whereas a full-blockchain node will construct a fully verified chain of thousands of blocks and transactions reaching down the blockchain (back in time) all the way to the genesis block, an SPV node will verify the chain of all blocks and link that chain to the transaction of interest.
|
||||
|
||||
For example, when examining a transaction in block 300,000, a full node links all 300,000 blocks down to the genesis block and builds a full database of UTXO, establishing the validity of the transaction by confirming that the UTXO remains unspent. An SPV node cannot validate whether the UTXO is unspent. Instead, the SPV node will establish a link between the transaction and the block that contains it, using a Merkle Path (see <<merkle_trees>>). Then, the SPV node waits until is sees the six blocks 300,001 through 300,006 piled on top of the block containing the transaction and verifies it by establishing its depth under blocks 300,006 to 300,001. The fact that other nodes on the network accepted block 300,000 and then did the necessary work to produce 6 more blocks on top of it is proof, by proxy, that the transaction was not a double-spend.
|
||||
|
||||
An SPV node cannot be persuaded that a transaction exists in a block, when it does not in fact exist. The SPV node establishes the existence of a transaction in a block by requesting a merkle path proof and by validating the proof-of-work in the chain of blocks. However, a transaction's existence can be "hidden" from an SPV node. An SPV node can definitely prove that a transaction exists but cannot verify that a transaction, such as a double-spend of the same UTXO, doesn't exist because it doesn't have a record of all transactions. This type of attack can be used as a Denial-of-Service attack or as a double-spending attack against SPV nodes. To defend against this, an SPV node needs to connect randomly to several nodes, to increase the probability that it is in contact with at least one honest node. SPV nodes are therefore vulnerable to network partitioning attacks or Sybil attacks, where they are connected to fake nodes or fake networks and do not have access to honest nodes or the real bitcoin network.
|
||||
|
||||
For most practical purposes, well-connected SPV nodes are secure enough, striking the right balance between resource needs, practicality, and security. For the truly security conscious, however, nothing beats running a full blockchain node.
|
||||
|
||||
[TIP]
|
||||
====
|
||||
A full blockchain node verifies a transaction by checking the chain of thousands of blocks below it and checks that the UTXO is not spent, whereas an SPV node checks how deep the block is buried by a handful of blocks above it.
|
||||
====
|
||||
|
||||
To get the block headers, SPV nodes use a +getheaders+ message instead of +getblocks+. The responding peer will send up to 2000 block headers using a single +headers+ message. The process is otherwise the same as that used by a full node to retrieve full blocks. SPV nodes also set a filter on the connection to peers, to filter the stream of future blocks and transactions sent by the peers. Any transactions of interest are retrieved using a +getdata+ request. The peer generates a +tx+ message containing the transactions, in response.
|
||||
|
||||
[[spv_synchronization]]
|
||||
.SPV Node synchronizing the block headers
|
||||
image::images/SPVSynchronization.png["SPVSynchronization"]
|
||||
|
||||
Because SPV nodes need to retrieve specific transactions in order to selectively verify them, they also create a privacy risk. Unlike full-blockchain nodes, which collect all transactions within each block, the SPV node's requests for specific data can inadvertently reveal the addresses in their wallet. For example, a third party monitoring a network could keep track of all the transactions requested by a wallet on an SPV node and use those to associate bitcoin addresses with the user of that wallet, destroying the user's privacy.
|
||||
|
||||
Shortly after the introduction of SPV/lightweight nodes, the bitcoin developers added a feature called _bloom filters_ to address the privacy risks of SPV nodes. Bloom filters allow SPV nodes to receive a subset of the transactions without revealing precisely which addresses they are interested in, through a filtering mechanism that uses probabilities rather than fixed patterns.
|
||||
|
||||
=== Bloom Filters
|
||||
|
||||
A bloom filter is a probabilistic search filter, a way to describe a desired pattern without specifying it exactly. Bloom filters offer an efficient way to express a search pattern while protecting privacy. They are used by SPV nodes to ask their peers for transactions matching a specific pattern, without revealing exactly which addresses they are searching for.
|
||||
|
||||
Using our previous analogy of a tourist without a map asking for directions to a specific address "23 Church St". If they ask strangers for directions to this street, they inadvertently reveal their destination. A bloom filter is like asking "Are there any streets in this neighborhood whose name ends in R-C-H". A question like that reveals slightly less about the desired destination, than asking for "23 Church St". Using this technique, a tourist could specify the desired address in more detail as "ending in U-R-C-H" or less detail as "ending in H". By varying the precision of the search, the tourist reveals more or less information, at the expense of getting more or less specific results. If they ask a less specific pattern, they get a lot more possible addresses and better privacy but many of the results are irrelevant. If they ask for a very specific pattern then they get fewer results but they lose privacy.
|
||||
|
||||
Bloom filters serve this function by allowing an SPV node to specify a search pattern for transactions that can be tuned towards precision or privacy. A more specific bloom filter will produce accurate results, but at the expense of revealing what addresses are used in the user's wallet. A less specific bloom filter will produce more data about more transactions, many irrelevant to the node, but will allow the node to maintain better privacy.
|
||||
|
||||
An SPV node will initialize a bloom filter as "empty" and in that state the bloom filter will not match any patterns. The SPV node will then make a list of all the addresses in its wallet and create a search pattern matching the transaction output that corresponds to each address. Usually, the search pattern is a Pay-to-Public-Key-Hash script that is the expected locking script that will be present in any transaction paying to the public-key-hash (address). If the SPV node is tracking the balance of a P2SH address, then the search pattern will be a Pay-to-Script-Hash script, instead. The SPV node then adds each of the search patterns to the bloom filter, so that the bloom filter can recognize the search pattern if it is present in a transaction. Finally, the bloom filter is sent to the peer and the peer uses it to match transactions for transmission to the SPV node.
|
||||
|
||||
Bloom filters are implemented as a variable-size array of N binary digits (a bit field) and a variable number of M hash functions. The hash functions are designed to always produce an output that is between 1 and N, corresponding to the array of binary digits. The hash functions are generated deterministically, so that any node implementing a bloom filter will always use the same hash functions and get the same results for a specific input. By choosing different length (N) bloom filters and a different number (M) of hash functions, the bloom filter can be tuned, varying the level of accuracy and therefore privacy.
|
||||
|
||||
In the example below, we use a very small array of 16 bits and a set of 3 hash functions to demonstrate how bloom filters work.
|
||||
|
||||
[[bloom1]]
|
||||
.An example of a simplistic bloom filter, with 16 bit field and 3 hash functions
|
||||
image::images/Bloom1.png["Bloom1"]
|
||||
|
||||
The bloom filter is initialized so that the array of bits is all zeros. To add a pattern to the bloom filter, the pattern is hashed by each hash function in turn. Applying the first hash function to the input results in a number between 1 and N. The corresponding bit in the array (indexed from 1 to N) is found and set to +1+, thereby recording the output of the hash function. Then, the next hash function is used to set another bit and so on and so forth. Once all M hash functions have been applied, the search pattern will be "recorded" in the bloom filter as M bits have been changed from +0+ to +1+.
|
||||
|
||||
Here's an example of adding a pattern "A" to the simple bloom filter shown above:
|
||||
|
||||
[[bloom2]]
|
||||
.Adding a pattern "A" to our simple bloom filter
|
||||
image::images/Bloom2.png["Bloom2"]
|
||||
|
||||
Adding a second pattern is as simple as repeating this process. The pattern is hashed by each hash function in turn and the result is recorded by setting the bits to +1+. Note that as a bloom filter is filled with more patterns, a hash function result may coincide with a bit that is already set to +1+ in which case the bit is not changed. In essence, as more patterns record on overlapping bits, the bloom filter starts to become saturated with more bits set to +1+ and the accuracy of the filter decreases. This is why the filter is a probabilistic data structure - it gets less accurate as more patterns are added. The accuracy depends on the number of patterns added versus the size of the bit array (N) and number of hash functions (M). A larger bit array and more hash functions can record more patterns with higher accuracy. A smaller bit array or fewer hash functions will record fewer patterns and produce less accuracy.
|
||||
|
||||
Below is an example of adding a second pattern "B" to the simple bloom filter:
|
||||
|
||||
[[bloom3]]
|
||||
.Adding a second pattern "B" to our simple bloom filter
|
||||
image::images/Bloom3.png["Bloom3"]
|
||||
|
||||
To test if a pattern is part of a bloom filter, the pattern is hashed by each hash function and the resulting bit pattern is tested against the bit array. If all the bits indexed by the hash functions are set to +1+, then the patten is _probably_ recorded in the bloom filter. Since the bits may be set because of overlap from multiple patterns, the answer is not certain, but is rather probabilistic. In simple terms, a bloom filter positive match is a "Maybe, Yes".
|
||||
|
||||
Below is an example of testing the existence of pattern "X" in the simple bloom filter. The corresponding bits are set to +1+, so the pattern is probably a match:
|
||||
|
||||
[[bloom4]]
|
||||
.Testing the existence of pattern "X" in the bloom filter. The result is probabilistic positive match, meaning "Maybe"
|
||||
image::images/Bloom4.png["Bloom4"]
|
||||
|
||||
On the contrary, if a pattern is tested against the bloom filter and any one of the bits is set to +0+, then this proves that the pattern was not recorded in the bloom filter. A negative result is not a probability, it is a certainty. In simple terms, a negative match on a bloom filter is a "Definitely No".
|
||||
|
||||
Below is an example of testing the existence of pattern "Y" in the simple bloom filter. One of the corresponding bits is set to +0+, so the pattern is definitely not a match:
|
||||
|
||||
[[bloom5]]
|
||||
.Testing the existence of pattern "Y" in the bloom filter. The result is a definitive negative match, meaning "Definitely No"
|
||||
image::images/Bloom5.png["Bloom5"]
|
||||
|
||||
Bitcoin's implementation of bloom filters is described in Bitcoin Improvement Proposal 37 (BIP0037). See <<bip0037>> or visit:
|
||||
https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki
|
||||
|
||||
=== Bloom Filters and Inventory Updates
|
||||
|
||||
Bloom filters are used to filter the transactions (and blocks containing them) that an SPV node receives from its peers. SPV nodes will create a filter that matches only the addresses held in the SPV node's wallet. The SPV node will then send a +filterload+ message to the peer, containing the bloom filter to use on the connection. After a filter is established, the peer will then test each transaction's outputs against the bloom filter. Only transactions which match the filter are sent to the node.
|
||||
|
||||
In response to a +getdata+ message from the node, peers will send a +merkleblock+ message that contains only block headers for blocks matching the filter and a merkle path (See <<merkle_trees>>) for each matching transaction. The peer will also then send +tx+ messages containing the transactions matched by the filter.
|
||||
|
||||
The node setting the bloom filter can interactively add patterns to the filter by sending a +filteradd+ message. To clear the bloom filter, the node can send a +filterclear+ message. Since it is not possible to remove a pattern from a bloom filter, a node has to clear and re-send a new bloom filter if a pattern is no longer desired.
|
||||
|
||||
[[transaction_pools]]
|
||||
=== Transaction Pools
|
||||
|
||||
Almost every node on the bitcoin network maintains a temporary list of unconfirmed transactions called the memory pool or transaction pool. Nodes use this pool to keep track of transactions that are known to the network but are not yet included in the blockchain. For example, a node that holds a user's wallet will use the transaction pool to track incoming payments to the user's wallet that have been received on the network but are not yet confirmed.
|
||||
|
||||
As transactions are received and verified, they are added to the transaction pool and relayed to the neighboring nodes to propagate on the network.
|
||||
|
||||
Some node implementations also maintain a separate pool of orphaned transactions as detailed in <<orphan_transactions>>. If a transaction's inputs refer to a transaction that is not yet known, a missing parent, then the orphan transaction will be stored temporarily in the orphan pool until the parent transaction arrives.
|
||||
|
||||
When a transaction is added to the transaction pool, the orphan pool is checked for any orphans that reference this transaction's outputs (its children). Any orphans found are pulled from the orphan pool and validated using the above checklist. If valid, they are also added to the transaction pool, completing the chain that started with the parent transaction. In light of the newly added transaction which is no longer an orphan, the process is repeated recursively looking for any further descendants, until no more descendants are found. Through this process, the arrival of a parent transaction triggers a cascade reconstruction of an entire chain of interdependent transactions by re-uniting the orphans with their parents all the way down the chain.
|
||||
|
||||
Both the transaction pool and orphan pool (where implemented) are stored in local memory and are not saved on persistent storage, rather they are dynamically populated from incoming network messages. When a node starts, both pools are empty and are gradually populated with new transactions received on the network.
|
||||
|
||||
Some implementations of the bitcoin client also maintain a UTXO database or UTXO pool which is the set of all unspent outputs on the blockchain. While the name "UTXO pool" sounds similar to the transaction pool, it represents a different set of data. Unlike the transaction and orphan pools, the UTXO pool is not initialized empty but instead contains millions of entries of unspent transaction outputs including some dating back to 2009. The UTXO pool may be housed in local memory or as an indexed database table on persistent storage.
|
||||
|
||||
Whereas the transaction and orphan pools represent a single node's local perspective and may vary significantly from node to node depending upon when the node was started or restarted, the UTXO pool represents the emergent consensus of the network and therefore will vary little between nodes. Furthermore the transaction and orphan pools only contain unconfirmed transactions, while the UTXO pool only contains confirmed outputs.
|
||||
|
||||
=== Alert Messages
|
||||
|
||||
Alert messages are a seldom used function, which is nevertheless implemented in most nodes. Alert messages are bitcoin's "emergency broadcast system", a means by which the core bitcoin developers can send an emergency text message to all bitcoin nodes. This feature is implemented to allow the core developer team to notify all bitcoin users of a serious problem in the bitcoin network, such as a critical bug that requires user action. The alert system has only been used a handful of times, most notably in the Spring of 2013 when a critical database bug caused a multi-block fork to occur in the bitcoin blockchain.
|
||||
|
||||
Alert messages are propagated by the +alert+ message. The alert message contains several fields, including:
|
||||
|
||||
* ID - An alert identified so that duplicate alerts can be detected
|
||||
* Expiration - a time after which the alert expires
|
||||
* RelayUntil - a time after which the alert should not be relayed
|
||||
* MinVer, MaxVer - the range of bitcoin protocol versions that this alert applies to
|
||||
* subVer - The client software version that this alert applies to
|
||||
* Priority - An alert priority level, currently unused
|
||||
|
||||
Alerts are cryptographically signed by a public key. The corresponding private key is held by a few selected members of the core development team. The digital signature ensures that fake alerts will not be propagated on the network.
|
||||
|
||||
Each node receiving this alert message will verify it, check for expiration, and propagate it to all its peers, thus ensuring rapid propagation across the entire network. In addition to propagating the alert, each node may implement a user interface function to present the alert to the user.
|
||||
|
||||
In the Bitcoin Core client, the alert is configured with the command line option +-alertnotify+, which specifies a command to run when an alert is received. The alert message is passed as a parameter to the alertnotify command. Most commonly, the alertnotify command is set to generate an email message to the administrator of the node, containing the alert message. The alert is also displayed as a pop-up dialog in the graphical user interface (bitcoin-Qt) if it is running.
|
||||
|
||||
Other implementations of the bitcoin protocol may handle the alert in different ways. Many hardware-embedded bitcoin mining systems do not implement the alert message function, as they have no user interface. It is strongly recommended that miners running such mining systems subscribe to alerts via a mining pool operator or by running a lightweight node just for alert purposes.
|
||||
|
||||
|
||||
[[ch6]]
|
||||
== The Bitcoin Network
|
||||
|
||||
=== Introduction
|
||||
|
||||
=== Peer-to-Peer Network Architecture
|
||||
|
||||
Bitcoin is structured as a peer-to-peer network architecture on top of the Internet. The term peer-to-peer or P2P means that the computers that participate in the network are peers to each other, they are all equal, there are no "special" nodes and all nodes share the burden of providing network services. The network nodes interconnect in a mesh network with a "flat" topology. There is no "server", no centralized service, and no hierarchy within the network. Nodes in a peer-to-peer network both provide and consume services at the same time, with reciprocity acting as the incentive for participation. Peer-to-peer networks are inherently resilient, de-centralized, and open. The pre-eminent example of a P2P network architecture was the early Internet itself, where nodes on the IP network were equal. Today's Internet architecture is more hierarchical, but the Internet Protocol still retains its flat-topology essence. Beyond bitcoin, the largest and most successful application of P2P technologies is file sharing, with Napster as the pioneer and bittorrent as the most recent evolution of the architecture.
|
||||
|
||||
Bitcoin's P2P network architecture is much more than a topology choice. Bitcoin is a peer-to-peer digital cash system by design, and the network architecture is both a reflection and a foundation of that core characteristic. De-centralization of control is a core design principle and that can only be achieved and maintained by a flat, de-centralized P2P consensus network.
|
||||
|
||||
The term "bitcoin network" refers to the collection of nodes running the bitcoin P2P protocol. In addition to the bitcoin P2P protocol, there are other protocols such as Stratum, that are used for mining and lightweight or mobile wallets. These additional protocols are provided by gateway routing servers that access the bitcoin network using the bitcoin P2P protocol and then extend that network to nodes running other protocols. For example, Stratum servers connect Stratum mining nodes via the Stratum protocol to the main bitcoin network and bridge the Stratum protocol to the bitcoin P2P protocol. We use the term "extended bitcoin network" to refer to the overall network that includes the bitcoin P2P protocol, pool mining protocols, the Stratum protocol, and any other related protocols connecting the components of the bitcoin system.
|
||||
|
||||
=== Nodes Types and Roles
|
||||
|
||||
While nodes in the bitcoin P2P network are equal, they may take on different "roles", depending on the functionality they are supporting. A bitcoin node is a collection of functions: routing, the blockchain database, mining, and wallet services. A full node with all four of these functions is shown below:
|
||||
|
||||
[[full_node_reference]]
|
||||
.A bitcoin network node with all four functions: Network routing, Blockchain database, Mining and Wallet
|
||||
image::images/FullNodeReferenceClient_Small.png["FullNodeReferenceClient_Small"]
|
||||
|
||||
All nodes include the routing function to participate in the network and may include other functionality. All nodes validate and propagate transactions and blocks, discover and maintain connections to peers. In the full node example above, the routing function is indicated by an orange circle named "Network Routing Node".
|
||||
|
||||
Some nodes, called full nodes, also maintain a complete and up-to-date copy of the blockchain. Full nodes can autonomously and authoritatively verify any transaction without external reference. Some nodes maintain only a subset of the blockchain and verify transactions using a method called _Simple Payment Verification_ or SPV. These nodes are known as SPV or Lightweight nodes. In the full node example above, the full node blockchain database function is indicated by a blue circle named "Blockchain Database". SPV nodes are drawn without the blue circle, showing that they do not have a full copy of the blockchain.
|
||||
|
||||
Mining nodes compete to create new blocks by running specialized hardware to solve the proof-of-work algorithm. Some mining nodes are also full nodes, maintaining a full copy of the blockchain while others are lightweight nodes participating in pool mining and depending on a pool server to maintain a full node. The mining function is shown in the full node above as a black circle named "Mining".
|
||||
|
||||
User wallets may be part of a full node, as is usually the case with desktop bitcoin clients. Increasingly many user wallets, especially those running on resource constrained devices such as smart phones, are SPV nodes. The wallet function is shown above as a green circle named "Wallet".
|
||||
|
||||
In addition to the main node types on the bitcoin P2P protocol, there are servers and nodes running other protocols, such as specialized mining pool protocols and lightweight client access protocols.
|
||||
|
||||
Here are the most common node types on the extended bitcoin network:
|
||||
|
||||
[[node_type_ledgend]]
|
||||
.Different types of nodes on the extended bitcoin network
|
||||
image::images/BitcoinNodeTypes.png["BitcoinNodeTypes"]
|
||||
|
||||
=== The Extended Bitcoin Network
|
||||
|
||||
The main bitcoin network, running the bitcoin P2P protocol consists of between 7,000 to 10,000 nodes running various versions of the bitcoin reference client (Bitcoin Core) and a few hundred nodes running various other implementations of the bitcoin P2P protocol, such as BitcoinJ, Libbitcoin, and btcd. A small percentage of the nodes on the bitcoin P2P network are also mining nodes, competing in the mining process, validating transactions, and creating new blocks. Various large companies interface with the bitcoin network by running full-node clients based on the Bitcoin Core client, with full copies of the blockchain and a network node, but without mining or wallet functions. These nodes act as network edge routers, allowing various other services (exchanges, wallets, block explorers, merchant payment processing) to be built on top.
|
||||
|
||||
The extended bitcoin network includes the network running the bitcoin P2P protocol, described above, as well as nodes running specialized protocols. Attached to the main bitcoin P2P network are a number of pool servers and protocol gateways that connect nodes running other protocols, mostly pool mining nodes (see <<mining>>) and lightweight wallet clients, which do not carry a full copy of the blockchain.
|
||||
|
||||
The diagram below shows the extended bitcoin network with the various types of nodes, gateway servers, edge routers, and wallet clients and the various protocols they use to connect to each other.
|
||||
|
||||
[[bitcoin_network]]
|
||||
.The extended bitcoin network showing various node types, gateways and protocols
|
||||
image::images/BitcoinNetwork.png["BitcoinNetwork"]
|
||||
|
||||
=== Network Discovery
|
||||
|
||||
When a new node boots up, it must discover other bitcoin nodes on the network in order to participate. To start this process, a new node must discover at least one existing node on the network and connect to it. The geographic location of the other nodes is irrelevant, the bitcoin network topology is not geographically defined. Therefore, any existing bitcoin nodes can be selected at random.
|
||||
|
||||
To connect to a known peer, nodes establish a TCP connection, usually to port 8333 (the bitcoin "well known" port), or an alternative port if one is provided. Upon establishing a connection, the node will start a "handshake" by transmitting a +version+ message, which contains basic identifying information, including:
|
||||
|
||||
* PROTOCOL_VERSION, a constant that defines the bitcoin P2P protocol version the client "speaks". E.g. 70002
|
||||
* nLocalServices, a list of local services supported by the node, currently just NODE_NETWORK
|
||||
* nTime, the current time
|
||||
* addrYou, the IP address of the remote node as seen from this node
|
||||
* addrMe, the IP address of the local node, as discovered by the local node
|
||||
* subver, a sub-version showing the type of software running on this node, e.g. "/Satoshi:0.9.2.1/"
|
||||
* BestHeight, the block height of this node's blockchain
|
||||
|
||||
(See https://github.com/bitcoin/bitcoin/blob/d3cb2b8acfce36d359262b4afd7e7235eff106b0/src/net.cpp#L562 for an example of the +version+ network message)
|
||||
|
||||
The peer node responds with +verack+ to acknowledge and establish a connection, and optionally sends its own +version+ message if it wishes to reciprocate the connection and connect back as a peer.
|
||||
|
||||
[[network_handshake]]
|
||||
.The initial handshake between peers
|
||||
image::images/NetworkHandshake.png["NetworkHandshake"]
|
||||
|
||||
How does a new node find peers? While there are no special nodes in bitcoin, there are some long running stable nodes that are listed in the client as _seed nodes_. While a new node does not have to connect with the seed nodes, it can use them to quickly discover other nodes in the network. In the Bitcoin Core client, the option to use the seed nodes is controlled by the option switch +-dnsseed+, which is set to 1, to use the seed nodes, by default. Alternatively, a bootstrapping node that knows nothing of the network must be given the IP address of at least one bitcoin node after which it can establish connections through further introductions. The command line argument +-seednode+ can be used to connect to one node just for introductions, using it as a DNS seed. After the initial seed node is used to form introductions, the client will disconnect from it and use the newly discovered peers.
|
||||
|
||||
Once one or more connections is established, the new node will send an +addr+ message containing its own IP address, to its neighbors. The neighbors will in turn forward the +addr+ message to their neighbors, ensuring that the newly connected node becomes well known and better connected. Additionally, the newly connected node can send +getaddr+ to the neighbors asking them to return a list of IP addresses of other peers. That way, a node can find peers to connect to and advertise its existence on the network for other nodes to find it.
|
||||
|
||||
|
||||
[[address_propagation]]
|
||||
.Address Propagation and Discovery
|
||||
image::images/AddressPropagation.png["AddressPropagation"]
|
||||
|
||||
A node must connect to a few different peers in order to establish diverse paths into the bitcoin network. These paths are not reliable, nodes come and go, and so the node must continue to discover new nodes as it loses old connections as well as assist other nodes when they bootstrap. Only one connection is needed to bootstrap, as the first node can offer introductions to its peer nodes and those peers can offer further introductions. Its also unnecessary and wasteful of network resources to connect to more than a handful of nodes. After bootstrapping a node will remember its most recent successful peer connections, so that if it is rebooted it can quickly reestablish connections with its former peer network. If none of the former peers respond to its connection request, the node can use the seed nodes to bootstrap again.
|
||||
|
||||
On a node running the Bitcoin Core client, you can list the peer connections with the command +getpeerinfo+:
|
||||
----
|
||||
$ bitcoin-cli getpeerinfo
|
||||
[
|
||||
{
|
||||
"addr" : "85.213.199.39:8333",
|
||||
"services" : "00000001",
|
||||
"lastsend" : 1405634126,
|
||||
"lastrecv" : 1405634127,
|
||||
"bytessent" : 23487651,
|
||||
"bytesrecv" : 138679099,
|
||||
"conntime" : 1405021768,
|
||||
"pingtime" : 0.00000000,
|
||||
"version" : 70002,
|
||||
"subver" : "/Satoshi:0.9.2.1/",
|
||||
"inbound" : false,
|
||||
"startingheight" : 310131,
|
||||
"banscore" : 0,
|
||||
"syncnode" : true
|
||||
},
|
||||
{
|
||||
"addr" : "58.23.244.20:8333",
|
||||
"services" : "00000001",
|
||||
"lastsend" : 1405634127,
|
||||
"lastrecv" : 1405634124,
|
||||
"bytessent" : 4460918,
|
||||
"bytesrecv" : 8903575,
|
||||
"conntime" : 1405559628,
|
||||
"pingtime" : 0.00000000,
|
||||
"version" : 70001,
|
||||
"subver" : "/Satoshi:0.8.6/",
|
||||
"inbound" : false,
|
||||
"startingheight" : 311074,
|
||||
"banscore" : 0,
|
||||
"syncnode" : false
|
||||
}
|
||||
]
|
||||
----
|
||||
|
||||
To override the automatic management of peers and to specify a list of IP addresses, users can provide the option +-connect=<IPAddress>+ and specify one or more IP addresses. If this option is used, the node will only connect to the selected IP addresses, instead of discovering and maintaining the peer connections automatically.
|
||||
|
||||
If there is no traffic on a connection, nodes will periodically send a message to maintain the connection. If a node has not communicated on a connection for more than 90 minutes, it is assumed to be disconnected and a new peer will be sought. Thus the network dynamically adjusts to transient nodes, network problems, and can organically grow and shrink as needed without any central control.
|
||||
|
||||
=== Full Nodes
|
||||
|
||||
Full nodes are nodes that maintain a full blockchain, with all transactions. More accurately they probably should be called "full blockchain nodes". In the early years of bitcoin, all nodes were full nodes and currently the Bitcoin Core client is a full blockchain node. In the last two years however, new forms of bitcoin clients have been introduced, which do not maintain a full blockchain but run as lightweight clients. These are examined in more detail in the next section.
|
||||
|
||||
Full blockchain nodes maintain a complete and up-to-date copy of the bitcoin blockchain, with all the transactions, which they independently build and verify, starting with the very first block (genesis block) and up to the latest known block in the network. A full blockchain node can independently and authoritatively verify any transaction, without recourse or reliance on any other node or source of information. The full blockchain node relies on the network to receive updates about new blocks of transactions, which it then verifies and incorporates into its local copy of the blockchain.
|
||||
|
||||
Running a full blockchain node gives you the pure bitcoin experience: independent verification of all transactions without the need to rely on, or trust, any other systems. It's easy to tell if you're running a full node because it requires several gigabytes of persistent storage (disk space) to store the full blockchain. If you need a lot of disk and it takes 2-3 days to "sync" to the network, you are running a full node. That is the price of complete independence and freedom from central authority.
|
||||
|
||||
There are a few alternative implementations of full-blockchain bitcoin clients, built using different programming languages and software architectures. However, the most common implementation is the reference client Bitcoin Core, also known as the Satoshi Client. More than 90% of the nodes on the bitcoin network run various versions of Bitcoin Core. It is identified as "Satoshi" in the sub-version string sent in the +version+ message and shown by the command +getpeerinfo+ as we saw above, for example +/Satoshi:0.8.6/+.
|
||||
|
||||
=== Exchanging "Inventory"
|
||||
|
||||
The first thing a full node will do once it connects to peers is try to construct a complete blockchain. If it is a brand-new node and has no blockchain at all, then it only knows one block (the genesis block), which is statically embedded in the client software. Starting with block #0, the genesis block, the new node will have to download hundreds of thousands of blocks to synchronize with the network and establish a full blockchain.
|
||||
|
||||
The process of "syncing" the blockchain starts with the +version+ message, as that contains +BestHeight+, a node's current blockchain height (number of blocks). A node will see the +version+ messages from its peers, know how many blocks they each have and be able to compare to how many blocks it has in its own blockchain. Peered nodes will exchange a +getblocks+ message that contains the hash (fingerprint) of the top block on their local blockchain. One of the peers will be able to identify the received hash as belonging to a block that is not at the top, but rather belongs to an older block, thus deducing that its own local blockchain is longer than its peer's.
|
||||
|
||||
The peer that has the longer blockchain, has more blocks than the other node and can identify which blocks the other node needs to "catch up". It will identify the first 500 blocks to share and transmit their hashes using an +inv+ (inventory) message. The node missing these blocks will then retrieve them, by issuing a series of +getdata+ messages requesting the full block data and identifying the requested blocks using the hashes from the +inv+ message.
|
||||
|
||||
Let's assume for example that a node only has the genesis block. It will then receive an +inv+ message from its peers containing the hashes of the next 500 blocks in the chain. It will start requesting blocks from all of its connected peers, spreading the load and ensuring that it doesn't overwhelm any peer with requests. The node keeps track of how many blocks are "in transit" per peer connection, meaning blocks that it has requested but not received, checking that it does not exceed a limit (MAX_BLOCKS_IN_TRANSIT_PER_PEER). This way, if it needs a lot of blocks, it will only request new ones as previous requests are fulfilled, allowing the peers to control the pace of updates and not overwhelming the network. As each block is received, it is added to the blockchain as we will see in the next chapter <<blockchain>>. The local blockchain is gradually built up, more blocks are requested and received and the process continues until the node catches up to the rest of the network.
|
||||
|
||||
This process of comparing the local blockchain with the peers and retrieving any missing blocks happens any time a node goes offline for any period of time. Whether a node has been offline for a few minutes and is missing a few blocks, or a month and is missing a few thousand blocks, it starts by sending +getblocks+, gets an +inv+ response, and starts downloading the missing blocks.
|
||||
|
||||
[[inventory_synchronization]]
|
||||
.Node synchronizing the blockchain by retrieving blocks from a peer
|
||||
image::images/InventorySynchronization.png["InventorySynchronization"]
|
||||
|
||||
=== Simple Payment Verification (SPV) Nodes
|
||||
|
||||
Not all nodes have the ability to store the full blockchain. Many bitcoin clients are designed to run on space- and power-constrained devices, such as smartphones, tablets or embedded systems. For such devices, a _simple payment verification_ (SPV) method is used to allow them to operate without storing the full blockchain. These types of clients are called SPV clients or lightweight clients. As bitcoin adoption surges, the SPV node is becoming the most common form of bitcoin node, especially for bitcoin wallets.
|
||||
|
||||
SPV nodes download only the block headers and do not download the transactions included in each block. The resulting chain of blocks, without transactions, is 1,000 times smaller than the full blockchain. SPV nodes cannot construct a full picture of all the UTXOs that are available for spending, as they do not know about all the transactions on the network. SPV nodes verify transactions using a slightly different methodology that relies on peers to provide partial views of relevant parts of the blockchain on-demand.
|
||||
|
||||
As an analogy, a full node is like a tourist in a strange city, equipped with a detailed map of every street and every address. By comparison, an SPV node is like a tourist in a strange city asking random strangers for turn-by-turn directions while knowing only one main avenue. While both tourists can verify the existence of a street by visiting it, the tourist without a map doesn't know what lies down any of the side streets and doesn't know what other streets exist. Positioned in front of 23 Church Street, the tourist without a map cannot know if there are a dozen other "23 Church Street" addresses in the city and whether this is the right one. The map-less tourist's best chance is to ask enough people and hope some of them are not trying to mug the tourist.
|
||||
|
||||
Simple Payment Verification verifies transactions by reference to their _depth_ in the blockchain instead of their _height_. Whereas a full-blockchain node will construct a fully verified chain of thousands of blocks and transactions reaching down the blockchain (back in time) all the way to the genesis block, an SPV node will verify the chain of all blocks and link that chain to the transaction of interest.
|
||||
|
||||
For example, when examining a transaction in block 300,000, a full node links all 300,000 blocks down to the genesis block and builds a full database of UTXO, establishing the validity of the transaction by confirming that the UTXO remains unspent. An SPV node cannot validate whether the UTXO is unspent. Instead, the SPV node will establish a link between the transaction and the block that contains it, using a Merkle Path (see <<merkle_trees>>). Then, the SPV node waits until is sees the six blocks 300,001 through 300,006 piled on top of the block containing the transaction and verifies it by establishing its depth under blocks 300,006 to 300,001. The fact that other nodes on the network accepted block 300,000 and then did the necessary work to produce 6 more blocks on top of it is proof, by proxy, that the transaction was not a double-spend.
|
||||
|
||||
An SPV node cannot be persuaded that a transaction exists in a block, when it does not in fact exist. The SPV node establishes the existence of a transaction in a block by requesting a merkle path proof and by validating the proof-of-work in the chain of blocks. However, a transaction's existence can be "hidden" from an SPV node. An SPV node can definitely prove that a transaction exists but cannot verify that a transaction, such as a double-spend of the same UTXO, doesn't exist because it doesn't have a record of all transactions. This type of attack can be used as a Denial-of-Service attack or as a double-spending attack against SPV nodes. To defend against this, an SPV node needs to connect randomly to several nodes, to increase the probability that it is in contact with at least one honest node. SPV nodes are therefore vulnerable to network partitioning attacks or Sybil attacks, where they are connected to fake nodes or fake networks and do not have access to honest nodes or the real bitcoin network.
|
||||
|
||||
For most practical purposes, well-connected SPV nodes are secure enough, striking the right balance between resource needs, practicality, and security. For the truly security conscious, however, nothing beats running a full blockchain node.
|
||||
|
||||
[TIP]
|
||||
====
|
||||
A full blockchain node verifies a transaction by checking the chain of thousands of blocks below it and checks that the UTXO is not spent, whereas an SPV node checks how deep the block is buried by a handful of blocks above it.
|
||||
====
|
||||
|
||||
To get the block headers, SPV nodes use a +getheaders+ message instead of +getblocks+. The responding peer will send up to 2000 block headers using a single +headers+ message. The process is otherwise the same as that used by a full node to retrieve full blocks. SPV nodes also set a filter on the connection to peers, to filter the stream of future blocks and transactions sent by the peers. Any transactions of interest are retrieved using a +getdata+ request. The peer generates a +tx+ message containing the transactions, in response.
|
||||
|
||||
[[spv_synchronization]]
|
||||
.SPV Node synchronizing the block headers
|
||||
image::images/SPVSynchronization.png["SPVSynchronization"]
|
||||
|
||||
Because SPV nodes need to retrieve specific transactions in order to selectively verify them, they also create a privacy risk. Unlike full-blockchain nodes, which collect all transactions within each block, the SPV node's requests for specific data can inadvertently reveal the addresses in their wallet. For example, a third party monitoring a network could keep track of all the transactions requested by a wallet on an SPV node and use those to associate bitcoin addresses with the user of that wallet, destroying the user's privacy.
|
||||
|
||||
Shortly after the introduction of SPV/lightweight nodes, the bitcoin developers added a feature called _bloom filters_ to address the privacy risks of SPV nodes. Bloom filters allow SPV nodes to receive a subset of the transactions without revealing precisely which addresses they are interested in, through a filtering mechanism that uses probabilities rather than fixed patterns.
|
||||
|
||||
=== Bloom Filters
|
||||
|
||||
A bloom filter is a probabilistic search filter, a way to describe a desired pattern without specifying it exactly. Bloom filters offer an efficient way to express a search pattern while protecting privacy. They are used by SPV nodes to ask their peers for transactions matching a specific pattern, without revealing exactly which addresses they are searching for.
|
||||
|
||||
Using our previous analogy of a tourist without a map asking for directions to a specific address "23 Church St". If they ask strangers for directions to this street, they inadvertently reveal their destination. A bloom filter is like asking "Are there any streets in this neighborhood whose name ends in R-C-H". A question like that reveals slightly less about the desired destination, than asking for "23 Church St". Using this technique, a tourist could specify the desired address in more detail as "ending in U-R-C-H" or less detail as "ending in H". By varying the precision of the search, the tourist reveals more or less information, at the expense of getting more or less specific results. If they ask a less specific pattern, they get a lot more possible addresses and better privacy but many of the results are irrelevant. If they ask for a very specific pattern then they get fewer results but they lose privacy.
|
||||
|
||||
Bloom filters serve this function by allowing an SPV node to specify a search pattern for transactions that can be tuned towards precision or privacy. A more specific bloom filter will produce accurate results, but at the expense of revealing what addresses are used in the user's wallet. A less specific bloom filter will produce more data about more transactions, many irrelevant to the node, but will allow the node to maintain better privacy.
|
||||
|
||||
An SPV node will initialize a bloom filter as "empty" and in that state the bloom filter will not match any patterns. The SPV node will then make a list of all the addresses in its wallet and create a search pattern matching the transaction output that corresponds to each address. Usually, the search pattern is a Pay-to-Public-Key-Hash script that is the expected locking script that will be present in any transaction paying to the public-key-hash (address). If the SPV node is tracking the balance of a P2SH address, then the search pattern will be a Pay-to-Script-Hash script, instead. The SPV node then adds each of the search patterns to the bloom filter, so that the bloom filter can recognize the search pattern if it is present in a transaction. Finally, the bloom filter is sent to the peer and the peer uses it to match transactions for transmission to the SPV node.
|
||||
|
||||
Bloom filters are implemented as a variable-size array of N binary digits (a bit field) and a variable number of M hash functions. The hash functions are designed to always produce an output that is between 1 and N, corresponding to the array of binary digits. The hash functions are generated deterministically, so that any node implementing a bloom filter will always use the same hash functions and get the same results for a specific input. By choosing different length (N) bloom filters and a different number (M) of hash functions, the bloom filter can be tuned, varying the level of accuracy and therefore privacy.
|
||||
|
||||
In the example below, we use a very small array of 16 bits and a set of 3 hash functions to demonstrate how bloom filters work.
|
||||
|
||||
[[bloom1]]
|
||||
.An example of a simplistic bloom filter, with 16 bit field and 3 hash functions
|
||||
image::images/Bloom1.png["Bloom1"]
|
||||
|
||||
The bloom filter is initialized so that the array of bits is all zeros. To add a pattern to the bloom filter, the pattern is hashed by each hash function in turn. Applying the first hash function to the input results in a number between 1 and N. The corresponding bit in the array (indexed from 1 to N) is found and set to +1+, thereby recording the output of the hash function. Then, the next hash function is used to set another bit and so on and so forth. Once all M hash functions have been applied, the search pattern will be "recorded" in the bloom filter as M bits have been changed from +0+ to +1+.
|
||||
|
||||
Here's an example of adding a pattern "A" to the simple bloom filter shown above:
|
||||
|
||||
[[bloom2]]
|
||||
.Adding a pattern "A" to our simple bloom filter
|
||||
image::images/Bloom2.png["Bloom2"]
|
||||
|
||||
Adding a second pattern is as simple as repeating this process. The pattern is hashed by each hash function in turn and the result is recorded by setting the bits to +1+. Note that as a bloom filter is filled with more patterns, a hash function result may coincide with a bit that is already set to +1+ in which case the bit is not changed. In essence, as more patterns record on overlapping bits, the bloom filter starts to become saturated with more bits set to +1+ and the accuracy of the filter decreases. This is why the filter is a probabilistic data structure - it gets less accurate as more patterns are added. The accuracy depends on the number of patterns added versus the size of the bit array (N) and number of hash functions (M). A larger bit array and more hash functions can record more patterns with higher accuracy. A smaller bit array or fewer hash functions will record fewer patterns and produce less accuracy.
|
||||
|
||||
Below is an example of adding a second pattern "B" to the simple bloom filter:
|
||||
|
||||
[[bloom3]]
|
||||
.Adding a second pattern "B" to our simple bloom filter
|
||||
image::images/Bloom3.png["Bloom3"]
|
||||
|
||||
To test if a pattern is part of a bloom filter, the pattern is hashed by each hash function and the resulting bit pattern is tested against the bit array. If all the bits indexed by the hash functions are set to +1+, then the patten is _probably_ recorded in the bloom filter. Since the bits may be set because of overlap from multiple patterns, the answer is not certain, but is rather probabilistic. In simple terms, a bloom filter positive match is a "Maybe, Yes".
|
||||
|
||||
Below is an example of testing the existence of pattern "X" in the simple bloom filter. The corresponding bits are set to +1+, so the pattern is probably a match:
|
||||
|
||||
[[bloom4]]
|
||||
.Testing the existence of pattern "X" in the bloom filter. The result is probabilistic positive match, meaning "Maybe"
|
||||
image::images/Bloom4.png["Bloom4"]
|
||||
|
||||
On the contrary, if a pattern is tested against the bloom filter and any one of the bits is set to +0+, then this proves that the pattern was not recorded in the bloom filter. A negative result is not a probability, it is a certainty. In simple terms, a negative match on a bloom filter is a "Definitely No".
|
||||
|
||||
Below is an example of testing the existence of pattern "Y" in the simple bloom filter. One of the corresponding bits is set to +0+, so the pattern is definitely not a match:
|
||||
|
||||
[[bloom5]]
|
||||
.Testing the existence of pattern "Y" in the bloom filter. The result is a definitive negative match, meaning "Definitely No"
|
||||
image::images/Bloom5.png["Bloom5"]
|
||||
|
||||
Bitcoin's implementation of bloom filters is described in Bitcoin Improvement Proposal 37 (BIP0037). See <<bip0037>> or visit:
|
||||
https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki
|
||||
|
||||
=== Bloom Filters and Inventory Updates
|
||||
|
||||
Bloom filters are used to filter the transactions (and blocks containing them) that an SPV node receives from its peers. SPV nodes will create a filter that matches only the addresses held in the SPV node's wallet. The SPV node will then send a +filterload+ message to the peer, containing the bloom filter to use on the connection. After a filter is established, the peer will then test each transaction's outputs against the bloom filter. Only transactions which match the filter are sent to the node.
|
||||
|
||||
In response to a +getdata+ message from the node, peers will send a +merkleblock+ message that contains only block headers for blocks matching the filter and a merkle path (See <<merkle_trees>>) for each matching transaction. The peer will also then send +tx+ messages containing the transactions matched by the filter.
|
||||
|
||||
The node setting the bloom filter can interactively add patterns to the filter by sending a +filteradd+ message. To clear the bloom filter, the node can send a +filterclear+ message. Since it is not possible to remove a pattern from a bloom filter, a node has to clear and re-send a new bloom filter if a pattern is no longer desired.
|
||||
|
||||
[[transaction_pools]]
|
||||
=== Transaction Pools
|
||||
|
||||
Almost every node on the bitcoin network maintains a temporary list of unconfirmed transactions called the memory pool or transaction pool. Nodes use this pool to keep track of transactions that are known to the network but are not yet included in the blockchain. For example, a node that holds a user's wallet will use the transaction pool to track incoming payments to the user's wallet that have been received on the network but are not yet confirmed.
|
||||
|
||||
As transactions are received and verified, they are added to the transaction pool and relayed to the neighboring nodes to propagate on the network.
|
||||
|
||||
Some node implementations also maintain a separate pool of orphaned transactions as detailed in <<orphan_transactions>>. If a transaction's inputs refer to a transaction that is not yet known, a missing parent, then the orphan transaction will be stored temporarily in the orphan pool until the parent transaction arrives.
|
||||
|
||||
When a transaction is added to the transaction pool, the orphan pool is checked for any orphans that reference this transaction's outputs (its children). Any orphans found are pulled from the orphan pool and validated using the above checklist. If valid, they are also added to the transaction pool, completing the chain that started with the parent transaction. In light of the newly added transaction which is no longer an orphan, the process is repeated recursively looking for any further descendants, until no more descendants are found. Through this process, the arrival of a parent transaction triggers a cascade reconstruction of an entire chain of interdependent transactions by re-uniting the orphans with their parents all the way down the chain.
|
||||
|
||||
Both the transaction pool and orphan pool (where implemented) are stored in local memory and are not saved on persistent storage, rather they are dynamically populated from incoming network messages. When a node starts, both pools are empty and are gradually populated with new transactions received on the network.
|
||||
|
||||
Some implementations of the bitcoin client also maintain a UTXO database or UTXO pool which is the set of all unspent outputs on the blockchain. While the name "UTXO pool" sounds similar to the transaction pool, it represents a different set of data. Unlike the transaction and orphan pools, the UTXO pool is not initialized empty but instead contains millions of entries of unspent transaction outputs including some dating back to 2009. The UTXO pool may be housed in local memory or as an indexed database table on persistent storage.
|
||||
|
||||
Whereas the transaction and orphan pools represent a single node's local perspective and may vary significantly from node to node depending upon when the node was started or restarted, the UTXO pool represents the emergent consensus of the network and therefore will vary little between nodes. Furthermore the transaction and orphan pools only contain unconfirmed transactions, while the UTXO pool only contains confirmed outputs.
|
||||
|
||||
=== Alert Messages
|
||||
|
||||
Alert messages are a seldom used function, which is nevertheless implemented in most nodes. Alert messages are bitcoin's "emergency broadcast system", a means by which the core bitcoin developers can send an emergency text message to all bitcoin nodes. This feature is implemented to allow the core developer team to notify all bitcoin users of a serious problem in the bitcoin network, such as a critical bug that requires user action. The alert system has only been used a handful of times, most notably in the Spring of 2013 when a critical database bug caused a multi-block fork to occur in the bitcoin blockchain.
|
||||
|
||||
Alert messages are propagated by the +alert+ message. The alert message contains several fields, including:
|
||||
|
||||
* ID - An alert identified so that duplicate alerts can be detected
|
||||
* Expiration - a time after which the alert expires
|
||||
* RelayUntil - a time after which the alert should not be relayed
|
||||
* MinVer, MaxVer - the range of bitcoin protocol versions that this alert applies to
|
||||
* subVer - The client software version that this alert applies to
|
||||
* Priority - An alert priority level, currently unused
|
||||
|
||||
Alerts are cryptographically signed by a public key. The corresponding private key is held by a few selected members of the core development team. The digital signature ensures that fake alerts will not be propagated on the network.
|
||||
|
||||
Each node receiving this alert message will verify it, check for expiration, and propagate it to all its peers, thus ensuring rapid propagation across the entire network. In addition to propagating the alert, each node may implement a user interface function to present the alert to the user.
|
||||
|
||||
In the Bitcoin Core client, the alert is configured with the command line option +-alertnotify+, which specifies a command to run when an alert is received. The alert message is passed as a parameter to the alertnotify command. Most commonly, the alertnotify command is set to generate an email message to the administrator of the node, containing the alert message. The alert is also displayed as a pop-up dialog in the graphical user interface (bitcoin-Qt) if it is running.
|
||||
|
||||
Other implementations of the bitcoin protocol may handle the alert in different ways. Many hardware-embedded bitcoin mining systems do not implement the alert message function, as they have no user interface. It is strongly recommended that miners running such mining systems subscribe to alerts via a mining pool operator or by running a lightweight node just for alert purposes.
|
||||
|
||||
|
||||
|
||||
412
ch07.asciidoc
412
ch07.asciidoc
@ -1,206 +1,206 @@
|
||||
[[ch7]]
|
||||
== The Blockchain
|
||||
|
||||
[[blockchain]]
|
||||
=== The Blockchain
|
||||
|
||||
The blockchain data structure is an ordered back-linked list of blocks of transactions. The blockchain can be stored as a flat file, or in a simple database. The bitcoin core client stores the blockchain metadata using Google's LevelDB database. Blocks are linked "back", each referring to the previous block in the chain. The blockchain is often visualized as a vertical stack, with blocks layered on top of each other and the first block ever serving as the foundation of the stack. The visualization of blocks stacked on top of each other results in the use of terms like "height", to refer to the distance from the first block, and "top" or "tip" to refer to the most recently added block.
|
||||
|
||||
Each block within the blockchain is identified by a hash, generated using the SHA256 cryptographic hash algorithm on the header of the block. Each block also references a previous block, known as the _parent_ block, through the "previous block hash" field in the block header. In other words, each block contains the hash of its parent inside its own header. The sequence of hashes linking each block to its parent, creates a chain going back all the way to the first block ever created, known as the _genesis block_.
|
||||
|
||||
While a block has just one parent, it can temporarily have multiple children. Each of the children refers to the same block as its parent and contains the same (parent) hash in the "previous block hash" field. Multiple children arise during a blockchain "fork", a temporary situation that occurs when different blocks are discovered almost simultaneously by different miners (see <<forks>>). Eventually, only one child block becomes part of the blockchain and the "fork" is resolved. Even though a block may have more than one child, each block can have only one parent. This is because a block has one single "previous block hash" field referencing its single parent.
|
||||
|
||||
The "previous block hash" field is inside the block header and thereby affects the _current_ block's hash. The child's own identity changes if the parent's identity changes. When the parent is modified in any way, the parent's hash changes. The parent's changed hash necessitates a change in the "previous block hash" pointer of the child. This in turn causes the child's hash to change, which requires a change in the pointer of the grandchild, which in turn changes the grandchild and so on. This cascade effect ensures that once a block has many generations following it, it cannot be changed without forcing a recalculation of all subsequent blocks. Because such a recalculation would require enormous computation, the existence of a long chain of blocks makes the blockchain's deep history immutable, a key feature of bitcoin's security.
|
||||
|
||||
One way to think about the blockchain is like layers in a geological formation, or glacier core sample. The surface layers may change with the seasons, or even be blown away before they have time to settle. But once you go a few inches deep, geological layers become more and more stable. By the time you look a few hundred feet down, you are looking at a snapshot of the past that has remained undisturbed for millennia or millions of years. In the blockchain, the most recent few blocks may be revised if there is a chain recalculation due to a fork. The top six blocks are like a few inches of topsoil. But once you go deeper into the blockchain, beyond 6 blocks, blocks are less and less likely to change. After 100 blocks back there is so much stability that the "coinbase" transaction, containing new bitcoin, can be spent. A few thousand blocks back (a month) and the blockchain is settled history. It will never change.
|
||||
|
||||
=== Structure of a Block
|
||||
|
||||
A block is a container data structure that aggregates transactions for inclusion in the public ledger, the blockchain. The block is made of a header, containing metadata, followed by a long list of transactions that make up the bulk of it's size. The block header is 80 bytes, whereas the average transaction is at least 250 bytes and the average block contains more than 500 transactions. A complete block, with all transactions, is therefore 1000 times larger than the block header.
|
||||
|
||||
[[block_structure]]
|
||||
.The structure of a block
|
||||
[options="header"]
|
||||
|=======
|
||||
|Size| Field | Description
|
||||
| 4 bytes | Block Size | The size of the block, in bytes, following this field
|
||||
| 80 bytes | Block Header | Several fields form the block header (see below)
|
||||
| 1-9 bytes (VarInt) | Transaction Counter | How many transactions follow
|
||||
| Variable | Transactions | The transactions recorded in this block
|
||||
|=======
|
||||
|
||||
[[block_header]]
|
||||
=== Block Header
|
||||
|
||||
The block header consists of three sets of block metadata. First, there is a reference to a previous block hash, which connects this block to the previous block in the blockchain. We will examine this in more detail in <<blockchain>>. The second set of metadata, namely the difficulty, timestamp and nonce, relate to the mining competition, as detailed in <<mining>>. The third piece of metadata is the Merkle Tree root, a data structure used to efficiently summarize all the transactions in the block.
|
||||
|
||||
[[block_structure]]
|
||||
.The structure of the block header
|
||||
[options="header"]
|
||||
|=======
|
||||
|Size| Field | Description
|
||||
| 4 bytes | Version | A version number to track software/protocol upgrades
|
||||
| 32 bytes | Previous Block Hash | A reference to the hash of the previous (parent) block in the chain
|
||||
| 32 bytes | Merkle Root | A hash of the root of the Merkle-Tree of this block's transactions
|
||||
| 4 bytes | Timestamp | The approximate creation time of this block (seconds from Unix Epoch)
|
||||
| 4 bytes | Difficulty Target | The proof-of-work algorithm difficulty target for this block
|
||||
| 4 bytes | Nonce | A counter used for the proof-of-work algorithm
|
||||
|=======
|
||||
|
||||
The Nonce, Difficulty Target and Timestamp are used in the mining process and will be discussed in more detail in <<mining>>.
|
||||
|
||||
[[block_hash]]
|
||||
=== Block Identifiers - Block Header Hash and Block Height
|
||||
|
||||
The primary identifier of a block is its cryptographic hash, a digital fingerprint, made by hashing the block header twice through the SHA256 algorithm. The resulting 32-byte hash, is called the _block hash_, but is more accurately the _block *header* hash_, as only the block header is used to compute it. For example, the block hash of the first bitcoin block ever created is +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+. The block hash identifies a block uniquely and unambiguously and can be independently derived by any node by simply hashing the block header.
|
||||
|
||||
Note that the block hash is not actually included inside the block's data structure, neither when the block is transmitted on the network, nor when it is stored on a node's persistence storage as part of the blockchain. Instead, the block's hash is computed by each node as the block is received from the network. The block hash may be stored in a separate database table as part of the block's metadata, to facilitate indexing and faster retrieval of blocks from disk.
|
||||
|
||||
A second way to identify a block is by its position in the blockchain, called the _block height_. The first block ever created is at block height 0 (zero), and is the same block that was referenced by the block hash +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+ above. A block can thus be identified two ways, either by referencing the block hash, or by referencing the block height. Each subsequent block added "on top" of that first block is one position "higher" in the blockchain, like boxes stacked one on top of the other. The block height on January 1st 2014 was approximately 278,000, meaning there were 278,000 blocks stacked on top of the first block created in January 2009.
|
||||
|
||||
Unlike the block hash, the block height is not a unique identifier. While a single block will always have a specific and invariant block height, the reverse is not true - the block height does not always identify a single block. Two or more blocks may have the same block height, competing for the same position in the blockchain. This scenario is discussed in detail in the section on <<forks>>. The block height is also not a part of the block's data structure, it is not stored within the block. Each node dynamically identifies a block's position (height) in the blockchain when it is received from the bitcoin network. The block height may also be stored as metadata in an indexed database table for faster retrieval.
|
||||
|
||||
[TIP]
|
||||
====
|
||||
A block's _block hash_ always identifies a single block uniquely. A block also always has a specific _block height_. However, it is not always the case that a specific block height can identify a single block, rather more than one blocks can compete for a single position in the blockchain.
|
||||
====
|
||||
|
||||
=== The Genesis Block
|
||||
|
||||
The first block in the blockchain is called the _genesis block_ and was created in 2009. It is the "common ancestor" of all the blocks in the blockchain, meaning that if you start at any block and follow the chain backwards in time you will eventually arrive at the _genesis block_.
|
||||
|
||||
Every node always starts with a blockchain of at least one block because the genesis block is statically encoded within the bitcoin client software, such that it cannot be altered. Every node always "knows" the genesis block's hash and structure, the fixed time it was created and even the single transaction within. Thus, every node has the starting point for the blockchain, a secure "root" from which to build a trusted blockchain.
|
||||
|
||||
See the statically encoded genesis block inside the Bitcoin Core client, in chainparams.cpp, line 123:
|
||||
https://github.com/bitcoin/bitcoin/blob/master/src/chainparams.cpp#L123
|
||||
|
||||
The genesis block has the identifier hash +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+. You can search for that block hash in any block explorer website, such as blockchain.info, and you will find a page describing the contents of this block, with a URL containing that hash:
|
||||
|
||||
https://blockchain.info/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
|
||||
|
||||
https://blockexplorer.com/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
|
||||
|
||||
Using the Bitcoin Core reference client on the command-line:
|
||||
|
||||
----
|
||||
$ bitcoind getblock 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
|
||||
{
|
||||
"hash" : "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
|
||||
"confirmations" : 308321,
|
||||
"size" : 285,
|
||||
"height" : 0,
|
||||
"version" : 1,
|
||||
"merkleroot" : "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
|
||||
"tx" : [
|
||||
"4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b"
|
||||
],
|
||||
"time" : 1231006505,
|
||||
"nonce" : 2083236893,
|
||||
"bits" : "1d00ffff",
|
||||
"difficulty" : 1.00000000,
|
||||
"nextblockhash" : "00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048"
|
||||
}
|
||||
----
|
||||
|
||||
The genesis block contains a hidden message within it. The coinbase transaction input contains the text "The Times 03/Jan/2009 Chancellor on brink of second bailout for banks". This message provides proof of the earliest date this block was created, by referencing the headline of the british newspaper _The Times_. It also serves as a tongue-in-cheek reminder of the importance of an independent monetary system, with bitcoin's launch occurring at the same time as an unprecedented worldwide monetary crisis. The message was embedded in the first block by Satoshi Nakamoto, bitcoin's creator.
|
||||
|
||||
=== Linking Blocks in the Blockchain
|
||||
|
||||
Bitcoin nodes maintain a local copy of the blockchain, starting at the genesis block. The local copy of the blockchain is constantly updated as new blocks are found and used to extend the chain. As a node receives incoming blocks from the network, it will validate these blocks and then link them to the existing blockchain. To establish a link, a node will examine the incoming block header and look for the "previous block hash".
|
||||
|
||||
Let's assume for example that a node has 277,314 blocks in the local copy of the blockchain. The last block the node knows about is block 277,314, with a block header hash of +00000000000000027e7ba6fe7bad39faf3b5a83daed765f05f7d1b71a1632249+.
|
||||
|
||||
The bitcoin node then receives a new block from the network, which it parses as follows:
|
||||
----
|
||||
{
|
||||
"size" : 43560,
|
||||
"version" : 2,
|
||||
"previousblockhash" :
|
||||
"00000000000000027e7ba6fe7bad39faf3b5a83daed765f05f7d1b71a1632249",
|
||||
"merkleroot" :
|
||||
"5e049f4030e0ab2debb92378f53c0a6e09548aea083f3ab25e1d94ea1155e29d",
|
||||
"time" : 1388185038,
|
||||
"difficulty" : 1180923195.25802612,
|
||||
"nonce" : 4215469401,
|
||||
"tx" : [
|
||||
"257e7497fb8bc68421eb2c7b699dbab234831600e7352f0d9e6522c7cf3f6c77",
|
||||
|
||||
[... many more transactions omitted ...]
|
||||
|
||||
"05cfd38f6ae6aa83674cc99e4d75a1458c165b7ab84725eda41d018a09176634"
|
||||
]
|
||||
}
|
||||
----
|
||||
|
||||
Looking at this new block, the node finds the "previousblockhash" field, which contains the hash of its parent block. It is a hash known to the node, that of the last block on the chain at height 277,314. Therefore, this new block is a child of the last block on the chain and extends the existing blockchain. The node adds this new block to the end of the chain, making the blockchain longer with a new height of 277,315.
|
||||
|
||||
[[chain_of_blocks]]
|
||||
.Blocks linked in a chain, by reference to the previous block header hash
|
||||
image::images/ChainOfBlocks.png["chain_of_blocks"]
|
||||
|
||||
[[merkle_trees]]
|
||||
=== Merkle Trees
|
||||
|
||||
Each block in the bitcoin blockchain contains a summary of all the transactions in the block, using a _Merkle Tree_.
|
||||
|
||||
A _Merkle Tree_, also known as a _Binary Hash Tree_ is a data structure used for efficiently summarizing and verifying the integrity of large sets of data. Merkle Trees are binary trees containing cryptographic hashes. The term "tree" is used in computer science to describe a branching data structure, but these trees are usually displayed upside down with the "root" at the top and the "leaves" at the bottom of a diagram, as you will see in the examples that follow.
|
||||
|
||||
Merkle trees are used in bitcoin to summarize all the transactions in a block, producing an overall digital fingerprint of the entire set of transactions, which can be used to prove that a transaction is included in the set. A merkle tree is constructed by recursively hashing pairs of nodes until there is only one hash, called the _root_, or _merkle root_. The cryptographic hash algorithm used in bitcoin's merkle trees is SHA256 applied twice, also known as double-SHA256.
|
||||
|
||||
When N data elements are hashed and summarized in a Merkle Tree, you can check to see if any one data element is included in the tree with at most +2*log~2~(N)+ calculations, making this a very efficient data structure.
|
||||
|
||||
The merkle tree is constructed bottom-up. In the example below, we start with four transactions A, B, C and D, which form the _leaves_ of the Merkle Tree, shown in the diagram at the bottom. The transactions are not stored in the merkle tree, rather their data is hashed and the resulting hash is stored in each leaf node as H~A~, H~B~, H~C~ and H~D~:
|
||||
|
||||
+H~A~ = SHA256(SHA256(Transaction A))+
|
||||
|
||||
Consecutive pairs of leaf nodes are then summarized in a parent node, by concatenating the two hashes and hashing them together. For example, to construct the parent node H~AB~, the two 32-byte hashes of the children are concatenated to create a 64-byte string. That string is then double-hashed to produce the parent node's hash:
|
||||
|
||||
+H~AB~ = SHA256(SHA256(H~A~ + H~B~))+
|
||||
|
||||
The process continues until there is only one node at the top, the node known as the Merkle Root. That 32-byte hash is stored in the block header and summarizes all the data in all four transactions.
|
||||
|
||||
[[simple_merkle]]
|
||||
.Calculating the nodes in a Merkle Tree
|
||||
image::images/MerkleTree.png["merkle_tree"]
|
||||
|
||||
Since the merkle tree is a binary tree, it needs an even number of leaf nodes. If there is an odd number of transactions to summarize, the last transaction hash is duplicated to create an even number of leaf nodes, also known as a _balanced tree_. This is shown in the example below, where transaction C is duplicated:
|
||||
|
||||
[[merkle_tree_odd]]
|
||||
.An even number of data elements, by duplicating one data element
|
||||
image::images/MerkleTreeOdd.png["merkle_tree_odd"]
|
||||
|
||||
The same method for constructing a tree from four transactions can be generalized to construct trees of any size. In bitcoin it is common to have several hundred to more than a thousand transactions in a single block, which are summarized in exactly the same way producing just 32-bytes of data from a single merkle root. In the diagram below, you will see a tree built from 16 transactions. Note that while the root looks bigger than the leaf nodes in the diagram, it is the exact same size, just 32 bytes. Whether there is one transaction or a hundred thousand transactions in the block, the merkle root always summarizes them into 32 bytes:
|
||||
|
||||
[[merkle_tree_large]]
|
||||
.A Merkle Tree summarizing many data elements
|
||||
image::images/MerkleTreeLarge.png["merkle_tree_large"]
|
||||
|
||||
To prove that a specific transaction is included in a block, a node need only produce +log~2~(N)+ 32-byte hashes, constituting an _authentication path_ or _merkle path_ connecting the specific transaction to the root of the tree. This is especially important as the number of transactions increases, because the base-2 logarithm of the number of transactions increases much more slowly. This allows bitcoin nodes to efficiently produce paths of ten or twelve hashes (320-384 bytes) which can provide proof of a single transaction out of more than a thousand transactions in a megabyte sized block. In the example below, a node can prove that a transaction K is included in the block by producing a merkle path that is only four 32-byte hashes long (128 bytes total). The path consists of the four hashes H~L~, H~IJ~, H~MNOP~ and H~ABCDEFGH~. With those four hashes provided as an authentication path, any node can prove that H~K~ is included in the merkle root by computing four additional pair-wise hashes H~KL~, H~IJKL~ and H~IJKLMNOP~ that lead to the merkle root.
|
||||
|
||||
[[merkle_tree_path]]
|
||||
.A Merkle Path used to prove inclusion of a data element
|
||||
image::images/MerkleTreePathToK.png["merkle_tree_path"]
|
||||
|
||||
The efficiency of merkle trees becomes obvious as the scale increases. For example, proving that a transaction is part of a block requires:
|
||||
|
||||
[[block_structure]]
|
||||
.Merkle Tree Efficiency
|
||||
[options="header"]
|
||||
|=======
|
||||
|Number of Transactions| Approx. Size of Block | Path Size (Hashes) | Path Size (Bytes)
|
||||
| 16 transactions | 4 kilobytes | 4 hashes | 128 bytes
|
||||
| 512 transactions | 128 kilobytes | 9 hashes | 288 bytes
|
||||
| 2048 transactions | 512 kilobytes | 11 hashes | 352 bytes
|
||||
| 65,535 transactions | 16 megabytes | 16 hashes | 512 bytes
|
||||
|=======
|
||||
|
||||
As you can see from the table above, while the block size increases rapidly, from 4KB with 16 transactions to a block size of 16 MB to fit 65,535 transactions, the merkle path required to prove the inclusion of a transaction increases much more slowly, from 128 bytes to only 512 bytes. With merkle trees, a node can download just the block headers (80 bytes per block) and still be able to identify a transaction's inclusion in a block by retrieving a small merkle path from a full node, without storing or transmitting the vast majority of the blockchain which may be several gigabytes in size. Nodes which do not maintain a full blockchain, called Simple Payment Verification or SPV nodes use merkle paths to verify transactions without downloading full blocks.
|
||||
|
||||
=== Merkle Trees and Simple Payment Verification (SPV)
|
||||
|
||||
Merkle trees are used extensively by Simple Payment Verification nodes. SPV nodes don't have all transactions and do not download full blocks, just block headers. In order to verify that a transaction is included in a block, without having to download all the transactions in the block, they use an _authentication path_, or merkle path.
|
||||
|
||||
Consider for example an SPV node that is interested in incoming payments to an address contained in its wallet. The SPV node will establish a bloom filter on its connections to peers to limit the transactions received to only those containing addresses of interest. When a peer sees a transaction that matches the bloom filter, it will send that block using a +merkleblock+ message. The +merkleblock+ message contains the block header as well as a merkle path that links the transaction of interest to the merkle root in the block. The SPV node can use this merkle path to connect the transaction to the block and verify that the transaction is included in the block. The SPV node also uses the block header to link the block to the rest of the blockchain. The combination of these two links, between the transaction and block, and between the block and blockchain, proves that the transaction is recorded in the blockchain. All in all, the SPV node will have received less than a kilobyte of data for the block header and merkle path, an amount of data that is a thousand times less than a full block (about 1 megabyte currently)
|
||||
|
||||
|
||||
|
||||
[[ch7]]
|
||||
== The Blockchain
|
||||
|
||||
[[blockchain]]
|
||||
=== The Blockchain
|
||||
|
||||
The blockchain data structure is an ordered back-linked list of blocks of transactions. The blockchain can be stored as a flat file, or in a simple database. The bitcoin core client stores the blockchain metadata using Google's LevelDB database. Blocks are linked "back", each referring to the previous block in the chain. The blockchain is often visualized as a vertical stack, with blocks layered on top of each other and the first block ever serving as the foundation of the stack. The visualization of blocks stacked on top of each other results in the use of terms like "height", to refer to the distance from the first block, and "top" or "tip" to refer to the most recently added block.
|
||||
|
||||
Each block within the blockchain is identified by a hash, generated using the SHA256 cryptographic hash algorithm on the header of the block. Each block also references a previous block, known as the _parent_ block, through the "previous block hash" field in the block header. In other words, each block contains the hash of its parent inside its own header. The sequence of hashes linking each block to its parent, creates a chain going back all the way to the first block ever created, known as the _genesis block_.
|
||||
|
||||
While a block has just one parent, it can temporarily have multiple children. Each of the children refers to the same block as its parent and contains the same (parent) hash in the "previous block hash" field. Multiple children arise during a blockchain "fork", a temporary situation that occurs when different blocks are discovered almost simultaneously by different miners (see <<forks>>). Eventually, only one child block becomes part of the blockchain and the "fork" is resolved. Even though a block may have more than one child, each block can have only one parent. This is because a block has one single "previous block hash" field referencing its single parent.
|
||||
|
||||
The "previous block hash" field is inside the block header and thereby affects the _current_ block's hash. The child's own identity changes if the parent's identity changes. When the parent is modified in any way, the parent's hash changes. The parent's changed hash necessitates a change in the "previous block hash" pointer of the child. This in turn causes the child's hash to change, which requires a change in the pointer of the grandchild, which in turn changes the grandchild and so on. This cascade effect ensures that once a block has many generations following it, it cannot be changed without forcing a recalculation of all subsequent blocks. Because such a recalculation would require enormous computation, the existence of a long chain of blocks makes the blockchain's deep history immutable, a key feature of bitcoin's security.
|
||||
|
||||
One way to think about the blockchain is like layers in a geological formation, or glacier core sample. The surface layers may change with the seasons, or even be blown away before they have time to settle. But once you go a few inches deep, geological layers become more and more stable. By the time you look a few hundred feet down, you are looking at a snapshot of the past that has remained undisturbed for millennia or millions of years. In the blockchain, the most recent few blocks may be revised if there is a chain recalculation due to a fork. The top six blocks are like a few inches of topsoil. But once you go deeper into the blockchain, beyond 6 blocks, blocks are less and less likely to change. After 100 blocks back there is so much stability that the "coinbase" transaction, containing new bitcoin, can be spent. A few thousand blocks back (a month) and the blockchain is settled history. It will never change.
|
||||
|
||||
=== Structure of a Block
|
||||
|
||||
A block is a container data structure that aggregates transactions for inclusion in the public ledger, the blockchain. The block is made of a header, containing metadata, followed by a long list of transactions that make up the bulk of it's size. The block header is 80 bytes, whereas the average transaction is at least 250 bytes and the average block contains more than 500 transactions. A complete block, with all transactions, is therefore 1000 times larger than the block header.
|
||||
|
||||
[[block_structure]]
|
||||
.The structure of a block
|
||||
[options="header"]
|
||||
|=======
|
||||
|Size| Field | Description
|
||||
| 4 bytes | Block Size | The size of the block, in bytes, following this field
|
||||
| 80 bytes | Block Header | Several fields form the block header (see below)
|
||||
| 1-9 bytes (VarInt) | Transaction Counter | How many transactions follow
|
||||
| Variable | Transactions | The transactions recorded in this block
|
||||
|=======
|
||||
|
||||
[[block_header]]
|
||||
=== Block Header
|
||||
|
||||
The block header consists of three sets of block metadata. First, there is a reference to a previous block hash, which connects this block to the previous block in the blockchain. We will examine this in more detail in <<blockchain>>. The second set of metadata, namely the difficulty, timestamp and nonce, relate to the mining competition, as detailed in <<mining>>. The third piece of metadata is the Merkle Tree root, a data structure used to efficiently summarize all the transactions in the block.
|
||||
|
||||
[[block_structure]]
|
||||
.The structure of the block header
|
||||
[options="header"]
|
||||
|=======
|
||||
|Size| Field | Description
|
||||
| 4 bytes | Version | A version number to track software/protocol upgrades
|
||||
| 32 bytes | Previous Block Hash | A reference to the hash of the previous (parent) block in the chain
|
||||
| 32 bytes | Merkle Root | A hash of the root of the Merkle-Tree of this block's transactions
|
||||
| 4 bytes | Timestamp | The approximate creation time of this block (seconds from Unix Epoch)
|
||||
| 4 bytes | Difficulty Target | The proof-of-work algorithm difficulty target for this block
|
||||
| 4 bytes | Nonce | A counter used for the proof-of-work algorithm
|
||||
|=======
|
||||
|
||||
The Nonce, Difficulty Target and Timestamp are used in the mining process and will be discussed in more detail in <<mining>>.
|
||||
|
||||
[[block_hash]]
|
||||
=== Block Identifiers - Block Header Hash and Block Height
|
||||
|
||||
The primary identifier of a block is its cryptographic hash, a digital fingerprint, made by hashing the block header twice through the SHA256 algorithm. The resulting 32-byte hash, is called the _block hash_, but is more accurately the _block *header* hash_, as only the block header is used to compute it. For example, the block hash of the first bitcoin block ever created is +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+. The block hash identifies a block uniquely and unambiguously and can be independently derived by any node by simply hashing the block header.
|
||||
|
||||
Note that the block hash is not actually included inside the block's data structure, neither when the block is transmitted on the network, nor when it is stored on a node's persistence storage as part of the blockchain. Instead, the block's hash is computed by each node as the block is received from the network. The block hash may be stored in a separate database table as part of the block's metadata, to facilitate indexing and faster retrieval of blocks from disk.
|
||||
|
||||
A second way to identify a block is by its position in the blockchain, called the _block height_. The first block ever created is at block height 0 (zero), and is the same block that was referenced by the block hash +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+ above. A block can thus be identified two ways, either by referencing the block hash, or by referencing the block height. Each subsequent block added "on top" of that first block is one position "higher" in the blockchain, like boxes stacked one on top of the other. The block height on January 1st 2014 was approximately 278,000, meaning there were 278,000 blocks stacked on top of the first block created in January 2009.
|
||||
|
||||
Unlike the block hash, the block height is not a unique identifier. While a single block will always have a specific and invariant block height, the reverse is not true - the block height does not always identify a single block. Two or more blocks may have the same block height, competing for the same position in the blockchain. This scenario is discussed in detail in the section on <<forks>>. The block height is also not a part of the block's data structure, it is not stored within the block. Each node dynamically identifies a block's position (height) in the blockchain when it is received from the bitcoin network. The block height may also be stored as metadata in an indexed database table for faster retrieval.
|
||||
|
||||
[TIP]
|
||||
====
|
||||
A block's _block hash_ always identifies a single block uniquely. A block also always has a specific _block height_. However, it is not always the case that a specific block height can identify a single block, rather more than one blocks can compete for a single position in the blockchain.
|
||||
====
|
||||
|
||||
=== The Genesis Block
|
||||
|
||||
The first block in the blockchain is called the _genesis block_ and was created in 2009. It is the "common ancestor" of all the blocks in the blockchain, meaning that if you start at any block and follow the chain backwards in time you will eventually arrive at the _genesis block_.
|
||||
|
||||
Every node always starts with a blockchain of at least one block because the genesis block is statically encoded within the bitcoin client software, such that it cannot be altered. Every node always "knows" the genesis block's hash and structure, the fixed time it was created and even the single transaction within. Thus, every node has the starting point for the blockchain, a secure "root" from which to build a trusted blockchain.
|
||||
|
||||
See the statically encoded genesis block inside the Bitcoin Core client, in chainparams.cpp, line 123:
|
||||
https://github.com/bitcoin/bitcoin/blob/master/src/chainparams.cpp#L123
|
||||
|
||||
The genesis block has the identifier hash +000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+. You can search for that block hash in any block explorer website, such as blockchain.info, and you will find a page describing the contents of this block, with a URL containing that hash:
|
||||
|
||||
https://blockchain.info/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
|
||||
|
||||
https://blockexplorer.com/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
|
||||
|
||||
Using the Bitcoin Core reference client on the command-line:
|
||||
|
||||
----
|
||||
$ bitcoind getblock 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
|
||||
{
|
||||
"hash" : "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
|
||||
"confirmations" : 308321,
|
||||
"size" : 285,
|
||||
"height" : 0,
|
||||
"version" : 1,
|
||||
"merkleroot" : "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
|
||||
"tx" : [
|
||||
"4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b"
|
||||
],
|
||||
"time" : 1231006505,
|
||||
"nonce" : 2083236893,
|
||||
"bits" : "1d00ffff",
|
||||
"difficulty" : 1.00000000,
|
||||
"nextblockhash" : "00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048"
|
||||
}
|
||||
----
|
||||
|
||||
The genesis block contains a hidden message within it. The coinbase transaction input contains the text "The Times 03/Jan/2009 Chancellor on brink of second bailout for banks". This message provides proof of the earliest date this block was created, by referencing the headline of the british newspaper _The Times_. It also serves as a tongue-in-cheek reminder of the importance of an independent monetary system, with bitcoin's launch occurring at the same time as an unprecedented worldwide monetary crisis. The message was embedded in the first block by Satoshi Nakamoto, bitcoin's creator.
|
||||
|
||||
=== Linking Blocks in the Blockchain
|
||||
|
||||
Bitcoin nodes maintain a local copy of the blockchain, starting at the genesis block. The local copy of the blockchain is constantly updated as new blocks are found and used to extend the chain. As a node receives incoming blocks from the network, it will validate these blocks and then link them to the existing blockchain. To establish a link, a node will examine the incoming block header and look for the "previous block hash".
|
||||
|
||||
Let's assume for example that a node has 277,314 blocks in the local copy of the blockchain. The last block the node knows about is block 277,314, with a block header hash of +00000000000000027e7ba6fe7bad39faf3b5a83daed765f05f7d1b71a1632249+.
|
||||
|
||||
The bitcoin node then receives a new block from the network, which it parses as follows:
|
||||
----
|
||||
{
|
||||
"size" : 43560,
|
||||
"version" : 2,
|
||||
"previousblockhash" :
|
||||
"00000000000000027e7ba6fe7bad39faf3b5a83daed765f05f7d1b71a1632249",
|
||||
"merkleroot" :
|
||||
"5e049f4030e0ab2debb92378f53c0a6e09548aea083f3ab25e1d94ea1155e29d",
|
||||
"time" : 1388185038,
|
||||
"difficulty" : 1180923195.25802612,
|
||||
"nonce" : 4215469401,
|
||||
"tx" : [
|
||||
"257e7497fb8bc68421eb2c7b699dbab234831600e7352f0d9e6522c7cf3f6c77",
|
||||
|
||||
[... many more transactions omitted ...]
|
||||
|
||||
"05cfd38f6ae6aa83674cc99e4d75a1458c165b7ab84725eda41d018a09176634"
|
||||
]
|
||||
}
|
||||
----
|
||||
|
||||
Looking at this new block, the node finds the "previousblockhash" field, which contains the hash of its parent block. It is a hash known to the node, that of the last block on the chain at height 277,314. Therefore, this new block is a child of the last block on the chain and extends the existing blockchain. The node adds this new block to the end of the chain, making the blockchain longer with a new height of 277,315.
|
||||
|
||||
[[chain_of_blocks]]
|
||||
.Blocks linked in a chain, by reference to the previous block header hash
|
||||
image::images/ChainOfBlocks.png["chain_of_blocks"]
|
||||
|
||||
[[merkle_trees]]
|
||||
=== Merkle Trees
|
||||
|
||||
Each block in the bitcoin blockchain contains a summary of all the transactions in the block, using a _Merkle Tree_.
|
||||
|
||||
A _Merkle Tree_, also known as a _Binary Hash Tree_ is a data structure used for efficiently summarizing and verifying the integrity of large sets of data. Merkle Trees are binary trees containing cryptographic hashes. The term "tree" is used in computer science to describe a branching data structure, but these trees are usually displayed upside down with the "root" at the top and the "leaves" at the bottom of a diagram, as you will see in the examples that follow.
|
||||
|
||||
Merkle trees are used in bitcoin to summarize all the transactions in a block, producing an overall digital fingerprint of the entire set of transactions, which can be used to prove that a transaction is included in the set. A merkle tree is constructed by recursively hashing pairs of nodes until there is only one hash, called the _root_, or _merkle root_. The cryptographic hash algorithm used in bitcoin's merkle trees is SHA256 applied twice, also known as double-SHA256.
|
||||
|
||||
When N data elements are hashed and summarized in a Merkle Tree, you can check to see if any one data element is included in the tree with at most +2*log~2~(N)+ calculations, making this a very efficient data structure.
|
||||
|
||||
The merkle tree is constructed bottom-up. In the example below, we start with four transactions A, B, C and D, which form the _leaves_ of the Merkle Tree, shown in the diagram at the bottom. The transactions are not stored in the merkle tree, rather their data is hashed and the resulting hash is stored in each leaf node as H~A~, H~B~, H~C~ and H~D~:
|
||||
|
||||
+H~A~ = SHA256(SHA256(Transaction A))+
|
||||
|
||||
Consecutive pairs of leaf nodes are then summarized in a parent node, by concatenating the two hashes and hashing them together. For example, to construct the parent node H~AB~, the two 32-byte hashes of the children are concatenated to create a 64-byte string. That string is then double-hashed to produce the parent node's hash:
|
||||
|
||||
+H~AB~ = SHA256(SHA256(H~A~ + H~B~))+
|
||||
|
||||
The process continues until there is only one node at the top, the node known as the Merkle Root. That 32-byte hash is stored in the block header and summarizes all the data in all four transactions.
|
||||
|
||||
[[simple_merkle]]
|
||||
.Calculating the nodes in a Merkle Tree
|
||||
image::images/MerkleTree.png["merkle_tree"]
|
||||
|
||||
Since the merkle tree is a binary tree, it needs an even number of leaf nodes. If there is an odd number of transactions to summarize, the last transaction hash is duplicated to create an even number of leaf nodes, also known as a _balanced tree_. This is shown in the example below, where transaction C is duplicated:
|
||||
|
||||
[[merkle_tree_odd]]
|
||||
.An even number of data elements, by duplicating one data element
|
||||
image::images/MerkleTreeOdd.png["merkle_tree_odd"]
|
||||
|
||||
The same method for constructing a tree from four transactions can be generalized to construct trees of any size. In bitcoin it is common to have several hundred to more than a thousand transactions in a single block, which are summarized in exactly the same way producing just 32-bytes of data from a single merkle root. In the diagram below, you will see a tree built from 16 transactions. Note that while the root looks bigger than the leaf nodes in the diagram, it is the exact same size, just 32 bytes. Whether there is one transaction or a hundred thousand transactions in the block, the merkle root always summarizes them into 32 bytes:
|
||||
|
||||
[[merkle_tree_large]]
|
||||
.A Merkle Tree summarizing many data elements
|
||||
image::images/MerkleTreeLarge.png["merkle_tree_large"]
|
||||
|
||||
To prove that a specific transaction is included in a block, a node need only produce +log~2~(N)+ 32-byte hashes, constituting an _authentication path_ or _merkle path_ connecting the specific transaction to the root of the tree. This is especially important as the number of transactions increases, because the base-2 logarithm of the number of transactions increases much more slowly. This allows bitcoin nodes to efficiently produce paths of ten or twelve hashes (320-384 bytes) which can provide proof of a single transaction out of more than a thousand transactions in a megabyte sized block. In the example below, a node can prove that a transaction K is included in the block by producing a merkle path that is only four 32-byte hashes long (128 bytes total). The path consists of the four hashes H~L~, H~IJ~, H~MNOP~ and H~ABCDEFGH~. With those four hashes provided as an authentication path, any node can prove that H~K~ is included in the merkle root by computing four additional pair-wise hashes H~KL~, H~IJKL~ and H~IJKLMNOP~ that lead to the merkle root.
|
||||
|
||||
[[merkle_tree_path]]
|
||||
.A Merkle Path used to prove inclusion of a data element
|
||||
image::images/MerkleTreePathToK.png["merkle_tree_path"]
|
||||
|
||||
The efficiency of merkle trees becomes obvious as the scale increases. For example, proving that a transaction is part of a block requires:
|
||||
|
||||
[[block_structure]]
|
||||
.Merkle Tree Efficiency
|
||||
[options="header"]
|
||||
|=======
|
||||
|Number of Transactions| Approx. Size of Block | Path Size (Hashes) | Path Size (Bytes)
|
||||
| 16 transactions | 4 kilobytes | 4 hashes | 128 bytes
|
||||
| 512 transactions | 128 kilobytes | 9 hashes | 288 bytes
|
||||
| 2048 transactions | 512 kilobytes | 11 hashes | 352 bytes
|
||||
| 65,535 transactions | 16 megabytes | 16 hashes | 512 bytes
|
||||
|=======
|
||||
|
||||
As you can see from the table above, while the block size increases rapidly, from 4KB with 16 transactions to a block size of 16 MB to fit 65,535 transactions, the merkle path required to prove the inclusion of a transaction increases much more slowly, from 128 bytes to only 512 bytes. With merkle trees, a node can download just the block headers (80 bytes per block) and still be able to identify a transaction's inclusion in a block by retrieving a small merkle path from a full node, without storing or transmitting the vast majority of the blockchain which may be several gigabytes in size. Nodes which do not maintain a full blockchain, called Simple Payment Verification or SPV nodes use merkle paths to verify transactions without downloading full blocks.
|
||||
|
||||
=== Merkle Trees and Simple Payment Verification (SPV)
|
||||
|
||||
Merkle trees are used extensively by Simple Payment Verification nodes. SPV nodes don't have all transactions and do not download full blocks, just block headers. In order to verify that a transaction is included in a block, without having to download all the transactions in the block, they use an _authentication path_, or merkle path.
|
||||
|
||||
Consider for example an SPV node that is interested in incoming payments to an address contained in its wallet. The SPV node will establish a bloom filter on its connections to peers to limit the transactions received to only those containing addresses of interest. When a peer sees a transaction that matches the bloom filter, it will send that block using a +merkleblock+ message. The +merkleblock+ message contains the block header as well as a merkle path that links the transaction of interest to the merkle root in the block. The SPV node can use this merkle path to connect the transaction to the block and verify that the transaction is included in the block. The SPV node also uses the block header to link the block to the rest of the blockchain. The combination of these two links, between the transaction and block, and between the block and blockchain, proves that the transaction is recorded in the blockchain. All in all, the SPV node will have received less than a kilobyte of data for the block header and merkle path, an amount of data that is a thousand times less than a full block (about 1 megabyte currently)
|
||||
|
||||
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user