<p><strong>Andreas M. Antonopoulos</strong> is a noted technologist and serial entrepreneur who has become one of the most well-known and well-respected figures in bitcoin. As an engaging public speaker, teacher, and writer, Andreas makes complex subjects accessible and easy to understand. As an advisor, he helps startups recognize, evaluate, and navigate security and business risks.</p>
<p>Andreas grew up with the Internet, starting his first company, an early BBS and proto-ISP, as a teenager in his home in Greece. He earned degrees in computer science, data communications, and distributed systems from University College London (UCL), recently ranked among the world’s top 10 universities. After moving to the US, Andreas co-founded and managed a successful technology research company, and in that role advised dozens of Fortune 500 company executives on networking, security, data centers, and cloud computing. More than 200 of his articles on security, cloud computing, and data centers have been published in print and syndicated worldwide. He holds two patents in networking and security.</p>
<p>Andreas grew up with the internet, starting his first company, an early BBS and proto-ISP, as a teenager in his home in Greece. He earned degrees in computer science, data communications, and distributed systems from University College London (UCL), recently ranked among the world’s top 10 universities. After moving to the US, Andreas co-founded and managed a successful technology research company, and in that role advised dozens of Fortune 500 company executives on networking, security, data centers, and cloud computing. More than 200 of his articles on security, cloud computing, and data centers have been published in print and syndicated worldwide. He holds two patents in networking and security.</p>
<p>In 1990, Andreas started teaching various IT topics in private, professional, and academic environments. He honed his speaking skills in front of audiences ranging in size from five executives in a boardroom to thousands of people in large conferences. With more than 400 speaking engagements under his belt he is considered a world-class and charismatic public speaker and teacher. In 2014, he was appointed as a teaching fellow with the University of Nicosia, the first university in the world to offer a masters degree in digital currency. In this role, he helped develop the curriculum and co-taught the Introduction to Digital Currencies course, offered as a massive open online course (MOOC) through the university.</p>
In the previous chapter, we introduced the basic elements of bitcoin transactions and looked at the most common type of transaction script, the Pay-to-Public-Key-Hash script. In this chapter we will look at more advanced scripting and how we can use it to build transactions with complex conditions.
First, we will look at _multi-signature_ scripts. Next we will examine the second most common transaction script, _Pay-to-Script-Hash_, which opens up a whole world of complex scripts. Then, we will examine new script operators that add a time-dimension to bitcoin, through _timelocks_.
First, we will look at _multisignature_ scripts. Next we will examine the second most common transaction script, _Pay-to-Script-Hash_, which opens up a whole world of complex scripts. Then, we will examine new script operators that add a time-dimension to bitcoin, through _timelocks_.
[[multisig]]
=== Multi-Signature
=== Multisignature
((("multi-signature scripts")))((("transactions","multi-signature scripts")))Multi-signature scripts set a condition where N public keys are recorded in the script and at least M of those must provide signatures to release the encumbrance. This is also known as an M-of-N scheme, where N is the total number of keys and M is the threshold of signatures required for validation. For example, a 2-of-3 multi-signature is one where three public keys are listed as potential signers and at least two of those must be used to create signatures for a valid transaction to spend the funds. ((("multi-signature scripts","limits on")))At this time, standard multi-signature scripts are limited to at most 15 listed public keys, meaning you can do anything from a 1-of-1 to a 15-of-15 multi-signature or any combination within that range. The limitation to 15 listed keys might be lifted by the time this book is published, so check the((("isStandard() function"))) +isStandard()+ function to see what is currently accepted by the network.
((("multi-signature scripts")))((("transactions","multi-signature scripts")))Multisignature scripts set a condition where N public keys are recorded in the script and at least M of those must provide signatures to release the encumbrance. This is also known as an M-of-N scheme, where N is the total number of keys and M is the threshold of signatures required for validation. For example, a 2-of-3 multisignature is one where three public keys are listed as potential signers and at least two of those must be used to create signatures for a valid transaction to spend the funds. ((("multi-signature scripts","limits on")))At this time, standard multisignature scripts are limited to at most 15 listed public keys, meaning you can do anything from a 1-of-1 to a 15-of-15 multisignature or any combination within that range. The limitation to 15 listed keys might be lifted by the time this book is published, so check the((("isStandard() function"))) +isStandard()+ function to see what is currently accepted by the network.
The general form of a locking script setting an M-of-N multi-signature condition is:
The general form of a locking script setting an M-of-N multisignature condition is:
----
M <Public Key 1> <Public Key 2> ... <Public Key N> N CHECKMULTISIG
@ -22,7 +22,7 @@ M <Public Key 1> <Public Key 2> ... <Public Key N> N CHECKMULTISIG
where N is the total number of listed public keys and M is the threshold of required signatures to spend the output.
A locking script setting a 2-of-3 multi-signature condition looks like this:
A locking script setting a 2-of-3 multisignature condition looks like this:
@ -46,15 +46,15 @@ When executed, this combined script will evaluate to TRUE if, and only if, the u
[[multisig_bug]]
===== A bug in CHECKMULTISIG execution
There is a bug in CHECKMULTISIG's execution that requires a slight workaround. When CHECKMULTISIG executes, it should consume M+N+2 items on the stack as parameters. However, due to the bug, CHECKMULTISIG will pop an extra value or one value more than expected.
There is a bug in ++CHECKMULTISIG++'s execution that requires a slight workaround. When +CHECKMULTISIG+ executes, it should consume M+N+2 items on the stack as parameters. However, due to the bug, +CHECKMULTISIG+ will pop an extra value or one value more than expected.
Let's look at this in greater detail using the validation example above.
Let's look at this in greater detail using the previous validation example:
First, CHECKMULTISIG pops the top item which is +N+, in this example "3". Then it pops +N+ items, which are the public keys that can sign. In this example, public keys A, B, and C. Then, it pops one item which is +M+, the quorum (how many signatures are needed). Here M = 2. At this point, CHECKMULTISIG should pop the final +M+ items, which are the signatures and see if they are valid. But, unfortunately, a bug in the implementation causes CHECKMULTISIG to pop one more item (M+1 total) than it should. The extra item is disregarded when checking the signatures so it has no direct effect on CHECKMULTISIG itself. However, an extra value must be present because if it is not present, when CHECKMULTISIG attempts to pop on an empty stack, it will cause a stack error and script failure (marking the transaction as invalid). Because the extra item is disregarded it can be anything, but customarily +0+ is used.
First, +CHECKMULTISIG+ pops the top item which is +N+, in this example "3". Then it pops +N+ items, which are the public keys that can sign. In this example, public keys A, B, and C. Then, it pops one item which is +M+, the quorum (how many signatures are needed). Here M = 2. At this point, +CHECKMULTISIG+ should pop the final +M+ items, which are the signatures, and see if they are valid. But, unfortunately, a bug in the implementation causes +CHECKMULTISIG+ to pop one more item (M+1 total) than it should. The extra item is disregarded when checking the signatures so it has no direct effect on +CHECKMULTISIG+ itself. However, an extra value must be present because if it is not present, when +CHECKMULTISIG+ attempts to pop on an empty stack, it will cause a stack error and script failure (marking the transaction as invalid). Because the extra item is disregarded it can be anything, but customarily +0+ is used.
Because this bug became part of the consensus rules, it must now be replicated forever. Therefore, the correct script validation would look like this:
@ -79,9 +79,9 @@ From now on, if you see a multisig unlocking script, you should expect to see an
[[p2sh]]
=== Pay-to-Script-Hash (P2SH)
((("multi-signature scripts","P2SH and", id="ix_ch05-asciidoc17", range="startofrange")))((("Pay-to-script-hash (P2SH)", id="ix_ch05-asciidoc18", range="startofrange")))((("transactions","Pay-to-script-hash", id="ix_ch05-asciidoc19", range="startofrange")))Pay-to-script-hash (P2SH) was introduced in 2012 as a powerful new type of transaction that greatly simplifies the use of complex transaction scripts. To explain the need for P2SH, let's look at a practical example.
((("multi-signature scripts","P2SH and", id="ix_ch05-asciidoc17", range="startofrange")))((("Pay-to-script-hash (P2SH)", id="ix_ch05-asciidoc18", range="startofrange")))((("transactions","Pay-to-script-hash", id="ix_ch05-asciidoc19", range="startofrange")))Pay-to-Script-Hash (P2SH) was introduced in 2012 as a powerful new type of transaction that greatly simplifies the use of complex transaction scripts. To explain the need for P2SH, let's look at a practical example.
In <<ch01_intro_what_is_bitcoin>> we introduced Mohammed, an electronics importer based in Dubai. Mohammed's company uses bitcoin's multi-signature feature extensively for its corporate accounts. Multi-signature scripts are one of the most common uses of bitcoin's advanced scripting capabilities and are a very powerful feature. Mohammed's company uses a multi-signature script for all customer payments, known in accounting terms as "accounts receivable," or AR. With the multi-signature scheme, any payments made by customers are locked in such a way that they require at least two signatures to release, from Mohammed and one of his partners or from his attorney who has a backup key. A multi-signature scheme like that offers corporate governance controls and protects against theft, embezzlement, or loss.
In <<ch01_intro_what_is_bitcoin>> we introduced Mohammed, an electronics importer based in Dubai. Mohammed's company uses bitcoin's multi-signature feature extensively for its corporate accounts. Multisignature scripts are one of the most common uses of bitcoin's advanced scripting capabilities and are a very powerful feature. Mohammed's company uses a multisignature script for all customer payments, known in accounting terms as "accounts receivable," or AR. With the multisignature scheme, any payments made by customers are locked in such a way that they require at least two signatures to release, from Mohammed and one of his partners or from his attorney who has a backup key. A multisignature scheme like that offers corporate governance controls and protects against theft, embezzlement, or loss.
The resulting script is quite long and looks like this:
@ -89,9 +89,9 @@ The resulting script is quite long and looks like this:
2 <Mohammed's Public Key> <Partner1 Public Key> <Partner2 Public Key> <Partner3 Public Key> <Attorney Public Key> 5 CHECKMULTISIG
----
Although multi-signature scripts are a powerful feature, they are cumbersome to use. Given the preceding script, Mohammed would have to communicate this script to every customer prior to payment. Each customer would have to use special bitcoin wallet software with the ability to create custom transaction scripts, and each customer would have to understand how to create a transaction using custom scripts. Furthermore, the resulting transaction would be about five times larger than a simple payment transaction, because this script contains very long public keys. The burden of that extra-large transaction would be borne by the customer in the form of fees. Finally, a large transaction script like this would be carried in the UTXO set in RAM in every full node, until it was spent. All of these issues make using complex locking scripts difficult in practice.
Although multisignature scripts are a powerful feature, they are cumbersome to use. Given the preceding script, Mohammed would have to communicate this script to every customer prior to payment. Each customer would have to use special bitcoin wallet software with the ability to create custom transaction scripts, and each customer would have to understand how to create a transaction using custom scripts. Furthermore, the resulting transaction would be about five times larger than a simple payment transaction, because this script contains very long public keys. The burden of that extra-large transaction would be borne by the customer in the form of fees. Finally, a large transaction script like this would be carried in the UTXO set in RAM in every full node, until it was spent. All of these issues make using complex locking scripts difficult in practice.
Pay-to-script-hash (P2SH) was developed to resolve these practical difficulties and to make the use of complex scripts as easy as a payment to a bitcoin address. With P2SH payments, the complex locking script is replaced with its digital fingerprint, a cryptographic hash. When a transaction attempting to spend the UTXO is presented later, it must contain the script that matches the hash, in addition to the unlocking script. In simple terms, P2SH means "pay to a script matching this hash, a script that will be presented later when this output is spent."
P2SH was developed to resolve these practical difficulties and to make the use of complex scripts as easy as a payment to a bitcoin address. With P2SH payments, the complex locking script is replaced with its digital fingerprint, a cryptographic hash. When a transaction attempting to spend the UTXO is presented later, it must contain the script that matches the hash, in addition to the unlocking script. In simple terms, P2SH means "pay to a script matching this hash, a script that will be presented later when this output is spent."
In P2SH transactions, the locking script that is replaced by a hash is referred to as the((("redeem script"))) _redeem script_ because it is presented to the system at redemption time rather than as a locking script. <<without_p2sh>> shows the script without P2SH and <<with_p2sh>> shows the same script encoded with P2SH.
@ -112,9 +112,9 @@ In P2SH transactions, the locking script that is replaced by a hash is referred
As you can see from the tables, with P2SH the complex script that details the conditions for spending the output (redeem script) is not presented in the locking script. Instead, only a hash of it is in the locking script and the redeem script itself is presented later, as part of the unlocking script when the output is spent. This shifts the burden in fees and complexity from the sender to the recipient (spender) of the transaction.
Let's look at Mohammed's company, the complex multi-signature script, and the resulting P2SH scripts.
Let's look at Mohammed's company, the complex multisignature script, and the resulting P2SH scripts.
First, the multi-signature script that Mohammed's company uses for all incoming payments from customers:
First, the multisignature script that Mohammed's company uses for all incoming payments from customers:
----
2 <Mohammed's Public Key> <Partner1 Public Key> <Partner2 Public Key> <Partner3 Public Key> <Attorney Public Key> 5 CHECKMULTISIG
@ -138,7 +138,8 @@ A P2SH transaction locks the output to this hash instead of the longer script, u
which, as you can see, is much shorter. Instead of "pay to this 5-key multi-signature script," the P2SH equivalent transaction is "pay to a script with this hash." A customer making a payment to Mohammed's company need only include this much shorter locking script in his payment. When Mohammed wants to spend this UTXO, they must present the original redeem script (the one whose hash locked the UTXO) and the signatures necessary to unlock it, like this:
which, as you can see, is much shorter. Instead of "pay to this 5-key multisignature script," the P2SH equivalent transaction is "pay to a script with this hash." A customer making a payment to Mohammed's company need only include this much shorter locking script in his payment. When Mohammed and his partners want to spend this UTXO, they must present the original redeem script (the one whose hash locked the UTXO) and the signatures necessary to unlock it, like this:
@ -157,15 +158,15 @@ If the redeem script hash matches, the unlocking script is executed on its own,
Almost all the scripts described in this chapter can only be implemented as Pay-to-Script-Hash scripts. They cannot be used directly in the locking script of a UTXO.
==== Pay-to-script-hash addresses
==== Pay-to-Script-Hash Addresses
((("addresses, bitcoin","Pay-to-Script-Hash (P2SH)")))((("Pay-to-script-hash (P2SH)","addresses")))Another important part of the P2SH feature is the ability to encode a script hash as an address, as defined in BIP-13. P2SH addresses are Base58Check encodings of the 20-byte hash of a script, just like bitcoin addresses are Base58Check encodings of the 20-byte hash of a public key. P2SH addresses use the version prefix "5", which results in Base58Check-encoded addresses that start with a "3". For example, Mohammed's complex script, hashed and Base58Check-encoded as a P2SH address becomes +39RF6JqABiHdYHkfChV6USGMe6Nsr66Gzw+. Now, Mohammed can give this "address" to his customers and they can use almost any bitcoin wallet to make a simple payment, as if it were a bitcoin address. The 3 prefix gives them a hint that this is a special type of address, one corresponding to a script instead of a public key, but otherwise it works in exactly the same way as a payment to a bitcoin address.
P2SH addresses hide all of the complexity, so that the person making a payment does not see the script.
==== Benefits of pay-to-script-hash
==== Benefits of Pay-to-Script-Hash
((("Pay-to-script-hash (P2SH)","benefits of")))The pay-to-script-hash feature offers the following benefits compared to the direct use of complex scripts in locking outputs:
((("Pay-to-script-hash (P2SH)","benefits of")))The P2SH feature offers the following benefits compared to the direct use of complex scripts in locking outputs:
* Complex scripts are replaced by shorter fingerprints in the transaction output, making the transaction smaller.
* Scripts can be coded as an address, so the sender and the sender's wallet don't need complex engineering to implement P2SH.
@ -174,13 +175,13 @@ P2SH addresses hide all of the complexity, so that the person making a payment d
* P2SH shifts the burden in data storage for the long script from the present time (payment) to a future time (when it is spent).
* P2SH shifts the transaction fee cost of a long script from the sender to the recipient, who has to include the long redeem script to spend it.
==== Redeem script and validation
==== Redeem Script and Validation
((("pay-to-script-hash (P2SH)","isStandard validation")))((("pay-to-script-hash (P2SH)","redeem script for")))Prior to version 0.9.2 of the Bitcoin Core client, pay-to-script-hash was limited to the standard types of bitcoin transaction scripts, by the +isStandard()+ function. That means that the redeem script presented in the spending transaction could only be one of the standard types: P2PK, P2PKH, or multi-sig nature, excluding +RETURN+ and P2SH itself.
((("pay-to-script-hash (P2SH)","isStandard validation")))((("pay-to-script-hash (P2SH)","redeem script for")))Prior to version 0.9.2 of the Bitcoin Core client, Pay-to-Script-Hash was limited to the standard types of bitcoin transaction scripts, by the +isStandard()+ function. That means that the redeem script presented in the spending transaction could only be one of the standard types: P2PK, P2PKH, or multisig nature, excluding +RETURN+ and P2SH itself.
As of version 0.9.2 of the Bitcoin Core client, P2SH transactions can contain any valid script, making the P2SH standard much more flexible and allowing for experimentation with many novel and complex types of transactions.
Note that you are not able to put a P2SH inside a P2SH redeem script, because the P2SH specification is not recursive. While it is technically possible to include +RETURN+ in a redeem script, as nothing in the rules prevents you from doing so, it is of no practical use as executing +RETURN+ during validation will cause the transaction to be marked invalid.
Note that you are not able to put a P2SH inside a P2SH redeem script, because the P2SH specification is not recursive. While it is technically possible to include +RETURN+ in a redeem script, as nothing in the rules prevents you from doing so, it is of no practical use because executing +RETURN+ during validation will cause the transaction to be marked invalid.
Note that because the redeem script is not presented to the network until you attempt to spend a P2SH output, if you lock an output with the hash of an invalid redeem script it will be processed regardless. The UTXO will be successfully locked. However, you will not be able to spend it because the spending transaction, which includes the redeem script, will not be accepted because it is an invalid script. This creates a risk, because you can lock bitcoin in a P2SH that cannot be spent later. The network will accept the P2SH locking script even if it corresponds to an invalid redeem script, because the script hash gives no indication of the script it represents.
@ -196,7 +197,7 @@ Note that because the redeem script is not presented to the network until you at
((("ledger, storing unrelated information in")))((("RETURN operator")))((("transactions","storing unrelated information in")))Bitcoin's distributed and timestamped ledger, the blockchain, has potential uses far beyond payments. Many developers have tried to use the transaction scripting language to take advantage of the security and resilience of the system for applications such as((("digital notary services")))((("smart contracts")))((("stock certificates"))) digital notary services, stock certificates, and smart contracts. Early attempts to use bitcoin's script language for these purposes involved creating transaction outputs that recorded data on the blockchain; for example, to record a digital fingerprint of a file in such a way that anyone could establish proof-of-existence of that file on a specific date by reference to that transaction.
((("blockchains","storing unrelated information in")))The use of bitcoin's blockchain to store data unrelated to bitcoin payments is a controversial subject. Many developers consider such use abusive and want to discourage it. Others view it as a demonstration of the powerful capabilities of blockchain technology and want to encourage such experimentation. Those who object to the inclusion of non-payment data argue that it causes "blockchain bloat," burdening those running full bitcoin nodes with carrying the cost of disk storage for data that the blockchain was not intended to carry. Moreover, such transactions create UTXO that cannot be spent, using the destination bitcoin address as a free-form 20-byte field. Because the address is used for data, it doesn't correspond to a private key and the resulting UTXO can _never_ be spent; it's a fake payment. These transactions that can never be spent are therefore never removed from the UTXO set and cause the size of the UTXO database to forever increase, or "bloat."
((("blockchains","storing unrelated information in")))The use of bitcoin's blockchain to store data unrelated to bitcoin payments is a controversial subject. Many developers consider such use abusive and want to discourage it. Others view it as a demonstration of the powerful capabilities of blockchain technology and want to encourage such experimentation. Those who object to the inclusion of nonpayment data argue that it causes "blockchain bloat," burdening those running full bitcoin nodes with carrying the cost of disk storage for data that the blockchain was not intended to carry. Moreover, such transactions create UTXO that cannot be spent, using the destination bitcoin address as a freeform 20-byte field. Because the address is used for data, it doesn't correspond to a private key and the resulting UTXO can _never_ be spent; it's a fake payment. These transactions that can never be spent are therefore never removed from the UTXO set and cause the size of the UTXO database to forever increase, or "bloat."
In version 0.9 of the Bitcoin Core client, a compromise was reached with the introduction of the +RETURN+ operator. +RETURN+ allows developers to add 80 bytes of nonpayment data to a transaction output. However, unlike the use of "fake" UTXO, the +RETURN+ operator creates an explicitly _provably unspendable_ output, which does not need to be stored in the UTXO set. +RETURN+ outputs are recorded on the blockchain, so they consume disk space and contribute to the increase in the blockchain's size, but they are not stored in the UTXO set and therefore do not bloat the UTXO memory pool and burden full nodes with the cost of more expensive RAM.
@ -208,59 +209,59 @@ RETURN <data>
The data portion is limited to 80 bytes and most often represents a hash, such as the output from the SHA256 algorithm (32 bytes). Many applications put a prefix in front of the data to help identify the application. For example, the http://proofofexistence.com[Proof of Existence] digital notarization service uses the 8-byte prefix +DOCPROOF+, which is ASCII encoded as +44 4f 43 50 52 4f 4f 46+ in hexadecimal.
Keep in mind that there is no "unlocking script" that corresponds to +RETURN+ that could possibly be used to "spend" a +RETURN+ output. The whole point of +RETURN+ is that you can't spend the money locked in that output, and therefore it does not need to be held in the UTXO set as potentially spendable—+RETURN+ is _provably un-spendable_. +RETURN+ is usually an output with a zero bitcoin amount, because any bitcoin assigned to such an output is effectively lost forever. If a +RETURN+ is referenced as an input in a transaction, the script validation engine will halt the execution of the validation script and marking the transaction as invalid. The execution of +RETURN+, essentially causes the script to "RETURN" with a FALSE and halt. Thus, if you accidentally reference a +RETURN+ output as an input in a transaction, that transaction is invalid.
Keep in mind that there is no "unlocking script" that corresponds to +RETURN+ that could possibly be used to "spend" a +RETURN+ output. The whole point of +RETURN+ is that you can't spend the money locked in that output, and therefore it does not need to be held in the UTXO set as potentially spendable—+RETURN+ is _provably unspendable_. +RETURN+ is usually an output with a zero bitcoin amount, because any bitcoin assigned to such an output is effectively lost forever. If a +RETURN+ is referenced as an input in a transaction, the script validation engine will halt the execution of the validation script and mark the transaction as invalid. The execution of +RETURN+ essentially causes the script to "RETURN" with a +FALSE+ and halt. Thus, if you accidentally reference a +RETURN+ output as an input in a transaction, that transaction is invalid.
A standard transaction (one that conforms to the +isStandard()+ checks) can have only one +RETURN+ output. However, a single +RETURN+ output can be combined in a transaction with outputs of any other type.
Two new command-line options have been added in Bitcoin Core as of version 0.10. The option +datacarrier+ controls relay and mining of RETURN transactions, with the default set to "1" to allow them. The option +datacarriersize+ takes a numeric argument specifying the maximum size in bytes of the RETURN script, 83 bytes by default which allows for a maximum of 80 bytes of RETURN data plus one byte of +RETURN+ opcode and two bytes of +PUSHDATA+ opcode.
Two new command-line options have been added in Bitcoin Core as of version 0.10. The option +datacarrier+ controls relay and mining of +RETURN+ transactions, with the default set to "1" to allow them. The option +datacarriersize+ takes a numeric argument specifying the maximum size in bytes of the +RETURN+ script, 83 bytes by default which, allows for a maximum of 80 bytes of +RETURN+ data plus one byte of +RETURN+ opcode and two bytes of +PUSHDATA+ opcode.
[NOTE]
====
RETURN was initially proposed with a limit of 80 bytes, but the limit was reduced to 40 bytes when the feature was released. In February 2015, in version 0.10 of Bitcoin Core, the limit was raised back to 80 bytes. Nodes may choose not to relay or mine RETURN, or only relay and mine RETURN containing less than 80 bytes of data.
+RETURN+ was initially proposed with a limit of 80 bytes, but the limit was reduced to 40 bytes when the feature was released. In February 2015, in version 0.10 of Bitcoin Core, the limit was raised back to 80 bytes. Nodes may choose not to relay or mine +RETURN+, or only relay and mine +RETURN+ containing less than 80 bytes of data.
====
=== Timelocks
Timelocks are restrictions on transactions or outputs that only allow spending after a point in time. Bitcoin has had a transaction-level timelock feature from the beginning. It is implemented by the nLocktime field in a transaction. Two new timelock features were introduced in late 2015 and mid-2016 that offer UTXO-level timelocks. These are +CHECKLOCKTIMEVERIFY+, and CHECKSEQUENCEVERIFY.
Timelocks are restrictions on transactions or outputs that only allow spending after a point in time. Bitcoin has had a transaction-level timelock feature from the beginning. It is implemented by the +nLocktime+ field in a transaction. Two new timelock features were introduced in late 2015 and mid-2016 that offer UTXO-level timelocks. These are +CHECKLOCKTIMEVERIFY+ and +CHECKSEQUENCEVERIFY+.
Timelocks are useful for post-dating transactions and locking funds to a date in the future. More importantly, timelocks extend bitcoin scripting into the dimension of time, opening the door for complex multi-step smart contracts. We will examine the use of timelocks for smart contracts in <<state_channels>>.
Timelocks are useful for post-dating transactions and locking funds to a date in the future. More importantly, timelocks extend bitcoin scripting into the dimension of time, opening the door for complex multistep smart contracts.
[[transaction_locktime_nlocktime]]
==== Transaction Locktime (nLocktime)
From the beginning, bitcoin has had a transaction-level timelock feature. ((("locktime")))((("transactions","nLockTime")))Transaction locktime is a transaction-level setting (a field in the transaction data structure) that defines the earliest time that a transaction is valid and can be relayed on the network or added to the blockchain. Locktime is also known as nLockTime from the variable name used in the Bitcoin Core codebase. It is set to zero in most transactions to indicate immediate propagation and execution. If nLockTime is nonzero and below 500 million, it is interpreted as a block height, meaning the transaction is not valid and is not relayed or included in the blockchain prior to the specified block height. If it is above 500 million, it is interpreted as a Unix Epoch timestamp (seconds since Jan-1-1970) and the transaction is not valid prior to the specified time. Transactions with nLockTime specifying a future block or time must be held by the originating system and transmitted to the bitcoin network only after they become valid. If a transaction is transmitted to the network before the specified nLockTime, the transaction will be rejected by the first node as invalid and will not be relayed to other nodes. The use of nLockTime is equivalent to postdating a paper check.
From the beginning, bitcoin has had a transaction-level timelock feature. ((("locktime")))((("transactions","nLockTime")))Transaction locktime is a transaction-level setting (a field in the transaction data structure) that defines the earliest time that a transaction is valid and can be relayed on the network or added to the blockchain. Locktime is also known as +nLocktime+ from the variable name used in the Bitcoin Core codebase. It is set to zero in most transactions to indicate immediate propagation and execution. If +nLocktime+ is nonzero and below 500 million, it is interpreted as a block height, meaning the transaction is not valid and is not relayed or included in the blockchain prior to the specified block height. If it is above 500 million, it is interpreted as a Unix Epoch timestamp (seconds since Jan-1-1970) and the transaction is not valid prior to the specified time. Transactions with +nLocktime+ specifying a future block or time must be held by the originating system and transmitted to the bitcoin network only after they become valid. If a transaction is transmitted to the network before the specified +nLocktime+, the transaction will be rejected by the first node as invalid and will not be relayed to other nodes. The use of nLockTime is equivalent to postdating a paper check.
[[locktime_limitations]]
===== Transaction Locktime Limitations
===== Transaction locktime limitations
nLocktime has the limitation that while it makes it possible to spend some outputs in the future, it does not make it impossible to spend them until that time. Let's explain that with the following example:
+nLocktime+ has the limitation that while it makes it possible to spend some outputs in the future, it does not make it impossible to spend them until that time. Let's explain that with the following example.
Alice signs a transaction spending one of her outputs to Bob's address, and sets the transaction nLocktime to 3 months in the future. Alice sends that transaction to Bob to hold. With this transaction Alice and Bob know that:
Alice signs a transaction spending one of her outputs to Bob's address, and sets the transaction +nLocktime+ to 3 months in the future. Alice sends that transaction to Bob to hold. With this transaction Alice and Bob know that:
* Bob cannot transmit the transaction to redeem the funds until 3 months have elapsed.
* Bob may transmit the transaction after 3 months
* Bob may transmit the transaction after 3 months.
However:
* Alice can create another transaction, double-spending the same inputs without a locktime. Thus, Alice can spend the same UTXO before the 3 months have elapsed
* Alice can create another transaction, double-spending the same inputs without a locktime. Thus, Alice can spend the same UTXO before the 3 months have elapsed.
* Bob has no guarantee that Alice won't do that.
It is important to understand the limitations of transaction nLockTime. The only guarantee is that Bob will not be able to redeem it before 3 months have elapsed. There is no guarantee that Bob will get the funds. To achieve such a guarantee, the timelock restriction must be placed on the UTXO itself and be part of the locking script, rather than on the transaction. This is achieved by the next form of timelock, below.
It is important to understand the limitations of transaction +nLocktime+. The only guarantee is that Bob will not be able to redeem it before 3 months have elapsed. There is no guarantee that Bob will get the funds. To achieve such a guarantee, the timelock restriction must be placed on the UTXO itself and be part of the locking script, rather than on the transaction. This is achieved by the next form of timelock, called Check Lock Time Verify.
==== Check Lock Time Verify (CLTV)
In December of 2015, a new form of timelock was introduced to bitcoin as a soft-fork upgrade. Based on a specification in Bitcoin Improvement Proposal 65 (BIP-65), a new script operator _CHECKLOCKTIMEVERIFY_ (known also as _CLTV_) was added to the scripting language. +CLTV+ is a per-output timelock, rather than a per-transaction timelock as is the case with nLocktime. This allows for much greater flexibility in the way timelocks are applied.
In December 2015, a new form of timelock was introduced to bitcoin as a soft-fork upgrade. Based on a specification in Bitcoin Improvement Proposal 65 (BIP-65), a new script operator _CHECKLOCKTIMEVERIFY_ (known also as _CLTV_) was added to the scripting language. +CLTV+ is a per-output timelock, rather than a per-transaction timelock as is the case with +nLocktime+. This allows for much greater flexibility in the way timelocks are applied.
In simple terms, by adding the +CLTV+ opcode in the redeem script of an output it restricts the output, so that it can only be spent after the specified time has elapsed.
[TIP]
====
While nLocktime is a transaction level timelock, CLTV is an output based timelock.
While +nLocktime+ is a transaction-level timelock, +CLTV+ is an output-based timelock.
====
+CLTV+ doesn't replace nLocktime, but rather restricts specific UTXO such that they can only be spent in a future transaction with nLocktime set to a greater or equal value.
The +CLTV+ opcode takes one parameter as input, expressed as a number in the same format as nLocktime (either a block height or Unix epoch time). As indicated by the +VERIFY+ suffix, +CLTV+ is the type of opcode that halts execution of the script if the outcome is +FALSE+. If it results in TRUE, execution continues.
+CLTV+ doesn't replace +nLocktime+, but rather restricts specific UTXO such that they can only be spent in a future transaction with +nLocktime+ set to a greater or equal value.
The +CLTV+ opcode takes one parameter as input, expressed as a number in the same format as +nLocktime+ (either a block height or Unix epoch time). As indicated by the +VERIFY+ suffix, +CLTV+ is the type of opcode that halts execution of the script if the outcome is +FALSE+. If it results in TRUE, execution continues.
In order to lock an output with +CLTV+, you insert it into the redeem script of the output, in the transaction that creates the output. For example, if Alice is paying Bob's address, the output would normally contain a P2PKH script like this:
@ -274,140 +275,129 @@ To lock it to a time, say 3 months from now, the transaction would be a P2SH tra
<now + 3 months> CHECKLOCKTIMEVERIFY DROP DUP HASH160 <Bob's Public Key Hash> EQUALVERIFY CHECKSIG
----
where +<now {plus} 3 months>+ is a block height or time value estimated 3 months from the time the transaction is mined: current block height {plus} 12,960 (blocks) or current Unix epoch time {plus} 7,760,000 (seconds). For now, don't worry about the +DROP+ opcode that follows +CHECKLOCKTIMEVERIFY+, it will be explained shortly.
where +<now {plus} 3 months>+ is a block height or time value estimated 3 months from the time the transaction is mined: current block height {plus} 12,960 (blocks) or current Unix epoch time {plus} 7,760,000 (seconds). For now, don't worry about the +DROP+ opcode that follows +CHECKLOCKTIMEVERIFY+; it will be explained shortly.
When Bob tries to spend this UTXO, he constructs a transaction which references the UTXO as an input. He uses his signature and public key in the unlocking script of that input and sets the transaction nLocktime to be equal or greater to the timelock in the +CHECKLOCKTIMEVERIFY+ Alice set. Bob then broadcasts the transaction on the bitcoin network.
Bob's transaction is evaluated as follows. If the +CHECKLOCKTIMEVERIFY+ parameter Alice set is less than or equal the spending transaction's nLocktime, script execution continues (acts as if a NOP opcode was executed). Otherwise, script execution halts and the transaction is deemed invalid.
Bob's transaction is evaluated as follows. If the +CHECKLOCKTIMEVERIFY+ parameter Alice set is less than or equal the spending transaction's +nLocktime+, script execution continues (acts as if a NOP opcode was executed). Otherwise, script execution halts and the transaction is deemed invalid.
More precisely, +CHECKLOCKTIMEVERIFY+ fails and halts execution (marking the transaction invalid) if:
More precisely, +CHECKLOCKTIMEVERIFY+ fails and halts execution, marking the transaction invalid, if (source: BIP-65):
1. the lock-time type (height vs. timestamp) of the top stack item and the nLockTime field are not the same; or
1. the top stack item is greater than the transaction's nLockTime field; or
1. the nSequence field of the input is 0xffffffff;
====
1. the lock-time type (height vs. timestamp) of the top stack item and the +nLocktime+ field are not the same; or
1. the top stack item is greater than the transaction's +nLocktime+ field; or
1. the +nSequence+ field of the input is 0xffffffff;
[NOTE]
====
CLTV and nLocktime use the same format to describe timelocks, either a block height or the time elapsed in seconds since Unix epoch. Critically, when used together, the format of nLocktime must match that of CHECKLOCKTIMEVERIFY in the inputs - they must both reference either block height or time in seconds.
+CLTV+ and +nLocktime+ use the same format to describe timelocks, either a block height or the time elapsed in seconds since Unix epoch. Critically, when used together, the format of +nLocktime+ must match that of +CLTV+ in the inputs—they must both reference either block height or time in seconds.
====
After execution, if CHECKLOCKTIMEVERIFY is satisfied, the time-parameter that preceded it remains as the top item on the stack and may need to be dropped, with +DROP+, for correct execution of subsequent script opcodes. You will often see +CHECKLOCKTIMEVERIFY+ followed by +DROP+ in scripts for this reason.
After execution, if +CLTV+ is satisfied, the time-parameter that preceded it remains as the top item on the stack and may need to be dropped, with +DROP+, for correct execution of subsequent script opcodes. You will often see +CHECKLOCKTIMEVERIFY+ followed by +DROP+ in scripts for this reason.
By using nLocktime in conjunction with +CLTV+, the scenario described in <<locktime_limitations>> changes. Because Alice locked the UTXO itself, it is now impossible for either Bob or Alice to spend it before the 3-month locktime has expired.
By introducing timelock functionality directly in the scripting language, +CLTV+ allows us to develop some very interesting complex scripts.
The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki[BIP-65 (CHECKLOCKTIMEVERIFY)]
==== Relative Timelocks
nLocktime and CLTV are both _absolute timelocks_ in that the specify an absolute point in time. The next two timelock features we will examine are _relative timelocks_ in that they specify, as a condition of spending an output, an elapsed time from the confirmation of the output in the blockchain.
+nLocktime+ and +CLTV+ are both _absolute timelocks_ in that they specify an absolute point in time. The next two timelock features we will examine are _relative timelocks_ in that they specify, as a condition of spending an output, an elapsed time from the confirmation of the output in the blockchain.
Relative timelocks are useful because they allow a chain of two or more interdependent transactions to be held off chain, while imposing a time constraint on one transaction that is dependent on the elapsed time from the confirmation of a previous transaction. In other words, the clock doesn't start counting until the UTXO is recorded on the blockchain. This functionality is especially useful in bi-directional state channels and Lightning network, as we will see in <<state_channels>>.
Relative timelocks are useful because they allow a chain of two or more interdependent transactions to be held off chain, while imposing a time constraint on one transaction that is dependent on the elapsed time from the confirmation of a previous transaction. In other words, the clock doesn't start counting until the UTXO is recorded on the blockchain. This functionality is especially useful in bidirectional state channels and Lightning Networks, as we will see in <<state_channels>>.
Relative timelocks, like absolute timelocks, are implemented with both a transaction-level feature and a script-level opcode. The transaction-level relative timelock is implemented as a consensus rule on the value of nSequence, a transaction field which is set in every transaction input. Script-level relative timelocks are implemented with the CHECKSEQUENCEVERIFY opcode.
Relative timelocks, like absolute timelocks, are implemented with both a transaction-level feature and a script-level opcode. The transaction-level relative timelock is implemented as a consensus rule on the value of +nSequence+, a transaction field that is set in every transaction input. Script-level relative timelocks are implemented with the +CHECKSEQUENCEVERIFY+ (CSV) opcode.
Relative timelocks are implemented according to the specifications in the following BIPs:
BIP-68 Relative lock-time using consensus-enforced sequence numbers:: https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki]
Relative timelocks are implemented according to the specifications in https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68 Relative lock-time using consensus-enforced sequence numbers] and https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[BIP-112 CHECKSEQUENCEVERIFY].
BIP-68 and BIP-112 were activated in May 2016 as a soft-fork upgrade to the consensus rules.
==== Relative Timelock with nSequence
Relative timelocks can be set on each input of a transaction, by setting the nSequence field in each input.
Relative timelocks can be set on each input of a transaction, by setting the +nSequence+ field in each input.
===== Original meaning of nSequence
The nSequence field was originally intended (but never properly implemented) to allow modification of transactions in the mempool. In that use, a transaction containing inputs with nSequence value below 2^32^ (0xFFFFFFFF) indicated a transaction that was not yet "finalized". Such a transaction would be held in the mempool until it was replaced by another transaction spending the same inputs with a higher nSequence value. Once a transaction was received whose inputs had an nSequence value of 2^32^ it would be considered "finalized" and mined.
The +nSequence+ field was originally intended (but never properly implemented) to allow modification of transactions in the mempool. In that use, a transaction containing inputs with +nSequence+ value below 2^32^ (0xFFFFFFFF) indicated a transaction that was not yet "finalized." Such a transaction would be held in the mempool until it was replaced by another transaction spending the same inputs with a higher +nSequence+ value. Once a transaction was received whose inputs had an +nSequence+ value of 2^32^ it would be considered "finalized" and mined.
The original meaning of nSequence was never properly implemented and the value of nSequence is customarily set to 2^32^ in transactions that do not utilize timelocks. For transactions with nLocktime or CHECKLOCKTIMEVEIRFY, the nSequence value must be set to less than 2^32^ for the timelock guards to have effect. Customarily, it is set to 2^32^ - 1 (0xFFFFFFFE).
The original meaning of +nSequence+ was never properly implemented and the value of +nSequence+ is customarily set to 2^32^ in transactions that do not utilize timelocks. For transactions with nLocktime or +CHECKLOCKTIMEVERIFY+, the +nSequence+ value must be set to less than 2^32^ for the timelock guards to have effect. Customarily, it is set to 2^32^ - 1 (0xFFFFFFFE).
===== nSequence as consensus-enforced relative timelock
Since the activation of BIP-68, new consensus rules apply for any transaction containing an input whose nSequence value is less than 2^31^ (bit 1<<31 is not set). Programmatically, that means that if the most significant bit (1<<31) is not set, it is a flag that means "relative locktime". Otherwise (bit 1<<31 set), the nSequnce value is reserved for other uses such as enabling CHECKLOCKTIMEVERIFY, nLocktime, Opt-In-Replace-By-Fee and other future developments.
Since the activation of BIP-68, new consensus rules apply for any transaction containing an input whose +nSequence+ value is less than 2^31^ (bit 1<<31 is not set). Programmatically, that means that if the most significant bit (1<<31) is not set, it is a flag that means "relative locktime." Otherwise (bit 1<<31 set), the nSequnce value is reserved for other uses such as enabling +CHECKLOCKTIMEVERIFY+, +nLocktime+, Opt-In-Replace-By-Fee, and other future developments.
Transaction inputs with nSequence values less than 2^31^ are interpreted as having a relative timelock. Meaning that the transaction that includes it is only valid once the input has aged by the relative timelock amount. For example, a transaction with one input with nSequence relative timelock of 30 blocks is only valid when at least 30 blocks have elapsed from the time the UTXO referenced in the input was mined. Since nSequence is a per input field, a transaction may contain any number of timelocked inputs, all of which must have sufficiently aged for the transaction to be valid. A valid transaction can include both timelocked inputs (nSequence < 2^31^) and inputs without a relative timelock (nSequence >= 2^31^).
Transaction inputs with +nSequence+ values less than 2^31^ are interpreted as having a relative timelock, meaning that the transaction that includes it is only valid once the input has aged by the relative timelock amount. For example, a transaction with one input with +nSequence+ relative timelock of 30 blocks is only valid when at least 30 blocks have elapsed from the time the UTXO referenced in the input was mined. Since +nSequence+ is a per-input field, a transaction may contain any number of timelocked inputs, all of which must have sufficiently aged for the transaction to be valid. A valid transaction can include both timelocked inputs (+nSequence+ < 2^31^) and inputs without a relative timelock (+nSequence+ >= 2^31^).
The nSequence value is specified in either blocks or seconds, but in a slightly different format than we saw used in nLocktime. A type-flag is used to differentiate between values counting blocks and values counting time in seconds. The type flag is set in the 23rd least significant bit (ie. value 1<<22). If the type-flag is set, then the nSequence value is interpreted as a multiple of 512 seconds. If the type-flag is not set, the nSequence value is interpreted as a number of blocks.
The +nSequence+ value is specified in either blocks or seconds, but in a slightly different format than we saw used in +nLocktime+. A type-flag is used to differentiate between values counting blocks and values counting time in seconds. The type-flag is set in the 23rd least significant bit (i.e., value 1<<22). If the type-flag is set, then the +nSequence+ value is interpreted as a multiple of 512 seconds. If the type-flag is not set, the +nSequence+ value is interpreted as a number of blocks.
When interpreting nSequence as a relative timelock, only the 16 least significant bits are considered. Once the flags (bits 32 and 23) are evaluated, the nSequence value is usually "masked" with a 16-bit mask (eg. nSequence & 0x0000FFFF).
When interpreting +nSequence+ as a relative timelock, only the 16 least significant bits are considered. Once the flags (bits 32 and 23) are evaluated, the +nSequence+ value is usually "masked" with a 16-bit mask (eg. +nSequence+ & 0x0000FFFF).
The following diagram shows the binary layout of the nSequence value, as defined by BIP-68:
<<bip_68_def_of_nseq>> shows the binary layout of the +nSequence+ value, as defined by BIP-68.
[[bip_68_def_of_nseq]]
.BIP-68 definition of nSequence encoding (Source: BIP-68)
image::images/mbc2_0701.png["BIP-68 definition of nSequence encoding"]
Relative timelocks based on consensus enforcement of the +nSequence+ value are defined in BIP-68:
Relative timelocks based on consensus enforcement of the nSequence value are defined in BIP-68:
BIP-68 Relative lock-time using consensus-enforced sequence numbers:: https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki]
The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68 Relative lock-time using consensus-enforced sequence numbers].
==== Relative timelocks with CHECKSEQUENCEVERIFY (CSV)
==== Relative Timelocks with CHECKSEQUENCEVERIFY (CSV)
Just like CLTV and nLocktime, there is a script opcode for relative timelocks that leverages the nSequence value in scripts. That opcode is +CHECKSEQUENCEVERIFY+ commonly referred to as +CSV_ for short.
Just like CLTV and +nLocktime+, there is a script opcode for relative timelocks that leverages the +nSequence+ value in scripts. That opcode is +CHECKSEQUENCEVERIFY+, commonly referred to as +CSV+ for short.
The CSV opcode when evaluated in a UTXO's redeem script, allows spending only in a transaction whose input nSequence value is greater than or equal to the CSV parameter. Essentially, this restricts spending the UTXO until a certain number of blocks or seconds have elapsed relative to the time the UTXO was mined.
The +CSV+ opcode when evaluated in a UTXO's redeem script, allows spending only in a transaction whose input +nSequence+ value is greater than or equal to the +CSV+ parameter. Essentially, this restricts spending the UTXO until a certain number of blocks or seconds have elapsed relative to the time the UTXO was mined.
As with CLTV, the value in CSV must match the format in the corresponding nSequence value. If CSV is specified in terms of blocks, then so must nSequence. If CSV is specified in terms of seconds, then so must nSequence.
As with CLTV, the value in +CSV+ must match the format in the corresponding +nSequence+ value. If +CSV+ is specified in terms of blocks, then so must nSequence. If +CSV+ is specified in terms of seconds, then so must +nSequence+.
Relative timelocks with CSV are especially useful when several (chained) transactions are created and signed, but not propagated, when they're kept "off-chain". A child transaction cannot be used until the parent transaction has been propagated, mined, and aged by the time specified in the relative timelock. One application of this use case can be seen in <<state_channels>> and <<lightning_network>>.
Relative timelocks with +CSV+ are especially useful when several (chained) transactions are created and signed, but not propagated, when they're kept "off-chain". A child transaction cannot be used until the parent transaction has been propagated, mined, and aged by the time specified in the relative timelock. One application of this use case can be seen in <<state_channels>> and <<lightning_network>>.
+CSV+ is defined in detail in BIP-112 https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[CHECKSEQUENCEVERIFY].
==== Median-Time-Past
As part of the activation of relative timelocks, there was also a change in the way "time" is calculated for timelocks (both absolute and relative). In bitcoin there is a subtle, but very significant difference, between wall time and consensus time. Bitcoin is a decentralized network, which means that each participant has their own perspective of time. Events on the network do not occur instantaneously everywhere. Network latency must be factored into the perspective of each node. Eventually everything is synchronized to create a common ledger. Bitcoin reaches consensus every 10 minutes about the state of the ledger as it existed in the _past_.
As part of the activation of relative timelocks, there was also a change in the way "time" is calculated for timelocks (both absolute and relative). In bitcoin there is a subtle, but very significant, difference between wall time and consensus time. Bitcoin is a decentralized network, which means that each participant has his or her own perspective of time. Events on the network do not occur instantaneously everywhere. Network latency must be factored into the perspective of each node. Eventually everything is synchronized to create a common ledger. Bitcoin reaches consensus every 10 minutes about the state of the ledger as it existed in the _past_.
The timestamps set in block headers are set by the miners. There is a certain degree of latitude allowed by the consensus rules to account for differences in clock accuracy between decentralized nodes. However, this creates an unfortunate incentive for miners to lie about the time in a block so as to earn extra fees by including time-locked transactions that are not yet mature. See <<fee_sniping>>.
The timestamps set in block headers are set by the miners. There is a certain degree of latitude allowed by the consensus rules to account for differences in clock accuracy between decentralized nodes. However, this creates an unfortunate incentive for miners to lie about the time in a block so as to earn extra fees by including timelocked transactions that are not yet mature. See <<fee_sniping>>.
To remove the incentive to lie and strengthen the security of timelocks, a Bitcoin Improvement Proposal was proposed and activated at the same time as the BIPs for relative timelocks. This is BIP-113 which defines a new consensus measurement of time called _Median Time Past_.
To remove the incentive to lie and strengthen the security of timelocks, a Bitcoin Improvement Proposal was proposed and activated at the same time as the BIPs for relative timelocks. This is BIP-113, which defines a new consensus measurement of time called _Median Time Past_.
Median-Time-Past is calculated by taking the timestamps of the last 11 blocks and finding the median. That median time then becomes consensus time and it is used for all timelock calculations. By taking the midpoint from approximately two hours in the past, the influence of any one block's timestamp is reduced. By incorporating 11 blocks, no single miner can influence the timestamps in such a way as to gain fees from transactions with a time lock that hasn't yet matured.
Median-Time-Past changes the implementation of time calculations for nLocktime, CLTV, nSequence and CSV. The consensus time calculated by Median-Time-Past is always approximately one hour behind wall clock time. If you create timelock transactions, you should account for it when estimating the desired value to encode in nLocktime, nSequence, CLTV, and CSV.
Median-Time-Past is specified in BIP-113:
Median-Time-Past changes the implementation of time calculations for +nLocktime+, +CLTV+, +nSequence+, and +CSV+. The consensus time calculated by Median-Time-Past is always approximately one hour behind wall clock time. If you create timelock transactions, you should account for it when estimating the desired value to encode in +nLocktime+, +nSequence+, +CLTV+, and +CSV+.
BIP-113 Median time-past as endpoint for lock-time calculations:: https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki]
Median-Time-Past is specified in https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki[BIP-113].
[[fee_sniping]]
==== Timelock defense against fee-sniping
==== Timelock Defense Against Fee-Sniping
Fee-sniping is a theoretical attack scenario, where miners attempting to rewrite past blocks, "snipe" higher-fee transactions from future blocks to maximize their profitability.
Fee-sniping is a theoretical attack scenario, where miners attempting to rewrite past blocks "snipe" higher-fee transactions from future blocks to maximize their profitability.
For example, let's say the highest block in existence is block #100,000. If instead of attempting to mine block #100,001 to extend the chain, some miners are attempting to re-mine #100,000. These miners can choose to include any valid transaction (that hasn't been mined yet) in their candidate block #100,000. They don't have to re-mine the block with the same transactions. In fact, they have the incentive to select the most profitable (highest fee per kB) transactions to include in their block. They can include any transactions that were in the "old" block #100,000, as well as any transactions from the current mempool. Essentially they have the option to pull transactions from the "present", into the rewritten "past" when they re-create block #100,000.
For example, let's say the highest block in existence is block #100,000. If instead of attempting to mine block #100,001 to extend the chain, some miners are attempting to re-mine #100,000. These miners can choose to include any valid transaction (that hasn't been mined yet) in their candidate block #100,000. They don't have to re-mine the block with the same transactions. In fact, they have the incentive to select the most profitable (highest fee per kB) transactions to include in their block. They can include any transactions that were in the "old" block #100,000, as well as any transactions from the current mempool. Essentially they have the option to pull transactions from the "present," into the rewritten "past" when they re-create block #100,000.
Today, this attack is not very lucrative, because block reward is much higher than total fees per block. But at some point in the future, transaction fees will be the majority of the reward (or even the entirety of the reward). At that time, this scenario becomes inevitable.
To prevent "fee sniping", when Bitcoin Core creates transactions, it uses nLocktime to limit them to the "next block", by default. In our scenario above, Bitcoin Core would set nLocktime to 100,001 on any transaction it created. Under normal circumstances, this nLocktime has no effect - the transactions could only be included in block #100,001 anyway, it's the next block.
To prevent "fee sniping," when Bitcoin Core creates transactions, it uses +nLocktime+ to limit them to the "next block", by default. In our scenario, Bitcoin Core would set +nLocktime+ to 100,001 on any transaction it created. Under normal circumstances, this +nLocktime+ has no effect—the transactions could only be included in block #100,001 anyway; it's the next block.
But under a blockchain fork attack, the miners would not be able to pull high-fee transactions from the mempool, because all those transactions would be timelocked to block #100,001. They can only re-mine #100,000 with whatever transactions were valid at that time, essentially gaining no new fees.
To achieve this, Bitcoin Core sets the nLocktime on all new transactions to <current block # + 1> and sets the nSequence on all the inputs to 0xFFFFFFFE, to enable nLocktime.
To achieve this, Bitcoin Core sets the +nLocktime+ on all new transactions to <current block # + 1> and sets the +nSequence+ on all the inputs to 0xFFFFFFFE, to enable +nLocktime+.
=== Scripts with Flow Control (Conditional Clauses)
One of the more powerful features of Bitcoin Script is flow control also known as conditional clauses. You are probably familiar with flow control in various programming languages that use the construct IF...THEN...ELSE. Bitcoin conditional clauses look a bit different, but are essentially the same construct.
One of the more powerful features of Bitcoin Script is flow control, also known as conditional clauses. You are probably familiar with flow control in various programming languages that use the construct +IF...THEN...ELSE+. Bitcoin conditional clauses look a bit different, but are essentially the same construct.
At a basic level, bitcoin conditional opcodes allow us to construct a redeem script that has two ways of being unlocked, depending on a TRUE/FALSE outcome of evaluating a logical condition. For example, if x is TRUE, redeem script is A, ELSE redeem script is B.
At a basic level, bitcoin conditional opcodes allow us to construct a redeem script that has two ways of being unlocked, depending on a +TRUE+/+FALSE+ outcome of evaluating a logical condition. For example, if x is +TRUE+, redeem script is A, ELSE redeem script is B.
Additionally, bitcoin conditional expressions can be "nested" indefinitely, meaning that a conditional clause can contain another within it, which contains another, etc. Bitcoin Script flow control can be used to construct very complex scripts with hundreds or even thousands of possible execution paths. There is no limit to nesting, but consensus rules impose a limit on the maximum size, in bytes, of a script.
Bitcoin implements flow control using the IF, ELSE, ENDIF, and NOTIF opcodes. Additionally, conditional expressions can contain boolean operators such as BOOLAND, BOOLOR, and NOT.
Bitcoin implements flow control using the +IF+, +ELSE+, +ENDIF+, and +NOTIF+ opcodes. Additionally, conditional expressions can contain boolean operators such as +BOOLAND+, +BOOLOR+, and +NOT+.
At first glance, you may find the bitcoin's flow control scripts confusing. That is because Bitcoin Script is a stack language. The same way that +1 {plus} 1+ looks "backwards" when expressed as +1 1 ADD+, flow control clauses in bitcoin also look "backwards".
At first glance, you may find the bitcoin's flow control scripts confusing. That is because Bitcoin Script is a stack language. The same way that +1 {plus} 1+ looks "backwards" when expressed as +1 1 ADD+, flow control clauses in bitcoin also look "backward."
In most traditional (procedural) programming languages, flow control looks like this:
@ -420,7 +410,7 @@ else:
code to run in either case
----
In a stack-based language like Bitcoin Script, the logical condition comes before the +IF+, which makes it look "backwards", like this:
In a stack-based language like Bitcoin Script, the logical condition comes before the +IF+, which makes it look "backward," like this:
.Bitcoin Script flow control
----
@ -435,11 +425,11 @@ code to run in either case
When reading Bitcoin Script, remember that the condition being evaluated comes *before* the +IF+ opcode.
==== Conditional clauses with VERIFY opcodes
==== Conditional Clauses with VERIFY Opcodes
Another form of conditional in Bitcoin Script is any opcode that ends in +VERIFY+. The +VERIFY+ suffix means that if the condition evaluated is not TRUE, execution of the script terminates immediately and the transaction is deemed invalid.
Another form of conditional in Bitcoin Script is any opcode that ends in +VERIFY+. The +VERIFY+ suffix means that if the condition evaluated is not +TRUE+, execution of the script terminates immediately and the transaction is deemed invalid.
Unlike an +IF+ clause which offers alternative execution paths, the +VERIFY+ suffix acts as a _guard clause_, continuing only if a precondition is met.
Unlike an +IF+ clause, which offers alternative execution paths, the +VERIFY+ suffix acts as a _guard clause_, continuing only if a precondition is met.
For example, the following script requires Bob's signature and a pre-image (secret) that produces a specific hash. Both conditions must be satisfied to unlock:
@ -474,20 +464,20 @@ Bob's unlocking script is identical:
<Bob's Sig> <hash pre-image>
----
The script with +IF+ does the same thing as using an opcode with a +VERIFY+ suffix, they both operate as guard clauses. However, the +VERIFY+ construction is more efficient, using one fewer opcode.
The script with +IF+ does the same thing as using an opcode with a +VERIFY+ suffix; they both operate as guard clauses. However, the +VERIFY+ construction is more efficient, using one fewer opcode.
So, when do we use +VERIFY+ and when do we use +IF+? If all we are trying to do is to attach a pre-condition (guard clause), then +VERIFY+ is better. If however, we want to have more than one execution path (flow control), then we need an +IF...ELSE+ flow control clause.
[TIP]
====
An opcode such as EQUAL will push the result (TRUE/FALSE) onto the stack, leaving it there for evaluation by subsequent opcodes. In contrast, the opcode EQUALVERIFY suffix does not leave anything on the stack. Opcodes that end in VERIFY do not leave the result on the stack.
An opcode such as +EQUAL+ will push the result (+TRUE+/+FALSE+) onto the stack, leaving it there for evaluation by subsequent opcodes. In contrast, the opcode +EQUALVERIFY+ suffix does not leave anything on the stack. Opcodes that end in +VERIFY+ do not leave the result on the stack.
====
==== Using Flow Control in Scripts
A very common use for flow control in Bitcoin Script, is to construct a redeem script that offers multiple execution paths, each a different way of redeeming the UTXO.
A very common use for flow control in Bitcoin Script is to construct a redeem script that offers multiple execution paths, each a different way of redeeming the UTXO.
Let's look at a simple example, where we have two signers, Alice and Bob and either one is able to redeem. With multi-sig, this would be expressed as a 1-of-2 multisig script. For the sake of demonstration, we will do the same thing with an +IF+ clause:
Let's look at a simple example, where we have two signers, Alice and Bob and either one is able to redeem. With multisig, this would be expressed as a 1-of-2 multisig script. For the sake of demonstration, we will do the same thing with an +IF+ clause:
----
IF
@ -506,7 +496,7 @@ Alice redeems this with the unlocking script:
<Alice's Sig> 1
----
The +1+ at the end serves as the condition (TRUE), that will make the +IF+ clause execute the first redemption path, for which Alice has a signature.
The +1+ at the end serves as the condition (+TRUE+), that will make the +IF+ clause execute the first redemption path, for which Alice has a signature.
For Bob to redeem this, he would have to choose the second execution path, by giving a +FALSE+ value to the +IF+ clause:
@ -530,7 +520,7 @@ ELSE
ENDIF
----
In this scenario, there are three execution paths (+script A+, +script B+ and +script C+). The unlocking script provides a path in the form of a sequence of +TRUE+ or +FALSE+ values. To select path +script B+ for example, the unlocking script must end in +1 0+ (+TRUE+, +FALSE+). These values will be pushed onto the stack, so that the second value (+FALSE+) ends up at the top of the stack. The outer +IF+ clause pops the +FALSE+ value and executes the first +ELSE+ clause. Then the +TRUE+ value moves to the top of the stack and is evaluated by the inner (nested) +IF+, selecting the +B+ execution path.
In this scenario, there are three execution paths (+script A+, +script B+, and +script C+). The unlocking script provides a path in the form of a sequence of +TRUE+ or +FALSE+ values. To select path +script B+, for example, the unlocking script must end in +1 0+ (+TRUE+, +FALSE+). These values will be pushed onto the stack, so that the second value (+FALSE+) ends up at the top of the stack. The outer +IF+ clause pops the +FALSE+ value and executes the first +ELSE+ clause. Then the +TRUE+ value moves to the top of the stack and is evaluated by the inner (nested) +IF+, selecting the +B+ execution path.
Using this construct, we can build redeem scripts with tens or hundreds of execution paths, each offering a different way to redeem the UTXO. To spend, we construct an unlocking script that navigates the execution path by putting the appropriate +TRUE+ and +FALSE+ values on the stack at each flow control point.
@ -540,7 +530,7 @@ In this section we combine many of the concepts from this chapter into a single
Our example uses the story of Mohammed, the company owner in Dubai who is operating an import/export business.
In this example, Mohammed wishes to construct a company capital account with flexible rules. The scheme he creates requires different levels of authorization depending on timelocks. The participants in the multisig scheme are Mohammed, his two partners Saeed and Zaira and their company lawyer Abdul. The three partners make decisions based on a majority rule, so two of three must agree. However, in the case of a problem with their keys, they want their lawyer to be able to recover the funds with one of the three partner signatures. Finally, if all partners are unavailable or incapacitated for a while, they want the lawyer to be able to manage the account directly.
In this example, Mohammed wishes to construct a company capital account with flexible rules. The scheme he creates requires different levels of authorization depending on timelocks. The participants in the multisig scheme are Mohammed, his two partners Saeed and Zaira, and their company lawyer Abdul. The three partners make decisions based on a majority rule, so two of three must agree. However, in the case of a problem with their keys, they want their lawyer to be able to recover the funds with one of the three partner signatures. Finally, if all partners are unavailable or incapacitated for a while, they want the lawyer to be able to manage the account directly.
Here's the script that Mohammed designs to achieve this:
@ -573,10 +563,10 @@ In the first execution path, this script operates as a simple 2-of-3 multisig wi
[TIP]
====
The +0+ at the beginning of this unlocking script is because of a bug in CHECKMULTISIG which pops an extra value from the stack. The extra value is disregarded by the CHECKMULTISIG, but it must be present or the script fails. Pushing +0+ (customarily) is a workaround to the bug, as described in <<multisig_bug>>
The +0+ at the beginning of this unlocking script is because of a bug in +CHECKMULTISIG+ that pops an extra value from the stack. The extra value is disregarded by the +CHECKMULTISIG+, but it must be present or the script fails. Pushing +0+ (customarily) is a workaround to the bug, as described in <<multisig_bug>>
====
The second execution path can only be used after 30 days have elapsed from the creation of the UTXO. At that time, it requires the signature of Abdul the lawyer and one of the three partners (a 1-of-3 multisig). This is achieved by line 7 which sets the quorum for the multisig to +1+. To select this execution path, the unlocking script would end in +FALSE TRUE+:
The second execution path can only be used after 30 days have elapsed from the creation of the UTXO. At that time, it requires the signature of Abdul the lawyer and one of the three partners (a 1-of-3 multisig). This is achieved by line 7, which sets the quorum for the multisig to +1+. To select this execution path, the unlocking script would end in +FALSE TRUE+:
.Unlocking script for the second execution path (Lawyer + 1-of-3)
----
@ -585,7 +575,7 @@ The second execution path can only be used after 30 days have elapsed from the c
[TIP]
====
Why +FALSE TRUE+? Isn't that backwards? Because the two values are pushed on to the stack, with +FALSE+ pushed first, then +TRUE+ pushed second. +TRUE+ is therefore popped *first* by the first +IF+ opcode.
Why +FALSE TRUE+? Isn't that backwards? Because the two values are pushed on to the stack, with +FALSE+ pushed first, then +TRUE+ pushed second. +TRUE+ is therefore popped _first_ by the first +IF+ opcode.
====
Finally, the third execution path allows Abdul the lawyer to spend the funds alone, but only after 90 days. To select this execution path, the unlocking script has to end in +FALSE+:
@ -601,9 +591,9 @@ A few more things to consider when reading this example. See if you can find the
* Why can't the lawyer redeem the third execution path at any time, by selecting it with +FALSE+ on the unlocking script?
* How many execution paths can be used 5, 35 and 105 days respectively after the UTXO is mined?
* How many execution paths can be used 5, 35, and 105 days, respectively, after the UTXO is mined?
* Are the funds lost if the lawyer loses their key? Does your answer change if 91 days have elapsed?
* Are the funds lost if the lawyer loses his key? Does your answer change if 91 days have elapsed?
* How do the partners "reset" the clock every 29 or 89 days to prevent the lawyer from accessing the funds?
((("bitcoin network", id="ix_ch08-asciidoc0", range="startofrange")))((("bitcoin network","architecture of")))((("peer-to-peer networks")))Bitcoin is structured as a peer-to-peer network architecture on top of the Internet. The term peer-to-peer, or P2P, means that the computers that participate in the network are peers to each other, that they are all equal, that there are no "special" nodes, and that all nodes share the burden of providing network services. The network nodes interconnect in a mesh network with a "flat" topology. There is no server, no centralized service, and no hierarchy within the network. Nodes in a peer-to-peer network both provide and consume services at the same time with reciprocity acting as the incentive for participation. Peer-to-peer networks are inherently resilient, decentralized, and open. The preeminent example of a P2P network architecture was the early Internet itself, where nodes on the IP network were equal. Today's Internet architecture is more hierarchical, but the Internet Protocol still retains its flat-topology essence. Beyond bitcoin, the largest and most successful application of P2P technologies is file sharing with Napster as the pioneer and BitTorrent as the most recent evolution of the architecture.
((("bitcoin network", id="ix_ch08-asciidoc0", range="startofrange")))((("bitcoin network","architecture of")))((("peer-to-peer networks")))Bitcoin is structured as a peer-to-peer network architecture on top of the internet. The term peer-to-peer, or P2P, means that the computers that participate in the network are peers to each other, that they are all equal, that there are no "special" nodes, and that all nodes share the burden of providing network services. The network nodes interconnect in a mesh network with a "flat" topology. There is no server, no centralized service, and no hierarchy within the network. Nodes in a peer-to-peer network both provide and consume services at the same time with reciprocity acting as the incentive for participation. Peer-to-peer networks are inherently resilient, decentralized, and open. The preeminent example of a P2P network architecture was the early internet itself, where nodes on the IP network were equal. Today's internet architecture is more hierarchical, but the Internet Protocol still retains its flat-topology essence. Beyond bitcoin, the largest and most successful application of P2P technologies is file sharing with Napster as the pioneer and BitTorrent as the most recent evolution of the architecture.
Bitcoin's P2P network architecture is much more than a topology choice. Bitcoin is a peer-to-peer digital cash system by design, and the network architecture is both a reflection and a foundation of that core characteristic. Decentralization of control is a core design principle and that can only be achieved and maintained by a flat, decentralized P2P consensus network.
((("bitcoin network","defined")))The term "bitcoin network" refers to the collection of nodes running the bitcoin P2P protocol. In addition to the bitcoin P2P protocol, there are other protocols such as((("Stratum (STM) mining protocol"))) Stratum, which are used for mining and lightweight or mobile wallets. These additional protocols are provided by gateway routing servers that access the bitcoin network using the bitcoin P2P protocol, and then extend that network to nodes running other protocols. For example, Stratum servers connect Stratum mining nodes via the Stratum protocol to the main bitcoin network and bridge the Stratum protocol to the bitcoin P2P protocol. We use the term "extended bitcoin network" to refer to the overall network that includes the bitcoin P2P protocol, pool-mining protocols, the Stratum protocol, and any other related protocols connecting the components of the bitcoin system.
=== Nodes Types and Roles
=== Node Types and Roles
((("bitcoin network","nodes")))((("nodes","roles of")))((("nodes","types of")))Although nodes in the bitcoin P2P network are equal, they may take on different roles depending on the functionality they are supporting. A bitcoin node is a collection of functions: routing, the blockchain database, mining, and wallet services. A full node with all four of these functions is shown in <<full_node_reference>>.
@ -17,11 +17,11 @@ Bitcoin's P2P network architecture is much more than a topology choice. Bitcoin
.A bitcoin network node with all four functions: wallet, miner, full blockchain database, and network routing
All nodes include the routing function to participate in the network and might include other functionality. All nodes validate and propagate transactions and blocks, and discover and maintain connections to peers. In the full-node example in <<full_node_reference>>, the routing function is indicated by an orange circle named "Network Routing Node.", or with the letter "N".
All nodes include the routing function to participate in the network and might include other functionality. All nodes validate and propagate transactions and blocks, and discover and maintain connections to peers. In the full-node example in <<full_node_reference>>, the routing function is indicated by an orange circle named "Network Routing Node", or with the letter "N".
Some nodes, called full nodes, also maintain a complete and up-to-date copy of the blockchain. Full nodes can autonomously and authoritatively verify any transaction without external reference. Some nodes maintain only a subset of the blockchain and verify transactions using a method called((("simplified payment verification (SPV) nodes","defined"))) _simplified payment verification_, or SPV. These nodes are known as SPV or lightweight nodes. In the full-node example in the figure, the full-node blockchain database function is indicated by a blue circle named "Full Blockchain.", or the letter "B". In <<bitcoin_network>>, SPV nodes are drawn without the blue circle, showing that they do not have a full copy of the blockchain.
Some nodes, called full nodes, also maintain a complete and up-to-date copy of the blockchain. Full nodes can autonomously and authoritatively verify any transaction without external reference. Some nodes maintain only a subset of the blockchain and verify transactions using a method called((("simplified payment verification (SPV) nodes","defined"))) _simplified payment verification_, or SPV. These nodes are known as SPV or lightweight nodes. In the full-node example in the figure, the full-node blockchain database function is indicated by a blue circle named "Full Blockchain" or the letter "B". In <<bitcoin_network>>, SPV nodes are drawn without the blue circle, showing that they do not have a full copy of the blockchain.
Mining nodes compete to create new blocks by running specialized hardware to solve the proof-of-work algorithm. Some mining nodes are also full nodes, maintaining a full copy of the blockchain, while others are lightweight nodes participating in pool mining and depending on a pool server to maintain a full node. The mining function is shown in the full node as a black circle named "Miner", or the letter "M".
Mining nodes compete to create new blocks by running specialized hardware to solve the Proof-of-Work algorithm. Some mining nodes are also full nodes, maintaining a full copy of the blockchain, while others are lightweight nodes participating in pool mining and depending on a pool server to maintain a full node. The mining function is shown in the full node as a black circle named "Miner" or the letter "M".
User wallets might be part of a full node, as is usually the case with desktop bitcoin clients. Increasingly, many user wallets, especially those running on resource-constrained devices such as smartphones, are SPV nodes. The wallet function is shown in <<full_node_reference>> as a green circle named "Wallet" or the letter "W".
@ -31,7 +31,7 @@ In addition to the main node types on the bitcoin P2P protocol, there are server
=== The Extended Bitcoin Network
((("bitcoin network","extended")))((("extended bitcoin network")))The main bitcoin network, running the bitcoin P2P protocol, consists of between 5,000 and 8,000 listening nodes running various versions of the bitcoin reference client (Bitcoin Core) and a few hundred nodes running various other implementations of the bitcoin P2P protocol, such as((("BitcoinJ library")))((("btcd")))((("libbitcoin library")))((("bcoin"))) BitcoinJ, Libbitcoin, btcd and bcoin. A small percentage of the nodes on the bitcoin P2P network are also mining nodes, competing in the mining process, validating transactions, and creating new blocks. Various large companies interface with the bitcoin network by running full-node clients based on the Bitcoin Core client, with full copies of the blockchain and a network node, but without mining or wallet functions. These nodes act as network edge routers, allowing various other services (exchanges, wallets, block explorers, merchant payment processing) to be built on top.
((("bitcoin network","extended")))((("extended bitcoin network")))The main bitcoin network, running the bitcoin P2P protocol, consists of between 5,000 and 8,000 listening nodes running various versions of the bitcoin reference client (Bitcoin Core) and a few hundred nodes running various other implementations of the bitcoin P2P protocol, such as((("BitcoinJ library")))((("btcd")))((("libbitcoin library")))((("bcoin"))) BitcoinJ, Libbitcoin, btcd, and bcoin. A small percentage of the nodes on the bitcoin P2P network are also mining nodes, competing in the mining process, validating transactions, and creating new blocks. Various large companies interface with the bitcoin network by running full-node clients based on the Bitcoin Core client, with full copies of the blockchain and a network node, but without mining or wallet functions. These nodes act as network edge routers, allowing various other services (exchanges, wallets, block explorers, merchant payment processing) to be built on top.
The extended bitcoin network includes the network running the bitcoin P2P protocol, described earlier, as well as nodes running specialized protocols. Attached to the main bitcoin P2P network are a number of((("mining pools","on the bitcoin network"))) pool servers and protocol gateways that connect nodes running other protocols. These other protocol nodes are mostly pool mining nodes (see <<mining>>) and lightweight wallet clients, which do not carry a full copy of the blockchain.
While the bitcoin P2P network serves the general needs of a broad variety of node types, it exhibits too high network latency for the specialized needs of bitcoin mining nodes.
Bitcoin miners are engaged in a time-sensitive competition to solve the Proof-of-Work problem and extend the blockchain (See <<mining>>). While participating in this competition, bitcoin miners must minimize the time between the propagation of a winning block and the beginning of the next round of competition. In mining, network latency is directly related to profit margins.
Bitcoin miners are engaged in a time-sensitive competition to solve the Proof-of-Work problem and extend the blockchain (see <<mining>>). While participating in this competition, bitcoin miners must minimize the time between the propagation of a winning block and the beginning of the next round of competition. In mining, network latency is directly related to profit margins.
A _Bitcoin Relay Network_ is a network that attempts minimize the latency in the transmission of blocks between miners. The original Bitcoin Relay Network (http://www.bitcoinrelaynetwork.org/) was created by core developer Matt Corallo in 2015 to enable fast synchronization of blocks between miners with very low latency. The network consisted of several specialized nodes hosted on Amazon Web Services infrastructure around the world and served to connect the majority of miners and mining pools.
A _Bitcoin Relay Network_ is a network that attempts minimize the latency in the transmission of blocks between miners. The original http://www.bitcoinrelaynetwork.org[Bitcoin Relay Network] was created by core developer Matt Corallo in 2015 to enable fast synchronization of blocks between miners with very low latency. The network consisted of several specialized nodes hosted on Amazon Web Services infrastructure around the world and served to connect the majority of miners and mining pools.
The original Bitcoin Relay Network was replaced in 2016 with the introduction of the _Fast Internet Bitcoin Relay Engine_ or _FIBRE_ (http://bitcoinfibre.org/), also created by core developer Matt Corallo. FIBRE is a UDP-based relay network that relays blocks within a network of nodes. FIBRE implements the _Compact Block_ (see <<compact_block>>) optimization to further reduce the amount of data transmitted and the network latency.
The original Bitcoin Relay Network was replaced in 2016 with the introduction of the _Fast Internet Bitcoin Relay Engine_ or http://bitcoinfibre.org[_FIBRE_], also created by core developer Matt Corallo. FIBRE is a UDP-based relay network that relays blocks within a network of nodes. FIBRE implements the _Compact Block_ (see <<compact_block>>) optimization to further reduce the amount of data transmitted and the network latency.
Another relay network (still in the proposal phase) is _Falcon_ (http://www.falcon-net.org/about), based on research at Cornell University. Falcon uses "cut-through-routing" instead of "store-and-forward" to reduce latency by propagating parts of blocks as they are received rather than waiting until a complete block is received.
Another relay network (still in the proposal phase) is http://www.falcon-net.org/about[_Falcon_], based on research at Cornell University. Falcon uses "cut-through-routing" instead of "store-and-forward" to reduce latency by propagating parts of blocks as they are received rather than waiting until a complete block is received.
Relay networks are not replacements for bitcoin's P2P network. Instead they are overlay networks that provide additional connectivity between nodes with specialized needs. Like freeways are not replacements for rural roads, but shortcuts between two points with heavy traffic, you still need small roads to connect to the freeways.
@ -70,14 +70,14 @@ Relay networks are not replacements for bitcoin's P2P network. Instead they are
+nTime+:: The current time
+addrYou+:: The IP address of the remote node as seen from this node
+addrMe+:: The IP address of the local node, as discovered by the local node
+subver+:: A sub-version showing the type of software running on this node (e.g., "/Satoshi:0.9.2.1/")+
+subver+:: A sub-version showing the type of software running on this node (e.g., +/Satoshi:0.9.2.1/+)
+BestHeight+:: The block height of this node's blockchain
(See http://bit.ly/1qlsC7w[GitHub] for an example of the +version+ network message.)
The +version+ message is always the first message sent by any peer to another peer. The local peer receiving a +version+ message will examine the remote peer's reported +nVersion+ and decide if the remote peer is compatible. If the remote peer is compatible, the local peer will acknowledge the +version+ message and establish a connection, by sending a +verack+.
How does a new node find peers? The first method is to query DNS using a number of ((("nodes","seed")))((("DNS seed")))"DNS seeds," which are DNS servers that provide a list of IP addresses of bitcoin nodes. Some of those DNS seeds provide a static list of IP addresses of stable bitcoin listening nodes. Some of the DNS seeds are custom implementations of BIND (Berkeley Internet Name Daemon) that return a random subset from a list of bitcoin node addresses collected by a crawler or a long-running bitcoin node. The Bitcoin Core client contains the names of five different DNS seeds. The diversity of ownership and diversity of implementation of the different DNS seeds offers a high level or reliability for the initial bootstrapping process. In the Bitcoin Core client, the option to use the DNS seeds is controlled by the option switch +-dnsseed+ (set to 1 by default, to use the DNS seed).
How does a new node find peers? The first method is to query DNS using a number of ((("nodes","seed")))((("DNS seed")))"DNS seeds," which are DNS servers that provide a list of IP addresses of bitcoin nodes. Some of those DNS seeds provide a static list of IP addresses of stable bitcoin listening nodes. Some of the DNS seeds are custom implementations of BIND (Berkeley Internet Name Daemon) that return a random subset from a list of bitcoin node addresses collected by a crawler or a long-running bitcoin node. The Bitcoin Core client contains the names of five different DNS seeds. The diversity of ownership and diversity of implementation of the different DNS seeds offers a high level of reliability for the initial bootstrapping process. In the Bitcoin Core client, the option to use the DNS seeds is controlled by the option switch +-dnsseed+ (set to 1 by default, to use the DNS seed).
Alternatively, a bootstrapping node that knows nothing of the network must be given the IP address of at least one bitcoin node, after which it can establish connections through further introductions. The command-line argument +-seednode+ can be used to connect to one node just for introductions, using it as a seed. After the initial seed node is used to form introductions, the client will disconnect from it and use the newly discovered peers.
@ -154,7 +154,7 @@ There are a few alternative implementations of full blockchain bitcoin clients,
=== Exchanging "Inventory"
((("blockchains","creating on nodes")))((("blockchains","on new nodes")))((("blocks","on new nodes")))((("full nodes","creating full blockchains on")))The first thing a full node will do once it connects to peers is try to construct a complete blockchain. If it is a brand-new node and has no blockchain at all, it only knows one block, the genesis block, which is statically embedded in the client software. Starting with block #0 (the genesis block), the new node will have to download hundreds of thousands of blocks to synchronize with the network and re-establish the full blockchain.
((("blockchains","creating on nodes")))((("blockchains","on new nodes")))((("blocks","on new nodes")))((("full nodes","creating full blockchains on")))The first thing a full node will do once it connects to peers is try to construct a complete blockchain. If it is a brand-new node and has no blockchain at all, it only knows one block, the genesis block, which is statically embedded in the client software. Starting with block #0 (the genesis block), the new node will have to download hundreds of thousands of blocks to synchronize with the network and reestablish the full blockchain.
((("syncing the blockchain")))The process of syncing the blockchain starts with the +version+ message, because that contains +BestHeight+, a node's current blockchain height (number of blocks). A node will see the +version+ messages from its peers, know how many blocks they each have, and be able to compare to how many blocks it has in its own blockchain. Peered nodes will exchange a +getblocks+ message that contains the hash (fingerprint) of the top block on their local blockchain. One of the peers will be able to identify the received hash as belonging to a block that is not at the top, but rather belongs to an older block, thus deducing that its own local blockchain is longer than its peer's.
@ -164,6 +164,10 @@ Let's assume, for example, that a node only has the genesis block. It will then
This process of comparing the local blockchain with the peers and retrieving any missing blocks happens any time a node goes offline for any period of time. Whether a node has been offline for a few minutes and is missing a few blocks, or a month and is missing a few thousand blocks, it starts by sending +getblocks+, gets an +inv+ response, and starts downloading the missing blocks. <<inventory_synchronization>> shows the inventory and block propagation protocol.
[[inventory_synchronization]]
.Node synchronizing the blockchain by retrieving blocks from a peer
@ -171,17 +175,13 @@ This process of comparing the local blockchain with the peers and retrieving any
((("blockchains","on SPV nodes")))SPV nodes download only the block headers and do not download the transactions included in each block. The resulting chain of blocks, without transactions, is 1,000 times smaller than the full blockchain. SPV nodes cannot construct a full picture of all the UTXOs that are available for spending because they do not know about all the transactions on the network. SPV nodes verify transactions using a slightly different methodology that relies on peers to provide partial views of relevant parts of the blockchain on demand.
[[inventory_synchronization]]
.Node synchronizing the blockchain by retrieving blocks from a peer
As an analogy, a full node is like a tourist in a strange city, equipped with a detailed map of every street and every address. By comparison, an SPV node is like a tourist in a strange city asking random strangers for turn-by-turn directions while knowing only one main avenue. Although both tourists can verify the existence of a street by visiting it, the tourist without a map doesn't know what lies down any of the side streets and doesn't know what other streets exist. Positioned in front of 23 Church Street, the tourist without a map cannot know if there are a dozen other "23 Church Street" addresses in the city and whether this is the right one. The mapless tourist's best chance is to ask enough people and hope some of them are not trying to mug him.
Simplified payment verification verifies transactions by reference to their _depth_ in the blockchain instead of their _height_. Whereas a full blockchain node will construct a fully verified chain of thousands of blocks and transactions reaching down the blockchain (back in time) all the way to the genesis block, an SPV node will verify the chain of all blocks (but not all transactions) and link that chain to the transaction of interest.
For example, when examining a transaction in block 300,000, a full node links all 300,000 blocks down to the genesis block and builds a full database of UTXO, establishing the validity of the transaction by confirming that the UTXO remains unspent. An SPV node cannot validate whether the UTXO is unspent. Instead, the SPV node will establish a link between the transaction and the block that contains it, using a((("merkle trees","SPV and"))) _merkle path_ (see <<merkle_trees>>). Then, the SPV node waits until it sees the six blocks 300,001 through 300,006 piled on top of the block containing the transaction and verifies it by establishing its depth under blocks 300,006 to 300,001. The fact that other nodes on the network accepted block 300,000 and then did the necessary work to produce six more blocks on top of it is proof, by proxy, that the transaction was not a double-spend.
An SPV node cannot be persuaded that a transaction exists in a block when the transaction does not in fact exist. The SPV node establishes the existence of a transaction in a block by requesting a merkle path proof and by validating the proof of work in the chain of blocks. However, a transaction's existence can be "hidden" from an SPV node. An SPV node can definitely prove that a transaction exists but cannot verify that a transaction, such as a double-spend of the same UTXO, doesn't exist because it doesn't have a record of all transactions. This vulnerability can be used in a denial-of-service attack or for a double-spending attack against SPV nodes. To defend against this, an SPV node needs to connect randomly to several nodes, to increase the probability that it is in contact with at least one honest node. This need to randomly connect means that SPV nodes also are vulnerable to network partitioning attacks or Sybil attacks, where they are connected to fake nodes or fake networks and do not have access to honest nodes or the real bitcoin network.
An SPV node cannot be persuaded that a transaction exists in a block when the transaction does not in fact exist. The SPV node establishes the existence of a transaction in a block by requesting a merkle path proof and by validating the Proof-of-Work in the chain of blocks. However, a transaction's existence can be "hidden" from an SPV node. An SPV node can definitely prove that a transaction exists but cannot verify that a transaction, such as a double-spend of the same UTXO, doesn't exist because it doesn't have a record of all transactions. This vulnerability can be used in a denial-of-service attack or for a double-spending attack against SPV nodes. To defend against this, an SPV node needs to connect randomly to several nodes, to increase the probability that it is in contact with at least one honest node. This need to randomly connect means that SPV nodes also are vulnerable to network partitioning attacks or Sybil attacks, where they are connected to fake nodes or fake networks and do not have access to honest nodes or the real bitcoin network.
For most practical purposes, well-connected SPV nodes are secure enough, striking a balance between resource needs, practicality, and security. For infallible security, however, nothing beats running a full blockchain node.
@ -201,15 +201,15 @@ Because SPV nodes need to retrieve specific transactions in order to selectively
Shortly after the introduction of SPV/lightweight nodes, the bitcoin developers added a feature called _bloom filters_ to address the privacy risks of SPV nodes. Bloom filters allow SPV nodes to receive a subset of the transactions without revealing precisely which addresses they are interested in, through a filtering mechanism that uses probabilities rather than fixed patterns.(((range="endofrange", startref="ix_ch08-asciidoc6")))(((range="endofrange", startref="ix_ch08-asciidoc5a")))(((range="endofrange", startref="ix_ch08-asciidoc5")))
[[bloom_filters]]
=== Bloom filters
=== Bloom Filters
((("bitcoin network","bloom filters and", id="ix_ch08-asciidoc7", range="startofrange")))((("bloom filters", id="ix_ch08-asciidoc8", range="startofrange")))((("Simplified Payment Verification (SPV) nodes","bloom filters and", id="ix_ch08-asciidoc9", range="startofrange")))A bloom filter is a probabilistic search filter, a way to describe a desired pattern without specifying it exactly. Bloom filters offer an efficient way to express a search pattern while protecting privacy. They are used by SPV nodes to ask their peers for transactions matching a specific pattern, without revealing exactly which addresses, keys or transactions they are searching for.
((("bitcoin network","bloom filters and", id="ix_ch08-asciidoc7", range="startofrange")))((("bloom filters", id="ix_ch08-asciidoc8", range="startofrange")))((("Simplified Payment Verification (SPV) nodes","bloom filters and", id="ix_ch08-asciidoc9", range="startofrange")))A bloom filter is a probabilistic search filter, a way to describe a desired pattern without specifying it exactly. Bloom filters offer an efficient way to express a search pattern while protecting privacy. They are used by SPV nodes to ask their peers for transactions matching a specific pattern, without revealing exactly which addresses, keys, or transactions they are searching for.
In our previous analogy, a tourist without a map is asking for directions to a specific address, "23 Church St." If she asks strangers for directions to this street, she inadvertently reveals her destination. A bloom filter is like asking, "Are there any streets in this neighborhood whose name ends in R-C-H?" A question like that reveals slightly less about the desired destination than asking for "23 Church St." Using this technique, a tourist could specify the desired address in more detail as "ending in U-R-C-H" or less detail as "ending in H." By varying the precision of the search, the tourist reveals more or less information, at the expense of getting more or less specific results. If she asks a less specific pattern, she gets a lot more possible addresses and better privacy, but many of the results are irrelevant. If she asks for a very specific pattern, she gets fewer results but loses privacy.
Bloom filters serve this function by allowing an SPV node to specify a search pattern for transactions that can be tuned toward precision or privacy. A more specific bloom filter will produce accurate results, but at the expense of revealing what patterns the SPV node is interested in, thus revealing the addresses owned by the user's wallet. A less specific bloom filter will produce more data about more transactions, many irrelevant to the node, but will allow the node to maintain better privacy.
==== How bloom filters work
==== How Bloom Filters Work
Bloom filters are implemented as a variable-size array of N binary digits (a bit field) and a variable number of M hash functions. The hash functions are designed to always produce an output that is between 1 and N, corresponding to the array of binary digits. The hash functions are generated deterministically, so that any node implementing a bloom filter will always use the same hash functions and get the same results for a specific input. By choosing different length (N) bloom filters and a different number (M) of hash functions, the bloom filter can be tuned, varying the level of accuracy and therefore privacy.
@ -251,20 +251,20 @@ On the contrary, if a pattern is tested against the bloom filter and any one of
.Testing the existence of pattern "Y" in the bloom filter. The result is a definitive negative match, meaning "Definitely Not!"
image::images/mbc2_0812.png[]
=== How SPV nodes use bloom filters
=== How SPV Nodes Use Bloom Filters
((("inventory updates, bloom filters and")))Bloom filters are used to filter the transactions (and blocks containing them) that an SPV node receives from its peers, selecting only transactions of interest to the SPV node without revealing which addresses or keys it is interested in.
An SPV node will initialize a bloom filter as "empty" and in that state the bloom filter will not match any patterns. The SPV node will then make a list of all the addresses, keys and hashes that it is interested in. It will do this by extracting the ((("pay-to-public-key-hash (P2PKH)","bloom filters and")))public-key-hash and ((("pay-to-script-hash (P2SH)","bloom filters and"))) script-hash and transaction IDs from any unspent transaction outputs (UTXO) controlled by its wallet. The SPV node then adds each of these to the bloom filter, so that the bloom filter will "match" if these patterns are present in a transaction, without revealing the patterns themselves.
An SPV node will initialize a bloom filter as "empty" and in that state the bloom filter will not match any patterns. The SPV node will then make a list of all the addresses, keys, and hashes that it is interested in. It will do this by extracting the ((("pay-to-public-key-hash (P2PKH)","bloom filters and")))public-key-hash and ((("pay-to-script-hash (P2SH)","bloom filters and"))) script-hash and transaction IDs from any unspent transaction outputs (UTXO) controlled by its wallet. The SPV node then adds each of these to the bloom filter, so that the bloom filter will "match" if these patterns are present in a transaction, without revealing the patterns themselves.
The SPV node will then send a((("filterload message"))) +filterload+ message to the peer, containing the bloom filter to use on the connection. On the peer bloom filters are checked against each incoming transaction. The full node checks several parts of the transaction against the bloom filter, looking for a match including:
* The transaction ID
* Each of the transaction outputs' locking scripts' data components (every key and hash in the script)
* The data components from the locking scripts of each of the transaction outputs (every key and hash in the script)
* Each of the transaction inputs
* Each of the input signatures data components (or witness scripts)
By checking against all these components, bloom filters can be used to match public key hashes, scripts, OP_RETURN values, public keys in signatures, or any future component of a smart contract or complex script.
By checking against all these components, bloom filters can be used to match public key hashes, scripts, +OP_RETURN+ values, public keys in signatures, or any future component of a smart contract or complex script.
After a filter is established, the peer will then test each transaction's outputs against the bloom filter. Only transactions that match the filter are sent to the node.
@ -274,14 +274,12 @@ As the full node sends transactions to the SPV node, the SPV node discards any f
The node setting the bloom filter can interactively add patterns to the filter by sending a((("filteradd message"))) +filteradd+ message. To clear the bloom filter, the node can send a((("filterclear message"))) +filterclear+ message. Because it is not possible to remove a pattern from a bloom filter, a node has to clear and resend a new bloom filter if a pattern is no longer desired.(((range="endofrange", startref="ix_ch08-asciidoc9")))(((range="endofrange", startref="ix_ch08-asciidoc8")))(((range="endofrange", startref="ix_ch08-asciidoc7")))
The network protocol and bloom filter mechanism for SPV nodes is defined in BIP-37:
The network protocol and bloom filter mechanism for SPV nodes is defined in https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki[BIP-37 (Peer Services)].
Nodes that implement Simple Payment Verification have weaker privacy than a full node. A full node receives all transactions and therefore reveals no information about whether it is using some address in its wallet. An SPV node receives a filtered list of transactions related to the addresses that are in its wallet. As a result, it reduces the privacy of the owner.
Nodes that implement SPV have weaker privacy than a full node. A full node receives all transactions and therefore reveals no information about whether it is using some address in its wallet. An SPV node receives a filtered list of transactions related to the addresses that are in its wallet. As a result, it reduces the privacy of the owner.
Bloom filters are a way to reduce the loss of privacy. Without them, an SPV node would have to explicitly list the addresses it was interested in, creating a serious breach of privacy. However, even with bloom filters, an adversary monitoring the traffic of an SPV client or connected to it directly as a node in the P2P network can collect enough information over time to learn the addresses in the wallet of the SPV client.
You should see "tor: ADD_ONION successful" in the logs, indicating that Bitcoin Core has added a hidden service to the Tor network.
You can find more instructions on running Bitcoin Core as a Tor hidden service in the Bitcoin Core documentation (+docs/tor.md+) and various online tutorials.
You can find more instructions on running Bitcoin Core as a Tor hidden service in the Bitcoin Core documentation (_docs/tor.md_) and various online tutorials.
==== Peer to peer authentication and encryption
==== Peer-to-Peer Authentication and Encryption
Two Bitcoin Improvement Proposals, BIP-150 and BIP-151, add support for Peer-to-Peer authentication and encryption in the bitcoin P2P network. These two BIPs define optional services that may be offered by compatible bitcoin nodes. BIP-151 enables negotiated encryption for all communications between two nodes that support BIP-151. BIP-150 offers optional peer-authentication which allows nodes to authenticate each other's identity using ECDSA and private keys. BIP-150 requires that prior to authentication the two nodes have established encrypted communications as per BIP-151.
Two Bitcoin Improvement Proposals, BIP-150 and BIP-151, add support for Peer-to-Peer authentication and encryption in the bitcoin P2P network. These two BIPs define optional services that may be offered by compatible bitcoin nodes. BIP-151 enables negotiated encryption for all communications between two nodes that support BIP-151. BIP-150 offers optional peer-authentication that allows nodes to authenticate each other's identity using ECDSA and private keys. BIP-150 requires that prior to authentication the two nodes have established encrypted communications as per BIP-151.
As of January 2017, BIP-150 and BIP-151 are not implemented in Bitcoin Core. However, the two proposals have been implemented by at least one alternative bitcoin client, named bcoin.((("bcoin")))
BIP-150 and BIP-151 allow users to run SPV clients that connect to a trusted full node, using encryption and authentication to protect the privacy of the SPV client.
Additionally, authentication can be used to create networks of trusted bitcoin nodes and prevent Man-In-The-Middle attacks. Finally, peer-to-peer encryption, if deployed broadly, would strengthen the resistance of bitcoin to traffic analysis and privacy eroding surveillance, especially in totalitarian countries where Internet use is heavily controlled and monitored.
Additionally, authentication can be used to create networks of trusted bitcoin nodes and prevent Man-in-the-Middle attacks. Finally, peer-to-peer encryption, if deployed broadly, would strengthen the resistance of bitcoin to traffic analysis and privacy eroding surveillance, especially in totalitarian countries where internet use is heavily controlled and monitored.
BIP-151 (Peer-to-Peer Communication Encryption):: https://github.com/bitcoin/bips/blob/master/bip-0151.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0151.mediawiki]
[[transaction_pools]]
The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0150.mediawiki[BIP-150 (Peer Authentication)] and https://github.com/bitcoin/bips/blob/master/bip-0151.mediawiki[BIP-151 (Peer-to-Peer Communication Encryption)].
@ -24,7 +24,7 @@ One way to think about the blockchain is like layers in a geological formation,
|Size| Field | Description
| 4 bytes | Block Size | The size of the block, in bytes, following this field
| 80 bytes | Block Header | Several fields form the block header
| 1-9 bytes (VarInt) | Transaction Counter | How many transactions follow
| 1–9 bytes (VarInt) | Transaction Counter | How many transactions follow
| Variable | Transactions | The transactions recorded in this block
|=======
@ -43,11 +43,11 @@ One way to think about the blockchain is like layers in a geological formation,
| 32 bytes | Previous Block Hash | A reference to the hash of the previous (parent) block in the chain
| 32 bytes | Merkle Root | A hash of the root of the merkle tree of this block's transactions
| 4 bytes | Timestamp | The approximate creation time of this block (seconds from Unix Epoch)
| 4 bytes | Difficulty Target | The proof-of-work algorithm difficulty target for this block
| 4 bytes | Nonce | A counter used for the proof-of-work algorithm
| 4 bytes | Difficulty Target | The Proof-of-Work algorithm difficulty target for this block
| 4 bytes | Nonce | A counter used for the Proof-of-Work algorithm
|=======
The nonce, difficulty target, and timestamp are used in the mining process and will be discussed in more detail in <<bitcoin_network_ch08>>.
The nonce, difficulty target, and timestamp are used in the mining process and will be discussed in more detail in <<ch10>>.
[[block_hash]]
=== Block Identifiers: Block Header Hash and Block Height
@ -157,11 +157,11 @@ A _merkle tree_, also known as a((("binary hash tree"))) _binary hash tree_, is
.Blocks linked in a chain, by reference to the previous block header hash
image::images/mbc2_0901.png[]
Merkle trees are used in bitcoin to summarize all the transactions in a block, producing an overall digital fingerprint of the entire set of transactions, providing a very efficient process to verify whether a transaction is included in a block. A((("Merkle trees","constructing"))) Merkle tree is constructed by recursively hashing pairs of nodes until there is only one hash, called the _root_, or _merkle root_. The cryptographic hash algorithm used in bitcoin's merkle trees is SHA256 applied twice, also known as double-SHA256.
Merkle trees are used in bitcoin to summarize all the transactions in a block, producing an overall digital fingerprint of the entire set of transactions, providing a very efficient process to verify whether a transaction is included in a block. A((("Merkle trees","constructing"))) merkle tree is constructed by recursively hashing pairs of nodes until there is only one hash, called the _root_, or _merkle root_. The cryptographic hash algorithm used in bitcoin's merkle trees is SHA256 applied twice, also known as double-SHA256.
When N data elements are hashed and summarized in a merkle tree, you can check to see if any one data element is included in the tree with at most +2*log~2~(N)+ calculations, making this a very efficient data structure.
The merkle tree is constructed bottom-up. In the following example, we start with four transactions, A, B, C and D, which form the _leaves_ of the Merkle tree, as shown in <<simple_merkle>>. The transactions are not stored in the merkle tree; rather, their data is hashed and the resulting hash is stored in each leaf node as H~A~, H~B~, H~C~, and H~D~:
The merkle tree is constructed bottom-up. In the following example, we start with four transactions, A, B, C and D, which form the _leaves_ of the merkle tree, as shown in <<simple_merkle>>. The transactions are not stored in the merkle tree; rather, their data is hashed and the resulting hash is stored in each leaf node as H~A~, H~B~, H~C~, and H~D~:
----
H~A~ = SHA256(SHA256(Transaction A))
@ -173,7 +173,7 @@ Consecutive pairs of leaf nodes are then summarized in a parent node, by concate
H~AB~ = SHA256(SHA256(H~A~ + H~B~))
----
The process continues until there is only one node at the top, the node known as the Merkle root. That 32-byte hash is stored in the block header and summarizes all the data in all four transactions.
The process continues until there is only one node at the top, the node known as the merkle root. That 32-byte hash is stored in the block header and summarizes all the data in all four transactions. <<simple_merkle>> shows how the root is calculated by pair-wise hashes of the nodes
[[simple_merkle]]
.Calculating the nodes in a merkle tree
@ -193,7 +193,7 @@ To prove that a specific transaction is included in a block, a node only needs t
.A merkle tree summarizing many data elements
image::images/mbc2_0904.png["merkle_tree_large"]
In <<merkle_tree_path>>, a node can prove that a transaction K is included in the block by producing a merkle path that is only four 32-byte hashes long (128 bytes total). The path consists of the four hashes (noted in blue in <<merkle_tree_path>>) H~L~, H~IJ~, H~MNOP~ and H~ABCDEFGH~. With those four hashes provided as an authentication path, any node can prove that H~K~ (noted in green in the diagram) is included in the merkle root by computing four additional pair-wise hashes H~KL~, H~IJKL~, H~IJKLMNOP~, and the merkle tree root (outlined in a dotted line in the diagram).
In <<merkle_tree_path>>, a node can prove that a transaction K is included in the block by producing a merkle path that is only four 32-byte hashes long (128 bytes total). The path consists of the four hashes (noted in blue in <<merkle_tree_path>>) H~L~, H~IJ~, H~MNOP~, and H~ABCDEFGH~. With those four hashes provided as an authentication path, any node can prove that H~K~ (noted in green in the diagram) is included in the merkle root by computing four additional pair-wise hashes H~KL~, H~IJKL~, H~IJKLMNOP~, and the merkle tree root (outlined in a dotted line in the diagram).
[[merkle_tree_path]]
.A merkle path used to prove inclusion of a data element
@ -247,7 +247,7 @@ The efficiency of merkle trees becomes obvious as the scale increases. <<block_s
As you can see from the table, while the block size increases rapidly, from 4 KB with 16 transactions to a block size of 16 MB to fit 65,535 transactions, the merkle path required to prove the inclusion of a transaction increases much more slowly, from 128 bytes to only 512 bytes. With merkle trees, a node can download just the block headers (80 bytes per block) and still be able to identify a transaction's inclusion in a block by retrieving a small merkle path from a full node, without storing or transmitting the vast majority of the blockchain, which might be several gigabytes in size. Nodes that do not maintain a full blockchain, called simplified payment verification (SPV nodes), use merkle paths to verify transactions without downloading full blocks.(((range="endofrange", startref="ix_ch09-asciidoc2")))(((range="endofrange", startref="ix_ch09-asciidoc1")))
As you can see from the table, while the block size increases rapidly, from 4 KB with 16 transactions to a block size of 16 MB to fit 65,535 transactions, the merkle path required to prove the inclusion of a transaction increases much more slowly, from 128 bytes to only 512 bytes. With merkle trees, a node can download just the block headers (80 bytes per block) and still be able to identify a transaction's inclusion in a block by retrieving a small merkle path from a full node, without storing or transmitting the vast majority of the blockchain, which might be several gigabytes in size. Nodes that do not maintain a full blockchain, called simplified payment verification (SPV) nodes, use merkle paths to verify transactions without downloading full blocks.(((range="endofrange", startref="ix_ch09-asciidoc2")))(((range="endofrange", startref="ix_ch09-asciidoc1")))
=== Merkle Trees and Simplified Payment Verification (SPV)
@ -257,24 +257,24 @@ Consider, for example, an SPV node that is interested in incoming payments to an
=== Bitcoin's Test Blockchains
You might be surprised to learn that there is more than one bitcoin blockchain. The "main" bitcoin blockchain, the one created by Satoshi Nakamoto on January 3rd 2009, the one with the genesis block we studied in this chapter, is called _mainnet_. ((("mainnet")))((("blockchain", "mainnet"))) There are other bitcoin blockchains that are used for testing purposes: at this time _testnet_, _segnet_ and _regtest_. Let's look at each in turn.((("testnet")))((("blockchain","testnet")))((("segnet")))
You might be surprised to learn that there is more than one bitcoin blockchain. The "main" bitcoin blockchain, the one created by Satoshi Nakamoto on January 3rd, 2009, the one with the genesis block we studied in this chapter, is called _mainnet_. ((("mainnet")))((("blockchain", "mainnet"))) There are other bitcoin blockchains that are used for testing purposes: at this time _testnet_, _segnet_ and _regtest_. Let's look at each in turn.((("testnet")))((("blockchain","testnet")))((("segnet")))
((("testnet")))((("blockchain","testnet"))) Testnet is the name of the test blockchain, network and currency that is used for testing purposes. The testnet is a fully featured live P2P network, with wallets, test bitcoins (testnet coins), mining and all the other features of mainnet. ((("testnet","coins")))((("testnet","mining"))) There are really only two differences: testnet coins are meant to be worthless and mining difficulty should be low enough that anyone can mine testnet coins relatively easily (keeping them worthless).
((("testnet")))((("blockchain","testnet"))) Testnet is the name of the test blockchain, network, and currency that is used for testing purposes. The testnet is a fully featured live P2P network, with wallets, test bitcoins (testnet coins), mining, and all the other features of mainnet. ((("testnet","coins")))((("testnet","mining"))) There are really only two differences: testnet coins are meant to be worthless and mining difficulty should be low enough that anyone can mine testnet coins relatively easily (keeping them worthless).
Any software development that is intended for production use on bitcoin's mainnet should first be tested on testnet with test coins. This protects both the developers from monetary losses due to bugs and the network from unintended behavior due to bugs.
Keeping the coins worthless and the mining easy, however, is not easy. Despite pleas from developers, some people use advanced mining equipment (GPUs and ASICs) to mine on testnet. This increases the difficulty, makes it impossible to mine with a CPU and eventually makes it difficult enough to get test coins that people start valuing them, so they're not worthless. As a result, every now and then, the testnet has to be scrapped and restarted from a new genesis block, resetting the difficulty.
Keeping the coins worthless and the mining easy, however, is not easy. Despite pleas from developers, some people use advanced mining equipment (GPUs and ASICs) to mine on testnet. This increases the difficulty, makes it impossible to mine with a CPU, and eventually makes it difficult enough to get test coins that people start valuing them, so they're not worthless. As a result, every now and then, the testnet has to be scrapped and restarted from a new genesis block, resetting the difficulty.
The current testnet is called _testnet3_, the third iteration of testnet, restarted in Feb 2011 to reset the difficulty from the previous testnet.
The current testnet is called _testnet3_, the third iteration of testnet, restarted in February 2011 to reset the difficulty from the previous testnet.
((("testnet","running a node")))Keep in mind that testnet3 is a large blockchain, in excess of 20GB in early 2017. It will take a day or so to sync fully and use up resources on your computer. Not as much as mainnet, but not exactly "lightweight" either. One good way to run a testnet node is as a virtual machine image (eg. virtualbox, docker, cloud server etc) dedicated for that purpose.
((("testnet","running a node")))Keep in mind that testnet3 is a large blockchain, in excess of 20GB in early 2017. It will take a day or so to sync fully and use up resources on your computer. Not as much as mainnet, but not exactly "lightweight" either. One good way to run a testnet node is as a virtual machine image (e.g., virtualbox, docker, cloud server, etc.) dedicated for that purpose.
===== Using testnet
((("testnet","running a node")))((("testing")))Bitcoin Core, like almost all other bitcoin software has full support for operation on testnet instead of mainnet, but also allows you to mine testnet coins and operate a full testnet node.
((("testnet","running a node")))((("testing")))Bitcoin Core, like almost all other bitcoin software, has full support for operation on testnet instead of mainnet, but also allows you to mine testnet coins and operate a full testnet node.
To start Bitcoin Core on testnet instead of mainnet you use the +testnet+ switch:
You can also run on testnet3 with other full node implementations, such as +btcd+ (written in Go) and +bcoin+ (written in Javascript), to experiment and learn in other programming languages and frameworks.
You can also run on testnet3 with other full-node implementations, such as +btcd+ (written in Go) and +bcoin+ (written in JavaScript), to experiment and learn in other programming languages and frameworks.
In early 2017, testnet3 supports all the features of mainnet, in addition to Segregated Witness (see <<segwit>>), which has yet to activate on mainnet. Therefore, testnet3 can also be used to test Segregated Witness features.
=== Segnet - The Segregated Witness Testnet
=== Segnet—The Segregated Witness Testnet
((("segnet")))
((("blockchain","segnet")))((("segiwt","testing")))In 2016, a special-purpose testnet was launched to aid in development and testing of Segregated Witness (aka. segwit see <<segwit>>). This test blockchain is called +segnet+ and can be joined by running a special version (branch) of Bitcoin Core.
((("blockchain","segnet")))((("segiwt","testing")))In 2016, a special-purpose testnet was launched to aid in development and testing of Segregated Witness (aka segwit; see <<segwit>>). This test blockchain is called +segnet+ and can be joined by running a special version (branch) of Bitcoin Core.
Since segwit was added to testnet3, it is no longer necessary to use segnet for testing of segwit features.
((("testing")))In the future it is likely we will see other testnet blockchains that are specifically designed to test a single feature or major architectural change, like segnet.
=== Regtest - The local blockchain
=== Regtest—The Local Blockchain
((("regtest")))((("blockchain","regtest")))((("testing")))Regtest, which stands for "Regression Testing" is a Bitcoin Core feature that allows you to create a local blockchain for testing purposes. Unlike testnet3, which is a public and share test blockchain, the regtest blockchains are intended to be run as closed systems for local testing. You launch a regtest blockchain from scratch, creating a local genesis block. You may add other nodes to the network, or run it with a single node only to test the Bitcoin Core software.
((("regtest")))((("blockchain","regtest")))((("testing")))Regtest, which stands for "Regression Testing," is a Bitcoin Core feature that allows you to create a local blockchain for testing purposes. Unlike testnet3, which is a public and shared test blockchain, the regtest blockchains are intended to be run as closed systems for local testing. You launch a regtest blockchain from scratch, creating a local genesis block. You may add other nodes to the network, or run it with a single node only to test the Bitcoin Core software.
To start Bitcoin Core in regtest mode, you use the +regtest+ flag:
@ -353,7 +353,7 @@ To start Bitcoin Core in regtest mode, you use the +regtest+ flag:
$ bitcoind -regtest
----
Just like with testnet, Bitcoin Core will initialize a new blockchain under the +regtest+ subdirectory of your bitcoind default directory:
Just like with testnet, Bitcoin Core will initialize a new blockchain under the _regtest_ subdirectory of your bitcoind default directory:
----
bitcoind: Using data directory /home/username/.bitcoin/regtest
Bitcoin's various blockchains (+regtest+, +segnet+, +testnet3+, +mainnet+) offer a range of testing environments for bitcoin development. ((("testing")))Use the test blockchains whether you are developing for Bitcoin Core, or another full-node consensus client; an application such as a wallet, exchange, e-commerce site; or even developing novel smart contracts and complex scripts.
Bitcoin's various blockchains (+regtest+, +segnet+, +testnet3+, +mainnet+) offer a range of testing environments for bitcoin development. ((("testing")))Use the test blockchains whether you are developing for Bitcoin Core, or another full-node consensus client; an application such as a wallet, exchange, ecommerce site; or even developing novel smart contracts and complex scripts.
You can use the test blockchains to establish a development pipeline. Test your code locally on a +regtest+ as you develop it. Once ready to try it on a public network, switch to +testnet+ to expose your code to a more dynamic environment with more diversity of code and applications. Finally, once you are confident your code works as expected, switch to +mainnet+ to deploy it in production. As you make changes, improvements, bug fixes etc., start the pipeline again deploying each change first on +regtest+, then on +testnet+ and finally into production.
You can use the test blockchains to establish a development pipeline. Test your code locally on a +regtest+ as you develop it. Once ready to try it on a public network, switch to +testnet+ to expose your code to a more dynamic environment with more diversity of code and applications. Finally, once you are confident your code works as expected, switch to +mainnet+ to deploy it in production. As you make changes, improvements, bug fixes, etc., start the pipeline again, deploying each change first on +regtest+, then on +testnet+, and finally into production.
((("consensus", id="ix_ch10-asciidoc0", range="startofrange")))((("mining", id="ix_ch10-asciidoc1", range="startofrange")))((("miners")))The word "mining" is somewhat misleading. By evoking the extraction of precious metals, it focuses our attention on the reward for mining, the new bitcoin created in each block. Although mining is incentivized by this reward, the primary purpose of mining is not the reward or the generation of new coins. If you view mining only as the process by which coins are created, you are mistaking the means (incentives) as the goal of the process. Mining is the mechanism that underpins the decentralized clearinghouse, by which transactions are validated and cleared. Mining is the invention that makes bitcoin special, a decentralized security mechanism that is the basis for peer-to-peer digital cash.
Mining *secures the bitcoin system* and enables the emergence of network-wide *consensus without a central authority*. The reward of newly minted coins and transaction fees is an incentive scheme that aligns the actions of miners with the security of the network, while simultaneously implementing the monetary supply.
Mining _secures the bitcoin system_ and enables the emergence of network-wide _consensus without a central authority_. The reward of newly minted coins and transaction fees is an incentive scheme that aligns the actions of miners with the security of the network, while simultaneously implementing the monetary supply.
[TIP]
====
The purpose of mining is not the creation of new bitcoin. That's the incentive system. Mining is the mechanism by which bitcoin's *security* is *decentralized*.
The purpose of mining is not the creation of new bitcoin. That's the incentive system. Mining is the mechanism by which bitcoin's _security_ is _decentralized_.
====
Miners validate new transactions and record them on the global ledger. A new block, containing transactions that occurred since the last block, is "mined" every 10 minutes on average, thereby adding those transactions to the blockchain. Transactions that become part of a block and added to the blockchain are considered "confirmed," which allows the new owners of bitcoin to spend the bitcoin they received in those transactions.
Miners receive two types of rewards in return for the security provided by mining: new coins created with each new block, and transaction fees from all the transactions included in the block. To earn this reward, the miners compete to solve a difficult mathematical problem based on a cryptographic hash algorithm. The solution to the problem, called the proof-of-work, is included in the new block and acts as proof that the miner expended significant computing effort. The competition to solve the proof-of-work algorithm to earn reward and the right to record transactions on the blockchain is the basis for bitcoin's security model.
Miners receive two types of rewards in return for the security provided by mining: new coins created with each new block, and transaction fees from all the transactions included in the block. To earn this reward, the miners compete to solve a difficult mathematical problem based on a cryptographic hash algorithm. The solution to the problem, called the Proof-of-Work, is included in the new block and acts as proof that the miner expended significant computing effort. The competition to solve the Proof-of-Work algorithm to earn reward and the right to record transactions on the blockchain is the basis for bitcoin's security model.
((("new coin generation")))The process is called mining because the reward (new coin generation) is designed to simulate diminishing returns, just like mining for precious metals. Bitcoin's money supply is created through mining, similar to how a central bank issues new money by printing bank notes. ((("bitcoin","rate of issuance")))The maximum amount of newly created bitcoin a miner can add to a block decreases approximately every four years (or precisely every 210,000 blocks). It started at 50 bitcoin per block in January of 2009 and halved to 25 bitcoin per block in November of 2012. It halved again to 12.5 bitcoin in July 2016. Based on this formula, bitcoin mining rewards decrease exponentially until approximately the year 2140, when all bitcoin (20.99999998 million) will have been issued. After 2140, no new bitcoin will be issued.
@ -73,7 +73,7 @@ Many economists argue that a deflationary economy is a disaster that should be a
Bitcoin experts argue that deflation is not bad per se. Rather, deflation is associated with a collapse in demand because that is the only example of deflation we have to study. In a fiat currency with the possibility of unlimited printing, it is very difficult to enter a deflationary spiral unless there is a complete collapse in demand and an unwillingness to print money. Deflation in bitcoin is not caused by a collapse in demand, but by a predictably constrained supply.
The positive aspect of deflation, of course, is that it is the opposite of inflation. Inflation causes a slow but inevitable debasement of currency, resulting in a form of hidden taxation that punishes savers in order to bail-out debtors (including the biggest debtors, governments themselves). Currencies under government control suffer from the moral hazard of easy debt issuance that can later be erased through debasement at the expense of savers.
The positive aspect of deflation, of course, is that it is the opposite of inflation. Inflation causes a slow but inevitable debasement of currency, resulting in a form of hidden taxation that punishes savers in order to bailout debtors (including the biggest debtors, governments themselves). Currencies under government control suffer from the moral hazard of easy debt issuance that can later be erased through debasement at the expense of savers.
It remains to be seen whether the deflationary aspect of the currency is a problem when it is not driven by rapid economic retraction, or an advantage because the protection from inflation and debasement far outweighs the risks of deflation.(((range="endofrange", startref="ix_ch10-asciidoc3")))(((range="endofrange", startref="ix_ch10-asciidoc2")))
****
@ -89,9 +89,9 @@ But how can everyone in the network agree on a single universal "truth" about wh
Bitcoin's decentralized consensus emerges from the interplay of four processes that occur independently on nodes across the network:
* Independent verification of each transaction, by every full node, based on a comprehensive list of criteria
* Independent aggregation of those transactions into new blocks by mining nodes, coupled with demonstrated computation through a proof-of-work algorithm
* Independent aggregation of those transactions into new blocks by mining nodes, coupled with demonstrated computation through a Proof-of-Work algorithm
* Independent verification of the new blocks by every node and assembly into a chain
* Independent selection, by every node, of the chain with the most cumulative computation demonstrated through proof-of-work
* Independent selection, by every node, of the chain with the most cumulative computation demonstrated through Proof-of-Work
In the next few sections we will examine these processes and how they interact to create the emergent property of network-wide consensus that allows any bitcoin node to assemble its own copy of the authoritative, trusted, public, global ledger.
@ -109,7 +109,7 @@ However, before forwarding transactions to its neighbors, every bitcoin node tha
* The transaction size in bytes is less than +MAX_BLOCK_SIZE+.
* Each output value, as well as the total, must be within the allowed range of values (less than 21m coins, more than the _dust_ threshold).
* None of the inputs have hash=0, N=–1 (coinbase transactions should not be relayed).
* +nLockTime+ is equal to +INT_MAX+, or nLocktime and nSequence values are satisfied according to MedianTimePast.
* +nLocktime+ is equal to +INT_MAX+, or +nLocktime+ and +nSequence+ values are satisfied according to +MedianTimePast+.
* The transaction size in bytes is greater than or equal to 100.
* The number of signature operations (SIGOPS) contained in the transaction is less than the signature operation limit.
* The unlocking script (+scriptSig+) can only push numbers on the stack, and the locking script (+scriptPubkey+) must match +isStandard+ forms (this rejects "nonstandard" transactions).
@ -120,12 +120,12 @@ However, before forwarding transactions to its neighbors, every bitcoin node tha
* For each input, the referenced output must exist and cannot already be spent.
* Using the referenced output transactions to get input values, check that each input value, as well as the sum, are in the allowed range of values (less than 21m coins, more than 0).
* Reject if the sum of input values is less than sum of output values.
* Reject if transaction fee would be too low (minRelayTxFee) to get into an empty block.
* Reject if transaction fee would be too low (+minRelayTxFee+) to get into an empty block.
* The unlocking scripts for each input must validate against the corresponding output locking scripts.
These conditions can be seen in detail in the functions +AcceptToMemoryPool+, +CheckTransaction+, and +CheckInputs+ in Bitcoin Core. Note that the conditions change over time, to address new types of denial-of-service attacks or sometimes to relax the rules so as to include more types of transactions.
By independently verifying each transaction as it is received and before propagating it, every node builds a pool of valid (but unconfirmed) transactions known as the _transaction pool_, _memory pool_ or _mempool_.
By independently verifying each transaction as it is received and before propagating it, every node builds a pool of valid (but unconfirmed) transactions known as the _transaction pool_, _memory pool_, or _mempool_.
=== Mining Nodes
@ -143,9 +143,12 @@ Jing's mining node maintains a local copy of the blockchain. By the time Alice b
During the previous 10 minutes, while Jing's node was searching for a solution to block 277,315, it was also collecting transactions in preparation for the next block. By now it has collected a few hundred transactions in the memory pool. Upon receiving block 277,315 and validating it, Jing's node will also compare it against all the transactions in the memory pool and remove any that were included in block 277,315. Whatever transactions remain in the memory pool are unconfirmed and are waiting to be recorded in a new block.
Jing's node immediately constructs a new empty block, a candidate for block 277,316. This block is called a _candidate block_ because it is not yet a valid block, as it does not contain a valid proof of work. The block becomes valid only if the miner succeeds in finding a solution to the proof-of-work algorithm.
Jing's node immediately constructs a new empty block, a candidate for block 277,316. This block is called a _candidate block_ because it is not yet a valid block, as it does not contain a valid Proof-of-Work. The block becomes valid only if the miner succeeds in finding a solution to the Proof-of-Work algorithm.
When Jing's node aggregates all the transactions from the memory pool, the new candidate block has 418 transactions with total transaction fees of 0.09094928 bitcoin. You can see this block in the blockchain using the Bitcoin Core client command-line interface, as shown in <<block277316>>.(((range="endofrange", startref="ix_ch10-asciidoc9")))(((range="endofrange", startref="ix_ch10-asciidoc8")))(((range="endofrange", startref="ix_ch10-asciidoc7")))(((range="endofrange", startref="ix_ch10-asciidoc6")))(((range="endofrange", startref="ix_ch10-asciidoc5")))(((range="endofrange", startref="ix_ch10-asciidoc4")))
When block #277,316 was mined, the reward was 25 bitcoin per block. Since then, one "halving" period has elapsed. The block reward changed to 12.5 bitcoin in July 2016. It will be halved again in 210,000 blocks, in the year 2020.
When block 277,316 was mined, the reward was 25 bitcoin per block. Since then, one "halving" period has elapsed. The block reward changed to 12.5 bitcoin in July 2016. It will be halved again in 210,000 blocks, in the year 2020.
====
Jing's node creates the coinbase transaction as a payment to his own wallet: "Pay Jing's address 25.09094928 bitcoin." The total amount of reward that Jing collects for mining a block is the sum of the coinbase reward (25 new bitcoin) and the transaction fees (0.09094928) from all the transactions included in the block as shown in <<generation_tx_example>>:
Jing's node creates the coinbase transaction as a payment to his own wallet: "Pay Jing's address 25.09094928 bitcoin." The total amount of reward that Jing collects for mining a block is the sum of the coinbase reward (25 new bitcoin) and the transaction fees (0.09094928) from all the transactions included in the block as shown in <<generation_tx_example>>.
@ -255,7 +255,7 @@ Next, Jing's node calculates the correct reward for the new block. The reward is
The calculation can be seen in function +GetBlockSubsidy+ in the Bitcoin Core client, as shown in <<getblocksubsidy_source>>.
[[getblocksubsidy_source]]
.Calculating the block reward — Function GetBlockSubsidy, Bitcoin Core Client, main.cpp
.Calculating the block reward—Function GetBlockSubsidy, Bitcoin Core Client, main.cpp
====
[source, cpp]
----
@ -289,7 +289,7 @@ Finally, the coinbase reward (+nSubsidy+) is added to the transaction fees (+nFe
If Jing's mining node writes the coinbase transaction, what stops Jing from "rewarding" himself 100 or 1000 bitcoin? The answer is that an incorrect reward would result in the block being deemed invalid by everyone else, wasting Jing's electricity used for Proof-of-Work. Jing only gets to spend the reward if the block is accepted by everyone
====
==== Structure of the coinbase transaction
==== Structure of the Coinbase Transaction
((("coinbase transaction","structure of")))With these calculations, Jing's node then constructs the coinbase transaction to pay himself 25.09094928 bitcoin.
@ -314,13 +314,13 @@ As you can see in <<generation_tx_example>>, the coinbase transaction has a spec
|Size| Field | Description
| 32 bytes | Transaction Hash | All bits are zero: Not a transaction hash reference
| 4 bytes | Output Index | All bits are ones: 0xFFFFFFFF
| 1-9 bytes (VarInt) | Coinbase Data Size | Length of the coinbase data, from 2 to 100 bytes
| 1–9 bytes (VarInt) | Coinbase Data Size | Length of the coinbase data, from 2 to 100 bytes
| Variable | Coinbase Data | Arbitrary data used for extra nonce and mining tags
in v2 blocks, must begin with block height
| 4 bytes | Sequence Number | Set to 0xFFFFFFFF
|=======
In a coinbase transaction, the first two fields are set to values that do not represent a UTXO reference. Instead of a "Transaction Hash," the first field is filled with 32 bytes all set to zero. The "Output Index" is filled with 4 bytes all set to 0xFF (255 decimal). The "Unlocking Script" (scriptSig) is replaced by coinbase data, a data field used by the miners, as we will see next.
In a coinbase transaction, the first two fields are set to values that do not represent a UTXO reference. Instead of a "Transaction Hash," the first field is filled with 32 bytes all set to zero. The "Output Index" is filled with 4 bytes all set to 0xFF (255 decimal). The "Unlocking Script" (+scriptSig+) is replaced by coinbase data, a data field used by the miners, as we will see next.
==== Coinbase Data
@ -334,7 +334,7 @@ In block 277,316 we see that the coinbase (see <<generation_tx_example>>), which
The first byte, +03+, instructs the script execution engine to push the next three bytes onto the script stack (see <<tx_script_ops_table_pushdata>>). The next three bytes, +0x443b04+, are the block height encoded in little-endian format (backward, least significant byte first). Reverse the order of the bytes and the result is +0x043b44+, which is 277,316 in decimal.
The next few hexadecimal digits (+0385840206+) are used to encode an extra _nonce_ (see <<extra_nonce>>), or random value, used to find a suitable proof of work solution.
The next few hexadecimal digits (+0385840206+) are used to encode an extra _nonce_ (see <<extra_nonce>>), or random value, used to find a suitable Proof-of-Work solution.
The final part of the coinbase data (+2f503253482f+) is the ASCII-encoded string +/P2SH/+, which indicates that the mining node that mined this block supports the((("pay-to-script-hash (P2SH)","coinbase data and"))) pay-to-script-hash (P2SH) improvement defined in BIP-16. The introduction of the P2SH capability required a "vote" by miners to endorse either BIP-16 or BIP-17. Those endorsing the BIP-16 implementation were to include +/P2SH/+ in their coinbase data. Those endorsing the BIP-17 implementation of P2SH were to include the string +p2sh/CHV+ in their coinbase data. The BIP-16 was elected as the winner, and many miners continued including the string +/P2SH/+ in their coinbase to indicate support for this feature.
@ -377,8 +377,8 @@ $ ./satoshi-words
| 32 bytes | Previous Block Hash | A reference to the hash of the previous (parent) block in the chain
| 32 bytes | Merkle Root | A hash of the root of the merkle tree of this block's transactions
| 4 bytes | Timestamp | The approximate creation time of this block (seconds from Unix Epoch)
| 4 bytes | Target | The proof-of-work algorithm target for this block
| 4 bytes | Nonce | A counter used for the proof-of-work algorithm
| 4 bytes | Target | The Proof-of-Work algorithm target for this block
| 4 bytes | Nonce | A counter used for the Proof-of-Work algorithm
|=======
At the time that block 277,316 was mined, the version number describing the block structure is version 2, which is encoded in little-endian format in 4 bytes as +0x02000000+.
((("timestamping blocks")))Jing's mining node will then add a 4-byte timestamp, encoded as a Unix "Epoch" timestamp, which is based on the number of seconds elapsed from January 1, 1970, midnight UTC/GMT. The time +1388185914+ is equal to Friday, 27 Dec 2013, 23:11:54 UTC/GMT.
((("proof-of-work target","constructing block headers and")))Jing's node then fills in the target, which defines the required proof-of-work to make this a valid block. The target is stored in the block as a "Target Bits" metric, which is a mantissa-exponent encoding of the target. The encoding has a 1-byte exponent, followed by a 3-byte mantissa (coefficient). In block 277,316, for example, the target bits value is +0x1903a30c+. The first part +0x19+ is a hexadecimal exponent, while the next part, +0x03a30c+, is the coefficient. The concept of a target is explained in <<target>> and the "target bits" representation is explained in <<target_bits>>.
((("proof-of-work target","constructing block headers and")))Jing's node then fills in the target, which defines the required Proof-of-Work to make this a valid block. The target is stored in the block as a "Target Bits" metric, which is a mantissa-exponent encoding of the target. The encoding has a 1-byte exponent, followed by a 3-byte mantissa (coefficient). In block 277,316, for example, the target bits value is +0x1903a30c+. The first part +0x19+ is a hexadecimal exponent, while the next part, +0x03a30c+, is the coefficient. The concept of a target is explained in <<target>> and the "target bits" representation is explained in <<target_bits>>.
((("nonce,","initializing in block header")))The final field is the nonce, which is initialized to zero.
@ -410,11 +410,11 @@ With all the other fields filled, the block header is now complete and the proce
=== Mining the Block
((("blocks","mining", id="ix_ch10-asciidoc13", range="startofrange")))((("mining","blocks", id="ix_ch10-asciidoc14", range="startofrange")))Now that a candidate block has been constructed by Jing's node, it is time for Jing's hardware mining rig to "mine" the block, to find a solution to the proof-of-work algorithm that makes the block valid. Throughout this book we have studied cryptographic hash functions as used in various aspects of the bitcoin system. The hash function SHA256 is the function used in bitcoin's mining process.
((("blocks","mining", id="ix_ch10-asciidoc13", range="startofrange")))((("mining","blocks", id="ix_ch10-asciidoc14", range="startofrange")))Now that a candidate block has been constructed by Jing's node, it is time for Jing's hardware mining rig to "mine" the block, to find a solution to the Proof-of-Work algorithm that makes the block valid. Throughout this book we have studied cryptographic hash functions as used in various aspects of the bitcoin system. The hash function SHA256 is the function used in bitcoin's mining process.
In the simplest terms, mining is the process of hashing the block header repeatedly, changing one parameter, until the resulting hash matches a specific target. The hash function's result cannot be determined in advance, nor can a pattern be created that will produce a specific hash value. This feature of hash functions means that the only way to produce a hash result matching a specific target is to try again and again, randomly modifying the input until the desired hash result appears by chance.
==== Proof-Of-Work Algorithm
==== Proof-of-Work Algorithm
((("mining","proof-of-work algorithm and", id="ix_ch10-asciidoc15", range="startofrange")))((("Proof-Of-Work algorithm", id="ix_ch10-asciidoc16", range="startofrange")))A hash algorithm takes an arbitrary-length data input and produces a fixed-length deterministic result, a digital fingerprint of the input. For any specific input, the resulting hash will always be the same and can be easily calculated and verified by anyone implementing the same hash algorithm. The key characteristic of a cryptographic hash algorithm is that it is computationally infeasible to find two different inputs that produce the same fingerprint (known as a _collision_). As a corollary, it is also virtually impossible to select an input in such a way as to produce a desired fingerprint, other than trying random inputs.
Running this will produce the hashes of several phrases, made different by adding a number at the end of the text. By incrementing the number, we can get different hashes, as shown in <<sha256_example_generator_output>>.
Running this will produce the hashes of several phrases, made different by adding a number at the end of the text. By incrementing the number, we can get different hashes, as ((("nonce")))shown in <<sha256_example_generator_output>>.
((("nonce")))
[[sha256_example_generator_output]]
.SHA256 output of a script for generating many hashes by iterating on a nonce
====
@ -488,31 +487,31 @@ Each phrase produces a completely different hash result. They seem completely ra
The number used as a variable in such a scenario is called a _nonce_. The nonce is used to vary the output of a cryptographic function, in this case to vary the SHA256 fingerprint of the phrase.
((("proof-of-work target","defined")))To make a challenge out of this algorithm, let's set target: find a phrase that produces a hexadecimal hash that starts with a zero. Fortunately, this isn't difficult! <<sha256_example_generator_output>> shows that the phrase "I am Satoshi Nakamoto13" produces the hash +0ebc56d59a34f5082aaef3d66b37a661696c2b618e62432727216ba9531041a5+, which fits our criteria. It took 13 attempts to find it. In terms of probabilities, if the output of the hash function is evenly distributed we would expect to find a result with a 0 as the hexadecimal prefix once every 16 hashes (one out of 16 hexadecimal digits 0 through F). In numerical terms, that means finding a hash value that is less than +0x1000000000000000000000000000000000000000000000000000000000000000+. We call this threshold the _target_ and the goal is to find a hash that is numerically less than the target. If we decrease the target, the task of finding a hash that is less than the target becomes more and more difficult.
((("proof-of-work target","defined")))To make a challenge out of this algorithm, let's set a target: find a phrase that produces a hexadecimal hash that starts with a zero. Fortunately, this isn't difficult! <<sha256_example_generator_output>> shows that the phrase "I am Satoshi Nakamoto13" produces the hash +0ebc56d59a34f5082aaef3d66b37a661696c2b618e62432727216ba9531041a5+, which fits our criteria. It took 13 attempts to find it. In terms of probabilities, if the output of the hash function is evenly distributed we would expect to find a result with a 0 as the hexadecimal prefix once every 16 hashes (one out of 16 hexadecimal digits 0 through F). In numerical terms, that means finding a hash value that is less than +0x1000000000000000000000000000000000000000000000000000000000000000+. We call this threshold the _target_ and the goal is to find a hash that is numerically less than the target. If we decrease the target, the task of finding a hash that is less than the target becomes more and more difficult.
To give a simple analogy, imagine a game where players throw a pair of dice repeatedly, trying to throw less than a specified target. In the first round, the target is 12. Unless you throw double-six, you win. In the next round the target is 11. Players must throw 10 or less to win, again an easy task. Let's say a few rounds later the target is down to 5. Now, more than half the dice throws will exceed the target and therefore be invalid. It takes exponentially more dice throws to win, the lower the target gets. Eventually, when the target is 2 (the minimum possible), only one throw out of every 36, or 2% of them, will produce a winning result.
From the perspective of an observer who knows that the target of the dice game is 2, if someone has succeeded in casting a winning throw it can be assumed that they attempted, on average, 36 throws. In other words, one can estimate the amount of work it takes to succeed from the difficulty imposed by the target. When the algorithm is a based on a deterministic function such as SHA256, the input itself constitutes _proof_ that a certain amount of _work_ was done to produce a result below the target. Hence, _Proof of Work_.
From the perspective of an observer who knows that the target of the dice game is 2, if someone has succeeded in casting a winning throw it can be assumed that they attempted, on average, 36 throws. In other words, one can estimate the amount of work it takes to succeed from the difficulty imposed by the target. When the algorithm is a based on a deterministic function such as SHA256, the input itself constitutes _proof_ that a certain amount of _work_ was done to produce a result below the target. Hence, _Proof-of-Work_.
[TIP]
====
Even though each attempt produces a random outcome, the probability of any possible outcome can be calculated in advance. Therefore, an outcome of specified difficulty constitutes proof of a specific amount of work.
====
In <<sha256_example_generator_output>>, the winning "nonce" is 13 and this result can be confirmed by anyone independently. Anyone can add the number 13 as a suffix to the phrase "I am Satoshi Nakamoto" and compute the hash, verifying that it is less than the target. The successful result is also proof of work, because it proves we did the work to find that nonce. While it only takes one hash computation to verify, it took us 13 hash computations to find a nonce that worked. If we had a lower target (higher difficulty) it would take many more hash computations to find a suitable nonce, but only one hash computation for anyone to verify. Furthermore, by knowing the target, anyone can estimate the difficulty using statistics and therefore know how much work was needed to find such a nonce.
In <<sha256_example_generator_output>>, the winning "nonce" is 13 and this result can be confirmed by anyone independently. Anyone can add the number 13 as a suffix to the phrase "I am Satoshi Nakamoto" and compute the hash, verifying that it is less than the target. The successful result is also Proof-of-Work, because it proves we did the work to find that nonce. While it only takes one hash computation to verify, it took us 13 hash computations to find a nonce that worked. If we had a lower target (higher difficulty) it would take many more hash computations to find a suitable nonce, but only one hash computation for anyone to verify. Furthermore, by knowing the target, anyone can estimate the difficulty using statistics and therefore know how much work was needed to find such a nonce.
[TIP]
====
The proof-of-work must produce a hash that is *less than* the target. A higher target means it is less difficult to find a hash that is below the target. A lower target means it is more difficult to find a hash below the target. The target and difficulty are inversely related.
The Proof-of-Work must produce a hash that is _less than_ the target. A higher target means it is less difficult to find a hash that is below the target. A lower target means it is more difficult to find a hash below the target. The target and difficulty are inversely related.
====
Bitcoin's proof-of-work is very similar to the challenge shown in <<sha256_example_generator_output>>. The miner constructs a candidate block filled with transactions. Next, the miner calculates the hash of this block's header and sees if it is smaller than the current _target_. If the hash is not less than the target, the miner will modify the nonce (usually just incrementing it by one) and try again. At the current difficulty in the bitcoin network, miners have to try quadrillions of times before finding a nonce that results in a low enough block header hash.
Bitcoin's Proof-of-Work is very similar to the challenge shown in <<sha256_example_generator_output>>. The miner constructs a candidate block filled with transactions. Next, the miner calculates the hash of this block's header and sees if it is smaller than the current _target_. If the hash is not less than the target, the miner will modify the nonce (usually just incrementing it by one) and try again. At the current difficulty in the bitcoin network, miners have to try quadrillions of times before finding a nonce that results in a low enough block header hash.
A very simplified proof-of-work algorithm is implemented in Python in <<pow_example1>>.((("proof of work")))
A very simplified Proof-of-Work algorithm is implemented in Python in <<pow_example1>>.((("proof of work")))
Running this code, you can set the desired difficulty (in bits, how many of the leading bits must be zero) and see how long it takes for your computer to find a solution. In <<pow_example_outputs>>, you can see how it works on an average laptop.
[[pow_example_outputs]]
.Running the proof of work example for various difficulties
.Running the Proof-of-Work example for various difficulties
====
[source, bash]
----
@ -593,7 +592,7 @@ At the time of writing, the network is attempting to find a block whose header h
[[target_bits]]
==== Target Representation
((("proof-of-work target")))((("mining","target bits")))((("mining","target")))In <<block277316>>, we saw that the block contains the target, in a notation called "target bits" or just "bits," which in block 277,316 has the value of +0x1903a30c+. This notation expresses the proof-of-work target as a coefficient/exponent format, with the first two hexadecimal digits for the exponent and the next six hex digits as the coefficient. In this block, therefore, the exponent is +0x19+ and the coefficient is +0x03a30c+.
((("proof-of-work target")))((("mining","target bits")))((("mining","target")))In <<block277316>>, we saw that the block contains the target, in a notation called "target bits" or just "bits," which in block 277,316 has the value of +0x1903a30c+. This notation expresses the Proof-of-Work target as a coefficient/exponent format, with the first two hexadecimal digits for the exponent and the next six hex digits as the coefficient. In this block, therefore, the exponent is +0x19+ and the coefficient is +0x03a30c+.
The formula to calculate the difficulty target from this representation is:
@ -630,11 +629,11 @@ This means that a valid block for height 277,316 is one that has a block header
[[target]]
==== Retargeting to Adjust Difficulty
((("target","retargeting", id="ix_ch10-asciidoc17", range="startofrange")))As we saw, the target determines the difficulty and therefore affects how long it takes to find a solution to the proof-of-work algorithm. This leads to the obvious questions: Why is the difficulty adjustable, who adjusts it, and how?
((("target","retargeting", id="ix_ch10-asciidoc17", range="startofrange")))As we saw, the target determines the difficulty and therefore affects how long it takes to find a solution to the Proof-of-Work algorithm. This leads to the obvious questions: Why is the difficulty adjustable, who adjusts it, and how?
((("retargeting")))((("target","block generation rate and")))Bitcoin's blocks are generated every 10 minutes, on average. This is bitcoin's heartbeat and underpins the frequency of currency issuance and the speed of transaction settlement. It has to remain constant not just over the short term, but over a period of many decades. Over this time, it is expected that computer power will continue to increase at a rapid pace. Furthermore, the number of participants in mining and the computers they use will also constantly change. To keep the block generation time at 10 minutes, the difficulty of mining must be adjusted to account for these changes. In fact, the proof-of-work target is a dynamic parameter that is periodically adjusted to meet a 10-minute block interval goal. In simple terms, the target is set so that the current mining power will result in a 10-minute block interval.
((("retargeting")))((("target","block generation rate and")))Bitcoin's blocks are generated every 10 minutes, on average. This is bitcoin's heartbeat and underpins the frequency of currency issuance and the speed of transaction settlement. It has to remain constant not just over the short term, but over a period of many decades. Over this time, it is expected that computer power will continue to increase at a rapid pace. Furthermore, the number of participants in mining and the computers they use will also constantly change. To keep the block generation time at 10 minutes, the difficulty of mining must be adjusted to account for these changes. In fact, the Proof-of-Work target is a dynamic parameter that is periodically adjusted to meet a 10-minute block interval goal. In simple terms, the target is set so that the current mining power will result in a 10-minute block interval.
How, then, is such an adjustment made in a completely decentralized network? Retargeting occurs automatically and on every node independently. Every 2,016 blocks, all nodes retarget the proof-of-work. The equation for retargeting measures the time it took to find the last 2,016 blocks and compares that to the expected time of 20,160 minutes (2,016 blocks times the desired 10-minute block interval). The ratio between the actual timespan and desired timespan is calculated and a proportionate adjustment (up or down) is made to the target. In simple terms: If the network is finding blocks faster than every 10 minutes, the difficulty increases (target decreases). If block discovery is slower than expected, the difficulty decreases (target increases).
How, then, is such an adjustment made in a completely decentralized network? Retargeting occurs automatically and on every node independently. Every 2,016 blocks, all nodes retarget the Proof-of-Work. The equation for retargeting measures the time it took to find the last 2,016 blocks and compares that to the expected time of 20,160 minutes (2,016 blocks times the desired 10-minute block interval). The ratio between the actual timespan and desired timespan is calculated and a proportionate adjustment (up or down) is made to the target. In simple terms: If the network is finding blocks faster than every 10 minutes, the difficulty increases (target decreases). If block discovery is slower than expected, the difficulty decreases (target increases).
The equation can be summarized as:
@ -645,7 +644,7 @@ New Target = Old Target * (Actual Time of Last 2016 Blocks / 20160 minutes)
<<retarget_code>> shows the code used in the Bitcoin Core client.
[[retarget_code]]
.Retargeting the proof-of-work — CalculateNextWorkRequired() in pow.cpp
.Retargeting the Proof-of-Work—CalculateNextWorkRequired() in pow.cpp
====
[source,cpp]
----
@ -675,11 +674,11 @@ New Target = Old Target * (Actual Time of Last 2016 Blocks / 20160 minutes)
[NOTE]
====
While the target calibration happens every 2,016 blocks, because of an off-by-one error in the original Bitcoin Core client it is based on the total time of the previous 2,015 blocks (not 2,016 as it should be), resulting in a retargeting bias towards higher difficulty by 0.05%.
While the target calibration happens every 2,016 blocks, because of an off-by-one error in the original Bitcoin Core client it is based on the total time of the previous 2,015 blocks (not 2,016 as it should be), resulting in a retargeting bias toward higher difficulty by 0.05%.
====
The parameters Interval (2,016 blocks) and TargetTimespan (two weeks as 1,209,600 seconds) are defined in _chainparams.cpp_.
The parameters +Interval+ (2,016 blocks) and +TargetTimespan+ (two weeks as 1,209,600 seconds) are defined in _chainparams.cpp_.
To avoid extreme volatility in the difficulty, the retargeting adjustment must be less than a factor of four (4) per cycle. If the required target adjustment is greater than a factor of four, it will be adjusted by a factor of 4 and not more. Any further adjustment will be accomplished in the next retargeting period because the imbalance will persist through the next 2,016 blocks. Therefore, large discrepancies between hashing power and difficulty might take several 2,016 block cycles to balance out.
@ -714,26 +713,26 @@ In the next section, we'll look at the process each node uses to validate a bloc
=== Validating a New Block
((("blocks","validating new")))((("consensus","validating new blocks")))((("mining","validating new blocks")))The third step in bitcoin's consensus mechanism is independent validation of each new block by every node on the network. As the newly solved block moves across the network, each node performs a series of tests to validate it before propagating it to its peers. This ensures that only valid blocks are propagated on the network. The independent validation also ensures that miners who act honestly get their blocks incorporated in the blockchain, thus earning the reward. Those miners who act dishonestly have their blocks rejected and not only lose the reward, but also waste the effort expended to find a proof-of-work solution, thus incurring the cost of electricity without compensation.
((("blocks","validating new")))((("consensus","validating new blocks")))((("mining","validating new blocks")))The third step in bitcoin's consensus mechanism is independent validation of each new block by every node on the network. As the newly solved block moves across the network, each node performs a series of tests to validate it before propagating it to its peers. This ensures that only valid blocks are propagated on the network. The independent validation also ensures that miners who act honestly get their blocks incorporated in the blockchain, thus earning the reward. Those miners who act dishonestly have their blocks rejected and not only lose the reward, but also waste the effort expended to find a Proof-of-Work solution, thus incurring the cost of electricity without compensation.
When a node receives a new block, it will validate the block by checking it against a long list of criteria that must all be met; otherwise, the block is rejected. These criteria can be seen in the Bitcoin Core client in the functions((("CheckBlock function (Bitcoin Core client)")))((("CheckBlockHeader function (Bitcoin Core client)"))) +CheckBlock+ and +CheckBlockHeader+ and include:
* The block data structure is syntactically valid
* The block header hash is less than the target (enforces the proof of work)
* The block header hash is less than the target (enforces the Proof-of-Work)
* The block timestamp is less than two hours in the future (allowing for time errors)
* The block size is within acceptable limits
* The first transaction (and only the first) is a coinbase transaction
* All transactions within the block are valid using the transaction checklist discussed in <<tx_verification>>
The independent validation of each new block by every node on the network ensures that the miners cannot cheat. In previous sections we saw how the miners get to write a transaction that awards them the new bitcoin created within the block and claim the transaction fees. Why don't miners write themselves a transaction for a thousand bitcoin instead of the correct reward? Because every node validates blocks according to the same rules. An invalid coinbase transaction would make the entire block invalid, which would result in the block being rejected and, therefore, that transaction would never become part of the ledger. The miners have to construct a perfect block, based on the shared rules that all nodes follow, and mine it with a correct solution to the proof of work. To do so, they expend a lot of electricity in mining, and if they cheat, all the electricity and effort is wasted. This is why independent validation is a key component of decentralized consensus.
The independent validation of each new block by every node on the network ensures that the miners cannot cheat. In previous sections we saw how the miners get to write a transaction that awards them the new bitcoin created within the block and claim the transaction fees. Why don't miners write themselves a transaction for a thousand bitcoin instead of the correct reward? Because every node validates blocks according to the same rules. An invalid coinbase transaction would make the entire block invalid, which would result in the block being rejected and, therefore, that transaction would never become part of the ledger. The miners have to construct a perfect block, based on the shared rules that all nodes follow, and mine it with a correct solution to the Proof-of-Work. To do so, they expend a lot of electricity in mining, and if they cheat, all the electricity and effort is wasted. This is why independent validation is a key component of decentralized consensus.
=== Assembling and Selecting Chains of Blocks
((("blockchains","assembling", id="ix_ch10-asciidoc18", range="startofrange")))((("blockchains","selecting", id="ix_ch10-asciidoc19", range="startofrange")))((("blocks","assembling chains of", id="ix_ch10-asciidoc20", range="startofrange")))((("blocks","selecting chains of", id="ix_ch10-asciidoc21", range="startofrange")))The final step in bitcoin's decentralized consensus mechanism is the assembly of blocks into chains and the selection of the chain with the most proof of work. Once a node has validated a new block, it will then attempt to assemble a chain by connecting the block to the existing blockchain.
((("blockchains","assembling", id="ix_ch10-asciidoc18", range="startofrange")))((("blockchains","selecting", id="ix_ch10-asciidoc19", range="startofrange")))((("blocks","assembling chains of", id="ix_ch10-asciidoc20", range="startofrange")))((("blocks","selecting chains of", id="ix_ch10-asciidoc21", range="startofrange")))The final step in bitcoin's decentralized consensus mechanism is the assembly of blocks into chains and the selection of the chain with the most Proof-of-Work. Once a node has validated a new block, it will then attempt to assemble a chain by connecting the block to the existing blockchain.
((("nodes","sets of blocks maintained by")))Nodes maintain three sets of blocks: those connected to the main blockchain, those that form branches off the main blockchain((("secondary chains"))) (secondary chains), and finally, blocks that do not have a known parent in the known chains((("orphan blocks"))) (orphans). Invalid blocks are rejected as soon as any one of the validation criteria fails and are therefore not included in any chain.
((("blockchains","main")))((("main blockchain")))The "main chain" at any time is whichever chain of blocks has the most cumulative proof-of-work associated with it. Under most circumstances this is also the chain with the most blocks in it, unless there are two equal-length chains and one has more proof-of-work. ((("sibling chains (to main chain)")))The main chain will also have branches with blocks that are "siblings" to the blocks on the main chain. These blocks are valid but not part of the main chain. They are kept for future reference, in case one of those chains is extended to exceed the main chain in work. In the next section (<<forks>>), we will see how secondary chains occur as a result of an almost simultaneous mining of blocks at the same height.
((("blockchains","main")))((("main blockchain")))The "main chain" at any time is whichever chain of blocks has the most cumulative Proof-of-Work associated with it. Under most circumstances this is also the chain with the most blocks in it, unless there are two equal-length chains and one has more Proof-of-Work. ((("sibling chains (to main chain)")))The main chain will also have branches with blocks that are "siblings" to the blocks on the main chain. These blocks are valid but not part of the main chain. They are kept for future reference, in case one of those chains is extended to exceed the main chain in work. In the next section (<<forks>>), we will see how secondary chains occur as a result of an almost simultaneous mining of blocks at the same height.
When a new block is received, a node will try to slot it into the existing blockchain. The node will look at the block's "previous block hash" field, which is the reference to the block's parent. Then, the node will attempt to find that parent in the existing blockchain. Most of the time, the parent will be the "tip" of the main chain, meaning this new block extends the main chain. For example, the new block 277,316 has a reference to the hash of its parent block 277,315. Most nodes that receive 277,316 will already have block 277,315 as the tip of their main chain and will therefore link the new block and extend that chain.
@ -748,23 +747,23 @@ In the next section we will look at how discrepancies between competing chains (
[[forks]]
==== Blockchain Forks
((("blockchains","forks", id="ix_ch10-asciidoc22", range="startofrange")))Because the blockchain is a decentralized data structure, different copies of it are not always consistent. Blocks might arrive at different nodes at different times, causing the nodes to have different perspectives of the blockchain. To resolve this, each node always selects and attempts to extend the chain of blocks that represents the most proof of work, also known as the longest chain or greatest cumulative work chain. By summing the work recorded in each block in a chain, a node can calculate the total amount of work that has been expended to create that chain. As long as all nodes select the greatest-cumulative-work chain, the global bitcoin network eventually converges to a consistent state. Forks occur as temporary inconsistencies between versions of the blockchain, which are resolved by eventual reconvergence as more blocks are added to one of the forks.
((("blockchains","forks", id="ix_ch10-asciidoc22", range="startofrange")))Because the blockchain is a decentralized data structure, different copies of it are not always consistent. Blocks might arrive at different nodes at different times, causing the nodes to have different perspectives of the blockchain. To resolve this, each node always selects and attempts to extend the chain of blocks that represents the most Proof-of-Work, also known as the longest chain or greatest cumulative work chain. By summing the work recorded in each block in a chain, a node can calculate the total amount of work that has been expended to create that chain. As long as all nodes select the greatest-cumulative-work chain, the global bitcoin network eventually converges to a consistent state. Forks occur as temporary inconsistencies between versions of the blockchain, which are resolved by eventual reconvergence as more blocks are added to one of the forks.
In the next few diagrams, we follow the progress of a "fork" event across the network. The diagram is a simplified representation of the bitcoin network. For illustration purposes, different blocks are shown as different shapes (star, triangle, upsidedown triangle, rhombus), spreading across the network. Each node in the network is represented as a circle.
In the next few diagrams, we follow the progress of a "fork" event across the network. The diagram is a simplified representation of the bitcoin network. For illustration purposes, different blocks are shown as different shapes (star, triangle, upside-down triangle, rhombus), spreading across the network. Each node in the network is represented as a circle.
Each node has its own perspective of the global blockchain. As each node receives blocks from its neighbors, it updates its own copy of the blockchain, selecting the greatest-cumulative-work chain. For illustration purposes, each node contains a shape that represents the block that it believes is currently the tip of the main chain. So, if you see a star shape in the node, that means that the star block is the tip of the main chain, as far as that node is concerned.
In the first diagram (<<fork1>>), the network has a unified perspective of the blockchain, with the star block as the tip of the main chain.
[[fork1]]
.Before the fork - all nodes have the same perspective
.Before the fork—all nodes have the same perspective
image::images/mbc2_1002.png["Before the fork - all nodes have the same perspective"]
A "fork" occurs whenever there are two candidate blocks competing to form the longest blockchain. This occurs under normal conditions whenever two miners solve the proof-of-work algorithm within a short period of time from each other. As both miners discover a solution for their respective candidate blocks, they immediately broadcast their own "winning" block to their immediate neighbors who begin propagating the block across the network. Each node that receives a valid block will incorporate it into its blockchain, extending the blockchain by one block. If that node later sees another candidate block extending the same parent, it connects the second candidate on a secondary chain. As a result, some nodes will "see" one candidate block first, while other nodes will see the other candidate block and two competing versions of the blockchain will emerge.
A "fork" occurs whenever there are two candidate blocks competing to form the longest blockchain. This occurs under normal conditions whenever two miners solve the Proof-of-Work algorithm within a short period of time from each other. As both miners discover a solution for their respective candidate blocks, they immediately broadcast their own "winning" block to their immediate neighbors who begin propagating the block across the network. Each node that receives a valid block will incorporate it into its blockchain, extending the blockchain by one block. If that node later sees another candidate block extending the same parent, it connects the second candidate on a secondary chain. As a result, some nodes will "see" one candidate block first, while other nodes will see the other candidate block and two competing versions of the blockchain will emerge.
In <<fork2>>, we see two miners (Node A and Node B) who mine two different blocks almost simultaneously. Both of these blocks are children of the star block, and extend the chain by building on top of the star block. To help us track it, one is visualized as a triangle block originating from Node A, and the other is shown as an upside-down black triangle block originating from Node B.
Let's assume, for example, that a miner Node A finds a proof-of-work solution for a block "triangle" that extends the blockchain, building on top of the parent block "star." Almost simultaneously, the miner Node B who was also extending the chain from block "star" finds a solution for block "upside-down black triangle," his candidate block. Now, there are two possible blocks, one we call "triangle," originating in Node A, and one we call "upside-down black triangle," originating in Node B. Both blocks are valid, both blocks contain a valid solution to the proof of work, and both blocks extend the same parent (block "star"). Both blocks likely contain most of the same transactions, with only perhaps a few differences in the order of transactions.
Let's assume, for example, that a miner Node A finds a Proof-of-Work solution for a block "triangle" that extends the blockchain, building on top of the parent block "star." Almost simultaneously, the miner Node B who was also extending the chain from block "star" finds a solution for block "upside-down black triangle," his candidate block. Now, there are two possible blocks, one we call "triangle," originating in Node A, and one we call "upside-down black triangle," originating in Node B. Both blocks are valid, both blocks contain a valid solution to the Proof-of-Work, and both blocks extend the same parent (block "star"). Both blocks likely contain most of the same transactions, with only perhaps a few differences in the order of transactions.
[[fork2]]
.Visualization of a blockchain fork event: two blocks found simultaneously
@ -772,11 +771,11 @@ image::images/mbc2_1003.png["Visualization of a blockchain fork event: two block
As the two blocks propagate, some nodes receive block "triangle" first and some receive block "upside-down black triangle" first. As shown in <<fork3>>, the network splits into two different perspectives of the blockchain, one side topped with a triangle block, the other with the upside-down-triangle block.
In the diagram, a randomly chosen "Node X" received the triangle block first and extended the star chain with it. Node X selected the chain with "triangle" block as the main chain. Later, Node X also received the "upside-down black triangle" block. Since it was received second, it is assumed to have "lost" the race. Yet, the "upside-down black triangle" block is not discarded. It is linked to the "star" block parent and forms a secondary chain. While Node X assumes it has correctly selected the winning chain, it keeps the "losing" chain so that it has the information needed to reconverge if the "losing" chain ends up "winning".
In the diagram, a randomly chosen "Node X" received the triangle block first and extended the star chain with it. Node X selected the chain with "triangle" block as the main chain. Later, Node X also received the "upside-down black triangle" block. Since it was received second, it is assumed to have "lost" the race. Yet, the "upside-down black triangle" block is not discarded. It is linked to the "star" block parent and forms a secondary chain. While Node X assumes it has correctly selected the winning chain, it keeps the "losing" chain so that it has the information needed to reconverge if the "losing" chain ends up "winning."
On the other side of the network, Node Y constructs a blockchain based on its own perspective of the sequence of events. It received "upside-down black triangle" first and elected that chain as the "winner". When it later received "triangle" block, it connected it to the "star" block parent as a secondary chain.
On the other side of the network, Node Y constructs a blockchain based on its own perspective of the sequence of events. It received "upside-down black triangle" first and elected that chain as the "winner." When it later received "triangle" block, it connected it to the "star" block parent as a secondary chain.
Neither side is "correct," or "incorrect". Both are valid perspectives of the blockchain. Only in hindsight will one prevail, based on how these two competing chains are extended by additional work.
Neither side is "correct," or "incorrect." Both are valid perspectives of the blockchain. Only in hindsight will one prevail, based on how these two competing chains are extended by additional work.
[[fork3]]
.Visualization of a blockchain fork event: two blocks propagate, splitting the network
@ -792,7 +791,7 @@ Forks are almost always resolved within one block. While part of the network's h
.Visualization of a blockchain fork event: a new block extends one fork, reconverging the network
image::images/mbc2_1005.png["Visualization of a blockchain fork event: a new block extends one fork"]
All nodes that had chosen "triangle" as the winner in the previous round will simply extend the chain one more block. The nodes that chose "upside-down black triangle" as the winner, however, will now see two chains: star-triangle-rhombus and star-upside-down-black-triangle. The chain star-triangle-rhombus is now longer (more cumulative work) than the other chain. As a result, those nodes will set the chain star-triangle-rhombus as main chain and change the star-upside-down-black-triangle chain to being a secondary chain, as shown in <<fork4>>. This is a chain reconvergence, because those nodes are forced to revise their view of the blockchain to incorporate the new evidence of a longer chain. Any miners working on extending the chain star-upside-down-black-triangle will now stop that work because their candidate block is an "orphan," as its parent "upside-down-black-triangle" is no longer on the longest chain. The transactions within "upside-down-black-triangle" are re-inserted in the mempool for inclusion in the next block, because the block they were in is no longer in the main chain. The entire network re-converges on a single blockchain star-triangle-rhombus, with "rhombus" as the last block in the chain. All miners immediately start working on candidate blocks that reference "rhombus" as their parent to extend the star-triangle-rhombus chain.
All nodes that had chosen "triangle" as the winner in the previous round will simply extend the chain one more block. The nodes that chose "upside-down black triangle" as the winner, however, will now see two chains: star-triangle-rhombus and star-upside-down-black-triangle. The chain star-triangle-rhombus is now longer (more cumulative work) than the other chain. As a result, those nodes will set the chain star-triangle-rhombus as main chain and change the star-upside-down-black-triangle chain to being a secondary chain, as shown in <<fork4>>. This is a chain reconvergence, because those nodes are forced to revise their view of the blockchain to incorporate the new evidence of a longer chain. Any miners working on extending the chain star-upside-down-black-triangle will now stop that work because their candidate block is an "orphan," as its parent "upside-down-black-triangle" is no longer on the longest chain. The transactions within "upside-down-black-triangle" are re-inserted in the mempool for inclusion in the next block, because the block they were in is no longer in the main chain. The entire network reconverges on a single blockchain star-triangle-rhombus, with "rhombus" as the last block in the chain. All miners immediately start working on candidate blocks that reference "rhombus" as their parent to extend the star-triangle-rhombus chain.
[[fork5]]
.Visualization of a blockchain fork event: the network reconverges on a new longest chain
@ -841,7 +840,7 @@ In the last two years, the ASIC mining chips have become increasingly denser, ap
((("hashing race","mining pools", id="ix_ch10-asciidoc26", range="startofrange")))((("mining pools", id="ix_ch10-asciidoc27", range="startofrange")))In this highly competitive environment,((("solo miners"))) individual miners working alone (also known as solo miners) don't stand a chance. The likelihood of them finding a block to offset their electricity and hardware costs is so low that it represents a gamble, like playing the lottery. Even the fastest consumer ASIC mining system cannot keep up with commercial systems that stack tens of thousands of these chips in giant warehouses near hydro-electric power stations. Miners now collaborate to form mining pools, pooling their hashing power and sharing the reward among thousands of participants. By participating in a pool, miners get a smaller share of the overall reward, but typically get rewarded every day, reducing uncertainty.
Let's look at a specific example. Assume a miner has purchased mining hardware with a combined hashing rate of 14,000 gigahashes per second (GH/s), or 14 TH/s. In 2017 this equipment costs approximately $2,500 USD. The hardware consumes 1375 watts (1.3 kW) of electricity when running, 32 kW-hours a day, at a cost of $1 to $2 per day on very low electricity rates. At current bitcoin difficulty, the miner will be able to solo mine a block approximately once every 4 years. If the miner does find a single block in that timeframe, the payout of 12.5 bitcoin, at approximately $1,000 per bitcoin, will result in a single payout of $12,500, which will not even cover the entire cost of the hardware and the electricity consumed over the time period, leaving a net loss of approximately $1,000. However, the chance of finding a block in a four-year period depends on the miner's luck. He might find two blocks in four years and make a very large profit. Or he might not find a block for 5 years and suffer a bigger financial loss. Even worse, the difficulty of the bitcoin proof-of-work algorithm is likely to go up significantly over that period, at the current rate of growth of hashing power, meaning the miner has, at most, one year to break even before the hardware is effectively obsolete and must be replaced by more powerful mining hardware. If this miner participates in a mining pool, instead of waiting for a once-in-four-years $12,500 windfall, he will be able to earn approximately $50 to $60 per week. The regular payouts from a mining pool will help him amortize the cost of hardware and electricity over time without taking an enormous risk. The hardware will still be obsolete in one or two years and the risk is still high, but the revenue is at least regular and reliable over that period. Financially this only makes sense at very low electricity cost (less than 1 cent per kW) and only at very large scale.
Let's look at a specific example. Assume a miner has purchased mining hardware with a combined hashing rate of 14,000 gigahashes per second (GH/s), or 14 TH/s. In 2017 this equipment costs approximately $2,500 USD. The hardware consumes 1375 watts (1.3 kW) of electricity when running, 32 kW-hours a day, at a cost of $1 to $2 per day on very low electricity rates. At current bitcoin difficulty, the miner will be able to solo mine a block approximately once every 4 years. If the miner does find a single block in that timeframe, the payout of 12.5 bitcoin, at approximately $1,000 per bitcoin, will result in a single payout of $12,500, which will not even cover the entire cost of the hardware and the electricity consumed over the time period, leaving a net loss of approximately $1,000. However, the chance of finding a block in a 4-year period depends on the miner's luck. He might find two blocks in 4 years and make a very large profit. Or he might not find a block for 5 years and suffer a bigger financial loss. Even worse, the difficulty of the bitcoin Proof-of-Work algorithm is likely to go up significantly over that period, at the current rate of growth of hashing power, meaning the miner has, at most, one year to break even before the hardware is effectively obsolete and must be replaced by more powerful mining hardware. If this miner participates in a mining pool, instead of waiting for a once-in-four-years $12,500 windfall, he will be able to earn approximately $50 to $60 per week. The regular payouts from a mining pool will help him amortize the cost of hardware and electricity over time without taking an enormous risk. The hardware will still be obsolete in one or two years and the risk is still high, but the revenue is at least regular and reliable over that period. Financially this only makes sense at very low electricity cost (less than 1 cent per kW) and only at very large scale.
Mining pools coordinate many hundreds or thousands of miners, over specialized pool-mining protocols. The individual miners configure their mining equipment to connect to a pool server, after creating an account with the pool. Their mining hardware remains connected to the pool server while mining, synchronizing their efforts with the other miners. Thus, the pool miners share the effort to mine a block and then share in the rewards.
@ -849,7 +848,7 @@ Successful blocks pay the reward to a pool bitcoin address, rather than individu
Miners participating in a pool split the work of searching for a solution to a candidate block, earning "shares" for their mining contribution. The mining pool sets a higher target (lower difficulty) for earning a share, typically more than 1,000 times easier than the bitcoin network's target. When someone in the pool successfully mines a block, the reward is earned by the pool and then shared with all miners in proportion to the number of shares they contributed to the effort.
Pools are open to any miner, big or small, professional or amateur. A pool will therefore have some participants with a single small mining machine, and others with a garage full of high-end mining hardware. Some will be mining with a few tens of a kilowatt of electricity, others will be running a data center consuming a megawatt of power. How does a mining pool measure the individual contributions, so as to fairly distribute the rewards, without the possibility of cheating? The answer is to use bitcoin's proof-of-work algorithm to measure each pool miner's contribution, but set at a lower difficulty so that even the smallest pool miners win a share frequently enough to make it worthwhile to contribute to the pool. By setting a lower difficulty for earning shares, the pool measures the amount of work done by each miner. Each time a pool miner finds a block header hash that is less than the pool target, she proves she has done the hashing work to find that result. More importantly, the work to find shares contributes, in a statistically measurable way, to the overall effort to find a hash lower than the bitcoin network's target. Thousands of miners trying to find low-value hashes will eventually find one low enough to satisfy the bitcoin network target.
Pools are open to any miner, big or small, professional or amateur. A pool will therefore have some participants with a single small mining machine, and others with a garage full of high-end mining hardware. Some will be mining with a few tens of a kilowatt of electricity, others will be running a data center consuming a megawatt of power. How does a mining pool measure the individual contributions, so as to fairly distribute the rewards, without the possibility of cheating? The answer is to use bitcoin's Proof-of-Work algorithm to measure each pool miner's contribution, but set at a lower difficulty so that even the smallest pool miners win a share frequently enough to make it worthwhile to contribute to the pool. By setting a lower difficulty for earning shares, the pool measures the amount of work done by each miner. Each time a pool miner finds a block header hash that is less than the pool target, she proves she has done the hashing work to find that result. More importantly, the work to find shares contributes, in a statistically measurable way, to the overall effort to find a hash lower than the bitcoin network's target. Thousands of miners trying to find low-value hashes will eventually find one low enough to satisfy the bitcoin network target.
Let's return to the analogy of a dice game. If the dice players are throwing dice with a goal of throwing less than four (the overall network difficulty), a pool would set an easier target, counting how many times the pool players managed to throw less than eight. When pool players throw less than eight (the pool share target), they earn shares, but they don't win the game because they don't achieve the game target (less than four). The pool players will achieve the easier pool target much more often, earning them shares very regularly, even when they don't achieve the harder target of winning the game. Every now and then, one of the pool players will throw a combined dice throw of less than four and the pool wins. Then, the earnings can be distributed to the pool players based on the shares they earned. Even though the target of eight-or-less wasn't winning, it was a fair way to measure dice throws for the players, and it occasionally produces a less-than-four throw.
@ -863,13 +862,13 @@ The pool server runs specialized software and a pool-mining protocol that coordi
Pool miners connect to the pool server using a mining protocol such as((("Stratum (STM) mining protocol"))) Stratum (STM) or((("GetBlockTemplate (GBT) mining protocol"))) GetBlockTemplate (GBT). An older standard called((("GetWork (GWK) mining protocol"))) GetWork (GWK) has been mostly obsolete since late 2012, because it does not easily support mining at hash rates above 4 GH/s. Both the STM and GBT protocols create((("block templates"))) block _templates_ that contain a template of a candidate block header. The pool server constructs a candidate block by aggregating transactions, adding a coinbase transaction (with extra nonce space), calculating the merkle root, and linking to the previous block hash. The header of the candidate block is then sent to each of the pool miners as a template. Each pool miner then mines using the block template, at a higher (easier) target than the bitcoin network target, and sends any successful results back to the pool server to earn shares.
===== Peer-to-Peer Mining Pool (P2Pool)
===== Peer-to-peer mining pool (P2Pool)
((("mining pools","P2Pool")))((("P2Pool")))Managed pools create the possibility of cheating by the pool operator, who might direct the pool effort to double-spend transactions or invalidate blocks (see <<consensus_attacks>>). Furthermore, centralized pool servers represent a single-point-of-failure. If the pool server is down or is slowed by a denial-of-service attack, the pool miners cannot mine. In 2011, to resolve these issues of centralization, a new pool mining method was proposed and implemented: P2Pool is a peer-to-peer mining pool, without a central operator.
P2Pool works by decentralizing the functions of the pool server, implementing a parallel blockchain-like system called a((("share chains"))) _share chain_. A share chain is a blockchain running at a lower difficulty than the bitcoin blockchain. The share chain allows pool miners to collaborate in a decentralized pool, by mining shares on the share chain at a rate of one share block every 30 seconds. Each of the blocks on the share chain records a proportionate share reward for the pool miners who contribute work, carrying the shares forward from the previous share block. When one of the share blocks also achieves the bitcoin network target, it is propagated and included on the bitcoin blockchain, rewarding all the pool miners who contributed to all the shares that preceded the winning share block. Essentially, instead of a pool server keeping track of pool miner shares and rewards, the share chain allows all pool miners to keep track of all shares using a decentralized consensus mechanism like bitcoin's blockchain consensus mechanism.
P2Pool mining is more complex than pool mining because it requires that the pool miners run a dedicated computer with enough disk space, memory, and Internet bandwidth to support a full bitcoin node and the P2Pool node software. P2Pool miners connect their mining hardware to their local P2Pool node, which simulates the functions of a pool server by sending block templates to the mining hardware. On P2Pool, individual pool miners construct their own candidate blocks, aggregating transactions much like solo miners, but then mine collaboratively on the share chain. P2Pool is a hybrid approach that has the advantage of much more granular payouts than solo mining, but without giving too much control to a pool operator like managed pools.
P2Pool mining is more complex than pool mining because it requires that the pool miners run a dedicated computer with enough disk space, memory, and internet bandwidth to support a full bitcoin node and the P2Pool node software. P2Pool miners connect their mining hardware to their local P2Pool node, which simulates the functions of a pool server by sending block templates to the mining hardware. On P2Pool, individual pool miners construct their own candidate blocks, aggregating transactions much like solo miners, but then mine collaboratively on the share chain. P2Pool is a hybrid approach that has the advantage of much more granular payouts than solo mining, but without giving too much control to a pool operator like managed pools.
Even though P2Pool reduces the concentration of power by mining pool operators, it is conceivably vulnerable to 51% attacks against the share chain itself. A much broader adoption of P2Pool does not solve the 51% attack problem for bitcoin itself. Rather, P2Pool makes bitcoin more robust overall, as part of a diversified mining ecosystem.
@ -887,7 +886,7 @@ Let's examine a practical example of a 51% attack. In the first chapter, we look
In our example, malicious attacker Mallory goes to Carol's gallery and purchases a beautiful triptych painting depicting Satoshi Nakamoto as Prometheus. Carol sells "The Great Fire" paintings for $250,000 in bitcoin, to Mallory. Instead of waiting for six or more confirmations on the transaction, Carol wraps and hands the paintings to Mallory after only one confirmation. Mallory works with an accomplice, Paul, who operates a large mining pool, and the accomplice launches a 51% attack as soon as Mallory's transaction is included in a block. Paul directs the mining pool to re-mine the same block height as the block containing Mallory's transaction, replacing Mallory's payment to Carol with a transaction that double-spends the same input as Mallory's payment. The double-spend transaction consumes the same UTXO and pays it back to Mallory's wallet, instead of paying it to Carol, essentially allowing Mallory to keep the bitcoin. Paul then directs the mining pool to mine an additional block, so as to make the chain containing the double-spend transaction longer than the original chain (causing a fork below the block containing Mallory's transaction). When the blockchain fork resolves in favor of the new (longer) chain, the double-spent transaction replaces the original payment to Carol. Carol is now missing the three paintings and also has no bitcoin payment. Throughout all this activity, Paul's mining pool participants might remain blissfully unaware of the double-spend attempt, because they mine with automated miners and cannot monitor every transaction or block.
To protect against this kind of attack, a merchant selling large-value items must wait at least six confirmations before giving the product to the buyer. Alternatively, the merchant should use an escrow((("multi-signature account"))) multi-signature account, again waiting for several confirmations after the escrow account is funded. The more confirmations elapse, the harder it becomes to invalidate a transaction with a 51% attack. For high-value items, payment by bitcoin will still be convenient and efficient even if the buyer has to wait 24 hours for delivery, which would correspond to approximately 144 confirmations.
To protect against this kind of attack, a merchant selling large-value items must wait at least six confirmations before giving the product to the buyer. Alternatively, the merchant should use an escrow((("multi-signature account"))) multisignature account, again waiting for several confirmations after the escrow account is funded. The more confirmations elapse, the harder it becomes to invalidate a transaction with a 51% attack. For high-value items, payment by bitcoin will still be convenient and efficient even if the buyer has to wait 24 hours for delivery, which would correspond to approximately 144 confirmations.
((("consensus attacks","denial of service attack")))((("denial of service attack")))In addition to a double-spend attack, the other scenario for a consensus attack is to deny service to specific bitcoin participants (specific bitcoin addresses). An attacker with a majority of the mining power can simply ignore specific transactions. If they are included in a block mined by another miner, the attacker can deliberately fork and re-mine that block, again excluding the specific transactions. This type of attack can result in a sustained denial of service against a specific address or set of addresses for as long as the attacker controls the majority of the mining power.
@ -895,7 +894,7 @@ Despite its name, the 51% attack scenario doesn't actually require 51% of the ha
The massive increase of total hashing power has arguably made bitcoin impervious to attacks by a single miner. There is no possible way for a solo miner to control more than a small percentage of the total mining power. However, the centralization of control caused by mining pools has introduced the risk of for-profit attacks by a mining pool operator. The pool operator in a managed pool controls the construction of candidate blocks and also controls which transactions are included. This gives the pool operator the power to exclude transactions or introduce double-spend transactions. If such abuse of power is done in a limited and subtle way, a pool operator could conceivably profit from a consensus attack without being noticed.
Not all attackers will be motivated by profit, however. One potential attack scenario is where an attacker intends to disrupt the bitcoin network without the possibility of profiting from such disruption. A malicious attack aimed at crippling bitcoin would require enormous investment and covert planning, but could conceivably be launched by a well-funded, most likely state-sponsored, attacker. Alternatively, a well-funded attacker could attack bitcoin's consensus by simultaneously amassing mining hardware, compromising pool operators and attacking other pools with denial-of-service. All of these scenarios are theoretically possible, but increasingly impractical as the bitcoin network's overall hashing power continues to grow exponentially.
Not all attackers will be motivated by profit, however. One potential attack scenario is where an attacker intends to disrupt the bitcoin network without the possibility of profiting from such disruption. A malicious attack aimed at crippling bitcoin would require enormous investment and covert planning, but could conceivably be launched by a well-funded, most likely state-sponsored, attacker. Alternatively, a well-funded attacker could attack bitcoin's consensus by simultaneously amassing mining hardware, compromising pool operators, and attacking other pools with denial-of-service. All of these scenarios are theoretically possible, but increasingly impractical as the bitcoin network's overall hashing power continues to grow exponentially.
Undoubtedly, a serious consensus attack would erode confidence in bitcoin in the short term, possibly causing a significant price decline. However, the bitcoin network and software are constantly evolving, so consensus attacks would be met with immediate countermeasures by the bitcoin community, making bitcoin hardier, stealthier, and more robust than ever.(((range="endofrange", startref="ix_ch10-asciidoc29")))(((range="endofrange", startref="ix_ch10-asciidoc28")))(((range="endofrange", startref="ix_ch10-asciidoc1")))(((range="endofrange", startref="ix_ch10-asciidoc0")))
@ -913,9 +912,9 @@ In <<forks>> we looked at how the bitcoin network may briefly diverge, with two
There is another scenario in which the network may diverge into following two chains: a change in the consensus rules. This type of fork is called a _hard fork_, because after the fork the network does not reconverge onto a single chain. Instead, the two chains evolve independently. Hard forks occur when part of the network is operating under a different set of consensus rules than the rest of the network. This may occur because of a bug or because of a deliberate change in the implementation of the consensus rules.
Hard forks can be used to change the rules of consensus, but they require coordination between all participants in the system. Any nodes that do not upgrade to the new consensus rules are unable to participate in the consensus mechanism and are forced onto a separate chain at the moment of the hard fork. Thus, a change introduced by hard fork can be thought of as not "forwards compatible", in that non-upgraded systems can no longer process the new consensus rules.
Hard forks can be used to change the rules of consensus, but they require coordination between all participants in the system. Any nodes that do not upgrade to the new consensus rules are unable to participate in the consensus mechanism and are forced onto a separate chain at the moment of the hard fork. Thus, a change introduced by hard fork can be thought of as not "forward compatible," in that non-upgraded systems can no longer process the new consensus rules.
Let's examine the mechanics of a hard fork with a specific example:
Let's examine the mechanics of a hard fork with a specific example.
[[blockchainwithforks]]
.A blockchain with forks
@ -931,17 +930,17 @@ The two chains continue to diverge from this point. Miners on the "b" chain will
==== Hard Forks: Software, Network, Mining, Chain
For software developers, the term "fork" has another meaning, adding confusion to the term "hard fork". In open source software, a fork occurs when a group of developers choose to follow a different software roadmap and start a competing implementation of an open source project. We've already discussed two circumstances that will lead to a hard fork: a bug in the consensus rules and a deliberate modification of the consensus rules. In the case of a deliberate change to the consensus rules, a software fork precedes the hard fork. However for this type of hard fork to occur, a new software implementation of the consensus rules must be developed, adopted, and launched.
For software developers, the term "fork" has another meaning, adding confusion to the term "hard fork." In open source software, a fork occurs when a group of developers choose to follow a different software roadmap and start a competing implementation of an open source project. We've already discussed two circumstances that will lead to a hard fork: a bug in the consensus rules and a deliberate modification of the consensus rules. In the case of a deliberate change to the consensus rules, a software fork precedes the hard fork. However, for this type of hard fork to occur, a new software implementation of the consensus rules must be developed, adopted, and launched.
Examples of software forks that have attempted to change consensus rules include Bitcoin XT, Bitcoin Classic, and most recently Bitcoin Unlimited. However none of these software forks have resulted in a hard fork. While a software fork is a necessary precondition, it is not in itself sufficient for a hard fork to occur. For a hard fork to occur, the competing implementation must be adopted and the new rules activated, by miners, wallets, and intermediary nodes. Conversely, there are numerous alternative implementations of Bitcoin Core, and even software forks, that do not change the consensus rules and barring a bug, these can co-exist on the network and inter-operate without causing a hard fork.
Examples of software forks that have attempted to change consensus rules include Bitcoin XT, Bitcoin Classic, and most recently Bitcoin Unlimited. However, none of these software forks have resulted in a hard fork. While a software fork is a necessary precondition, it is not in itself sufficient for a hard fork to occur. For a hard fork to occur, the competing implementation must be adopted and the new rules activated, by miners, wallets, and intermediary nodes. Conversely, there are numerous alternative implementations of Bitcoin Core, and even software forks, that do not change the consensus rules and barring a bug, these can coexist on the network and inter-operate without causing a hard fork.
Consensus rules may differ in obvious and explicit ways, in the validation of transactions or blocks. The rules may also differ in more subtle ways, in the implementation of the consensus rules as they apply to bitcoin scripts or cryptographic primitives such as digital signatures. Finally, the consensus rules may differ in unanticipated ways because of implicit consensus constraints imposed by system limitations or implementation details. An example of the latter was seen in the unanticipated hard fork during the upgrade of Bitcoin Core 0.7 to 0.8, which was caused by a limitation in the Berkley DB implementation used to store blocks.
Conceptually, we can think of a hard fork as developing in four stages: a software fork, a network fork, a mining fork and a chain fork.
Conceptually, we can think of a hard fork as developing in four stages: a software fork, a network fork, a mining fork, and a chain fork.
The process begins when an alternative implementation of the client, with modified consensus rules, is created by developers.
When this forked implementation is deployed in the network, a certain percentage of miners, wallet users and intermediate nodes may adopt and run this implementation. A resulting fork will depend upon whether the new consensus rules apply to blocks, transactions, or some other aspect of the system. If the new consensus rules pertain to transactions, then a wallet creating a transaction under the new rules may precipitate a network fork, followed by a hard fork when the transaction is mined into a block. If the new rules pertain to blocks, then the hard fork process will begin when a block is mined under the new rules.
When this forked implementation is deployed in the network, a certain percentage of miners, wallet users, and intermediate nodes may adopt and run this implementation. A resulting fork will depend upon whether the new consensus rules apply to blocks, transactions, or some other aspect of the system. If the new consensus rules pertain to transactions, then a wallet creating a transaction under the new rules may precipitate a network fork, followed by a hard fork when the transaction is mined into a block. If the new rules pertain to blocks, then the hard fork process will begin when a block is mined under the new rules.
First, the network will fork. Nodes based on the original implementation of the consensus rules will reject any transactions and blocks that are created under the new rules. Furthermore, the nodes following the original consensus rules will temporarily ban and disconnect from any nodes that are sending them these invalid transactions and blocks. As a result, the network will partition into two: old nodes will only remain connected to old nodes and new nodes will only be connected to new nodes. A single transaction or block based on the new rules will ripple through the network and result in the partition into two networks.
@ -951,9 +950,9 @@ Once a miner using the new rules mines a block, the mining power and chain will
As miners diverge into mining two different chains, the hashing power is split between the chains. The mining power can be split in any proportion between the two chains. The new rules may only be followed by a minority, or by the vast majority of the mining power.
Let's assume for example an 80%-20% split, with the majority of the mining power using the new consensus rules. Let's also assume that the fork occurs immediately after a retargetting period.
Let's assume, for example, an 80%-20% split, with the majority of the mining power using the new consensus rules. Let's also assume that the fork occurs immediately after a retargetting period.
The two chains would each inherit the difficulty from the retargetting period. The new consensus rules would have 80% of the previously available mining power committed to them. From the perspective of this chain, the mining power has suddenly declined by 20% vis-a-vis the previous period. Blocks will be found on average every 12 minutes, representing the 20% decline in mining power available to extend this chain. This rate of block issuance will continue (barring any changes in hashing power) until 2016 blocks are mined, which will take approximately 24,192 minutes (at 12 minutes per block), or 16.8 days. After 16.8 days, a retarget will occur and the difficulty will adjust (reduced by 20%) to produce 10minute blocks again, based on the reduced amount of hashing power in this chain.
The two chains would each inherit the difficulty from the retargetting period. The new consensus rules would have 80% of the previously available mining power committed to them. From the perspective of this chain, the mining power has suddenly declined by 20% vis-a-vis the previous period. Blocks will be found on average every 12 minutes, representing the 20% decline in mining power available to extend this chain. This rate of block issuance will continue (barring any changes in hashing power) until 2016 blocks are mined, which will take approximately 24,192 minutes (at 12 minutes per block), or 16.8 days. After 16.8 days, a retarget will occur and the difficulty will adjust (reduced by 20%) to produce 10-minute blocks again, based on the reduced amount of hashing power in this chain.
The minority chain, mining under the old rules with only 20% of the hashing power will face a much more difficult task. On this chain, blocks will now be mined every 50 minutes on average. The difficulty will not be adjusted for 2016 blocks, which will take 100,800 minutes, or approximately 10 weeks to mine. Assuming a fixed capacity per block, this will also result in a reduction of transaction capacity by a factor of 5, as there are fewer blocks per hour available to record transactions.
@ -969,70 +968,70 @@ Already we have seen the emergence of new methodologies to address the risks of
==== Soft Forks
Not all consensus rule changes cause a hard fork. Only consensus changes that are forwards-incompatible cause a fork. If the change is implemented in such a way that an unmodified client still sees the transaction or block as valid under the previous rules, the change can happen without a fork.
Not all consensus rule changes cause a hard fork. Only consensus changes that are forward-incompatible cause a fork. If the change is implemented in such a way that an unmodified client still sees the transaction or block as valid under the previous rules, the change can happen without a fork.
The term _soft fork_ was introduced to distinguish this upgrade method from a "hard fork". In practice, a soft fork is not a fork at all. A soft fork is a forwards compatible change to the consensus rules that allows un-upgraded clients to continue to operate in consensus with the new rules.
The term _soft fork_ was introduced to distinguish this upgrade method from a "hard fork." In practice, a soft fork is not a fork at all. A soft fork is a forward compatible change to the consensus rules that allows un-upgraded clients to continue to operate in consensus with the new rules.
One aspect of soft forks that is not immediately obvious is that soft fork upgrades can only be used to constrain the consensus rules, not to expand them. In order to be forwards compatible, transactions and blocks created under the new rules must be valid under the old rules too, but not vice-versa. The new rules can only limit what is valid, otherwise they will trigger a hard fork when rejected under the old rules.
One aspect of soft forks that is not immediately obvious is that soft fork upgrades can only be used to constrain the consensus rules, not to expand them. In order to be forward compatible, transactions and blocks created under the new rules must be valid under the old rules too, but not viceversa. The new rules can only limit what is valid, otherwise they will trigger a hard fork when rejected under the old rules.
Soft forks can be implemented in a number of ways - the term does not define a single method, rather a set of methods which all have one thing in common: they don't require all nodes to upgrade or force non-upgraded nodes out of consensus.
Soft forks can be implemented in a number of ways—the term does not define a single method, rather a set of methods that all have one thing in common: they don't require all nodes to upgrade or force nonupgraded nodes out of consensus.
===== Soft Forks re-defining NOP opcodes
===== Soft forks redefining NOP opcodes
A number of soft forks have been implemented in bitcoin, based on the re-interpretation of NOP opcodes. Bitcoin Script had ten opcodes reserved for future use, NOP1 through NOP10 respectively. Under the consensus rules, the presence of these opcodes in a script is interpreted as a null-potent operator, meaning they have no effect. Execution continues after the NOP opcode as if it wasn't there.
A number of soft forks have been implemented in bitcoin, based on the re-interpretation of NOP opcodes. Bitcoin Script had ten opcodes reserved for future use, NOP1 through NOP10, respectively. Under the consensus rules, the presence of these opcodes in a script is interpreted as a null-potent operator, meaning they have no effect. Execution continues after the NOP opcode as if it wasn't there.
A soft fork therefore can modify the semantics of a NOP code to give it new meaning. For example, BIP-65 (CHECKLOCKTIMEVERIFY) reinterpreted the NOP2 opcode. Clients implementing BIP-65 interpret NOP2 as OP_CHECKLOCKTIMEVERIFY and impose an absolute locktime consensus rule on UTXO that contain this opcode in their locking scripts. This change is a soft fork because a transaction that is valid under BIP-65 is also valid on any client that is not implementing (ignorant of) BIP-65. To the old clients, the script contains an NOP code which is ignored.
A soft fork therefore can modify the semantics of a NOP code to give it new meaning. For example, BIP-65 (+CHECKLOCKTIMEVERIFY+) reinterpreted the NOP2 opcode. Clients implementing BIP-65 interpret NOP2 as +OP_CHECKLOCKTIMEVERIFY+ and impose an absolute locktime consensus rule on UTXO that contain this opcode in their locking scripts. This change is a soft fork because a transaction that is valid under BIP-65 is also valid on any client that is not implementing (ignorant of) BIP-65. To the old clients, the script contains an NOP code, which is ignored.
===== Other ways to soft fork upgrade
The re-interpretation of NOP opcodes was both planned for and an obvious mechanism for consensus upgrades. Recently, however, another soft fork mechanism was introduced that does not rely on NOP opcodes, for a very specific type of consensus change. This is examined in more detail in <<segwit>>. Segregated Witness is an architectural change to the structure of a transaction, which moves the unlocking script (witness) from inside the transaction to an external data structure (segregating it). Segwit was initially envisioned as a hard fork upgrade, as it modified a fundamental structure (transactions). In November 2015, a developer working on Bitcoin Core proposed a mechanism by which segwit could be introduced as a soft fork. The mechanism used for this is a modification of the locking script of UTXO created under segwit rules, such that unmodified client see the locking script as redeemable with any unlocking script whatsoever. As a result, segwit can be introduced without requiring every node to upgrade or split from the chain: a soft fork.
The re-interpretation of NOP opcodes was both planned for and an obvious mechanism for consensus upgrades. Recently, however, another soft fork mechanism was introduced that does not rely on NOP opcodes, for a very specific type of consensus change. This is examined in more detail in <<segwit>>. Segregated Witness is an architectural change to the structure of a transaction, which moves the unlocking script (witness) from inside the transaction to an external data structure (segregating it). Segwit was initially envisioned as a hard fork upgrade, as it modified a fundamental structure (transactions). In November 2015, a developer working on Bitcoin Core proposed a mechanism by which segwit could be introduced as a soft fork. The mechanism used for this is a modification of the locking script of UTXO created under segwit rules, such that unmodified clients see the locking script as redeemable with any unlocking script whatsoever. As a result, segwit can be introduced without requiring every node to upgrade or split from the chain: a soft fork.
It is likely that there are other, yet to be discovered, mechanisms by which upgrades can be made in a forward-compatible way as a soft fork.
==== Criticisms of soft forks
==== Criticisms of Soft Forks
Soft forks based on the NOP opcodes are relatively uncontroversial. The NOP opcodes were placed in Bitcoin Script with the explicit goal of allowing non-disruptive upgrades.
However, many developers are concerned that other methods of soft fork upgrades make unacceptable tradeoffs. Common criticisms of soft fork changes include:
However, many developers are concerned that other methods of soft fork upgrades make unacceptable trade-offs. Common criticisms of soft fork changes include:
Technical debt:: Because soft forks are more technically complex than a hard fork upgrade, they introduce _technical debt_, a term that refers to increasing the future cost of code maintenance because of design tradeoffs made in the past. Code complexity in turn increases the likelihood of bugs and security vulnerabilities.
Technical debt:: Because soft forks are more technically complex than a hard fork upgrade, they introduce _technical debt_, a term that refers to increasing the future cost of code maintenance because of design trade-offs made in the past. Code complexity in turn increases the likelihood of bugs and security vulnerabilities.
Validation relaxation:: Unmodified clients see transactions as valid, without evaluating the modified consensus rules. In effect, the unmodified clients are not validating using the full range of consensus rules, as they are blind to the new rules. This applies to NOP-based upgrades, as well as other soft fork upgrades.
Irreversible Upgrades:: Because soft forks create transactions with additional consensus constraints, they become irreversible upgrades in practice. If a soft fork upgrade were to be reversed after being activated, any transactions created under the new rules could result in a loss of funds under the old rules. For example, if a CLTV transaction is evaluated under the old rules, there is no timelock constraint and it can be spent whenever. Therefore, critics contend that a failed soft fork that had to be reversed because of a bug would almost certainly lead to loss of funds.
Irreversible upgrades:: Because soft forks create transactions with additional consensus constraints, they become irreversible upgrades in practice. If a soft fork upgrade were to be reversed after being activated, any transactions created under the new rules could result in a loss of funds under the old rules. For example, if a CLTV transaction is evaluated under the old rules, there is no timelock constraint and it can be spent whenever. Therefore, critics contend that a failed soft fork that had to be reversed because of a bug would almost certainly lead to loss of funds.
[[softforksignaling]]
=== Soft Fork Signaling with Block Version
Since soft forks allow unmodified clients to continue to operate within consensus, the mechanism for "activating" a soft fork is through a vote by miners. A majority of miners must agree that they are ready and willing to enforce the new consensus rules. To coordinate their actions, there is a voting mechanism that allows them to signal their support for a consensus rule change. This mechanism was introduced with the activation of BIP-34 in March of 2013 and replaced by the activation of BIP-9 in July of 2016.
Since soft forks allow unmodified clients to continue to operate within consensus, the mechanism for "activating" a soft fork is through a vote by miners. A majority of miners must agree that they are ready and willing to enforce the new consensus rules. To coordinate their actions, there is a voting mechanism that allows them to signal their support for a consensus rule change. This mechanism was introduced with the activation of BIP-34 in March 2013 and replaced by the activation of BIP-9 in July 2016.
==== BIP-34 Signaling and Activation
The first implementation, in BIP-34, used the block version field to allow miners to signal readiness for a specific consensus rule change. Prior to BIP-34, block version was set to "1" by *convention* not enforced by *consensus*.
The first implementation, in BIP-34, used the block version field to allow miners to signal readiness for a specific consensus rule change. Prior to BIP-34, block version was set to "1" by _convention_ not enforced by _consensus_.
BIP-34 defined a consensus rule change that required the coinbase field (input) of the coinbase transaction to contain the block height. Prior to BIP-34, the coinbase could contain any arbitrary data the miners chose to include. After activation of BIP-34, valid blocks would have to contain a specific block-height at the beginning of the coinbase and be identified with a version number greater than or equal to "2".
To signal the change and activation of BIP-34, miners set the block version to "2", instead of "1". This did not immediately make version "1" blocks invalid. Once activated, version "1" blocks would become invalid and all version "2" blocks would be required to contain the block height in the coinbase to be valid.
BIP-34 defined a two-step activation mechanism, based on a rolling window of 1000 blocks. A miner would signal their individual readiness and vote yes on BIP-34 by constructing blocks with "2" as the version number. Strictly speaking, these blocks did not yet have to comply with the new consensus rule of including the block-height in the coinbase transaction because the consensus rule had not yet activated. The consensus rules activated in two steps:
BIP-34 defined a two-step activation mechanism, based on a rolling window of 1000 blocks. A miner would signal his or her individual readiness and vote yes on BIP-34 by constructing blocks with "2" as the version number. Strictly speaking, these blocks did not yet have to comply with the new consensus rule of including the block-height in the coinbase transaction because the consensus rule had not yet activated. The consensus rules activated in two steps:
* If 75% percent (750 of the most recent 1000 blocks) are marked with version "2", then version "2" blocks must contain block-height in the coinbase transaction or they are rejected as invalid. Version "1" blocks are still accepted by the network and do not need contain block-height. The old and new consensus rules co-exist during this period.
* If 75% percent (750 of the most recent 1000 blocks) are marked with version "2", then version "2" blocks must contain block-height in the coinbase transaction or they are rejected as invalid. Version "1" blocks are still accepted by the network and do not need contain block-height. The old and new consensus rules coexist during this period.
* When 95% percent (950 of the most recent 1000 blocks) are version "2", version "1" blocks are no longer considered valid. Version "2" blocks are valid only if they contain the block-height in the coinbase (as per the previous threshold). Thereafter, all blocks must comply with the new consensus rules, all valid blocks must contain block-height in the coinbase transaction.
* When 95% percent (950 of the most recent 1000 blocks) are version "2", version "1" blocks are no longer considered valid. Version "2" blocks are valid only if they contain the block-height in the coinbase (as per the previous threshold). Thereafter, all blocks must comply with the new consensus rules, and all valid blocks must contain block-height in the coinbase transaction.
After the successful signaling and activation under the BIP-34 rules, this mechanism was used twice more to activate soft forks:
* BIP-66 Strict DER Encoding of Signatures was activated by BIP-34 style signaling with a block version "3" and invalidating version "2" blocks
* BIP-65 CHECKLOCKTIMEVERIFY was activated by BIP-34 style signaling with a block version "4" and invalidating version "3" blocks.
* BIP-65 +CHECKLOCKTIMEVERIFY+ was activated by BIP-34 style signaling with a block version "4" and invalidating version "3" blocks
After the activation of BIP-65, the signaling and activation mechanism of BIP-34 was retired and replaced with the BIP-9 signaling mechanism described next.
BIP-34 (Block v2, Height in Coinbase):: https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki]
The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki[BIP-34 (Block v2, Height in Coinbase)].
==== BIP-9 Signaling and Activation
The mechanism used by BIP-34, BIP-66 and BIP-65 was successful in activating 3 soft forks. However, it was replaced because it had several limitations:
The mechanism used by BIP-34, BIP-66, and BIP-65 was successful in activating three soft forks. However, it was replaced because it had several limitations:
* By using the integer value of the block version, only one soft fork could be activated at a time, so it required coordination between soft fork proposals and agreement on their prioritization and sequencing.
@ -1042,15 +1041,15 @@ The mechanism used by BIP-34, BIP-66 and BIP-65 was successful in activating 3 s
BIP-9 was proposed to overcome these challenges and improve the rate and ease of implementing future changes.
BIP-9 interprets the block version as a bit field instead of an integer. Because the block version was originally used as an integer, versions 1 through 4, only 29 bits remain available to be used as a bit field. This leaves 29 bits which can be used to independently and simultaneously vote on 29 different proposals.
BIP-9 interprets the block version as a bit field instead of an integer. Because the block version was originally used as an integer, versions 1 through 4, only 29 bits remain available to be used as a bit field. This leaves 29 bits that can be used to independently and simultaneously vote on 29 different proposals.
BIP-9 also sets a maximum time for signaling and activation. Thus a change is not voted on perpetually. If it is not activated within the TIMEOUT period (defined in the proposal), the proposal is considered rejected.
BIP-9 also sets a maximum time for signaling and activation. Thus a change is not voted on perpetually. If it is not activated within the +TIMEOUT+ period (defined in the proposal), the proposal is considered rejected.
Furthermore, after the TIMEOUT has past and a feature has been activated or rejected, the signaling bit can be reused for another feature without confusion. Therefore, up to 29 changes can be voted on in parallel and after TIMEOUT the bits can be "recycled" to propose new changes.
Furthermore, after the +TIMEOUT+ has passed and a feature has been activated or rejected, the signaling bit can be reused for another feature without confusion. Therefore, up to 29 changes can be voted on in parallel and after +TIMEOUT+ the bits can be "recycled" to propose new changes.
[NOTE]
====
While signaling bits can be reused or recycled, as long as the voting period does not overlap, the authors of BIP-9 recommend that bits are reused only when necessary, unexpected behavior could occur due to bugs in older software. In short, we should not expect to see reuse until all 29 bits have been used once.
While signaling bits can be reused or recycled, as long as the voting period does not overlap, the authors of BIP-9 recommend that bits are reused only when necessary; unexpected behavior could occur due to bugs in older software. In short, we should not expect to see reuse until all 29 bits have been used once.
====
Proposed changes are identified by a data structure that contains the following fields:
@ -1061,26 +1060,26 @@ bit:: 0 through 28, the bit in the block version that miners use to signal appro
starttime:: The time (based on Median Time Past) that signaling starts after which the bit's value is interpreted as signaling readiness for the proposal.
endtime:: The time (based on Median Time Past) after which the change is considered rejected if it has not reach the activation threshold.
endtime:: The time (based on Median Time Past) after which the change is considered rejected if it has not reached the activation threshold.
Unlike BIP-34, BIP-9 counts votes in whole intervals based on the difficulty retarget period of 2016 blocks. For every retarget period, if the sum of blocks signaling for a proposal exceeds 95% (1916 of 2016), the proposal will be activated one retarget period later.
BIP-9 offers a proposal state diagram to illustrate the various stages and transitions for a proposal:
BIP-9 offers a proposal state diagram to illustrate the various stages and transitions for a proposal, as shown in <<bip9states>>.
[[bip9states]]
.BIP-9 Proposal State Transition Diagram
.BIP-9 proposal state transition diagram
image::images/mbc2_1010.png[BIP-9 Proposal State Transition Diagram]
Proposals start in the DEFINED state, once their parameters are known (defined) in the bitcoin software. For blocks with MTP after the start time, the proposal state transitions to STARTED. If the voting threshold is exceeded within a retarget period and the timeout has not been exceeded, the proposal state transitions to LOCKED_IN. One retarget period later, the proposal becomes ACTIVE. Proposals remain in the ACTIVE state perpetually once they reach that state. If the timeout is elapsed before the voting threshold has been reached, the proposal state changes to FAILED, indicating a rejected proposal. REJECTED proposals remain in that state perpetually.
Proposals start in the +DEFINED+ state, once their parameters are known (defined) in the bitcoin software. For blocks with MTP after the start time, the proposal state transitions to +STARTED+. If the voting threshold is exceeded within a retarget period and the timeout has not been exceeded, the proposal state transitions to +LOCKED_IN+. One retarget period later, the proposal becomes +ACTIVE+. Proposals remain in the +ACTIVE+ state perpetually once they reach that state. If the timeout is elapsed before the voting threshold has been reached, the proposal state changes to +FAILED+, indicating a rejected proposal. +REJECTED+ proposals remain in that state perpetually.
BIP-9 was first implemented for the activation of CHECKSEQUENCEVERIFY and associated BIPs (68, 112, 113). The proposal named "csv" was activated successfully in July of 2016.
BIP-9 was first implemented for the activation of +CHECKSEQUENCEVERIFY+ and associated BIPs (68, 112, 113). The proposal named "csv" was activated successfully in July of 2016.
BIP-9 (Version bits with timeout and delay):: https://github.com/bitcoin/bips/blob/master/bip-0009.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0009.mediawiki]
The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0009.mediawiki[BIP-9 (Version bits with timeout and delay)]
=== Consensus Software Development
Consensus software development continues to evolve and there is much discussion on the various mechanisms for changing the consensus rules. By its very nature, bitcoin sets a very high bar on coordination and consensus for changes. As a decentralized system, it has no "authority" that can impose its will on the participants of the network. Power is diffused between multiple constituencies such as miners, core developers, wallet developers, exchanges, merchants and end-users. Decisions cannot be made unilaterally by any of these constituencies. For example, while miners can theoretically change the rules by simple majority (51%), they are constrained by the consent of the other constituencies. If they act unilaterally, the rest of the participants may simply refuse to follow them, keeping the economic activity on a minority chain. Without economic activity (transactions, merchants, wallets, exchanges), the miners will be mining a worthless coin with empty blocks. This diffusion of power means that all the participants must coordinate, or no changes can be made. Status quo is the stable state of this system with only a few changes possible if there is strong consensus by a very large majority. The 95% threshold for soft forks is reflective of this reality.
Consensus software development continues to evolve and there is much discussion on the various mechanisms for changing the consensus rules. By its very nature, bitcoin sets a very high bar on coordination and consensus for changes. As a decentralized system, it has no "authority" that can impose its will on the participants of the network. Power is diffused between multiple constituencies such as miners, core developers, wallet developers, exchanges, merchants, and end users. Decisions cannot be made unilaterally by any of these constituencies. For example, while miners can theoretically change the rules by simple majority (51%), they are constrained by the consent of the other constituencies. If they act unilaterally, the rest of the participants may simply refuse to follow them, keeping the economic activity on a minority chain. Without economic activity (transactions, merchants, wallets, exchanges), the miners will be mining a worthless coin with empty blocks. This diffusion of power means that all the participants must coordinate, or no changes can be made. Status quo is the stable state of this system with only a few changes possible if there is strong consensus by a very large majority. The 95% threshold for soft forks is reflective of this reality.
It is important to recognize that there is no perfect solution for consensus development. Both hard forks and soft forks involve tradeoffs. For some types of changes, soft forks may be a better choice, for others hard forks may be a better choice. There is no perfect choice; both carry risks. The one constant characteristic of consensus software development is that change is difficult and consensus forces compromise.
It is important to recognize that there is no perfect solution for consensus development. Both hard forks and soft forks involve trade-offs. For some types of changes, soft forks may be a better choice, for others hard forks may be a better choice. There is no perfect choice; both carry risks. The one constant characteristic of consensus software development is that change is difficult and consensus forces compromise.
Some see this as a weakness of consensus systems. In time, you may come to see it as I do, as the system's greatest strength.
@ -7,49 +7,49 @@ However, bitcoin has capabilities that cash, gold, and bank accounts do not. A b
=== Security Principles
((("security","principles of")))The core principle in bitcoin is decentralization and it has important implications for security. A centralized model, such as a traditional bank or payment network, depends on access control and vetting to keep bad actors out of the system. By comparison, a decentralized system like bitcoin pushes the responsibility and control to the users. Because security of the network is based on proof of work, not access control, the network can be open and no encryption is required for bitcoin traffic.
((("security","principles of")))The core principle in bitcoin is decentralization and it has important implications for security. A centralized model, such as a traditional bank or payment network, depends on access control and vetting to keep bad actors out of the system. By comparison, a decentralized system like bitcoin pushes the responsibility and control to the users. Because security of the network is based on Proof-of-Work, not access control, the network can be open and no encryption is required for bitcoin traffic.
On a((("credit card payment system")))((("payment networks, traditional"))) traditional payment network, such as a credit card system, the payment is open-ended because it contains the user's private identifier (the credit card number). After the initial charge, anyone with access to the identifier can "pull" funds and charge the owner again and again. Thus, the payment network has to be secured end-to-end with encryption and must ensure that no((("eavesdroppers"))) eavesdroppers or intermediaries can compromise the payment traffic, in transit or when it is stored (at rest). If a bad actor gains access to the system, he can compromise current transactions _and_ payment tokens that can be used to create new transactions. Worse, when customer data is compromised, the customers are exposed to identity theft and must take action to prevent fraudulent use of the compromised accounts.
Bitcoin is dramatically different. A bitcoin transaction authorizes only a specific value to a specific recipient and cannot be forged or modified. It does not reveal any private information, such as the identities of the parties, and cannot be used to authorize additional payments. Therefore, a bitcoin payment network does not need to be encrypted or protected from eavesdropping. In fact, you can broadcast bitcoin transactions over an open public channel, such as unsecured WiFi or Bluetooth, with no loss of security.
Bitcoin's decentralized security model puts a lot of power in the hands of the users. With that power comes responsibility for maintaining the secrecy of the keys. For most users that is not easy to do, especially on general-purpose computing devices such as Internet-connected smartphones or laptops. Although bitcoin's decentralized model prevents the type of mass compromise seen with credit cards, many users are not able to adequately secure their keys and get hacked, one by one.
Bitcoin's decentralized security model puts a lot of power in the hands of the users. With that power comes responsibility for maintaining the secrecy of the keys. For most users that is not easy to do, especially on general-purpose computing devices such as internet-connected smartphones or laptops. Although bitcoin's decentralized model prevents the type of mass compromise seen with credit cards, many users are not able to adequately secure their keys and get hacked, one by one.
==== Developing Bitcoin Systems Securely
((("bitcoin","system security")))((("security","centralized controls and")))The most important principle for bitcoin developers is decentralization. Most developers will be familiar with centralized security models and might be tempted to apply these models to their bitcoin applications, with disastrous results.
Bitcoin's security relies on decentralized control over keys and on independent transaction validation by miners. If you want to leverage Bitcoin's security, you need to ensure that you remain within the Bitcoin security model. In simple terms: don't take control of keys away from users and don't take transactions off the blockchain.
Bitcoin's security relies on decentralized control over keys and on independent transaction validation by miners. If you want to leverage bitcoin's security, you need to ensure that you remain within the bitcoin security model. In simple terms: don't take control of keys away from users and don't take transactions off the blockchain.
For example, many early bitcoin exchanges concentrated all user funds in a single "hot" wallet with keys stored on a single server. Such a design removes control from users and centralizes control over keys in a single system. Many such systems have been hacked, with disastrous consequences for their customers.
((("transactions","taking off blockchain")))Another common mistake is to take transactions "off blockchain" in a misguided effort to reduce transaction fees or accelerate transaction processing. An "off blockchain" system will record transactions on an internal, centralized ledger and only occasionally synchronize them to the bitcoin blockchain. This practice, again, substitutes decentralized bitcoin security with a proprietary and centralized approach. When transactions are off blockchain, improperly secured centralized ledgers can be falsified, diverting funds and depleting reserves, unnoticed.
Unless you are prepared to invest heavily in operational security, multiple layers of access control, and audits (as the traditional banks do) you should think very carefully before taking funds outside of Bitcoin's decentralized security context. Even if you have the funds and discipline to implement a robust security model, such a design merely replicates the fragile model of traditional financial networks, plagued by identity theft, corruption, and embezzlement. To take advantage of Bitcoin's unique decentralized security model, you have to avoid the temptation of centralized architectures that might feel familiar but ultimately subvert Bitcoin's security.
Unless you are prepared to invest heavily in operational security, multiple layers of access control, and audits (as the traditional banks do) you should think very carefully before taking funds outside of bitcoin's decentralized security context. Even if you have the funds and discipline to implement a robust security model, such a design merely replicates the fragile model of traditional financial networks, plagued by identity theft, corruption, and embezzlement. To take advantage of bitcoin's unique decentralized security model, you have to avoid the temptation of centralized architectures that might feel familiar but ultimately subvert bitcoin's security.
==== The Root of Trust
((("root of trust")))((("security","root of trust")))Traditional security architecture is based upon a concept called the _root of trust_, which is a trusted core used as the foundation for the security of the overall system or application. Security architecture is developed around the root of trust as a series of concentric circles, like layers in an onion, extending trust outward from the center. Each layer builds upon the more-trusted inner layer using access controls, digital signatures, encryption, and other security primitives. As software systems become more complex, they are more likely to contain bugs, which make them vulnerable to security compromise. As a result, the more complex a software system becomes, the harder it is to secure. The root of trust concept ensures that most of the trust is placed within the least complex part of the system, and therefore least vulnerable, parts of the system, while more complex software is layered around it. This security architecture is repeated at different scales, first establishing a root of trust within the hardware of a single system, then extending that root of trust through the operating system to higher-level system services, and finally across many servers layered in concentric circles of diminishing trust.
Bitcoin security architecture is different. In Bitcoin, the consensus system creates a trusted public ledger that is completely decentralized. A correctly validated blockchain uses the genesis block as the root of trust, building a chain of trust up to the current block. Bitcoin systems can and should use the blockchain as their root of trust. When designing a complex bitcoin application that consists of services on many different systems, you should carefully examine the security architecture in order to ascertain where trust is being placed. Ultimately, the only thing that should be explicitly trusted is a fully validated blockchain. If your application explicitly or implicitly vests trust in anything but the blockchain, that should be a source of concern because it introduces vulnerability. A good method to evaluate the security architecture of your application is to consider each individual component and evaluate a hypothetical scenario where that component is completely compromised and under the control of a malicious actor. Take each component of your application, in turn, and assess the impacts on the overall security if that component is compromised. If your application is no longer secure when components are compromised, that shows you have misplaced trust in those components. A bitcoin application without vulnerabilities should be vulnerable only to a compromise of the bitcoin consensus mechanism, meaning that its root of trust is based on the strongest part of the bitcoin security architecture.
Bitcoin security architecture is different. In bitcoin, the consensus system creates a trusted public ledger that is completely decentralized. A correctly validated blockchain uses the genesis block as the root of trust, building a chain of trust up to the current block. Bitcoin systems can and should use the blockchain as their root of trust. When designing a complex bitcoin application that consists of services on many different systems, you should carefully examine the security architecture in order to ascertain where trust is being placed. Ultimately, the only thing that should be explicitly trusted is a fully validated blockchain. If your application explicitly or implicitly vests trust in anything but the blockchain, that should be a source of concern because it introduces vulnerability. A good method to evaluate the security architecture of your application is to consider each individual component and evaluate a hypothetical scenario where that component is completely compromised and under the control of a malicious actor. Take each component of your application, in turn, and assess the impacts on the overall security if that component is compromised. If your application is no longer secure when components are compromised, that shows you have misplaced trust in those components. A bitcoin application without vulnerabilities should be vulnerable only to a compromise of the bitcoin consensus mechanism, meaning that its root of trust is based on the strongest part of the bitcoin security architecture.
The numerous examples of hacked bitcoin exchanges serve to underscore this point because their security architecture and design fails even under the most casual scrutiny. These centralized implementations had invested trust explicitly in numerous components outside the bitcoin blockchain, such as hot wallets, centralized ledger databases, vulnerable encryption keys, and similar schemes.
=== User Security Best Practices
((("security","user", id="ix_ch11-asciidoc1", range="startofrange")))((("user security", id="ix_ch11-asciidoc2", range="startofrange")))Humans have used physical security controls for thousands of years. By comparison, our experience with digital security is less than 50 years old. ((("operating systems, bitcoin security and")))Modern general-purpose operating systems are not very secure and not particularly suited to storing digital money. Our computers are constantly exposed to external threats via always-on Internet connections. They run thousands of software components from hundreds of authors, often with unconstrained access to the user's files. A single piece of rogue software, among the many thousands installed on your computer, can compromise your keyboard and files, stealing any bitcoin stored in wallet applications. The level of computer maintenance required to keep a computer virus-free and trojan-free is beyond the skill level of all but a tiny minority of computer users.
((("security","user", id="ix_ch11-asciidoc1", range="startofrange")))((("user security", id="ix_ch11-asciidoc2", range="startofrange")))Humans have used physical security controls for thousands of years. By comparison, our experience with digital security is less than 50 years old. ((("operating systems, bitcoin security and")))Modern general-purpose operating systems are not very secure and not particularly suited to storing digital money. Our computers are constantly exposed to external threats via always-on internet connections. They run thousands of software components from hundreds of authors, often with unconstrained access to the user's files. A single piece of rogue software, among the many thousands installed on your computer, can compromise your keyboard and files, stealing any bitcoin stored in wallet applications. The level of computer maintenance required to keep a computer virus-free and trojan-free is beyond the skill level of all but a tiny minority of computer users.
Despite decades of research and advancements in information security, digital assets are still woefully vulnerable to a determined adversary. Even the most highly protected and restricted systems, in financial services companies, intelligence agencies, and defense contractors, are frequently breached. Bitcoin creates digital assets that have intrinsic value and can be stolen and diverted to new owners instantly and irrevocably. ((("hackers")))This creates a massive incentive for hackers. Until now, hackers had to convert identity information or account tokens—such as credit cards, and bank accounts—into value after compromising them. Despite the difficulty of fencing and laundering financial information, we have seen ever-escalating thefts. Bitcoin escalates this problem because it doesn't need to be fenced or laundered; it is intrinsic value within a digital asset.
Despite decades of research and advancements in information security, digital assets are still woefully vulnerable to a determined adversary. Even the most highly protected and restricted systems, in financial services companies, intelligence agencies, and defense contractors, are frequently breached. Bitcoin creates digital assets that have intrinsic value and can be stolen and diverted to new owners instantly and irrevocably. ((("hackers")))This creates a massive incentive for hackers. Until now, hackers had to convert identity information or account tokens—such as credit cards and bank accounts—into value after compromising them. Despite the difficulty of fencing and laundering financial information, we have seen ever-escalating thefts. Bitcoin escalates this problem because it doesn't need to be fenced or laundered; it is intrinsic value within a digital asset.
Fortunately, bitcoin also creates the incentives to improve computer security. Whereas previously the risk of computer compromise was vague and indirect, bitcoin makes these risks clear and obvious. Holding bitcoin on a computer serves to focus the user's mind on the need for improved computer security. As a direct result of the proliferation and increased adoption of bitcoin and other digital currencies, we have seen an escalation in both hacking techniques and security solutions. In simple terms, hackers now have a very juicy target and users have a clear incentive to defend themselves.
Over the past three years, as a direct result of bitcoin adoption, we have seen tremendous innovation in the realm of information security in the form of hardware encryption, key storage and hardware wallets, multi-signature technology, and digital escrow. In the following sections we will examine various best practices for practical user security.
Over the past three years, as a direct result of bitcoin adoption, we have seen tremendous innovation in the realm of information security in the form of hardware encryption, key storage and hardware wallets, multisignature technology, and digital escrow. In the following sections we will examine various best practices for practical user security.
==== Physical Bitcoin Storage
((("backups","cold-storage wallets")))((("bitcoin","storage, physical")))((("cold-storage wallets")))((("paper wallets")))((("user security","physical bitcoin storage")))Because most users are far more comfortable with physical security than information security, a very effective method for protecting bitcoin is to convert them into physical form. Bitcoin keys are nothing more than long numbers. This means that they can be stored in a physical form, such as printed on paper or etched on a metal coin. Securing the keys then becomes as simple as physically securing the printed copy of the bitcoin keys. A set of bitcoin keys that is printed on paper is called a "paper wallet," and there are many free tools that can be used to create them. I personally keep the vast majority of my bitcoin (99% or more) stored on paper wallets, encrypted with BIP-38, with multiple copies locked in safes. Keeping bitcoin offline is called _cold storage_ and it is one of the most effective security techniques. A cold storage system is one where the keys are generated on an offline system (one never connected to the Internet) and stored offline either on paper or on digital media, such as a USB memory stick.
((("backups","cold-storage wallets")))((("bitcoin","storage, physical")))((("cold-storage wallets")))((("paper wallets")))((("user security","physical bitcoin storage")))Because most users are far more comfortable with physical security than information security, a very effective method for protecting bitcoin is to convert them into physical form. Bitcoin keys are nothing more than long numbers. This means that they can be stored in a physical form, such as printed on paper or etched on a metal coin. Securing the keys then becomes as simple as physically securing the printed copy of the bitcoin keys. A set of bitcoin keys that is printed on paper is called a "paper wallet," and there are many free tools that can be used to create them. I personally keep the vast majority of my bitcoin (99% or more) stored on paper wallets, encrypted with BIP-38, with multiple copies locked in safes. Keeping bitcoin offline is called _cold storage_ and it is one of the most effective security techniques. A cold storage system is one where the keys are generated on an offline system (one never connected to the internet) and stored offline either on paper or on digital media, such as a USB memory stick.
==== Hardware Wallets
@ -57,15 +57,15 @@ Over the past three years, as a direct result of bitcoin adoption, we have seen
==== Balancing Risk
((("risk, security")))((("user security","risk, balancing")))Although most users are rightly concerned about bitcoin theft, there is an even bigger risk. Data files get lost all the time. If they contain bitcoin, the loss is much more painful. In the effort to secure their bitcoin wallets, users must be very careful not to go too far and end up losing the bitcoin. In July of 2011, a well-known bitcoin awareness and education project lost almost 7,000 bitcoin. In their effort to prevent theft, the owners had implemented a complex series of encrypted backups. In the end they accidentally lost the encryption keys, making the backups worthless and losing a fortune. Like hiding money by burying it in the desert, if you secure your bitcoin too well you might not be able to find it again.
((("risk, security")))((("user security","risk, balancing")))Although most users are rightly concerned about bitcoin theft, there is an even bigger risk. Data files get lost all the time. If they contain bitcoin, the loss is much more painful. In the effort to secure their bitcoin wallets, users must be very careful not to go too far and end up losing the bitcoin. In July 2011, a well-known bitcoin awareness and education project lost almost 7,000 bitcoin. In their effort to prevent theft, the owners had implemented a complex series of encrypted backups. In the end they accidentally lost the encryption keys, making the backups worthless and losing a fortune. Like hiding money by burying it in the desert, if you secure your bitcoin too well you might not be able to find it again.
==== Diversifying Risk
((("user security","risk, diversifying")))Would you carry your entire net worth in cash in your wallet? Most people would consider that reckless, yet bitcoin users often keep all their bitcoin in a single wallet. Instead, users should spread the risk among multiple and diverse bitcoin wallets. Prudent users will keep only a small fraction, perhaps less than 5%, of their bitcoin in an online or mobile wallet as "pocket change." The rest should be split between a few different storage mechanisms, such as a desktop wallet and offline (cold storage).
==== Multi-sig and Governance
==== Multisig and Governance
((("corporations, multi-sig governance and")))((("governance")))((("multi-signature addresses","security and")))((("security","governance")))((("security","multi-signature addresses and")))Whenever a company or individual stores large amounts of bitcoin, they should consider using a multi-signature bitcoin address. Multi-signature addresses secure funds by requiring more than one signature to make a payment. The signing keys should be stored in a number of different locations and under the control of different people. In a corporate environment, for example, the keys should be generated independently and held by several company executives, to ensure no single person can compromise the funds. Multi-signature addresses can also offer redundancy, where a single person holds several keys that are stored in different locations.
((("corporations, multi-sig governance and")))((("governance")))((("multi-signature addresses","security and")))((("security","governance")))((("security","multi-signature addresses and")))Whenever a company or individual stores large amounts of bitcoin, they should consider using a multisignature bitcoin address. Multisignature addresses secure funds by requiring more than one signature to make a payment. The signing keys should be stored in a number of different locations and under the control of different people. In a corporate environment, for example, the keys should be generated independently and held by several company executives, to ensure no single person can compromise the funds. Multisignature addresses can also offer redundancy, where a single person holds several keys that are stored in different locations.
Let's now build on our understanding of bitcoin by looking at it as an _application platform_. Nowadays, many people use the term "blockchain" to refer to any application platform that shares the design principles of bitcoin. The term is often misused and applied to many things that fail to deliver the primary features that bitcoin's blockchain delivers.
In this chapter we will look at the features offered by the bitcoin blockchain, as an application platform. We will consider the application building _primitives_, which form the building blocks of any blockchain application. We will look at several important applications that use these primitives, such as: colored coins, payment (state) channels and routed payment channels (Lightning Network).
In this chapter we will look at the features offered by the bitcoin blockchain, as an application platform. We will consider the application building _primitives_, which form the building blocks of any blockchain application. We will look at several important applications that use these primitives, such as colored coins, payment (state) channels, and routed payment channels (Lightning Network).
=== Introduction
((("blockchain applications", id="ix_ch12-asciidoc0", range="startofrange")))The bitcoin system was designed as a decentralized currency and payment system. However, most of its functionality is derived from much lowerlevel constructs that can be used for much broader applications. Bitcoin wasn't built with components such as accounts, users, balances and payments. Instead, it uses a transactional scripting language with lowlevel cryptographic functions, as we saw in <<transactions>>. Just like the higher-level concepts of accounts, balances and payments can be derived from these basic primitives, so can many other complex applications. Thus, the bitcoin blockchain can become an application platform offering trust services to applications, such as smart contracts, far surpassing the original purpose of digital currency and payments.
((("blockchain applications", id="ix_ch12-asciidoc0", range="startofrange")))The bitcoin system was designed as a decentralized currency and payment system. However, most of its functionality is derived from much lower-level constructs that can be used for much broader applications. Bitcoin wasn't built with components such as accounts, users, balances, and payments. Instead, it uses a transactional scripting language with low-level cryptographic functions, as we saw in <<transactions>>. Just like the higher-level concepts of accounts, balances, and payments can be derived from these basic primitives, so can many other complex applications. Thus, the bitcoin blockchain can become an application platform offering trust services to applications, such as smart contracts, far surpassing the original purpose of digital currency and payments.
=== Building Blocks (Primitives)
@ -15,7 +15,7 @@ When operating correctly and over the long term, the bitcoin system offers certa
No Double-Spend:: The most fundamental guarantee of bitcoin's decentralized consensus algorithm ensures that no UTXO can be spent twice.
Immutability:: Once a transaction is recorded in the blockchain and sufficient work has been added with subsequent blocks, the transaction's data becomes immutable. Immutability is underwritten by energy, as rewriting the blockchain requires the expenditure of energy to produce proof of work. The energy required and therefore the degree of immutability increases with the amount of work committed on top of the block containing a transaction.
Immutability:: Once a transaction is recorded in the blockchain and sufficient work has been added with subsequent blocks, the transaction's data becomes immutable. Immutability is underwritten by energy, as rewriting the blockchain requires the expenditure of energy to produce Proof-of-Work. The energy required and therefore the degree of immutability increases with the amount of work committed on top of the block containing a transaction.
Neutrality:: The decentralized bitcoin network propagates valid transactions regardless of the origin or content of those transactions. This means that anyone can create a valid transaction with sufficient fees and trust they will be able to transmit that transaction and have it included in the blockchain at anytime.
@ -29,13 +29,13 @@ Accounting:: In any transaction (except the coinbase transaction) the value of i
Non-Expiration:: A valid transaction does not expire. If it is valid today, it will be valid in the near future, as long as the inputs remain unspent and the consensus rules do not change.
Integrity:: A bitcoin transaction signed with SIGHASH_ALL or parts of a transaction signed by another SIGHASH type cannot be modified without invalidating the signature, thus invalidating the transaction itself.
Integrity:: A bitcoin transaction signed with +SIGHASH_ALL+ or parts of a transaction signed by another +SIGHASH+ type cannot be modified without invalidating the signature, thus invalidating the transaction itself.
Transaction Atomicity:: Bitcoin transactions are atomic. They are either valid and confirmed (mined) or not. Partial transactions cannot be mined and there is no interim state for a transaction. At any point in time a transaction is either mined, or not.
Discrete (Indivisible) Units of Value:: Transaction outputs are discrete and indivisible units of value. They can either be spent or unspent, in full. They cannot be divided or partially spent.
Quorum of Control:: Multi-signature constraints in scripts impose a quorum of authorization, predefined in the multi-signature scheme. The M-of-N requirement is enforced by the consensus rules.
Quorum of Control:: Multisignature constraints in scripts impose a quorum of authorization, predefined in the multisignature scheme. The M-of-N requirement is enforced by the consensus rules.
Timelock/Aging:: Any script clause containing a relative or absolute timelock can only be executed after its age exceeds the time specified.
@ -45,11 +45,11 @@ Forgery Protection:: A transaction can only spend existing, validated outputs. I
Consistency:: In the absence of miner partitions, blocks that are recorded in the blockchain are subject to reorganization or disagreement with exponentially decreasing likelihood, based on the depth at which they are recorded. Once deeply recorded, the computation and energy required to change makes change practically infeasible.
Recording External State:: A transaction can commit a data value, via OP_RETURN, representing a state transition in an external state machine.
Recording External State:: A transaction can commit a data value, via +OP_RETURN+, representing a state transition in an external state machine.
Predictable Issuance:: Less than 21 million bitcoin will be issued, at a predictable rate.
The list of building blocks above is not complete and more are added with each new feature introduced into bitcoin.
The list of building blocks is not complete and more are added with each new feature introduced into bitcoin.
=== Applications from Building Blocks
@ -59,25 +59,25 @@ Proof-of-Existence (Digital Notary):: Immutability + Timestamp + Durability. A d
Kickstarter (Lighthouse):: Consistency + Atomicity + Integrity. If you sign one input and the output (Integrity) of a fundraiser transaction, others can contribute to the fundraiser but it cannot be spent (Atomicity) until the goal (output value) is funded (Consistency).
Payment Channels:: Quorum of Control + Timelock + No Double Spend + Non-Expiration + Censorship Resistance + Authorization. A multi-sig 2-of-2 (Quorum) with a timelock (Timelock), used as the "settlement" transaction of a payment channel can be held (Non-Expiration) and spent whenever (Censorship Resistance) by either party (Authorization). The two parties can then create update transactions that double-spend (No DoubleSpend) the settlement on a shorter timelock (Timelock).
Payment Channels:: Quorum of Control + Timelock + No Double Spend + Non-Expiration + Censorship Resistance + Authorization. A multisig 2-of-2 (Quorum) with a timelock (Timelock), used as the "settlement" transaction of a payment channel can be held (Non-Expiration) and spent whenever (Censorship Resistance) by either party (Authorization). The two parties can then create update transactions that double-spend (No Double-Spend) the settlement on a shorter timelock (Timelock).
=== Colored Coins
The first blockchain application we will discuss is _Colored Coins_
The first blockchain application we will discuss is _Colored coins_.
Colored coins refers to a set of similar technologies that use bitcoin transactions to record the creation, ownership and transfer of extrinsic assets other than bitcoin. By "extrinsic" we mean assets that are not stored directly on the bitcoin blockchain, as opposed to bitcoin itself which is an asset intrinsic to the blockchain.
Colored coins refers to a set of similar technologies that use bitcoin transactions to record the creation, ownership, and transfer of extrinsic assets other than bitcoin. By "extrinsic" we mean assets that are not stored directly on the bitcoin blockchain, as opposed to bitcoin itself, which is an asset intrinsic to the blockchain.
Colored coins are used to track digital assets as well as physical assets held by third parties and traded through colored coins certificates of ownership. Digital asset colored coins can represent intangible assets such as a stock certificate, license, virtual property (game items), or most any form of licensed intellectual property (trademarks, copyrights, etc). Tangible asset colored coins can represent certificates of ownership of commodities (gold, silver, oil), land title, automobiles, boats, aircraft, etc.
The term derives from the idea of "coloring" or marking a nominal amount of bitcoin, for example a single satoshi, to represent something other than the bitcoin value itself. As an analogy, consider stamping a $1 note with a message saying "This is a stock certificate of ACME" or "This note can be redeemed for 1 oz of silver" and then trading the $1 note as a certificate of ownership of this other asset. The first implementation of colored coins, named _Enhanced Padded-Order-Based Coloring_ or _EPOBC_ assigned extrinsic assets to a 1satoshi output. In this way, it was a true "colored coin", as each asset was added as an attribute (color) of a single satoshi.
The term derives from the idea of "coloring" or marking a nominal amount of bitcoin, for example a single satoshi, to represent something other than the bitcoin value itself. As an analogy, consider stamping a $1 note with a message saying "This is a stock certificate of ACME" or "This note can be redeemed for 1 oz of silver" and then trading the $1 note as a certificate of ownership of this other asset. The first implementation of colored coins, named _Enhanced Padded-Order-Based Coloring_ or _EPOBC_ assigned extrinsic assets to a 1-satoshi output. In this way, it was a true "colored coin," as each asset was added as an attribute (color) of a single satoshi.
After _EPOBC_, more recent implementations of colored coins use the OP_RETURN script opcode to store metadata in a transaction, in conjunction with external data stores which associate the metadata to specific assets.
After _EPOBC_, more recent implementations of colored coins use the +OP_RETURN+ script opcode to store metadata in a transaction, in conjunction with external data stores that associate the metadata to specific assets.
The two most prominent implementations of colored coins today are http://www.openassets.org/[_Open Assets_] and http://coloredcoins.org[_Colored Coins by Colu_]. These two systems use different approaches to colored coins and are not compatible. Colored coins created in one system cannot be seen or used in the other system.
==== Using Colored Coins
Colored coins are created, transfered and generally viewed in special wallets that can interpret the colored coins protocol metadata attached to bitcoin transactions. Special care must be taken to avoid using a colored coin related key in a regular bitcoin wallet, as the regular wallet may destroy the metadata. Similarly, colored coins should not be sent to addresses managed by regular wallets, but only to addresses that are managed by wallets that are colored-coin-aware. Both Colu and Open Assets systems use special colored-coin addresses to mitigate this risk and to ensure that colored coins are not sent to unaware wallets.
Colored coins are created, transferred, and generally viewed in special wallets that can interpret the colored coins protocol metadata attached to bitcoin transactions. Special care must be taken to avoid using a colored-coin–related key in a regular bitcoin wallet, as the regular wallet may destroy the metadata. Similarly, colored coins should not be sent to addresses managed by regular wallets, but only to addresses that are managed by wallets that are colored-coin-aware. Both Colu and Open Assets systems use special colored-coin addresses to mitigate this risk and to ensure that colored coins are not sent to unaware wallets.
Colored coins are also not visible to most general-purpose blockchain explorers. Instead, you must use a colored-coins explorer to interpret the metadata of a colored coins transaction.
@ -89,60 +89,63 @@ A Colu Colored Coins compatible wallet application and blockchain explorer can b
Each of the colored coins implementations has a different way of creating colored coins, but they all provide similar functionality. The process of creating a colored coin asset is called _issuance_. An initial transaction, the _issuance transaction_ registers the asset on the bitcoin blockchain and creates an _asset ID_ that is used to reference the asset. Once issued, assets can be transferred between addresses using _transfer transactions_.
Assets issued as colored coins can have multiple properties. They can be _divisible_ or _indivisible_, meaning that the amount of asset in a transfer can be an integer (eg. 5) or have decimal subdivision (eg. 4.321). Assets can also have _fixed issuance_, meaning a certain amount are issued only once, or can be _reissued_, meaning that new units of the asset can be issued by the original issuer after the initial issuance.
Assets issued as colored coins can have multiple properties. They can be _divisible_ or _indivisible_, meaning that the amount of asset in a transfer can be an integer (e.g., 5) or have decimal subdivision (e.g., 4.321). Assets can also have _fixed issuance_, meaning a certain amount are issued only once, or can be _reissued_, meaning that new units of the asset can be issued by the original issuer after the initial issuance.
Finally, some colored coins enable _dividends_, allowing the distribution of bitcoin payments to the owners of a colored coin asset in proportion to their ownership.
==== Colored Coins Transactions
The metadata that gives meaning to a colored coin transaction is usually stored in one of the outputs using the OP_RETURN opcode. Different colored coins protocols use different encodings for the content of the OP_RETURN data. The output containing the OP_RETURN is called the _marker output_.
The metadata that gives meaning to a colored coin transaction is usually stored in one of the outputs using the +OP_RETURN+ opcode. Different colored coins protocols use different encodings for the content of the +OP_RETURN+ data. The output containing the +OP_RETURN+ is called the _marker output_.
The order of the outputs and position of the marker output may have special meaning in the colored coins protocol. In Open Assets, for example, any outputs before the marker output represent asset issuance. Any outputs after the marker represent asset transfer. The marker output assigns specific values and colors to the other outputs by referencing their order in the transaction.
In Colored Coins (Colu), by comparison, the marker output encodes an opcode that determines how the metadata is interpreted. Opcodes 0x01 through 0x0F indicate an issuance transaction. An issuance opcode is usually followed by an asset ID or other identifier that can be used to retrieve the asset information from an external source (eg. bittorrent).
In Colored Coins (Colu), by comparison, the marker output encodes an opcode that determines how the metadata is interpreted. Opcodes 0x01 through 0x0F indicate an issuance transaction. An issuance opcode is usually followed by an asset ID or other identifier that can be used to retrieve the asset information from an external source (e.g., bittorrent).
Opcodes 0x10 through 0x1F represent a transfer transaction. Transfer transaction metadata contain simple scripts that transfer specific amounts of assets from inputs to outputs, by reference to their index. Ordering of inputs and outputs is therefore important in the interpretation of the script.
If the metadata is too long to fit in OP_RETURN, the colored coins protocol may use other "tricks" to store metadata in a transaction. Examples include putting metadata in a redeem script, followed by OP_DROP opcodes to ensure the script ignores the metadata. Another mechanism used is a 1-of-N multi-sig script where only the first public key is a real public key that can spend the output and subsequent "keys" are replaced by encoded metadata.
If the metadata is too long to fit in +OP_RETURN+, the colored coins protocol may use other "tricks" to store metadata in a transaction. Examples include putting metadata in a redeem script, followed by +OP_DROP+ opcodes to ensure the script ignores the metadata. Another mechanism used is a 1-of-N multisig script where only the first public key is a real public key that can spend the output and subsequent "keys" are replaced by encoded metadata.
In order to correctly interpret the metadata in a colored coins transaction you must use a compatible wallet or block explorer. Otherwise, the transaction looks like a "normal" bitcoin transaction with an OP_RETURN output.
In order to correctly interpret the metadata in a colored coins transaction you must use a compatible wallet or block explorer. Otherwise, the transaction looks like a "normal" bitcoin transaction with an +OP_RETURN+ output.
As an example, I created and issued a MasterBTC asset using colored coins. The MasterBTC asset represents a voucher for a free copy of this book. These vouchers can be transferred, traded and redeemed using a colored coins compatible wallet.
As an example, I created and issued a MasterBTC asset using colored coins. The MasterBTC asset represents a voucher for a free copy of this book. These vouchers can be transferred, traded, and redeemed using a colored coins compatible wallet.
For this particular example, I used the wallet and explorer at http://coinprism.info/[https://coinprism.info], which uses the Open Assets colored coin protocol.
Let's look at the issuance transaction using the Coinprism block explorer:
<<the_issuance_transaction_as_viewed_on_coinprismio>> shows the issuance transaction using the Coinprism block explorer:
.The issuance transaction as viewed on coinprism.info
image::images/mbc2_1201.png[The Issuance Transaction - as viewed on coinprism.info]
As you can see, coinprism shows the issuance of 20 units of "Free copy of Mastering Bitcoin", the MasterBTC asset, to a special colored coin address +akTnsDt5uzpioRST76VFRQM8q8sBFnQiwcx+
As you can see, coinprism shows the issuance of 20 units of "Free copy of Mastering Bitcoin," the MasterBTC asset, to a special colored coin address +akTnsDt5uzpioRST76VFRQM8q8sBFnQiwcx+
[WARNING]
====
Any funds or colored assets sent to this address will be lost forever. Do not send value to this example address!
====
The transaction ID of the issuance transaction is a "normal" bitcoin transaction ID. Let's look at that same transaction in a block explorer that doesn't decode colored coins. We'll use blockchain.info
The transaction ID of the issuance transaction is a "normal" bitcoin transaction ID. <<the_issuance_transaction_on_a_block>> shows that same transaction in a block explorer that doesn't decode colored coins. We'll use blockchain.info:
.The Issuance Transaction - on a block explorer that doesn't decode colored coins
[[the_issuance_transaction_on_a_block]]
.The issuance transaction on a block explorer that doesn't decode colored coins
image::images/mbc2_1202.png[The Issuance Transaction - on a block explorer that doesn't decode colored coins]
As you can see, blockchain.info doesn't recognize this as a colored coins transaction. In fact, it marks the second output with "Unable to decode output address" in red letters.
If you select "Show scripts & coinbase" on that screen, you can see more detail about the transaction:
If you select "Show scripts & coinbase" on that screen, you can see more detail about the transaction (<<the_scripts_in_the_issuance_transaction>>).
.The scripts in the Issuance Transaction
[[the_scripts_in_the_issuance_transaction]]
.The scripts in the issuance transaction
image::images/mbc2_1203.png[The scripts in the Issuance Transaction]
Once again, blockchain.info doesn't understand the second output. It marks it with "Strange" in red letters. However, we can see some of the metadata in the marker output is human-readable:
@ -170,13 +173,13 @@ Stripping out the rest of the transaction, the second output looks like this:
}
----
The prefix +4F41+ represents the letters "OA" which stands for "Open Assets" and helps us identify that what follows is metadata defined by the Open Assets protocol. The ASCIIencoded string that follows is a link to an asset definition:
The prefix +4F41+ represents the letters "OA", which stands for "Open Assets" and helps us identify that what follows is metadata defined by the Open Assets protocol. The ASCII-encoded string that follows is a link to an asset definition:
----
u=https://cpr.sm/FoykwrH6UY
----
If we retrieve this URL, we get a JSON encoded asset definition, as shown below:
If we retrieve this URL, we get a JSON-encoded asset definition, as shown here:
[source,json]
----
@ -203,56 +206,56 @@ If we retrieve this URL, we get a JSON encoded asset definition, as shown below:
Counterparty is a protocol layer built on top of bitcoin. The Counterparty protocol, similarly to colored coins, offers the ability to create and trade virtual assets and tokens. In addition, Counterparty offers a decentralized exchange for assets. Counterparty is also implementing smart contracts, based on the Ethereum Virtual Machine (EVM).
Like the colored coins protocols, Counterparty embeds metadata in bitcoin transactions, using the OP_RETURN opcode or 1-of-N multisignature addresses that encode metadata in the place of public keys. Using these mechanisms, Counterparty implements a protocol layer encoded in bitcoin transactions. The additional protocol layer can be interpreted by applications that are Counterparty-aware, such as wallets and blockchain explorers, or any application built using the Counterparty libraries.
Like the colored coins protocols, Counterparty embeds metadata in bitcoin transactions, using the +OP_RETURN+ opcode or 1-of-N multisignature addresses that encode metadata in the place of public keys. Using these mechanisms, Counterparty implements a protocol layer encoded in bitcoin transactions. The additional protocol layer can be interpreted by applications that are Counterparty-aware, such as wallets and blockchain explorers, or any application built using the Counterparty libraries.
Counterparty can be used as a platform for other applications and services, in turn. For example, Tokenly, is a platform built on top of Counterparty that allows content creators, artists and companies to issue tokens that express digital ownership and can be used to rent, access, trade or shop for content, products and services. Other applications leveraging Counterparty include games (Spells of Genesis) and grid computing projects (Folding Coin).
Counterparty can be used as a platform for other applications and services, in turn. For example, Tokenly is a platform built on top of Counterparty that allows content creators, artists, and companies to issue tokens that express digital ownership and can be used to rent, access, trade, or shop for content, products, and services. Other applications leveraging Counterparty include games (Spells of Genesis) and grid computing projects (Folding Coin).
More details about Counterparty can be found at https://counterparty.io. The open source project can be found at https://github.com/CounterpartyXCP
[[state_channels]]
=== Payment Channels and State Channels
_Payment Channels_ are a trustless mechanism for exchanging bitcoin transactions between two parties, outside of the bitcoin blockchain. These transactions, which would be valid if settled on the bitcoin blockchain, are held off-chain instead, acting as _promissory notes_ for eventual batch settlement. Because the transactions are not settled, they can be exchanged without the usual settlement latency, allowing extremely high transaction throughput, low (sub-millisecond) latency and fine (satoshi-level) granularity.
_Payment channels_ are a trustless mechanism for exchanging bitcoin transactions between two parties, outside of the bitcoin blockchain. These transactions, which would be valid if settled on the bitcoin blockchain, are held off-chain instead, acting as _promissory notes_ for eventual batch settlement. Because the transactions are not settled, they can be exchanged without the usual settlement latency, allowing extremely high transaction throughput, low (sub-millisecond) latency, and fine (satoshi-level) granularity.
Actually, the term channel is a metaphor. State channels are virtual constructs represented by the exchange of state between two parties, outside of the blockchain. There are no "channels", per-se and the underlying data transport mechanism is not the channel. We use the term channel to represent the relationship and shared state between two parties, outside of the blockchain.
Actually, the term _channel_ is a metaphor. State channels are virtual constructs represented by the exchange of state between two parties, outside of the blockchain. There are no "channels," per-se and the underlying data transport mechanism is not the channel. We use the term channel to represent the relationship and shared state between two parties, outside of the blockchain.
To further explain this concept, think of a TCP stream. From the perspective of higher-level protocols it is a "socket" connecting two applications across the Internet. But if you look at the network traffic, a TCP stream is just a virtual channel over IP packets. Each endpoint of the TCP stream sequences and assembles IP packets to create the illusion of a stream of bytes. Underneath, it's all disconnected packets. Similarly, a payment channel is just a series of transactions. If properly sequenced and connected, they create redeemable obligations that you can trust even though you don't trust the other side of the channel.
To further explain this concept, think of a TCP stream. From the perspective of higher-level protocols it is a "socket" connecting two applications across the internet. But if you look at the network traffic, a TCP stream is just a virtual channel over IP packets. Each endpoint of the TCP stream sequences and assembles IP packets to create the illusion of a stream of bytes. Underneath, it's all disconnected packets. Similarly, a payment channel is just a series of transactions. If properly sequenced and connected, they create redeemable obligations that you can trust even though you don't trust the other side of the channel.
In this section we will look at various forms of payment channels. First we will examine the mechanisms used to construct a one-way (uni-directional) payment channel for a metered micro-payment service, such as streaming video. Then, we will expand on this mechanism and introduce bi-directional payment channels. Finally, we will look at how bi-directional channels can be connected end-to-end to form multi-hop channels in a routed network, first proposed under the name _Lightning Network_.
In this section we will look at various forms of payment channels. First we will examine the mechanisms used to construct a one-way (unidirectional) payment channel for a metered micro-payment service, such as streaming video. Then, we will expand on this mechanism and introduce bidirectional payment channels. Finally, we will look at how bidirectional channels can be connected end-to-end to form multihop channels in a routed network, first proposed under the name _Lightning Network_.
Payment channels are part of the broader concept of a _State Channel_, which represents an off-chain alteration of state, secured by eventual settlement in a blockchain. A payment channel is a state channel where the state being altered is the balance of a virtual currency.
Payment channels are part of the broader concept of a _state channel_, which represents an off-chain alteration of state, secured by eventual settlement in a blockchain. A payment channel is a state channel where the state being altered is the balance of a virtual currency.
==== State Channels - Basic Concepts and Terminology
==== State Channels—Basic Concepts and Terminology
A state channel is established between two parties, through a transaction that locks a shared state on the blockchain. This is called the _funding transaction_ or _anchor transaction_. This single transaction must be transmitted to the network and mined to establish the channel. In the example of a payment channel, the locked state is the initial balance (in currency) of the channel.
The two parties then exchange signed transactions, called _commitment transactions_ that alter the initial state. These transactions are valid transactions in that they _could_ be submitted for settlement by either party, but instead are held off-chain by each party pending the channel closure. State updates can be created as fast as each party can create, sign and transmit a transaction to the other party. In practice this means that thousands of transactions per second can be exchanged.
The two parties then exchange signed transactions, called _commitment transactions_ that alter the initial state. These transactions are valid transactions in that they _could_ be submitted for settlement by either party, but instead are held off-chain by each party pending the channel closure. State updates can be created as fast as each party can create, sign, and transmit a transaction to the other party. In practice this means that thousands of transactions per second can be exchanged.
When exchanging commitment transactions the two parties also invalidate the previous states, so that the most up-to-date commitment transaction is always the only one that can be redeemed. This prevents either party from cheating by unilaterally closing the channel with an expired prior state that is more favorable to them than the current state. We will examine the various mechanisms that can be used to invalidate prior state in the rest of this chapter.
Finally, the channel can be closed either cooperatively, by submitting a final _settlement transaction_ to the blockchain, or unilaterally by either party submitting the last commitment transaction to the blockchain. A unilateral close option is needed in case one of the parties unexpectedly disconnects. The settlement transaction represents the final state of the channel and is settled on the blockchain.
Finally, the channel can be closed either cooperatively, by submitting a final _settlement transaction_ to the blockchain, or unilaterally, by either party submitting the last commitment transaction to the blockchain. A unilateral close option is needed in case one of the parties unexpectedly disconnects. The settlement transaction represents the final state of the channel and is settled on the blockchain.
In the entire lifetime of the channel, only two transactions need to be submitted for mining on the blockchain: the funding and settlement transactions. In between these two states, the two parties can exchange any number of commitment transactions that are never seen by anyone else, nor submitted to the blockchain.
==== Simple Payment Channel Example
To explain state channels, we have to start with a very simple example. We demonstrate a one-way channel, meaning that value is flowing in one direction only. We will also start with the naive assumption that no one is trying to cheat, to keep things simple. Once we have the basic channel idea explained, we will then look at what it takes to make it trustless so that neither party *can* cheat, even if they are trying to.
To explain state channels, we have to start with a very simple example. We demonstrate a one-way channel, meaning that value is flowing in one direction only. We will also start with the naive assumption that no one is trying to cheat, to keep things simple. Once we have the basic channel idea explained, we will then look at what it takes to make it trustless so that neither party _can_ cheat, even if they are trying to.
For this example we will assume two participants: Emma and Fabian. Fabian offers a video streaming service that is billed by the second, using a micro-payment channel. Fabian charges 0.01 millibit (0.00001 BTC) per second of video, equivalent to 36 millibits (0.036 BTC) per hour of video. Emma is a user who purchases this streaming video service from Fabian.
In this example, Fabian and Emma are using special software that handles both the payment channel and the video streaming. Emma is running the software in her browser, Fabian is running it on a server. The software includes basic bitcoin wallet functionality and can create and sign bitcoin transactions. Both the concept and the term "payment channel" are completely hidden from the users. What they see is video that is paid for by the second.
To setup the payment channel, Emma and Fabian establish a 2-of-2 multisignature address, with each of them holding one of the keys. From Emma's perspective, the software in her browser presents a QR code with a P2SH address (starting with "3"), and asks her to submit a "deposit" for up to 1 hour of video. The address is then funded by Emma. Emma's transaction, paying to the multisignature address is the funding or anchor transaction for the payment channel.
To setup the payment channel, Emma and Fabian establish a 2-of-2 multisignature address, with each of them holding one of the keys. From Emma's perspective, the software in her browser presents a QR code with a P2SH address (starting with "3"), and asks her to submit a "deposit" for up to 1 hour of video. The address is then funded by Emma. Emma's transaction, paying to the multisignature address, is the funding or anchor transaction for the payment channel.
For this example, let's say that Emma funds the channel with 36 millibits (0.036 BTC). This will allow Emma to consume *up to* 1 hour of streaming video. The funding transaction in this case sets the maximum amount that can be transmitted in this channel, setting the _channel capacity_.
For this example, let's say that Emma funds the channel with 36 millibits (0.036 BTC). This will allow Emma to consume _up to_ 1 hour of streaming video. The funding transaction in this case sets the maximum amount that can be transmitted in this channel, setting the _channel capacity_.
The funding transaction consumes one or more inputs from Emma's wallet, sourcing the funds. It creates one output with a value of 36 millibits paid to the multisignature 2-of-2 address controlled jointly between Emma and Fabian. It may have additional outputs for change back to Emma's wallet.
Once the funding transaction is confirmed, Emma can start streaming video. Emma's software creates and signs a commitment transaction that changes the channel balance to credit 0.01 millibit to Fabian's address and refund 35.99 millibits back to Emma. The transaction signed by Emma consumes the 36 millibits output created by the funding transaction and creates two outputs: one for her refund, the other for Fabian's payment. The transaction is only partially signed - it requires 2 signatures (2-of-2), but only has Emma's signature. When Fabian's server receives this transaction, it adds the second signature (for the 2-of-2 input) and returns it to Emma together with 1 second worth of video. Now both parties have a fully signed commitment transaction that either can redeem, representing the correct up-to-date balance of the channel. Neither party broadcasts this transaction to the network.
Once the funding transaction is confirmed, Emma can start streaming video. Emma's software creates and signs a commitment transaction that changes the channel balance to credit 0.01 millibit to Fabian's address and refund 35.99 millibits back to Emma. The transaction signed by Emma consumes the 36 millibits output created by the funding transaction and creates two outputs: one for her refund, the other for Fabian's payment. The transaction is only partially signed—it requires two signatures (2-of-2), but only has Emma's signature. When Fabian's server receives this transaction, it adds the second signature (for the 2-of-2 input) and returns it to Emma together with 1 second worth of video. Now both parties have a fully signed commitment transaction that either can redeem, representing the correct up-to-date balance of the channel. Neither party broadcasts this transaction to the network.
In the next round, Emma's software creates and signs another commitment transaction (commitment #2) that consumes the *same* 2-of-2 output from the funding transaction. The second commitment transaction allocates one output of 0.2 millibits to Fabian's address and one output of 35.98 millibits back to Emma's address. This new transaction is payment for two cumulative seconds of video. Fabian's software signs and returns the second commitment transaction, together with the another second of video.
In the next round, Emma's software creates and signs another commitment transaction (commitment #2) that consumes the _same_ 2-of-2 output from the funding transaction. The second commitment transaction allocates one output of 0.2 millibits to Fabian's address and one output of 35.98 millibits back to Emma's address. This new transaction is payment for two cumulative seconds of video. Fabian's software signs and returns the second commitment transaction, together with the another second of video.
In this way, Emma's software continues to send commitment transactions to Fabian's server in exchange for streaming video. The balance of the channel gradually accumulates in favor of Fabian, as Emma consumes more seconds of video. Let's say Emma watches 600 seconds (10 minutes) of video, creating and signing 600 commitment transactions. The last commitment transaction (#600) will have two outputs, splitting the balance of the channel 6 millibits to Fabian and 30 millibits to Emma.
In this way, Emma's software continues to send commitment transactions to Fabian's server in exchange for streaming video. The balance of the channel gradually accumulates in favor of Fabian, as Emma consumes more seconds of video. Let's say Emma watches 600 seconds (10 minutes) of video, creating and signing 600 commitment transactions. The last commitment transaction (#600) will have two outputs, splitting the balance of the channel, 6 millibits to Fabian and 30 millibits to Emma.
Finally, Emma clicks "Stop" to stop streaming video. Either Fabian or Emma can now transmit the final state transaction for settlement. This last transaction is the _settlement transaction_ and pays Fabian for all the video Emma consumed, refunding the remainder of the funding transaction to Emma.
@ -260,35 +263,35 @@ In the end, only two transactions are recorded on the blockchain: the funding tr
==== Making Trustless Channels
The channel we described above works, but only if both parties cooperate, without any failures or attempts to cheat. Let's look at some of the scenarios that break this channel and see what is needed to fix those:
The channel we just described works, but only if both parties cooperate, without any failures or attempts to cheat. Let's look at some of the scenarios that break this channel and see what is needed to fix those:
* Once the funding transaction happens, Emma needs Fabian's signature to get any money back. If Fabian disappears, Emma's funds are locked in a 2-of-2 and effectively lost. This channel, as constructed, leads to a loss of funds if one of the parties disconnects before there is at least one commitment transaction signed by both parties.
* While the channel is running, Emma can take any of the commitment transactions Fabian has countersigned and transmit one to the blockchain. Why pay for 600 seconds of video, if she can transmit commitment transaction #1 and only pay for 1 second of video? The channel fails because Emma can cheat by broadcasting a prior commitment that is in her favor.
Both of the above problems can be solved with timelocks, let's look at how we could use transaction level timelocks (nLockTime).
Both of these problems can be solved with timelocks—let's look at how we could use transaction level timelocks (+nLocktime+).
Emma cannot risk funding a 2-of-2 multisig unless she has a guaranteed refund. To solve this problem, Emma constructs the funding and refund transaction at the same time. She signs the funding transaction but doesn't transmit it to anyone. Emma transmits the refund transaction, only, to Fabian and obtains his signature.
The refund transaction acts as the first commitment transaction and its timelock establishes the upper bound for the channel's life. In this case, Emma could set the nLockTime to 30 days or 4320 blocks into the future. All subsequent commitment transactions must have a shorter timelock, so that they can be redeemed before the refund transaction.
The refund transaction acts as the first commitment transaction and its timelock establishes the upper bound for the channel's life. In this case, Emma could set the +nLocktime+ to 30 days or 4320 blocks into the future. All subsequent commitment transactions must have a shorter timelock, so that they can be redeemed before the refund transaction.
Now that Emma has a fully signed refund transaction, she can confidently transmit the signed funding transaction knowing that she can eventually, after the timelock expires, redeem the refund transaction even if Fabian disappears.
Every commitment transaction the parties exchange during the life of the channel will be timelocked into the future. But the delay will be slightly shorter for each commitment so the most recent commitment can be redeemed before the prior commitment it invalidates. Because of the nLockTime, neither party can successfully propagate any of the commitment transactions until their timelock expires. If all goes well, they will cooperate and close the channel gracefully with a settlement transaction, making it unnecessary to transmit an intermediate commitment transaction. In essence, the commitment transactions are only used in
Every commitment transaction the parties exchange during the life of the channel will be timelocked into the future. But the delay will be slightly shorter for each commitment so the most recent commitment can be redeemed before the prior commitment it invalidates. Because of the +nLocktime+, neither party can successfully propagate any of the commitment transactions until their timelock expires. If all goes well, they will cooperate and close the channel gracefully with a settlement transaction, making it unnecessary to transmit an intermediate commitment transaction. In essence, the commitment transactions are only used in the case one party disconnects and the other party has to close the channel unilaterally.
For example, commitment transaction 1 cannot be spent before 1000 blocks, then if commitment transaction 2 is timelocked to 995 blocks. Commitment transaction 600 can be spent 600 blocks earlier than commitment transaction 1.
For example, if commitment transaction #1 is timelocked to 1000 blocks in the future, then commitment transaction #2 is timelocked to 999 blocks. Commitment transaction #600 can be spent #600 blocks before commitment transaction #1 becomes valid.
Each subsequent commitment transaction must have a shorter timelock so that it may be broadcast before its predecessors and before the refund transaction. The ability to broadcast a commitment earlier ensures it will be able to spend the funding output and preclude any other commitment transaction from being redeemed by spending the output. The guarantees offered by the bitcoin blockchain, preventing doublespends and enforcing timelocks, effectively allow each commitment transaction to invalidate its predecessors.
Each subsequent commitment transaction must have a shorter timelock so that it may be broadcast before its predecessors and before the refund transaction. The ability to broadcast a commitment earlier ensures it will be able to spend the funding output and preclude any other commitment transaction from being redeemed by spending the output. The guarantees offered by the bitcoin blockchain, preventing double-spends and enforcing timelocks, effectively allow each commitment transaction to invalidate its predecessors.
State channels use timelocks to establish smart contracts across a time dimension. In this example we saw how the time dimension guarantees that a fair commitment, representing the correct channel balance can be transmitted and confirmed before an unfair commitment transaction, representing a channel's prior state, can be transmitted and confirmed. This implementation, needs nothing more than absolute transaction level timelocks (nLockTime). Next we will see how script level timelocks, CHECKLOCKTIMEVERIFY and CHECKSEQUENCEVERIFY can be used to construct more flexible, useful, and sophisticated state channels.
State channels use timelocks to enforce smart contracts across a time dimension. In this example we saw how the time dimension guarantees that the most recent commitment transaction becomes valid before any earlier commitments. Thus, the most recent commitment transaction can be transmitted, spending the inputs and invalidating prior commitment transactions. The enforcement of smart contracts with absolute timelocks protects against cheating by one of the parties. This implementation, needs nothing more than absolute transaction-level timelocks (+nLocktime+). Next we will see how script-level timelocks, +CHECKLOCKTIMEVERIFY+ and +CHECKSEQUENCEVERIFY+ can be used to construct more flexible, useful, and sophisticated state channels.
The first form of unidirectional payment channel was demonstrated as a prototype video streaming application in 2015 by an Argentinian team of developers. You can still see it at streamium.io.
Timelocks are not the only way to invalidate prior commitment transactions. In the next sections we will see how a revocation key can be used to achieve the same result. Timelocks are effective but they have two distinct disadvantages. By establishing a maximum timelock when the channel is first opened, they limit the lifetime of the channel. Worse they force channel implementations to strike a balance between allowing longlived channels and forcing one of the participants to wait a very long time for a refund in case of premature closure. For example, if you allow the channel to remain open for 30 days, by setting the refund timelock to 30 days, if one of the parties disappears immediately the other party must wait 30 days for a refund. The more distant the endpoint, the more distant the refund.
Timelocks are not the only way to invalidate prior commitment transactions. In the next sections we will see how a revocation key can be used to achieve the same result. Timelocks are effective but they have two distinct disadvantages. By establishing a maximum timelock when the channel is first opened, they limit the lifetime of the channel. Worse, they force channel implementations to strike a balance between allowing long-lived channels and forcing one of the participants to wait a very long time for a refund in case of premature closure. For example, if you allow the channel to remain open for 30 days, by setting the refund timelock to 30 days, if one of the parties disappears immediately the other party must wait 30 days for a refund. The more distant the endpoint, the more distant the refund.
The second problem is that since each subsequent commitment transaction must decrement the timelock, there is an explicit limit on the number of commitment transactions that can be exchanged between the parties. For example, a 30 day channel, setting a timelock of 4320 blocks into the future, can only accommodate 4320 intermediate commitment transactions before it must be closed. There is a danger in setting the timelock commitment transaction interval at 1 block. By setting the timelock interval between commitment transactions to 1 block, a developer is creating a very high burden for the channel participants who have to be vigilant, remain online and watching, and be ready to transmit the right commitment transaction at anytime.
Now that we understand how timelocks can be used to invalidate prior commitments, we can see the difference between closing the channel cooperatively and closing it unilaterally by broadcasting a commitment transaction. All commitment transactions are timelocked, therefore broadcasting a commitment transaction will always involve waiting until the timelock has expired. But if the two parties agree on what the final balance is and know they both hold commitment transactions that will make eventually make that balance a reality, they can construct a settlement transaction without a timelock representing that same balance. In a cooperative close, either party takes the most recent commitment transaction, and builds a settlement transaction that is identical in every way except it omits the timelock. Both parties can sign this settlement transaction knowing there is no way to cheat and get a more favorable balance. By cooperatively signing and transmitting the settlement transaction they can close the channel and redeem their balance immediately. Worst case, one of the parties can be petty, refuse to cooperate and force the other party to do a unilateral close with the most recent commitment transaction. But if they do that, they have to wait for their funds too.
Now that we understand how timelocks can be used to invalidate prior commitments, we can see the difference between closing the channel cooperatively and closing it unilaterally by broadcasting a commitment transaction. All commitment transactions are timelocked, therefore broadcasting a commitment transaction will always involve waiting until the timelock has expired. But if the two parties agree on what the final balance is and know they both hold commitment transactions that will eventually make that balance a reality, they can construct a settlement transaction without a timelock representing that same balance. In a cooperative close, either party takes the most recent commitment transaction, and builds a settlement transaction that is identical in every way except it omits the timelock. Both parties can sign this settlement transaction knowing there is no way to cheat and get a more favorable balance. By cooperatively signing and transmitting the settlement transaction they can close the channel and redeem their balance immediately. Worst case, one of the parties can be petty, refuse to cooperate, and force the other party to do a unilateral close with the most recent commitment transaction. But if they do that, they have to wait for their funds too.
==== Asymmetric Revocable Commitments
@ -303,9 +306,9 @@ The initial balance is 5 bitcoin for Hitesh and 5 bitcoin for Irene. The funding
The funding transaction may have one or more inputs from Hitesh (adding up to 5 bitcoin or more), and one or more inputs from Irene (adding up to 5 bitcoin or more). The inputs have to slightly exceed the channel capacity in order to cover the transaction fees. The transaction has one output that locks the 10 total bitcoin to a 2-of-2 multisig address controlled by both Hitesh and Irene. The funding transaction may also have one or more outputs returning change to Hitesh and Irene if their inputs exceeded their intended channel contribution. This is a single transaction with inputs offered and signed by two parties. It has to be constructed in collaboration and signed by each party before it is transmitted.
Now, instead of creating a single commitment transaction that both parties sign, Hitesh and Irene create two different commitment transactions that are *asymmetric*:
Now, instead of creating a single commitment transaction that both parties sign, Hitesh and Irene create two different commitment transactions that are _asymmetric_.
Hitesh has a commitment transaction with two outputs. The first output pays Irene the 5 bitcoin she is owed *immediately*. The second output pays Hitesh the 5 bitcoin he is owed, but only after a timelock of 1000 blocks. The transaction outputs look like this:
Hitesh has a commitment transaction with two outputs. The first output pays Irene the 5 bitcoin she is owed _immediately_. The second output pays Hitesh the 5 bitcoin he is owed, but only after a timelock of 1000 blocks. The transaction outputs look like this:
----
Input: 2-of-2 funding output, signed by Irene
@ -320,7 +323,7 @@ Output 1:
<Hitesh's Public Key> CHECKSIG
----
Irene has a different commitment transaction with two outputs. The first output pays Hitesh the 5 bitcoin he is owed immediately. The second output pays Irene the 5 bitcoin she is own but only after a timelock of 1000 blocks. The commitment transaction Irene holds (signed by Hitesh), looks like this:
Irene has a different commitment transaction with two outputs. The first output pays Hitesh the 5 bitcoin he is owed immediately. The second output pays Irene the 5 bitcoin she is owed but only after a timelock of 1000 blocks. The commitment transaction Irene holds (signed by Hitesh), looks like this:
----
Input: 2-of-2 funding output, signed by Hitesh
@ -335,11 +338,11 @@ Output 1:
<Irene's Public Key> CHECKSIG
----
This way, each party has a commitment transaction, spending the 2-of-2 funding output. This input is signed by the *other* party. At any time the party holding the transaction can also sign (completing the 2-of-2) and broadcast. However, if they broadcast the commitment transaction, it pays the other party immediately whereas they have to wait for a short timelock to expire. By imposing a delay on the redemption of one of the outputs, we put each party at a slight disadvantage when they choose to unilaterally broadcast a commitment transaction. But a time delay, alone, isn't enough to encourage fair conduct.
This way, each party has a commitment transaction, spending the 2-of-2 funding output. This input is signed by the _other_ party. At any time the party holding the transaction can also sign (completing the 2-of-2) and broadcast. However, if they broadcast the commitment transaction, it pays the other party immediately whereas they have to wait for a short timelock to expire. By imposing a delay on the redemption of one of the outputs, we put each party at a slight disadvantage when they choose to unilaterally broadcast a commitment transaction. But a time delay, alone, isn't enough to encourage fair conduct.
Now we introduce the final element of this scheme: a revocation key that allows a wronged party to punish a cheater by taking the entire balance of the channel.
Each of the commitment transactions has a "delayed" output. The redemption script for that output allows one party to redeem it after 1000 blocks *or* the other party to redeem it if they have a revocation key. So when Hitesh creates a commitment transaction for Irene to sign, he makes the second output payable to himself after 1000 blocks, or to whoever can present a revocation key. Hitesh constructs this transaction and creates a revocation key that he keeps secret. He will only reveal it to Irene when he is ready to move to a new channel state and wants to revoke this commitment. The second output's script looks like this:
Each of the commitment transactions has a "delayed" output. The redemption script for that output allows one party to redeem it after 1000 blocks _or_ the other party to redeem it if they have a revocation key. So when Hitesh creates a commitment transaction for Irene to sign, he makes the second output payable to himself after 1000 blocks, or to whoever can present a revocation key. Hitesh constructs this transaction and creates a revocation key that he keeps secret. He will only reveal it to Irene when he is ready to move to a new channel state and wants to revoke this commitment. The second output's script looks like this:
----
Output 0 <5 bitcoin>:
@ -360,9 +363,9 @@ CHECKSIG
Irene can confidently sign this transaction, since if transmitted it will immediately pay her what she is owed. Hitesh holds the transaction, but knows that if he transmits it in a unilateral channel closing, he will have to wait 1000 blocks to get paid.
When the channel is advanced to the next state, Hitesh has to _revoke_ this commitment transaction, before Irene agrees to sign the next commitment transaction. To do that, all he has to do is send the _revocation key_ to Irene. Once Irene has the revocation key for this commitment, she can sign the next commitment with confidence. She knows that if Hitesh tries to cheat by publishing the prior commitment, she can use the revocation key to redeem Hitesh's delayed output. *If Hitesh cheats, Irene gets BOTH outputs*.
When the channel is advanced to the next state, Hitesh has to _revoke_ this commitment transaction, before Irene agrees to sign the next commitment transaction. To do that, all he has to do is send the _revocation key_ to Irene. Once Irene has the revocation key for this commitment, she can sign the next commitment with confidence. She knows that if Hitesh tries to cheat by publishing the prior commitment, she can use the revocation key to redeem Hitesh's delayed output. _If Hitesh cheats, Irene gets BOTH outputs_.
The revocation protocol is bilateral, meaning that in each round, as the channel state is advanced, the two parties exchange new commitments, exchange revocation keys for the previous commitment and sign each other's commitment transactions. As they accept a new state, they make the prior state impossible to use, by giving each other the necessary revocation keys to punish any cheating.
The revocation protocol is bilateral, meaning that in each round, as the channel state is advanced, the two parties exchange new commitments, exchange revocation keys for the previous commitment, and sign each other's commitment transactions. As they accept a new state, they make the prior state impossible to use, by giving each other the necessary revocation keys to punish any cheating.
Let's look at an example of how it works. One of Irene's customers wants to send 2 bitcoin to one of Hitesh's customers. To transmit 2 bitcoin across the channel, Hitesh and Irene must advance the channel state to reflect the new balance. They will commit to a new state (state number 2) where the channel's 10 bitcoin are split 7 bitcoin to Hitesh and 3 bitcoin to Irene. To advance the state of the channel, they will each create new commitment transactions reflecting the new channel balance.
@ -370,11 +373,11 @@ As before, these commitment transactions are asymmetric so that the commitment t
Importantly, the revocation doesn't happen automatically. While Hitesh has the ability to punish Irene for cheating, he has to watch the blockchain diligently for signs of cheating. If he sees a prior commitment transaction broadcast, he has 1000 blocks to take action and use the revocation key to thwart Irene's cheating and punish her by taking the entire balance, all 10 bitcoin.
Asymmetric revocable commitments with relative time locks (CSV) are a much better way to implement payment channels and a very significant innovation in this technology. With this construct, the channel can remain open indefinitely and can have billions of intermediate commitment transactions. In prototype implementations of Lightning Network, the commitment state is identified by a 48-bit index, allowing more than 281 trillion (2.8 x 10^14^) state transitions in any single channel!
Asymmetric revocable commitments with relative time locks (+CSV+) are a much better way to implement payment channels and a very significant innovation in this technology. With this construct, the channel can remain open indefinitely and can have billions of intermediate commitment transactions. In prototype implementations of Lightning Network, the commitment state is identified by a 48-bit index, allowing more than 281 trillion (2.8 x 10^14^) state transitions in any single channel!
==== Hash Time Lock Contracts (HTLC)
Payment channels can be further extended with a special type of smart contract that allows the participants to commit funds to a redeemable secret, with an expiration time. This feature is called a _Hash Time Lock Contract_ or _HTLC_ and is used in both bi-directional and routed payment channels.
Payment channels can be further extended with a special type of smart contract that allows the participants to commit funds to a redeemable secret, with an expiration time. This feature is called a _Hash Time Lock Contract_ or _HTLC_ and is used in both bidirectional and routed payment channels.
Let's first explain the "hash" part of the HTLC. To create an HTLC, the intended recipient of the payment will first create a secret +R+. They then calculate the hash of this secret +H+:
@ -384,13 +387,7 @@ H = Hash(R)
This produces a hash +H+ that can be included in an output's locking script. Whoever knows the secret can use it to redeem the output. The secret +R+ is also referred to as a _pre-image_ to the hash function. Pre-image just means the data that is used as input to a hash function.
[TIP]
====
The hash function used....
====
The second part of an HTLC is the "time lock" component. If the secret is not revealed, the payer of the HTLC can get a "refund" after some time. This is achieved with an absolute time lock using CHECKLOCKTIMEVERIFY.
The second part of an HTLC is the "time lock" component. If the secret is not revealed, the payer of the HTLC can get a "refund" after some time. This is achieved with an absolute time lock using +CHECKLOCKTIMEVERIFY+.
The script implementing an HTLC might look like this:
@ -409,13 +406,14 @@ Anyone who knows the secret +R+, which when hashed equals to +H+, can redeem thi
If the secret is not revealed and the HTLC claimed, after a certain number of blocks the payee can claim a refund using the second clause in the +IF+ flow.
This is a basic implementation of an HTLC. This type of HTLC can be redeemed by *anyone* who has the secret R. An HTLC can take many different forms with slight variations to the script. For example, adding a CHECKSIG operator and a public key in the first clause restricts redemption of the hash to a named recipient, who must also know the secret R.
This is a basic implementation of an HTLC. This type of HTLC can be redeemed by _anyone_ who has the secret +R+. An HTLC can take many different forms with slight variations to the script. For example, adding a +CHECKSIG+ operator and a public key in the first clause restricts redemption of the hash to a named recipient, who must also know the secret +R+.
[[lightning_network]]
=== Routed Payment Channels (Lightning Network)
The Lightning Network is a proposed routed network of bi-directional payment channels connected end-to-end. A network like this can allow any participant to route a payment from channel to channel without trusting any of the intermediaries. The Lightning Network was first described by Joseph Poon and Thadeus Dryja in February 2015, building on the concept of payment channels as proposed and elaborated upon by many others:
The Lightning Network is a proposed routed network of bidirectional payment channels connected end-to-end. A network like this can allow any participant to route a payment from channel to channel without trusting any of the intermediaries. The Lightning Network was first described by Joseph Poon and Thadeus Dryja in February 2015, building on the concept of payment channels as proposed and elaborated upon by many others:
"Lightning Network" refers to a specific design for a routed payment channel network, which has now been implemented by at least five different open source teams. The independent implementations are coordinated by a set of interoperability standards described in the _Basics of Lightning Technology (BOLT)_ papers:
@ -429,23 +427,23 @@ The Lightning Network is one possible way of implementing routed payment channel
Let's see how this works.
In this example, we have 5 participants: Alice, Bob, Carol, Diana, and Eric. These 5 participants have opened payment channels with each other, in pairs. Alice has a payment channel with Bob. Bob is connected to Carol, Carol to Diana and Diana to Eric. For simplicity let's assume each channel is funded with 2 bitcoin by each participant, for a total capacity of 4 bitcoin in each channel.
In this example, we have five participants: Alice, Bob, Carol, Diana, and Eric. These five participants have opened payment channels with each other, in pairs. Alice has a payment channel with Bob. Bob is connected to Carol, Carol to Diana, and Diana to Eric. For simplicity let's assume each channel is funded with 2 bitcoin by each participant, for a total capacity of 4 bitcoin in each channel.
Alice wants to pay Eric 1 bitcoin. However, Alice is not connected to Eric by a payment channel. Creating a payment channel requires a funding transaction, which must be committed to the bitcoin blockchain. Alice does not want to open a new payment channel and commit more of her funds. Is there a way to pay Eric, indirectly?
Alice is running a Lightning Network (LN) node that is keeping track of her payment channel to Bob and has the ability to discover routes between payment channels. Alice's LN node also has the ability to connect over the Internet to Eric's LN node. Eric's LN node creates a secret R using a random number generator. Eric's node does not reveal this secret to anyone. Instead, Eric's node calculates a hash +H+ of the secret +R+ and transmits this hash to Alice's node.
Alice is running a Lightning Network (LN) node that is keeping track of her payment channel to Bob and has the ability to discover routes between payment channels. Alice's LN node also has the ability to connect over the internet to Eric's LN node. Eric's LN node creates a secret +R+ using a random number generator. Eric's node does not reveal this secret to anyone. Instead, Eric's node calculates a hash +H+ of the secret +R+ and transmits this hash to Alice's node.
Now Alice's LN node constructs a route between Alice's LN node and Eric's LN node. The routing algorithm used will be examined in more detail later, but for now let's assume that Alice's node can find an efficient route.
Alice's node then constructs an HTLC, payable to the hash +H+, with a 10 block refund timeout (current block + 10), for an amount of 1.003 bitcoin. The extra 0.003 will be used to compensate the intermediate nodes for their participation in this payment route. Alice offers this HTLC to Bob, deducting 1.003 bitcoin from her channel balance with Bob and committing it to the HTLC. The HTLC has the following meaning: _"Alice is committing 1.003 of her channel balance to be paid to Bob if Bob knows the secret, or refunded back to Alice's balance if 10 blocks elapse."_. The channel balance between Alice and Bob is now expressed by commitment transactions with 3 outputs: 2 bitcoin balance to Bob, 0.997 bitcoin balance to Alice, 1.003 bitcoin committed in Alice's HTLC. Alice's balance is reduced by the amount committed to the HTLC.
Alice's node then constructs an HTLC, payable to the hash +H+, with a 10 block refund timeout (current block + 10), for an amount of 1.003 bitcoin. The extra 0.003 will be used to compensate the intermediate nodes for their participation in this payment route. Alice offers this HTLC to Bob, deducting 1.003 bitcoin from her channel balance with Bob and committing it to the HTLC. The HTLC has the following meaning: _"Alice is committing 1.003 of her channel balance to be paid to Bob if Bob knows the secret, or refunded back to Alice's balance if 10 blocks elapse."_ The channel balance between Alice and Bob is now expressed by commitment transactions with three outputs: 2 bitcoin balance to Bob, 0.997 bitcoin balance to Alice, 1.003 bitcoin committed in Alice's HTLC. Alice's balance is reduced by the amount committed to the HTLC.
Bob now has a commitment that if he is able to get the secret +R+ within the next 10 blocks, he can claim the 1.003 locked by Alice. With this commitment in hand, Bob's node constructs an HTLC on his payment channel with Carol. Bob's HTLC commits 1.002 bitcoin to hash +H+ for 9 blocks, which Carol can redeem if she has secret +R+. Bob knows that if Carol can claim his HTLC, she has to produce +R+. If Bob has +R+ in 9 blocks, he can use it to claim Alice's HTLC to him. He also makes 0.001 bitcoin for committing his channel balance for 9 blocks. If Carol is unable to claim his HTLC and he is unable to claim Alice's HTLC, everything reverts back to the prior channel balances and no one is at a loss. The channel balance between Bob and Carol is now: 2 to Carol, 0.998 to Bob, 1.002 committed by Bob to the HTLC.
Bob now has a commitment that if he is able to get the secret +R+ within the next 10 blocks, he can claim the 1.003 locked by Alice. With this commitment in hand, Bob's node constructs an HTLC on his payment channel with Carol. Bob's HTLC commits 1.002 bitcoin to hash +H+ for 9 blocks, which Carol can redeem if she has secret +R+. Bob knows that if Carol can claim his HTLC, she has to produce +R+. If Bob has +R+ in nine blocks, he can use it to claim Alice's HTLC to him. He also makes 0.001 bitcoin for committing his channel balance for nine blocks. If Carol is unable to claim his HTLC and he is unable to claim Alice's HTLC, everything reverts back to the prior channel balances and no one is at a loss. The channel balance between Bob and Carol is now: 2 to Carol, 0.998 to Bob, 1.002 committed by Bob to the HTLC.
Carol now has a commitment that if she gets +R+ within the next 9 blocks, she can claim 1.002 bitcoin locked by Bob. Now she can make an HTLC commitment on her channel with Diana. She commits an HTLC of 1.001 bitcoin to hash +H+, for 8 blocks, which Diana can redeem if she has secret +R+. From Carol's perspective, if this works she is 0.001 bitcoin better off and if it doesn't she loses nothing. Her HTLC to Diana is only viable if +R+ is revealed, at which point she can claim the HTLC from Bob. The channel balance between Carol and Diana is now: 2 to Diana, 0.999 to Carol, 1.001 committed by Carol to the HTLC.
Carol now has a commitment that if she gets +R+ within the next nine blocks, she can claim 1.002 bitcoin locked by Bob. Now she can make an HTLC commitment on her channel with Diana. She commits an HTLC of 1.001 bitcoin to hash +H+, for eight blocks, which Diana can redeem if she has secret +R+. From Carol's perspective, if this works she is 0.001 bitcoin better off and if it doesn't she loses nothing. Her HTLC to Diana is only viable if +R+ is revealed, at which point she can claim the HTLC from Bob. The channel balance between Carol and Diana is now: 2 to Diana, 0.999 to Carol, 1.001 committed by Carol to the HTLC.
Finally, Diana can offer an HTLC to Eric, committing 1 bitcoin for 7 blocks to hash +H+. The channel balance between Diana and Eric is now: 2 to Eric, 1 to Diana, 1 committed by Diana to the HTLC.
Finally, Diana can offer an HTLC to Eric, committing 1 bitcoin for seven blocks to hash +H+. The channel balance between Diana and Eric is now: 2 to Eric, 1 to Diana, 1 committed by Diana to the HTLC.
However, at this hop in the route, Eric *has* secret +R+. He can therefore claim the HTLC offered by Diana. He sends +R+ to Diana and claims the 1 bitcoin, adding it to his channel balance. The channel balance is now: 1 to Diana, 3 to Eric.
However, at this hop in the route, Eric _has_ secret +R+. He can therefore claim the HTLC offered by Diana. He sends +R+ to Diana and claims the 1 bitcoin, adding it to his channel balance. The channel balance is now: 1 to Diana, 3 to Eric.
Now, Diana has secret +R+. Therefore, she can now claim the HTLC from Carol. Diana transmits +R+ to Carol and adds the 1.001 bitcoin to her channel balance. Now the channel balance between Carol and Diana is: 0.999 to Carol, 3.001 to Diana. Diana has "earned" 0.001 for participating in this payment route.
@ -453,17 +451,17 @@ Flowing back through the route, the secret +R+ allows each participant to claim
Alice has paid Eric 1 bitcoin without opening a channel to Eric. None of the intermediate parties in the payment route had to trust each other. For the short-term commitment of their funds in the channel they are able to earn a small fee, with the only risk being a small delay in refund if the channel was closed or the routed payment failed.
==== Lightning Network Transport & Routing
==== Lightning Network Transport and Routing
All communications between Lightning Network nodes are encrypted point-to-point. In addition, nodes have a long-term public key that they http://bitfury.com/content/5-white-papers-research/whitepaper_flare_an_approach_to_routing_in_lightning_network_7_7_2016.pdf[use as an identifier and to authenticate each other].
Whenever a node wishes to send a payment to another node, it must first construct a _path_ through the network by connecting payment channels with sufficient capacity. Nodes advertise routing information, including what channels they have open, how much capacity each channel has and what fees they charge to route payments. The routing information can be shared in a variety of ways and different routing protocols are likely to emerge as Lightning Network technology advances. Some Lightning Network implementations use the IRC protocol as a convenient mechanism for nodes to announce routing information. Another implementation of route discovery uses a P2P model where nodes propagate channel announcements to their peers, in a "flooding" model, similar to how bitcoin propagates transactions. Future plans include a proposal called http://bitfury.com/content/5-white-papers-research/whitepaper_flare_an_approach_to_routing_in_lightning_network_7_7_2016.pdf[Flare], which is a hybrid routing model with local node "neighborhoods" and longer-range beacon nodes.
Whenever a node wishes to send a payment to another node, it must first construct a _path_ through the network by connecting payment channels with sufficient capacity. Nodes advertise routing information, including what channels they have open, how much capacity each channel has, and what fees they charge to route payments. The routing information can be shared in a variety of ways and different routing protocols are likely to emerge as Lightning Network technology advances. Some Lightning Network implementations use the IRC protocol as a convenient mechanism for nodes to announce routing information. Another implementation of route discovery uses a P2P model where nodes propagate channel announcements to their peers, in a "flooding" model, similar to how bitcoin propagates transactions. Future plans include a proposal called http://bitfury.com/content/5-white-papers-research/whitepaper_flare_an_approach_to_routing_in_lightning_network_7_7_2016.pdf[Flare], which is a hybrid routing model with local node "neighborhoods" and longer-range beacon nodes.
In our previous example, Alice's node uses one of these route discovery mechanisms to find one or more paths connecting her node to Eric's node. Once Alice's node has constructed a path, she will initialize that path through the network, by propagating a series of encrypted and nested instructions to connect each of the adjacent payment channels.
Importantly, this path is only known to Alice's node. All other participants in the payment route see only the adjacent nodes. From Carol's perspective, this looks like a payment from Bob to Diana. Carol does not know that Bob is actually relaying a payment from Alice. She also doesn't know that Diana will be relaying a payment to Eric.
This is a critical feature of the Lightning Network, because it ensures privacy of payments and makes it very difficult to apply surveillance, censorship or blacklists. But how does Alice establish this payment path, without revealing anything to the intermediary nodes?
This is a critical feature of the Lightning Network, because it ensures privacy of payments and makes it very difficult to apply surveillance, censorship, or blacklists. But how does Alice establish this payment path, without revealing anything to the intermediary nodes?
The Lightning Network implements an _Onion Routed Protocol_ based on a scheme called https://cypherpunks.ca/~iang/pubs/Sphinx_Oakland09.pdf[Sphinx]. This routing protocol ensures that a payment sender can construct and communicate a path through the Lightning Network such that:
@ -475,19 +473,19 @@ The Lightning Network implements an _Onion Routed Protocol_ based on a scheme ca
* Each part of the path is encrypted in such a way that a network-level attacker cannot associate the packets from different parts of the path to each other
* Unlike Tor, (an onion routed anonymization protocol on the Internet), there are no "exit nodes" that can be placed under surveillance. The payments do not need to be transmitted to the bitcoin blockchain, the nodes just update channel balances.
* Unlike Tor (an onion routed anonymization protocol on the internet), there are no "exit nodes" that can be placed under surveillance. The payments do not need to be transmitted to the bitcoin blockchain; the nodes just update channel balances.
Using this onion routed protocol, Alice wraps each element of the path in a layer of encryption, starting with the end and working backwards. She encrypts a message to Eric with Eric's public key. This message is wrapped in a message encrypted to Diana, identifying Eric as the next recipient. The message to Diana is wrapped in a message encrypted to Carol's public key and identifying Diana as the next recipient. The message to Carol is encrypted to Bob's key. Thus, Alice has constructed this encrypted multi-layer "onion" of messages. She sends this to Bob, who can only decrypt and unwrap the outer layer. Inside, Bob finds a message addressed to Carol that he can forward to Carol but cannot decipher himself. Following the path, the messages get forwarded, decrypted, forwarded etc, all the way to Eric. Each participant knows only the previous and next node in each hop.
Using this onion routed protocol, Alice wraps each element of the path in a layer of encryption, starting with the end and working backwards. She encrypts a message to Eric with Eric's public key. This message is wrapped in a message encrypted to Diana, identifying Eric as the next recipient. The message to Diana is wrapped in a message encrypted to Carol's public key and identifying Diana as the next recipient. The message to Carol is encrypted to Bob's key. Thus, Alice has constructed this encrypted multilayer "onion" of messages. She sends this to Bob, who can only decrypt and unwrap the outer layer. Inside, Bob finds a message addressed to Carol that he can forward to Carol but cannot decipher himself. Following the path, the messages get forwarded, decrypted, forwarded, etc., all the way to Eric. Each participant knows only the previous and next node in each hop.
Each element of the path contains information on the HTLC that must be extended to the next hop, the amount that is being sent, the fee to include and the CLTV locktime (in blocks) expiration of the HTLC. As the route information propagates, the nodes make HTLC commitments forward to the next hop.
Each element of the path contains information on the HTLC that must be extended to the next hop, the amount that is being sent, the fee to include, and the CLTV locktime (in blocks) expiration of the HTLC. As the route information propagates, the nodes make HTLC commitments forward to the next hop.
At this point, you might be wondering how it is possible that the nodes do not know the length of the path and their position in that path. After all, they receive a message and forward it to the next hop. Doesn't it get shorter, allowing them to deduce the path size and their position? To prevent this, the path is always fixed at 20 hops and padded with random data. Each node sees the next hop and a fixed length encrypted message to forward. Only the final recipient sees that there is no next hop. To everyone else it seems as if there are always 20 more hops to go.
==== Lightning Network Benefits
A Lightning Network is a second-layer routing technology. It can be applied to any blockchain that supports some basic capabilities, such as multi-signature transactions, timelocks and basic smart contracts.
A Lightning Network is a second-layer routing technology. It can be applied to any blockchain that supports some basic capabilities, such as multisignature transactions, timelocks, and basic smart contracts.
If a Lightning Network is layered on top of the bitcoin network, the bitcoin network can gain a significant increase in capacity, privacy, granularity and speed, without sacrificing the principles of trustless operation without intermediaries.
If a Lightning Network is layered on top of the bitcoin network, the bitcoin network can gain a significant increase in capacity, privacy, granularity, and speed, without sacrificing the principles of trustless operation without intermediaries:
Privacy:: LN payments are much more private than payments on the bitcoin blockchain, as they are not public. While participants in a route can see payments propagated across their channels, they do not know the sender or recipient.
@ -501,7 +499,7 @@ Capacity:: LN increases the capacity of the bitcoin system by several orders of
Trustless Operation:: LN uses bitcoin transactions between nodes that operate as peers without trusting each other. Thus, LN preserves the principles of the bitcoin system, while expanding its operating parameters significantly.
Of course, as mentioned previously, the Lightning Network protocol is not the only way to implement routed payment channels. Other proposed systems include Tumblebit and Teechan. At this time, however, the Lightning Network has already been deployed on testnet. Several different teams have developed competing implementations of LN and are working towards a common interoperability standard (called BOLT). It is likely that Lightning Network will be the first routed payment channel network to be deployed in production.
Of course, as mentioned previously, the Lightning Network protocol is not the only way to implement routed payment channels. Other proposed systems include Tumblebit and Teechan. At this time, however, the Lightning Network has already been deployed on testnet. Several different teams have developed competing implementations of LN and are working toward a common interoperability standard (called BOLT). It is likely that Lightning Network will be the first routed payment channel network to be deployed in production.
Nick Adams
7 years ago