diff --git a/ch06.asciidoc b/ch06.asciidoc index f163ba8e..154c290a 100644 --- a/ch06.asciidoc +++ b/ch06.asciidoc @@ -5,16 +5,35 @@ [[ch06_intro]] === Introduction -((("transactions", "defined")))((("warnings and cautions", see="also security")))Transactions are the most important part of the Bitcoin system. Everything else in bitcoin is designed to ensure that transactions can be created, propagated on the network, validated, and finally added to the global ledger of transactions (the blockchain). Transactions are data structures that encode the transfer of value between participants in the Bitcoin system. Each transaction is a public entry in bitcoin's blockchain, the global double-entry bookkeeping ledger. - -In this chapter we will examine all the various forms of transactions, what they contain, how to create them, how they are verified, and how they become part of the permanent record of all transactions. When we use the term "wallet" in this chapter, we are referring to the software that constructs transactions, not just the database of keys. +((("transactions", "defined")))((("warnings and cautions", see="also +security")))Transactions are the most important part of the Bitcoin +system. Everything else in bitcoin is designed to ensure that +transactions can be created, propagated on the network, validated, and +finally added to the global ledger of transactions (the blockchain). +Transactions are data structures that encode the transfer of value +between participants in the Bitcoin system. Each transaction is a public +entry in bitcoin's blockchain, the global double-entry bookkeeping +ledger. + +In this chapter we will examine all the various forms of transactions, +what they contain, how to create them, how they are verified, and how +they become part of the permanent record of all transactions. When we +use the term "wallet" in this chapter, we are referring to the software +that constructs transactions, not just the database of keys. [[tx_structure]] === Transactions in Detail -((("use cases", "buying coffee", id="alicesix")))In <>, we looked at the transaction Alice used to pay for coffee at Bob's coffee shop using a block explorer (<>). +((("use cases", "buying coffee", id="alicesix")))In +<>, we looked at the transaction Alice used to +pay for coffee at Bob's coffee shop using a block explorer +(<>). -The block explorer application shows a transaction from Alice's "address" to Bob's "address." This is a much simplified view of what is contained in a transaction. In fact, as we will see in this chapter, much of the information shown is constructed by the block explorer and is not actually in the transaction. +The block explorer application shows a transaction from Alice's +"address" to Bob's "address." This is a much simplified view of what is +contained in a transaction. In fact, as we will see in this chapter, +much of the information shown is constructed by the block explorer and +is not actually in the transaction. [[alices_transactions_to_bobs_cafe]] .Alice's transaction to Bob's Cafe @@ -23,9 +42,15 @@ image::images/mbc2_0208.png["Alice Coffee Transaction"] [[transactions_behind_the_scenes]] ==== Transactions—Behind the Scenes -((("transactions", "behind the scenes details of")))Behind the scenes, an actual transaction looks very different from a transaction provided by a typical block explorer. In fact, most of the high-level constructs we see in the various bitcoin application user interfaces _do not actually exist_ in the Bitcoin system. +((("transactions", "behind the scenes details of")))Behind the scenes, +an actual transaction looks very different from a transaction provided +by a typical block explorer. In fact, most of the high-level constructs +we see in the various bitcoin application user interfaces _do not +actually exist_ in the Bitcoin system. -We can use Bitcoin Core's command-line interface (+getrawtransaction+ and +decoderawtransaction+) to retrieve Alice's "raw" transaction, decode it, and see what it contains. The result looks like this: +We can use Bitcoin Core's command-line interface (+getrawtransaction+ +and +decoderawtransaction+) to retrieve Alice's "raw" transaction, +decode it, and see what it contains. The result looks like this: [[alice_tx]] .Alice's transaction decoded @@ -55,53 +80,148 @@ We can use Bitcoin Core's command-line interface (+getrawtransaction+ and +decod } ---- -You may notice a few things about this transaction, mostly the things that are missing! Where is Alice's address? Where is Bob's address? Where is the 0.1 input "sent" by Alice? In bitcoin, there are no coins, no senders, no recipients, no balances, no accounts, and no addresses. All those things are constructed at a higher level for the benefit of the user, to make things easier to understand. +You may notice a few things about this transaction, mostly the things +that are missing! Where is Alice's address? Where is Bob's address? +Where is the 0.1 input "sent" by Alice? In bitcoin, there are no coins, +no senders, no recipients, no balances, no accounts, and no addresses. +All those things are constructed at a higher level for the benefit of +the user, to make things easier to understand. -You may also notice a lot of strange and indecipherable fields and hexadecimal strings. Don't worry, we will explain each field shown here in detail in this chapter. +You may also notice a lot of strange and indecipherable fields and +hexadecimal strings. Don't worry, we will explain each field shown here +in detail in this chapter. [[tx_inputs_outputs]] === Transaction Outputs and Inputs -((("transactions", "outputs and inputs", id="Tout06")))((("outputs and inputs", "outputs defined")))((("unspent transaction outputs (UTXO)")))((("UTXO sets")))((("transactions", "outputs and inputs", "output characteristics")))((("outputs and inputs", "output characteristics")))The fundamental building block of a bitcoin transaction is a _transaction output_. Transaction outputs are indivisible chunks of bitcoin currency, recorded on the blockchain, and recognized as valid by the entire network. Bitcoin full nodes track all available and spendable outputs, known as _unspent transaction outputs_, or _UTXO_. The collection of all UTXO is known as the _UTXO set_ and currently numbers in the millions of UTXO. The UTXO set grows as new UTXO is created and shrinks when UTXO is consumed. Every transaction represents a change (state transition) in the UTXO set. - -((("balances")))When we say that a user's wallet has "received" bitcoin, what we mean is that the wallet has detected an UTXO that can be spent with one of the keys controlled by that wallet. Thus, a user's bitcoin "balance" is the sum of all UTXO that user's wallet can spend and which may be scattered among hundreds of transactions and hundreds of blocks. The concept of a balance is created by the wallet application. The wallet calculates the user's balance by scanning the blockchain and aggregating the value of any UTXO the wallet can spend with the keys it controls. Most wallets maintain a database or use a database service to store a quick reference set of all the UTXO they can spend with the keys they control. - -((("satoshis")))A transaction output can have an arbitrary (integer) value denominated as a multiple of satoshis. Just as dollars can be divided down to two decimal places as cents, bitcoin can be divided down to eight decimal places as satoshis. Although an output can have any arbitrary value, once created it is indivisible. This is an important characteristic of outputs that needs to be emphasized: outputs are _discrete_ and _indivisible_ units of value, denominated in integer satoshis. An unspent output can only be consumed in its entirety by a transaction. - -((("change, making")))If an UTXO is larger than the desired value of a transaction, it must still be consumed in its entirety and change must be generated in the transaction. In other words, if you have an UTXO worth 20 bitcoin and want to pay only 1 bitcoin, your transaction must consume the entire 20-bitcoin UTXO and produce two outputs: one paying 1 bitcoin to your desired recipient and another paying 19 bitcoin in change back to your wallet. As a result of the indivisible nature of transaction outputs, most bitcoin transactions will have to generate change. - -Imagine a shopper buying a $1.50 beverage, reaching into her wallet and trying to find a combination of coins and bank notes to cover the $1.50 cost. The shopper will choose exact change if available e.g. a dollar bill and two quarters (a quarter is $0.25), or a combination of smaller denominations (six quarters), or if necessary, a larger unit such as a $5 note. If she hands too much money, say $5, to the shop owner, she will expect $3.50 change, which she will return to her wallet and have available for future transactions. - -Similarly, a bitcoin transaction must be created from a user's UTXO in whatever denominations that user has available. Users cannot cut an UTXO in half any more than they can cut a dollar bill in half and use it as currency. The user's wallet application will typically select from the user's available UTXO to compose an amount greater than or equal to the desired transaction amount. - -As with real life, the bitcoin application can use several strategies to satisfy the purchase amount: combining several smaller units, finding exact change, or using a single unit larger than the transaction value and making change. All of this complex assembly of spendable UTXO is done by the user's wallet automatically and is invisible to users. It is only relevant if you are programmatically constructing raw transactions from UTXO. - -A transaction consumes previously recorded unspent transaction outputs and creates new transaction outputs that can be consumed by a future transaction. This way, chunks of bitcoin value move forward from owner to owner in a chain of transactions consuming and creating UTXO. - -((("transactions", "coinbase transactions")))((("coinbase transactions")))((("mining and consensus", "coinbase transactions")))The exception to the output and input chain is a special type of transaction called the _coinbase_ transaction, which is the first transaction in each block. This transaction is placed there by the "winning" miner and creates brand-new bitcoin payable to that miner as a reward for mining. This special coinbase transaction does not consume UTXO; instead, it has a special type of input called the "coinbase." This is how bitcoin's money supply is created during the mining process, as we will see in <>. +((("transactions", "outputs and inputs", id="Tout06")))((("outputs and +inputs", "outputs defined")))((("unspent transaction outputs +(UTXO)")))((("UTXO sets")))((("transactions", "outputs and inputs", +"output characteristics")))((("outputs and inputs", "output +characteristics")))The fundamental building block of a bitcoin +transaction is a _transaction output_. Transaction outputs are +indivisible chunks of bitcoin currency, recorded on the blockchain, and +recognized as valid by the entire network. Bitcoin full nodes track all +available and spendable outputs, known as _unspent transaction outputs_, +or _UTXO_. The collection of all UTXO is known as the _UTXO set_ and +currently numbers in the millions of UTXO. The UTXO set grows as new +UTXO is created and shrinks when UTXO is consumed. Every transaction +represents a change (state transition) in the UTXO set. + +((("balances")))When we say that a user's wallet has "received" bitcoin, +what we mean is that the wallet has detected an UTXO that can be spent +with one of the keys controlled by that wallet. Thus, a user's bitcoin +"balance" is the sum of all UTXO that user's wallet can spend and which +may be scattered among hundreds of transactions and hundreds of blocks. +The concept of a balance is created by the wallet application. The +wallet calculates the user's balance by scanning the blockchain and +aggregating the value of any UTXO the wallet can spend with the keys it +controls. Most wallets maintain a database or use a database service to +store a quick reference set of all the UTXO they can spend with the keys +they control. + +((("satoshis")))A transaction output can have an arbitrary (integer) +value denominated as a multiple of satoshis. Just as dollars can be +divided down to two decimal places as cents, bitcoin can be divided down +to eight decimal places as satoshis. Although an output can have any +arbitrary value, once created it is indivisible. This is an important +characteristic of outputs that needs to be emphasized: outputs are +_discrete_ and _indivisible_ units of value, denominated in integer +satoshis. An unspent output can only be consumed in its entirety by a +transaction. + +((("change, making")))If an UTXO is larger than the desired value of a +transaction, it must still be consumed in its entirety and change must +be generated in the transaction. In other words, if you have an UTXO +worth 20 bitcoin and want to pay only 1 bitcoin, your transaction must +consume the entire 20-bitcoin UTXO and produce two outputs: one paying 1 +bitcoin to your desired recipient and another paying 19 bitcoin in +change back to your wallet. As a result of the indivisible nature of +transaction outputs, most bitcoin transactions will have to generate +change. + +Imagine a shopper buying a $1.50 beverage, reaching into her wallet and +trying to find a combination of coins and bank notes to cover the $1.50 +cost. The shopper will choose exact change if available e.g. a dollar +bill and two quarters (a quarter is $0.25), or a combination of smaller +denominations (six quarters), or if necessary, a larger unit such as a +$5 note. If she hands too much money, say $5, to the shop owner, she +will expect $3.50 change, which she will return to her wallet and have +available for future transactions. + +Similarly, a bitcoin transaction must be created from a user's UTXO in +whatever denominations that user has available. Users cannot cut an UTXO +in half any more than they can cut a dollar bill in half and use it as +currency. The user's wallet application will typically select from the +user's available UTXO to compose an amount greater than or equal to the +desired transaction amount. + +As with real life, the bitcoin application can use several strategies to +satisfy the purchase amount: combining several smaller units, finding +exact change, or using a single unit larger than the transaction value +and making change. All of this complex assembly of spendable UTXO is +done by the user's wallet automatically and is invisible to users. It is +only relevant if you are programmatically constructing raw transactions +from UTXO. + +A transaction consumes previously recorded unspent transaction outputs +and creates new transaction outputs that can be consumed by a future +transaction. This way, chunks of bitcoin value move forward from owner +to owner in a chain of transactions consuming and creating UTXO. + +((("transactions", "coinbase transactions")))((("coinbase +transactions")))((("mining and consensus", "coinbase transactions")))The +exception to the output and input chain is a special type of transaction +called the _coinbase_ transaction, which is the first transaction in +each block. This transaction is placed there by the "winning" miner and +creates brand-new bitcoin payable to that miner as a reward for mining. +This special coinbase transaction does not consume UTXO; instead, it has +a special type of input called the "coinbase." This is how bitcoin's +money supply is created during the mining process, as we will see in +<>. [TIP] ==== -What comes first? Inputs or outputs, the chicken or the egg? Strictly speaking, outputs come first because coinbase transactions, which generate new bitcoin, have no inputs and create outputs from nothing. +What comes first? Inputs or outputs, the chicken or the egg? Strictly +speaking, outputs come first because coinbase transactions, which +generate new bitcoin, have no inputs and create outputs from nothing. ==== [[tx_outs]] ==== Transaction Outputs -((("transactions", "outputs and inputs", "output components")))((("outputs and inputs", "output parts")))Every bitcoin transaction creates outputs, which are recorded on the bitcoin ledger. Almost all of these outputs, with one exception (see <>) create spendable chunks of bitcoin called UTXO, which are then recognized by the whole network and available for the owner to spend in a future transaction. +((("transactions", "outputs and inputs", "output +components")))((("outputs and inputs", "output parts")))Every bitcoin +transaction creates outputs, which are recorded on the bitcoin ledger. +Almost all of these outputs, with one exception (see <>) +create spendable chunks of bitcoin called UTXO, which are then +recognized by the whole network and available for the owner to spend in +a future transaction. -UTXO are tracked by every full-node Bitcoin client in the UTXO set. New transactions consume (spend) one or more of these outputs from the UTXO set. +UTXO are tracked by every full-node Bitcoin client in the UTXO set. New +transactions consume (spend) one or more of these outputs from the UTXO +set. Transaction outputs consist of two parts: -* An amount of bitcoin, denominated in _satoshis_, the smallest bitcoin unit -* A cryptographic puzzle that determines the conditions required to spend the output +- An amount of bitcoin, denominated in _satoshis_, the smallest bitcoin + unit -((("locking scripts")))((("scripting", "locking scripts")))((("witnesses")))((("scriptPubKey")))The cryptographic puzzle is also known as a _locking script_, a _witness script_, or a +scriptPubKey+. +- A cryptographic puzzle that determines the conditions required to + spend the output -The transaction scripting language, used in the locking script mentioned previously, is discussed in detail in <>. +((("locking scripts")))((("scripting", "locking +scripts")))((("witnesses")))((("scriptPubKey")))The cryptographic puzzle +is also known as a _locking script_, a _witness script_, or a ++scriptPubKey+. -Now, let's look at Alice's transaction (shown previously in <>) and see if we can identify the outputs. In the JSON encoding, the outputs are in an array (list) named +vout+: +The transaction scripting language, used in the locking script mentioned +previously, is discussed in detail in <>. + +Now, let's look at Alice's transaction (shown previously in +<>) and see if we can identify the +outputs. In the JSON encoding, the outputs are in an array (list) named ++vout+: [source,json] ---- @@ -118,13 +238,31 @@ Now, let's look at Alice's transaction (shown previously in <>. The scripting language that is used for the script in +scriptPubKey+ is discussed in <>. But before we delve into those topics, we need to understand the overall structure of transaction inputs and outputs. +The topic of locking and unlocking UTXO will be discussed later, in +<>. The scripting language that is used for the script +in +scriptPubKey+ is discussed in <>. But before we delve +into those topics, we need to understand the overall structure of +transaction inputs and outputs. ===== Transaction serialization—outputs -((("transactions", "outputs and inputs", "structure of")))((("outputs and inputs", "structure of")))((("serialization", "outputs")))When transactions are transmitted over the network or exchanged between applications, they are _serialized_. Serialization is the process of converting the internal representation of a data structure into a format that can be transmitted one byte at a time, also known as a byte stream. Serialization is most commonly used for encoding data structures for transmission over a network or for storage in a file. The serialization format of a transaction output is shown in <>. +((("transactions", "outputs and inputs", "structure of")))((("outputs +and inputs", "structure of")))((("serialization", "outputs")))When +transactions are transmitted over the network or exchanged between +applications, they are _serialized_. Serialization is the process of +converting the internal representation of a data structure into a format +that can be transmitted one byte at a time, also known as a byte stream. +Serialization is most commonly used for encoding data structures for +transmission over a network or for storage in a file. The serialization +format of a transaction output is shown in <>. [[tx_out_structure]] .Transaction output serialization @@ -136,11 +274,25 @@ The topic of locking and unlocking UTXO will be discussed later, in <> to help you: +Most bitcoin libraries and frameworks do not store transactions +internally as byte-streams, as that would require complex parsing every +time you needed to access a single field. For convenience and +readability, bitcoin libraries store transactions internally in data +structures (usually object-oriented structures). + +((("deserialization")))((("parsing")))((("transactions", "parsing")))The +process of converting from the byte-stream representation of a +transaction to a library's internal representation data structure is +called _deserialization_ or _transaction parsing_. The process of +converting back to a byte-stream for transmission over the network, for +hashing, or for storage on disk is called _serialization_. Most bitcoin +libraries have built-in functions for transaction serialization and +deserialization. + +See if you can manually decode Alice's transaction from the serialized +hexadecimal form, finding some of the elements we saw previously. The +section containing the two outputs is highlighted in <> to +help you: [[example_6_1]] .Alice's transaction, serialized and presented in hexadecimal notation @@ -159,21 +311,45 @@ See if you can manually decode Alice's transaction from the serialized hexadecim Here are some hints: -* There are two outputs in the highlighted section, each serialized as shown in <>. -* The value of 0.015 bitcoin is 1,500,000 satoshis. That's +16 e3 60+ in hexadecimal. -* In the serialized transaction, the value +16 e3 60+ is encoded in little-endian (least-significant-byte-first) byte order, so it looks like +60 e3 16+. -* The +scriptPubKey+ length is 25 bytes, which is +19+ in hexadecimal. +- There are two outputs in the highlighted section, each serialized as + shown in <>. -[[tx_inputs]] -==== Transaction Inputs +- The value of 0.015 bitcoin is 1,500,000 satoshis. That's +16 e3 60+ in + hexadecimal. -((("transactions", "outputs and inputs", "input components")))((("outputs and inputs", "input components")))((("unspent transaction outputs (UTXO)")))((("UTXO sets")))Transaction inputs identify (by reference) which UTXO will be consumed and provide proof of ownership through an unlocking script. +- In the serialized transaction, the value +16 e3 60+ is encoded in + little-endian (least-significant-byte-first) byte order, so it looks + like +60 e3 16+. -To build a transaction, a wallet selects from the UTXO it controls, UTXO with enough value to make the requested payment. Sometimes one UTXO is enough, other times more than one is needed. For each UTXO that will be consumed to make this payment, the wallet creates one input pointing to the UTXO and unlocks it with an unlocking script. +- The +scriptPubKey+ length is 25 bytes, which is +19+ in hexadecimal. -Let's look at the components of an input in greater detail. The first part of an input is a pointer to an UTXO by reference to the transaction hash and an output index, which identifies the specific UTXO in that transaction. The second part is an unlocking script, which the wallet constructs in order to satisfy the spending conditions set in the UTXO. Most often, the unlocking script is a digital signature and public key proving ownership of the bitcoin. However, not all unlocking scripts contain signatures. The third part is a sequence number, which will be discussed later. +[[tx_inputs]] +==== Transaction Inputs -Consider our example in <>. The transaction inputs are an array (list) called +vin+: +((("transactions", "outputs and inputs", "input +components")))((("outputs and inputs", "input components")))((("unspent +transaction outputs (UTXO)")))((("UTXO sets")))Transaction inputs +identify (by reference) which UTXO will be consumed and provide proof of +ownership through an unlocking script. + +To build a transaction, a wallet selects from the UTXO it controls, UTXO +with enough value to make the requested payment. Sometimes one UTXO is +enough, other times more than one is needed. For each UTXO that will be +consumed to make this payment, the wallet creates one input pointing to +the UTXO and unlocks it with an unlocking script. + +Let's look at the components of an input in greater detail. The first +part of an input is a pointer to an UTXO by reference to the transaction +hash and an output index, which identifies the specific UTXO in that +transaction. The second part is an unlocking script, which the wallet +constructs in order to satisfy the spending conditions set in the UTXO. +Most often, the unlocking script is a digital signature and public key +proving ownership of the bitcoin. However, not all unlocking scripts +contain signatures. The third part is a sequence number, which will be +discussed later. + +Consider our example in <>. The +transaction inputs are an array (list) called +vin+: [[vin]] .The transaction inputs in Alice's transaction @@ -189,12 +365,20 @@ Consider our example in <>. The transaction inpu ] ---- -As you can see, there is only one input in the list (because one UTXO contained sufficient value to make this payment). The input contains four elements: +As you can see, there is only one input in the list (because one UTXO +contained sufficient value to make this payment). The input contains +four elements: + +- A ((("transaction IDs (txd)")))transaction ID, referencing the + transaction that contains the UTXO being spent + +- An output index (+vout+), identifying which UTXO from that transaction + is referenced (first one is zero) -* A ((("transaction IDs (txd)")))transaction ID, referencing the transaction that contains the UTXO being spent -* An output index (+vout+), identifying which UTXO from that transaction is referenced (first one is zero) -* A +scriptSig+, which satisfies the conditions placed on the UTXO, unlocking it for spending -* A sequence number (to be discussed later) +- A +scriptSig+, which satisfies the conditions placed on the UTXO, + unlocking it for spending + +- A sequence number (to be discussed later) In Alice's transaction, the input points to the transaction ID: @@ -202,15 +386,44 @@ In Alice's transaction, the input points to the transaction ID: 7957a35fe64f80d234d76d83a2a8f1a0d8149a41d81de548f0a65a8a999f6f18 ---- -and output index +0+ (i.e., the first UTXO created by that transaction). The unlocking script is constructed by Alice's wallet by first retrieving the referenced UTXO, examining its locking script, and then using it to build the necessary unlocking script to satisfy it. - -Looking just at the input you may have noticed that we don't know anything about this UTXO, other than a reference to the transaction containing it. We don't know its value (amount in satoshi), and we don't know the locking script that sets the conditions for spending it. To find this information, we must retrieve the referenced UTXO by retrieving the underlying transaction. Notice that because the value of the input is not explicitly stated, we must also use the referenced UTXO in order to calculate the fees that will be paid in this transaction (see <>). - -It's not just Alice's wallet that needs to retrieve UTXO referenced in the inputs. Once this transaction is broadcast to the network, every validating node will also need to retrieve the UTXO referenced in the transaction inputs in order to validate the transaction. - -Transactions on their own seem incomplete because they lack context. They reference UTXO in their inputs but without retrieving that UTXO we cannot know the value of the inputs or their locking conditions. When writing bitcoin software, anytime you decode a transaction with the intent of validating it or counting the fees or checking the unlocking script, your code will first have to retrieve the referenced UTXO from the blockchain in order to build the context implied but not present in the UTXO references of the inputs. For example, to calculate the amount paid in fees, you must know the sum of the values of inputs and outputs. But without retrieving the UTXO referenced in the inputs, you do not know their value. So a seemingly simple operation like counting fees in a single transaction in fact involves multiple steps and data from multiple transactions. - -We can use the same sequence of commands with Bitcoin Core as we used when retrieving Alice's transaction (+getrawtransaction+ and +decoderawtransaction+). With that we can get the UTXO referenced in the preceding input and take a look: +and output index +0+ (i.e., the first UTXO created by that transaction). +The unlocking script is constructed by Alice's wallet by first +retrieving the referenced UTXO, examining its locking script, and then +using it to build the necessary unlocking script to satisfy it. + +Looking just at the input you may have noticed that we don't know +anything about this UTXO, other than a reference to the transaction +containing it. We don't know its value (amount in satoshi), and we don't +know the locking script that sets the conditions for spending it. To +find this information, we must retrieve the referenced UTXO by +retrieving the underlying transaction. Notice that because the value of +the input is not explicitly stated, we must also use the referenced UTXO +in order to calculate the fees that will be paid in this transaction +(see <>). + +It's not just Alice's wallet that needs to retrieve UTXO referenced in +the inputs. Once this transaction is broadcast to the network, every +validating node will also need to retrieve the UTXO referenced in the +transaction inputs in order to validate the transaction. + +Transactions on their own seem incomplete because they lack context. +They reference UTXO in their inputs but without retrieving that UTXO we +cannot know the value of the inputs or their locking conditions. When +writing bitcoin software, anytime you decode a transaction with the +intent of validating it or counting the fees or checking the unlocking +script, your code will first have to retrieve the referenced UTXO from +the blockchain in order to build the context implied but not present in +the UTXO references of the inputs. For example, to calculate the amount +paid in fees, you must know the sum of the values of inputs and outputs. +But without retrieving the UTXO referenced in the inputs, you do not +know their value. So a seemingly simple operation like counting fees in +a single transaction in fact involves multiple steps and data from +multiple transactions. + +We can use the same sequence of commands with Bitcoin Core as we used +when retrieving Alice's transaction (+getrawtransaction+ and ++decoderawtransaction+). With that we can get the UTXO referenced in the +preceding input and take a look: [[alice_input_tx]] .Alice's UTXO from the previous transaction, referenced in the input @@ -224,16 +437,24 @@ We can use the same sequence of commands with Bitcoin Core as we used when retri ] ---- -We see that this UTXO has a value of 0.1 BTC and that it has a locking script (+scriptPubKey+) that contains "OP_DUP OP_HASH160...". +We see that this UTXO has a value of 0.1 BTC and that it has a locking +script (+scriptPubKey+) that contains "OP_DUP OP_HASH160...". [TIP] ==== -To fully understand Alice's transaction we had to retrieve the previous transaction(s) referenced as inputs. A function that retrieves previous transactions and unspent transaction outputs is very common and exists in almost every bitcoin library and API. +To fully understand Alice's transaction we had to retrieve the previous +transaction(s) referenced as inputs. A function that retrieves previous +transactions and unspent transaction outputs is very common and exists +in almost every bitcoin library and API. ==== ===== Transaction serialization—inputs -((("serialization", "inputs")))((("transactions", "outputs and inputs", "input serialization")))((("outputs and inputs", "input serialization")))When transactions are serialized for transmission on the network, their inputs are encoded into a byte stream as shown in <>. +((("serialization", "inputs")))((("transactions", "outputs and inputs", +"input serialization")))((("outputs and inputs", "input +serialization")))When transactions are serialized for transmission on +the network, their inputs are encoded into a byte stream as shown in +<>. [[tx_in_structure]] .Transaction input serialization @@ -247,7 +468,8 @@ To fully understand Alice's transaction we had to retrieve the previous transact | 4 bytes | Sequence Number | Used for locktime or disabled (0xFFFFFFFF) |======= -As with the outputs, let's see if we can find the inputs from Alice's transaction in the serialized format. First, the inputs decoded: +As with the outputs, let's see if we can find the inputs from Alice's +transaction in the serialized format. First, the inputs decoded: [source,json] ---- @@ -261,7 +483,8 @@ As with the outputs, let's see if we can find the inputs from Alice's transactio ], ---- -Now, let's see if we can identify these fields in the serialized hex encoding in <>: +Now, let's see if we can identify these fields in the serialized hex +encoding in <>: [[example_6_2]] .Alice's transaction, serialized and presented in hexadecimal notation @@ -280,40 +503,124 @@ Now, let's see if we can identify these fields in the serialized hex encoding in Hints: -* The transaction ID is serialized in reversed byte order, so it starts with (hex) +18+ and ends with +79+ -* The output index is a 4-byte group of zeros, easy to identify -* The length of the +scriptSig+ is 139 bytes, or +8b+ in hex -* The sequence number is set to +FFFFFFFF+, again easy to identify((("", startref="alicesix"))) - -[[tx_fees]] -==== Transaction Fees - -((("transactions", "outputs and inputs", "transaction fees")))((("fees", "transaction fees")))((("mining and consensus", "rewards and fees")))Most transactions include transaction fees, which compensate the bitcoin miners for securing the network. Fees also serve as a security mechanism themselves, by making it economically infeasible for attackers to flood the network with transactions. Mining and the fees and rewards collected by miners are discussed in more detail in <>. +- The transaction ID is serialized in reversed byte order, so it starts + with (hex) +18+ and ends with +79+ -This section examines how transaction fees are included in a typical transaction. Most wallets calculate and include transaction fees automatically. However, if you are constructing transactions programmatically, or using a command-line interface, you must manually account for and include these fees. +- The output index is a 4-byte group of zeros, easy to identify -Transaction fees serve as an incentive to include (mine) a transaction into the next block and also as a disincentive against abuse of the system by imposing a small cost on every transaction. Transaction fees are collected by the miner who mines the block that records the transaction on the blockchain. +- The length of the +scriptSig+ is 139 bytes, or +8b+ in hex -Transaction fees are calculated based on the size of the transaction in kilobytes, not the value of the transaction in bitcoin. Overall, transaction fees are set based on market forces within the Bitcoin network. Miners prioritize transactions based on many different criteria, including fees, and might even process transactions for free under certain circumstances. Transaction fees affect the processing priority, meaning that a transaction with sufficient fees is likely to be included in the next block mined, whereas a transaction with insufficient or no fees might be delayed, processed on a best-effort basis after a few blocks, or not processed at all. Transaction fees are not mandatory, and transactions without fees might be processed eventually; however, including transaction fees encourages priority processing. +- The sequence number is set to +FFFFFFFF+, again easy to identify((("", + startref="alicesix"))) -Over time, the way transaction fees are calculated and the effect they have on transaction prioritization has evolved. At first, transaction fees were fixed and constant across the network. Gradually, the fee structure relaxed and may be influenced by market forces, based on network capacity and transaction volume. Since at least the beginning of 2016, capacity limits in bitcoin have created competition between transactions, resulting in higher fees and effectively making free transactions a thing of the past. Zero fee or very low fee transactions rarely get mined and sometimes will not even be propagated across the network. - -((("fees", "fee relay policies")))((("minrelaytxfee option")))In Bitcoin Core, fee relay policies are set by the +minrelaytxfee+ option. The current default +minrelaytxfee+ is 0.00001 bitcoin or a hundredth of a millibitcoin per kilobyte. Therefore, by default, transactions with a fee less than 0.00001 bitcoin are treated as free and are only relayed if there is space in the mempool; otherwise, they are dropped. Bitcoin nodes can override the default fee relay policy by adjusting the value of +minrelaytxfee+. - -((("dynamic fees")))((("fees", "dynamic fees")))Any bitcoin service that creates transactions, including wallets, exchanges, retail applications, etc., _must_ implement dynamic fees. Dynamic fees can be implemented through a third-party fee estimation service or with a built-in fee estimation algorithm. If you're unsure, begin with a third-party service and as you gain experience design and implement your own algorithm if you wish to remove the third-party dependency. - -Fee estimation algorithms calculate the appropriate fee, based on capacity and the fees offered by "competing" transactions. These algorithms range from simplistic (average or median fee in the last block) to sophisticated (statistical analysis). They estimate the necessary fee (in satoshis per byte) that will give a transaction a high probability of being selected and included within a certain number of blocks. Most services offer users the option of choosing high, medium, or low priority fees. High priority means users pay higher fees but the transaction is likely to be included in the next block. Medium and low priority means users pay lower transaction fees but the transactions may take much longer to confirm. +[[tx_fees]] +==== Transaction Fees -((("bitcoinfees (third-party service)")))Many wallet applications use third-party services for fee calculations. One popular service is http://bitcoinfees.21.co/[_http://bitcoinfees.21.co_], which provides an API and a visual chart showing the fee in satoshi/byte for different priorities. +((("transactions", "outputs and inputs", "transaction fees")))((("fees", +"transaction fees")))((("mining and consensus", "rewards and +fees")))Most transactions include transaction fees, which compensate the +bitcoin miners for securing the network. Fees also serve as a security +mechanism themselves, by making it economically infeasible for attackers +to flood the network with transactions. Mining and the fees and rewards +collected by miners are discussed in more detail in <>. + +This section examines how transaction fees are included in a typical +transaction. Most wallets calculate and include transaction fees +automatically. However, if you are constructing transactions +programmatically, or using a command-line interface, you must manually +account for and include these fees. + +Transaction fees serve as an incentive to include (mine) a transaction +into the next block and also as a disincentive against abuse of the +system by imposing a small cost on every transaction. Transaction fees +are collected by the miner who mines the block that records the +transaction on the blockchain. + +Transaction fees are calculated based on the size of the transaction in +kilobytes, not the value of the transaction in bitcoin. Overall, +transaction fees are set based on market forces within the Bitcoin +network. Miners prioritize transactions based on many different +criteria, including fees, and might even process transactions for free +under certain circumstances. Transaction fees affect the processing +priority, meaning that a transaction with sufficient fees is likely to +be included in the next block mined, whereas a transaction with +insufficient or no fees might be delayed, processed on a best-effort +basis after a few blocks, or not processed at all. Transaction fees are +not mandatory, and transactions without fees might be processed +eventually; however, including transaction fees encourages priority +processing. + +Over time, the way transaction fees are calculated and the effect they +have on transaction prioritization has evolved. At first, transaction +fees were fixed and constant across the network. Gradually, the fee +structure relaxed and may be influenced by market forces, based on +network capacity and transaction volume. Since at least the beginning of +2016, capacity limits in bitcoin have created competition between +transactions, resulting in higher fees and effectively making free +transactions a thing of the past. Zero fee or very low fee transactions +rarely get mined and sometimes will not even be propagated across the +network. + +((("fees", "fee relay policies")))((("minrelaytxfee option")))In Bitcoin +Core, fee relay policies are set by the +minrelaytxfee+ option. The +current default +minrelaytxfee+ is 0.00001 bitcoin or a hundredth of a +millibitcoin per kilobyte. Therefore, by default, transactions with a +fee less than 0.00001 bitcoin are treated as free and are only relayed +if there is space in the mempool; otherwise, they are dropped. Bitcoin +nodes can override the default fee relay policy by adjusting the value +of +minrelaytxfee+. + +((("dynamic fees")))((("fees", "dynamic fees")))Any bitcoin service that +creates transactions, including wallets, exchanges, retail applications, +etc., _must_ implement dynamic fees. Dynamic fees can be implemented +through a third-party fee estimation service or with a built-in fee +estimation algorithm. If you're unsure, begin with a third-party service +and as you gain experience design and implement your own algorithm if +you wish to remove the third-party dependency. + +Fee estimation algorithms calculate the appropriate fee, based on +capacity and the fees offered by "competing" transactions. These +algorithms range from simplistic (average or median fee in the last +block) to sophisticated (statistical analysis). They estimate the +necessary fee (in satoshis per byte) that will give a transaction a high +probability of being selected and included within a certain number of +blocks. Most services offer users the option of choosing high, medium, +or low priority fees. High priority means users pay higher fees but the +transaction is likely to be included in the next block. Medium and low +priority means users pay lower transaction fees but the transactions may +take much longer to confirm. + +((("bitcoinfees (third-party service)")))Many wallet applications use +third-party services for fee calculations. One popular service is +http://bitcoinfees.21.co/[_http://bitcoinfees.21.co_], which provides an +API and a visual chart showing the fee in satoshi/byte for different +priorities. [TIP] ==== -((("static fees")))((("fees", "static fees")))Static fees are no longer viable on the Bitcoin network. Wallets that set static fees will produce a poor user experience as transactions will often get "stuck" and remain unconfirmed. Users who don't understand bitcoin transactions and fees are dismayed by "stuck" transactions because they think they've lost their money. +((("static fees")))((("fees", "static fees")))Static fees are no longer +viable on the Bitcoin network. Wallets that set static fees will produce +a poor user experience as transactions will often get "stuck" and remain +unconfirmed. Users who don't understand bitcoin transactions and fees +are dismayed by "stuck" transactions because they think they've lost +their money. ==== -The chart in <> shows the real-time estimate of fees in 10 satoshi/byte increments and the expected confirmation time (in minutes and number of blocks) for transactions with fees in each range. For each fee range (e.g., 61–70 satoshi/byte), two horizontal bars show the number of unconfirmed transactions (1405) and total number of transactions in the past 24 hours (102,975), with fees in that range. Based on the graph, the recommended high-priority fee at this time was 80 satoshi/byte, a fee likely to result in the transaction being mined in the very next block (zero block delay). For perspective, the median transaction size is 226 bytes, so the recommended fee for a transaction size would be 18,080 satoshis (0.00018080 BTC). - -The fee estimation data can be retrieved via a simple HTTP REST API, at https://bitcoinfees.21.co/api/v1/fees/recommended[https://bitcoinfees.21.co/api/v1/fees/recommended]. For example, on the command line using the +curl+ command: +The chart in <> shows the real-time estimate of fees in +10 satoshi/byte increments and the expected confirmation time (in +minutes and number of blocks) for transactions with fees in each range. +For each fee range (e.g., 61–70 satoshi/byte), two horizontal +bars show the number of unconfirmed transactions (1405) and total number +of transactions in the past 24 hours (102,975), with fees in that range. +Based on the graph, the recommended high-priority fee at this time was +80 satoshi/byte, a fee likely to result in the transaction being mined +in the very next block (zero block delay). For perspective, the median +transaction size is 226 bytes, so the recommended fee for a transaction +size would be 18,080 satoshis (0.00018080 BTC). + +The fee estimation data can be retrieved via a simple HTTP REST API, at +https://bitcoinfees.21.co/api/v1/fees/recommended[https://bitcoinfees.21.co/api/v1/fees/recommended]. +For example, on the command line using the +curl+ command: .Using the fee estimation API ---- @@ -322,7 +629,9 @@ $ curl https://bitcoinfees.21.co/api/v1/fees/recommended {"fastestFee":80,"halfHourFee":80,"hourFee":60} ---- -The API returns a JSON object with the current fee estimate for fastest confirmation (+fastestFee+), confirmation within three blocks (+halfHourFee+) and six blocks (+hourFee+), in satoshi per byte. +The API returns a JSON object with the current fee estimate for fastest +confirmation (+fastestFee+), confirmation within three blocks +(+halfHourFee+) and six blocks (+hourFee+), in satoshi per byte. [[bitcoinfees21co]] .Fee estimation service bitcoinfees.21.co @@ -330,7 +639,11 @@ image::images/mbc2_0602.png[Fee Estimation Service bitcoinfees.21.co] ==== Adding Fees to Transactions -The data structure of transactions does not have a field for fees. Instead, fees are implied as the difference between the sum of inputs and the sum of outputs. Any excess amount that remains after all outputs have been deducted from all inputs is the fee that is collected by the miners: +The data structure of transactions does not have a field for fees. +Instead, fees are implied as the difference between the sum of inputs +and the sum of outputs. Any excess amount that remains after all outputs +have been deducted from all inputs is the fee that is collected by the +miners: [[tx_fee_equation]] .Transaction fees are implied, as the excess of inputs minus outputs: @@ -338,63 +651,195 @@ The data structure of transactions does not have a field for fees. Instead, fees Fees = Sum(Inputs) – Sum(Outputs) ---- -This is a somewhat confusing element of transactions and an important point to understand, because if you are constructing your own transactions you must ensure you do not inadvertently include a very large fee by underspending the inputs. That means that you must account for all inputs, if necessary by creating change, or you will end up giving the miners a very big tip! +This is a somewhat confusing element of transactions and an important +point to understand, because if you are constructing your own +transactions you must ensure you do not inadvertently include a very +large fee by underspending the inputs. That means that you must account +for all inputs, if necessary by creating change, or you will end up +giving the miners a very big tip! -For example, if you consume a 20-bitcoin UTXO to make a 1-bitcoin payment, you must include a 19-bitcoin change output back to your wallet. Otherwise, the 19-bitcoin "leftover" will be counted as a transaction fee and will be collected by the miner who mines your transaction in a block. Although you will receive priority processing and make a miner very happy, this is probably not what you intended. +For example, if you consume a 20-bitcoin UTXO to make a 1-bitcoin +payment, you must include a 19-bitcoin change output back to your +wallet. Otherwise, the 19-bitcoin "leftover" will be counted as a +transaction fee and will be collected by the miner who mines your +transaction in a block. Although you will receive priority processing +and make a miner very happy, this is probably not what you intended. [WARNING] ==== -((("warnings and cautions", "change outputs")))If you forget to add a change output in a manually constructed transaction, you will be paying the change as a transaction fee. "Keep the change!" might not be what you intended. +((("warnings and cautions", "change outputs")))If you forget to add a +change output in a manually constructed transaction, you will be paying +the change as a transaction fee. "Keep the change!" might not be what +you intended. ==== -((("use cases", "buying coffee")))Let's see how this works in practice, by looking at Alice's coffee purchase again. Alice wants to spend 0.015 bitcoin to pay for coffee. To ensure this transaction is processed promptly, she will want to include a transaction fee, say 0.001. That will mean that the total cost of the transaction will be 0.016. Her wallet must therefore source a set of UTXO that adds up to 0.016 bitcoin or more and, if necessary, create change. Let's say her wallet has a 0.2-bitcoin UTXO available. It will therefore need to consume this UTXO, create one output to Bob's Cafe for 0.015, and a second output with 0.184 bitcoin in change back to her own wallet, leaving 0.001 bitcoin unallocated, as an implicit fee for the transaction. - -((("use cases", "charitable donations")))((("charitable donations")))Now let's look at a different scenario. Eugenia, our children's charity director in the Philippines, has completed a fundraiser to purchase schoolbooks for the children. She received several thousand small donations from people all around the world, totaling 50 bitcoin, so her wallet is full of very small payments (UTXO). Now she wants to purchase hundreds of schoolbooks from a local publisher, paying in bitcoin. - -As Eugenia's wallet application tries to construct a single larger payment transaction, it must source from the available UTXO set, which is composed of many smaller amounts. That means that the resulting transaction will source from more than a hundred small-value UTXO as inputs and only one output, paying the book publisher. A transaction with that many inputs will be larger than one kilobyte, perhaps several kilobytes in size. As a result, it will require a much higher fee than the median-sized transaction. - -Eugenia's wallet application will calculate the appropriate fee by measuring the size of the transaction and multiplying that by the per-kilobyte fee. Many wallets will overpay fees for larger transactions to ensure the transaction is processed promptly. The higher fee is not because Eugenia is spending more money, but because her transaction is more complex and larger in size--the fee is independent of the transaction's bitcoin value.((("", startref="Tout06"))) +((("use cases", "buying coffee")))Let's see how this works in practice, +by looking at Alice's coffee purchase again. Alice wants to spend 0.015 +bitcoin to pay for coffee. To ensure this transaction is processed +promptly, she will want to include a transaction fee, say 0.001. That +will mean that the total cost of the transaction will be 0.016. Her +wallet must therefore source a set of UTXO that adds up to 0.016 bitcoin +or more and, if necessary, create change. Let's say her wallet has a +0.2-bitcoin UTXO available. It will therefore need to consume this UTXO, +create one output to Bob's Cafe for 0.015, and a second output with +0.184 bitcoin in change back to her own wallet, leaving 0.001 bitcoin +unallocated, as an implicit fee for the transaction. + +((("use cases", "charitable donations")))((("charitable donations")))Now +let's look at a different scenario. Eugenia, our children's charity +director in the Philippines, has completed a fundraiser to purchase +schoolbooks for the children. She received several thousand small +donations from people all around the world, totaling 50 bitcoin, so her +wallet is full of very small payments (UTXO). Now she wants to purchase +hundreds of schoolbooks from a local publisher, paying in bitcoin. + +As Eugenia's wallet application tries to construct a single larger +payment transaction, it must source from the available UTXO set, which +is composed of many smaller amounts. That means that the resulting +transaction will source from more than a hundred small-value UTXO as +inputs and only one output, paying the book publisher. A transaction +with that many inputs will be larger than one kilobyte, perhaps several +kilobytes in size. As a result, it will require a much higher fee than +the median-sized transaction. + +Eugenia's wallet application will calculate the appropriate fee by +measuring the size of the transaction and multiplying that by the +per-kilobyte fee. Many wallets will overpay fees for larger transactions +to ensure the transaction is processed promptly. The higher fee is not +because Eugenia is spending more money, but because her transaction is +more complex and larger in size--the fee is independent of the +transaction's bitcoin value.((("", startref="Tout06"))) [[tx_script]] [role="pagebreak-before less_space_h1"] === Transaction Scripts and Script Language -((("transactions", "scripts and Script language", id="Tsript06")))((("scripting", "transactions and", id="Stransact06")))The bitcoin transaction script language, called _Script_, is a Forth-like reverse-polish notation stack-based execution language. If that sounds like gibberish, you probably haven't studied 1960s programming languages, but that's ok—we will explain it all in this chapter. Both the locking script placed on an UTXO and the unlocking script are written in this scripting language. When a transaction is validated, the unlocking script in each input is executed alongside the corresponding locking script to see if it satisfies the spending condition. - -Script is a very simple language that was designed to be limited in scope and executable on a range of hardware, perhaps as simple as an embedded device. It requires minimal processing and cannot do many of the fancy things modern programming languages can do. For its use in validating programmable money, this is a deliberate security feature. - -((("Pay-to-Public-Key-Hash (P2PKH)")))Today, most transactions processed through the Bitcoin network have the form "Payment to Bob's Bitcoin address" and are based on a script called a Pay-to-Public-Key-Hash script. However, bitcoin transactions are not limited to the "Payment to Bob's Bitcoin address" script. In fact, locking scripts can be written to express a vast variety of complex conditions. In order to understand these more complex scripts, we must first understand the basics of transaction scripts and script language. - -In this section, we will demonstrate the basic components of the bitcoin transaction scripting language and show how it can be used to express simple conditions for spending and how those conditions can be satisfied by unlocking scripts. +((("transactions", "scripts and Script language", +id="Tsript06")))((("scripting", "transactions and", +id="Stransact06")))The bitcoin transaction script language, called +_Script_, is a Forth-like reverse-polish notation stack-based execution +language. If that sounds like gibberish, you probably haven't studied +1960s programming languages, but that's ok—we will explain it all +in this chapter. Both the locking script placed on an UTXO and the +unlocking script are written in this scripting language. When a +transaction is validated, the unlocking script in each input is executed +alongside the corresponding locking script to see if it satisfies the +spending condition. + +Script is a very simple language that was designed to be limited in +scope and executable on a range of hardware, perhaps as simple as an +embedded device. It requires minimal processing and cannot do many of +the fancy things modern programming languages can do. For its use in +validating programmable money, this is a deliberate security feature. + +((("Pay-to-Public-Key-Hash (P2PKH)")))Today, most transactions processed +through the Bitcoin network have the form "Payment to Bob's Bitcoin +address" and are based on a script called a Pay-to-Public-Key-Hash +script. However, bitcoin transactions are not limited to the "Payment +to Bob's Bitcoin address" script. In fact, locking scripts can be +written to express a vast variety of complex conditions. In order to +understand these more complex scripts, we must first understand the +basics of transaction scripts and script language. + +In this section, we will demonstrate the basic components of the bitcoin +transaction scripting language and show how it can be used to express +simple conditions for spending and how those conditions can be satisfied +by unlocking scripts. [TIP] ==== -((("programmable money")))Bitcoin transaction validation is not based on a static pattern, but instead is achieved through the execution of a scripting language. This language allows for a nearly infinite variety of conditions to be expressed. This is how bitcoin gets the power of "programmable money." +((("programmable money")))Bitcoin transaction validation is not based on +a static pattern, but instead is achieved through the execution of a +scripting language. This language allows for a nearly infinite variety +of conditions to be expressed. This is how bitcoin gets the power of +"programmable money." ==== - ==== Turing Incompleteness -((("Turing incompleteness")))The bitcoin transaction script language contains many operators, but is deliberately limited in one important way--there are no loops or complex flow control capabilities other than conditional flow control. This ensures that the language is not _Turing Complete_, meaning that scripts have limited complexity and predictable execution times. Script is not a general-purpose language. ((("denial-of-service attacks")))((("denial-of-service attacks", see="also security")))((("security", "denial-of-service attacks")))These limitations ensure that the language cannot be used to create an infinite loop or other form of "logic bomb" that could be embedded in a transaction in a way that causes a denial-of-service attack against the Bitcoin network. Remember, every transaction is validated by every full node on the Bitcoin network. A limited language prevents the transaction validation mechanism from being used as a vulnerability. +((("Turing incompleteness")))The bitcoin transaction script language +contains many operators, but is deliberately limited in one important +way--there are no loops or complex flow control capabilities other than +conditional flow control. This ensures that the language is not _Turing +Complete_, meaning that scripts have limited complexity and predictable +execution times. Script is not a general-purpose language. +((("denial-of-service attacks")))((("denial-of-service attacks", +see="also security")))((("security", "denial-of-service attacks")))These +limitations ensure that the language cannot be used to create an +infinite loop or other form of "logic bomb" that could be embedded in a +transaction in a way that causes a denial-of-service attack against the +Bitcoin network. Remember, every transaction is validated by every full +node on the Bitcoin network. A limited language prevents the transaction +validation mechanism from being used as a vulnerability. ==== Stateless Verification -((("stateless verification")))The bitcoin transaction script language is stateless, in that there is no state prior to execution of the script, or state saved after execution of the script. Therefore, all the information needed to execute a script is contained within the script. A script will predictably execute the same way on any system. If your system verifies a script, you can be sure that every other system in the Bitcoin network will also verify the script, meaning that a valid transaction is valid for everyone and everyone knows this. This predictability of outcomes is an essential benefit of the Bitcoin system. +((("stateless verification")))The bitcoin transaction script language is +stateless, in that there is no state prior to execution of the script, +or state saved after execution of the script. Therefore, all the +information needed to execute a script is contained within the script. A +script will predictably execute the same way on any system. If your +system verifies a script, you can be sure that every other system in the +Bitcoin network will also verify the script, meaning that a valid +transaction is valid for everyone and everyone knows this. This +predictability of outcomes is an essential benefit of the Bitcoin +system. [[tx_lock_unlock]] ==== Script Construction (Lock + Unlock) -Bitcoin's transaction validation engine relies on two types of scripts to validate transactions: a locking script and an unlocking script. - -((("locking scripts")))((("unlocking scripts")))((("scripting", "locking scripts")))A locking script is a spending condition placed on an output: it specifies the conditions that must be met to spend the output in the future. ((("scriptPubKey")))Historically, the locking script was called a _scriptPubKey_, because it usually contained a public key or Bitcoin address (public key hash). In this book we refer to it as a "locking script" to acknowledge the much broader range of possibilities of this scripting technology. In most bitcoin applications, what we refer to as a locking script will appear in the source code as +scriptPubKey+. ((("witnesses")))((("cryptographic puzzles")))You will also see the locking script referred to as a _witness script_ (see <>) or more generally as a _cryptographic puzzle_. These terms all mean the same thing, at different levels of abstraction. - -An unlocking script is a script that "solves," or satisfies, the conditions placed on an output by a locking script and allows the output to be spent. Unlocking scripts are part of every transaction input. Most of the time they contain a digital signature produced by the user's wallet from his or her private key. ((("scriptSig")))Historically, the unlocking script was called _scriptSig_, because it usually contained a digital signature. In most bitcoin applications, the source code refers to the unlocking script as +scriptSig+. You will also see the unlocking script referred to as a _witness_ (see <>). In this book, we refer to it as an "unlocking script" to acknowledge the much broader range of locking script requirements, because not all unlocking scripts must contain signatures. - -Every bitcoin validating node will validate transactions by executing the locking and unlocking scripts together. Each input contains an unlocking script and refers to a previously existing UTXO. The validation software will copy the unlocking script, retrieve the UTXO referenced by the input, and copy the locking script from that UTXO. The unlocking and locking script are then executed in sequence. The input is valid if the unlocking script satisfies the locking script conditions (see <>). All the inputs are validated independently, as part of the overall validation of the transaction. - -Note that the UTXO is permanently recorded in the blockchain, and therefore is invariable and is unaffected by failed attempts to spend it by reference in a new transaction. Only a valid transaction that correctly satisfies the conditions of the output results in the output being considered as "spent" and removed from the set of unspent transaction outputs (UTXO set). - -<> is an example of the unlocking and locking scripts for the most common type of bitcoin transaction (a payment to a public key hash), showing the combined script resulting from the concatenation of the unlocking and locking scripts prior to script validation. +Bitcoin's transaction validation engine relies on two types of scripts +to validate transactions: a locking script and an unlocking script. + +((("locking scripts")))((("unlocking scripts")))((("scripting", "locking +scripts")))A locking script is a spending condition placed on an output: +it specifies the conditions that must be met to spend the output in the +future. ((("scriptPubKey")))Historically, the locking script was called +a _scriptPubKey_, because it usually contained a public key or Bitcoin +address (public key hash). In this book we refer to it as a "locking +script" to acknowledge the much broader range of possibilities of this +scripting technology. In most bitcoin applications, what we refer to as +a locking script will appear in the source code as +scriptPubKey+. +((("witnesses")))((("cryptographic puzzles")))You will also see the +locking script referred to as a _witness script_ (see <>) or +more generally as a _cryptographic puzzle_. These terms all mean the +same thing, at different levels of abstraction. + +An unlocking script is a script that "solves," or satisfies, the +conditions placed on an output by a locking script and allows the output +to be spent. Unlocking scripts are part of every transaction input. Most +of the time they contain a digital signature produced by the user's +wallet from his or her private key. ((("scriptSig")))Historically, the +unlocking script was called _scriptSig_, because it usually contained a +digital signature. In most bitcoin applications, the source code refers +to the unlocking script as +scriptSig+. You will also see the unlocking +script referred to as a _witness_ (see <>). In this book, we +refer to it as an "unlocking script" to acknowledge the much broader +range of locking script requirements, because not all unlocking scripts +must contain signatures. + +Every bitcoin validating node will validate transactions by executing +the locking and unlocking scripts together. Each input contains an +unlocking script and refers to a previously existing UTXO. The +validation software will copy the unlocking script, retrieve the UTXO +referenced by the input, and copy the locking script from that UTXO. The +unlocking and locking script are then executed in sequence. The input is +valid if the unlocking script satisfies the locking script conditions +(see <>). All the inputs are validated independently, as +part of the overall validation of the transaction. + +Note that the UTXO is permanently recorded in the blockchain, and +therefore is invariable and is unaffected by failed attempts to spend it +by reference in a new transaction. Only a valid transaction that +correctly satisfies the conditions of the output results in the output +being considered as "spent" and removed from the set of unspent +transaction outputs (UTXO set). + +<> is an example of the unlocking and +locking scripts for the most common type of bitcoin transaction (a +payment to a public key hash), showing the combined script resulting +from the concatenation of the unlocking and locking scripts prior to +script validation. [[scriptSig_and_scriptPubKey]] .Combining scriptSig and scriptPubKey to evaluate a transaction script @@ -402,19 +847,47 @@ image::images/mbc2_0603.png["scriptSig_and_scriptPubKey"] ===== The script execution stack -Bitcoin's scripting language is called a stack-based language because it uses a data structure called a _stack_. A stack is a very simple data structure that can be visualized as a stack of cards. A stack allows two operations: push and pop. Push adds an item on top of the stack. Pop removes the top item from the stack. Operations on a stack can only act on the topmost item on the stack. A stack data structure is also called a Last-In-First-Out, or "LIFO" queue. - -The scripting language executes the script by processing each item from left to right. Numbers (data constants) are pushed onto the stack. Operators push or pop one or more parameters from the stack, act on them, and might push a result onto the stack. For example, +OP_ADD+ will pop two items from the stack, add them, and push the resulting sum onto the stack. - -Conditional operators evaluate a condition, producing a boolean result of TRUE or FALSE. For example, +OP_EQUAL+ pops two items from the stack and pushes TRUE (TRUE is represented by the number 1) if they are equal or FALSE (represented by zero) if they are not equal. Bitcoin transaction scripts usually contain a conditional operator, so that they can produce the TRUE result that signifies a valid transaction. +Bitcoin's scripting language is called a stack-based language because it +uses a data structure called a _stack_. A stack is a very simple data +structure that can be visualized as a stack of cards. A stack allows two +operations: push and pop. Push adds an item on top of the stack. Pop +removes the top item from the stack. Operations on a stack can only act +on the topmost item on the stack. A stack data structure is also called +a Last-In-First-Out, or "LIFO" queue. + +The scripting language executes the script by processing each item from +left to right. Numbers (data constants) are pushed onto the stack. +Operators push or pop one or more parameters from the stack, act on +them, and might push a result onto the stack. For example, +OP_ADD+ will +pop two items from the stack, add them, and push the resulting sum onto +the stack. + +Conditional operators evaluate a condition, producing a boolean result +of TRUE or FALSE. For example, +OP_EQUAL+ pops two items from the stack +and pushes TRUE (TRUE is represented by the number 1) if they are equal +or FALSE (represented by zero) if they are not equal. Bitcoin +transaction scripts usually contain a conditional operator, so that they +can produce the TRUE result that signifies a valid transaction. ===== A simple script Now let's apply what we've learned about scripts and stacks to some simple examples. -In <>, the script +2 3 OP_ADD 5 OP_EQUAL+ demonstrates the arithmetic addition operator +OP_ADD+, adding two numbers and putting the result on the stack, followed by the conditional operator +OP_EQUAL+, which checks that the resulting sum is equal to +5+. For brevity, the +OP_+ prefix is omitted in the step-by-step example. For more details on the available script operators and functions, see <>. - -Although most locking scripts refer to a public key hash (essentially, a Bitcoin address), thereby requiring proof of ownership to spend the funds, the script does not have to be that complex. Any combination of locking and unlocking scripts that results in a TRUE value is valid. The simple arithmetic we used as an example of the scripting language is also a valid locking script that can be used to lock a transaction output. +In <>, the script +2 3 OP_ADD 5 OP_EQUAL+ +demonstrates the arithmetic addition operator +OP_ADD+, adding two +numbers and putting the result on the stack, followed by the conditional +operator +OP_EQUAL+, which checks that the resulting sum is equal to ++5+. For brevity, the +OP_+ prefix is omitted in the step-by-step +example. For more details on the available script operators and +functions, see <>. + +Although most locking scripts refer to a public key hash (essentially, a +Bitcoin address), thereby requiring proof of ownership to spend the +funds, the script does not have to be that complex. Any combination of +locking and unlocking scripts that results in a TRUE value is valid. The +simple arithmetic we used as an example of the scripting language is +also a valid locking script that can be used to lock a transaction +output. Use part of the arithmetic example script as the locking script: @@ -422,23 +895,36 @@ Use part of the arithmetic example script as the locking script: 3 OP_ADD 5 OP_EQUAL ---- -which can be satisfied by a transaction containing an input with the unlocking script: +which can be satisfied by a transaction containing an input with the +unlocking script: ---- 2 ---- -The validation software combines the locking and unlocking scripts and the resulting script is: +The validation software combines the locking and unlocking scripts and +the resulting script is: ---- 2 3 OP_ADD 5 OP_EQUAL ---- -As we saw in the step-by-step example in <>, when this script is executed, the result is +OP_TRUE+, making the transaction valid. Not only is this a valid transaction output locking script, but the resulting UTXO could be spent by anyone with the arithmetic skills to know that the number 2 satisfies the script. +As we saw in the step-by-step example in <>, when +this script is executed, the result is +OP_TRUE+, making the transaction +valid. Not only is this a valid transaction output locking script, but +the resulting UTXO could be spent by anyone with the arithmetic skills +to know that the number 2 satisfies the script. [TIP] ==== -((("transactions", "valid and invalid")))Transactions are valid if the top result on the stack is +TRUE+ (noted as ++{0x01}++), any other nonzero value, or if the stack is empty after script execution. Transactions are invalid if the top value on the stack is +FALSE+ (a zero-length empty value, noted as ++{}++) or if script execution is halted explicitly by an operator, such as +OP_VERIFY+, +OP_RETURN+, or a conditional terminator such as +OP_ENDIF+. See <> for details. +((("transactions", "valid and invalid")))Transactions are valid if the +top result on the stack is +TRUE+ (noted as ++{0x01}++), any +other nonzero value, or if the stack is empty after script execution. +Transactions are invalid if the top value on the stack is +FALSE+ (a +zero-length empty value, noted as ++{}++) or if script +execution is halted explicitly by an operator, such as +OP_VERIFY+, ++OP_RETURN+, or a conditional terminator such as +OP_ENDIF+. See +<> for details. ==== [[simplemath_script]] @@ -446,51 +932,95 @@ As we saw in the step-by-step example in <>, when this script image::images/mbc2_0604.png["TxScriptSimpleMathExample"] [role="pagebreak-before"] -The following is a slightly more complex script, which calculates ++2 + 7 -- 3 + 1++. Notice that when the script contains several operators in a row, the stack allows the results of one operator to be acted upon by the next operator: +The following is a slightly more complex script, which calculates ++2 + +7 -- 3 + 1++. Notice that when the script contains several operators in +a row, the stack allows the results of one operator to be acted upon by +the next operator: ---- 2 7 OP_ADD 3 OP_SUB 1 OP_ADD 7 OP_EQUAL ---- -Try validating the preceding script yourself using pencil and paper. When the script execution ends, you should be left with the value +TRUE+ on the stack. +Try validating the preceding script yourself using pencil and paper. +When the script execution ends, you should be left with the value +TRUE+ +on the stack. [[script_exec]] ===== Separate execution of unlocking and locking scripts -((("security", "locking and unlocking scripts")))In the original Bitcoin client, the unlocking and locking scripts were concatenated and executed in sequence. For security reasons, this was changed in 2010, because of a vulnerability that allowed a malformed unlocking script to push data onto the stack and corrupt the locking script. In the current implementation, the scripts are executed separately with the stack transferred between the two executions, as described next. - -First, the unlocking script is executed, using the stack execution engine. If the unlocking script is executed without errors (e.g., it has no "dangling" operators left over), the main stack is copied and the locking script is executed. If the result of executing the locking script with the stack data copied from the unlocking script is "TRUE," the unlocking script has succeeded in resolving the conditions imposed by the locking script and, therefore, the input is a valid authorization to spend the UTXO. If any result other than "TRUE" remains after execution of the combined script, the input is invalid because it has failed to satisfy the spending conditions placed on the UTXO. +((("security", "locking and unlocking scripts")))In the original Bitcoin +client, the unlocking and locking scripts were concatenated and executed +in sequence. For security reasons, this was changed in 2010, because of +a vulnerability that allowed a malformed unlocking script to push data +onto the stack and corrupt the locking script. In the current +implementation, the scripts are executed separately with the stack +transferred between the two executions, as described next. + +First, the unlocking script is executed, using the stack execution +engine. If the unlocking script is executed without errors (e.g., it has +no "dangling" operators left over), the main stack is copied and the +locking script is executed. If the result of executing the locking +script with the stack data copied from the unlocking script is "TRUE," +the unlocking script has succeeded in resolving the conditions imposed +by the locking script and, therefore, the input is a valid authorization +to spend the UTXO. If any result other than "TRUE" remains after +execution of the combined script, the input is invalid because it has +failed to satisfy the spending conditions placed on the UTXO. [[p2pkh]] ==== Pay-to-Public-Key-Hash (P2PKH) -((("Pay-to-Public-Key-Hash (P2PKH)")))The vast majority of transactions processed on the Bitcoin network spend outputs locked with a Pay-to-Public-Key-Hash or "P2PKH" script. These outputs contain a locking script that locks the output to a public key hash, more commonly known as a Bitcoin address. An output locked by a P2PKH script can be unlocked (spent) by presenting a public key and a digital signature created by the corresponding private key (see <>). +((("Pay-to-Public-Key-Hash (P2PKH)")))The vast majority of transactions +processed on the Bitcoin network spend outputs locked with a +Pay-to-Public-Key-Hash or "P2PKH" script. These outputs contain a +locking script that locks the output to a public key hash, more commonly +known as a Bitcoin address. An output locked by a P2PKH script can be +unlocked (spent) by presenting a public key and a digital signature +created by the corresponding private key (see <>). -((("use cases", "buying coffee")))For example, let's look at Alice's payment to Bob's Cafe again. Alice made a payment of 0.015 bitcoin to the cafe's Bitcoin address. That transaction output would have a locking script of the form: +((("use cases", "buying coffee")))For example, let's look at Alice's +payment to Bob's Cafe again. Alice made a payment of 0.015 bitcoin to +the cafe's Bitcoin address. That transaction output would have a locking +script of the form: ---- OP_DUP OP_HASH160 OP_EQUALVERIFY OP_CHECKSIG ---- -The +Cafe Public Key Hash+ is equivalent to the Bitcoin address of the cafe, without the Base58Check encoding. Most applications would show the _public key hash_ in hexadecimal encoding and not the familiar Bitcoin address Base58Check format that begins with a "1." +The +Cafe Public Key Hash+ is equivalent to the Bitcoin address of the +cafe, without the Base58Check encoding. Most applications would show the +_public key hash_ in hexadecimal encoding and not the familiar Bitcoin +address Base58Check format that begins with a "1." -The preceding locking script can be satisfied with an unlocking script of the form: +The preceding locking script can be satisfied with an unlocking script +of the form: ---- ---- -The two scripts together would form the following combined validation script: +The two scripts together would form the following combined validation +script: ---- OP_DUP OP_HASH160 OP_EQUALVERIFY OP_CHECKSIG ---- -When executed, this combined script will evaluate to TRUE if, and only if, the unlocking script matches the conditions set by the locking script. In other words, the result will be TRUE if the unlocking script has a valid signature from the cafe's private key that corresponds to the public key hash set as an encumbrance. +When executed, this combined script will evaluate to TRUE if, and only +if, the unlocking script matches the conditions set by the locking +script. In other words, the result will be TRUE if the unlocking script +has a valid signature from the cafe's private key that corresponds to +the public key hash set as an encumbrance. -Figures pass:[#P2PubKHash1] and pass:[#P2PubKHash2] show (in two parts) a step-by-step execution of the combined script, which will prove this is a valid transaction.((("", startref="Tsript06")))((("", startref="Stransact06"))) +Figures pass:[#P2PubKHash1] and pass:[#P2PubKHash2] show (in two parts) a step-by-step +execution of the combined script, which will prove this is a valid +transaction.((("", startref="Tsript06")))((("", +startref="Stransact06"))) [[P2PubKHash1]] .Evaluating a script for a P2PKH transaction (part 1 of 2) @@ -503,34 +1033,72 @@ image::images/mbc2_0606.png["Tx_Script_P2PubKeyHash_2"] [[digital_sigs]] === Digital Signatures (ECDSA) -((("transactions", "digital signatures and", id="Tdigsig06")))So far, we have not delved into any detail about "digital signatures." In this section we look at how digital signatures work and how they can present proof of ownership of a private key without revealing that private key. - -((("digital signatures", "algorithm used")))((("Elliptic Curve Digital Signature Algorithm (ECDSA)")))The digital signature algorithm used in bitcoin is the _Elliptic Curve Digital Signature Algorithm_, or _ECDSA_. ECDSA is the algorithm used for digital signatures based on elliptic curve private/public key pairs, as described in <>. ECDSA is used by the script functions +OP_CHECKSIG+, +OP_CHECKSIGVERIFY+, +OP_CHECKMULTISIG+, and +OP_CHECKMULTISIGVERIFY+. Any time you see those in a locking script, the unlocking script must contain an ECDSA signature. - -((("digital signatures", "purposes of")))A digital signature serves three purposes in bitcoin (see the following sidebar). First, the signature proves that the owner of the private key, who is by implication the owner of the funds, has _authorized_ the spending of those funds. Secondly, the proof of authorization is _undeniable_ (nonrepudiation). Thirdly, the signature proves that the transaction (or specific parts of the transaction) have not and _cannot be modified_ by anyone after it has been signed. - -Note that each transaction input is signed independently. This is critical, as neither the signatures nor the inputs have to belong to or be applied by the same "owners." In fact, a specific transaction scheme called "CoinJoin" uses this fact to create multi-party transactions for privacy. +((("transactions", "digital signatures and", id="Tdigsig06")))So far, we +have not delved into any detail about "digital signatures." In this +section we look at how digital signatures work and how they can present +proof of ownership of a private key without revealing that private key. + +((("digital signatures", "algorithm used")))((("Elliptic Curve Digital +Signature Algorithm (ECDSA)")))The digital signature algorithm used in +bitcoin is the _Elliptic Curve Digital Signature Algorithm_, or _ECDSA_. +ECDSA is the algorithm used for digital signatures based on elliptic +curve private/public key pairs, as described in <>. +ECDSA is used by the script functions +OP_CHECKSIG+, ++OP_CHECKSIGVERIFY+, +OP_CHECKMULTISIG+, and +OP_CHECKMULTISIGVERIFY+. +Any time you see those in a locking script, the unlocking script must +contain an ECDSA signature. + +((("digital signatures", "purposes of")))A digital signature serves +three purposes in bitcoin (see the following sidebar). First, the +signature proves that the owner of the private key, who is by +implication the owner of the funds, has _authorized_ the spending of +those funds. Secondly, the proof of authorization is _undeniable_ +(nonrepudiation). Thirdly, the signature proves that the transaction (or +specific parts of the transaction) have not and _cannot be modified_ by +anyone after it has been signed. + +Note that each transaction input is signed independently. This is +critical, as neither the signatures nor the inputs have to belong to or +be applied by the same "owners." In fact, a specific transaction scheme +called "CoinJoin" uses this fact to create multi-party transactions for +privacy. [NOTE] ==== -Each transaction input and any signature it may contain is _completely_ independent of any other input or signature. Multiple parties can collaborate to construct transactions and sign only one input each. +Each transaction input and any signature it may contain is _completely_ +independent of any other input or signature. Multiple parties can +collaborate to construct transactions and sign only one input each. ==== [[digital_signature_definition]] .Wikipedia's Definition of a "Digital Signature" **** -((("digital signatures", "defined")))A digital signature is a mathematical scheme for demonstrating the authenticity of a digital message or documents. A valid digital signature gives a recipient reason to believe that the message was created by a known sender (authentication), that the sender cannot deny having sent the message (nonrepudiation), and that the message was not altered in transit (integrity). +((("digital signatures", "defined")))A digital signature is a +mathematical scheme for demonstrating the authenticity of a digital +message or documents. A valid digital signature gives a recipient reason +to believe that the message was created by a known sender +(authentication), that the sender cannot deny having sent the message +(nonrepudiation), and that the message was not altered in transit +(integrity). _Source: https://en.wikipedia.org/wiki/Digital_signature_ **** ==== How Digital Signatures Work -((("digital signatures", "how they work")))A digital signature is a _mathematical scheme_ that consists of two parts. The first part is an algorithm for creating a signature, using a private key (the signing key), from a message (the transaction). The second part is an algorithm that allows anyone to verify the signature, given also the message and a public key. +((("digital signatures", "how they work")))A digital signature is a +_mathematical scheme_ that consists of two parts. The first part is an +algorithm for creating a signature, using a private key (the signing +key), from a message (the transaction). The second part is an algorithm +that allows anyone to verify the signature, given also the message and a +public key. ===== Creating a digital signature -In bitcoin's implementation of the ECDSA algorithm, the "message" being signed is the transaction, or more accurately a hash of a specific subset of the data in the transaction (see <>). The signing key is the user's private key. The result is the signature: +In bitcoin's implementation of the ECDSA algorithm, the "message" being +signed is the transaction, or more accurately a hash of a specific +subset of the data in the transaction (see <>). The +signing key is the user's private key. The result is the signature: latexmath:[\(Sig = F_{sig}(F_{hash}(m), dA)\)] @@ -544,24 +1112,34 @@ where: More details on the mathematics of ECDSA can be found in <>. -The function _F_~_sig_~ produces a signature +Sig+ that is composed of two values, commonly referred to as +R+ and +S+: +The function _F_~_sig_~ produces a signature +Sig+ that is composed of +two values, commonly referred to as +R+ and +S+: ---- Sig = (R, S) ---- -((("Distinguished Encoding Rules (DER)")))Now that the two values +R+ and +S+ have been calculated, they are serialized into a byte-stream using an international standard encoding scheme called the _Distinguished Encoding Rules_, or _DER_. +((("Distinguished Encoding Rules (DER)")))Now that the two values +R+ +and +S+ have been calculated, they are serialized into a byte-stream +using an international standard encoding scheme called the +_Distinguished Encoding Rules_, or _DER_. [[seralization_of_signatures_der]] ===== Serialization of signatures (DER) -Let's look at the transaction Alice ((("use cases", "buying coffee", id="alicesixtwo")))created again. In the transaction input there is an unlocking script that contains the following DER-encoded signature from Alice's wallet: +Let's look at the transaction Alice ((("use cases", "buying coffee", +id="alicesixtwo")))created again. In the transaction input there is an +unlocking script that contains the following DER-encoded signature from +Alice's wallet: ---- 3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e381301 ---- -That signature is a serialized byte-stream of the +R+ and +S+ values produced by Alice's wallet to prove she owns the private key authorized to spend that output. The serialization format consists of nine elements as follows: +That signature is a serialized byte-stream of the +R+ and +S+ values +produced by Alice's wallet to prove she owns the private key authorized +to spend that output. The serialization format consists of nine elements +as follows: * +0x30+—indicating the start of a DER sequence * +0x45+—the length of the sequence (69 bytes) @@ -573,25 +1151,60 @@ That signature is a serialized byte-stream of the +R+ and +S+ values produced by * +S+—++4b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813++ * A suffix (+0x01+) indicating the type of hash used (+SIGHASH_ALL+) -See if you can decode Alice's serialized (DER-encoded) signature using this list. The important numbers are +R+ and +S+; the rest of the data is part of the DER encoding scheme. +See if you can decode Alice's serialized (DER-encoded) signature using +this list. The important numbers are +R+ and +S+; the rest of the data +is part of the DER encoding scheme. ==== Verifying the Signature -((("digital signatures", "verifying")))To verify the signature, one must have the signature (+R+ and +S+), the serialized transaction, and the public key (that corresponds to the private key used to create the signature). Essentially, verification of a signature means "Only the owner of the private key that generated this public key could have produced this signature on this transaction." +((("digital signatures", "verifying")))To verify the signature, one must +have the signature (+R+ and +S+), the serialized transaction, and the +public key (that corresponds to the private key used to create the +signature). Essentially, verification of a signature means "Only the +owner of the private key that generated this public key could have +produced this signature on this transaction." -The signature verification algorithm takes the message (a hash of the transaction or parts of it), the signer's public key and the signature (+R+ and +S+ values), and returns TRUE if the signature is valid for this message and public key. +The signature verification algorithm takes the message (a hash of the +transaction or parts of it), the signer's public key and the signature +(+R+ and +S+ values), and returns TRUE if the signature is valid for +this message and public key. [[sighash_types]] ==== Signature Hash Types (SIGHASH) -((("digital signatures", "signature hash types")))((("commitment")))Digital signatures are applied to messages, which in the case of bitcoin, are the transactions themselves. The signature implies a _commitment_ by the signer to specific transaction data. In the simplest form, the signature applies to the entire transaction, thereby committing all the inputs, outputs, and other transaction fields. However, a signature can commit to only a subset of the data in a transaction, which is useful for a number of scenarios as we will see in this section. - -((("SIGHASH flags")))Bitcoin signatures have a way of indicating which part of a transaction's data is included in the hash signed by the private key using a +SIGHASH+ flag. The +SIGHASH+ flag is a single byte that is appended to the signature. Every signature has a +SIGHASH+ flag and the flag can be different from input to input. A transaction with three signed inputs may have three signatures with different +SIGHASH+ flags, each signature signing (committing) different parts of the transaction. - -Remember, each input may contain a signature in its unlocking script. As a result, a transaction that contains several inputs may have signatures with different +SIGHASH+ flags that commit different parts of the transaction in each of the inputs. Note also that bitcoin transactions may contain inputs from different "owners," who may sign only one input in a partially constructed (and invalid) transaction, collaborating with others to gather all the necessary signatures to make a valid transaction. Many of the +SIGHASH+ flag types only make sense if you think of multiple participants collaborating outside the Bitcoin network and updating a partially signed transaction. +((("digital signatures", "signature hash +types")))((("commitment")))Digital signatures are applied to messages, +which in the case of bitcoin, are the transactions themselves. The +signature implies a _commitment_ by the signer to specific transaction +data. In the simplest form, the signature applies to the entire +transaction, thereby committing all the inputs, outputs, and other +transaction fields. However, a signature can commit to only a subset of +the data in a transaction, which is useful for a number of scenarios as +we will see in this section. + +((("SIGHASH flags")))Bitcoin signatures have a way of indicating which +part of a transaction's data is included in the hash signed by the +private key using a +SIGHASH+ flag. The +SIGHASH+ flag is a single byte +that is appended to the signature. Every signature has a +SIGHASH+ flag +and the flag can be different from input to input. A transaction with +three signed inputs may have three signatures with different +SIGHASH+ +flags, each signature signing (committing) different parts of the +transaction. + +Remember, each input may contain a signature in its unlocking script. As +a result, a transaction that contains several inputs may have signatures +with different +SIGHASH+ flags that commit different parts of the +transaction in each of the inputs. Note also that bitcoin transactions +may contain inputs from different "owners," who may sign only one input +in a partially constructed (and invalid) transaction, collaborating with +others to gather all the necessary signatures to make a valid +transaction. Many of the +SIGHASH+ flag types only make sense if you +think of multiple participants collaborating outside the Bitcoin network +and updating a partially signed transaction. [role="pagebreak-before"] -There are three +SIGHASH+ flags: +ALL+, +NONE+, and +SINGLE+, as shown in <>. +There are three +SIGHASH+ flags: +ALL+, +NONE+, and +SINGLE+, as shown +in <>. [[sighash_types_and_their]] .SIGHASH types and their meanings @@ -603,7 +1216,12 @@ There are three +SIGHASH+ flags: +ALL+, +NONE+, and +SINGLE+, as shown in <>. +In addition, there is a modifier flag +SIGHASH_ANYONECANPAY+, which can +be combined with each of the preceding flags. When +ANYONECANPAY+ is +set, only one input is signed, leaving the rest (and their sequence +numbers) open for modification. The +ANYONECANPAY+ has the value +0x80+ +and is applied by bitwise OR, resulting in the combined flags as shown +in <>. [[sighash_types_with_modifiers]] .SIGHASH types with modifiers and their meanings @@ -615,40 +1233,100 @@ In addition, there is a modifier flag +SIGHASH_ANYONECANPAY+, which can be combi | SINGLE\|ANYONECANPAY | 0x83 | Signature applies to one input and the output with the same index number |======================= -The way +SIGHASH+ flags are applied during signing and verification is that a copy of the transaction is made and certain fields within are truncated (set to zero length and emptied). The resulting transaction is serialized. The +SIGHASH+ flag is added to the end of the serialized transaction and the result is hashed. The hash itself is the "message" that is signed. Depending on which +SIGHASH+ flag is used, different parts of the transaction are truncated. The resulting hash depends on different subsets of the data in the transaction. By including the +SIGHASH+ as the last step before hashing, the signature commits the +SIGHASH+ type as well, so it can't be changed (e.g., by a miner). +The way +SIGHASH+ flags are applied during signing and verification is +that a copy of the transaction is made and certain fields within are +truncated (set to zero length and emptied). The resulting transaction is +serialized. The +SIGHASH+ flag is added to the end of the serialized +transaction and the result is hashed. The hash itself is the "message" +that is signed. Depending on which +SIGHASH+ flag is used, different +parts of the transaction are truncated. The resulting hash depends on +different subsets of the data in the transaction. By including the ++SIGHASH+ as the last step before hashing, the signature commits the ++SIGHASH+ type as well, so it can't be changed (e.g., by a miner). [NOTE] ==== -All +SIGHASH+ types sign the transaction +nLocktime+ field (see <>). In addition, the +SIGHASH+ type itself is appended to the transaction before it is signed, so that it can't be modified once signed. +All +SIGHASH+ types sign the transaction +nLocktime+ field (see +<>). In addition, the +SIGHASH+ type +itself is appended to the transaction before it is signed, so that it +can't be modified once signed. ==== -In the example of Alice's transaction (see the list in <>), we saw that the last part of the DER-encoded signature was +01+, which is the +SIGHASH_ALL+ flag. This locks the transaction data, so Alice's signature is committing the state of all inputs and outputs. This is the most common signature form. - -Let's look at some of the other +SIGHASH+ types and how they can be used in practice: - -+ALL|ANYONECANPAY+ :: ((("charitable donations")))((("use cases", "charitable donations")))This construction can be used to make a "crowdfunding”-style transaction. Someone attempting to raise funds can construct a transaction with a single output. The single output pays the "goal" amount to the fundraiser. Such a transaction is obviously not valid, as it has no inputs. However, others can now amend it by adding an input of their own, as a donation. They sign their own input with +ALL|ANYONECANPAY+. Unless enough inputs are gathered to reach the value of the output, the transaction is invalid. Each donation is a "pledge," which cannot be collected by the fundraiser until the entire goal amount is raised. - -+NONE+ :: This construction can be used to create a "bearer check" or "blank check" of a specific amount. It commits to the input, but allows the output locking script to be changed. Anyone can write their own Bitcoin address into the output locking script and redeem the transaction. However, the output value itself is locked by the signature. - -+NONE|ANYONECANPAY+ :: This construction can be used to build a "dust collector." Users who have tiny UTXO in their wallets can't spend these without the cost in fees exceeding the value of the dust. With this type of signature, the dust UTXO can be donated for anyone to aggregate and spend whenever they want. - -((("Bitmask Sighash Modes")))There are some proposals to modify or expand the +SIGHASH+ system. One such proposal is _Bitmask Sighash Modes_ by Blockstream's Glenn Willen, as part of the Elements project. This aims to create a flexible replacement for +SIGHASH+ types that allows "arbitrary, miner-rewritable bitmasks of inputs and outputs" that can express "more complex contractual precommitment schemes, such as signed offers with change in a distributed asset exchange." +In the example of Alice's transaction (see the list in +<>), we saw that the last part of the +DER-encoded signature was +01+, which is the +SIGHASH_ALL+ flag. This +locks the transaction data, so Alice's signature is committing the state +of all inputs and outputs. This is the most common signature form. + +Let's look at some of the other +SIGHASH+ types and how they can be used +in practice: + ++ALL|ANYONECANPAY+ :: ((("charitable donations")))((("use cases", +"charitable donations")))This construction can be used to make a +"crowdfunding”-style transaction. Someone attempting to raise +funds can construct a transaction with a single output. The single +output pays the "goal" amount to the fundraiser. Such a transaction is +obviously not valid, as it has no inputs. However, others can now amend +it by adding an input of their own, as a donation. They sign their own +input with +ALL|ANYONECANPAY+. Unless enough inputs are gathered to +reach the value of the output, the transaction is invalid. Each donation +is a "pledge," which cannot be collected by the fundraiser until the +entire goal amount is raised. + ++NONE+ :: This construction can be used to create a "bearer check" or +"blank check" of a specific amount. It commits to the input, but allows +the output locking script to be changed. Anyone can write their own +Bitcoin address into the output locking script and redeem the +transaction. However, the output value itself is locked by the +signature. + ++NONE|ANYONECANPAY+ :: This construction can be used to build a "dust +collector." Users who have tiny UTXO in their wallets can't spend these +without the cost in fees exceeding the value of the dust. With this type +of signature, the dust UTXO can be donated for anyone to aggregate and +spend whenever they want. + +((("Bitmask Sighash Modes")))There are some proposals to modify or +expand the +SIGHASH+ system. One such proposal is _Bitmask Sighash +Modes_ by Blockstream's Glenn Willen, as part of the Elements project. +This aims to create a flexible replacement for +SIGHASH+ types that +allows "arbitrary, miner-rewritable bitmasks of inputs and outputs" that +can express "more complex contractual precommitment schemes, such as +signed offers with change in a distributed asset exchange." [NOTE] ==== -You will not see +SIGHASH+ flags presented as an option in a user's wallet application. With few exceptions, wallets construct P2PKH scripts and sign with +SIGHASH_ALL+ flags. To use a different +SIGHASH+ flag, you would have to write software to construct and sign transactions. More importantly, +SIGHASH+ flags can be used by special-purpose bitcoin applications that enable novel uses. +You will not see +SIGHASH+ flags presented as an option in a user's +wallet application. With few exceptions, wallets construct P2PKH scripts +and sign with +SIGHASH_ALL+ flags. To use a different +SIGHASH+ flag, +you would have to write software to construct and sign transactions. +More importantly, +SIGHASH+ flags can be used by special-purpose bitcoin +applications that enable novel uses. ==== [[ecdsa_math]] ==== ECDSA Math -((("Elliptic Curve Digital Signature Algorithm (ECDSA)")))As mentioned previously, signatures are created by a mathematical function _F_~_sig_~ that produces a signature composed of two values _R_ and _S_. In this section we look at the function _F_~_sig_~ in more detail. +((("Elliptic Curve Digital Signature Algorithm (ECDSA)")))As mentioned +previously, signatures are created by a mathematical function _F_~_sig_~ +that produces a signature composed of two values _R_ and _S_. In this +section we look at the function _F_~_sig_~ in more detail. -((("public and private keys", "key pairs", "ephemeral")))The signature algorithm first generates an _ephemeral_ (temporary) private public key pair. This temporary key pair is used in the calculation of the _R_ and _S_ values, after a transformation involving the signing private key and the transaction hash. +((("public and private keys", "key pairs", "ephemeral")))The signature +algorithm first generates an _ephemeral_ (temporary) private public key +pair. This temporary key pair is used in the calculation of the _R_ and +_S_ values, after a transformation involving the signing private key and +the transaction hash. -The temporary key pair is based on a random number _k_, which is used as the temporary private key. From _k_, we generate the corresponding temporary public key _P_ (calculated as _P = k*G_, in the same way bitcoin public keys are derived; see <>). The _R_ value of the digital signature is then the x coordinate of the ephemeral public key _P_. +The temporary key pair is based on a random number _k_, which is used as +the temporary private key. From _k_, we generate the corresponding +temporary public key _P_ (calculated as _P = k*G_, in the same way +bitcoin public keys are derived; see <>). The _R_ value of the +digital signature is then the x coordinate of the ephemeral public key +_P_. -From there, the algorithm calculates the _S_ value of the signature, such that: +From there, the algorithm calculates the _S_ value of the signature, +such that: _S_ = __k__^-1^ (__Hash__(__m__) + __dA__ * __R__) _mod p_ @@ -660,75 +1338,180 @@ where: * _m_ is the transaction data * _p_ is the prime order of the elliptic curve -Verification is the inverse of the signature generation function, using the _R_, _S_ values and the public key to calculate a value _P_, which is a point on the elliptic curve (the ephemeral public key used in signature creation): +Verification is the inverse of the signature generation function, using +the _R_, _S_ values and the public key to calculate a value _P_, which +is a point on the elliptic curve (the ephemeral public key used in +signature creation): _P_ = __S__^-1^ * __Hash__(__m__) * _G_ + __S__^-1^ * _R_ * _Qa_ where: -* _R_ and _S_ are the signature values -* _Qa_ is Alice's public key -* _m_ is the transaction data that was signed -* _G_ is the elliptic curve generator point +- _R_ and _S_ are the signature values +- _Qa_ is Alice's public key +- _m_ is the transaction data that was signed +- _G_ is the elliptic curve generator point -If the x coordinate of the calculated point _P_ is equal to _R_, then the verifier can conclude that the signature is valid. +If the x coordinate of the calculated point _P_ is equal to _R_, then +the verifier can conclude that the signature is valid. -Note that in verifying the signature, the private key is neither known nor revealed. +Note that in verifying the signature, the private key is neither known +nor revealed. [TIP] ==== -ECDSA is necessarily a fairly complicated piece of math; a full explanation is beyond the scope of this book. A number of great guides online take you through it step by step: search for "ECDSA explained" or try this one: http://bit.ly/2r0HhGB[]. +ECDSA is necessarily a fairly complicated piece of math; a full +explanation is beyond the scope of this book. A number of great guides +online take you through it step by step: search for "ECDSA explained" or +try this one: http://bit.ly/2r0HhGB[]. ==== ==== The Importance of Randomness in Signatures -((("digital signatures", "randomness in")))As we saw in <>, the signature generation algorithm uses a random key _k_, as the basis for an ephemeral private/public key pair. The value of _k_ is not important, _as long as it is random_. If the same value _k_ is used to produce two signatures on different messages (transactions), then the signing _private key_ can be calculated by anyone. Reuse of the same value for _k_ in a signature algorithm leads to exposure of the private key! +((("digital signatures", "randomness in")))As we saw in <>, +the signature generation algorithm uses a random key _k_, as the basis +for an ephemeral private/public key pair. The value of _k_ is not +important, _as long as it is random_. If the same value _k_ is used to +produce two signatures on different messages (transactions), then the +signing _private key_ can be calculated by anyone. Reuse of the same +value for _k_ in a signature algorithm leads to exposure of the private +key! [WARNING] ==== -((("warnings and cautions", "digital signatures")))If the same value _k_ is used in the signing algorithm on two different transactions, the private key can be calculated and exposed to the world! +((("warnings and cautions", "digital signatures")))If the same value _k_ +is used in the signing algorithm on two different transactions, the +private key can be calculated and exposed to the world! ==== -This is not just a theoretical possibility. We have seen this issue lead to exposure of private keys in a few different implementations of transaction-signing algorithms in bitcoin. People have had funds stolen because of inadvertent reuse of a _k_ value. The most common reason for reuse of a _k_ value is an improperly initialized random-number generator. - -((("random numbers", "random number generation")))((("entropy", "random number generation")))((("deterministic initialization")))To avoid this vulnerability, the industry best practice is to not generate _k_ with a random-number generator seeded with entropy, but instead to use a deterministic-random process seeded with the transaction data itself. This ensures that each transaction produces a different _k_. The industry-standard algorithm for deterministic initialization of _k_ is defined in https://tools.ietf.org/html/rfc6979[RFC 6979], published by the Internet Engineering Task Force. - -If you are implementing an algorithm to sign transactions in bitcoin, you _must_ use RFC 6979 or a similarly deterministic-random algorithm to ensure you generate a different _k_ for each transaction.((("", startref="Tdigsig06"))) +This is not just a theoretical possibility. We have seen this issue lead +to exposure of private keys in a few different implementations of +transaction-signing algorithms in bitcoin. People have had funds stolen +because of inadvertent reuse of a _k_ value. The most common reason for +reuse of a _k_ value is an improperly initialized random-number +generator. + +((("random numbers", "random number generation")))((("entropy", "random +number generation")))((("deterministic initialization")))To avoid this +vulnerability, the industry best practice is to not generate _k_ with a +random-number generator seeded with entropy, but instead to use a +deterministic-random process seeded with the transaction data itself. +This ensures that each transaction produces a different _k_. The +industry-standard algorithm for deterministic initialization of _k_ is +defined in https://tools.ietf.org/html/rfc6979[RFC 6979], published by +the Internet Engineering Task Force. + +If you are implementing an algorithm to sign transactions in bitcoin, +you _must_ use RFC 6979 or a similarly deterministic-random algorithm to +ensure you generate a different _k_ for each transaction.((("", +startref="Tdigsig06"))) === Bitcoin Addresses, Balances, and Other Abstractions -((("transactions", "higher-level abstractions", id="Thigher06")))We began this chapter with the discovery that transactions look very different "behind the scenes" than how they are presented in wallets, blockchain explorers, and other user-facing applications. Many of the simplistic and familiar concepts from the earlier chapters, such as Bitcoin addresses and balances, seem to be absent from the transaction structure. We saw that transactions don't contain Bitcoin addresses, per se, but instead operate through scripts that lock and unlock discrete values of bitcoin. Balances are not present anywhere in this system and yet every wallet application prominently displays the balance of the user's wallet. - -Now that we have explored what is actually included in a bitcoin transaction, we can examine how the higher-level abstractions are derived from the seemingly primitive components of the transaction. - -Let's look again at how Alice's transaction was presented on a popular block explorer (<>). +((("transactions", "higher-level abstractions", id="Thigher06")))We +began this chapter with the discovery that transactions look very +different "behind the scenes" than how they are presented in wallets, +blockchain explorers, and other user-facing applications. Many of the +simplistic and familiar concepts from the earlier chapters, such as +Bitcoin addresses and balances, seem to be absent from the transaction +structure. We saw that transactions don't contain Bitcoin addresses, per +se, but instead operate through scripts that lock and unlock discrete +values of bitcoin. Balances are not present anywhere in this system and +yet every wallet application prominently displays the balance of the +user's wallet. + +Now that we have explored what is actually included in a bitcoin +transaction, we can examine how the higher-level abstractions are +derived from the seemingly primitive components of the transaction. + +Let's look again at how Alice's transaction was presented on a popular +block explorer (<>). [[alice_transaction_to_bobs_cafe]] .Alice's transaction to Bob's Cafe image::images/mbc2_0208.png["Alice Coffee Transaction"] -On the left side of the transaction, the blockchain explorer shows Alice's Bitcoin address as the "sender." In fact, this information is not in the transaction itself. When the blockchain explorer retrieved the transaction it also retrieved the previous transaction referenced in the input and extracted the first output from that older transaction. Within that output is a locking script that locks the UTXO to Alice's public key hash (a P2PKH script). The blockchain explorer extracted the public key hash and encoded it using Base58Check encoding to produce and display the Bitcoin address that represents that public key. - -Similarly, on the right side, the blockchain explorer shows the two outputs; the first to Bob's Bitcoin address and the second to Alice's Bitcoin address (as change). Once again, to create these Bitcoin addresses, the blockchain explorer extracted the locking script from each output, recognized it as a P2PKH script, and extracted the public-key-hash from within. Finally, the blockchain explorer reencoded that public key hash with Base58Check to produce and display the Bitcoin addresses. - -If you were to click on Bob's Bitcoin address, the blockchain explorer would show you the view in <>. +On the left side of the transaction, the blockchain explorer shows +Alice's Bitcoin address as the "sender." In fact, this information is +not in the transaction itself. When the blockchain explorer retrieved +the transaction it also retrieved the previous transaction referenced in +the input and extracted the first output from that older transaction. +Within that output is a locking script that locks the UTXO to Alice's +public key hash (a P2PKH script). The blockchain explorer extracted the +public key hash and encoded it using Base58Check encoding to produce and +display the Bitcoin address that represents that public key. + +Similarly, on the right side, the blockchain explorer shows the two +outputs; the first to Bob's Bitcoin address and the second to Alice's +Bitcoin address (as change). Once again, to create these Bitcoin +addresses, the blockchain explorer extracted the locking script from +each output, recognized it as a P2PKH script, and extracted the +public-key-hash from within. Finally, the blockchain explorer reencoded +that public key hash with Base58Check to produce and display the Bitcoin +addresses. + +If you were to click on Bob's Bitcoin address, the blockchain explorer +would show you the view in <>. [[the_balance_of_bobs_bitcoin_address]] .The balance of Bob's Bitcoin address image::images/mbc2_0608.png["The balance of Bob's Bitcoin address"] -The blockchain explorer displays the balance of Bob's Bitcoin address. But nowhere in the Bitcoin system is there a concept of a "balance." Rather, the values displayed here are constructed by the blockchain explorer as follows. - -To construct the "Total Received" amount, the blockchain explorer first will decode the Base58Check encoding of the Bitcoin address to retrieve the 160-bit hash of Bob's public key that is encoded within the address. Then, the blockchain explorer will search through the database of transactions, looking for outputs with P2PKH locking scripts that contain Bob's public key hash. By summing up the value of all the outputs, the blockchain explorer can produce the total value received. - -Constructing the current balance (displayed as "Final Balance") requires a bit more work. The blockchain explorer keeps a separate database of the outputs that are currently unspent, the UTXO set. To maintain this database, the blockchain explorer must monitor the Bitcoin network, add newly created UTXO, and remove spent UTXO, in real time, as they appear in unconfirmed transactions. This is a complicated process that depends on keeping track of transactions as they propagate, as well as maintaining consensus with the Bitcoin network to ensure that the correct chain is followed. Sometimes, the blockchain explorer goes out of sync and its perspective of the UTXO set is incomplete or incorrect. - -From the UTXO set, the blockchain explorer sums up the value of all unspent outputs referencing Bob's public key hash and produces the "Final Balance" number shown to the user. - -In order to produce this one image, with these two "balances," the blockchain explorer has to index and search through dozens, hundreds, or even hundreds of thousands of transactions. - -In summary, the information presented to users through wallet applications, blockchain explorers, and other bitcoin user interfaces is often composed of higher-level abstractions that are derived by searching many different transactions, inspecting their content, and manipulating the data contained within them. By presenting this simplistic view of bitcoin transactions that resemble bank checks from one sender to one recipient, these applications have to abstract a lot of underlying detail. They mostly focus on the common types of transactions: P2PKH with SIGHASH_ALL signatures on every input. Thus, while bitcoin applications can present more than 80% of all transactions in an easy-to-read manner, they are sometimes stumped by transactions that deviate from the norm. Transactions that contain more complex locking scripts, or different SIGHASH flags, or many inputs and outputs, demonstrate the simplicity and weakness of these abstractions. - -Every day, hundreds of transactions that do not contain P2PKH outputs are confirmed on the blockchain. The blockchain explorers often present these with red warning messages saying they cannot decode an address. The following link contains the most recent "strange transactions" that were not fully decoded: https://blockchain.info/strange-transactions[]. - -As we will see in the next chapter, these are not necessarily strange transactions. They are transactions that contain more complex locking scripts than the common P2PKH. We will learn how to decode and understand more complex scripts and the applications they support next.((("", startref="Thigher06")))((("", startref="alicesixtwo"))) +The blockchain explorer displays the balance of Bob's Bitcoin address. +But nowhere in the Bitcoin system is there a concept of a "balance." +Rather, the values displayed here are constructed by the blockchain +explorer as follows. + +To construct the "Total Received" amount, the blockchain explorer first +will decode the Base58Check encoding of the Bitcoin address to retrieve +the 160-bit hash of Bob's public key that is encoded within the address. +Then, the blockchain explorer will search through the database of +transactions, looking for outputs with P2PKH locking scripts that +contain Bob's public key hash. By summing up the value of all the +outputs, the blockchain explorer can produce the total value received. + +Constructing the current balance (displayed as "Final Balance") requires +a bit more work. The blockchain explorer keeps a separate database of +the outputs that are currently unspent, the UTXO set. To maintain this +database, the blockchain explorer must monitor the Bitcoin network, add +newly created UTXO, and remove spent UTXO, in real time, as they appear +in unconfirmed transactions. This is a complicated process that depends +on keeping track of transactions as they propagate, as well as +maintaining consensus with the Bitcoin network to ensure that the +correct chain is followed. Sometimes, the blockchain explorer goes out +of sync and its perspective of the UTXO set is incomplete or incorrect. + +From the UTXO set, the blockchain explorer sums up the value of all +unspent outputs referencing Bob's public key hash and produces the +"Final Balance" number shown to the user. + +In order to produce this one image, with these two "balances," the +blockchain explorer has to index and search through dozens, hundreds, or +even hundreds of thousands of transactions. + +In summary, the information presented to users through wallet +applications, blockchain explorers, and other bitcoin user interfaces is +often composed of higher-level abstractions that are derived by +searching many different transactions, inspecting their content, and +manipulating the data contained within them. By presenting this +simplistic view of bitcoin transactions that resemble bank checks from +one sender to one recipient, these applications have to abstract a lot +of underlying detail. They mostly focus on the common types of +transactions: P2PKH with SIGHASH_ALL signatures on every input. Thus, +while bitcoin applications can present more than 80% of all transactions +in an easy-to-read manner, they are sometimes stumped by transactions +that deviate from the norm. Transactions that contain more complex +locking scripts, or different SIGHASH flags, or many inputs and outputs, +demonstrate the simplicity and weakness of these abstractions. + +Every day, hundreds of transactions that do not contain P2PKH outputs +are confirmed on the blockchain. The blockchain explorers often present +these with red warning messages saying they cannot decode an address. +The following link contains the most recent "strange transactions" that +were not fully decoded: https://blockchain.info/strange-transactions[]. + +As we will see in the next chapter, these are not necessarily strange +transactions. They are transactions that contain more complex locking +scripts than the common P2PKH. We will learn how to decode and +understand more complex scripts and the applications they support +next.((("", startref="Thigher06")))((("", startref="alicesixtwo"))) diff --git a/ch07.asciidoc b/ch07.asciidoc index 5e9d9490..def52a58 100644 --- a/ch07.asciidoc +++ b/ch07.asciidoc @@ -5,37 +5,71 @@ [[ch07_intro]] === Introduction -In the previous chapter, we introduced the basic elements of bitcoin transactions and looked at the most common type of transaction script, the P2PKH script. In this chapter we will look at more advanced scripting and how we can use it to build transactions with complex conditions. - -First, we will look at _multisignature_ scripts. Next, we will examine the second most common transaction script, _Pay-to-Script-Hash_, which opens up a whole world of complex scripts. Then, we will examine new script operators that add a time dimension to bitcoin, through _timelocks_. Finally, we will look at _Segregated Witness_, an architectural change to the structure of transactions. +In the previous chapter, we introduced the basic elements of bitcoin +transactions and looked at the most common type of transaction script, +the P2PKH script. In this chapter we will look at more advanced +scripting and how we can use it to build transactions with complex +conditions. + +First, we will look at _multisignature_ scripts. Next, we will examine +the second most common transaction script, _Pay-to-Script-Hash_, which +opens up a whole world of complex scripts. Then, we will examine new +script operators that add a time dimension to bitcoin, through +_timelocks_. Finally, we will look at _Segregated Witness_, an +architectural change to the structure of transactions. [[multisig]] === Multisignature -((("transactions", "advanced", "multisignature scripts")))((("transactions", "advanced", id="Tadv07")))((("scripting", "multisignature scripts", id="Smulti07")))((("multisignature scripts")))Multisignature scripts set a condition where N public keys are recorded in the script and at least M of those must provide signatures to unlock the funds. This is also known as an M-of-N scheme, where N is the total number of keys and M is the threshold of signatures required for validation. For example, a 2-of-3 multisignature is one where three public keys are listed as potential signers and at least two of those must be used to create signatures for a valid transaction to spend the funds. - -At this time, _standard_ multisignature scripts are limited to at most 3 listed public keys, meaning you can do anything from a 1-of-1 to a 3-of-3 multisignature or any combination within that range. The limitation to 3 listed keys might be lifted by the time this book is published, so check the +IsStandard()+ function to see what is currently accepted by the network. Note that the limit of 3 keys applies only to standard (also known as "bare") multisignature scripts, not to multisignature scripts wrapped in a Pay-to-Script-Hash (P2SH) script. P2SH multisignature scripts are limited to 15 keys, allowing for up to 15-of-15 multisignature. We will learn about P2SH in <>. - -The general form of a locking script setting an M-of-N multisignature condition is: +((("transactions", "advanced", "multisignature +scripts")))((("transactions", "advanced", id="Tadv07")))((("scripting", +"multisignature scripts", id="Smulti07")))((("multisignature +scripts")))Multisignature scripts set a condition where N public keys +are recorded in the script and at least M of those must provide +signatures to unlock the funds. This is also known as an M-of-N scheme, +where N is the total number of keys and M is the threshold of signatures +required for validation. For example, a 2-of-3 multisignature is one +where three public keys are listed as potential signers and at least two +of those must be used to create signatures for a valid transaction to +spend the funds. + +At this time, _standard_ multisignature scripts are limited to at most 3 +listed public keys, meaning you can do anything from a 1-of-1 to a +3-of-3 multisignature or any combination within that range. The +limitation to 3 listed keys might be lifted by the time this book is +published, so check the +IsStandard()+ function to see what is currently +accepted by the network. Note that the limit of 3 keys applies only to +standard (also known as "bare") multisignature scripts, not to +multisignature scripts wrapped in a Pay-to-Script-Hash (P2SH) script. +P2SH multisignature scripts are limited to 15 keys, allowing for up to +15-of-15 multisignature. We will learn about P2SH in <>. + +The general form of a locking script setting an M-of-N multisignature +condition is: ---- M ... N CHECKMULTISIG ---- -where N is the total number of listed public keys and M is the threshold of required signatures to spend the output. +where N is the total number of listed public keys and M is the threshold +of required signatures to spend the output. -A locking script setting a 2-of-3 multisignature condition looks like this: +A locking script setting a 2-of-3 multisignature condition looks like +this: ---- 2 3 CHECKMULTISIG ---- -The preceding locking script can be satisfied with an unlocking script containing pairs of signatures and public keys: +The preceding locking script can be satisfied with an unlocking script +containing pairs of signatures and public keys: ---- ---- -or any combination of two signatures from the private keys corresponding to the three listed public keys. + +or any combination of two signatures from the private keys corresponding +to the three listed public keys. The two scripts together would form the combined validation script: @@ -43,22 +77,47 @@ The two scripts together would form the combined validation script: 2 3 CHECKMULTISIG ---- -When executed, this combined script will evaluate to TRUE if, and only if, the unlocking script matches the conditions set by the locking script. In this case, the condition is whether the unlocking script has a valid signature from the two private keys that correspond to two of the three public keys set as an encumbrance. +When executed, this combined script will evaluate to TRUE if, and only +if, the unlocking script matches the conditions set by the locking +script. In this case, the condition is whether the unlocking script has +a valid signature from the two private keys that correspond to two of +the three public keys set as an encumbrance. [[multisig_bug]] ===== A bug in CHECKMULTISIG execution -((("scripting", "multisignature scripts", "CHECKMULTISIG bug")))((("CHECKMULTISIG bug workaround")))There is a bug in ++CHECKMULTISIG++'s execution that requires a slight workaround. When +CHECKMULTISIG+ executes, it should consume M+N+2 items on the stack as parameters. However, due to the bug, +CHECKMULTISIG+ will pop an extra value or one value more than expected. +((("scripting", "multisignature scripts", "CHECKMULTISIG +bug")))((("CHECKMULTISIG bug workaround")))There is a bug in +++CHECKMULTISIG++'s execution that requires a slight workaround. When ++CHECKMULTISIG+ executes, it should consume M+N+2 items on the stack as +parameters. However, due to the bug, +CHECKMULTISIG+ will pop an extra +value or one value more than expected. -Let's look at this in greater detail using the previous validation example: +Let's look at this in greater detail using the previous validation +example: ---- 2 3 CHECKMULTISIG ---- -First, +CHECKMULTISIG+ pops the top item, which is +N+ (in this example "3"). Then it pops +N+ items, which are the public keys that can sign. In this example, public keys A, B, and C. Then, it pops one item, which is +M+, the quorum (how many signatures are needed). Here M = 2. At this point, +CHECKMULTISIG+ should pop the final +M+ items, which are the signatures, and see if they are valid. However, unfortunately, a bug in the implementation causes +CHECKMULTISIG+ to pop one more item (M+1 total) than it should. The extra item is disregarded when checking the signatures so it has no direct effect on +CHECKMULTISIG+ itself. However, an extra value must be present because if it is not present, when +CHECKMULTISIG+ attempts to pop on an empty stack, it will cause a stack error and script failure (marking the transaction as invalid). Because the extra item is disregarded it can be anything, but customarily +0+ is used. +First, +CHECKMULTISIG+ pops the top item, which is +N+ (in this example +"3"). Then it pops +N+ items, which are the public keys that can sign. +In this example, public keys A, B, and C. Then, it pops one item, which +is +M+, the quorum (how many signatures are needed). Here M = 2. At this +point, +CHECKMULTISIG+ should pop the final +M+ items, which are the +signatures, and see if they are valid. However, unfortunately, a bug in +the implementation causes +CHECKMULTISIG+ to pop one more item (M+1 +total) than it should. The extra item is disregarded when checking the +signatures so it has no direct effect on +CHECKMULTISIG+ itself. +However, an extra value must be present because if it is not present, +when +CHECKMULTISIG+ attempts to pop on an empty stack, it will cause a +stack error and script failure (marking the transaction as invalid). +Because the extra item is disregarded it can be anything, but +customarily +0+ is used. -Because this bug became part of the consensus rules, it must now be replicated forever. Therefore the correct script validation would look like this: +Because this bug became part of the consensus rules, it must now be +replicated forever. Therefore the correct script validation would look +like this: ---- 0 2 3 CHECKMULTISIG @@ -76,14 +135,35 @@ but instead it is: 0 ---- -From now on, if you see a multisig unlocking script, you should expect to see an extra +0+ in the beginning, whose only purpose is as a workaround to a bug that accidentally became a consensus rule.((("", startref="Smulti07"))) +From now on, if you see a multisig unlocking script, you should expect +to see an extra +0+ in the beginning, whose only purpose is as a +workaround to a bug that accidentally became a consensus rule.((("", +startref="Smulti07"))) [[p2sh]] === Pay-to-Script-Hash (P2SH) -((("transactions", "advanced", "Pay-to-Script-Hash")))((("scripting", "Pay-to-Script-Hash", id="Spay07")))Pay-to-Script-Hash (P2SH) was introduced in 2012 as a powerful new type of transaction that greatly simplifies the use of complex transaction scripts. To explain the need for P2SH, let's look at a practical example. - -((("use cases", "import/export", id="mohamseven")))((("scripting", "Pay-to-Script-Hash", "import/export example")))((("Pay-to-Script-Hash (P2SH)", "import/export example")))In <> we introduced Mohammed, an electronics importer based in Dubai. Mohammed's company uses bitcoin's multisignature feature extensively for its corporate accounts. Multisignature scripts are one of the most common uses of bitcoin's advanced scripting capabilities and are a very powerful feature. ((("accounts receivable (AR)")))Mohammed's company uses a multisignature script for all customer payments, known in accounting terms as "accounts receivable," or AR. With the multisignature scheme, any payments made by customers are locked in such a way that they require at least two signatures to release, from Mohammed and one of his partners or from his attorney who has a backup key. A multisignature scheme like that offers corporate governance controls and protects against theft, embezzlement, or loss. +((("transactions", "advanced", "Pay-to-Script-Hash")))((("scripting", +"Pay-to-Script-Hash", id="Spay07")))Pay-to-Script-Hash (P2SH) was +introduced in 2012 as a powerful new type of transaction that greatly +simplifies the use of complex transaction scripts. To explain the need +for P2SH, let's look at a practical example. + +((("use cases", "import/export", id="mohamseven")))((("scripting", +"Pay-to-Script-Hash", "import/export example")))((("Pay-to-Script-Hash +(P2SH)", "import/export example")))In <> we +introduced Mohammed, an electronics importer based in Dubai. Mohammed's +company uses bitcoin's multisignature feature extensively for its +corporate accounts. Multisignature scripts are one of the most common +uses of bitcoin's advanced scripting capabilities and are a very +powerful feature. ((("accounts receivable (AR)")))Mohammed's company +uses a multisignature script for all customer payments, known in +accounting terms as "accounts receivable," or AR. With the +multisignature scheme, any payments made by customers are locked in such +a way that they require at least two signatures to release, from +Mohammed and one of his partners or from his attorney who has a backup +key. A multisignature scheme like that offers corporate governance +controls and protects against theft, embezzlement, or loss. The resulting script is quite long and looks like this: @@ -91,11 +171,35 @@ The resulting script is quite long and looks like this: 2 5 CHECKMULTISIG ---- -Although multisignature scripts are a powerful feature, they are cumbersome to use. Given the preceding script, Mohammed would have to communicate this script to every customer prior to payment. Each customer would have to use special bitcoin wallet software with the ability to create custom transaction scripts, and each customer would have to understand how to create a transaction using custom scripts. Furthermore, the resulting transaction would be about five times larger than a simple payment transaction, because this script contains very long public keys. The burden of that extra-large transaction would be borne by the customer in the form of fees. Finally, a large transaction script like this would be carried in the UTXO set in RAM in every full node, until it was spent. All of these issues make using complex locking scripts difficult in practice. - -P2SH was developed to resolve these practical difficulties and to make the use of complex scripts as easy as a payment to a Bitcoin address. With P2SH payments, the complex locking script is replaced with its digital fingerprint, a cryptographic hash. When a transaction attempting to spend the UTXO is presented later, it must contain the script that matches the hash, in addition to the unlocking script. In simple terms, P2SH means "pay to a script matching this hash, a script that will be presented later when this output is spent." - -((("redeem scripts")))((("scripting", "redeem scripts")))In P2SH transactions, the locking script that is replaced by a hash is referred to as the _redeem script_ because it is presented to the system at redemption time rather than as a locking script. <> shows the script without P2SH and <> shows the same script encoded with P2SH. +Although multisignature scripts are a powerful feature, they are +cumbersome to use. Given the preceding script, Mohammed would have to +communicate this script to every customer prior to payment. Each +customer would have to use special bitcoin wallet software with the +ability to create custom transaction scripts, and each customer would +have to understand how to create a transaction using custom scripts. +Furthermore, the resulting transaction would be about five times larger +than a simple payment transaction, because this script contains very +long public keys. The burden of that extra-large transaction would be +borne by the customer in the form of fees. Finally, a large transaction +script like this would be carried in the UTXO set in RAM in every full +node, until it was spent. All of these issues make using complex locking +scripts difficult in practice. + +P2SH was developed to resolve these practical difficulties and to make +the use of complex scripts as easy as a payment to a Bitcoin address. +With P2SH payments, the complex locking script is replaced with its +digital fingerprint, a cryptographic hash. When a transaction attempting +to spend the UTXO is presented later, it must contain the script that +matches the hash, in addition to the unlocking script. In simple terms, +P2SH means "pay to a script matching this hash, a script that will be +presented later when this output is spent." + +((("redeem scripts")))((("scripting", "redeem scripts")))In P2SH +transactions, the locking script that is replaced by a hash is referred +to as the _redeem script_ because it is presented to the system at +redemption time rather than as a locking script. <> shows +the script without P2SH and <> shows the same script encoded +with P2SH. [[without_p2sh]] .Complex script without P2SH @@ -112,26 +216,39 @@ P2SH was developed to resolve these practical difficulties and to make the use o | Unlocking Script | Sig1 Sig2 |======= -As you can see from the tables, with P2SH the complex script that details the conditions for spending the output (redeem script) is not presented in the locking script. Instead, only a hash of it is in the locking script and the redeem script itself is presented later, as part of the unlocking script when the output is spent. This shifts the burden in fees and complexity from the sender to the recipient (spender) of the transaction. +As you can see from the tables, with P2SH the complex script that +details the conditions for spending the output (redeem script) is not +presented in the locking script. Instead, only a hash of it is in the +locking script and the redeem script itself is presented later, as part +of the unlocking script when the output is spent. This shifts the burden +in fees and complexity from the sender to the recipient (spender) of the +transaction. -Let's look at Mohammed's company, the complex multisignature script, and the resulting P2SH scripts. +Let's look at Mohammed's company, the complex multisignature script, and +the resulting P2SH scripts. -First, the multisignature script that Mohammed's company uses for all incoming payments from customers: +First, the multisignature script that Mohammed's company uses for all +incoming payments from customers: ---- 2 5 CHECKMULTISIG ---- -If the placeholders are replaced by actual public keys (shown here as 520-bit numbers starting with 04) you can see that this script becomes very long: +If the placeholders are replaced by actual public keys (shown here as +520-bit numbers starting with 04) you can see that this script becomes +very long: ---- 2 04C16B8698A9ABF84250A7C3EA7EEDEF9897D1C8C6ADF47F06CF73370D74DCCA01CDCA79DCC5C395D7EEC6984D83F1F50C900A24DD47F569FD4193AF5DE762C58704A2192968D8655D6A935BEAF2CA23E3FB87A3495E7AF308EDF08DAC3C1FCBFC2C75B4B0F4D0B1B70CD2423657738C0C2B1D5CE65C97D78D0E34224858008E8B49047E63248B75DB7379BE9CDA8CE5751D16485F431E46117B9D0C1837C9D5737812F393DA7D4420D7E1A9162F0279CFC10F1E8E8F3020DECDBC3C0DD389D99779650421D65CBD7149B255382ED7F78E946580657EE6FDA162A187543A9D85BAAA93A4AB3A8F044DADA618D087227440645ABE8A35DA8C5B73997AD343BE5C2AFD94A5043752580AFA1ECED3C68D446BCAB69AC0BA7DF50D56231BE0AABF1FDEEC78A6A45E394BA29A1EDF518C022DD618DA774D207D137AAB59E0B000EB7ED238F4D800 5 CHECKMULTISIG ---- -This entire script can instead be represented by a 20-byte cryptographic hash, by first applying the SHA256 hashing algorithm and then applying the RIPEMD160 algorithm on the result. +This entire script can instead be represented by a 20-byte cryptographic +hash, by first applying the SHA256 hashing algorithm and then applying +the RIPEMD160 algorithm on the result. -We use +libbitcoin-explorer+ (+bx+) on the command-line to produce the script hash, as follows: +We use +libbitcoin-explorer+ (+bx+) on the command-line to produce the +script hash, as follows: ---- echo \ @@ -146,7 +263,10 @@ echo \ 54c557e07dde5bb6cb791c7a540e0a4796f5e97e ---- -The series of commands above first encodes Mohammed's multisig redeem script as a serialized hex-encoded bitcoin Script. The next +bx+ command calculates the SHA256 hash of that. The next +bx+ command hashes again with RIPEMD160, producing the final script-hash: +The series of commands above first encodes Mohammed's multisig redeem +script as a serialized hex-encoded bitcoin Script. The next +bx+ command +calculates the SHA256 hash of that. The next +bx+ command hashes again +with RIPEMD160, producing the final script-hash: The 20-byte hash of Mohammed's redeem script is: @@ -154,36 +274,58 @@ The 20-byte hash of Mohammed's redeem script is: 54c557e07dde5bb6cb791c7a540e0a4796f5e97e ---- -A P2SH transaction locks the output to this hash instead of the longer redeem script, using the locking script: +A P2SH transaction locks the output to this hash instead of the longer +redeem script, using the locking script: ---- HASH160 54c557e07dde5bb6cb791c7a540e0a4796f5e97e EQUAL ---- -which, as you can see, is much shorter. Instead of "pay to this 5-key multisignature script," the P2SH equivalent transaction is "pay to a script with this hash." A customer making a payment to Mohammed's company need only include this much shorter locking script in his payment. When Mohammed and his partners want to spend this UTXO, they must present the original redeem script (the one whose hash locked the UTXO) and the signatures necessary to unlock it, like this: +which, as you can see, is much shorter. Instead of "pay to this 5-key +multisignature script," the P2SH equivalent transaction is "pay to a +script with this hash." A customer making a payment to Mohammed's +company need only include this much shorter locking script in his +payment. When Mohammed and his partners want to spend this UTXO, they +must present the original redeem script (the one whose hash locked the +UTXO) and the signatures necessary to unlock it, like this: ---- <2 PK1 PK2 PK3 PK4 PK5 5 CHECKMULTISIG> ---- -The two scripts are combined in two stages. First, the redeem script is checked against the locking script to make sure the hash matches: +The two scripts are combined in two stages. First, the redeem script is +checked against the locking script to make sure the hash matches: ---- <2 PK1 PK2 PK3 PK4 PK5 5 CHECKMULTISIG> HASH160 EQUAL ---- -If the redeem script hash matches, the unlocking script is executed on its own, to unlock the redeem script: + +If the redeem script hash matches, the unlocking script is executed on +its own, to unlock the redeem script: ---- 2 PK1 PK2 PK3 PK4 PK5 5 CHECKMULTISIG ---- -Almost all the scripts described in this chapter can only be implemented as P2SH scripts. They cannot be used directly in the locking script of an UTXO.((("", startref="mohamseven"))) +Almost all the scripts described in this chapter can only be implemented +as P2SH scripts. They cannot be used directly in the locking script of +an UTXO.((("", startref="mohamseven"))) ==== P2SH Addresses -((("scripting", "Pay-to-Script-Hash", "addresses")))((("Pay-to-Script-Hash (P2SH)", "addresses")))((("bitcoin improvement proposals", "Address Format for P2SH (BIP-13)")))Another important part of the P2SH feature is the ability to encode a script hash as an address, as defined in BIP-13. P2SH addresses are Base58Check encodings of the 20-byte hash of a script, just like Bitcoin addresses are Base58Check encodings of the 20-byte hash of a public key. P2SH addresses use the version prefix "5," which results in Base58Check-encoded addresses that start with a "3." +((("scripting", "Pay-to-Script-Hash", +"addresses")))((("Pay-to-Script-Hash (P2SH)", "addresses")))((("bitcoin +improvement proposals", "Address Format for P2SH (BIP-13)")))Another +important part of the P2SH feature is the ability to encode a script +hash as an address, as defined in BIP-13. P2SH addresses are Base58Check +encodings of the 20-byte hash of a script, just like Bitcoin addresses +are Base58Check encodings of the 20-byte hash of a public key. P2SH +addresses use the version prefix "5," which results in +Base58Check-encoded addresses that start with a "3." -For example, Mohammed's complex script, hashed and Base58Check-encoded as a P2SH address, becomes +39RF6JqABiHdYHkfChV6USGMe6Nsr66Gzw+. We can confirm that with the +bx+ command: +For example, Mohammed's complex script, hashed and Base58Check-encoded +as a P2SH address, becomes +39RF6JqABiHdYHkfChV6USGMe6Nsr66Gzw+. We can +confirm that with the +bx+ command: ---- echo \ @@ -192,47 +334,129 @@ echo \ 39RF6JqABiHdYHkfChV6USGMe6Nsr66Gzw ---- +Now, Mohammed can give this "address" to his customers and they can use +almost any bitcoin wallet to make a simple payment, as if it were a +Bitcoin address. The 3 prefix gives them a hint that this is a special +type of address, one corresponding to a script instead of a public key, +but otherwise it works in exactly the same way as a payment to a Bitcoin +address. -Now, Mohammed can give this "address" to his customers and they can use almost any bitcoin wallet to make a simple payment, as if it were a Bitcoin address. The 3 prefix gives them a hint that this is a special type of address, one corresponding to a script instead of a public key, but otherwise it works in exactly the same way as a payment to a Bitcoin address. - -P2SH addresses hide all of the complexity, so that the person making a payment does not see the script. +P2SH addresses hide all of the complexity, so that the person making a +payment does not see the script. ==== Benefits of P2SH ((("scripting", "Pay-to-Script-Hash", "benefits of")))((("Pay-to-Script-Hash (P2SH)", "benefits of")))The P2SH feature offers the following benefits compared to the direct use of complex scripts in locking outputs: -* Complex scripts are replaced by shorter fingerprints in the transaction output, making the transaction smaller. -* Scripts can be coded as an address, so the sender and the sender's wallet don't need complex engineering to implement P2SH. -* P2SH shifts the burden of constructing the script to the recipient, not the sender. -* P2SH shifts the burden in data storage for the long script from the output (which additionally to being stored on the blockchain is in the UTXO set) to the input (only stored on the blockchain). -* P2SH shifts the burden in data storage for the long script from the present time (payment) to a future time (when it is spent). -* P2SH shifts the transaction fee cost of a long script from the sender to the recipient, who has to include the long redeem script to spend it. +- Complex scripts are replaced by shorter fingerprints in the + transaction output, making the transaction smaller. -==== Redeem Script and Validation +- Scripts can be coded as an address, so the sender and the sender's + wallet don't need complex engineering to implement P2SH. + +- P2SH shifts the burden of constructing the script to the recipient, + not the sender. -((("scripting", "Pay-to-Script-Hash", "redeem scripts and validation")))((("Pay-to-Script-Hash (P2SH)", "redeem scripts and validation")))((("redeem scripts")))((("validation")))Prior to version 0.9.2 of the Bitcoin Core client, Pay-to-Script-Hash was limited to the standard types of bitcoin transaction scripts, by the +IsStandard()+ function. That means that the redeem script presented in the spending transaction could only be one of the standard types: P2PK, P2PKH, or multisig. +- P2SH shifts the burden in data storage for the long script from the + output (which additionally to being stored on the blockchain is in the + UTXO set) to the input (only stored on the blockchain). -As of version 0.9.2 of the Bitcoin Core client, P2SH transactions can contain any valid script, making the P2SH standard much more flexible and allowing for experimentation with many novel and complex types of transactions. +- P2SH shifts the burden in data storage for the long script from the + present time (payment) to a future time (when it is spent). -You are not able to put a P2SH inside a P2SH redeem script, because the P2SH specification is not recursive. Also, while it is technically possible to include +RETURN+ (see <>) in a redeem script, as nothing in the rules prevents you from doing so, it is of no practical use because executing +RETURN+ during validation will cause the transaction to be marked invalid. +- P2SH shifts the transaction fee cost of a long script from the sender + to the recipient, who has to include the long redeem script to spend + it. -Note that because the redeem script is not presented to the network until you attempt to spend a P2SH output, if you lock an output with the hash of an invalid redeem script it will be processed regardless. The UTXO will be successfully locked. However, you will not be able to spend it because the spending transaction, which includes the redeem script, will not be accepted because it is an invalid script. This creates a risk, because you can lock bitcoin in a P2SH that cannot be spent later. The network will accept the P2SH locking script even if it corresponds to an invalid redeem script, because the script hash gives no indication of the script it represents.((("", startref="Spay07"))) +==== Redeem Script and Validation + +((("scripting", "Pay-to-Script-Hash", "redeem scripts and +validation")))((("Pay-to-Script-Hash (P2SH)", "redeem scripts and +validation")))((("redeem scripts")))((("validation")))Prior to version +0.9.2 of the Bitcoin Core client, Pay-to-Script-Hash was limited to the +standard types of bitcoin transaction scripts, by the +IsStandard()+ +function. That means that the redeem script presented in the spending +transaction could only be one of the standard types: P2PK, P2PKH, or +multisig. + +As of version 0.9.2 of the Bitcoin Core client, P2SH transactions can +contain any valid script, making the P2SH standard much more flexible +and allowing for experimentation with many novel and complex types of +transactions. + +You are not able to put a P2SH inside a P2SH redeem script, because the +P2SH specification is not recursive. Also, while it is technically +possible to include +RETURN+ (see <>) in a redeem script, as +nothing in the rules prevents you from doing so, it is of no practical +use because executing +RETURN+ during validation will cause the +transaction to be marked invalid. + +Note that because the redeem script is not presented to the network +until you attempt to spend a P2SH output, if you lock an output with the +hash of an invalid redeem script it will be processed regardless. The +UTXO will be successfully locked. However, you will not be able to spend +it because the spending transaction, which includes the redeem script, +will not be accepted because it is an invalid script. This creates a +risk, because you can lock bitcoin in a P2SH that cannot be spent later. +The network will accept the P2SH locking script even if it corresponds +to an invalid redeem script, because the script hash gives no indication +of the script it represents.((("", startref="Spay07"))) [WARNING] ==== -((("warnings and cautions", "accidental bitcoin locking")))P2SH locking scripts contain the hash of a redeem script, which gives no clues as to the content of the redeem script itself. The P2SH transaction will be considered valid and accepted even if the redeem script is invalid. You might accidentally lock bitcoin in such a way that it cannot later be spent. +((("warnings and cautions", "accidental bitcoin locking")))P2SH locking +scripts contain the hash of a redeem script, which gives no clues as to +the content of the redeem script itself. The P2SH transaction will be +considered valid and accepted even if the redeem script is invalid. You +might accidentally lock bitcoin in such a way that it cannot later be +spent. ==== - - [[op_return]] === Data Recording Output (RETURN) -((("transactions", "advanced", "data recording output")))((("scripting", "data recording output")))((("RETURN operator")))((("data recording (nonpayment data)")))((("nonpayment data")))((("blockchain (the)", "nonpayment data recording")))((("digital notary services")))Bitcoin's distributed and timestamped ledger, the blockchain, has potential uses far beyond payments. Many developers have tried to use the transaction scripting language to take advantage of the security and resilience of the system for applications such as digital notary services, stock certificates, and smart contracts. Early attempts to use bitcoin's script language for these purposes involved creating transaction outputs that recorded data on the blockchain; for example, to record a digital fingerprint of a file in such a way that anyone could establish proof-of-existence of that file on a specific date by reference to that transaction. - -((("blockchain bloat")))((("bloat")))((("unspent transaction outputs (UTXO)")))((("UTXO sets")))The use of bitcoin's blockchain to store data unrelated to bitcoin payments is a controversial subject. Many developers consider such use abusive and want to discourage it. Others view it as a demonstration of the powerful capabilities of blockchain technology and want to encourage such experimentation. Those who object to the inclusion of nonpayment data argue that it causes "blockchain bloat," burdening those running full Bitcoin nodes with carrying the cost of disk storage for data that the blockchain was not intended to carry. Moreover, such transactions create UTXO that cannot be spent, using the destination Bitcoin address as a freeform 20-byte field. Because the address is used for data, it doesn't correspond to a private key and the resulting UTXO can _never_ be spent; it's a fake payment. These transactions that can never be spent are therefore never removed from the UTXO set and cause the size of the UTXO database to forever increase, or "bloat." - -In version 0.9 of the Bitcoin Core client, a compromise was reached with the introduction of the +RETURN+ operator. +RETURN+ allows developers to add 80 bytes of nonpayment data to a transaction output. However, unlike the use of "fake" UTXO, the +RETURN+ operator creates an explicitly _provably unspendable_ output, which does not need to be stored in the UTXO set. +RETURN+ outputs are recorded on the blockchain, so they consume disk space and contribute to the increase in the blockchain's size, but they are not stored in the UTXO set and therefore do not bloat the UTXO memory pool and burden full nodes with the cost of more expensive RAM. +((("transactions", "advanced", "data recording output")))((("scripting", +"data recording output")))((("RETURN operator")))((("data recording +(nonpayment data)")))((("nonpayment data")))((("blockchain (the)", +"nonpayment data recording")))((("digital notary services")))Bitcoin's +distributed and timestamped ledger, the blockchain, has potential uses +far beyond payments. Many developers have tried to use the transaction +scripting language to take advantage of the security and resilience of +the system for applications such as digital notary services, stock +certificates, and smart contracts. Early attempts to use bitcoin's +script language for these purposes involved creating transaction outputs +that recorded data on the blockchain; for example, to record a digital +fingerprint of a file in such a way that anyone could establish +proof-of-existence of that file on a specific date by reference to that +transaction. + +((("blockchain bloat")))((("bloat")))((("unspent transaction outputs +(UTXO)")))((("UTXO sets")))The use of bitcoin's blockchain to store data +unrelated to bitcoin payments is a controversial subject. Many +developers consider such use abusive and want to discourage it. Others +view it as a demonstration of the powerful capabilities of blockchain +technology and want to encourage such experimentation. Those who object +to the inclusion of nonpayment data argue that it causes "blockchain +bloat," burdening those running full Bitcoin nodes with carrying the +cost of disk storage for data that the blockchain was not intended to +carry. Moreover, such transactions create UTXO that cannot be spent, +using the destination Bitcoin address as a freeform 20-byte field. +Because the address is used for data, it doesn't correspond to a private +key and the resulting UTXO can _never_ be spent; it's a fake payment. +These transactions that can never be spent are therefore never removed +from the UTXO set and cause the size of the UTXO database to forever +increase, or "bloat." + +In version 0.9 of the Bitcoin Core client, a compromise was reached with +the introduction of the +RETURN+ operator. +RETURN+ allows developers to +add 80 bytes of nonpayment data to a transaction output. However, unlike +the use of "fake" UTXO, the +RETURN+ operator creates an explicitly +_provably unspendable_ output, which does not need to be stored in the +UTXO set. +RETURN+ outputs are recorded on the blockchain, so they +consume disk space and contribute to the increase in the blockchain's +size, but they are not stored in the UTXO set and therefore do not bloat +the UTXO memory pool and burden full nodes with the cost of more +expensive RAM. +RETURN+ scripts look like this: @@ -240,81 +464,193 @@ In version 0.9 of the Bitcoin Core client, a compromise was reached with the int RETURN ---- -((("Proof of Existence")))((("DOCPROOF prefix")))The data portion is limited to 80 bytes and most often represents a hash, such as the output from the SHA256 algorithm (32 bytes). Many applications put a prefix in front of the data to help identify the application. For example, the http://proofofexistence.com[Proof of Existence] digital notarization service uses the 8-byte prefix +DOCPROOF+, which is ASCII encoded as +44 4f 43 50 52 4f 4f 46+ in hexadecimal. - -Keep in mind that there is no "unlocking script" that corresponds to +RETURN+ that could possibly be used to "spend" a +RETURN+ output. The whole point of +RETURN+ is that you can't spend the money locked in that output, and therefore it does not need to be held in the UTXO set as potentially spendable—+RETURN+ is _provably unspendable_. +RETURN+ is usually an output with a zero bitcoin amount, because any bitcoin assigned to such an output is effectively lost forever. If a +RETURN+ is referenced as an input in a transaction, the script validation engine will halt the execution of the validation script and mark the transaction as invalid. The execution of +RETURN+ essentially causes the script to "RETURN" with a +FALSE+ and halt. Thus, if you accidentally reference a +RETURN+ output as an input in a transaction, that transaction is invalid. - -A standard transaction (one that conforms to the +IsStandard()+ checks) can have only one +RETURN+ output. However, a single +RETURN+ output can be combined in a transaction with outputs of any other type. - -Two new command-line options have been added in Bitcoin Core as of version 0.10. The option +datacarrier+ controls relay and mining of +RETURN+ transactions, with the default set to "1" to allow them. The option +datacarriersize+ takes a numeric argument specifying the maximum size in bytes of the +RETURN+ script, 83 bytes by default, which, allows for a maximum of 80 bytes of +RETURN+ data plus one byte of +RETURN+ opcode and two bytes of +PUSHDATA+ opcode. +((("Proof of Existence")))((("DOCPROOF prefix")))The data portion is +limited to 80 bytes and most often represents a hash, such as the output +from the SHA256 algorithm (32 bytes). Many applications put a prefix in +front of the data to help identify the application. For example, the +http://proofofexistence.com[Proof of Existence] digital notarization +service uses the 8-byte prefix +DOCPROOF+, which is ASCII encoded as +44 +4f 43 50 52 4f 4f 46+ in hexadecimal. + +Keep in mind that there is no "unlocking script" that corresponds to ++RETURN+ that could possibly be used to "spend" a +RETURN+ output. The +whole point of +RETURN+ is that you can't spend the money locked in that +output, and therefore it does not need to be held in the UTXO set as +potentially spendable—+RETURN+ is _provably unspendable_. +RETURN+ is +usually an output with a zero bitcoin amount, because any bitcoin +assigned to such an output is effectively lost forever. If a +RETURN+ is +referenced as an input in a transaction, the script validation engine +will halt the execution of the validation script and mark the +transaction as invalid. The execution of +RETURN+ essentially causes the +script to "RETURN" with a +FALSE+ and halt. Thus, if you accidentally +reference a +RETURN+ output as an input in a transaction, that +transaction is invalid. + +A standard transaction (one that conforms to the +IsStandard()+ checks) +can have only one +RETURN+ output. However, a single +RETURN+ output can +be combined in a transaction with outputs of any other type. + +Two new command-line options have been added in Bitcoin Core as of +version 0.10. The option +datacarrier+ controls relay and mining of ++RETURN+ transactions, with the default set to "1" to allow them. The +option +datacarriersize+ takes a numeric argument specifying the maximum +size in bytes of the +RETURN+ script, 83 bytes by default, which, allows +for a maximum of 80 bytes of +RETURN+ data plus one byte of +RETURN+ +opcode and two bytes of +PUSHDATA+ opcode. [NOTE] ==== -+RETURN+ was initially proposed with a limit of 80 bytes, but the limit was reduced to 40 bytes when the feature was released. In February 2015, in version 0.10 of Bitcoin Core, the limit was raised back to 80 bytes. Nodes may choose not to relay or mine +RETURN+, or only relay and mine +RETURN+ containing less than 80 bytes of data. ++RETURN+ was initially proposed with a limit of 80 bytes, but the limit +was reduced to 40 bytes when the feature was released. In February 2015, +in version 0.10 of Bitcoin Core, the limit was raised back to 80 bytes. +Nodes may choose not to relay or mine +RETURN+, or only relay and mine ++RETURN+ containing less than 80 bytes of data. ==== === Timelocks -((("transactions", "advanced", "timelocks")))((("scripting", "timelocks", id="Stimelock07")))((("nLocktime field")))((("scripting", "timelocks", "uses for")))((("timelocks", "uses for")))Timelocks are restrictions on transactions or outputs that only allow spending after a point in time. Bitcoin has had a transaction-level timelock feature from the beginning. It is implemented by the +nLocktime+ field in a transaction. Two new timelock features were introduced in late 2015 and mid-2016 that offer UTXO-level timelocks. These are +CHECKLOCKTIMEVERIFY+ and +CHECKSEQUENCEVERIFY+. - -Timelocks are useful for postdating transactions and locking funds to a date in the future. More importantly, timelocks extend bitcoin scripting into the dimension of time, opening the door for complex multistep smart contracts. +((("transactions", "advanced", "timelocks")))((("scripting", +"timelocks", id="Stimelock07")))((("nLocktime field")))((("scripting", +"timelocks", "uses for")))((("timelocks", "uses for")))Timelocks are +restrictions on transactions or outputs that only allow spending after a +point in time. Bitcoin has had a transaction-level timelock feature from +the beginning. It is implemented by the +nLocktime+ field in a +transaction. Two new timelock features were introduced in late 2015 and +mid-2016 that offer UTXO-level timelocks. These are ++CHECKLOCKTIMEVERIFY+ and +CHECKSEQUENCEVERIFY+. + +Timelocks are useful for postdating transactions and locking funds to a +date in the future. More importantly, timelocks extend bitcoin scripting +into the dimension of time, opening the door for complex multistep smart +contracts. [[transaction_locktime_nlocktime]] ==== Transaction Locktime (nLocktime) -((("scripting", "timelocks", "nLocktime")))((("timelocks", "nLocktime")))From the beginning, Bitcoin has had a transaction-level timelock feature. Transaction locktime is a transaction-level setting (a field in the transaction data structure) that defines the earliest time that a transaction is valid and can be relayed on the network or added to the blockchain. Locktime is also known as +nLocktime+ from the variable name used in the Bitcoin Core codebase. It is set to zero in most transactions to indicate immediate propagation and execution. If +nLocktime+ is nonzero and below 500 million, it is interpreted as a block height, meaning the transaction is not valid and is not relayed or included in the blockchain prior to the specified block height. If it is above 500 million, it is interpreted as a Unix Epoch timestamp (seconds since Jan-1-1970) and the transaction is not valid prior to the specified time. Transactions with +nLocktime+ specifying a future block or time must be held by the originating system and transmitted to the Bitcoin network only after they become valid. If a transaction is transmitted to the network before the specified +nLocktime+, the transaction will be rejected by the first node as invalid and will not be relayed to other nodes. The use of +nLocktime+ is equivalent to postdating a paper check. +((("scripting", "timelocks", "nLocktime")))((("timelocks", +"nLocktime")))From the beginning, Bitcoin has had a transaction-level +timelock feature. Transaction locktime is a transaction-level setting (a +field in the transaction data structure) that defines the earliest time +that a transaction is valid and can be relayed on the network or added +to the blockchain. Locktime is also known as +nLocktime+ from the +variable name used in the Bitcoin Core codebase. It is set to zero in +most transactions to indicate immediate propagation and execution. If ++nLocktime+ is nonzero and below 500 million, it is interpreted as a +block height, meaning the transaction is not valid and is not relayed or +included in the blockchain prior to the specified block height. If it is +above 500 million, it is interpreted as a Unix Epoch timestamp (seconds +since Jan-1-1970) and the transaction is not valid prior to the +specified time. Transactions with +nLocktime+ specifying a future block +or time must be held by the originating system and transmitted to the +Bitcoin network only after they become valid. If a transaction is +transmitted to the network before the specified +nLocktime+, the +transaction will be rejected by the first node as invalid and will not +be relayed to other nodes. The use of +nLocktime+ is equivalent to +postdating a paper check. [[locktime_limitations]] ===== Transaction locktime limitations -+nLocktime+ has the limitation that while it makes it possible to spend some outputs in the future, it does not make it impossible to spend them until that time. Let's explain that with the following example. ++nLocktime+ has the limitation that while it makes it possible to spend +some outputs in the future, it does not make it impossible to spend them +until that time. Let's explain that with the following example. -((("use cases", "buying coffee", id="alicesseven")))Alice signs a transaction spending one of her outputs to Bob's address, and sets the transaction +nLocktime+ to 3 months in the future. Alice sends that transaction to Bob to hold. With this transaction Alice and Bob know that: +((("use cases", "buying coffee", id="alicesseven")))Alice signs a +transaction spending one of her outputs to Bob's address, and sets the +transaction +nLocktime+ to 3 months in the future. Alice sends that +transaction to Bob to hold. With this transaction Alice and Bob know +that: -* Bob cannot transmit the transaction to redeem the funds until 3 months have elapsed. -* Bob may transmit the transaction after 3 months. +- Bob cannot transmit the transaction to redeem the funds until 3 months + have elapsed. + +- Bob may transmit the transaction after 3 months. However: -* Alice can create another transaction, double-spending the same inputs without a locktime. Thus, Alice can spend the same UTXO before the 3 months have elapsed. -* Bob has no guarantee that Alice won't do that. +- Alice can create another transaction, double-spending the same inputs + without a locktime. Thus, Alice can spend the same UTXO before the 3 + months have elapsed. -It is important to understand the limitations of transaction +nLocktime+. The only guarantee is that Bob will not be able to redeem it before 3 months have elapsed. There is no guarantee that Bob will get the funds. To achieve such a guarantee, the timelock restriction must be placed on the UTXO itself and be part of the locking script, rather than on the transaction. This is achieved by the next form of timelock, called Check Lock Time Verify. +- Bob has no guarantee that Alice won't do that. -==== Check Lock Time Verify (CLTV) +It is important to understand the limitations of transaction ++nLocktime+. The only guarantee is that Bob will not be able to redeem +it before 3 months have elapsed. There is no guarantee that Bob will get +the funds. To achieve such a guarantee, the timelock restriction must be +placed on the UTXO itself and be part of the locking script, rather than +on the transaction. This is achieved by the next form of timelock, +called Check Lock Time Verify. -((("Check Lock Time Verify (CLTV)", id="cltv07")))((("timelocks", "Check Lock Time Verify (CLTV)")))((("scripting", "timelocks", "Check Lock Time Verify (CLTV)")))((("bitcoin improvement proposals", "CHECKLOCKTIMEVERIFY (BIP-65)")))In December 2015, a new form of timelock was introduced to Bitcoin as a soft fork upgrade. Based on a specification in BIP-65, a new script operator called _CHECKLOCKTIMEVERIFY_ (_CLTV_) was added to the scripting language. +CLTV+ is a per-output timelock, rather than a per-transaction timelock as is the case with +nLocktime+. This allows for much greater flexibility in the way timelocks are applied. +==== Check Lock Time Verify (CLTV) -In simple terms, by adding the +CLTV+ opcode in the redeem script of an output it restricts the output, so that it can only be spent after the specified time has elapsed. +((("Check Lock Time Verify (CLTV)", id="cltv07")))((("timelocks", "Check +Lock Time Verify (CLTV)")))((("scripting", "timelocks", "Check Lock Time +Verify (CLTV)")))((("bitcoin improvement proposals", +"CHECKLOCKTIMEVERIFY (BIP-65)")))In December 2015, a new form of +timelock was introduced to Bitcoin as a soft fork upgrade. Based on a +specification in BIP-65, a new script operator called +_CHECKLOCKTIMEVERIFY_ (_CLTV_) was added to the scripting language. ++CLTV+ is a per-output timelock, rather than a per-transaction timelock +as is the case with +nLocktime+. This allows for much greater +flexibility in the way timelocks are applied. + +In simple terms, by adding the +CLTV+ opcode in the redeem script of an +output it restricts the output, so that it can only be spent after the +specified time has elapsed. [TIP] ==== -While +nLocktime+ is a transaction-level timelock, +CLTV+ is an output-based timelock. +While +nLocktime+ is a transaction-level timelock, +CLTV+ is an +output-based timelock. ==== -+CLTV+ doesn't replace +nLocktime+, but rather restricts specific UTXO such that they can only be spent in a future transaction with +nLocktime+ set to a greater or equal value. ++CLTV+ doesn't replace +nLocktime+, but rather restricts specific UTXO +such that they can only be spent in a future transaction with ++nLocktime+ set to a greater or equal value. -The +CLTV+ opcode takes one parameter as input, expressed as a number in the same format as +nLocktime+ (either a block height or Unix epoch time). As indicated by the +VERIFY+ suffix, +CLTV+ is the type of opcode that halts execution of the script if the outcome is +FALSE+. If it results in TRUE, execution continues. +The +CLTV+ opcode takes one parameter as input, expressed as a number in +the same format as +nLocktime+ (either a block height or Unix epoch +time). As indicated by the +VERIFY+ suffix, +CLTV+ is the type of opcode +that halts execution of the script if the outcome is +FALSE+. If it +results in TRUE, execution continues. -In order to lock an output with +CLTV+, you insert it into the redeem script of the output in the transaction that creates the output. For example, if Alice is paying Bob's address, the output would normally contain a P2PKH script like this: +In order to lock an output with +CLTV+, you insert it into the redeem +script of the output in the transaction that creates the output. For +example, if Alice is paying Bob's address, the output would normally +contain a P2PKH script like this: ---- DUP HASH160 EQUALVERIFY CHECKSIG ---- -To lock it to a time, say 3 months from now, the transaction would be a P2SH transaction with a redeem script like this: +To lock it to a time, say 3 months from now, the transaction would be a +P2SH transaction with a redeem script like this: ---- CHECKLOCKTIMEVERIFY DROP DUP HASH160 EQUALVERIFY CHECKSIG ---- -where ++ is a block height or time value estimated 3 months from the time the transaction is mined: current block height {plus} 12,960 (blocks) or current Unix epoch time {plus} 7,760,000 (seconds). For now, don't worry about the +DROP+ opcode that follows +CHECKLOCKTIMEVERIFY+; it will be explained shortly. +where ++ is a block height or time value estimated +3 months from the time the transaction is mined: current block height +{plus} 12,960 (blocks) or current Unix epoch time {plus} 7,760,000 +(seconds). For now, don't worry about the +DROP+ opcode that follows ++CHECKLOCKTIMEVERIFY+; it will be explained shortly. -When Bob tries to spend this UTXO, he constructs a transaction that references the UTXO as an input. He uses his signature and public key in the unlocking script of that input and sets the transaction +nLocktime+ to be equal or greater to the timelock in the +CHECKLOCKTIMEVERIFY+ Alice set. Bob then broadcasts the transaction on the Bitcoin network. +When Bob tries to spend this UTXO, he constructs a transaction that +references the UTXO as an input. He uses his signature and public key in +the unlocking script of that input and sets the transaction +nLocktime+ +to be equal or greater to the timelock in the +CHECKLOCKTIMEVERIFY+ +Alice set. Bob then broadcasts the transaction on the Bitcoin network. -Bob's transaction is evaluated as follows. If the +CHECKLOCKTIMEVERIFY+ parameter Alice set is less than or equal the spending transaction's +nLocktime+, script execution continues (acts as if a “no operation” or NOP opcode was executed). Otherwise, script execution halts and the transaction is deemed invalid. +Bob's transaction is evaluated as follows. If the +CHECKLOCKTIMEVERIFY+ +parameter Alice set is less than or equal the spending transaction's ++nLocktime+, script execution continues (acts as if a “no +operation” or NOP opcode was executed). Otherwise, script +execution halts and the transaction is deemed invalid. -More precisely, +CHECKLOCKTIMEVERIFY+ fails and halts execution, marking the transaction invalid if (source: BIP-65): +More precisely, +CHECKLOCKTIMEVERIFY+ fails and halts execution, marking +the transaction invalid if (source: BIP-65): 1. the stack is empty; or 1. the top item on the stack is less than 0; or @@ -324,115 +660,302 @@ More precisely, +CHECKLOCKTIMEVERIFY+ fails and halts execution, marking the tra [NOTE] ==== -+CLTV+ and +nLocktime+ use the same format to describe timelocks, either a block height or the time elapsed in seconds since Unix epoch. Critically, when used together, the format of +nLocktime+ must match that of +CLTV+ in the outputs—they must both reference either block height or time in seconds. ++CLTV+ and +nLocktime+ use the same format to describe timelocks, either +a block height or the time elapsed in seconds since Unix epoch. +Critically, when used together, the format of +nLocktime+ must match +that of +CLTV+ in the outputs—they must both reference either +block height or time in seconds. ==== -After execution, if +CLTV+ is satisfied, the time parameter that preceded it remains as the top item on the stack and may need to be dropped, with +DROP+, for correct execution of subsequent script opcodes. You will often see +CHECKLOCKTIMEVERIFY+ followed by +DROP+ in scripts for this reason. +After execution, if +CLTV+ is satisfied, the time parameter that +preceded it remains as the top item on the stack and may need to be +dropped, with +DROP+, for correct execution of subsequent script +opcodes. You will often see +CHECKLOCKTIMEVERIFY+ followed by +DROP+ in +scripts for this reason. -By using nLocktime in conjunction with +CLTV+, the scenario described in <> changes. Alice can no longer spend the money (because it's locked with Bob's key) and Bob cannot spend it before the 3-month locktime has expired.((("", startref="alicesseven"))) +By using nLocktime in conjunction with +CLTV+, the scenario described in +<> changes. Alice can no longer spend the money +(because it's locked with Bob's key) and Bob cannot spend it before the +3-month locktime has expired.((("", startref="alicesseven"))) -By introducing timelock functionality directly into the scripting language, +CLTV+ allows us to develop some very interesting complex scripts.((("", startref="cltv07"))) +By introducing timelock functionality directly into the scripting +language, +CLTV+ allows us to develop some very interesting complex +scripts.((("", startref="cltv07"))) -The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki[BIP-65 (CHECKLOCKTIMEVERIFY)]. +The standard is defined in +https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki[BIP-65 +(CHECKLOCKTIMEVERIFY)]. ==== Relative Timelocks -+nLocktime+ and +CLTV+ are ((("timelocks", "relative timelocks", id="Trelative07")))((("scripting", "timelocks", "relative timelocks")))((("relative timelocks", id="relativetime07")))both _absolute timelocks_ in that they specify an absolute point in time. The next two timelock features we will examine are _relative timelocks_ in that they specify, as a condition of spending an output, an elapsed time from the confirmation of the output in the blockchain. - -Relative timelocks are useful because they allow a chain of two or more interdependent transactions to be held off chain, while imposing a time constraint on one transaction that is dependent on the elapsed time from the confirmation of a previous transaction. In other words, the clock doesn't start counting until the UTXO is recorded on the blockchain. This functionality is especially useful in bidirectional state channels and Lightning Networks, as we will see in <>. - -Relative timelocks, like absolute timelocks, are implemented with both a transaction-level feature and a script-level opcode. The transaction-level relative timelock is implemented as a consensus rule on the value of +nSequence+, a transaction field that is set in every transaction input. Script-level relative timelocks are implemented with the +CHECKSEQUENCEVERIFY+ (CSV) opcode. - -((("bitcoin improvement proposals", "Relative lock-time using consensus-enforced sequence numbers (BIP-68)")))((("bitcoin improvement proposals", "CHECKSEQUENCEVERIFY (BIP-112)")))Relative timelocks are implemented according to the specifications in https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68, Relative lock-time using consensus-enforced sequence numbers] and https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[BIP-112, CHECKSEQUENCEVERIFY]. - -BIP-68 and BIP-112 were activated in May 2016 as a soft fork upgrade to the consensus rules. ++nLocktime+ and +CLTV+ are ((("timelocks", "relative timelocks", +id="Trelative07")))((("scripting", "timelocks", "relative +timelocks")))((("relative timelocks", id="relativetime07")))both +_absolute timelocks_ in that they specify an absolute point in time. The +next two timelock features we will examine are _relative timelocks_ in +that they specify, as a condition of spending an output, an elapsed time +from the confirmation of the output in the blockchain. + +Relative timelocks are useful because they allow a chain of two or more +interdependent transactions to be held off chain, while imposing a time +constraint on one transaction that is dependent on the elapsed time from +the confirmation of a previous transaction. In other words, the clock +doesn't start counting until the UTXO is recorded on the blockchain. +This functionality is especially useful in bidirectional state channels +and Lightning Networks, as we will see in <>. + +Relative timelocks, like absolute timelocks, are implemented with both a +transaction-level feature and a script-level opcode. The +transaction-level relative timelock is implemented as a consensus rule +on the value of +nSequence+, a transaction field that is set in every +transaction input. Script-level relative timelocks are implemented with +the +CHECKSEQUENCEVERIFY+ (CSV) opcode. + +((("bitcoin improvement proposals", "Relative lock-time using +consensus-enforced sequence numbers (BIP-68)")))((("bitcoin improvement +proposals", "CHECKSEQUENCEVERIFY (BIP-112)")))Relative timelocks are +implemented according to the specifications in +https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68, +Relative lock-time using consensus-enforced sequence numbers] and +https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[BIP-112, +CHECKSEQUENCEVERIFY]. + +BIP-68 and BIP-112 were activated in May 2016 as a soft fork upgrade to +the consensus rules. ==== Relative Timelocks with nSequence -((("nSequence field")))((("scripting", "timelocks", "relative timelocks with nSequence")))Relative timelocks can be set on each input of a transaction, by setting the +nSequence+ field in each input. +((("nSequence field")))((("scripting", "timelocks", "relative timelocks +with nSequence")))Relative timelocks can be set on each input of a +transaction, by setting the +nSequence+ field in each input. ===== Original meaning of nSequence -The +nSequence+ field was originally intended (but never properly implemented) to allow modification of transactions in the mempool. In that use, a transaction containing inputs with +nSequence+ value below 2^32^ - 1 (0xFFFFFFFF) indicated a transaction that was not yet "finalized." Such a transaction would be held in the mempool until it was replaced by another transaction spending the same inputs with a higher +nSequence+ value. Once a transaction was received whose inputs had an +nSequence+ value of 0xFFFFFFFF it would be considered "finalized" and mined. - -The original meaning of +nSequence+ was never properly implemented and the value of +nSequence+ is customarily set to 0xFFFFFFFF in transactions that do not utilize timelocks. For transactions with nLocktime or +CHECKLOCKTIMEVERIFY+, the +nSequence+ value must be set to less than 2^31^ for the timelock guards to have an effect, as explained below. +The +nSequence+ field was originally intended (but never properly +implemented) to allow modification of transactions in the mempool. In +that use, a transaction containing inputs with +nSequence+ value below +2^32^ - 1 (0xFFFFFFFF) indicated a transaction that was not yet +"finalized." Such a transaction would be held in the mempool until it +was replaced by another transaction spending the same inputs with a +higher +nSequence+ value. Once a transaction was received whose inputs +had an +nSequence+ value of 0xFFFFFFFF it would be considered +"finalized" and mined. + +The original meaning of +nSequence+ was never properly implemented and +the value of +nSequence+ is customarily set to 0xFFFFFFFF in +transactions that do not utilize timelocks. For transactions with +nLocktime or +CHECKLOCKTIMEVERIFY+, the +nSequence+ value must be set to +less than 2^31^ for the timelock guards to have an effect, as explained +below. ===== nSequence as a consensus-enforced relative timelock -Since the activation of BIP-68, new consensus rules apply for any transaction containing an input whose +nSequence+ value is less than 2^31^ (bit 1<<31 is not set). Programmatically, that means that if the most significant (bit 1<<31) is not set, it is a flag that means "relative locktime." Otherwise (bit 1<<31 set), the +nSequence+ value is reserved for other uses such as enabling +CHECKLOCKTIMEVERIFY+, +nLocktime+, Opt-In-Replace-By-Fee, and other future developments. - -Transaction inputs with +nSequence+ values less than 2^31^ are interpreted as having a relative timelock. Such a transaction is only valid once the input has aged by the relative timelock amount. For example, a transaction with one input with an +nSequence+ relative timelock of 30 blocks is only valid when at least 30 blocks have elapsed from the time the UTXO referenced in the input was mined. Since +nSequence+ is a per-input field, a transaction may contain any number of timelocked inputs, all of which must have sufficiently aged for the transaction to be valid. A transaction can include both timelocked inputs (+nSequence+ < 2^31^) and inputs without a relative timelock (+nSequence+ >= 2^31^). - -The +nSequence+ value is specified in either blocks or seconds, but in a slightly different format than we saw used in +nLocktime+. A type-flag is used to differentiate between values counting blocks and values counting time in seconds. The type-flag is set in the 23rd least-significant bit (i.e., value 1<<22). If the type-flag is set, then the +nSequence+ value is interpreted as a multiple of 512 seconds. If the type-flag is not set, the +nSequence+ value is interpreted as a number of blocks. - -When interpreting +nSequence+ as a relative timelock, only the 16 least significant bits are considered. Once the flags (bits 32 and 23) are evaluated, the +nSequence+ value is usually "masked" with a 16-bit mask (e.g., +nSequence+ & 0x0000FFFF). - -<> shows the binary layout of the +nSequence+ value, as defined by BIP-68. +Since the activation of BIP-68, new consensus rules apply for any +transaction containing an input whose +nSequence+ value is less than +2^31^ (bit 1<<31 is not set). Programmatically, that means that if the +most significant (bit 1<<31) is not set, it is a flag that means +"relative locktime." Otherwise (bit 1<<31 set), the +nSequence+ value is +reserved for other uses such as enabling +CHECKLOCKTIMEVERIFY+, ++nLocktime+, Opt-In-Replace-By-Fee, and other future developments. + +Transaction inputs with +nSequence+ values less than 2^31^ are +interpreted as having a relative timelock. Such a transaction is only +valid once the input has aged by the relative timelock amount. For +example, a transaction with one input with an +nSequence+ relative +timelock of 30 blocks is only valid when at least 30 blocks have elapsed +from the time the UTXO referenced in the input was mined. Since ++nSequence+ is a per-input field, a transaction may contain any number +of timelocked inputs, all of which must have sufficiently aged for the +transaction to be valid. A transaction can include both timelocked +inputs (+nSequence+ < 2^31^) and inputs without a relative timelock +(+nSequence+ >= 2^31^). + +The +nSequence+ value is specified in either blocks or seconds, but in a +slightly different format than we saw used in +nLocktime+. A type-flag +is used to differentiate between values counting blocks and values +counting time in seconds. The type-flag is set in the 23rd +least-significant bit (i.e., value 1<<22). If the type-flag is set, then +the +nSequence+ value is interpreted as a multiple of 512 seconds. If +the type-flag is not set, the +nSequence+ value is interpreted as a +number of blocks. + +When interpreting +nSequence+ as a relative timelock, only the 16 least +significant bits are considered. Once the flags (bits 32 and 23) are +evaluated, the +nSequence+ value is usually "masked" with a 16-bit mask +(e.g., +nSequence+ & 0x0000FFFF). + +<> shows the binary layout of the +nSequence+ value, +as defined by BIP-68. [[bip_68_def_of_nseq]] .BIP-68 definition of nSequence encoding (Source: BIP-68) image::images/mbc2_0701.png["BIP-68 definition of nSequence encoding"] +Relative timelocks based on consensus enforcement of the +nSequence+ +value are defined in BIP-68. -Relative timelocks based on consensus enforcement of the +nSequence+ value are defined in BIP-68. - -The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68, Relative lock-time using consensus-enforced sequence numbers]. +The standard is defined in +https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68, +Relative lock-time using consensus-enforced sequence numbers]. ==== Relative Timelocks with CSV -((("scripting", "timelocks", "relative timelocks with CHECKSEQUENCEVERIFY")))((("CHECKSEQUENCEVERIFY (CSV)")))Just like CLTV and +nLocktime+, there is a script opcode for relative timelocks that leverages the +nSequence+ value in scripts. That opcode is +CHECKSEQUENCEVERIFY+, commonly referred to as +CSV+ for short. - -The +CSV+ opcode when evaluated in an UTXO's redeem script allows spending only in a transaction whose input +nSequence+ value is greater than or equal to the +CSV+ parameter. Essentially, this restricts spending the UTXO until a certain number of blocks or seconds have elapsed relative to the time the UTXO was mined. - -As with CLTV, the value in +CSV+ must match the format in the corresponding +nSequence+ value. If +CSV+ is specified in terms of blocks, then so must +nSequence+. If +CSV+ is specified in terms of seconds, then so must +nSequence+. - -Relative timelocks with +CSV+ are especially useful when several (chained) transactions are created and signed, but not propagated, when they're kept "off-chain." A child transaction cannot be used until the parent transaction has been propagated, mined, and aged by the time specified in the relative timelock. One application of this use case can be seen in <> and <>.((("", startref="relativetime07")))((("", startref="Trelative07"))) - -+CSV+ is defined in detail in https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[BIP-112, CHECKSEQUENCEVERIFY]. - +((("scripting", "timelocks", "relative timelocks with +CHECKSEQUENCEVERIFY")))((("CHECKSEQUENCEVERIFY (CSV)")))Just like CLTV +and +nLocktime+, there is a script opcode for relative timelocks that +leverages the +nSequence+ value in scripts. That opcode is ++CHECKSEQUENCEVERIFY+, commonly referred to as +CSV+ for short. + +The +CSV+ opcode when evaluated in an UTXO's redeem script allows +spending only in a transaction whose input +nSequence+ value is greater +than or equal to the +CSV+ parameter. Essentially, this restricts +spending the UTXO until a certain number of blocks or seconds have +elapsed relative to the time the UTXO was mined. + +As with CLTV, the value in +CSV+ must match the format in the +corresponding +nSequence+ value. If +CSV+ is specified in terms of +blocks, then so must +nSequence+. If +CSV+ is specified in terms of +seconds, then so must +nSequence+. + +Relative timelocks with +CSV+ are especially useful when several +(chained) transactions are created and signed, but not propagated, when +they're kept "off-chain." A child transaction cannot be used until the +parent transaction has been propagated, mined, and aged by the time +specified in the relative timelock. One application of this use case can +be seen in <> and <>.((("", +startref="relativetime07")))((("", startref="Trelative07"))) + ++CSV+ is defined in detail in +https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[BIP-112, +CHECKSEQUENCEVERIFY]. ==== Median-Time-Past -((("scripting", "timelocks", "Median-Tme-Past")))((("Median-Tme-Past")))((("timelocks", "Median-Tme-Past")))As part of the activation of relative timelocks, there was also a change in the way "time" is calculated for timelocks (both absolute and relative). In bitcoin there is a subtle, but very significant, difference between wall time and consensus time. Bitcoin is a decentralized network, which means that each participant has his or her own perspective of time. Events on the network do not occur instantaneously everywhere. Network latency must be factored into the perspective of each node. Eventually everything is synchronized to create a common ledger. Bitcoin reaches consensus every 10 minutes about the state of the ledger as it existed in the _past_. - -The timestamps set in block headers are set by the miners. There is a certain degree of latitude allowed by the consensus rules to account for differences in clock accuracy between decentralized nodes. However, this creates an unfortunate incentive for miners to lie about the time in a block so as to earn extra fees by including timelocked transactions that are not yet mature. See the following section for more information. - -To remove the incentive to lie and strengthen the security of timelocks, a BIP was proposed and activated at the same time as the BIPs for relative timelocks. This is BIP-113, which defines a new consensus measurement of time called _Median-Time-Past_. - -Median-Time-Past is calculated by taking the timestamps of the last 11 blocks and finding the median. That median time then becomes consensus time and is used for all timelock calculations. By taking the midpoint from approximately two hours in the past, the influence of any one block's timestamp is reduced. By incorporating 11 blocks, no single miner can influence the timestamps in order to gain fees from transactions with a timelock that hasn't yet matured. - -Median-Time-Past changes the implementation of time calculations for +nLocktime+, +CLTV+, +nSequence+, and +CSV+. The consensus time calculated by Median-Time-Past is always approximately one hour behind wall clock time. If you create timelock transactions, you should account for it when estimating the desired value to encode in +nLocktime+, +nSequence+, +CLTV+, and +CSV+. - -Median-Time-Past is specified in https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki[BIP-113]. +((("scripting", "timelocks", +"Median-Tme-Past")))((("Median-Tme-Past")))((("timelocks", +"Median-Tme-Past")))As part of the activation of relative timelocks, +there was also a change in the way "time" is calculated for timelocks +(both absolute and relative). In bitcoin there is a subtle, but very +significant, difference between wall time and consensus time. Bitcoin is +a decentralized network, which means that each participant has his or +her own perspective of time. Events on the network do not occur +instantaneously everywhere. Network latency must be factored into the +perspective of each node. Eventually everything is synchronized to +create a common ledger. Bitcoin reaches consensus every 10 minutes about +the state of the ledger as it existed in the _past_. + +The timestamps set in block headers are set by the miners. There is a +certain degree of latitude allowed by the consensus rules to account for +differences in clock accuracy between decentralized nodes. However, this +creates an unfortunate incentive for miners to lie about the time in a +block so as to earn extra fees by including timelocked transactions that +are not yet mature. See the following section for more information. + +To remove the incentive to lie and strengthen the security of timelocks, +a BIP was proposed and activated at the same time as the BIPs for +relative timelocks. This is BIP-113, which defines a new consensus +measurement of time called _Median-Time-Past_. + +Median-Time-Past is calculated by taking the timestamps of the last 11 +blocks and finding the median. That median time then becomes consensus +time and is used for all timelock calculations. By taking the midpoint +from approximately two hours in the past, the influence of any one +block's timestamp is reduced. By incorporating 11 blocks, no single +miner can influence the timestamps in order to gain fees from +transactions with a timelock that hasn't yet matured. + +Median-Time-Past changes the implementation of time calculations for ++nLocktime+, +CLTV+, +nSequence+, and +CSV+. The consensus time +calculated by Median-Time-Past is always approximately one hour behind +wall clock time. If you create timelock transactions, you should account +for it when estimating the desired value to encode in +nLocktime+, ++nSequence+, +CLTV+, and +CSV+. + +Median-Time-Past is specified in +https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki[BIP-113]. [[fee_sniping]] ==== Timelock Defense Against Fee Sniping -((("scripting", "timelocks", "defense against fee-sniping")))((("timelocks", "defense against fee-sniping")))((("fees", "fee sniping")))((("security", "defense against fee-sniping")))((("sniping")))Fee-sniping is a theoretical attack scenario, where miners attempting to rewrite past blocks "snipe" higher-fee transactions from future blocks to maximize their profitability. - -For example, let's say the highest block in existence is block #100,000. If instead of attempting to mine block #100,001 to extend the chain, some miners attempt to remine #100,000. These miners can choose to include any valid transaction (that hasn't been mined yet) in their candidate block #100,000. They don't have to remine the block with the same transactions. In fact, they have the incentive to select the most profitable (highest fee per kB) transactions to include in their block. They can include any transactions that were in the "old" block #100,000, as well as any transactions from the current mempool. Essentially they have the option to pull transactions from the "present" into the rewritten "past" when they re-create block #100,000. - -Today, this attack is not very lucrative, because block reward is much higher than total fees per block. But at some point in the future, transaction fees will be the majority of the reward (or even the entirety of the reward). At that time, this scenario becomes inevitable. - -To prevent "fee sniping," when Bitcoin Core creates transactions, it uses +nLocktime+ to limit them to the "next block," by default. In our scenario, Bitcoin Core would set +nLocktime+ to 100,001 on any transaction it created. Under normal circumstances, this +nLocktime+ has no effect—the transactions could only be included in block #100,001 anyway; it's the next block. - -But under a blockchain fork attack, the miners would not be able to pull high-fee transactions from the mempool, because all those transactions would be timelocked to block #100,001. They can only remine #100,000 with whatever transactions were valid at that time, essentially gaining no new fees. - -To achieve this, Bitcoin Core sets the +nLocktime+ on all new transactions to and sets the +nSequence+ on all the inputs to 0xFFFFFFFE to enable +nLocktime+.((("", startref="Stimelock07"))) +((("scripting", "timelocks", "defense against +fee-sniping")))((("timelocks", "defense against +fee-sniping")))((("fees", "fee sniping")))((("security", "defense +against fee-sniping")))((("sniping")))Fee-sniping is a theoretical +attack scenario, where miners attempting to rewrite past blocks "snipe" +higher-fee transactions from future blocks to maximize their +profitability. + +For example, let's say the highest block in existence is block +#100,000. If instead of attempting to mine block #100,001 to extend the +chain, some miners attempt to remine #100,000. These miners can choose +to include any valid transaction (that hasn't been mined yet) in their +candidate block #100,000. They don't have to remine the block with the +same transactions. In fact, they have the incentive to select the most +profitable (highest fee per kB) transactions to include in their block. +They can include any transactions that were in the "old" block +#100,000, as well as any transactions from the current mempool. +Essentially they have the option to pull transactions from the "present" +into the rewritten "past" when they re-create block #100,000. + +Today, this attack is not very lucrative, because block reward is much +higher than total fees per block. But at some point in the future, +transaction fees will be the majority of the reward (or even the +entirety of the reward). At that time, this scenario becomes inevitable. + +To prevent "fee sniping," when Bitcoin Core creates transactions, it +uses +nLocktime+ to limit them to the "next block," by default. In our +scenario, Bitcoin Core would set +nLocktime+ to 100,001 on any +transaction it created. Under normal circumstances, this +nLocktime+ has +no effect—the transactions could only be included in block +#100,001 anyway; it's the next block. + +But under a blockchain fork attack, the miners would not be able to pull +high-fee transactions from the mempool, because all those transactions +would be timelocked to block #100,001. They can only remine #100,000 +with whatever transactions were valid at that time, essentially gaining +no new fees. + +To achieve this, Bitcoin Core sets the +nLocktime+ on all new +transactions to and sets the +nSequence+ on all +the inputs to 0xFFFFFFFE to enable +nLocktime+.((("", +startref="Stimelock07"))) === Scripts with Flow Control (Conditional Clauses) -((("transactions", "advanced", "flow control scripts")))((("scripting", "flow control scripts", id="Sflow07")))((("conditional clauses", id="condition07")))((("flow control", id="flow07")))One of the more powerful features of Bitcoin Script is flow control, also known as conditional clauses. You are probably familiar with flow control in various programming languages that use the construct +IF...THEN...ELSE+. Bitcoin conditional clauses look a bit different, but are essentially the same construct. - -At a basic level, bitcoin conditional opcodes allow us to construct a redeem script that has two ways of being unlocked, depending on a +TRUE+/+FALSE+ outcome of evaluating a logical condition. For example, if x is +TRUE+, the redeem script is A and the ELSE redeem script is B. - -Additionally, bitcoin conditional expressions can be "nested" indefinitely, meaning that a conditional clause can contain another within it, which contains another, etc. Bitcoin Script flow control can be used to construct very complex scripts with hundreds or even thousands of possible execution paths. There is no limit to nesting, but consensus rules impose a limit on the maximum size, in bytes, of a script. - -Bitcoin implements flow control using the +IF+, +ELSE+, +ENDIF+, and +NOTIF+ opcodes. Additionally, conditional expressions can contain boolean operators such as +BOOLAND+, pass:[BOOLOR], and +NOT+. - -At first glance, you may find the bitcoin's flow control scripts confusing. That is because Bitcoin Script is a stack language. The same way that +1 {plus} 1+ looks "backward" when expressed as +1 1 ADD+, flow control clauses in bitcoin also look "backward." - -In most traditional (procedural) programming languages, flow control looks like this: +((("transactions", "advanced", "flow control scripts")))((("scripting", +"flow control scripts", id="Sflow07")))((("conditional clauses", +id="condition07")))((("flow control", id="flow07")))One of the more +powerful features of Bitcoin Script is flow control, also known as +conditional clauses. You are probably familiar with flow control in +various programming languages that use the construct +IF...THEN...ELSE+. +Bitcoin conditional clauses look a bit different, but are essentially +the same construct. + +At a basic level, bitcoin conditional opcodes allow us to construct a +redeem script that has two ways of being unlocked, depending on a ++TRUE+/+FALSE+ outcome of evaluating a logical condition. For example, +if x is +TRUE+, the redeem script is A and the ELSE redeem script is B. + +Additionally, bitcoin conditional expressions can be "nested" +indefinitely, meaning that a conditional clause can contain another +within it, which contains another, etc. Bitcoin Script flow control can +be used to construct very complex scripts with hundreds or even +thousands of possible execution paths. There is no limit to nesting, but +consensus rules impose a limit on the maximum size, in bytes, of a +script. + +Bitcoin implements flow control using the +IF+, +ELSE+, +ENDIF+, and ++NOTIF+ opcodes. Additionally, conditional expressions can contain +boolean operators such as +BOOLAND+, pass:[BOOLOR], and +NOT+. + +At first glance, you may find the bitcoin's flow control scripts +confusing. That is because Bitcoin Script is a stack language. The same +way that +1 {plus} 1+ looks "backward" when expressed as +1 1 ADD+, flow +control clauses in bitcoin also look "backward." + +In most traditional (procedural) programming languages, flow control +looks like this: .Pseudocode of flow control in most programming languages ---- @@ -443,7 +966,8 @@ else: code to run in either case ---- -In a stack-based language like Bitcoin Script, the logical condition comes before the +IF+, which makes it look "backward," like this: +In a stack-based language like Bitcoin Script, the logical condition +comes before the +IF+, which makes it look "backward," like this: .Bitcoin Script flow control ---- @@ -456,29 +980,40 @@ ENDIF code to run in either case ---- -When reading Bitcoin Script, remember that the condition being evaluated comes _before_ the +IF+ opcode. +When reading Bitcoin Script, remember that the condition being evaluated +comes _before_ the +IF+ opcode. ==== Conditional Clauses with VERIFY Opcodes -((("VERIFY opcodes")))((("IF clauses")))((("opcodes", "VERIFY")))Another form of conditional in Bitcoin Script is any opcode that ends in +VERIFY+. The +VERIFY+ suffix means that if the condition evaluated is not +TRUE+, execution of the script terminates immediately and the transaction is deemed invalid. +((("VERIFY opcodes")))((("IF clauses")))((("opcodes", "VERIFY")))Another +form of conditional in Bitcoin Script is any opcode that ends in ++VERIFY+. The +VERIFY+ suffix means that if the condition evaluated is +not +TRUE+, execution of the script terminates immediately and the +transaction is deemed invalid. -((("guard clauses")))Unlike an +IF+ clause, which offers alternative execution paths, the +VERIFY+ suffix acts as a _guard clause_, continuing only if a precondition is met. +((("guard clauses")))Unlike an +IF+ clause, which offers alternative +execution paths, the +VERIFY+ suffix acts as a _guard clause_, +continuing only if a precondition is met. -For example, the following script requires Bob's signature and a pre-image (secret) that produces a specific hash. Both conditions must be satisfied to unlock: +For example, the following script requires Bob's signature and a +pre-image (secret) that produces a specific hash. Both conditions must +be satisfied to unlock: .A redeem script with an +EQUALVERIFY+ guard clause. ---- HASH160 EQUALVERIFY CHECKSIG ---- -To redeem this, Bob must construct an unlocking script that presents a valid pre-image and a signature: +To redeem this, Bob must construct an unlocking script that presents a +valid pre-image and a signature: .An unlocking script to satisfy the above redeem script ---- ---- -Without presenting the pre-image, Bob can't get to the part of the script that checks for his signature. +Without presenting the pre-image, Bob can't get to the part of the +script that checks for his signature. [role="pagebreak-after"] This script can be written with an +IF+ instead: @@ -498,20 +1033,36 @@ Bob's unlocking script is identical: ---- -The script with +IF+ does the same thing as using an opcode with a +VERIFY+ suffix; they both operate as guard clauses. However, the +VERIFY+ construction is more efficient, using two fewer opcodes. +The script with +IF+ does the same thing as using an opcode with a ++VERIFY+ suffix; they both operate as guard clauses. However, the ++VERIFY+ construction is more efficient, using two fewer opcodes. -So, when do we use +VERIFY+ and when do we use +IF+? If all we are trying to do is to attach a precondition (guard clause), then +VERIFY+ is better. If, however, we want to have more than one execution path (flow control), then we need an +IF...ELSE+ flow control clause. +So, when do we use +VERIFY+ and when do we use +IF+? If all we are +trying to do is to attach a precondition (guard clause), then +VERIFY+ +is better. If, however, we want to have more than one execution path +(flow control), then we need an +IF...ELSE+ flow control clause. [TIP] ==== -((("EQUAL opcode")))((("opcodes", "EQUAL")))((("EQUALVERIFY opcode")))((("opcodes", "EQUALVERIFY")))An opcode such as +EQUAL+ will push the result (+TRUE+/+FALSE+) onto the stack, leaving it there for evaluation by subsequent opcodes. In contrast, the opcode +EQUALVERIFY+ suffix does not leave anything on the stack. Opcodes that end in +VERIFY+ do not leave the result on the stack. +((("EQUAL opcode")))((("opcodes", "EQUAL")))((("EQUALVERIFY +opcode")))((("opcodes", "EQUALVERIFY")))An opcode such as +EQUAL+ will +push the result (+TRUE+/+FALSE+) onto the stack, leaving it there for +evaluation by subsequent opcodes. In contrast, the opcode +EQUALVERIFY+ +suffix does not leave anything on the stack. Opcodes that end in ++VERIFY+ do not leave the result on the stack. ==== ==== Using Flow Control in Scripts -A very common use for flow control in Bitcoin Script is to construct a redeem script that offers multiple execution paths, each a different way of redeeming the UTXO. +A very common use for flow control in Bitcoin Script is to construct a +redeem script that offers multiple execution paths, each a different way +of redeeming the UTXO. -((("use cases", "buying coffee")))Let's look at a simple example, where we have two signers, Alice and Bob, and either one is able to redeem. With multisig, this would be expressed as a 1-of-2 multisig script. For the sake of demonstration, we will do the same thing with an +IF+ clause: +((("use cases", "buying coffee")))Let's look at a simple example, where +we have two signers, Alice and Bob, and either one is able to redeem. +With multisig, this would be expressed as a 1-of-2 multisig script. For +the sake of demonstration, we will do the same thing with an +IF+ +clause: ---- IF @@ -521,26 +1072,35 @@ ELSE ENDIF ---- -Looking at this redeem script, you may be wondering: "Where is the condition? There is nothing preceding the +IF+ clause!" +Looking at this redeem script, you may be wondering: "Where is the +condition? There is nothing preceding the +IF+ clause!" -The condition is not part of the redeem script. Instead, the condition will be offered in the unlocking script, allowing Alice and Bob to "choose" which execution path they want. +The condition is not part of the redeem script. Instead, the condition +will be offered in the unlocking script, allowing Alice and Bob to +"choose" which execution path they want. Alice redeems this with the unlocking script: ---- 1 ---- -The +1+ at the end serves as the condition (+TRUE+) that will make the +IF+ clause execute the first redemption path for which Alice has a signature. +The +1+ at the end serves as the condition (+TRUE+) that will make the ++IF+ clause execute the first redemption path for which Alice has a +signature. -For Bob to redeem this, he would have to choose the second execution path by giving a +FALSE+ value to the +IF+ clause: +For Bob to redeem this, he would have to choose the second execution +path by giving a +FALSE+ value to the +IF+ clause: ---- 0 ---- -Bob's unlocking script puts a +0+ on the stack, causing the +IF+ clause to execute the second (+ELSE+) script, which requires Bob's signature. +Bob's unlocking script puts a +0+ on the stack, causing the +IF+ clause +to execute the second (+ELSE+) script, which requires Bob's signature. -Since +IF+ clauses can be nested, we can create a "maze" of execution paths. The unlocking script can provide a "map" selecting which execution path is actually executed: +Since +IF+ clauses can be nested, we can create a "maze" of execution +paths. The unlocking script can provide a "map" selecting which +execution path is actually executed: ---- IF @@ -554,19 +1114,49 @@ ELSE ENDIF ---- -In this scenario, there are three execution paths (+script A+, +script B+, and +script C+). The unlocking script provides a path in the form of a sequence of +TRUE+ or +FALSE+ values. To select path +script B+, for example, the unlocking script must end in +1 0+ (+TRUE+, +FALSE+). These values will be pushed onto the stack, so that the second value (+FALSE+) ends up at the top of the stack. The outer +IF+ clause pops the +FALSE+ value and executes the first +ELSE+ clause. Then the +TRUE+ value moves to the top of the stack and is evaluated by the inner (nested) +IF+, selecting the +B+ execution path. +In this scenario, there are three execution paths (+script A+, +script +B+, and +script C+). The unlocking script provides a path in the form of +a sequence of +TRUE+ or +FALSE+ values. To select path +script B+, for +example, the unlocking script must end in +1 0+ (+TRUE+, +FALSE+). These +values will be pushed onto the stack, so that the second value (+FALSE+) +ends up at the top of the stack. The outer +IF+ clause pops the +FALSE+ +value and executes the first +ELSE+ clause. Then the +TRUE+ value moves +to the top of the stack and is evaluated by the inner (nested) +IF+, +selecting the +B+ execution path. -Using this construct, we can build redeem scripts with tens or hundreds of execution paths, each offering a different way to redeem the UTXO. To spend, we construct an unlocking script that navigates the execution path by putting the appropriate +TRUE+ and +FALSE+ values on the stack at each flow control point.((("", startref="Sflow07")))((("", startref="flow07")))((("", startref="condition07"))) +Using this construct, we can build redeem scripts with tens or hundreds +of execution paths, each offering a different way to redeem the UTXO. To +spend, we construct an unlocking script that navigates the execution +path by putting the appropriate +TRUE+ and +FALSE+ values on the stack +at each flow control point.((("", startref="Sflow07")))((("", +startref="flow07")))((("", startref="condition07"))) === Complex Script Example -((("transactions", "advanced", "example")))((("scripting", "complex script example", id="Scomplex07")))In this section we combine many of the concepts from this chapter into a single example. - -((("use cases", "import/export", id="mohamseventwo")))Our example uses the story of Mohammed, the company owner in Dubai who is operating an import/export business. - -((("transactions", "advanced", "multisignature scripts")))((("scripting", "multisignature scripts", "import/export example")))((("multisignature scripts")))In this example, Mohammed wishes to construct a company capital account with flexible rules. The scheme he creates requires different levels of authorization depending on timelocks. The participants in the multisig scheme are Mohammed, his two partners Saeed and Zaira, and their company lawyer Abdul. The three partners make decisions based on a majority rule, so two of the three must agree. However, in the case of a problem with their keys, they want their lawyer to be able to recover the funds with one of the three partner signatures. Finally, if all partners are unavailable or incapacitated for a while, they want the lawyer to be able to manage the account directly. - -Here's the redeem script that Mohammed designs to achieve this (line number prefix as XX): +((("transactions", "advanced", "example")))((("scripting", "complex +script example", id="Scomplex07")))In this section we combine many of +the concepts from this chapter into a single example. + +((("use cases", "import/export", id="mohamseventwo")))Our example uses +the story of Mohammed, the company owner in Dubai who is operating an +import/export business. + +((("transactions", "advanced", "multisignature +scripts")))((("scripting", "multisignature scripts", "import/export +example")))((("multisignature scripts")))In this example, Mohammed +wishes to construct a company capital account with flexible rules. The +scheme he creates requires different levels of authorization depending +on timelocks. The participants in the multisig scheme are Mohammed, his +two partners Saeed and Zaira, and their company lawyer Abdul. The three +partners make decisions based on a majority rule, so two of the three +must agree. However, in the case of a problem with their keys, they want +their lawyer to be able to recover the funds with one of the three +partner signatures. Finally, if all partners are unavailable or +incapacitated for a while, they want the lawyer to be able to manage the +account directly. + +Here's the redeem script that Mohammed designs to achieve this (line +number prefix as XX): .Variable Multi-Signature with Timelock ---- @@ -585,9 +1175,14 @@ Here's the redeem script that Mohammed designs to achieve this (line number pref 13 ENDIF ---- -Mohammed's script implements three execution paths using nested +IF...ELSE+ flow control clauses. +Mohammed's script implements three execution paths using nested ++IF...ELSE+ flow control clauses. -In the first execution path, this script operates as a simple 2-of-3 multisig with the three partners. This execution path consists of lines 3 and 9. Line 3 sets the quorum of the multisig to +2+ (2-of-3). This execution path can be selected by putting +TRUE TRUE+ at the end of the unlocking script: +In the first execution path, this script operates as a simple 2-of-3 +multisig with the three partners. This execution path consists of lines +3 and 9. Line 3 sets the quorum of the multisig to +2+ (2-of-3). This +execution path can be selected by putting +TRUE TRUE+ at the end of the +unlocking script: .Unlocking script for the first execution path (2-of-3 multisig) ---- @@ -597,10 +1192,19 @@ In the first execution path, this script operates as a simple 2-of-3 multisig wi [TIP] ==== -The +0+ at the beginning of this unlocking script is because of a bug in +CHECKMULTISIG+ that pops an extra value from the stack. The extra value is disregarded by the +CHECKMULTISIG+, but it must be present or the script fails. Pushing +0+ (customarily) is a workaround to the bug, as described in <>. +The +0+ at the beginning of this unlocking script is because of a bug in ++CHECKMULTISIG+ that pops an extra value from the stack. The extra value +is disregarded by the +CHECKMULTISIG+, but it must be present or the +script fails. Pushing +0+ (customarily) is a workaround to the bug, as +described in <>. ==== -The second execution path can only be used after 30 days have elapsed from the creation of the UTXO. At that time, it requires the signature of Abdul the lawyer and one of the three partners (a 1-of-3 multisig). This is achieved by line 7, which sets the quorum for the multisig to +1+. To select this execution path, the unlocking script would end in +FALSE TRUE+: +The second execution path can only be used after 30 days have elapsed +from the creation of the UTXO. At that time, it requires the signature +of Abdul the lawyer and one of the three partners (a 1-of-3 multisig). +This is achieved by line 7, which sets the quorum for the multisig to ++1+. To select this execution path, the unlocking script would end in ++FALSE TRUE+: .Unlocking script for the second execution path (Lawyer + 1-of-3) ---- @@ -609,10 +1213,14 @@ The second execution path can only be used after 30 days have elapsed from the c [TIP] ==== -Why +FALSE TRUE+? Isn't that backward? Because the two values are pushed on to the stack, with +FALSE+ pushed first, then +TRUE+ pushed second. +TRUE+ is therefore popped _first_ by the first +IF+ opcode. +Why +FALSE TRUE+? Isn't that backward? Because the two values are pushed +on to the stack, with +FALSE+ pushed first, then +TRUE+ pushed second. ++TRUE+ is therefore popped _first_ by the first +IF+ opcode. ==== -Finally, the third execution path allows Abdul the lawyer to spend the funds alone, but only after 90 days. To select this execution path, the unlocking script has to end in +FALSE+: +Finally, the third execution path allows Abdul the lawyer to spend the +funds alone, but only after 90 days. To select this execution path, the +unlocking script has to end in +FALSE+: .Unlocking script for the third execution path (Lawyer only) ---- @@ -621,32 +1229,60 @@ Finally, the third execution path allows Abdul the lawyer to spend the funds alo Try running the script on paper to see how it behaves on the stack. -A few more things to consider when reading this example. See if you can find the answers: +A few more things to consider when reading this example. See if you can +find the answers: -* Why can't the lawyer redeem the third execution path at any time by selecting it with +FALSE+ on the unlocking script? +* Why can't the lawyer redeem the third execution path at any time by +* selecting it with +FALSE+ on the unlocking script? -* How many execution paths can be used 5, 35, and 105 days, respectively, after the UTXO is mined? +* How many execution paths can be used 5, 35, and 105 days, +* respectively, after the UTXO is mined? -* Are the funds lost if the lawyer loses his key? Does your answer change if 91 days have elapsed? +* Are the funds lost if the lawyer loses his key? Does your answer +* change if 91 days have elapsed? -* How do the partners "reset" the clock every 29 or 89 days to prevent the lawyer from accessing the funds? +* How do the partners "reset" the clock every 29 or 89 days to prevent +* the lawyer from accessing the funds? -* Why do some +CHECKSIG+ opcodes in this script have the +VERIFY+ suffix while others don't?((("", startref="Scomplex07")))((("", startref="mohamseventwo"))) +* Why do some +CHECKSIG+ opcodes in this script have the +VERIFY+ suffix +* while others don't?((("", startref="Scomplex07")))((("", +* startref="mohamseventwo"))) [[segwit]] === Segregated Witness -((("segwit (Segregated Witness)", id="Ssegwit07")))Segregated Witness (segwit) is an upgrade to the bitcoin consensus rules and network protocol, proposed and implemented as a BIP-9 soft-fork that was activated on bitcoin's mainnet on August 1st, 2017. - -In cryptography, the term "witness" is used to describe a solution to a cryptographic puzzle. In bitcoin terms, the witness satisfies a cryptographic condition placed on a unspent transaction output (UTXO). - -In the context of bitcoin, a digital signature is _one type of witness_, but a witness is more broadly any solution that can satisfy the conditions imposed on an UTXO and unlock that UTXO for spending. The term “witness” is a more general term for an “unlocking script” or “scriptSig.” - -Before segwit’s introduction, every input in a transaction was followed by the witness data that unlocked it. The witness data was embedded in the transaction as part of each input. The term _segregated witness_, or _segwit_ for short, simply means separating the signature or unlocking script of a specific output. Think "separate scriptSig," or “separate signature” in the simplest form. - -Segregated Witness therefore is an architectural change to bitcoin that aims to move the witness data from the +scriptSig+ (unlocking script) field of a transaction into a separate _witness_ data structure that accompanies a transaction. Clients may request transaction data with or without the accompanying witness data. - -In this section we will look at some of the benefits of Segregated Witness, describe the mechanism used to deploy and implement this architecture change, and demonstrate the use of Segregated Witness in transactions and addresses. +((("segwit (Segregated Witness)", id="Ssegwit07")))Segregated Witness +(segwit) is an upgrade to the bitcoin consensus rules and network +protocol, proposed and implemented as a BIP-9 soft-fork that was +activated on bitcoin's mainnet on August 1st, 2017. + +In cryptography, the term "witness" is used to describe a solution to a +cryptographic puzzle. In bitcoin terms, the witness satisfies a +cryptographic condition placed on a unspent transaction output (UTXO). + +In the context of bitcoin, a digital signature is _one type of witness_, +but a witness is more broadly any solution that can satisfy the +conditions imposed on an UTXO and unlock that UTXO for spending. The +term “witness” is a more general term for an “unlocking script” or +“scriptSig.” + +Before segwit’s introduction, every input in a transaction was followed +by the witness data that unlocked it. The witness data was embedded in +the transaction as part of each input. The term _segregated witness_, or +_segwit_ for short, simply means separating the signature or unlocking +script of a specific output. Think "separate scriptSig," or “separate +signature” in the simplest form. + +Segregated Witness therefore is an architectural change to bitcoin that +aims to move the witness data from the +scriptSig+ (unlocking script) +field of a transaction into a separate _witness_ data structure that +accompanies a transaction. Clients may request transaction data with or +without the accompanying witness data. + +In this section we will look at some of the benefits of Segregated +Witness, describe the mechanism used to deploy and implement this +architecture change, and demonstrate the use of Segregated Witness in +transactions and addresses. Segregated Witness is defined by the following BIPs: @@ -660,59 +1296,149 @@ https://github.com/bitcoin/bips/blob/master/bip-0145.mediawiki[BIP-145] :: getbl https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki[BIP-173]:: Base32 address format for native v0-16 witness outputs - ==== Why Segregated Witness? -Segregated Witness is an architectural change that has several effects on the scalability, security, economic incentives, and performance of bitcoin: - -Transaction Malleability :: By moving the witness outside the transaction, the transaction hash used as an identifier no longer includes the witness data. Since the witness data is the only part of the transaction that can be modified by a third party (see <>), removing it also removes the opportunity for transaction malleability attacks. With Segregated Witness, transaction hashes become immutable by anyone other than the creator of the transaction, which greatly improves the implementation of many other protocols that rely on advanced bitcoin transaction construction, such as payment channels, chained transactions, and lightning networks. - -Script Versioning :: With the introduction of Segregated Witness scripts, every locking script is preceded by a _script version_ number, similar to how transactions and blocks have version numbers. The addition of a script version number allows the scripting language to be upgraded in a backward-compatible way (i.e., using soft fork upgrades) to introduce new script operands, syntax, or semantics. The ability to upgrade the scripting language in a nondisruptive way will greatly accelerate the rate of innovation in bitcoin. - -Network and Storage Scaling :: The witness data is often a big contributor to the total size of a transaction. More complex scripts such as those used for multisig or payment channels are very large. In some cases these scripts account for the majority (more than 75%) of the data in a transaction. By moving the witness data outside the transaction, Segregated Witness improves bitcoin’s scalability. Nodes can prune the witness data after validating the signatures, or ignore it altogether when doing simplified payment verification. The witness data doesn’t need to be transmitted to all nodes and does not need to be stored on disk by all nodes. - -Signature Verification Optimization :: Segregated Witness upgrades the signature functions (+CHECKSIG+, +CHECKMULTISIG+, etc.) to reduce the algorithm's computational complexity. Before segwit, the algorithm used to produce a signature required a number of hash operations that was proportional to the size of the transaction. Data-hashing computations increased in O(n^2^) with respect to the number of signature operations, introducing a substantial computational burden on all nodes verifying the signature. With segwit, the algorithm is changed to reduce the complexity to O(n). - -Offline Signing Improvement :: Segregated Witness signatures incorporate the value (amount) referenced by each input in the hash that is signed. Previously, an offline signing device, such as a hardware wallet, would have to verify the amount of each input before signing a transaction. This was usually accomplished by streaming a large amount of data about the previous transactions referenced as inputs. Since the amount is now part of the commitment hash that is signed, an offline device does not need the previous transactions. If the amounts do not match (are misrepresented by a compromised online system), the signature will be invalid. +Segregated Witness is an architectural change that has several effects +on the scalability, security, economic incentives, and performance of +bitcoin: + +Transaction Malleability :: By moving the witness outside the +transaction, the transaction hash used as an identifier no longer +includes the witness data. Since the witness data is the only part of +the transaction that can be modified by a third party (see +<>), removing it also removes the opportunity for +transaction malleability attacks. With Segregated Witness, transaction +hashes become immutable by anyone other than the creator of the +transaction, which greatly improves the implementation of many other +protocols that rely on advanced bitcoin transaction construction, such +as payment channels, chained transactions, and lightning networks. + +Script Versioning :: With the introduction of Segregated Witness +scripts, every locking script is preceded by a _script version_ number, +similar to how transactions and blocks have version numbers. The +addition of a script version number allows the scripting language to be +upgraded in a backward-compatible way (i.e., using soft fork upgrades) +to introduce new script operands, syntax, or semantics. The ability to +upgrade the scripting language in a nondisruptive way will greatly +accelerate the rate of innovation in bitcoin. + +Network and Storage Scaling :: The witness data is often a big +contributor to the total size of a transaction. More complex scripts +such as those used for multisig or payment channels are very large. In +some cases these scripts account for the majority (more than 75%) of the +data in a transaction. By moving the witness data outside the +transaction, Segregated Witness improves bitcoin’s scalability. Nodes +can prune the witness data after validating the signatures, or ignore it +altogether when doing simplified payment verification. The witness data +doesn’t need to be transmitted to all nodes and does not need to be +stored on disk by all nodes. + +Signature Verification Optimization :: Segregated Witness upgrades the +signature functions (+CHECKSIG+, +CHECKMULTISIG+, etc.) to reduce the +algorithm's computational complexity. Before segwit, the algorithm used +to produce a signature required a number of hash operations that was +proportional to the size of the transaction. Data-hashing computations +increased in O(n^2^) with respect to the number of signature operations, +introducing a substantial computational burden on all nodes verifying +the signature. With segwit, the algorithm is changed to reduce the +complexity to O(n). + +Offline Signing Improvement :: Segregated Witness signatures incorporate +the value (amount) referenced by each input in the hash that is signed. +Previously, an offline signing device, such as a hardware wallet, would +have to verify the amount of each input before signing a transaction. +This was usually accomplished by streaming a large amount of data about +the previous transactions referenced as inputs. Since the amount is now +part of the commitment hash that is signed, an offline device does not +need the previous transactions. If the amounts do not match (are +misrepresented by a compromised online system), the signature will be +invalid. ==== How Segregated Witness Works -At first glance, Segregated Witness appears to be a change to how transactions are constructed and therefore a transaction-level feature, but it is not. Rather, Segregated Witness is a change to how individual UTXO are spent and therefore is a per-output feature. +At first glance, Segregated Witness appears to be a change to how +transactions are constructed and therefore a transaction-level feature, +but it is not. Rather, Segregated Witness is a change to how individual +UTXO are spent and therefore is a per-output feature. -A transaction can spend Segregated Witness outputs or traditional (inline-witness) outputs or both. Therefore, it does not make much sense to refer to a transaction as a “Segregated Witness transaction.” Rather we should refer to specific transaction outputs as “Segregated Witness outputs." +A transaction can spend Segregated Witness outputs or traditional +(inline-witness) outputs or both. Therefore, it does not make much sense +to refer to a transaction as a “Segregated Witness transaction.” Rather +we should refer to specific transaction outputs as “Segregated Witness +outputs." -When a transaction spends an UTXO, it must provide a witness. In a traditional UTXO, the locking script requires that witness data be provided _inline_ in the input part of the transaction that spends the UTXO. A Segregated Witness UTXO, however, specifies a locking script that can be satisfied with witness data outside of the input (segregated). +When a transaction spends an UTXO, it must provide a witness. In a +traditional UTXO, the locking script requires that witness data be +provided _inline_ in the input part of the transaction that spends the +UTXO. A Segregated Witness UTXO, however, specifies a locking script +that can be satisfied with witness data outside of the input +(segregated). ==== Soft Fork (Backward Compatibility) -Segregated Witness is a significant change to the way outputs and transactions are architected. Such a change would normally require a simultaneous change in every Bitcoin node and wallet to change the consensus rules—what is known as a hard fork. Instead, segregated witness is introduced with a much less disruptive change, which is backward compatible, known as a soft fork. This type of upgrade allows nonupgraded software to ignore the changes and continue to operate without any disruption. - -Segregated Witness outputs are constructed so that older systems that are not segwit-aware can still validate them. To an old wallet or node, a Segregated Witness output looks like an output that _anyone can spend_. Such outputs can be spent with an empty signature, therefore the fact that there is no signature inside the transaction (it is segregated) does not invalidate the transaction. Newer wallets and mining nodes, however, see the Segregated Witness output and expect to find a valid witness for it in the transaction’s witness data. +Segregated Witness is a significant change to the way outputs and +transactions are architected. Such a change would normally require a +simultaneous change in every Bitcoin node and wallet to change the +consensus rules—what is known as a hard fork. Instead, segregated +witness is introduced with a much less disruptive change, which is +backward compatible, known as a soft fork. This type of upgrade allows +nonupgraded software to ignore the changes and continue to operate +without any disruption. + +Segregated Witness outputs are constructed so that older systems that +are not segwit-aware can still validate them. To an old wallet or node, +a Segregated Witness output looks like an output that _anyone can +spend_. Such outputs can be spent with an empty signature, therefore the +fact that there is no signature inside the transaction (it is +segregated) does not invalidate the transaction. Newer wallets and +mining nodes, however, see the Segregated Witness output and expect to +find a valid witness for it in the transaction’s witness data. ==== Segregated Witness Output and Transaction Examples -Let’s look at some of our example transactions and see how they would change with Segregated Witness. We’ll first look at how a Pay-to-Public-Key-Hash (P2PKH) payment is transformed with the Segregated Witness program. Then, we’ll look at the Segregated Witness equivalent for Pay-to-Script-Hash (P2SH) scripts. Finally, we’ll look at how both of the preceding Segregated Witness programs can be embedded inside a P2SH script. +Let’s look at some of our example transactions and see how they would +change with Segregated Witness. We’ll first look at how a +Pay-to-Public-Key-Hash (P2PKH) payment is transformed with the +Segregated Witness program. Then, we’ll look at the Segregated Witness +equivalent for Pay-to-Script-Hash (P2SH) scripts. Finally, we’ll look at +how both of the preceding Segregated Witness programs can be embedded +inside a P2SH script. [[p2wpkh]] ===== Pay-to-Witness-Public-Key-Hash (P2WPKH) -In <>, ((("use cases", "buying coffee", id="aliced")))Alice created a transaction to pay Bob for a cup of coffee. That transaction created a P2PKH output with a value of 0.015 BTC that was spendable by Bob. The output’s script looks like this: +In <>, ((("use cases", "buying coffee", +id="aliced")))Alice created a transaction to pay Bob for a cup of +coffee. That transaction created a P2PKH output with a value of 0.015 +BTC that was spendable by Bob. The output’s script looks like this: .Example P2PKH output script ---- DUP HASH160 ab68025513c3dbd2f7b92a94e0581f5d50f654e7 EQUALVERIFY CHECKSIG ---- -With Segregated Witness, Alice would create a Pay-to-Witness-Public-Key-Hash (P2WPKH) script, which looks like this: +With Segregated Witness, Alice would create a +Pay-to-Witness-Public-Key-Hash (P2WPKH) script, which looks like this: .Example P2WPKH output script ---- 0 ab68025513c3dbd2f7b92a94e0581f5d50f654e7 ---- -As you can see, a Segregated Witness output’s locking script is much simpler than a traditional output. It consists of two values that are pushed on to the script evaluation stack. To an old (nonsegwit-aware) Bitcoin client, the two pushes would look like an output that anyone can spend and does not require a signature (or rather, can be spent with an empty signature). To a newer, segwit-aware client, the first number (0) is interpreted as a version number (the _witness version_) and the second part (20 bytes) is the equivalent of a locking script known as a _witness program_. The 20-byte witness program is simply the hash of the public key, as in a P2PKH script +As you can see, a Segregated Witness output’s locking script is much +simpler than a traditional output. It consists of two values that are +pushed on to the script evaluation stack. To an old (nonsegwit-aware) +Bitcoin client, the two pushes would look like an output that anyone can +spend and does not require a signature (or rather, can be spent with an +empty signature). To a newer, segwit-aware client, the first number (0) +is interpreted as a version number (the _witness version_) and the +second part (20 bytes) is the equivalent of a locking script known as a +_witness program_. The 20-byte witness program is simply the hash of the +public key, as in a P2PKH script -Now, let’s look at the corresponding transaction that Bob uses to spend this output. For the original script (nonsegwit), Bob’s transaction would have to include a signature within the transaction input: +Now, let’s look at the corresponding transaction that Bob uses to spend +this output. For the original script (nonsegwit), Bob’s transaction +would have to include a signature within the transaction input: .Decoded transaction showing a P2PKH output being spent with a signature ---- @@ -725,7 +1451,10 @@ Now, let’s look at the corresponding transaction that Bob uses to spend this o [...] ---- -However, to spend the Segregated Witness output, the transaction has no signature on that input. Instead, Bob’s transaction has an empty +scriptSig+ and includes a Segregated Witness, outside the transaction itself: +However, to spend the Segregated Witness output, the transaction has no +signature on that input. Instead, Bob’s transaction has an empty ++scriptSig+ and includes a Segregated Witness, outside the transaction +itself: .Decoded transaction showing a P2WPKH output being spent with separate witness data ---- @@ -742,26 +1471,47 @@ However, to spend the Segregated Witness output, the transaction has no signatur ===== Wallet construction of P2WPKH -It is extremely important to note that P2WPKH should only be created by the payee (recipient) and not converted by the sender from a known public key, P2PKH script, or address. The sender has no way of knowing if the recipient's wallet has the ability to construct segwit transactions and spend P2WPKH outputs. +It is extremely important to note that P2WPKH should only be created by +the payee (recipient) and not converted by the sender from a known +public key, P2PKH script, or address. The sender has no way of knowing +if the recipient's wallet has the ability to construct segwit +transactions and spend P2WPKH outputs. -Additionally, P2WPKH outputs must be constructed from the hash of a _compressed_ public key. Uncompressed public keys are nonstandard in segwit and may be explicitly disabled by a future soft fork. If the hash used in the P2WPKH came from an uncompressed public key, it may be unspendable and you may lose funds. P2WPKH outputs should be created by the payee's wallet by deriving a compressed public key from their private key. +Additionally, P2WPKH outputs must be constructed from the hash of a +_compressed_ public key. Uncompressed public keys are nonstandard in +segwit and may be explicitly disabled by a future soft fork. If the hash +used in the P2WPKH came from an uncompressed public key, it may be +unspendable and you may lose funds. P2WPKH outputs should be created by +the payee's wallet by deriving a compressed public key from their +private key. [WARNING] ==== -P2WPKH should be constructed by the payee (recipient) by converting a compressed public key to a P2WPKH hash. You should never transform a P2PKH script, Bitcoin address, or uncompressed public key to a P2WPKH witness script. +P2WPKH should be constructed by the payee (recipient) by converting a +compressed public key to a P2WPKH hash. You should never transform a +P2PKH script, Bitcoin address, or uncompressed public key to a P2WPKH +witness script. ==== [[p2wsh]] ===== Pay-to-Witness-Script-Hash (P2WSH) -The ((("use cases", "import/export", id="mohamappd")))second type of witness program corresponds to a Pay-to-Script-Hash (P2SH) script. We saw this type of script in <>. In that example, P2SH was used by Mohammed's company to express a multisignature script. Payments to Mohammed's company were encoded with a locking script like this: +The ((("use cases", "import/export", id="mohamappd")))second type of +witness program corresponds to a Pay-to-Script-Hash (P2SH) script. We +saw this type of script in <>. In that example, P2SH was used by +Mohammed's company to express a multisignature script. Payments to +Mohammed's company were encoded with a locking script like this: .Example P2SH output script ---- HASH160 54c557e07dde5bb6cb791c7a540e0a4796f5e97e EQUAL ---- -This P2SH script references the hash of a _redeem script_ that defines a 2-of-3 multisignature requirement to spend funds. To spend this output, Mohammed's company would present the redeem script (whose hash matches the script hash in the P2SH output) and the signatures necessary to satisfy that redeem script, all inside the transaction input: +This P2SH script references the hash of a _redeem script_ that defines a +2-of-3 multisignature requirement to spend funds. To spend this output, +Mohammed's company would present the redeem script (whose hash matches +the script hash in the P2SH output) and the signatures necessary to +satisfy that redeem script, all inside the transaction input: .Decoded transaction showing a P2SH output being spent ---- @@ -773,22 +1523,39 @@ This P2SH script references the hash of a _redeem script_ that defines a 2-of-3 ] ---- -Now, let's look at how this entire example would be upgraded to segwit. If Mohammed's customers were using a segwit-compatible wallet, they would make a payment, creating a Pay-to-Witness-Script-Hash (P2WSH) output that would look like this: +Now, let's look at how this entire example would be upgraded to segwit. +If Mohammed's customers were using a segwit-compatible wallet, they +would make a payment, creating a Pay-to-Witness-Script-Hash (P2WSH) +output that would look like this: .Example P2WSH output script ---- 0 a9b7b38d972cabc7961dbfbcb841ad4508d133c47ba87457b4a0e8aae86dbb89 ---- -Again, as with the example of P2WPKH, you can see that the Segregated Witness equivalent script is a lot simpler and omits the various script operands that you see in P2SH scripts. Instead, the Segregated Witness program consists of two values pushed to the stack: a witness version (0) and the 32-byte SHA256 hash of the redeem script. +Again, as with the example of P2WPKH, you can see that the Segregated +Witness equivalent script is a lot simpler and omits the various script +operands that you see in P2SH scripts. Instead, the Segregated Witness +program consists of two values pushed to the stack: a witness version +(0) and the 32-byte SHA256 hash of the redeem script. [TIP] ==== -While P2SH uses the 20-byte +RIPEMD160(SHA256(script))+ hash, the P2WSH witness program uses a 32-byte +SHA256(script)+ hash. This difference in the selection of the hashing algorithm is deliberate and is used to differentiate between the two types of witness programs (P2WPKH and P2WSH) by the length of the hash and to provide stronger security to P2WSH (128 bits of security in P2WSH versus 80 bits of security in P2SH). - +While P2SH uses the 20-byte +RIPEMD160(SHA256(script))+ hash, the P2WSH +witness program uses a 32-byte +SHA256(script)+ hash. This difference in +the selection of the hashing algorithm is deliberate and is used to +differentiate between the two types of witness programs (P2WPKH and +P2WSH) by the length of the hash and to provide stronger security to +P2WSH (128 bits of security in P2WSH versus 80 bits of security in +P2SH). ==== -Mohammed's company can spend outputs the P2WSH output by presenting the correct redeem script and sufficient signatures to satisfy it. Both the redeem script and the signatures would be segregated _outside_ the spending transaction as part of the witness data. Within the transaction input, Mohammed's ((("", startref="mohamappd")))wallet would put an empty +scriptSig+: +Mohammed's company can spend outputs the P2WSH output by presenting the +correct redeem script and sufficient signatures to satisfy it. Both the +redeem script and the signatures would be segregated _outside_ the +spending transaction as part of the witness data. Within the transaction +input, Mohammed's ((("", startref="mohamappd")))wallet would put an +empty +scriptSig+: .Decoded transaction showing a P2WSH output being spent with separate witness data ---- @@ -805,41 +1572,89 @@ Mohammed's company can spend outputs the P2WSH output by presenting the correct ===== Differentiating between P2WPKH and P2WSH -In the previous two sections, we demonstrated two types of witness programs: <> and <>. Both types of witness programs consist of a single byte version number followed by a longer hash. They look very similar, but are interpreted very differently: one is interpreted as a public key hash, which is satisfied by a signature and the other as a script hash, which is satisfied by a redeem script. The critical difference between them is the length of the hash: +In the previous two sections, we demonstrated two types of witness +programs: <> and <>. Both types of witness programs +consist of a single byte version number followed by a longer hash. They +look very similar, but are interpreted very differently: one is +interpreted as a public key hash, which is satisfied by a signature and +the other as a script hash, which is satisfied by a redeem script. The +critical difference between them is the length of the hash: -* The public key hash in P2WPKH is 20 bytes -* The script hash in P2WSH is 32 bytes +- The public key hash in P2WPKH is 20 bytes +- The script hash in P2WSH is 32 bytes -This is the one difference that allows a wallet to differentiate between the two types of witness programs. By looking at the length of the hash, a wallet can determine what type of witness program it is, P2WPKH or P2WSH. +This is the one difference that allows a wallet to differentiate between +the two types of witness programs. By looking at the length of the hash, +a wallet can determine what type of witness program it is, P2WPKH or +P2WSH. ==== Upgrading to Segregated Witness -As we can see from the previous examples, upgrading to Segregated Witness is a two-step process. First, wallets must create special segwit type outputs. Then, these outputs can be spent by wallets that know how to construct Segregated Witness transactions. In the examples, Alice's wallet was segwit-aware and able to create special outputs with Segregated Witness scripts. Bob's wallet is also segwit-aware and able to spend those outputs. What may not be obvious from the example is that in practice, Alice's wallet needs to _know_ that Bob uses a segwit-aware wallet and can spend these outputs. Otherwise, if Bob's wallet is not upgraded and Alice tries to make segwit payments to Bob, Bob's wallet will not be able to detect these payments. +As we can see from the previous examples, upgrading to Segregated +Witness is a two-step process. First, wallets must create special segwit +type outputs. Then, these outputs can be spent by wallets that know how +to construct Segregated Witness transactions. In the examples, Alice's +wallet was segwit-aware and able to create special outputs with +Segregated Witness scripts. Bob's wallet is also segwit-aware and able +to spend those outputs. What may not be obvious from the example is that +in practice, Alice's wallet needs to _know_ that Bob uses a segwit-aware +wallet and can spend these outputs. Otherwise, if Bob's wallet is not +upgraded and Alice tries to make segwit payments to Bob, Bob's wallet +will not be able to detect these payments. [TIP] ==== -For P2WPKH and P2WSH payment types, both the sender and the recipient wallets need to be upgraded to be able to use segwit. Furthermore, the sender's wallet needs to know that the recipient's wallet is segwit-aware. +For P2WPKH and P2WSH payment types, both the sender and the recipient +wallets need to be upgraded to be able to use segwit. Furthermore, the +sender's wallet needs to know that the recipient's wallet is +segwit-aware. ==== -Segregated Witness will not be implemented simultaneously across the entire network. Rather, Segregated Witness is implemented as a backward-compatible upgrade, where _old and new clients can coexist_. Wallet developers will independently upgrade wallet software to add segwit capabilities. The P2WPKH and P2WSH payment types are used when both sender and recipient are segwit-aware. The traditional P2PKH and P2SH will continue to work for nonupgraded wallets. That leaves two important scenarios, which are addressed in the next section: +Segregated Witness will not be implemented simultaneously across the +entire network. Rather, Segregated Witness is implemented as a +backward-compatible upgrade, where _old and new clients can coexist_. +Wallet developers will independently upgrade wallet software to add +segwit capabilities. The P2WPKH and P2WSH payment types are used when +both sender and recipient are segwit-aware. The traditional P2PKH and +P2SH will continue to work for nonupgraded wallets. That leaves two +important scenarios, which are addressed in the next section: -* Ability of a sender's wallet that is not segwit-aware to make a payment to a recipient's wallet that can process segwit transactions +- Ability of a sender's wallet that is not segwit-aware to make a + payment to a recipient's wallet that can process segwit transactions -* Ability of a sender's wallet that is segwit-aware to recognize and distinguish between recipients that are segwit-aware and ones that are not, by their _addresses_. +- Ability of a sender's wallet that is segwit-aware to recognize and + distinguish between recipients that are segwit-aware and ones that are + not, by their _addresses_. ===== Embedding Segregated Witness inside P2SH -Let's assume, for example, that Alice's wallet is not upgraded to segwit, but Bob's wallet is upgraded and can handle segwit transactions. Alice and Bob can use "old" non-segwit transactions. But Bob would likely want to use segwit to reduce transaction fees, taking advantage of the discount that applies to witness data. +Let's assume, for example, that Alice's wallet is not upgraded to +segwit, but Bob's wallet is upgraded and can handle segwit transactions. +Alice and Bob can use "old" non-segwit transactions. But Bob would +likely want to use segwit to reduce transaction fees, taking advantage +of the discount that applies to witness data. -In this case Bob's wallet can construct a P2SH address that contains a segwit script inside it. Alice's wallet sees this as a "normal" P2SH address and can make payments to it without any knowledge of segwit. Bob's wallet can then spend this payment with a segwit transaction, taking full advantage of segwit and reducing transaction fees. +In this case Bob's wallet can construct a P2SH address that contains a +segwit script inside it. Alice's wallet sees this as a "normal" P2SH +address and can make payments to it without any knowledge of segwit. +Bob's wallet can then spend this payment with a segwit transaction, +taking full advantage of segwit and reducing transaction fees. -Both forms of witness scripts, P2WPKH and P2WSH, can be embedded in a P2SH address. The first is noted as P2SH(P2WPKH) and the second is noted as P2SH(P2WSH). +Both forms of witness scripts, P2WPKH and P2WSH, can be embedded in a +P2SH address. The first is noted as P2SH(P2WPKH) and the second is noted +as P2SH(P2WSH). ===== Pay-to-Witness-Public-Key-Hash inside Pay-to-Script-Hash -The first form of witness script we will examine is P2SH(P2WPKH). This is a Pay-to-Witness-Public-Key-Hash witness program, embedded inside a Pay-to-Script-Hash script, so that it can be used by a wallet that is not aware of segwit. +The first form of witness script we will examine is P2SH(P2WPKH). This +is a Pay-to-Witness-Public-Key-Hash witness program, embedded inside a +Pay-to-Script-Hash script, so that it can be used by a wallet that is +not aware of segwit. -Bob's wallet constructs a P2WPKH witness program with Bob's public key. This witness program is then hashed and the resulting hash is encoded as a P2SH script. The P2SH script is converted to a Bitcoin address, one that starts with a "3," as we saw in the <> section. +Bob's wallet constructs a P2WPKH witness program with Bob's public key. +This witness program is then hashed and the resulting hash is encoded as +a P2SH script. The P2SH script is converted to a Bitcoin address, one +that starts with a "3," as we saw in the <> section. Bob's wallet starts with the P2WPKH witness program we saw earlier: @@ -848,9 +1663,11 @@ Bob's wallet starts with the P2WPKH witness program we saw earlier: 0 ab68025513c3dbd2f7b92a94e0581f5d50f654e7 ---- -The P2WPKH witness program consists of the witness version and Bob's 20-byte public key hash. +The P2WPKH witness program consists of the witness version and Bob's +20-byte public key hash. -Bob's wallet then hashes the preceding witness program, first with SHA256, then with RIPEMD160, producing another 20-byte hash. +Bob's wallet then hashes the preceding witness program, first with +SHA256, then with RIPEMD160, producing another 20-byte hash. Let's use +bx+ on the command-line to replicate that: @@ -862,8 +1679,8 @@ echo \ 3e0547268b3b19288b3adef9719ec8659f4b2b0b ---- - -Next, the redeem script hash is converted to a Bitcoin address. Let's use +bx+ on the command-line again: +Next, the redeem script hash is converted to a Bitcoin address. Let's +use +bx+ on the command-line again: .P2SH address ---- @@ -873,22 +1690,36 @@ echo \ 37Lx99uaGn5avKBxiW26HjedQE3LrDCZru ---- -Now, Bob can display this address for customers to pay for their coffee. Alice's wallet can make a payment to +37Lx99uaGn5avKBxiW26HjedQE3LrDCZru+, just as it would to any other Bitcoin address. +Now, Bob can display this address for customers to pay for their coffee. +Alice's wallet can make a payment to ++37Lx99uaGn5avKBxiW26HjedQE3LrDCZru+, just as it would to any other +Bitcoin address. To pay Bob, Alice's wallet would lock the output with a P2SH script: ---- HASH160 3e0547268b3b19288b3adef9719ec8659f4b2b0b EQUAL ---- -Even though Alice's wallet has no support for segwit, the payment it creates can be spent by Bob with a segwit transaction.((("", startref="aliced"))) +Even though Alice's wallet has no support for segwit, the payment it +creates can be spent by Bob with a segwit transaction.((("", +startref="aliced"))) ===== Pay-to-Witness-Script-Hash inside Pay-to-Script-Hash -Similarly, a P2WSH witness program for a multisig script or other complicated script can be embedded inside a P2SH script and address, making it possible for any wallet to make payments that are segwit compatible. +Similarly, a P2WSH witness program for a multisig script or other +complicated script can be embedded inside a P2SH script and address, +making it possible for any wallet to make payments that are segwit +compatible. -As we saw in <>, Mohammed's ((("use cases", "import/export")))company is using Segregated Witness payments to multisignature scripts. To make it possible for any client to pay his company, regardless of whether their wallets are upgraded for segwit, Mohammed's wallet can embed the P2WSH witness program inside a P2SH script. +As we saw in <>, Mohammed's ((("use cases", +"import/export")))company is using Segregated Witness payments to +multisignature scripts. To make it possible for any client to pay his +company, regardless of whether their wallets are upgraded for segwit, +Mohammed's wallet can embed the P2WSH witness program inside a P2SH +script. -First, Mohammed's wallet hashes the redeem script with SHA256 (just once). Let's use +bx+ to do that on the command-line: +First, Mohammed's wallet hashes the redeem script with SHA256 (just +once). Let's use +bx+ to do that on the command-line: .Mohammed's wallet creates a P2WSH witness program ---- @@ -909,7 +1740,9 @@ Next, the hashed redeem script is turned into a P2WSH witness program: 0 9592d601848d04b172905e0ddb0adde59f1590f1e553ffc81ddc4b0ed927dd73 ---- -Then, the witness program itself is hashed with SHA256 and RIPEMD160, producing a new 20-byte hash, as used in traditional P2SH. Let's use +bx+ on the command-line to do that: +Then, the witness program itself is hashed with SHA256 and RIPEMD160, +producing a new 20-byte hash, as used in traditional P2SH. Let's use ++bx+ on the command-line to do that: .The HASH160 of the P2WSH witness program ---- @@ -919,7 +1752,8 @@ Then, the witness program itself is hashed with SHA256 and RIPEMD160, producing 86762607e8fe87c0c37740cddee880988b9455b2 ---- -Next, the wallet constructs a P2SH Bitcoin address from this hash. Again, we use +bx+ to calculate on the command-line: +Next, the wallet constructs a P2SH Bitcoin address from this hash. +Again, we use +bx+ to calculate on the command-line: .P2SH Bitcoin address ---- @@ -929,30 +1763,52 @@ echo \ 3Dwz1MXhM6EfFoJChHCxh1jWHb8GQqRenG ---- -Now, Mohammed's clients can make payments to this address without any need to support segwit. To send a payment to Mohammed, a wallet would lock the output with the following P2SH script: +Now, Mohammed's clients can make payments to this address without any +need to support segwit. To send a payment to Mohammed, a wallet would +lock the output with the following P2SH script: .P2SH script used to lock payments to Mohammed's multisig ---- HASH160 86762607e8fe87c0c37740cddee880988b9455b2 EQUAL ---- -Mohammed's company can then construct segwit transactions to spend these payments, taking advantage of segwit features including lower transaction fees. +Mohammed's company can then construct segwit transactions to spend these +payments, taking advantage of segwit features including lower +transaction fees. ===== Segregated Witness addresses -Even after segwit activation, it will take some time until most wallets are upgraded. At first, segwit will be embedded in P2SH, as we saw in the previous section, to ease compatibility between segit-aware and unaware wallets. +Even after segwit activation, it will take some time until most wallets +are upgraded. At first, segwit will be embedded in P2SH, as we saw in +the previous section, to ease compatibility between segit-aware and +unaware wallets. -However, once wallets are broadly supporting segwit, it makes sense to encode witness scripts directly in a native address format designed for segwit, rather than embed it in P2SH. +However, once wallets are broadly supporting segwit, it makes sense to +encode witness scripts directly in a native address format designed for +segwit, rather than embed it in P2SH. The native segwit address format is defined in BIP-173: -https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki[BIP-173]:: Base32 address format for native v0-16 witness outputs +https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki[BIP-173]:: +Base32 address format for native v0-16 witness outputs -BIP-173 only encodes witness (P2WPKH and P2WSH) scripts. It is not compatible with non-segwit P2PKH or P2SH scripts. BIP-173 is a checksummed Base32 encoding, as compared to the Base58 encoding of a "traditional" Bitcoin address. BIP-173 addesses are also called _bech32_ addresses, pronounced "beh-ch thirty two", alluding to the use of a "BCH" error detection algorithm and 32-character encoding set. +BIP-173 only encodes witness (P2WPKH and P2WSH) scripts. It is not +compatible with non-segwit P2PKH or P2SH scripts. BIP-173 is a +checksummed Base32 encoding, as compared to the Base58 encoding of a +"traditional" Bitcoin address. BIP-173 addesses are also called _bech32_ +addresses, pronounced "beh-ch thirty two", alluding to the use of a +"BCH" error detection algorithm and 32-character encoding set. -BIP-173 addresses use 32 lower-case-only alphanumeric character set, carefully selected to reduce errors from misreading or mistyping. By choosing a lower-case-only character set, bech32 is easier to read, speak, and 45% more efficient to encode in QR codes. +BIP-173 addresses use 32 lower-case-only alphanumeric character set, +carefully selected to reduce errors from misreading or mistyping. By +choosing a lower-case-only character set, bech32 is easier to read, +speak, and 45% more efficient to encode in QR codes. -The BCH error detection algorithm is a vast improvement over the previous checksum algorithm (from Base58Check), allowing not only detection but also _correction_ of errors. Address-input interfaces (such as text-fields in forms) can detect and highlight which character was most likely mistyped when they detect an error. +The BCH error detection algorithm is a vast improvement over the +previous checksum algorithm (from Base58Check), allowing not only +detection but also _correction_ of errors. Address-input interfaces +(such as text-fields in forms) can detect and highlight which character +was most likely mistyped when they detect an error. From the BIP-173 specification, here are some examples of bech32 addresses: @@ -961,69 +1817,196 @@ Testnet P2WPKH:: tb1qw508d6qejxtdg4y5r3zarvary0c5xw7kxpjzsx Mainnet P2WSH:: bc1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3qccfmv3 Testnet P2WSH:: tb1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3q0sl5k7 -As you can see in these examples, a segwit bech32 string is up to 90 characters long and consists of three parts: +As you can see in these examples, a segwit bech32 string is up to 90 +characters long and consists of three parts: -The human readable part:: This prefix "bc" or "tb" identifying mainnet or testnet. +The human readable part:: This prefix "bc" or "tb" identifying mainnet +or testnet. -The separator:: The digit "1", which is not part of the 32-character encoding set and can only appear in this position as a separator +The separator:: The digit "1", which is not part of the 32-character +encoding set and can only appear in this position as a separator -The data part:: A minimum of 6 alphanumeric characters, the checksum encoded witness script +The data part:: A minimum of 6 alphanumeric characters, the checksum +encoded witness script -At this time, only a few wallets accept or produce native segwit bech32 addresses, but as segwit adoption increases, you will see these more and more often. +At this time, only a few wallets accept or produce native segwit bech32 +addresses, but as segwit adoption increases, you will see these more and +more often. [[segwit_txid]] ===== Transaction identifiers -((("transaction IDs (txid)")))One of the greatest benefits of Segregated Witness is that it eliminates third-party transaction malleability. - -Before segwit, transactions could have their signatures subtly modified by third parties, changing their transaction ID (hash) without changing any fundamental properties (inputs, outputs, amounts). This created opportunities for denial-of-service attacks as well as attacks against poorly written wallet software that assumed unconfirmed transaction hashes were immutable. - -With the introduction of Segregated Witness, transactions have two identifiers, +txid+ and +wtxid+. The traditional transaction ID +txid+ is the double-SHA256 hash of the serialized transaction, without the witness data. A transaction +wtxid+ is the double-SHA256 hash of the new serialization format of the transaction with witness data. - -The traditional +txid+ is calculated in exactly the same way as with a nonsegwit transaction. However, since the segwit transaction has empty ++scriptSig++s in every input, there is no part of the transaction that can be modified by a third party. Therefore, in a segwit transaction, the +txid+ is immutable by a third party, even when the transaction is unconfirmed. - -The +wtxid+ is like an "extended" ID, in that the hash also incorporates the witness data. If a transaction is transmitted without witness data, then the +wtxid+ and +txid+ are identical. Note than since the +wtxid+ includes witness data (signatures) and since witness data may be malleable, the +wtxid+ should be considered malleable until the transaction is confirmed. Only the +txid+ of a segwit transaction can be considered immutable by third parties and only if _all_ the inputs of the transaction are segwit inputs. +((("transaction IDs (txid)")))One of the greatest benefits of Segregated +Witness is that it eliminates third-party transaction malleability. + +Before segwit, transactions could have their signatures subtly modified +by third parties, changing their transaction ID (hash) without changing +any fundamental properties (inputs, outputs, amounts). This created +opportunities for denial-of-service attacks as well as attacks against +poorly written wallet software that assumed unconfirmed transaction +hashes were immutable. + +With the introduction of Segregated Witness, transactions have two +identifiers, +txid+ and +wtxid+. The traditional transaction ID +txid+ +is the double-SHA256 hash of the serialized transaction, without the +witness data. A transaction +wtxid+ is the double-SHA256 hash of the new +serialization format of the transaction with witness data. + +The traditional +txid+ is calculated in exactly the same way as with a +nonsegwit transaction. However, since the segwit transaction has empty +++scriptSig++s in every input, there is no part of the transaction that +can be modified by a third party. Therefore, in a segwit transaction, +the +txid+ is immutable by a third party, even when the transaction is +unconfirmed. + +The +wtxid+ is like an "extended" ID, in that the hash also incorporates +the witness data. If a transaction is transmitted without witness data, +then the +wtxid+ and +txid+ are identical. Note than since the +wtxid+ +includes witness data (signatures) and since witness data may be +malleable, the +wtxid+ should be considered malleable until the +transaction is confirmed. Only the +txid+ of a segwit transaction can be +considered immutable by third parties and only if _all_ the inputs of +the transaction are segwit inputs. [TIP] ==== -Segregated Witness transactions have two IDs: +txid+ and +wtxid+. The +txid+ is the hash of the transaction without the witness data and the +wtxid+ is the hash inclusive of witness data. The +txid+ of a transaction where all inputs are segwit inputs is not susceptible to third-party transaction malleability. +Segregated Witness transactions have two IDs: +txid+ and +wtxid+. The ++txid+ is the hash of the transaction without the witness data and the ++wtxid+ is the hash inclusive of witness data. The +txid+ of a +transaction where all inputs are segwit inputs is not susceptible to +third-party transaction malleability. ==== ==== Segregated Witness' New Signing Algorithm -Segregated Witness modifies the semantics of the four signature verification functions (+CHECKSIG+, +CHECKSIGVERIFY+, +CHECKMULTISIG+, and +CHECKMULTISIGVERIFY+), changing the way a transaction commitment hash is calculated. - -Signatures in bitcoin transactions are applied on a _commitment hash_, which is calculated from the transaction data, locking specific parts of the data indicating the signer's commitment to those values. For example, in a simple +SIGHASH_ALL+ type signature, the commitment hash includes all inputs and outputs. - -Unfortunately, the way the commitment hash was calculated introduced the possibility that a node verifying the signature can be forced to perform a significant number of hash computations. Specifically, the hash operations increase in O(n^2^) with respect to the number of signature operations in the transaction. An attacker could therefore create a transaction with a very large number of signature operations, causing the entire Bitcoin network to have to perform hundreds or thousands of hash operations to verify the transaction. - -Segwit represented an opportunity to address this problem by changing the way the commitment hash is calculated. For segwit version 0 witness programs, signature verification occurs using an improved commitment hash algorithm as specified in BIP-143. - -The new algorithm achieves two important goals. Firstly, the number of hash operations increases by a much more gradual O(n) to the number of signature operations, reducing the opportunity to create denial-of-service attacks with overly complex transactions. Secondly, the commitment hash now also includes the value (amounts) of each input as part of the commitment. This means that a signer can commit to a specific input value without needing to "fetch" and check the previous transaction referenced by the input. In the case of offline devices, such as hardware wallets, this greatly simplifies the communication between the host and the hardware wallet, removing the need to stream previous transactions for validation. A hardware wallet can accept the input value "as stated" by an untrusted host. Since the signature is invalid if that input value is not correct, the hardware wallet doesn't need to validate the value before signing the input. +Segregated Witness modifies the semantics of the four signature +verification functions (+CHECKSIG+, +CHECKSIGVERIFY+, +CHECKMULTISIG+, +and +CHECKMULTISIGVERIFY+), changing the way a transaction commitment +hash is calculated. + +Signatures in bitcoin transactions are applied on a _commitment hash_, +which is calculated from the transaction data, locking specific parts of +the data indicating the signer's commitment to those values. For +example, in a simple +SIGHASH_ALL+ type signature, the commitment hash +includes all inputs and outputs. + +Unfortunately, the way the commitment hash was calculated introduced the +possibility that a node verifying the signature can be forced to perform +a significant number of hash computations. Specifically, the hash +operations increase in O(n^2^) with respect to the number of signature +operations in the transaction. An attacker could therefore create a +transaction with a very large number of signature operations, causing +the entire Bitcoin network to have to perform hundreds or thousands of +hash operations to verify the transaction. + +Segwit represented an opportunity to address this problem by changing +the way the commitment hash is calculated. For segwit version 0 witness +programs, signature verification occurs using an improved commitment +hash algorithm as specified in BIP-143. + +The new algorithm achieves two important goals. Firstly, the number of +hash operations increases by a much more gradual O(n) to the number of +signature operations, reducing the opportunity to create +denial-of-service attacks with overly complex transactions. Secondly, +the commitment hash now also includes the value (amounts) of each input +as part of the commitment. This means that a signer can commit to a +specific input value without needing to "fetch" and check the previous +transaction referenced by the input. In the case of offline devices, +such as hardware wallets, this greatly simplifies the communication +between the host and the hardware wallet, removing the need to stream +previous transactions for validation. A hardware wallet can accept the +input value "as stated" by an untrusted host. Since the signature is +invalid if that input value is not correct, the hardware wallet doesn't +need to validate the value before signing the input. ==== Economic Incentives for Segregated Witness -Bitcoin mining nodes and full nodes incur costs for the resources used to support the Bitcoin network and the blockchain. As the volume of bitcoin transactions increases, so does the cost of resources (CPU, network bandwidth, disk space, memory). Miners are compensated for these costs through fees that are proportional to the size (in bytes) of each transaction. Nonmining full nodes are not compensated, so they incur these costs because they have a need to run an authoritative fully validating full-index node, perhaps because they use the node to operate a bitcoin business. - -Without transaction fees, the growth in bitcoin data would arguably increase dramatically. Fees are intended to align the needs of bitcoin users with the burden their transactions impose on the network, through a market-based price discovery mechanism. - -The calculation of fees based on transaction size treats all the data in the transaction as equal in cost. But from the perspective of full nodes and miners, some parts of a transaction carry much higher costs. Every transaction added to the Bitcoin network affects the consumption of four resources on nodes: - -Disk Space :: Every transaction is stored in the blockchain, adding to the total size of the blockchain. The blockchain is stored on disk, but the storage can be optimized by “pruning” older transactions. +Bitcoin mining nodes and full nodes incur costs for the resources used +to support the Bitcoin network and the blockchain. As the volume of +bitcoin transactions increases, so does the cost of resources (CPU, +network bandwidth, disk space, memory). Miners are compensated for these +costs through fees that are proportional to the size (in bytes) of each +transaction. Nonmining full nodes are not compensated, so they incur +these costs because they have a need to run an authoritative fully +validating full-index node, perhaps because they use the node to operate +a bitcoin business. + +Without transaction fees, the growth in bitcoin data would arguably +increase dramatically. Fees are intended to align the needs of bitcoin +users with the burden their transactions impose on the network, through +a market-based price discovery mechanism. + +The calculation of fees based on transaction size treats all the data in +the transaction as equal in cost. But from the perspective of full nodes +and miners, some parts of a transaction carry much higher costs. Every +transaction added to the Bitcoin network affects the consumption of four +resources on nodes: + +Disk Space :: Every transaction is stored in the blockchain, adding to +the total size of the blockchain. The blockchain is stored on disk, but +the storage can be optimized by “pruning” older transactions. CPU :: Every transaction must be validated, which requires CPU time. -Bandwidth :: Every transaction is transmitted (through flood propagation) across the network at least once. Without any optimization in the block propagation protocol, transactions are transmitted again as part of a block, doubling the impact on network capacity. - -Memory :: Nodes that validate transactions keep the UTXO index or the entire UTXO set in memory to speed up validation. Because memory is at least one order of magnitude more expensive than disk, growth of the UTXO set contributes disproportionately to the cost of running a node. - -As you can see from the list, not every part of a transaction has an equal impact on the cost of running a node or on the ability of bitcoin to scale to support more transactions. The most expensive part of a transaction are the newly created outputs, as they are added to the in-memory UTXO set. By comparison, signatures (aka witness data) add the least burden to the network and the cost of running a node, because witness data are only validated once and then never used again. Furthermore, immediately after receiving a new transaction and validating witness data, nodes can discard that witness data. If fees are calculated on transaction size, without discriminating between these two types of data, then the market incentives of fees are not aligned with the actual costs imposed by a transaction. In fact, the current fee structure actually encourages the opposite behavior, because witness data is the largest part of a transaction. - -The incentives created by fees matter because they affect the behavior of wallets. All wallets must implement some strategy for assembling transactions that takes into consideration a number of factors, such as privacy (reducing address reuse), fragmentation (making lots of loose change), and fees. If the fees are overwhelmingly motivating wallets to use as few inputs as possible in transactions, this can lead to UTXO picking and change address strategies that inadvertently bloat the UTXO set. - -Transactions consume UTXO in their inputs and create new UTXO with their outputs. A transaction, therefore, that has more inputs than outputs will result in a decrease in the UTXO set, whereas a transaction that has more outputs than inputs will result in an increase in the UTXO set. Let’s consider the _difference_ between inputs and outputs and call that the “Net-new-UTXO.” That’s an important metric, as it tells us what impact a transaction will have on the most expensive network-wide resource, the in-memory UTXO set. A transaction with positive Net-new-UTXO adds to that burden. A transaction with a negative Net-new-UTXO reduces the burden. We would therefore want to encourage transactions that are either negative Net-new-UTXO or neutral with zero Net-new-UTXO. - -Let’s look at an example of what incentives are created by the transaction fee calculation, with and without Segregated Witness. We will look at two different transactions. Transaction A is a 3-input, 2-output transaction, which has a Net-new-UTXO metric of –1, meaning it consumes one more UTXO than it creates, reducing the UTXO set by one. Transaction B is a 2-input, 3-output transaction, which has a Net-new-UTXO metric of 1, meaning it adds one UTXO to the UTXO set, imposing additional cost on the entire Bitcoin network. Both transactions use multisignature (2-of-3) scripts to demonstrate how complex scripts increase the impact of segregated witness on fees. Let’s assume a transaction fee of 30 satoshi per byte and a 75% fee discount on witness data: +Bandwidth :: Every transaction is transmitted (through flood +propagation) across the network at least once. Without any optimization +in the block propagation protocol, transactions are transmitted again as +part of a block, doubling the impact on network capacity. + +Memory :: Nodes that validate transactions keep the UTXO index or the +entire UTXO set in memory to speed up validation. Because memory is at +least one order of magnitude more expensive than disk, growth of the +UTXO set contributes disproportionately to the cost of running a node. + +As you can see from the list, not every part of a transaction has an +equal impact on the cost of running a node or on the ability of bitcoin +to scale to support more transactions. The most expensive part of a +transaction are the newly created outputs, as they are added to the +in-memory UTXO set. By comparison, signatures (aka witness data) add the +least burden to the network and the cost of running a node, because +witness data are only validated once and then never used again. +Furthermore, immediately after receiving a new transaction and +validating witness data, nodes can discard that witness data. If fees +are calculated on transaction size, without discriminating between these +two types of data, then the market incentives of fees are not aligned +with the actual costs imposed by a transaction. In fact, the current fee +structure actually encourages the opposite behavior, because witness +data is the largest part of a transaction. + +The incentives created by fees matter because they affect the behavior +of wallets. All wallets must implement some strategy for assembling +transactions that takes into consideration a number of factors, such as +privacy (reducing address reuse), fragmentation (making lots of loose +change), and fees. If the fees are overwhelmingly motivating wallets to +use as few inputs as possible in transactions, this can lead to UTXO +picking and change address strategies that inadvertently bloat the UTXO +set. + +Transactions consume UTXO in their inputs and create new UTXO with their +outputs. A transaction, therefore, that has more inputs than outputs +will result in a decrease in the UTXO set, whereas a transaction that +has more outputs than inputs will result in an increase in the UTXO set. +Let’s consider the _difference_ between inputs and outputs and call that +the “Net-new-UTXO.” That’s an important metric, as it tells us what +impact a transaction will have on the most expensive network-wide +resource, the in-memory UTXO set. A transaction with positive +Net-new-UTXO adds to that burden. A transaction with a negative +Net-new-UTXO reduces the burden. We would therefore want to encourage +transactions that are either negative Net-new-UTXO or neutral with zero +Net-new-UTXO. + +Let’s look at an example of what incentives are created by the +transaction fee calculation, with and without Segregated Witness. We +will look at two different transactions. Transaction A is a 3-input, +2-output transaction, which has a Net-new-UTXO metric of –1, +meaning it consumes one more UTXO than it creates, reducing the UTXO set +by one. Transaction B is a 2-input, 3-output transaction, which has a +Net-new-UTXO metric of 1, meaning it adds one UTXO to the UTXO set, +imposing additional cost on the entire Bitcoin network. Both +transactions use multisignature (2-of-3) scripts to demonstrate how +complex scripts increase the impact of segregated witness on fees. Let’s +assume a transaction fee of 30 satoshi per byte and a 75% fee discount +on witness data: ++++
@@ -1041,7 +2024,17 @@ Let’s look at an example of what incentives are created by the transaction fee
++++ - -Both transactions are less expensive when segregated witness is implemented. But comparing the costs between the two transactions, we see that before Segregated Witness, the fee is higher for the transaction that has a negative Net-new-UTXO. After Segregated Witness, the transaction fees align with the incentive to minimize new UTXO creation by not inadvertently penalizing transactions with many inputs. - -Segregated Witness therefore has two main effects on the fees paid by Bitcoin users. Firstly, segwit reduces the overall cost of transactions by discounting witness data and increasing the capacity of the Bitcoin blockchain. Secondly, segwit’s discount on witness data corrects a misalignment of incentives that may have inadvertently created more bloat in the UTXO set.((("", startref="Tadv07")))((("", startref="Ssegwit07"))) +Both transactions are less expensive when segregated witness is +implemented. But comparing the costs between the two transactions, we +see that before Segregated Witness, the fee is higher for the +transaction that has a negative Net-new-UTXO. After Segregated Witness, +the transaction fees align with the incentive to minimize new UTXO +creation by not inadvertently penalizing transactions with many inputs. + +Segregated Witness therefore has two main effects on the fees paid by +Bitcoin users. Firstly, segwit reduces the overall cost of transactions +by discounting witness data and increasing the capacity of the Bitcoin +blockchain. Secondly, segwit’s discount on witness data corrects a +misalignment of incentives that may have inadvertently created more +bloat in the UTXO set.((("", startref="Tadv07")))((("", +startref="Ssegwit07")))