Segregated Witness (segwit) is an upgrade to the bitcoin consensus rules and network protocol, scheduled for implementation in the second half of 2016.
In cryptography, the term "witness" is used to describe a solution to a cryptographic puzzle. In bitcoin terms, the witness satisfies a cryptographic condition placed on a Unspent Transaction Output (UTXO).
In the context of bitcoin, a digital signature is _one type of witness_, but a witness is more broadly any solution that can satisfy the conditions imposed on a UTXO and unlock that UTXO for spending. The term “witness” is a more general term for an “unlocking script” or “scriptSig”.
Before segwit’s introduction, every input in a transaction was followed by the witness data that unlocked it. The witness data was embedded in the transaction as part of each input, The term _segregated witness_ or _segwit_ for short, simply means separating the signature or unlocking script of a specific output. Think "separate scriptSig", or “separate signature” in the simplest form.
Segregated Witness therefore is an architectural change to bitcoin that aims to move the witness data from the scriptSig (unlocking script) field of a transaction into separate a _witness_ data structure that accompanies a transaction. Clients may request transaction data with or without the accompanying witness data.
In this section we will look at some of the benefits of segregated witness, describe the mechanism used to deploy and implement this architecture change and demonstrate the use of segregated witness in transactions and addresses.
Segregated Witness is defined by the following Bitcoin Improvement Proposals (BIPs):
BIP141 :: The main definition of Segregated Witness. https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki
BIP143 :: Transaction Signature Verification for Version 0 Witness Program
Segregated witness is an architectural change that has several effects on the scalability, security, economic incentives and performance of bitcoin.
Transaction malleability :: By moving the witness outside the transaction, the transaction hash used as an identifier no longer includes the witness data. Since the witness data is the only part of the transaction that can be modified by a third party (see <<transaction malleability>> and <<segwit_txid>>), removing it also removes the opportunity for transaction malleability attacks. With segregated witness, transaction hashes become immutable by anyone other than the creator of the transaction, which greatly improves the implementation of many other protocols that rely on advanced bitcoin transaction construction, such as payment channels, chained transactions and lightning networks.
Script Versioning :: With the introduction of segregated witness scripts, every locking script is preceded by a _script version_ number, similar to how transactions and blocks have version numbers. The addition of a script version number allows the scripting language to be upgraded in a backwards compatible way (ie. using soft-fork upgrades), to introduce new script operands, syntax or semantics. The ability to upgrade the scripting language in a non-disruptive way will greatly accelerate the rate of innovation in bitcoin.
Network and Storage Scaling :: The witness data is often a big contributor to the total size of a transaction. More complex scripts such as those used for multi-sig or payment channels are very large. In some cases these scripts account for the majority (more than 75%) of the data in a transaction. By moving the witness data outside the transaction, segregated witness improves bitcoin’s scalability. Nodes can prune the witness data after validating the signatures, or ignore it altogether when doing simplified payment verification. The witness data doesn’t need to be transmitted to all nodes and does not need to be stored on disk by all nodes.
Signature Verification Optimization :: Segregated Witness upgrades the signature functions (CHECKSIG, CHECKMULTISIG etc), to reduce the algorithm's computational complexity. Before segwit, the algorithm used to produce a signature required a number of hash operations that was proportional to the size of the transaction. Data-hasing computations increased in O(n^2^) with respect to the number of signature operations, introducing a substantial computational burden on all nodes verifying the signature. With segwit, the algorithm is changed to reduce the complexity to O(n).
Offline Signing Improvement :: Segregated Witness signatures incorporate the value (amount) referenced by each input in the hash that is signed. Previously, an offline signing device, such as a hardware wallet, would have to verify the amount of each input before signing a transaction. This was usually accomplished by streaming a large amount of data about the previous transactions referenced as inputs. Since the amount is now part of the commitment hash that is signed, an offline device does not need the previous transactions. If the amounts do not match (are misrepresented by a compromised online system), the signature will be invalid.
==== How Segregated Witness Works
At first glance, segregated witness appears to be a change to how transactions are constructed and therefore a transaction-level feature, but it is not. In fact, segregated witness is also a change to how individual UTXO are spent and therefore is a per-output feature.
A transaction can spend segregated witness outputs or traditional (inline-witness) outputs or both. Therefore, it does not make much sense to refer to a transaction as a “segregated witness transaction”. Rather we should refer to specific transaction inputs as “segregated witness inputs".
When a transaction spends a UTXO, it must provide a witness. In a traditional UTXO, the locking script requires that witness data be provided _inline_ in the input part of the transaction that spends the UTXO. A segregated witness UTXO, however, specifies a locking script that can be satisfied with witness data outside of the input (segregated).
==== Soft-fork (backwards compatibility)
Segregated witness is a significant change to the way outputs and transactions are architected. Such a change would normally require a simultaneous change in every bitcoin node and wallet, to change the consensus rules -- what is known as a hard fork. Instead, segregated witness is introduced with a much less disruptive change, which is backwards compatible, known as a soft fork. This type of upgrade allows non-upgraded software to ignore the changes and continue to operate without any disruption.
Segregated witness outputs are constructed so that older systems that are not segwit-aware can still validate them. To an old wallet or node, a segregated witness output looks like an output that _anyone can spend_. Such outputs can be spent with an empty signature, therefore the fact that there is no signature inside the transaction (it is segregated), does not invalidate the transaction. Newer wallets & mining nodes however see the segregated witness output and expect to find a valid witness for it in the transaction’s witness data.
==== Segregated Witness Output and Transaction Examples
Let’s look at some of our example transactions and see how they would change with segregated witness. We’ll first look at how a Pay-to-Public-Key-Hash (P2PKH) payment is transformed with segregated witness program. Then, we’ll look at the segregated witness equivalent for Pay-to-Script-Hash (P2SH) scripts. Finally, we’ll look at how both of the above segregated witness programs can be embedded inside a P2SH script.
[[p2wpkh]]
===== Pay-to-Witness-Public-Key-Hash (P2WPKH)
In <<cup_of_coffee>>, Alice created a transaction to pay Bob for a cup of coffee. That transaction created a Pay-to-Public-Key-Hash (P2PKH) output with a value of 0.015 BTC that was spendable by Bob. The output’s script looks like this:
With segregated witness, a Pay-to-Public-Key-Hash output, is created instead a Pay-to-Witness-Public-Key-Hash (P2WPKH), which looks like this:
.Example P2WPKH output script
----
0 ab68025513c3dbd2f7b92a94e0581f5d50f654e7
----
As you can see, a segregated witness output’s locking script is much simpler than a traditional output. It consists of two values that are pushed on to the script evaluation stack. To an old (non-segwit-aware) bitcoin client, the two pushes would look like an output that anyone can spend and does not require a signature (or rather, can be spent with an empty signature). To a newer, segwit-aware client, the first number (0) is interpreted as a version number (the _witness version_) and the second part (20 bytes) is the equivalent of a locking script known as a _witness program_. The 20-byte witness program is simply the hash of the public key, as in a P2PKH script
Now, let’s look at the corresponding transaction that Bob uses to spend this output. For the original script (non-segwit), Bob’s transaction would have to include a signature within the transaction input:
.Decoded transaction showing a P2PKH output being spent with a signature
However, to spend the segregated witness output, the transaction has no signature on that input. Instead, Bob’s transaction has an empty scriptSig and includes a segregated witness, outside the transaction itself:
.Decoded transaction showing a P2WPKH output being spent with separate witness data
It is extremely important to note that P2WPKH should only be created by the payee (recipient) and not converted by the sender from a known public key, P2PKH script or address. The sender has no way of knowing if the recipient's wallet has the ability to construct segwit transactions and spend P2WPKH outputs.
Additionally, P2WPKH outputs must be constructed from the hash of a _compressed_ public key. Uncompressed public keys are non-standard in segwit and may be explicitly disabled by a future soft fork. If the hash used in the P2WPKH came from an uncompressed public key, it may be unspendable and you may lose funds. P2WPKH outputs should be created by the payee's wallet by deriving a compressed public key from their private key.
[WARNING]
====
P2WPKH should be constructed by the payee (recipient), by converting a compressed public key to a P2WPKH hash. You should never transform a P2PKH script, bitcoin address or uncompressed public key to a P2WPKH witness script.
====
[[p2wsh]]
===== Pay-to-Witness-Script-Hash (P2WSH)
The second type of witness program corresponds to a Pay-to-Script-Hash (P2SH) script. We saw this type of script in <<p2sh>>. In that example, P2SH was used by Mohammed's company to express a multi-signature script. Payments to Mohammed's company were encoded with a locking script like this:
The P2SH script above references the hash of a _redeem script_ that defines a 2-of-3 multi-signature requirement to spend funds. To spend this output, Mohammed's company would present the redeem script (whose hash matches the script hash in the P2SH output) and the signatures necessary to satisfy that redeem script, all inside the transaction input:
.Decoded transaction showing a P2SH output being spent
Now, let's look at how this entire example would be upgraded to segwit. If Mohammed's customers were using a segwit compatible wallet, they would make a payment, creating a Pay-to-Witness-Script-Hash (P2WSH) output that would look like this:
Again, as with the example of P2WPKH, you can see that the segregated witness equivalent script is a lot simpler and omits the various script operands that you see in P2SH scripts. Instead, the segregated witness program consists of two values pushed to the stack: a witness version (0) and the 32-byte SHA256 hash of the redeem script.
[TIP]
====
While P2SH uses the 20-byte +RIPEMD160(SHA256(script))+ hash, the P2WSH witness program uses a 32-byte +SHA256(script)+ hash. This difference in the selection of the hashing algorithm is deliberate and used to differentiate between the two types of witness programs (P2WPKH and P2WSH) by the length of the hash, and to provide stronger security to P2WSH (128bits vs. 80bits of P2SH).
====
Mohammed's company can spend outputs the Pay-to-Witness-Script-Hash output by presenting the correct redeem script and sufficient signatures to satisfy the redeem script. Both the redeem script and the signatures would be segregated _outside_ the spending transaction as part of the witness data. Within the transaction input, Mohammed's wallet would put an empty scriptSig:
.Decoded transaction showing a P2WSH output being spent with separate witness data
In the previous two sections, we demonstrated two types of witness programs: <<p2wpkh>> and <<p2wsh>>. Both types of witness programs consist of single byte version number followed by a longer hash. They look very similar, but are interpreted very differently: one is interpreted as a public key hash, which is satisfied by a signature and the other as a script hash, which is satisfied by a redeem script. The critical difference between them is the length of the hash:
* The public key hash in P2WPKH is 20 bytes
* The script hash in P2WSH is 32 bytes
This is the one difference that allows a wallet to differentiate between the two types of witness programs. By looking at the length of the hash, a wallet can determine what type of witness program this is, P2WPKH or P2WSH.
==== Upgrading to Segregated Witness
As we can see from the examples above, upgrading to segregated witness is a two-step process. First, wallets must create special segwit type outputs. Then, these outputs can be spent by wallets that know how to construct segregated witness transactions. In the examples above, Alice's wallet was segwit-aware and able to create special outputs with segregated witness scripts. Bob's wallet is also segwit-aware and able to spend those outputs. What may not be obvious from the example is that in practice, Alice's wallet needs to _know_ that Bob uses a segwit-aware wallet and can spend these outputs. Otherwise, if Bob's wallet is not upgraded and Alice tries to make segwit payments to Bob, Bob's wallet will not be able to detect these payments.
[TIP]
====
For P2WPKH and P2WSH payment types, both the sender and the recipient wallets need to be upgraded to be able to use segwit. Furthermore, the sender's wallet needs to know that the recipient's wallet is segwit-aware.
====
Segregated witness will not be implemented simultaneously across the entire network. Rather, segregated witness is implemented as a backwards compatible upgrade, where _old and new clients can coexist_. Wallet developers will independently upgrade wallet software to add segwit capabilities. The P2WPKH and P2WSH payment types are intended for when both sender and recipient are segwit-aware. The traditional P2PKH and P2SH will continue to work for non-upgraded wallets. That leaves two important scenarios which are addressed in the next section:
* Ability of a sender's wallet that is not segwit-aware to make a payment to a recipient's wallet that can process segwit transactions.
* Ability of a sender's wallet that is segwit-aware to recognize and distinguish between recipients that are segwit-aware and ones that are not, by their _addresses_.
===== Embedding Segregated Witness Inside P2SH
Let's assume, for example, that Alice's wallet is not upgraded to segwit, but Bob's wallet is upgraded and can handle segwit transactions. Alice and Bob can use "old" non-segwit transactions. But Bob would likely want to use segwit to reduce transaction fees, taking advantage of the discount that applies to witness data.
In this case Bob's wallet can construct a P2SH address that contains a segwit script inside it. Alice's wallet sees this as a "normal" P2SH address and can make payments to it without any knowledge of segwit. Bob's wallet can then spend this payment with a segwit transaction, taking full advantage of segwit and reducing transaction fees.
Both forms of witness scripts, P2WPKH and P2WSH, can be embedded in a P2SH address. The first is noted as P2SH(P2WPKH) and the second is noted as P2SH(P2WSH).
The first form of witness script we will examine is P2SH(P2WPKH). This is a Pay-to-Witness-Public-Key-Hash witness program, embedded inside a Pay-to-Script-Hash script, so that it can be used by a wallet that is not aware of segwit.
Bob's wallet constructs a Pay-to-Witness-Public-Key-Hash (P2WPKH) witness program with Bob's public key. This witness program is then hashed and the resulting hash is encoded as a Pay-to-Script-Hash (P2SH) script. The P2SH script is converted to a bitcoin address, one which starts with a "3", as we saw in the <<p2sh>> section.
Bob's wallet starts with the P2WPKH witness program we saw earlier:
.Bob's P2WPKH witness program
----
0 ab68025513c3dbd2f7b92a94e0581f5d50f654e7
----
The P2WPKH witness program consists of the witness version and Bob's 20-byte public key hash.
Bob's wallet then hashes the above witness program, first with SHA256, then with RIPEMD160, producing another 20-byte hash:
.HASH160 of the P2WPKH witness program
----
660a5ab01c8468ac2e3cd58e9b17f8ba637867a4
----
The hash of the witness program is then embedded in a P2SH script:
.P2SH script containing the hash of a P2WPKH witness program
Finally, the P2SH script is converted to a P2SH bitcoin address:
.P2SH address
----
3AzZFY4WJJZbVr2A6qBTbdkYRpMLbdg6gD
----
Now, Bob can display this address for customers to pay for their coffee. Alice's wallet can make a payment to +3deadbeef+, just as it would to any other bitcoin address. Even though Alice's wallet has no support for segwit, the payment it creates can be spent by Bob with a segwit transaction.
Similarly, a P2WSH witness program for a multisig script or other complicated script can be embedded inside a Pay-to-Script-Hash script and address, making it possible for any wallet to make payments that are segwit compatible.
As we saw in <<p2wsh>>, Mohammed's company is using segregated witness payments to multi-signature scripts. To make it possible for any client to pay his company, regardless of whether their wallets are upgraded for segwit, Mohammed's wallet can embed the P2WSH witness program inside a P2SH script.
First, Mohammed's wallet creates the P2WSH witness program that corresponds to the multi-signature script, hashed with SHA256:
.Mohammed's wallet creates a P2WSH witness program
Finally, the wallet constructs a bitcoin address from this script:
.P2SH bitcoin address
----
3NTWTcFE88p26GTPoxcWef9Q5ncKt6CY2E
----
Now, Mohammed's clients can make payments to this address without any need to support segwit. Mohammed's company can then construct segwit transactions to spend these payments, taking advantage of segwit features including lower transaction fees.
===== Segregated Witness Addresses
After segwit is deployed on the bitcoin network, it will take some time until wallets are upgraded. It is quite likely therefore that segwit will mostly be used embedded in P2SH, as we saw in the previous section, at least for several months.
Eventually however, almost all wallets will be able to support segwit payments. At that time it will no longer be necessary to embed segwit in P2SH. It is therefore likely that a new form of bitcoin address will be created, one that indicates the recipient is segwit-aware and which directly encodes a witness program. There have been a number of proposals for a segregated witness address scheme, but none have been actively pursued at this time.
[[segwit_txid]]
===== Transaction Identifiers
One of the greatest benefits of Segregated Witness is that it eliminates third-party transaction malleability.
Before segwit, transactions could have their signatures subtly modified by third parties, changing their transaction ID (hash) without changing any fundamental properties (inputs, outputs, amounts). This created opportunities for Denial-of-Service attacks as well as attacks against poorly written wallet software that assumed unconfirmed transaction-hashes were immutable.
With the introduction of Segregated Witness, transactions have two identifiers, +txid+ and +wtxid+. The traditional transaction ID +txid+ is the double-SHA256 hash of the serialized transaction, without the witness data. A transaction +wtxid+ is the double-SHA256 hash of the new serialization format of the transaction with witness data.
The traditional +txid+ is calculated in exactly the same way as with a non-segwit transaction. However, since the segwit transaction has empty scriptSig's in every input, there is no part of the transaction that can be modified by a third party. Therefore, in a segwit transaction, the +txid+ is immutable by a third party, even when the transaction is unconfirmed.
The +wtxid+ is like an "extended" ID, in that the hash also incorporates the witness data. If a transaction is transmitted without witness data, then the +wtxid+ and +txid+ are identical. Note than since the +wtxid+ includes witness data (signatures) and since witness data may be malleable, the +wtxid+ should be considered malleable until the transaction is confirmed. Only the +txid+ of a segwit transaction can be considered immutable by third parties and only if _all_ the inputs of the transaction are segwit inputs.
[TIP]
====
Segregated Witness transactions have two IDs: +txid+ and +wtxid+. The +txid+ is the hash of the transaction without the witness data and the +wtxid+ is the hash inclusive of witness data. The +txid+ of a transaction where all inputs are segwit inputs, is not susceptible to third-party transaction malleability
====
==== Segregated Witness' New Signing Algorithm
Segregated Witness modifies the semantics of the four signature verification functions (CHECKSIG, CHECKSIGVERIFY, CHECKMULTISIG and CHECKMULTISIGVERIFY), changing the way a transaction commitment hash is calculated.
Signatures in bitcoin transactions are applied on a _commitment hash_ which is calculated from the transaction data, locking specific parts of the data indicating the signer's commitment to those values. For example, in a simple SIGHASH_ALL type signature, the commitment hash includes all inputs and outputs.
Unfortunately, the way the commitment hash was calculated introduced the possibility that a node verifying the signature can be forced to perform a significant number of hash computations. Specifically, the hash operations increase in O(n^2^) with respect to the number of signature operations in the transaction. An attacker could therefore create a transaction with a very large number of signature operations, causing the entire bitcoin network to have to perform hundreds or thousands of hash operations to verify the transaction.
Segwit represented an opportunity to address this problem by changing the way the commitment hash is calculated. For segwit version 0 witness programs, signature verification occurs using an improved commitment hash algorithm as specified in Bitcoin Improvement Proposal 143 (BIP143).
The new algorithm achieves two important goals. Firstly, the number of hash operations increases by a much more gradual O(n) to the number of signature operations, reducing the opportunity to create Denial-of-Service attacks with overly complex transactions. Secondly, the commitment hash now also includes the value (amounts) of each input as part of the commitment. This means that a signer can commit to a specific input value without needing to "fetch" and check the previous transaction referenced by the input. In the case of offline devices, such as hardware wallets, this greatly simplifies the communication between the host and the hardware wallet, removing the need to stream previous transactions for validation. A hardware wallet can accept the input value "as stated" by an untrusted host. Since the signature is invalid if that input value is not correct, the hardware wallet doesn't need to validate the value before signing the input.
==== Economic Incentives for Segregated Witness
Bitcoin mining nodes and full nodes incur costs for the resources used to support the bitcoin network and the blockchain. As the volume of bitcoin transactions increases, so does the cost of resources (CPU, network bandwidth, disk space, memory). Miners are compensated for these costs through fees that are proportional to the size (in bytes) of each transaction. Non-mining full nodes are not compensated, so they incur these costs because they have a need to run an authoritative fully-validating full-index node, perhaps because they use the node to operate a bitcoin business.
Without transaction fees, the growth in bitcoin data would arguably increase dramatically. Fees are intended to align the needs of bitcoin users with the burden their transactions impose on the network, through a market-based price discovery mechanism.
The calculation of fees based on transaction size treats all the data in the transaction as equal in cost. But from the perspective of full nodes and miners, some parts of a transaction carry much higher costs. Every transaction added to the bitcoin network affects the consumption of four resources on nodes:
Disk Space :: Every transaction is stored in the blockchain, adding to the total size of the blockchain. The blockchain is stored on disk, but the storage can be optimized by “pruning” older transactions.
CPU :: Every transaction must be validated, which requires CPU time.
Bandwidth :: Every transaction is transmitted (through flood propagation) across the network at least once. Without any optimization in the block propagation protocol, transactions are transmitted again as part of a block, doubling the impact on network capacity
Memory :: Nodes that validate transactions keep the UTXO index or the entire UTXO set in memory to speed up validation. Because memory is at least one order of magnitude more expensive than disk, growth of the UTXO set contributes disproportionately to the cost of running a node.
As you can see from the list above, not every part of a transaction has an equal impact on the cost of running a node or on the ability of bitcoin to scale to support more transactions. The most expensive part of a transaction are the newly created outputs, as they are added to the in-memory UTXO set. By comparison, signatures (aka witness data) add the least burden to the network and the cost of running a node, because witness data are only validated once and then never used again. Furthermore, immediately after receiving a new transaction and validating witness data, nodes can discard that witness data. If fees are calculated on transaction size, without discriminating between these two types of data, then the market incentives of fees are not aligned with the actual costs imposed by a transaction. In fact, the current fee structure actually encourages the opposite behavior, because witness data is the largest part of a transaction.
The incentives created by fees matter because they affect the behavior of wallets. All wallets must implement some strategy for assembling transactions that takes into considerations a number of factors, such as privacy (reducing address re-use), fragmentation (making lots of loose change) and fees. If the fees are overwhelmingly motivating wallets to use as few inputs as possible in transactions, this can lead to UTXO picking and change address strategies that inadvertently bloat the UTXO set.
Transactions consume UTXO in their inputs and create new UTXO with their outputs. A transaction, therefore, that has more inputs than outputs will result in a decrease in the UTXO set, whereas a transaction that has more outputs than inputs will result in an increase in the UTXO set. Let’s consider the _difference_ between inputs and outputs and call that the “Net new UTXO”. That’s an important metric, as it tells us what impact a transaction will have on the most expensive network-wide resource, the in-memory UTXO set. A transaction with positive Net-new-UTXO, adds to that burden. A transaction with a negative Net-new-UTXO reduces the burden. We would therefore want to encourage transactions that are either negative Net-new-UTXO or neutral with zero Net-new-UTXO.
Let’s look at an example of what incentives are created by the transaction fee calculation, with and without segregated witness. We will look at two different transactions. Transaction A is a 3-input, 2-output transaction, which has a Net-new-UTXO metric of -1, meaning it consumes one more UTXO than it creates, reducing the UTXO set by one. Transaction B is a 2-input, 3-output transaction, which has a Net-new-UTXO metric of 1, meaning it adds one UTXO to the UTXO set, imposing additional cost on the entire bitcoin network. Both transactions use multi-signature (2-of-3) scripts, to demonstrate how complex scripts increase the impact of segregated witness on fees. Let’s assume a transaction fee of 30 satoshi per byte and a 75% fee discount on witness data:
Without Segregated Witness
Transaction A fee: 25,710 satoshi
Transaction B fee: 18,990 satoshi
With Segregated Witness
Transaction A fee: 8,130 satoshi
Transaction B fee: 12,045 satoshi
Both transactions are less expensive when segregated witness is implemented. But comparing the costs between the two transactions, we see that before segregated witness, the fee is higher for the transaction that has a negative Net-new-UTXO. After segregated witness, the transaction fees align with the incentive to minimize new UTXO creation, by not inadvertently penalizing transactions with many inputs.
Segregated witness therefore has two main effects on the fees paid by bitcoin users. Firstly, segwit reduces the overall cost of transactions by discounting witness data and increasing the capacity of the bitcoin blockchain. Secondly, segwit’s discount on witness data correcting a misalignment of incentives that may have inadvertently created more bloat in the UTXO set.
=== Proposed Future Scripting and Transaction Improvements
==== Confidential Transactions
==== Schnorr Signatures
==== Merkleized Abstract Syntax Trees (Pay-to-Merkle-Root)
The preceding locking script can be satisfied with an unlocking script containing pairs of signatures and public keys:
----
OP_0 <Signature B> <Signature C>
<Signature B> <Signature C>
----
or any combination of two signatures from the private keys corresponding to the three listed public keys.
[NOTE]
====
((("CHECKMULTISIG implementation")))The prefix +OP_0+ is required because of a bug in the original implementation of +CHECKMULTISIG+ where one item too many is popped off the stack. It is ignored by +CHECKMULTISIG+ and is simply a placeholder.
====
The two scripts together would form the combined validation script:
When executed, this combined script will evaluate to TRUE if, and only if, the unlocking script matches the conditions set by the locking script. In this case, the condition is whether the unlocking script has a valid signature from the two private keys that correspond to two of the three public keys set as an encumbrance.
[[multisig_bug]]
===== A bug in CHECKMULTISIG execution
There is a bug in CHECKMULTISIG's execution that requires a slight workaround. When CHECKMULTISIG executes, it should consume M+N+2 items on the stack as parameters. However, due to the bug, CHECKMULTISIG attempts to pop one value more than expected.
Let's look at this in greater detail using the validation example above.
First, CHECKMULTISIG pops the top item which is +N+, in this example "3". Then it pops +N+ items, which are the public keys that can sign. In this example, public keys A, B, and C. Then, it pops one item which is +M+, the quorum (how many signatures are needed). Here M = 2. At this point, CHECKMULTISIG should pop the final +M+ items, which are the signatures and see if they are valid. But, unfortunately, a bug in the implementation causes CHECKMULTISIG to pop one more item (M+1 total) than it should. The extra item is disregarded when checking the signatures so it has no direct effect on CHECKMULTISIG itself. However, an extra value must be present because if it is not present, when CHECKMULTISIG attempts to pop on an empty stack, it will cause a stack error and script failure (marking the transaction as invalid). Because the extra item is disregarded it can be anything, but customarily +0+ is used.
Because this bug became part of the consensus rules, it must now be replicated forever. Therefore, the correct script validation would look like this:
Therefore, the unlocking script actually used in multisig is not:
----
<Signature B> <Signature C>
----
but instead it is:
----
0 <Signature B> <Signature C>
----
From now on, if you see a multisig unlocking script, you should expect to see an extra +0+ in the beginning, whose only purpose is as a workaround to a bug that accidentally became a consensus rule.
[[p2sh]]
=== Pay-to-Script-Hash (P2SH)
@ -58,7 +86,7 @@ In <<ch01_intro_what_is_bitcoin>> we introduced Mohammed, an electronics importe
The resulting script is quite long and looks like this:
----
2 <Mohammed's Public Key> <Partner1 Public Key> <Partner2 Public Key> <Partner3 Public Key> <Attorney Public Key> 5 OP_CHECKMULTISIG
2 <Mohammed's Public Key> <Partner1 Public Key> <Partner2 Public Key> <Partner3 Public Key> <Attorney Public Key> 5 CHECKMULTISIG
----
Although multi-signature scripts are a powerful feature, they are cumbersome to use. Given the preceding script, Mohammed would have to communicate this script to every customer prior to payment. Each customer would have to use special bitcoin wallet software with the ability to create custom transaction scripts, and each customer would have to understand how to create a transaction using custom scripts. Furthermore, the resulting transaction would be about five times larger than a simple payment transaction, because this script contains very long public keys. The burden of that extra-large transaction would be borne by the customer in the form of fees. Finally, a large transaction script like this would be carried in the UTXO set in RAM in every full node, until it was spent. All of these issues make using complex output scripts difficult in practice.
@ -70,15 +98,15 @@ In P2SH transactions, the locking script that is replaced by a hash is referred
@ -89,14 +117,14 @@ Let's look at Mohammed's company, the complex multi-signature script, and the re
First, the multi-signature script that Mohammed's company uses for all incoming payments from customers:
----
2 <Mohammed's Public Key> <Partner1 Public Key> <Partner2 Public Key> <Partner3 Public Key> <Attorney Public Key> 5 OP_CHECKMULTISIG
2 <Mohammed's Public Key> <Partner1 Public Key> <Partner2 Public Key> <Partner3 Public Key> <Attorney Public Key> 5 CHECKMULTISIG
----
If the placeholders are replaced by actual public keys (shown here as 520-bit numbers starting with 04) you can see that this script becomes very long:
This entire script can instead be represented by a 20-byte cryptographic hash, by first applying the SHA256 hashing algorithm and then applying the RIPEMD160 algorithm on the result. The 20-byte hash of the preceding script is:
@ -108,25 +136,27 @@ This entire script can instead be represented by a 20-byte cryptographic hash, b
A P2SH transaction locks the output to this hash instead of the longer script, using the locking script:
which, as you can see, is much shorter. Instead of "pay to this 5-key multi-signature script," the P2SH equivalent transaction is "pay to a script with this hash." A customer making a payment to Mohammed's company need only include this much shorter locking script in his payment. When Mohammed wants to spend this UTXO, they must present the original redeem script (the one whose hash locked the UTXO) and the signatures necessary to unlock it, like this:
Almost all the scripts described in this chapter can only be implemented as Pay-to-Script-Hash scripts. They cannot be used directly in the locking script of a UTXO.
==== Pay-to-script-hash addresses
((("addresses, bitcoin","Pay-to-Script-Hash (P2SH)")))((("Pay-to-script-hash (P2SH)","addresses")))Another important part of the P2SH feature is the ability to encode a script hash as an address, as defined in BIP-13. P2SH addresses are Base58Check encodings of the 20-byte hash of a script, just like bitcoin addresses are Base58Check encodings of the 20-byte hash of a public key. P2SH addresses use the version prefix "5", which results in Base58Check-encoded addresses that start with a "3". For example, Mohammed's complex script, hashed and Base58Check-encoded as a P2SH address becomes +39RF6JqABiHdYHkfChV6USGMe6Nsr66Gzw+. Now, Mohammed can give this "address" to his customers and they can use almost any bitcoin wallet to make a simple payment, as if it were a bitcoin address. The 3 prefix gives them a hint that this is a special type of address, one corresponding to a script instead of a public key, but otherwise it works in exactly the same way as a payment to a bitcoin address.
@ -144,18 +174,13 @@ P2SH addresses hide all of the complexity, so that the person making a payment d
* P2SH shifts the burden in data storage for the long script from the present time (payment) to a future time (when it is spent).
* P2SH shifts the transaction fee cost of a long script from the sender to the recipient, who has to include the long redeem script to spend it.
==== Limitations on redeem scripts
////
10,000 bytes, 201 ops
////
==== Redeem script and validation
((("pay-to-script-hash (P2SH)","isStandard validation")))((("pay-to-script-hash (P2SH)","redeem script for")))Prior to version 0.9.2 of the Bitcoin Core client, pay-to-script-hash was limited to the standard types of bitcoin transaction scripts, by the +isStandard()+ function. That means that the redeem script presented in the spending transaction could only be one of the standard types: P2PK, P2PKH, or multi-sig nature, excluding +OP_RETURN+ and P2SH itself.
((("pay-to-script-hash (P2SH)","isStandard validation")))((("pay-to-script-hash (P2SH)","redeem script for")))Prior to version 0.9.2 of the Bitcoin Core client, pay-to-script-hash was limited to the standard types of bitcoin transaction scripts, by the +isStandard()+ function. That means that the redeem script presented in the spending transaction could only be one of the standard types: P2PK, P2PKH, or multi-sig nature, excluding +RETURN+ and P2SH itself.
As of version 0.9.2 of the Bitcoin Core client, P2SH transactions can contain any valid script, making the P2SH standard much more flexible and allowing for experimentation with many novel and complex types of transactions.
Note that you are not able to put a P2SH inside a P2SH redeem script, because the P2SH specification is not recursive. You are also still not able to use +OP_RETURN+ in a redeem script because +OP_RETURN+ cannot be redeemed by definition.
Note that you are not able to put a P2SH inside a P2SH redeem script, because the P2SH specification is not recursive. You are also still not able to use +RETURN+ in a redeem script because +RETURN+ cannot be redeemed by definition.
Note that because the redeem script is not presented to the network until you attempt to spend a P2SH output, if you lock an output with the hash of an invalid transaction it will be processed regardless. However, you will not be able to spend it because the spending transaction, which includes the redeem script, will not be accepted because it is an invalid script. This creates a risk, because you can lock bitcoin in a P2SH that cannot be spent later. The network will accept the P2SH encumbrance even if it corresponds to an invalid redeem script, because the script hash gives no indication of the script it represents.
@ -164,50 +189,39 @@ Note that because the redeem script is not presented to the network until you at
((("Pay-to-Script-Hash (P2SH)","locking scripts")))P2SH locking scripts contain the hash of a redeem script, which gives no clues as to the content of the redeem script itself. The P2SH transaction will be considered valid and accepted even if the redeem script is invalid. You might accidentally lock bitcoin in such a way that it cannot later be spent.(((range="endofrange", startref="ix_ch05-asciidoc19")))(((range="endofrange", startref="ix_ch05-asciidoc18")))(((range="endofrange", startref="ix_ch05-asciidoc17")))(((range="endofrange", startref="ix_ch05-asciidoc0")))
====
=== Scripts with Conditional Clauses
////
// script to put a number on the stack
OP_IF
// script that runs only if the number isn't zero
OP_ELSE
// script that runs only if the number is zero
OP_ENDIF
// script that runs no matter what
////
[[op_return]]
=== Data Recording Output (OP_RETURN)
=== Data Recording Output (RETURN)
((("ledger, storing unrelated information in")))((("OP_RETURN operator")))((("transactions","storing unrelated information in")))Bitcoin's distributed and timestamped ledger, the blockchain, has potential uses far beyond payments. Many developers have tried to use the transaction scripting language to take advantage of the security and resilience of the system for applications such as((("digital notary services")))((("smart contracts")))((("stock certificates"))) digital notary services, stock certificates, and smart contracts. Early attempts to use bitcoin's script language for these purposes involved creating transaction outputs that recorded data on the blockchain; for example, to record a digital fingerprint of a file in such a way that anyone could establish proof-of-existence of that file on a specific date by reference to that transaction.
((("ledger, storing unrelated information in")))((("RETURN operator")))((("transactions","storing unrelated information in")))Bitcoin's distributed and timestamped ledger, the blockchain, has potential uses far beyond payments. Many developers have tried to use the transaction scripting language to take advantage of the security and resilience of the system for applications such as((("digital notary services")))((("smart contracts")))((("stock certificates"))) digital notary services, stock certificates, and smart contracts. Early attempts to use bitcoin's script language for these purposes involved creating transaction outputs that recorded data on the blockchain; for example, to record a digital fingerprint of a file in such a way that anyone could establish proof-of-existence of that file on a specific date by reference to that transaction.
((("blockchains","storing unrelated information in")))The use of bitcoin's blockchain to store data unrelated to bitcoin payments is a controversial subject. Many developers consider such use abusive and want to discourage it. Others view it as a demonstration of the powerful capabilities of blockchain technology and want to encourage such experimentation. Those who object to the inclusion of non-payment data argue that it causes "blockchain bloat," burdening those running full bitcoin nodes with carrying the cost of disk storage for data that the blockchain was not intended to carry. Moreover, such transactions create UTXO that cannot be spent, using the destination bitcoin address as a free-form 20-byte field. Because the address is used for data, it doesn't correspond to a private key and the resulting UTXO can _never_ be spent; it's a fake payment. These transactions that can never be spent are therefore never removed from the UTXO set and cause the size of the UTXO database to forever increase, or "bloat."
In version 0.9 of the Bitcoin Core client, a compromise was reached with the introduction of the +OP_RETURN+ operator. +OP_RETURN+ allows developers to add 80 bytes of nonpayment data to a transaction output. However, unlike the use of "fake" UTXO, the +OP_RETURN+ operator creates an explicitly _provably unspendable_ output, which does not need to be stored in the UTXO set. +OP_RETURN+ outputs are recorded on the blockchain, so they consume disk space and contribute to the increase in the blockchain's size, but they are not stored in the UTXO set and therefore do not bloat the UTXO memory pool and burden full nodes with the cost of more expensive RAM.
In version 0.9 of the Bitcoin Core client, a compromise was reached with the introduction of the +RETURN+ operator. +RETURN+ allows developers to add 80 bytes of nonpayment data to a transaction output. However, unlike the use of "fake" UTXO, the +RETURN+ operator creates an explicitly _provably unspendable_ output, which does not need to be stored in the UTXO set. +RETURN+ outputs are recorded on the blockchain, so they consume disk space and contribute to the increase in the blockchain's size, but they are not stored in the UTXO set and therefore do not bloat the UTXO memory pool and burden full nodes with the cost of more expensive RAM.
+OP_RETURN+ scripts look like this:
+RETURN+ scripts look like this:
----
OP_RETURN <data>
RETURN <data>
----
The data portion is limited to 80 bytes and most often represents a hash, such as the output from the SHA256 algorithm (32 bytes). Many applications put a prefix in front of the data to help identify the application. For example, the http://proofofexistence.com[Proof of Existence] digital notarization service uses the 8-byte prefix +DOCPROOF+, which is ASCII encoded as +44 4f 43 50 52 4f 4f 46+ in hexadecimal.
Keep in mind that there is no "unlocking script" that corresponds to +OP_RETURN+ that could possibly be used to "spend" an +OP_RETURN+ output. The whole point of +OP_RETURN+ is that you can't spend the money locked in that output, and therefore it does not need to be held in the UTXO set as potentially spendable—+OP_RETURN+ is _provably un-spendable_. +OP_RETURN+ is usually an output with a zero bitcoin amount, because any bitcoin assigned to such an output is effectively lost forever. If an +OP_RETURN+ is referenced as an input in a transaction, the script validation engine will halt the execution of the validation script and marking the transaction as invalid. The execution of OP_RETURN, essentially causes the script to "RETURN" with a FALSE and halt. Thus, if you accidentally reference an +OP_RETURN+ output as an input in a transaction, that transaction is invalid.
Keep in mind that there is no "unlocking script" that corresponds to +RETURN+ that could possibly be used to "spend" an +RETURN+ output. The whole point of +RETURN+ is that you can't spend the money locked in that output, and therefore it does not need to be held in the UTXO set as potentially spendable—+RETURN+ is _provably un-spendable_. +RETURN+ is usually an output with a zero bitcoin amount, because any bitcoin assigned to such an output is effectively lost forever. If an +RETURN+ is referenced as an input in a transaction, the script validation engine will halt the execution of the validation script and marking the transaction as invalid. The execution of RETURN, essentially causes the script to "RETURN" with a FALSE and halt. Thus, if you accidentally reference an +RETURN+ output as an input in a transaction, that transaction is invalid.
A standard transaction (one that conforms to the +isStandard()+ checks) can have only one +OP_RETURN+ output. However, a single +OP_RETURN+ output can be combined in a transaction with outputs of any other type.
A standard transaction (one that conforms to the +isStandard()+ checks) can have only one +RETURN+ output. However, a single +RETURN+ output can be combined in a transaction with outputs of any other type.
Two new command-line options have been added in Bitcoin Core as of version 0.10. The option +datacarrier+ controls relay and mining of OP_RETURN transactions, with the default set to "1" to allow them. The option +datacarriersize+ takes a numeric argument specifying the maximum size in bytes of the OP_RETURN data, 40 bytes by default.
Two new command-line options have been added in Bitcoin Core as of version 0.10. The option +datacarrier+ controls relay and mining of RETURN transactions, with the default set to "1" to allow them. The option +datacarriersize+ takes a numeric argument specifying the maximum size in bytes of the RETURN data, 40 bytes by default.
[NOTE]
====
OP_RETURN was initially proposed with a limit of 80 bytes, but the limit was reduced to 40 bytes when the feature was released. In February 2015, in version 0.10 of Bitcoin Core, the limit was raised back to 80 bytes. Nodes may choose not to relay or mine OP_RETURN, or only relay and mine OP_RETURN containing less than 80 bytes of data.
RETURN was initially proposed with a limit of 80 bytes, but the limit was reduced to 40 bytes when the feature was released. In February 2015, in version 0.10 of Bitcoin Core, the limit was raised back to 80 bytes. Nodes may choose not to relay or mine RETURN, or only relay and mine RETURN containing less than 80 bytes of data.
====
=== Timelocks
Timelocks are restrictions on transactions or outputs that only allow spending after a point in time. Bitcoin has had a transaction-level timelock feature from the beginning. It is implemented by the nLocktime filed in a transaction. Two new timelock features were introduced in late 2015 and mid-2016 that offer UTXO-level timelocks. These are OP_+CHECKLOCKTIMEVERIFY+, and OP_CHECKSEQUENCEVERIFY.
Timelocks are restrictions on transactions or outputs that only allow spending after a point in time. Bitcoin has had a transaction-level timelock feature from the beginning. It is implemented by the nLocktime filed in a transaction. Two new timelock features were introduced in late 2015 and mid-2016 that offer UTXO-level timelocks. These are +CHECKLOCKTIMEVERIFY+, and CHECKSEQUENCEVERIFY.
Timelocks are useful for post-dating transactions and locking funds to a date in the future. More importantly, timelocks extend bitcoin scripting into the dimension of time, opening the door for complex multi-step smart contracts. We will examine the use of timelocks for smart contracts in <<state_channels>>.
@ -234,9 +248,9 @@ It is important to understand the limitations of transaction nLockTime. The only
==== Check Lock Time Verify (CLTV)
In December of 2015, a new form of timelock was introduced to bitcoin as a soft-fork upgrade. Based on a specification in Bitcoin Improvement Proposal 65 (BIP-65), a new script operator _OP_CHECKLOCKTIMEVERIFY_ (known also as _CLTV_) was added to the scripting language. +CLTV+ is a per-output timelock, rather than a per-transaction timelock as is the case with nLocktime. This allows for much greater flexibility in the way timelocks are applied.
In December of 2015, a new form of timelock was introduced to bitcoin as a soft-fork upgrade. Based on a specification in Bitcoin Improvement Proposal 65 (BIP-65), a new script operator _CHECKLOCKTIMEVERIFY_ (known also as _CLTV_) was added to the scripting language. +CLTV+ is a per-output timelock, rather than a per-transaction timelock as is the case with nLocktime. This allows for much greater flexibility in the way timelocks are applied.
In simple terms, by adding the +CLTV+ opcode in the locking script of an output it restricts the output, so that it can only be spent after the specified time has elapsed.
In simple terms, by adding the +CLTV+ opcode in the redeem script of an output it restricts the output, so that it can only be spent after the specified time has elapsed.
[TIP]
====
@ -245,23 +259,21 @@ While nLocktime is a transaction level timelock, CLTV is an output based timeloc
====
+CLTV+ doesn't replace nLocktime, but rather restricts specific UTXO such that they can only be spent in a future transaction with nLocktime set to a greater or equal value.
As Peter Todd, the author of BIP-65, astutely says: "CHECKLOCKTIMEVERIFY works how you thought nLocktime worked"
The +CLTV+ opcode takes one parameter as input, expressed as a number in the same format as nLocktime (either a block height or Unix epoch time). As indicated by the +VERIFY+ suffix, +CLTV+ is the type of opcode that halts execution of the script if the outcome is +FALSE+. If it results in TRUE, execution continues.
In order to lock a output with +CLTV+, you insert it into the locking script of the output, in the transaction that creates the output. For example, if Alice is paying Bob's address, the output would normally contain a P2PKH script like this:
In order to lock a output with +CLTV+, you insert it into the redeem script of the output, in the transaction that creates the output. For example, if Alice is paying Bob's address, the output would normally contain a P2PKH script like this:
----
DUP HASH160 <Bob's Public Key Hash> EQUALVERIFY
----
To lock it to a time, say 3 months from now, the locking script would instead look like this:
To lock it to a time, say 3 months from now, the transaction would be a P2SH transaction with a redeem script like this:
----
<now + 3 months> CHECKLOCKTIMEVERIFY DROP DUP HASH160 <Bob's Public Key Hash> EQUALVERIFY
----
where +<now {plus} 3 months>+ is a block height or time value estimated 3 months from the time the transaction is mined: current block height {plus} 12,960 or current Unix epoch time {plus} 7,760,000 seconds. For now, don't worry about the +DROP+ opcode that follows +CHECKLOCKTIMEVERIFY+, it will be explained shortly.
where +<now {plus} 3 months>+ is a block height or time value estimated 3 months from the time the transaction is mined: current block height {plus} 12,960 (blocks) or current Unix epoch time {plus} 7,760,000 (seconds). For now, don't worry about the +DROP+ opcode that follows +CHECKLOCKTIMEVERIFY+, it will be explained shortly.
When Bob tries to spend this UTXO, he constructs a transaction which references the UTXO as an input. He uses his signature and public key in the unlocking script of that input and sets the transaction nLocktime to be equal or greater to the timelock in the +CHECKLOCKTIMEVERIFY+ Alice set. Bob then broadcasts the transaction on the bitcoin network.
@ -287,417 +299,292 @@ After execution, if CHECKLOCKTIMEVERIFY is satisfied, the time-parameter that pr
By using nLocktime in conjunction with +CLTV+, the scenario described in <<locktime_limitations>> changes. Because Alice locked the UTXO itself, it is now impossible for either Bob or Alice to spend it before the 3-month locktime has expired.
By introducing timelock functionality directly in the scripting language, +CLTV+ allows us to develop some very interesting complex scripts, as we will see in the next example.
BIP-65 offers an example of a multi-signature script with varying conditions depending on when it is redeemed. Alice and Bob run a business together. They want to store their funds in a multi-signature wallet, but want to add a key held by a third-party as a backup in case they lose one of the keys or one of them is incapacitated. They want to give the third key to their attorney, Lenny.
If Alice and Bob use a 2-of-3 multi-sig with Lenny as the third key, they always run the risk that Lenny could collude with either one of them to steal the funds. Instead they implement a system which is a 2-of-2 multisig, unless the funds are not spent within 3 months. At that point, Lenny can add a signature and unlock the funds with either Alice or Bob's agreement.
It looks like this:
.Variable Multi-Signature with Timelock
----
IF
<now + 3 months> CHECKLOCKTIMEVERIFY DROP
<Lenny's pubkey> CHECKSIGVERIFY
1
ELSE
2
ENDIF
<Alice's pubkey> <Bob's pubkey> 2 CHECKMULTISIG
----
This locking script can be unlocked, anytime with:
By introducing timelock functionality directly in the scripting language, +CLTV+ allows us to develop some very interesting complex scripts.
In that "mode" it operates a lot like a 2-of-2 multi-sig script.
=== Relative Timelocks
However, if Alice and Bob leave these funds unspent for 3 months, after that time, the locking script can also be unlocked with:
nLocktime and CLTV are both _absolute timelocks_ in that the specify an absolute point in time. The next two timelock features we will examine are _relative timelocks_ in that they specify, as a condition of spending an output, an elapsed time from the confirmation of the output in the blockchain.
----
0 <Alice or Bob's signature> <Lenny's signature> 1
----
Relative timelocks are useful because they allow a chain of two or more interdependent transactions to be held off chain, while imposing a time constraint on one transaction that is dependent on the elapsed time from the confirmation of a previous transaction. In other words, the clock doesn't start counting until the UTXO is recorded on the blockchain. This functionality is especially useful in bi-directional state channels and Lightning network, as we will see in <<state_channels>>.
So, if the funds are unspent for 3 months, the script behaves like a 2-of-3 multi-sig.
Relative timelocks, like absolute timelocks are implemented with both a transaction-level feature and a script-level opcode. The transaction-level relative timelock is implemented as a consensus rule on the value of nSequence, a transaction field which is set in every transaction input. Script-level relative timelocks are implemented with the CHECKSEQUENCEVERIFY opcode.
To understand what is happening in the locking script, examine the +IF...ELSE+ conditional clauses. You will notice that in the first clause there is a trailing +1+, just before the ELSE. In the second conditional clause there is a +2+ right before the +ENDIF+.
Relative timelocks are implemented according to the specifications in the following BIPs:
You will also notice that the +CHECKMULTISIG+ script seems incomplete. It should have the +M+ parameter (the quorum), before Alice's pubkey. Now if you connect these two facts, you will see that the conditional clauses leave the number +1+ or the number +2+ on the stack, which serves as the quorum number for the multisig.
BIP-68 Relative lock-time using consensus-enforced sequence numbers:: https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki]
Essentially, the conditional clause changes the script from a 2-of-2 Alice and Bob multisig to a Lenny plus 1-of-2 Alice and Bob multisig depending on whether it is executed before or after 3 months. In addition, Lenny's key is required in the first conditional clause, only if 3 months have elapsed.
Additionally, you might be wondering how the conditional clauses work. What is the IF testing? Look at the unlocking scripts and you will see that one of them ends in +0+, the other ends in +1+. Essentially, the unlocking scripts are *choosing* which branch of the conditional clause to execute. Anytime, Alice and Bob can use the unlocking script that ends in +0+ for "FALSE", executing the second clause (the ELSE clause). After 3 months, they may instead also use the other unlocking script, ending in +1+ for TRUE, activating the first clause (the IF clause).
BIP-68 and BIP-112 were activated in May 2016 as a soft-fork upgrade to the consensus rules.
Finally, notice how the +CHECKLOCKTIMEVERIFY+ serves as a "guard" of the first conditional clause. If you try to execute it before three months have elapsed, by putting a +1+ on the end of any unlocking script, it will simply mark the transaction invalid, as the +CHECKLOCKTIMEVERIFY+ will terminate validation.
Try running the script on paper to see how it behaves on the stack.
The example contained in BIP-65 is one of several offered in that specification:
Relative timelocks can be set on each input of a transaction, by setting the nSequence field in each input.
===== Original meaning of nSequence
[[segwit]]
=== Segregated Witness
The nSequence field was originally intended (but never properly implemented) to allow modification of transactions in the mempool. In that use, a transaction containing inputs with nSequence value below 2^32^ (0xFFFFFFFF) indicated a transaction that was not yet "finalized". Such a transaction would be held in the mempool until it was replaced by another transaction spending the same inputs with a higher nSequence value. Once a transaction was received whose inputs had an nSequence value of 2^32^ it would be considered "finalized" and mined.
Segregated Witness (segwit) is an upgrade to the bitcoin consensus rules and network protocol, scheduled for implementation in the second half of 2016.
The original meaning of nSequence was never properly implemented and the value of nSequence is customarily set to 2^32^ in transactions that do not utilize timelocks. For transactions with nLocktime or CHECKLOCKTIMEVEIRFY, the nSequence value must be set to less than 2^32^. Customarily, it is set to 2^32^ - 1 (0xFFFFFFFE).
In cryptography, the term "witness" is used to describe a solution to a cryptographic puzzle. In bitcoin terms, the witness satisfies a cryptographic condition placed on a Unspent Transaction Output (UTXO).
Programmatically, that means that if the most significant bit (1<<31) is not set, it is a flag that means "relative locktime". Otherwise (bit 1<<31 set), the nSequnce value is reserved for other uses such as enabling CHECKLOCKTIMEVERIFY, nLocktime, Opt-In-Replace-By-Fee and other future developments.
In the context of bitcoin, a digital signature is _one type of witness_, but a witness is more broadly any solution that can satisfy the conditions imposed on a UTXO and unlock that UTXO for spending. The term “witness” is a more general term for an “unlocking script” or “scriptSig”.
===== nSequence as consensus-enforced relative timelock
Before segwit’s introduction, every input in a transaction was followed by the witness data that unlocked it. The witness data was embedded in the transaction as part of each input, The term _segregated witness_ or _segwit_ for short, simply means separating the signature or unlocking script of a specific output. Think "separate scriptSig", or “separate signature” in the simplest form.
Since the activation of BIP-68, the new consensus rules apply for any transaction containing an input whose nSequence value is less than 2^31^ (bit 1<<31 is not set).
Segregated Witness therefore is an architectural change to bitcoin that aims to move the witness data from the scriptSig (unlocking script) field of a transaction into separate a _witness_ data structure that accompanies a transaction. Clients may request transaction data with or without the accompanying witness data.
For transactions setting nSequence as a relative timelock, the UTXO referenced by that input must be "older" than the value of nSequence or the transaction is invalid.
In this section we will look at some of the benefits of segregated witness, describe the mechanism used to deploy and implement this architecture change and demonstrate the use of segregated witness in transactions and addresses.
The nSequence value is specified in either blocks or seconds, but in a slightly different format than we saw used in nLocktime. A type-flag is used to differentiate between values counting blocks and values counting time in seconds. The type flag is set in the 23rd least significant bit (ie. value 1<<22). if the type-flag is set, then the nSequence value is interpreted as a multiple of 512 seconds. If the type-flag is not set, the nSequence value is interpreted as a number of blocks.
Segregated Witness is defined by the following Bitcoin Improvement Proposals (BIPs):
When interpreting nSequence as a relative timelock, only the 16 least significant bits are considered. Once the flags (bits 32 and 23) are evaluted, the nSequence value is usually "masked" with a 16-bit mask (eg. nSequence & 0x0000FFFF).
BIP141 :: The main definition of Segregated Witness. https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki
The following diagram shows the binary layout of the nSequence value, as defined by BIP-68:
BIP143 :: Transaction Signature Verification for Version 0 Witness Program
BIP-68 Relative lock-time using consensus-enforced sequence numbers:: https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki]
==== Check Sequence Verify (CSV)
==== Why Segregated Witness?
Just like CLTV and nLocktime, there is a script opcode for relative timelocks that leverages the nSequence value in scripts. That opcode is +CHECKSEQUENCEVERIFY+ commonly referred to as +CSV_ for short.
Segregated witness is an architectural change that has several effects on the scalability, security, economic incentives and performance of bitcoin.
The CSV opcode when evaluated in a UTXO's redeem script, allows spending only in a transaction whose input nSequence value is lower than the CSV parameter. Essentially, this restricts spending the UTXO until a certain number of blocks or seconds have elapsed relative to the time the UTXO was mined.
Transaction malleability :: By moving the witness outside the transaction, the transaction hash used as an identifier no longer includes the witness data. Since the witness data is the only part of the transaction that can be modified by a third party (see <<transaction malleability>> and <<segwit_txid>>), removing it also removes the opportunity for transaction malleability attacks. With segregated witness, transaction hashes become immutable by anyone other than the creator of the transaction, which greatly improves the implementation of many other protocols that rely on advanced bitcoin transaction construction, such as payment channels, chained transactions and lightning networks.
As with CLTV, the value in CSV must match the format in the corresponding nSequence value. If CSV is specified in terms of blocks, then so must nSequence. If CSV is specified in terms of seconds, then so must nSequence.
Script Versioning :: With the introduction of segregated witness scripts, every locking script is preceded by a _script version_ number, similar to how transactions and blocks have version numbers. The addition of a script version number allows the scripting language to be upgraded in a backwards compatible way (ie. using soft-fork upgrades), to introduce new script operands, syntax or semantics. The ability to upgrade the scripting language in a non-disruptive way will greatly accelerate the rate of innovation in bitcoin.
Relative timelocks with CSV are especially useful when several (chained) transactions are created and signed, but not submitted to the blockchain, but kept "off-chain". A child transaction cannot be used until the parent transaction has been propagated, mined, and aged by the time specified in the relative timelock. One application of this use case can be seen in <<state_channels>> and <<lightning_network>>.
Network and Storage Scaling :: The witness data is often a big contributor to the total size of a transaction. More complex scripts such as those used for multi-sig or payment channels are very large. In some cases these scripts account for the majority (more than 75%) of the data in a transaction. By moving the witness data outside the transaction, segregated witness improves bitcoin’s scalability. Nodes can prune the witness data after validating the signatures, or ignore it altogether when doing simplified payment verification. The witness data doesn’t need to be transmitted to all nodes and does not need to be stored on disk by all nodes.
CSV is defined in detail in BIP-112:
Signature Verification Optimization :: Segregated Witness upgrades the signature functions (OP_CHECKSIG, OP_CHECKMULTISIG etc), to reduce the algorithm's computational complexity. Before segwit, the algorithm used to produce a signature required a number of hash operations that was proportional to the size of the transaction. Data-hasing computations increased in O(n^2^) with respect to the number of signature operations, introducing a substantial computational burden on all nodes verifying the signature. With segwit, the algorithm is changed to reduce the complexity to O(n).
Offline Signing Improvement :: Segregated Witness signatures incorporate the value (amount) referenced by each input in the hash that is signed. Previously, an offline signing device, such as a hardware wallet, would have to verify the amount of each input before signing a transaction. This was usually accomplished by streaming a large amount of data about the previous transactions referenced as inputs. Since the amount is now part of the commitment hash that is signed, an offline device does not need the previous transactions. If the amounts do not match (are misrepresented by a compromised online system), the signature will be invalid.
==== How Segregated Witness Works
=== Median-Time-Past
At first glance, segregated witness appears to be a change to how transactions are constructed and therefore a transaction-level feature, but it is not. In fact, segregated witness is also a change to how individual UTXO are spent and therefore is a per-output feature.
As part of the activation of relative timelocks, there was also a change in the way "time" is calculated for timelocks (both absolute and relative).
A transaction can spend segregated witness outputs or traditional (inline-witness) outputs or both. Therefore, it does not make much sense to refer to a transaction as a “segregated witness transaction”. Rather we should refer to specific transaction inputs as “segregated witness inputs".
The timestamps set in block headers are set by the miners. There is a certain degree of latitude allowed by the consensus rules to account for differences in clock accuracy between decentralized nodes. However, this creates an unfortunate incentive for miners to lie about the time in a block so as to earn extra fees by including time-locked transactions that are not yet mature.
When a transaction spends a UTXO, it must provide a witness. In a traditional UTXO, the locking script requires that witness data be provided _inline_ in the input part of the transaction that spends the UTXO. A segregated witness UTXO, however, specifies a locking script that can be satisfied with witness data outside of the input (segregated).
To remove the incentive to lie and strengthen the security of time locks, a Bitcoin Improvement Proposal was proposed and activated at the same time as the BIPs for relative timelocks. This is BIP-113 which defines a new consensus measurement of time called _Median Time Past_.
==== Soft-fork (backwards compatibility)
Median-Time-Past replaces time in any calculations for timelocks as the median time of the last 11 blocks, instead of the timestamp in the current block.
Segregated witness is a significant change to the way outputs and transactions are architected. Such a change would normally require a simultaneous change in every bitcoin node and wallet, to change the consensus rules -- what is known as a hard fork. Instead, segregated witness is introduced with a much less disruptive change, which is backwards compatible, known as a soft fork. This type of upgrade allows non-upgraded software to ignore the changes and continue to operate without any disruption.
By incorporating 11 blocks, no single miner can influence the timestamps in such a way as to gain fees from transactions with a time lock that hasn't yet matured.
Segregated witness outputs are constructed so that older systems that are not segwit-aware can still validate them. To an old wallet or node, a segregated witness output looks like an output that _anyone can spend_. Such outputs can be spent with an empty signature, therefore the fact that there is no signature inside the transaction (it is segregated), does not invalidate the transaction. Newer wallets & mining nodes however see the segregated witness output and expect to find a valid witness for it in the transaction’s witness data.
Median-Time-Past changes the implementation of time calculations for nLocktime, CLTV, nSequence and CSV.
==== Segregated Witness Output and Transaction Examples
Median-Time-Past is specified in BIP-113:
Let’s look at some of our example transactions and see how they would change with segregated witness. We’ll first look at how a Pay-to-Public-Key-Hash (P2PKH) payment is transformed with segregated witness program. Then, we’ll look at the segregated witness equivalent for Pay-to-Script-Hash (P2SH) scripts. Finally, we’ll look at how both of the above segregated witness programs can be embedded inside a P2SH script.
BIP-113 Median time-past as endpoint for lock-time calculations:: https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki[https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki]
[[p2wpkh]]
===== Pay-to-Witness-Public-Key-Hash (P2WPKH)
=== Scripts with Flow Control (Conditional Clauses)
In <<cup_of_coffee>>, Alice created a transaction to pay Bob for a cup of coffee. That transaction created a Pay-to-Public-Key-Hash (P2PKH) output with a value of 0.015 BTC that was spendable by Bob. The output’s script looks like this:
One of the more powerful features of Bitcoin Script is flow control also known as conditional clauses. You are probably familiar with flow control in various programming languages that use the construct IF...THEN...ELSE. Bitcoin conditional clauses look a bit different, but are essentially the same construct.
At a basic level, bitcoin conditional opcodes allow us to construct a redeem script that has two ways of being unlocked, depending on a TRUE/FALSE outcome of evaluating a logical condition. For example, if x is TRUE, redeem script is A, ELSE redeem script is B.
With segregated witness, a Pay-to-Public-Key-Hash output, is created instead a Pay-to-Witness-Public-Key-Hash (P2WPKH), which looks like this:
Additionally, bitcoin conditional expressions can be "nested" indefinitely, meaning that a conditional clause can contain another within it, which contains another, etc. Bitcoin Script flow control can be used to construct very complex scripts with hundreds or even thousands of possible execution paths. There is no limit to nesting, but consensus rules impose a limit on the maximum size, in bytes, of a script.
.Example P2WPKH output script
----
0 ab68025513c3dbd2f7b92a94e0581f5d50f654e7
----
Bitcoin implements flow control using the IF, ELSE, ENDIF, and NOTIF opcodes. Additionally, conditional expressions can contain boolean operators such as BOOLAND, BOOLOR, and NOT.
As you can see, a segregated witness output’s locking script is much simpler than a traditional output. It consists of two values that are pushed on to the script evaluation stack. To an old (non-segwit-aware) bitcoin client, the two pushes would look like an output that anyone can spend and does not require a signature (or rather, can be spent with an empty signature). To a newer, segwit-aware client, the first number (0) is interpreted as a version number (the _witness version_) and the second part (20 bytes) is the equivalent of a locking script known as a _witness program_. The 20-byte witness program is simply the hash of the public key, as in a P2PKH script
At first glance, you may find the bitcoin's flow control scripts confusing. That is because Bitcoin Script is a stack language. The same way that +1 {plus} 1+ looks "backwards" when expressed as +1 1 ADD+, flow control clauses in bitcoin also look "backwards".
Now, let’s look at the corresponding transaction that Bob uses to spend this output. For the original script (non-segwit), Bob’s transaction would have to include a signature within the transaction input:
In most traditional (procedural) programming languages, flow control looks like this:
.Decoded transaction showing a P2PKH output being spent with a signature
.Pseudocode of flow control in most programming languages
However, to spend the segregated witness output, the transaction has no signature on that input. Instead, Bob’s transaction has an empty scriptSig and includes a segregated witness, outside the transaction itself:
In a stack-based language like Bitcoin Script, the logical condition comes before the +IF+, which makes it look "backwards", like this:
.Decoded transaction showing a P2WPKH output being spent with separate witness data
When reading Bitcoin Script, remember that the condition being evaluated comes *before* the +IF+ opcode.
It is extremely important to note that P2WPKH should only be created by the payee (recipient) and not converted by the sender from a known public key, P2PKH script or address. The sender has no way of knowing if the recipient's wallet has the ability to construct segwit transactions and spend P2WPKH outputs.
==== Flow Control with VERIFY opcodes
Additionally, P2WPKH outputs must be constructed from the hash of a _compressed_ public key. Uncompressed public keys are non-standard in segwit and may be explicitly disabled by a future soft fork. If the hash used in the P2WPKH came from an uncompressed public key, it may be unspendable and you may lose funds. P2WPKH outputs should be created by the payee's wallet by deriving a compressed public key from their private key.
Another form of flow control in Bitcoin Script is any opcode that ends in +VERIFY+. The +VERIFY+ suffix means that if the condition evaluated is not TRUE, execution of the script terminates immediately and the transaction is deemed invalid.
[WARNING]
====
P2WPKH should be constructed by the payee (recipient), by converting a compressed public key to a P2WPKH hash. You should never transform a P2PKH script, bitcoin address or uncompressed public key to a P2WPKH witness script.
====
Unlike an +IF+ clause which offers alternative execution paths, the +VERIFY+ suffix acts as a _guard clause_, continuing only if a precondition is met.
[[p2wsh]]
===== Pay-to-Witness-Script-Hash (P2WSH)
For example, the following script requires Bob's signature and a pre-image (secret) that produces a specific hash. Both conditions must be satisfied to unlock:
The second type of witness program corresponds to a Pay-to-Script-Hash (P2SH) script. We saw this type of script in <<p2sh>>. In that example, P2SH was used by Mohammed's company to express a multi-signature script. Payments to Mohammed's company were encoded with a locking script like this:
.Example P2SH output script
.A redeem script with an +EQUALVERIFY+ guard clause.
The P2SH script above references the hash of a _redeem script_ that defines a 2-of-3 multi-signature requirement to spend funds. To spend this output, Mohammed's company would present the redeem script (whose hash matches the script hash in the P2SH output) and the signatures necessary to satisfy that redeem script, all inside the transaction input:
To redeem this, Bob must construct an unlocking script that presents a valid pre-image and a signature:
.Decoded transaction showing a P2SH output being spent
.An unlocking script to satisfy the above redeem script
Now, let's look at how this entire example would be upgraded to segwit. If Mohammed's customers were using a segwit compatible wallet, they would make a payment, creating a Pay-to-Witness-Script-Hash (P2WSH) output that would look like this:
Without presenting the pre-image, Bob can't get to the part of the script that checks for his signature.
Again, as with the example of P2WPKH, you can see that the segregated witness equivalent script is a lot simpler and omits the various script operands that you see in P2SH scripts. Instead, the segregated witness program consists of two values pushed to the stack: a witness version (0) and the 32-byte SHA256 hash of the redeem script.
[TIP]
====
While P2SH uses the 20-byte +RIPEMD160(SHA256(script))+ hash, the P2WSH witness program uses a 32-byte +SHA256(script)+ hash. This difference in the selection of the hashing algorithm is deliberate and used to differentiate between the two types of witness programs (P2WPKH and P2WSH) by the length of the hash, and to provide stronger security to P2WSH (128bits vs. 80bits of P2SH).
====
Mohammed's company can spend outputs the Pay-to-Witness-Script-Hash output by presenting the correct redeem script and sufficient signatures to satisfy the redeem script. Both the redeem script and the signatures would be segregated _outside_ the spending transaction as part of the witness data. Within the transaction input, Mohammed's wallet would put an empty scriptSig:
Bob's unlocking script is identical:
.Decoded transaction showing a P2WSH output being spent with separate witness data
.An unlocking script to satisfy the above redeem script
In the previous two sections, we demonstrated two types of witness programs: <<p2wpkh>> and <<p2wsh>>. Both types of witness programs consist of single byte version number followed by a longer hash. They look very similar, but are interpreted very differently: one is interpreted as a public key hash, which is satisfied by a signature and the other as a script hash, which is satisfied by a redeem script. The critical difference between them is the length of the hash:
* The public key hash in P2WPKH is 20 bytes
* The script hash in P2WSH is 32 bytes
This is the one difference that allows a wallet to differentiate between the two types of witness programs. By looking at the length of the hash, a wallet can determine what type of witness program this is, P2WPKH or P2WSH.
The script with +IF+ does the same thing as using an opcode with a +VERIFY+ suffix, they both operate as guard clauses. However, the +VERIFY+ construction is more efficient, using one fewer opcode.
==== Upgrading to Segregated Witness
So, when do we use +VERIFY+ and when do we use +IF+? If all we are trying to do is to attach a pre-condition (guard clause), then +VERIFY+ is better. If however, we want to have more than one execution path, then we need an +IF...ELSE+ flow control clause.
As we can see from the examples above, upgrading to segregated witness is a two-step process. First, wallets must create special segwit type outputs. Then, these outputs can be spent by wallets that know how to construct segregated witness transactions. In the examples above, Alice's wallet was segwit-aware and able to create special outputs with segregated witness scripts. Bob's wallet is also segwit-aware and able to spend those outputs. What may not be obvious from the example is that in practice, Alice's wallet needs to _know_ that Bob uses a segwit-aware wallet and can spend these outputs. Otherwise, if Bob's wallet is not upgraded and Alice tries to make segwit payments to Bob, Bob's wallet will not be able to detect these payments.
[TIP]
====
For P2WPKH and P2WSH payment types, both the sender and the recipient wallets need to be upgraded to be able to use segwit. Furthermore, the sender's wallet needs to know that the recipient's wallet is segwit-aware.
====
==== Using Flow Control in Scripts
Segregated witness will not be implemented simultaneously across the entire network. Rather, segregated witness is implemented as a backwards compatible upgrade, where _old and new clients can coexist_. Wallet developers will independently upgrade wallet software to add segwit capabilities. The P2WPKH and P2WSH payment types are intended for when both sender and recipient are segwit-aware. The traditional P2PKH and P2SH will continue to work for non-upgraded wallets. That leaves two important scenarios which are addressed in the next section:
A very common use for flow control in Bitcoin Script, is to construct a redeem script that offers multiple execution paths, each a different way of redeeming the UTXO.
* Ability of a sender's wallet that is not segwit-aware to make a payment to a recipient's wallet that can process segwit transactions.
Let's look at a simple example, where we have two signers, Alice and Bob and either one is able to redeem. With multi-sig, this would be expressed as a 1-of-2 multisig script. For the sake of demonstration, we will do the same thing with an +IF+ clause:
* Ability of a sender's wallet that is segwit-aware to recognize and distinguish between recipients that are segwit-aware and ones that are not, by their _addresses_.
===== Embedding Segregated Witness Inside P2SH
Let's assume, for example, that Alice's wallet is not upgraded to segwit, but Bob's wallet is upgraded and can handle segwit transactions. Alice and Bob can use "old" non-segwit transactions. But Bob would likely want to use segwit to reduce transaction fees, taking advantage of the discount that applies to witness data.
In this case Bob's wallet can construct a P2SH address that contains a segwit script inside it. Alice's wallet sees this as a "normal" P2SH address and can make payments to it without any knowledge of segwit. Bob's wallet can then spend this payment with a segwit transaction, taking full advantage of segwit and reducing transaction fees.
Both forms of witness scripts, P2WPKH and P2WSH, can be embedded in a P2SH address. The first is noted as P2SH(P2WPKH) and the second is noted as P2SH(P2WSH).
The first form of witness script we will examine is P2SH(P2WPKH). This is a Pay-to-Witness-Public-Key-Hash witness program, embedded inside a Pay-to-Script-Hash script, so that it can be used by a wallet that is not aware of segwit.
----
IF
<Alice's Pubkey> CHECKSIG
ELSE
<Bob's Pubkey> CHECKSIG
ENDIF
----
Bob's wallet constructs a Pay-to-Witness-Public-Key-Hash (P2WPKH) witness program with Bob's public key. This witness program is then hashed and the resulting hash is encoded as a Pay-to-Script-Hash (P2SH) script. The P2SH script is converted to a bitcoin address, one which starts with a "3", as we saw in the <<p2sh>> section.
Looking at this redeem script, you may be wondering: "Where is the condition? There is nothing preceeding the +IF+ clause!"
Bob's wallet starts with the P2WPKH witness program we saw earlier:
The condition is not part of the redeem script. Instead, the condition will be offered in the unlocking script, allowing Alice and Bob to "choose" which execution path they want.
.Bob's P2WPKH witness program
Alice redeems this with the unlocking script:
----
0 ab68025513c3dbd2f7b92a94e0581f5d50f654e7
<Alice's Sig> 1
----
The P2WPKH witness program consists of the witness version and Bob's 20-byte public key hash.
The +1+ at the end serves as the condition (TRUE), that will make the +IF+ clause execute the first redemption path, for which Alice has a signature.
Bob's wallet then hashes the above witness program, first with SHA256, then with RIPEMD160, producing another 20-byte hash:
For Bob to redeem this, he would have to choose the second execution path, by giving a +FALSE+ value to the +IF+ clause:
.HASH160 of the P2WPKH witness program
----
660a5ab01c8468ac2e3cd58e9b17f8ba637867a4
<Bob's Sig> 0
----
The hash of the witness program is then embedded in a P2SH script:
.P2SH script containing the hash of a P2WPKH witness program
Bob's unlocking script puts a +0+ on the stack, causing the +IF+ clause to execute the second (+ELSE+) script, which requires Bob's signature.
Finally, the P2SH script is converted to a P2SH bitcoin address:
Since +IF+ clauses can be nested, we can create a "maze" of execution paths. The unlocking script can provide a "map" selecting which execution path is actually executed:
.P2SH address
----
3AzZFY4WJJZbVr2A6qBTbdkYRpMLbdg6gD
IF
script A
ELSE
IF
script B
ELSE
script C
ENDIF
ENDIF
----
Now, Bob can display this address for customers to pay for their coffee. Alice's wallet can make a payment to +3deadbeef+, just as it would to any other bitcoin address. Even though Alice's wallet has no support for segwit, the payment it creates can be spent by Bob with a segwit transaction.
In this scenario, there are three execution paths (+script A+, +script B+ and +script C+). The unlocking script provides a path in the form of a sequence of +TRUE+ or +FALSE+ values. To select path +script B+ for example, the unlocking script must end in +1 0+ (+TRUE+, +FALSE+). These values will be pushed onto the stack, so that the second value (+FALSE+) ends up at the top of the stack. The outer +IF+ clause pops the +FALSE+ value and executes the first +ELSE+ clause. Then the +TRUE+ value moves to the top of the stack and is evaluated by the inner (nested) +IF+, selecting the +B+ execution path.
Using this construct, we can build redeem scripts with tens or hundreds of execution paths, each offering a different way to redeem the UTXO. To spend, we construct an unlocking script that navigates the execution path by putting the appropriate +TRUE+ and +FALSE+ values on the stack at each flow control point.
Similarly, a P2WSH witness program for a multisig script or other complicated script can be embedded inside a Pay-to-Script-Hash script and address, making it possible for any wallet to make payments that are segwit compatible.
=== Complex Script Example
As we saw in <<p2wsh>>, Mohammed's company is using segregated witness payments to multi-signature scripts. To make it possible for any client to pay his company, regardless of whether their wallets are upgraded for segwit, Mohammed's wallet can embed the P2WSH witness program inside a P2SH script.
In this section we combine many of the concepts from this chapter into a single example.
First, Mohammed's wallet creates the P2WSH witness program that corresponds to the multi-signature script, hashed with SHA256:
Our example uses the story of Mohammed, the company owner in Dubai who is operating an import/export business.
.Mohammed's wallet creates a P2WSH witness program
In this example, Mohammed wishes to construct a company capital account with flexible rules. The scheme he creates requires different levels of authorization depending on timelocks. The participants in the multisig scheme are Mohammed, his two partners Saeed and Zaira and their company lawyer Abdul. The three partners make decisions based on a majority rule, so two of three must agree. However, in the case of a problem with their keys, they want their lawyer to be able to recover the funds with one of the three partner signatures. Finally, if all partners are unavailable or incapacitated for a while, they want the lawyer to be able to manage the account directly.
Then, the witness program itself is hashed with SHA256 and RIPEMD160, producing a new 20-byte hash, as used in traditional P2SH:
Here's the script that Mohammed designs to achieve this:
Mohammed's script implements three execution paths, using nested +IF...ELSE+ flow control clauses.
Finally, the wallet constructs a bitcoin address from this script:
In the first execution path, this script operates as a simple 2-of-3 multisig with the three partners. This execution path consists of lines 3 and 9. Line 3 sets the quorum of the multisig to +2+ (2-of-3). This execution path can be selected by putting +TRUE TRUE+ at the end of the unlocking script:
.P2SH bitcoin address
.Unlocking script for the first execution path (2-of-3 multisig)
----
3NTWTcFE88p26GTPoxcWef9Q5ncKt6CY2E
0 <Mohammed's Sig> <Zaira's Sig> TRUE TRUE
----
Now, Mohammed's clients can make payments to this address without any need to support segwit. Mohammed's company can then construct segwit transactions to spend these payments, taking advantage of segwit features including lower transaction fees.
===== Segregated Witness Addresses
After segwit is deployed on the bitcoin network, it will take some time until wallets are upgraded. It is quite likely therefore that segwit will mostly be used embedded in P2SH, as we saw in the previous section, at least for several months.
Eventually however, almost all wallets will be able to support segwit payments. At that time it will no longer be necessary to embed segwit in P2SH. It is therefore likely that a new form of bitcoin address will be created, one that indicates the recipient is segwit-aware and which directly encodes a witness program. There have been a number of proposals for a segregated witness address scheme, but none have been actively pursued at this time.
[[segwit_txid]]
===== Transaction Identifiers
One of the greatest benefits of Segregated Witness is that it eliminates third-party transaction malleability.
Before segwit, transactions could have their signatures subtly modified by third parties, changing their transaction ID (hash) without changing any fundamental properties (inputs, outputs, amounts). This created opportunities for Denial-of-Service attacks as well as attacks against poorly written wallet software that assumed unconfirmed transaction-hashes were immutable.
With the introduction of Segregated Witness, transactions have two identifiers, +txid+ and +wtxid+. The traditional transaction ID +txid+ is the double-SHA256 hash of the serialized transaction, without the witness data. A transaction +wtxid+ is the double-SHA256 hash of the new serialization format of the transaction with witness data.
The traditional +txid+ is calculated in exactly the same way as with a non-segwit transaction. However, since the segwit transaction has empty scriptSig's in every input, there is no part of the transaction that can be modified by a third party. Therefore, in a segwit transaction, the +txid+ is immutable by a third party, even when the transaction is unconfirmed.
The +wtxid+ is like an "extended" ID, in that the hash also incorporates the witness data. If a transaction is transmitted without witness data, then the +wtxid+ and +txid+ are identical. Note than since the +wtxid+ includes witness data (signatures) and since witness data may be malleable, the +wtxid+ should be considered malleable until the transaction is confirmed. Only the +txid+ of a segwit transaction can be considered immutable by third parties and only if _all_ the inputs of the transaction are segwit inputs.
[TIP]
====
Segregated Witness transactions have two IDs: +txid+ and +wtxid+. The +txid+ is the hash of the transaction without the witness data and the +wtxid+ is the hash inclusive of witness data. The +txid+ of a transaction where all inputs are segwit inputs, is not susceptible to third-party transaction malleability
The +0+ at the beginning of this unlocking script is because of a bug in CHECKMULTISIG which pops an extra value from the stack. The extra value is disregarded by the CHECKMULTISIG, but it must be present or the script fails. Pushing +0+ (customarily) is a workaround to the bug, as described in <<multisig_bug>>
====
==== Segregated Witness' New Signing Algorithm
Segregated Witness modifies the semantics of the four signature verification functions (OP_CHECKSIG, OP_CHECKSIGVERIFY, OP_CHECKMULTISIG and OP_CHECKMULTISIGVERIFY), changing the way a transaction commitment hash is calculated.
Signatures in bitcoin transactions are applied on a _commitment hash_ which is calculated from the transaction data, locking specific parts of the data indicating the signer's commitment to those values. For example, in a simple SIGHASH_ALL type signature, the commitment hash includes all inputs and outputs.
Unfortunately, the way the commitment hash was calculated introduced the possibility that a node verifying the signature can be forced to perform a significant number of hash computations. Specifically, the hash operations increase in O(n^2^) with respect to the number of signature operations in the transaction. An attacker could therefore create a transaction with a very large number of signature operations, causing the entire bitcoin network to have to perform hundreds or thousands of hash operations to verify the transaction.
Segwit represented an opportunity to address this problem by changing the way the commitment hash is calculated. For segwit version 0 witness programs, signature verification occurs using an improved commitment hash algorithm as specified in Bitcoin Improvement Proposal 143 (BIP143).
The new algorithm achieves two important goals. Firstly, the number of hash operations increases by a much more gradual O(n) to the number of signature operations, reducing the opportunity to create Denial-of-Service attacks with overly complex transactions. Secondly, the commitment hash now also includes the value (amounts) of each input as part of the commitment. This means that a signer can commit to a specific input value without needing to "fetch" and check the previous transaction referenced by the input. In the case of offline devices, such as hardware wallets, this greatly simplifies the communication between the host and the hardware wallet, removing the need to stream previous transactions for validation. A hardware wallet can accept the input value "as stated" by an untrusted host. Since the signature is invalid if that input value is not correct, the hardware wallet doesn't need to validate the value before signing the input.
==== Economic Incentives for Segregated Witness
Bitcoin mining nodes and full nodes incur costs for the resources used to support the bitcoin network and the blockchain. As the volume of bitcoin transactions increases, so does the cost of resources (CPU, network bandwidth, disk space, memory). Miners are compensated for these costs through fees that are proportional to the size (in bytes) of each transaction. Non-mining full nodes are not compensated, so they incur these costs because they have a need to run an authoritative fully-validating full-index node, perhaps because they use the node to operate a bitcoin business.
Without transaction fees, the growth in bitcoin data would arguably increase dramatically. Fees are intended to align the needs of bitcoin users with the burden their transactions impose on the network, through a market-based price discovery mechanism.
The calculation of fees based on transaction size treats all the data in the transaction as equal in cost. But from the perspective of full nodes and miners, some parts of a transaction carry much higher costs. Every transaction added to the bitcoin network affects the consumption of four resources on nodes:
The second execution path can only be used after 30 days have elapsed from the creation of the UTXO. At that time, it requires the signature of Abdul the lawyer and one of the three partners (a 1-of-3 multisig). This is achieved by line 7 which sets the quorum for the multisig to +1+. To select this execution path, the unlocking script would end in +FALSE TRUE+:
Disk Space :: Every transaction is stored in the blockchain, adding to the total size of the blockchain. The blockchain is stored on disk, but the storage can be optimized by “pruning” older transactions.
CPU :: Every transaction must be validated, which requires CPU time.
Bandwidth :: Every transaction is transmitted (through flood propagation) across the network at least once. Without any optimization in the block propagation protocol, transactions are transmitted again as part of a block, doubling the impact on network capacity
Memory :: Nodes that validate transactions keep the UTXO index or the entire UTXO set in memory to speed up validation. Because memory is at least one order of magnitude more expensive than disk, growth of the UTXO set contributes disproportionately to the cost of running a node.
As you can see from the list above, not every part of a transaction has an equal impact on the cost of running a node or on the ability of bitcoin to scale to support more transactions. The most expensive part of a transaction are the newly created outputs, as they are added to the in-memory UTXO set. By comparison, signatures (aka witness data) add the least burden to the network and the cost of running a node, because witness data are only validated once and then never used again. Furthermore, immediately after receiving a new transaction and validating witness data, nodes can discard that witness data. If fees are calculated on transaction size, without discriminating between these two types of data, then the market incentives of fees are not aligned with the actual costs imposed by a transaction. In fact, the current fee structure actually encourages the opposite behavior, because witness data is the largest part of a transaction.
The incentives created by fees matter because they affect the behavior of wallets. All wallets must implement some strategy for assembling transactions that takes into considerations a number of factors, such as privacy (reducing address re-use), fragmentation (making lots of loose change) and fees. If the fees are overwhelmingly motivating wallets to use as few inputs as possible in transactions, this can lead to UTXO picking and change address strategies that inadvertently bloat the UTXO set.
Transactions consume UTXO in their inputs and create new UTXO with their outputs. A transaction, therefore, that has more inputs than outputs will result in a decrease in the UTXO set, whereas a transaction that has more outputs than inputs will result in an increase in the UTXO set. Let’s consider the _difference_ between inputs and outputs and call that the “Net new UTXO”. That’s an important metric, as it tells us what impact a transaction will have on the most expensive network-wide resource, the in-memory UTXO set. A transaction with positive Net-new-UTXO, adds to that burden. A transaction with a negative Net-new-UTXO reduces the burden. We would therefore want to encourage transactions that are either negative Net-new-UTXO or neutral with zero Net-new-UTXO.
Let’s look at an example of what incentives are created by the transaction fee calculation, with and without segregated witness. We will look at two different transactions. Transaction A is a 3-input, 2-output transaction, which has a Net-new-UTXO metric of -1, meaning it consumes one more UTXO than it creates, reducing the UTXO set by one. Transaction B is a 2-input, 3-output transaction, which has a Net-new-UTXO metric of 1, meaning it adds one UTXO to the UTXO set, imposing additional cost on the entire bitcoin network. Both transactions use multi-signature (2-of-3) scripts, to demonstrate how complex scripts increase the impact of segregated witness on fees. Let’s assume a transaction fee of 30 satoshi per byte and a 75% fee discount on witness data:
Without Segregated Witness
Transaction A fee: 25,710 satoshi
Transaction B fee: 18,990 satoshi
.Unlocking script for the second execution path (Lawyer + 1-of-3)
----
0 <Saeed's Sig> <Abdul's Sig> FALSE TRUE
----
With Segregated Witness
Transaction A fee: 8,130 satoshi
Transaction B fee: 12,045 satoshi
[TIP]
====
Why +FALSE TRUE+? Isn't that backwards? Because the two values are pushed on to the stack, with +FALSE+ pushed first, then +TRUE+ pushed second. +TRUE+ is therefore popped *first* by the first +IF+ opcode.
====
Finally, the third execution path allows Abdul the lawyer to spend the funds alone, but only after 90 days. To select this execution path, the unlocking script has to end in +FALSE+:
Both transactions are less expensive when segregated witness is implemented. But comparing the costs between the two transactions, we see that before segregated witness, the fee is higher for the transaction that has a negative Net-new-UTXO. After segregated witness, the transaction fees align with the incentive to minimize new UTXO creation, by not inadvertently penalizing transactions with many inputs.
.Unlocking script for the third execution path (Lawyer only)
----
<Abdul's Sig> FALSE
----
Segregated witness therefore has two main effects on the fees paid by bitcoin users. Firstly, segwit reduces the overall cost of transactions by discounting witness data and increasing the capacity of the bitcoin blockchain. Secondly, segwit’s discount on witness data correcting a misalignment of incentives that may have inadvertently created more bloat in the UTXO set.
Try running the script on paper to see how it behaves on the stack.
=== Proposed Future Scripting and Transaction Improvements
A few more things to consider when reading this example. See if you can find the answers:
==== Confidential Transactions
* Why can't the lawyer redeem the third execution path at any time, by selecting it with +FALSE+ on the unlocking script?
==== Schnorr Signatures
* How many execution paths can be used 5, 35 and 105 days respectively after the UTXO is mined?
==== Merkleized Abstract Syntax Trees (Pay-to-Merkle-Root)
* Are the funds lost if the lawyer loses their key? Does your answer change if 91 days have elapsed?
==== Covenants
* How do the partners "reset" the clock every 29 or 89 days to prevent the lawyer from accessing the funds?
Andreas M. Antonopoulos
7 years ago
