diff --git a/.gitattributes b/.gitattributes deleted file mode 100644 index 09e2d14d..00000000 --- a/.gitattributes +++ /dev/null @@ -1 +0,0 @@ -*.asciidoc linguist-detectable diff --git a/.gitignore b/.gitignore index 440ef0e6..a4f6a28a 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,6 @@ code/python-env .debris _build/ dist/ +_build/ +build/ +book.html diff --git a/LICENSE b/LICENSE index c2c0d841..033410a0 100644 --- a/LICENSE +++ b/LICENSE @@ -13,4 +13,4 @@ Different editions of this book are covered by different licenses. Contributions These "Free Culture" licenses were approved by the publisher O'Reilly Media (http://oreilly.com). O'Reilly Media is not just the world's best publisher of technical books but also a strong supporter of an open culture and the sharing of knowledge. -Thank you O'Reilly! \ No newline at end of file +Thank you O'Reilly! diff --git a/README.md b/README.md index cad607e8..294bd6f9 100644 --- a/README.md +++ b/README.md @@ -14,20 +14,11 @@ If you know how to make a pull request to contribute a fix, please write the cor To read this book, see [book.asciidoc](https://github.com/bitcoinbook/bitcoinbook/blob/develop/book.asciidoc). Click on each of the chapters to read in your browser. Other parties may choose to release PDFs of the book online, for those editions where a PDF derivative is licensed. -## Chapters - -+ Chapter 1: '[Introduction](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch01.asciidoc)' -+ Chapter 2: '[How Bitcoin Works](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch02.asciidoc)' -+ Chapter 3: '[Bitcoin Core: The Reference Implementation](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch03.asciidoc)' -+ Chapter 4: '[Keys, Addresses](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch04.asciidoc)' -+ Chapter 5: '[Wallets](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch05.asciidoc)' -+ Chapter 6: '[Transactions](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch06.asciidoc)' -+ Chapter 7: '[Advanced Transactions and Scripting](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch07.asciidoc)' -+ Chapter 8: '[The Bitcoin Network](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch08.asciidoc)' -+ Chapter 9: '[The Blockchain](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch09.asciidoc)' -+ Chapter 10: '[Mining and Consensus](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch10.asciidoc)' -+ Chapter 11: '[Bitcoin Security](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch11.asciidoc)' -+ Chapter 12: '[Blockchain Applications](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch12.asciidoc)' +The 2nd edition of "Mastering Bitcoin" is available under a CC-BY-NC-ND license, not a CC-BY-SA license. + +It is deliberately not available as a PDF. Why? Because a PDF is a "derivative" product, which is what the ND prohibits. That's because the publisher (O'Reilly Media) is a for-profit publisher who puts considerable resources behind distributing the book. The book will eventually (within a year of publication) be released under a CC-BY-SA license, at which point the PDF format and translations will be allowed. Until then, making PDF copies violates the license and interferes with the publisher's (and the author's) ability to make a living. Furthermore, if you make it so the publisher can't recoup their investment, they will delay the release into CC-BY-SA. + +Please don't create or distribute PDFs until the license is changed to CC-BY-SA. It is rare for a publisher to even agree to a CC-BY-NC-ND license. Don't make it harder for free culture by violating even that, already generous, license. # Published @@ -47,9 +38,9 @@ The book's source code, found in this repository, is kept synchronized with the The tags [Edition1Print1](https://github.com/bitcoinbook/bitcoinbook/releases/tag/Edition1Print1), [Edition1Print2](https://github.com/bitcoinbook/bitcoinbook/releases/tag/Edition1Print2) correspond to the two existing prints of Mastering Bitcoin (First Edition) as published by O'Reilly Media. -Creative Commons License
Mastering Bitcoin - First Edition by Andreas M. Antonopoulos LLC is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. +Creative Commons License
Mastering Bitcoin - First Edition by Andreas M. Antonopoulos LLC is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. -This "Free Culture" compliant license was approved by my publisher O'Reilly Media (http://oreilly.com), who understands the value of open source. O'Reilly Media is not just the world's best publisher of technical books, but is also a strong supporter of this open culture and the sharing of knowledge. +This "Free Culture" compliant license was approved by my publisher O'Reilly Media (https://oreilly.com), who understands the value of open source. O'Reilly Media is not just the world's best publisher of technical books, but is also a strong supporter of this open culture and the sharing of knowledge. Thank you O'Reilly Media! @@ -57,8 +48,10 @@ Thank you O'Reilly Media! The tags, [second_edition_print_1](https://github.com/bitcoinbook/bitcoinbook/releases/tag/second_edition_print_1) [second_edition_print2](https://github.com/bitcoinbook/bitcoinbook/releases/tag/second_edition_print2), [second_edition_print3](https://github.com/bitcoinbook/bitcoinbook/releases/tag/second_edition_print3), correspond to the first (June 8th, 2017), second (July 20th, 2017) and third (March 23rd, 2018) print of Mastering Bitcoin (Second Edition), as published by O'Reilly Media. -Creative Commons License
Mastering Bitcoin - Second Edition by Andreas M. Antonopoulos LLC is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. +Creative Commons License
Mastering Bitcoin - Second Edition by Andreas M. Antonopoulos LLC is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. + +It is expected that the second edition will be released under a CC-BY-SA license within a year of publication. # Translations -If you are interested in translating this book, please join our team of volunteers at: https://www.transifex.com/aantonop/mastering-bitcoin +If you are interested in translating this book, please join our team of volunteers at: https://www.transifex.com/bitcoinbook/mastering-bitcoin/ diff --git a/appdx-bitcoinwhitepaper.asciidoc b/appa_whitepaper.adoc similarity index 51% rename from appdx-bitcoinwhitepaper.asciidoc rename to appa_whitepaper.adoc index 3c9e1938..6ec5d506 100644 --- a/appdx-bitcoinwhitepaper.asciidoc +++ b/appa_whitepaper.adoc @@ -1,10 +1,11 @@ [[satoshi_whitepaper]] [appendix] -== The Bitcoin Whitepaper by Satoshi Nakamoto +== The Bitcoin Whitepaper [.keep-together]#by Satoshi Nakamoto# + [NOTE] ==== -((("whitepaper", id="whitethirteen")))((("bitcoin whitepaper", id="BCwhitethirteen")))((("Nakamoto, Satoshi", id="nakamatothirteen")))This is the original whitepaper, reproduced in its entirety exactly as it was published by Satoshi Nakamoto in October 2008. +This is the((("Bitcoin whitepaper", "original version", id="bitcoin-whitepaper-original")))((("whitepaper (Bitcoin)", "original version", id="whitepaper-original")))((("Nakamoto, Satoshi"))) original whitepaper, reproduced in its entirety exactly as it was published by Satoshi Nakamoto in October 2008. ==== === Bitcoin - A Peer-to-Peer Electronic Cash System @@ -23,23 +24,23 @@ Commerce on the Internet has come to rely almost exclusively on financial instit What is needed is an electronic payment system based on cryptographic proof instead of trust, allowing any two willing parties to transact directly with each other without the need for a trusted third party. Transactions that are computationally impractical to reverse would protect sellers from fraud, and routine escrow mechanisms could easily be implemented to protect buyers. In this paper, we propose a solution to the double-spending problem using a peer-to-peer distributed timestamp server to generate computational proof of the chronological order of transactions. The system is secure as long as honest nodes collectively control more CPU power than any cooperating group of attacker nodes. ==== Transactions -We define an electronic coin as a chain of digital signatures. Each owner transfers the coin to the next by digitally signing a hash of the previous transaction and the public key of the next owner and adding these to the end of the coin. A payee can verify the signatures to verify the chain of ownership. +We ((("transactions", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper", id="transaction-whitepaper")))define an electronic coin as a chain of digital signatures. Each owner transfers the coin to the next by digitally signing a hash of the previous transaction and the public key of the next owner and adding these to the end of the coin. A payee can verify the signatures to verify the chain of ownership. -image::images/mbc2_abin01.png["Transactions"] +image::images/mbc3_0401.png["Transactions"] The problem of course is the payee can't verify that one of the owners did not double-spend the coin. A common solution is to introduce a trusted central authority, or mint, that checks every transaction for double spending. After each transaction, the coin must be returned to the mint to issue a new coin, and only coins issued directly from the mint are trusted not to be double-spent. The problem with this solution is that the fate of the entire money system depends on the company running the mint, with every transaction having to go through them, just like a bank. -We need a way for the payee to know that the previous owners did not sign any earlier transactions. For our purposes, the earliest transaction is the one that counts, so we don't care about later attempts to double-spend. The only way to confirm the absence of a transaction is to be aware of all transactions. In the mint based model, the mint was aware of all transactions and decided which arrived first. To accomplish this without a trusted party, transactions must be publicly announced [1], and we need a system for participants to agree on a single history of the order in which they were received. The payee needs proof that at the time of each transaction, the majority of nodes agreed it was the first received. +We need a way for the payee to know that the previous owners did not sign any earlier transactions. For our purposes, the earliest transaction is the one that counts, so we don't care about later attempts to double-spend. The only way to confirm the absence of a transaction is to be aware of all transactions. In the mint based model, the mint was aware of all transactions and decided which arrived first. To accomplish this without a trusted party, transactions must be publicly announced [1], and we need a system for participants to agree on a single history of the order in which they were received. The payee needs proof that at the time of each transaction, the majority of nodes agreed it was the ((("transactions", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper", startref="transaction-whitepaper")))first received. ==== Timestamp Server -The solution we propose begins with a timestamp server. A timestamp server works by taking a hash of a block of items to be timestamped and widely publishing the hash, such as in a newspaper or Usenet post [2-5]. The timestamp proves that the data must have existed at the time, obviously, in order to get into the hash. Each timestamp includes the previous timestamp in its hash, forming a chain, with each additional timestamp reinforcing the ones before it. +The ((("timestamp servers")))solution we propose begins with a timestamp server. A timestamp server works by taking a hash of a block of items to be timestamped and widely publishing the hash, such as in a newspaper or Usenet post [2-5]. The timestamp proves that the data must have existed at the time, obviously, in order to get into the hash. Each timestamp includes the previous timestamp in its hash, forming a chain, with each additional timestamp reinforcing the ones before it. -image::images/mbc2_abin02.png["timestamp server"] +image::images/mbc3_aain02.png["timestamp server"] ==== Proof-of-Work -To implement a distributed timestamp server on a peer-to-peer basis, we will need to use a proof-of-work system similar to Adam Back's Hashcash [6], rather than newspaper or Usenet posts. The proof-of-work involves scanning for a value that when hashed, such as with SHA-256, the hash begins with a number of zero bits. The average work required is exponential in the number of zero bits required and can be verified by executing a single hash. For our timestamp network, we implement the proof-of-work by incrementing a nonce in the block until a value is found that gives the block's hash the required zero bits. Once the CPU effort has been expended to make it satisfy the proof-of-work, the block cannot be changed without redoing the work. As later blocks are chained after it, the work to change the block would include redoing all the blocks after it. +To ((("proof-of-work algorithm", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper")))implement a distributed timestamp server on a peer-to-peer basis, we will need to use a proof-of-work system similar to Adam Back's Hashcash [6], rather than newspaper or Usenet posts. The proof-of-work involves scanning for a value that when hashed, such as with SHA-256, the hash begins with a number of zero bits. The average work required is exponential in the number of zero bits required and can be verified by executing a single hash. For our timestamp network, we implement the proof-of-work by incrementing a nonce in the block until a value is found that gives the block's hash the required zero bits. Once the CPU effort has been expended to make it satisfy the proof-of-work, the block cannot be changed without redoing the work. As later blocks are chained after it, the work to change the block would include redoing all the blocks after it. -image::images/mbc2_abin03.png["pow"] +image::images/mbc3_aain03.png["pow"] The proof-of-work also solves the problem of determining representation in majority decision making. If the majority were based on one-IP-address-one-vote, it could be subverted by anyone able to allocate many IPs. Proof-of-work is essentially one-CPU-one-vote. The majority decision is represented by the longest chain, which has the greatest proof-of-work effort invested in it. If a majority of CPU power is controlled by honest nodes, the honest chain will grow the fastest and outpace any competing chains. To modify a past block, an attacker would have to redo the proof-of-work of the block and all blocks after it and then catch up with and surpass the work of the honest nodes. We will show later that the probability of a slower attacker catching up diminishes exponentially as subsequent blocks are added. @@ -47,7 +48,7 @@ To compensate for increasing hardware speed and varying interest in running node ==== Network -The steps to run the network are as follows: +The steps((("nodes", "in Bitcoin", secondary-sortas="Bitcoin whitepaper")))((("Bitcoin network", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper"))) to run the network are as follows: 1. New transactions are broadcast to all nodes. 2. Each node collects new transactions into a block. @@ -61,50 +62,50 @@ Nodes always consider the longest chain to be the correct one and will keep work New transaction broadcasts do not necessarily need to reach all nodes. As long as they reach many nodes, they will get into a block before long. Block broadcasts are also tolerant of dropped messages. If a node does not receive a block, it will request it when it receives the next block and realizes it missed one. ==== Incentive -By convention, the first transaction in a block is a special transaction that starts a new coin owned by the creator of the block. This adds an incentive for nodes to support the network, and provides a way to initially distribute coins into circulation, since there is no central authority to issue them. The steady addition of a constant of amount of new coins is analogous to gold miners expending resources to add gold to circulation. In our case, it is CPU time and electricity that is expended. +By((("incentives", id="incentive-whitepaper"))) convention, the first transaction in a block is a special transaction that starts a new coin owned by the creator of the block. This adds an incentive for nodes to support the network, and provides a way to initially distribute coins into circulation, since there is no central authority to issue them. The steady addition of a constant of amount of new coins is analogous to gold miners expending resources to add gold to circulation. In our case, it is CPU time and electricity that is expended. The incentive can also be funded with transaction fees. If the output value of a transaction is less than its input value, the difference is a transaction fee that is added to the incentive value of the block containing the transaction. Once a predetermined number of coins have entered circulation, the incentive can transition entirely to transaction fees and be completely inflation free. -The incentive may help encourage nodes to stay honest. If a greedy attacker is able to assemble more CPU power than all the honest nodes, he would have to choose between using it to defraud people by stealing back his payments, or using it to generate new coins. He ought to find it more profitable to play by the rules, such rules that favour him with more new coins than everyone else combined, than to undermine the system and the validity of his own wealth. +The incentive may help encourage nodes to stay honest. If a greedy attacker is able to assemble more CPU power than all the honest nodes, he would have to choose between using it to defraud people by stealing back his payments, or using it to generate new coins. He ought to find it more profitable to play by the rules, such rules that favour him with more new coins than everyone else combined, than to undermine the system and the validity of his own((("incentives", startref="incentive-whitepaper"))) wealth. ==== Reclaiming Disk Space ++++ -

Once the latest transaction in a coin is buried under enough blocks, the spent transactions before it can be discarded to save disk space. To facilitate this without breaking the block's hash, transactions are hashed in a Merkle Tree [7] [2] [5], with only the root included in the block's hash. Old blocks can then be compacted by stubbing off branches of the tree. The interior hashes do not need to be stored.

+

Once ((("disk space", "reclaiming")))((("reclaiming", "disk space")))((("blocks", "reclaiming disk space")))the latest transaction in a coin is buried under enough blocks, the spent transactions before it can be discarded to save disk space. To facilitate this without breaking the block's hash, transactions are hashed in a Merkle Tree [7] [2] [5], with only the root included in the block's hash. Old blocks can then be compacted by stubbing off branches of the tree. The interior hashes do not need to be stored.

++++ -image::images/mbc2_abin04.png["disk"] +image::images/mbc3_aain04.png["disk"] A block header with no transactions would be about 80 bytes. If we suppose blocks are generated every 10 minutes, +80 bytes * 6 * 24 * 365 = 4.2MB+ per year. With computer systems typically selling with 2GB of RAM as of 2008, and Moore's Law predicting current growth of 1.2GB per year, storage should not be a problem even if the block headers must be kept in memory. ==== Simplified Payment Verification -It is possible to verify payments without running a full network node. A user only needs to keep a copy of the block headers of the longest proof-of-work chain, which he can get by querying network nodes until he's convinced he has the longest chain, and obtain the Merkle branch linking the transaction to the block it's timestamped in. He can't check the transaction for himself, but by linking it to a place in the chain, he can see that a network node has accepted it, and blocks added after it further confirm the network has accepted it. +It is ((("payment verification", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper", id="payment-verify")))((("verifying", "payment", "in Bitcoin whitepaper", tertiary-sortas="Bitcoin whitepaper", id="verify-payment")))possible to verify payments without running a full network node. A user only needs to keep a copy of the block headers of the longest proof-of-work chain, which he can get by querying network nodes until he's convinced he has the longest chain, and obtain the Merkle branch linking the transaction to the block it's timestamped in. He can't check the transaction for himself, but by linking it to a place in the chain, he can see that a network node has accepted it, and blocks added after it further confirm the network has accepted it. -image::images/mbc2_abin05.png["spv"] +image::images/mbc3_aain05.png["spv"] -As such, the verification is reliable as long as honest nodes control the network, but is more vulnerable if the network is overpowered by an attacker. While network nodes can verify transactions for themselves, the simplified method can be fooled by an attacker's fabricated transactions for as long as the attacker can continue to overpower the network. One strategy to protect against this would be to accept alerts from network nodes when they detect an invalid block, prompting the user's software to download the full block and alerted transactions to confirm the inconsistency. Businesses that receive frequent payments will probably still want to run their own nodes for more independent security and quicker verification. +As such, the verification is reliable as long as honest nodes control the network, but is more vulnerable if the network is overpowered by an attacker. While network nodes can verify transactions for themselves, the simplified method can be fooled by an attacker's fabricated transactions for as long as the attacker can continue to overpower the network. One strategy to protect against this would be to accept alerts from network nodes when they detect an invalid block, prompting the user's software to download the full block and alerted transactions to confirm the inconsistency. Businesses that receive frequent payments will probably still want to run their own nodes for more independent security and quicker ((("payment verification", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper", startref="payment-verify")))((("verifying", "payment", "in Bitcoin whitepaper", tertiary-sortas="Bitcoin whitepaper", startref="verify-payment")))verification. ==== Combining and Splitting Value -Although it would be possible to handle coins individually, it would be unwieldy to make a separate transaction for every cent in a transfer. To allow value to be split and combined, transactions contain multiple inputs and outputs. Normally there will be either a single input from a larger previous transaction or multiple inputs combining smaller amounts, and at most two outputs: one for the payment, and one returning the change, if any, back to the sender. +Although ((("transactions", "inputs", "in Bitcoin whitepaper", tertiary-sortas="Bitcoin whitepaper")))((("transactions", "outputs", "in Bitcoin whitepaper", tertiary-sortas="Bitcoin whitepaper")))((("inputs", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper")))((("outputs", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper")))it would be possible to handle coins individually, it would be unwieldy to make a separate transaction for every cent in a transfer. To allow value to be split and combined, transactions contain multiple inputs and outputs. Normally there will be either a single input from a larger previous transaction or multiple inputs combining smaller amounts, and at most two outputs: one for the payment, and one returning the change, if any, back to the sender. -image::images/mbc2_abin06.png["combining-splitting"] +image::images/mbc3_aain06.png["combining-splitting"] It should be noted that fan-out, where a transaction depends on several transactions, and those transactions depend on many more, is not a problem here. There is never the need to extract a complete standalone copy of a transaction's history. ==== Privacy -The traditional banking model achieves a level of privacy by limiting access to information to the parties involved and the trusted third party. The necessity to announce all transactions publicly precludes this method, but privacy can still be maintained by breaking the flow of information in another place: by keeping public keys anonymous. The public can see that someone is sending an amount to someone else, but without information linking the transaction to anyone. This is similar to the level of information released by stock exchanges, where the time and size of individual trades, the "tape", is made public, but without telling who the parties were. +The traditional((("privacy", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper"))) banking model achieves a level of privacy by limiting access to information to the parties involved and the trusted third party. The necessity to announce all transactions publicly precludes this method, but privacy can still be maintained by breaking the flow of information in another place: by keeping public keys anonymous. The public can see that someone is sending an amount to someone else, but without information linking the transaction to anyone. This is similar to the level of information released by stock exchanges, where the time and size of individual trades, the "tape", is made public, but without telling who the parties were. -image::images/mbc2_abin07.png["privacy"] +image::images/mbc3_aain07.png["privacy"] As an additional firewall, a new key pair should be used for each transaction to keep them from being linked to a common owner. Some linking is still unavoidable with multi-input transactions, which necessarily reveal that their inputs were owned by the same owner. The risk is that if the owner of a key is revealed, linking could reveal other transactions that belonged to the same owner. ==== Calculations -We consider the scenario of an attacker trying to generate an alternate chain faster than the honest chain. Even if this is accomplished, it does not throw the system open to arbitrary changes, such as creating value out of thin air or taking money that never belonged to the attacker. Nodes are not going to accept an invalid transaction as payment, and honest nodes will never accept a block containing them. An attacker can only try to change one of his own transactions to take back money he recently spent. +We consider ((("calculations", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper", id="calculate-whitepaper")))the scenario of an attacker trying to generate an alternate chain faster than the honest chain. Even if this is accomplished, it does not throw the system open to arbitrary changes, such as creating value out of thin air or taking money that never belonged to the attacker. Nodes are not going to accept an invalid transaction as payment, and honest nodes will never accept a block containing them. An attacker can only try to change one of his own transactions to take back money he recently spent. -The race between the honest chain and an attacker chain can be characterized as a Binomial Random Walk. The success event is the honest chain being extended by one block, increasing its lead by +1, and the failure event is the attacker's chain being extended by one block, reducing the gap by -1. +The race between the honest chain and an attacker chain can be characterized as a ((("Binomial Random Walk")))Binomial Random Walk. The success event is the honest chain being extended by one block, increasing its lead by +1, and the failure event is the attacker's chain being extended by one block, reducing the gap by -1. ++++ -

The probability of an attacker catching up from a given deficit is analogous to a Gambler's Ruin problem. Suppose a gambler with unlimited credit starts at a deficit and plays potentially an infinite number of trials to try to reach breakeven. We can calculate the probability he ever reaches breakeven, or that an attacker ever catches up with the honest chain, as follows [8]:

+

The probability of an attacker catching up from a given deficit is analogous to a ((("Gambler's Ruin problem")))Gambler's Ruin problem. Suppose a gambler with unlimited credit starts at a deficit and plays potentially an infinite number of trials to try to reach breakeven. We can calculate the probability he ever reaches breakeven, or that an attacker ever catches up with the honest chain, as follows [8]:

++++ p = probability an honest node finds the next block @@ -113,7 +114,63 @@ q = probability the attacker finds the next block q~z~ = probability the attacker will ever catch up from z blocks behind -image::images/mbc2_abin08.png["eq1"] + +++++ +
+ + + + q + z + + = + + { + + + + 1 + + + + if + + p + ≤ + q + + + + + + + ( + q + ⁄ + p + + ) + z + + + + + + if + + p + > + q + + + + + } + + + +
+++++ Given our assumption that p > q, the probability drops exponentially as the number of blocks the attacker has to catch up with increases. With the odds against him, if he doesn't make a lucky lunge forward early on, his chances become vanishingly small as he falls further behind. @@ -123,15 +180,175 @@ The receiver generates a new key pair and gives the public key to the sender sho The recipient waits until the transaction has been added to a block and z blocks have been linked after it. He doesn't know the exact amount of progress the attacker has made, but assuming the honest blocks took the average expected time per block, the attacker's potential progress will be a Poisson distribution with expected value: -image::images/mbc2_abin09.png["eq2"] +++++ +
+ + + λ + = + z + + q + p + + + +
+++++ To get the probability the attacker could still catch up now, we multiply the Poisson density for each amount of progress he could have made by the probability he could catch up from that point: -image::images/mbc2_abin10.png["eq3"] +++++ +
+ + + + + ∑ + + k + = + 0 + + ∞ + + + + + + λ + k + + + e + + − + λ + + + + + k + ! + + + ⋅ + + { + + + + + ( + q + ⁄ + p + + ) + + ( + z + − + k + ) + + + + + + + if + + k + ≤ + z + + + + + + 1 + + + + if + + k + > + z + + + + + } + + + +
+++++ Rearranging to avoid summing the infinite tail of the distribution... -image::images/mbc2_abin11.png["eq4"] +++++ +
+ + + 1 + − + + + ∑ + + k + = + 0 + + z + + + + + + λ + k + + + e + + − + λ + + + + + k + ! + + + + ( + 1 + − + ( + q + ⁄ + p + + ) + + ( + z + − + k + ) + + + ) + + + +
+++++ Converting to C code... @@ -198,7 +415,7 @@ q=0.45 z=340 ---- ==== Conclusion -We have proposed a system for electronic transactions without relying on trust. We started with the usual framework of coins made from digital signatures, which provides strong control of ownership, but is incomplete without a way to prevent double-spending. To solve this, we proposed a peer-to-peer network using proof-of-work to record a public history of transactions that quickly becomes computationally impractical for an attacker to change if honest nodes control a majority of CPU power. The network is robust in its unstructured simplicity. Nodes work all at once with little coordination. They do not need to be identified, since messages are not routed to any particular place and only need to be delivered on a best effort basis. Nodes can leave and rejoin the network at will, accepting the proof-of-work chain as proof of what happened while they were gone. They vote with their CPU power, expressing their acceptance of valid blocks by working on extending them and rejecting invalid blocks by refusing to work on them. Any needed rules and incentives can be enforced with this consensus mechanism. +We ((("calculations", "in Bitcoin whitepaper", secondary-sortas="Bitcoin whitepaper", startref="calculate-whitepaper")))have proposed a system for electronic transactions without relying on trust. We started with the usual framework of coins made from digital signatures, which provides strong control of ownership, but is incomplete without a way to prevent double-spending. To solve this, we proposed a peer-to-peer network using proof-of-work to record a public history of transactions that quickly becomes computationally impractical for an attacker to change if honest nodes control a majority of CPU power. The network is robust in its unstructured simplicity. Nodes work all at once with little coordination. They do not need to be identified, since messages are not routed to any particular place and only need to be delivered on a best effort basis. Nodes can leave and rejoin the network at will, accepting the proof-of-work chain as proof of what happened while they were gone. They vote with their CPU power, expressing their acceptance of valid blocks by working on extending them and rejecting invalid blocks by refusing to work on them. Any needed rules and incentives can be enforced with this consensus mechanism. ==== References ++++ @@ -235,7 +452,7 @@ We have proposed a system for electronic transactions without relying on trust. === License -((("open source licenses")))This whitepaper was published in October 2008 by Satoshi Nakamoto. It was later (2009) added as supporting documentation to the bitcoin software, which carries the same MIT license. It has been reproduced in this book, without modification other than formatting, under the terms of the MIT license: +This whitepaper was published in October 2008 by Satoshi Nakamoto. It was later (2009) added as supporting documentation to the bitcoin software and carries the same MIT license. It has been reproduced in this book, without modification other than formatting, under the terms of the MIT license: The MIT License (MIT) Copyright (c) 2008 Satoshi Nakamoto @@ -244,4 +461,4 @@ Permission is hereby granted, free of charge, to any person obtaining a copy of The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. -THE SOFTWARE IS PROVIDED "AS IS," WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.((("", startref="whitethirteen")))((("", startref="nakamatothirteen")))((("", startref="BCwhitethirteen"))) +THE SOFTWARE IS PROVIDED "AS IS," WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN((("Bitcoin whitepaper", "original version", startref="bitcoin-whitepaper-original")))((("whitepaper (Bitcoin)", "original version", startref="whitepaper-original"))) THE SOFTWARE. diff --git a/appb_errata.adoc b/appb_errata.adoc new file mode 100644 index 00000000..a02d7c67 --- /dev/null +++ b/appb_errata.adoc @@ -0,0 +1,245 @@ +[appendix] +== Errata to the Bitcoin Whitepaper + +This ((("Bitcoin whitepaper", "errata", id="bitcoin-whitepaper-errata")))((("whitepaper (Bitcoin)", "errata", id="whitepaper-errata")))appendix contains a description of known problems in Satoshi Nakamoto’s paper, "Bitcoin: +A Peer-to-Peer Electronic Cash System," as well as notes on terminology +changes and how Bitcoin's implementation differs from that described in +the paper. + +This document was originally published by a coauthor of this book in +2016; it is reproduced here with updates. The names of +sections in this errata correspond to the names of the +sections in Nakamoto's original paper. + +=== Abstract + +____ +"The longest chain not only serves as proof of the sequence of events +witnessed, but proof that it came from the largest pool of CPU power." +____ + +* *Implementation detail:* If each link in the chain (called "blocks" +in Bitcoin) was built using the same amount of _proof of work_ (PoW), the +longest chain would be the one backed by the largest pool of +computational power. However, Bitcoin was implemented in such a way that +the amount of PoW can vary between blocks, so it became important not to +check for the "the longest chain" but rather "the chain demonstrating +the most PoW"; this is often shortened to "most-work chain." ++ +The +https://oreil.ly/XYZzx[change] +from checking for the longest chain to checking for the most-work chain +occurred in July 2010, long after Bitcoin’s initial release: ++ +[source,diff] +---- +- if (pindexNew->nHeight > nBestHeight) ++ if (pindexNew->bnChainWork > bnBestChainWork) +---- + +[role="less_space pagebreak-before"] +* *Terminology change:* General CPUs were used to generate the PoW for +the earliest Bitcoin blocks, but PoW generation today is mostly performed +by specialist Application Specific Integrated Circuits (ASICs), so +instead of saying "CPU power" it is perhaps more correct to say +"computational power" or, simply, "hash rate" for the hashing used +in generating the PoW. + +____ +"As long as a majority of CPU power is controlled by nodes that are not +cooperating to attack the network, they’ll generate the longest chain +and outpace attackers." +____ + +* *Terminology change:* The term "nodes" today is used to refer to +full validation nodes, which are programs that enforce all the rules of +the system. Programs (and hardware) that extend the chain today are +called "miners" based on Nakamoto’s analogy to gold miners in section +6 of the paper. Nakamoto expected all miners to be nodes but the +software he released did not require all nodes to be miners. In the +original software, a simple menu item in the node GUI allowed toggling +the mining function on or off. ++ +Today it is the case that the overwhelming number of nodes are not +miners and that many individuals who own mining hardware do not use it +with their own nodes (and even those that do mine with their own nodes +often mine for short periods of time on top of newly discovered blocks +without ensuring their node considers the new block valid). The early +parts of the paper where "nodes" is mostly used without modification +refer to mining using a full validation node; the later parts of the +paper which refer to "network nodes" is mainly about what nodes can do +even if they aren’t mining. +* *Post-publication discovery:* When a new block is produced, the miner +who produces that block can begin working on its sequel immediately but +all other miners are unaware of the new block and cannot begin working +on it until it has propagated across the +network to them. This gives miners who produce many blocks an edge over +miners who produce fewer blocks, and this can be exploited in what’s +known as the _selfish mining attack_ to allow an attacker with around +30% of total network hash rate to make other miners less profitable, +perhaps driving them into following the attacking miner’s policy. So +instead of saying "a majority of CPU power is controlled by nodes that +are not cooperating to attack the network," it is perhaps more correct +to say "as long as nodes cooperating to attack the network control less +than about 30% of the network." + +=== Transactions + +____ +"We define((("transactions", "errata in Bitcoin whitepaper", id="transaction-errata"))) an electronic coin as a chain of digital signatures. Each +owner transfers the coin to the next by digitally signing a hash of the +previous transaction and the public key of the next owner and adding +these to the end of the coin." +____ + +* *Implementation detail:* Bitcoin implements a more general version of +this system where digital signatures are not used directly but rather a +"deterministic expression" is used instead. Just as a signature that +matches a known public key can be used to enable a payment, the data +that satisfies a known expression can also enable a payment. +Generically, the expression that must be satisfied in Bitcoin in order +to spend a coin is known as an "encumbrance." Almost all encumbrances +in Bitcoin to date require providing at least one signature. So instead +of saying "a chain of digital signatures," it is more correct to say +"a chain of encumbrances." Given that transactions often have more +than one input and more than one output, the structure is not very +chain-like; it’s more accurately described as a directed acyclic ((("transactions", "errata in Bitcoin whitepaper", startref="transaction-errata")))graph +(DAG). + +=== Proof of Work + +____ +"...we((("proof-of-work algorithm", "errata in Bitcoin whitepaper", id="proof-errata"))) implement the proof-of-work by incrementing a nonce in the block +until a value is found that gives the block’s hash the required zero +bits." +____ + +* *Implementation detail:* Adam Back’s Hashcash implementation requires +finding a hash with the required number of leading zero bits. Bitcoin +treats the hash as an integer and requires that it be less than a +specified integer, which effectively allows a fractional number of bits +to be specified. + +____ +"Proof-of-work is essentially one-CPU-one-vote." +____ + +* *Important note:* The vote here is not on the rules of the system but +merely on the ordering of the transactions in order to provide +assurances that an "electronic coin" cannot be easily double spent. +This is described in more detail in section 11 of the paper where it +says, "We consider the scenario of an attacker trying to generate an +alternate chain faster than the honest chain. Even if this is +accomplished, it does not throw the system open to arbitrary changes, +such as creating value out of thin air or taking money that never +belonged to the attacker. Nodes are not going to accept an invalid +transaction as payment, and honest nodes will never accept a block +containing them." + +____ +"...proof-of-work difficulty is determined by a moving average targeting an +average number of blocks per hour." +____ + +* *Implementation detail:* A moving average is not used. Instead, every +2,016th block has its reported generation time compared to the +generation time for an earlier block, and the difference between them is +used to calculate the average used for adjustment. ++ +Further, the average implemented in Bitcoin targets an average number of +blocks per two weeks (not per hour as might be implied by the text). +Other implemented rules may further slow adjustments, such as a rule +that the adjustment cannot increase block production speed by more than +300% per period, nor slow it by more ((("proof-of-work algorithm", "errata in Bitcoin whitepaper", startref="proof-errata")))than 75%. + +=== Reclaiming Disk Space + +____ +"Once the ((("disk space, reclaiming")))((("reclaiming disk space")))((("blocks", "reclaiming disk space")))latest transaction in a coin is buried under enough blocks, the +spent transactions before it can be discarded to save disk space." +____ + +* *Possible post-publication discovery:* Although the merkle tree +structure described in this section can prove a transaction was included +in a particular block, there is currently no way in Bitcoin to prove +that a transaction has not been spent except to process all subsequent +data in the blockchain. This means the method described here cannot be +universally used for reclaiming disk space among all nodes, as all new +nodes will need to process all transactions. + +=== Simplified Payment Verification + +____ +"One strategy((("payment verification", "errata in Bitcoin whitepaper")))((("verifying", "payment", "errata in Bitcoin whitepaper"))) to protect against this would be to accept alerts from +network nodes when they detect an invalid block, prompting the user’s +software to download the full block and alerted transactions to confirm +the inconsistency." +____ + +* *Important Note:* Although software has been produced that implements +some parts of this section and calls that Simplified Payment +Verification (SPV), none of these programs currently accepts alerts from +network nodes (full validation nodes) when invalid blocks have been +detected. This has placed bitcoins in so-called SPV wallets at risk in +the past. + +=== Privacy + +____ +"Some linking((("privacy", "errata in Bitcoin whitepaper"))) is still unavoidable with multi-input transactions, which +necessarily reveal that their inputs were owned by the same owner." +____ + +* *Post-publication invention:* It isn't clear that different inputs +in the same transaction have the same owner if owners often mix their +inputs with +inputs belonging to other owners. For example, there’s no public +difference between Alice and Bob each contributing one of their inputs +toward paying Charlie and Dan than there is between just Alice +contributing two of her inputs toward paying Charlie and Dan. ++ +This technique is known today as +https://oreil.ly/UBEJX[CoinJoin], and software implementing +it has been in use since 2015. + +=== Calculations + +____ +"The receiver ((("calculations", "errata in Bitcoin whitepaper")))generates a new key pair and gives the public key to the +sender shortly before signing. This prevents the sender from preparing a +chain of blocks ahead of time by working on it continuously until he is +lucky enough to get far enough ahead, then executing the transaction at +that moment." +____ + +* *Post-publication discovery:* Nothing about the receiver generating a +public key shortly before the spender signs a transaction prevents the +spender from preparing a chain of blocks ahead of time. Early Bitcoin +user Hal Finney discovered this attack and +https://oreil.ly/kg_Xe[described +it]: "Suppose the attacker is generating blocks occasionally. In each +block he generates, he includes a transfer from address A to address B, +both of which he controls. ++ +"To cheat you, when he generates a block, he doesn’t broadcast it. +Instead, he runs down to your store and makes a payment to your address +C with his address A. You wait a few seconds, don’t hear anything, and +transfer the goods. He broadcasts his block now, and his transaction +will take precedence over yours." ++ +The attack works for any number of confirmations, and is sometimes named +the Finney Attack. + +''''' + +*Disclaimer:* The author of this document was not the first person to +identify any of the problems described here—he has merely collected them +into a single document. + +*License:* This errata document is released under the +https://oreil.ly/xZeBR[CC0] 1.0 Universal +Public Domain Dedication + +For updates made ((("Bitcoin whitepaper", "errata", startref="bitcoin-whitepaper-errata")))((("whitepaper (Bitcoin)", "errata", startref="whitepaper-errata")))after the publication of this book, please see the +https://oreil.ly/ygExa[Original +document]. diff --git a/appc_bips.adoc b/appc_bips.adoc new file mode 100644 index 00000000..b98bb090 --- /dev/null +++ b/appc_bips.adoc @@ -0,0 +1,73 @@ +[[appdxbitcoinimpproposals]] +[appendix] +== Bitcoin Improvement Proposals + +Bitcoin Improvement Proposals are design documents providing information to the Bitcoin community or describing a new feature for Bitcoin or its processes or environment. + +As per BIP1 _BIP Purpose and Guidelines_, there are three((("BIPs (Bitcoin Improvement Proposals)", "types of"))) kinds of BIPs: + +_Standard_ BIP:: Describes any change that affects most or all Bitcoin implementations, such as a change to the network protocol, a change in block or transaction validity rules, or any change or addition that affects the interoperability of applications using Bitcoin. +_Informational_ BIP:: Describes a Bitcoin design issue or provides general guidelines or information to the Bitcoin community, but does not propose a new feature. Informational BIPs do not necessarily represent a Bitcoin community consensus or recommendation, so users and implementors may ignore informational BIPs or follow their advice. +_Process_ BIP:: Describes a Bitcoin process or proposes a change to (or an event in) a process. Process BIPs are like standard BIPs but apply to areas other than the Bitcoin protocol itself. They might propose an implementation but not to Bitcoin's codebase; they often require community consensus. Unlike informational BIPs, they are more than recommendations, and users are typically not free to ignore them. Examples include procedures, guidelines, changes to the decision-making process, and changes to the tools or environment used in Bitcoin development. Any meta-BIP is also considered a process BIP. + +BIPs are recorded in a https://oreil.ly/jjO0R[versioned repository on GitHub]. +An MIT-licensed document from the open source Bitcoin Core project, +reproduced here in edited form, describes which BIPs it implements, including listing +the Pull Request (PR) and version of Bitcoin Core where support for each BIP was added or +significantly changed. + +BIPs that are ((("BIPs (Bitcoin Improvement Proposals)", "implemented by Bitcoin Core", id="bips-implement")))((("Bitcoin Core", "BIPs implemented by", id="bitcoin-core-bips")))implemented by Bitcoin Core: + +- BIP9: The changes allowing multiple soft forks to be deployed in parallel have been implemented since v0.12.1 (PR #7575). +- BIP11: Multisig outputs are standard since v0.6.0 (PR #669). +- BIP13: The address format for P2SH addresses has been implemented since v0.6.0 (PR #669). +- BIP14: The subversion string is being used as User Agent since v0.6.0 (PR #669). +- BIP16: The pay-to-script-hash evaluation rules have been implemented since v0.6.0, and took effect on April 1st 2012 (PR #748). +- BIP21: The URI format for Bitcoin payments has been implemented since v0.6.0 (PR #176). +- BIP22: The 'getblocktemplate' (GBT) RPC protocol for mining has been implemented since v0.7.0 (PR #936). +- BIP23: Some extensions to GBT have been implemented since v0.10.0rc1, including longpolling and block proposals (PR #1816). +- BIP30: The evaluation rules to forbid creating new transactions with the same txid as previous not-fully-spent transactions were implemented since v0.6.0, and the rule took effect on March 15th 2012 (PR #915). +- BIP31: The 'pong' protocol message (and the protocol version bump to 60001) has been implemented since v0.6.1 (PR #1081). +- BIP32: Hierarchical Deterministic Wallets has been implemented since v0.13.0 (PR #8035). +- BIP34: The rule that requires blocks to contain their height (number) in the coinbase input, and the introduction of version 2 blocks has been implemented since v0.7.0. The rule took effect for version 2 blocks as of block 224413 (March 5th 2013), and version 1 blocks are no longer allowed since block 227931 (March 25th 2013) (PR #1526). +- BIP35: The 'mempool' protocol message (and the protocol version bump to 60002) has been implemented since v0.7.0 (PR #1641). As of v0.13.0, this is only available for +NODE_BLOOM+ (BIP111) peers. + +[role="less_space pagebreak-before"] +- BIP37: The bloom filtering for transaction relaying, partial Merkle trees for blocks, and the protocol version bump to 70001 (enabling low-bandwidth lightweight clients) has been implemented since v0.8.0 (PR #1795). Disabled by default since v0.19.0, can be enabled by the +-peerbloomfilters+ option. +- BIP42: The bug that would have caused the subsidy schedule to resume after block 13440000 was fixed in v0.9.2 (PR #3842). +- BIP43: The experimental descriptor wallets introduced in v0.21.0 by default use the Hierarchical Deterministic Wallet derivation proposed by BIP43 (PR #16528). +- BIP44: The experimental descriptor wallets introduced in v0.21.0 by default use the Hierarchical Deterministic Wallet derivation proposed by BIP44 (PR #16528). +- BIP49: The experimental descriptor wallets introduced in v0.21.0 by default use the Hierarchical Deterministic Wallet derivation proposed by BIP49 (PR #16528). +- BIP61: The 'reject' protocol message (and the protocol version bump to 70002) was added in v0.9.0 (PR #3185). Starting v0.17.0, whether to send reject messages can be configured with the ++-enablebip61++ option, and support is deprecated (disabled by default) as of v0.18.0. Support was removed in v0.20.0 (PR #15437). +- BIP65: The ++CHECKLOCKTIMEVERIFY++ soft fork was merged in v0.12.0 (PR #6351), and backported to v0.11.2 and v0.10.4. Mempool-only +CLTV+ was added in PR #6124. +- BIP66: The strict DER rules and associated version 3 blocks have been implemented since v0.10.0 (PR #5713). +- BIP68: Sequence locks have been implemented as of v0.12.1 (PR #7184), and have been buried since v0.19.0 (PR #16060). +- BIP70 71 72: Payment Protocol support has been available in Bitcoin Core GUI since v0.9.0 (PR #5216). Support can be optionally disabled at build time since v0.18.0 (PR 14451), and it is disabled by default at build time since v0.19.0 (PR #15584). It has been removed as of v0.20.0 (PR 17165). +- BIP84: The experimental descriptor wallets introduced in v0.21.0 by default use the Hierarchical Deterministic Wallet derivation proposed by BIP84. (PR #16528) +- BIP86: Descriptor wallets by default use the Hierarchical Deterministic Wallet derivation proposed by BIP86 since v23.0 (PR #22364). +- BIP90: Trigger mechanism for activation of BIPs 34, 65, and 66 has been simplified to block height checks since v0.14.0 (PR #8391). +- BIP111: +NODE_BLOOM+ service bit added and enforced for all peer versions as of v0.13.0 (PR #6579 and PR #6641). +- BIP112: The +CHECKSEQUENCEVERIFY+ opcode has been implemented since v0.12.1 (PR #7524), and has been buried since v0.19.0 (PR #16060). +- BIP113: Median time past lock-time calculations have been implemented since v0.12.1 (PR #6566), and has been buried since v0.19.0 (PR #16060). +- BIP125: Opt-in full replace-by-fee signaling partially implemented. +- BIP130: direct headers announcement is negotiated with peer versions ≥70012 as of v0.12.0 (PR 6494). +- BIP133: feefilter messages are respected and sent for peer versions ≥70013 as of v0.13.0 (PR 7542). +- BIP141: Segregated Witness (Consensus Layer) as of v0.13.0 (PR 8149), defined for mainnet as of v0.13.1 (PR 8937), and buried since v0.19.0 (PR #16060). +- BIP143: Transaction Signature Verification for Version 0 Witness Program as of v0.13.0 (PR 8149), defined for mainnet as of v0.13.1 (PR 8937), and buried since v0.19.0 (PR #16060). +- BIP144: Segregated Witness as of 0.13.0 (PR 8149). +- BIP145: getblocktemplate updates for Segregated Witness as of v0.13.0 (PR 8149). +- BIP147: +NULLDUMMY+ soft fork as of v0.13.1 (PR 8636 and PR 8937), buried since v0.19.0 (PR #16060). +- BIP152: Compact block transfer and related optimizations are used as of v0.13.0 (PR 8068). +- BIP155: The 'addrv2' and 'sendaddrv2' messages which enable relay of Tor V3 addresses (and other networks) are supported as of v0.21.0 (PR 19954). +- BIP157 158: Compact Block Filters for Light Clients can be indexed as of v0.19.0 (PR #14121) and served to peers on the P2P network as of v0.21.0 (PR #16442). +- BIP159: The +NODE_NETWORK_LIMITED+ service bit is signalled as of v0.16.0 (PR 11740), and such nodes are connected to as of v0.17.0 (PR 10387). +- BIP173: Bech32 addresses for native Segregated Witness outputs are supported as of v0.16.0 (PR 11167). Bech32 addresses are generated by default as of v0.20.0 (PR 16884). +- BIP174: RPCs to operate on Partially Signed Bitcoin Transactions (PSBT) are present as of v0.17.0 (PR 13557). +- BIP176: Bits Denomination [QT only] is supported as of v0.16.0 (PR 12035). +- BIP325: Signet test network is supported as of v0.21.0 (PR 18267). +- BIP339: Relay of transactions by wtxid is supported as of v0.21.0 (PR 18044). +- BIP340 341 342: Validation rules for Taproot (including Schnorr signatures and Tapscript leaves) are implemented as of v0.21.0 (PR 19953), with mainnet activation as of v0.21.1 (PR 21377, PR 21686). +- BIP350: Addresses for native v1+ segregated Witness outputs use bech32m instead of bech32 as of v22.0 (PR 20861). +- BIP371: Taproot fields for PSBT as of v24.0 (PR 22558). +- BIP380 381 382 383 384 385: Output Script Descriptors, and most of Script Expressions are implemented as of v0.17.0 (PR 13697). +- BIP386: +tr()+ Output Script Descriptors are implemented as((("BIPs (Bitcoin Improvement Proposals)", "implemented by Bitcoin Core", startref="bips-implement")))((("Bitcoin Core", "BIPs implemented by", startref="bitcoin-core-bips"))) of v22.0 (PR 22051). diff --git a/appdx-bips.asciidoc b/appdx-bips.asciidoc deleted file mode 100644 index c6021c97..00000000 --- a/appdx-bips.asciidoc +++ /dev/null @@ -1,116 +0,0 @@ -[[appdxbitcoinimpproposals]] -[appendix] -== Bitcoin Improvement Proposals - -((("bitcoin improvement proposals", "types of")))Bitcoin Improvement Proposals are design documents providing information to the bitcoin community, or for describing a new feature for bitcoin or its processes or environment. - -As per BIP-01 _BIP Purpose and Guidelines_, there are three kinds of BIPs: - -_Standard_ BIP:: Describes any change that affects most or all bitcoin implementations, such as a change to the network protocol, a change in block or transaction validity rules, or any change or addition that affects the interoperability of applications using bitcoin. -_Informational_ BIP:: Describes a bitcoin design issue, or provides general guidelines or information to the bitcoin community, but does not propose a new feature. Informational BIPs do not necessarily represent a bitcoin community consensus or recommendation, so users and implementors may ignore informational BIPs or follow their advice. -_Process_ BIP:: Describes a bitcoin process, or proposes a change to (or an event in) a process. Process BIPs are like standard BIPs but apply to areas other than the Bitcoin protocol itself. They might propose an implementation, but not to bitcoin's codebase; they often require community consensus; and unlike informational BIPs, they are more than recommendations, and users are typically not free to ignore them. Examples include procedures, guidelines, changes to the decision-making process, and changes to the tools or environment used in bitcoin development. Any meta-BIP is also considered a process BIP. - -((("bitcoin improvement proposals", "repository of")))BIPs are recorded in a versioned repository on GitHub: https://github.com/bitcoin/bips[https://github.com/bitcoin/bips]. <> shows a snapshot of the BIPs in April 2017. Consult the authoritative repository for up-to-date information on existing BIPs and their contents.((("bitcoin improvement proposals", "snapshot of", id="BIPsnap15"))) - -[[table_d-1]] -.Snapshot of BIPs -[options="header"] -|======================================================================= -|BIP# | Title |Owner |Type |Status -|[[bip-1]]https://github.com/bitcoin/bips/blob/master/bip-0001.mediawiki[BIP-1] |BIP Purpose and Guidelines |Amir Taaki |Process |Replaced -|[[bip-2]]https://github.com/bitcoin/bips/blob/master/bip-0002.mediawiki[BIP-2] |BIP process, revised |Luke Dashjr |Process |Active -|[[bip-8]]https://github.com/bitcoin/bips/blob/master/bip-0008.mediawiki[BIP-8] |Version bits with guaranteed lock-in |Shaolin Fry |Informational |Draft -|[[bip-9]]https://github.com/bitcoin/bips/blob/master/bip-0009.mediawiki[BIP-9] |Version bits with timeout and delay |Pieter Wuille, Peter Todd, Greg Maxwell, Rusty Russell |Informational |Final -|[[bip-10]]https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki[BIP-10] |Multi-Sig Transaction Distribution |Alan Reiner |Informational |Withdrawn -|[[bip-11]]https://github.com/bitcoin/bips/blob/master/bip-0011.mediawiki[BIP-11] |M-of-N Standard Transactions |Gavin Andresen |Standard |Final -|[[bip-12]]https://github.com/bitcoin/bips/blob/master/bip-0012.mediawiki[BIP-12] |OP_EVAL |Gavin Andresen |Standard |Withdrawn -|[[bip-13]]https://github.com/bitcoin/bips/blob/master/bip-0013.mediawiki[BIP-13] |Address Format for pay-to-script-hash |Gavin Andresen |Standard |Final -|[[bip-14]]https://github.com/bitcoin/bips/blob/master/bip-0014.mediawiki[BIP-14] |Protocol Version and User Agent |Amir Taaki, Patrick Strateman |Standard |Final -|[[bip-15]]https://github.com/bitcoin/bips/blob/master/bip-0015.mediawiki[BIP-15] |Aliases |Amir Taaki |Standard |Deferred -|[[bip-16]]https://github.com/bitcoin/bips/blob/master/bip-0016.mediawiki[BIP-16] |Pay to Script Hash |Gavin Andresen |Standard |Final -|[[bip-17]]https://github.com/bitcoin/bips/blob/master/bip-0017.mediawiki[BIP-17] |OP_CHECKHASHVERIFY (CHV) |Luke Dashjr |Standard |Withdrawn -|[[bip-18]]https://github.com/bitcoin/bips/blob/master/bip-0018.mediawiki[BIP-18] |hashScriptCheck |Luke Dashjr |Standard |Proposed -|[[bip-19]]https://github.com/bitcoin/bips/blob/master/bip-0019.mediawiki[BIP-19] |M-of-N Standard Transactions (Low SigOp) |Luke Dashjr |Standard |Draft -|[[bip-20]]https://github.com/bitcoin/bips/blob/master/bip-0020.mediawiki[BIP-20] |URI Scheme |Luke Dashjr |Standard |Replaced -|[[bip-21]]https://github.com/bitcoin/bips/blob/master/bip-0021.mediawiki[BIP-21] |URI Scheme |Nils Schneider, Matt Corallo |Standard |Final -|[[bip-22]]https://github.com/bitcoin/bips/blob/master/bip-0022.mediawiki[BIP-22] |getblocktemplate - Fundamentals |Luke Dashjr |Standard |Final -|[[bip-23]]https://github.com/bitcoin/bips/blob/master/bip-0023.mediawiki[BIP-23] |getblocktemplate - Pooled Mining |Luke Dashjr |Standard |Final -|[[bip-30]]https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki[BIP-30] |Duplicate transactions |Pieter Wuille |Standard |Final -|[[bip-31]]https://github.com/bitcoin/bips/blob/master/bip-0031.mediawiki[BIP-31] |Pong message |Mike Hearn |Standard |Final -|[[bip-32]]https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki[BIP-32] |Hierarchical Deterministic Wallets |Pieter Wuille |Informational |Final -|[[bip-33]]https://github.com/bitcoin/bips/blob/master/bip-0033.mediawiki[BIP-33] |Stratized Nodes |Amir Taaki |Standard |Draft -|[[bip-34]]https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki[BIP-34] |Block v2, Height in Coinbase |Gavin Andresen |Standard |Final -|[[bip-35]]https://github.com/bitcoin/bips/blob/master/bip-0035.mediawiki[BIP-35] |mempool message |Jeff Garzik |Standard |Final -|[[bip-36]]https://github.com/bitcoin/bips/blob/master/bip-0036.mediawiki[BIP-36] |Custom Services |Stefan Thomas |Standard |Draft -|[[bip-37]]https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki[BIP-37] |Connection Bloom filtering |Mike Hearn, Matt Corallo |Standard |Final -|[[bip-39]]https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki[BIP-39] |Mnemonic code for generating deterministic keys |Marek Palatinus, Pavol Rusnak, Aaron Voisine, Sean Bowe |Standard |Proposed -|[[bip-40]]https://github.com/bitcoin/bips/blob/master/bip-0040.mediawiki[BIP-40] |Stratum wire protocol |Marek Palatinus |Standard |BIP number allocated -|[[bip-41]]https://github.com/bitcoin/bips/blob/master/bip-0041.mediawiki[BIP-41] |Stratum mining protocol |Marek Palatinus |Standard |BIP number allocated -|[[bip-42]]https://github.com/bitcoin/bips/blob/master/bip-0042.mediawiki[BIP-42] |A finite monetary supply for Bitcoin |Pieter Wuille |Standard |Draft -|[[bip-43]]https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki[BIP-43] |Purpose Field for Deterministic Wallets |Marek Palatinus, Pavol Rusnak |Informational |Draft -|[[bip-44]]https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki[BIP-44] |Multi-Account Hierarchy for Deterministic Wallets |Marek Palatinus, Pavol Rusnak |Standard |Proposed -|[[bip-45]]https://github.com/bitcoin/bips/blob/master/bip-0045.mediawiki[BIP-45] |Structure for Deterministic P2SH Multisignature Wallets |Manuel Araoz, Ryan X. Charles, Matias Alejo Garcia |Standard |Proposed -|[[bip-47]]https://github.com/bitcoin/bips/blob/master/bip-0047.mediawiki[BIP-47] |Reusable Payment Codes for Hierarchical Deterministic Wallets |Justus Ranvier |Informational |Draft -|[[bip-49]]https://github.com/bitcoin/bips/blob/master/bip-0049.mediawiki[BIP-49] |Derivation scheme for P2WPKH-nested-in-P2SH based accounts |Daniel Weigl |Informational |Draft -|[[bip-50]]https://github.com/bitcoin/bips/blob/master/bip-0050.mediawiki[BIP-50] |March 2013 Chain Fork Post-Mortem |Gavin Andresen |Informational |Final -|[[bip-60]]https://github.com/bitcoin/bips/blob/master/bip-0060.mediawiki[BIP-60] |Fixed Length "version" Message (Relay-Transactions Field) |Amir Taaki |Standard |Draft -|[[bip-61]]https://github.com/bitcoin/bips/blob/master/bip-0061.mediawiki[BIP-61] |Reject P2P message |Gavin Andresen |Standard |Final -|[[bip-62]]https://github.com/bitcoin/bips/blob/master/bip-0062.mediawiki[BIP-62] |Dealing with malleability |Pieter Wuille |Standard |Withdrawn -|[[bip-63]]https://github.com/bitcoin/bips/blob/master/bip-0063.mediawiki[BIP-63] |Stealth Addresses |Peter Todd |Standard |BIP number allocated -|[[bip-64]]https://github.com/bitcoin/bips/blob/master/bip-0064.mediawiki[BIP-64] |getutxo message |Mike Hearn |Standard |Draft -|[[bip-65]]https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki[BIP-65] |OP_CHECKLOCKTIMEVERIFY |Peter Todd |Standard |Final -|[[bip-66]]https://github.com/bitcoin/bips/blob/master/bip-0066.mediawiki[BIP-66] |Strict DER signatures |Pieter Wuille |Standard |Final -|[[bip-67]]https://github.com/bitcoin/bips/blob/master/bip-0067.mediawiki[BIP-67] |Deterministic Pay-to-script-hash multi-signature addresses through public key sorting |Thomas Kerin, Jean-Pierre Rupp, Ruben de Vries |Standard |Proposed -|[[bip-68]]https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68] |Relative lock-time using consensus-enforced sequence numbers |Mark Friedenbach, BtcDrak, Nicolas Dorier, kinoshitajona |Standard |Final -|[[bip-69]]https://github.com/bitcoin/bips/blob/master/bip-0069.mediawiki[BIP-69] |Lexicographical Indexing of Transaction Inputs and Outputs |Kristov Atlas |Informational |Proposed -|[[bip-70]]https://github.com/bitcoin/bips/blob/master/bip-0070.mediawiki[BIP-70] |Payment Protocol |Gavin Andresen, Mike Hearn |Standard |Final -|[[bip-71]]https://github.com/bitcoin/bips/blob/master/bip-0071.mediawiki[BIP-71] |Payment Protocol MIME types |Gavin Andresen |Standard |Final -|[[bip-72]]https://github.com/bitcoin/bips/blob/master/bip-0072.mediawiki[BIP-72] |bitcoin: uri extensions for Payment Protocol |Gavin Andresen |Standard |Final -|[[bip-73]]https://github.com/bitcoin/bips/blob/master/bip-0073.mediawiki[BIP-73] |Use "Accept" header for response type negotiation with Payment Request URLs |Stephen Pair |Standard |Final -|[[bip-74]]https://github.com/bitcoin/bips/blob/master/bip-0074.mediawiki[BIP-74] |Allow zero value OP_RETURN in Payment Protocol |Toby Padilla |Standard |Draft -|[[bip-75]]https://github.com/bitcoin/bips/blob/master/bip-0075.mediawiki[BIP-75] |Out of Band Address Exchange using Payment Protocol Encryption |Justin Newton, Matt David, Aaron Voisine, James MacWhyte |Standard |Draft -|[[bip-80]]https://github.com/bitcoin/bips/blob/master/bip-0080.mediawiki[BIP-80] |Hierarchy for Non-Colored Voting Pool Deterministic Multisig Wallets |Justus Ranvier, Jimmy Song |Informational |Deferred -|[[bip-81]]https://github.com/bitcoin/bips/blob/master/bip-0081.mediawiki[BIP-81] |Hierarchy for Colored Voting Pool Deterministic Multisig Wallets |Justus Ranvier, Jimmy Song |Informational |Deferred -|[[bip-83]]https://github.com/bitcoin/bips/blob/master/bip-0083.mediawiki[BIP-83] |Dynamic Hierarchical Deterministic Key Trees |Eric Lombrozo |Standard |Draft -|[[bip-90]]https://github.com/bitcoin/bips/blob/master/bip-0090.mediawiki[BIP-90] |Buried Deployments |Suhas Daftuar |Informational |Draft -|[[bip-99]]https://github.com/bitcoin/bips/blob/master/bip-0099.mediawiki[BIP-99] |Motivation and deployment of consensus rule changes ([soft/hard]forks) |Jorge Timón |Informational |Draft -|[[bip-101]]https://github.com/bitcoin/bips/blob/master/bip-0101.mediawiki[BIP-101] |Increase maximum block size |Gavin Andresen |Standard |Withdrawn -|[[bip-102]]https://github.com/bitcoin/bips/blob/master/bip-0102.mediawiki[BIP-102] |Block size increase to 2MB |Jeff Garzik |Standard |Draft -|[[bip-103]]https://github.com/bitcoin/bips/blob/master/bip-0103.mediawiki[BIP-103] |Block size following technological growth |Pieter Wuille |Standard |Draft -|[[bip-104]]https://github.com/bitcoin/bips/blob/master/bip-0104.mediawiki[BIP-104] |'Block75' - Max block size like difficulty |t.khan |Standard |Draft -|[[bip-105]]https://github.com/bitcoin/bips/blob/master/bip-0105.mediawiki[BIP-105] |Consensus based block size retargeting algorithm |BtcDrak |Standard |Draft -|[[bip-106]]https://github.com/bitcoin/bips/blob/master/bip-0106.mediawiki[BIP-106] |Dynamically Controlled Bitcoin Block Size Max Cap |Upal Chakraborty |Standard |Draft -|[[bip-107]]https://github.com/bitcoin/bips/blob/master/bip-0107.mediawiki[BIP-107] |Dynamic limit on the block size |Washington Y. Sanchez |Standard |Draft -|[[bip-109]]https://github.com/bitcoin/bips/blob/master/bip-0109.mediawiki[BIP-109] |Two million byte size limit with sigop and sighash limits |Gavin Andresen |Standard |Rejected -|[[bip-111]]https://github.com/bitcoin/bips/blob/master/bip-0111.mediawiki[BIP-111] |NODE_BLOOM service bit |Matt Corallo, Peter Todd |Standard |Proposed -|[[bip-112]]https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[BIP-112] |CHECKSEQUENCEVERIFY |BtcDrak, Mark Friedenbach, Eric Lombrozo |Standard |Final -|[[bip-113]]https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki[BIP-113] |Median time-past as endpoint for lock-time calculations |Thomas Kerin, Mark Friedenbach |Standard |Final -|[[bip-114]]https://github.com/bitcoin/bips/blob/master/bip-0114.mediawiki[BIP-114] |Merkelized Abstract Syntax Tree |Johnson Lau |Standard |Draft -|[[bip-120]]https://github.com/bitcoin/bips/blob/master/bip-0120.mediawiki[BIP-120] |Proof of Payment |Kalle Rosenbaum |Standard |Draft -|[[bip-121]]https://github.com/bitcoin/bips/blob/master/bip-0121.mediawiki[BIP-121] |Proof of Payment URI scheme |Kalle Rosenbaum |Standard |Draft -|[[bip-122]]https://github.com/bitcoin/bips/blob/master/bip-0122.mediawiki[BIP-122] |URI scheme for Blockchain references / exploration |Marco Pontello |Standard |Draft -|[[bip-123]]https://github.com/bitcoin/bips/blob/master/bip-0123.mediawiki[BIP-123] |BIP Classification |Eric Lombrozo |Process |Active -|[[bip-124]]https://github.com/bitcoin/bips/blob/master/bip-0124.mediawiki[BIP-124] |Hierarchical Deterministic Script Templates |Eric Lombrozo, William Swanson |Informational |Draft -|[[bip-125]]https://github.com/bitcoin/bips/blob/master/bip-0125.mediawiki[BIP-125] |Opt-in Full Replace-by-Fee Signaling |David A. Harding, Peter Todd |Standard |Proposed -|[[bip-126]]https://github.com/bitcoin/bips/blob/master/bip-0126.mediawiki[BIP-126] |Best Practices for Heterogeneous Input Script Transactions |Kristov Atlas |Informational |Draft -|[[bip-130]]https://github.com/bitcoin/bips/blob/master/bip-0130.mediawiki[BIP-130] |sendheaders message |Suhas Daftuar |Standard |Proposed -|[[bip-131]]https://github.com/bitcoin/bips/blob/master/bip-0131.mediawiki[BIP-131] |"Coalescing Transaction" Specification (wildcard inputs) |Chris Priest |Standard |Draft -|[[bip-132]]https://github.com/bitcoin/bips/blob/master/bip-0132.mediawiki[BIP-132] |Committee-based BIP Acceptance Process |Andy Chase |Process |Withdrawn -|[[bip-133]]https://github.com/bitcoin/bips/blob/master/bip-0133.mediawiki[BIP-133] |feefilter message |Alex Morcos |Standard |Draft -|[[bip-134]]https://github.com/bitcoin/bips/blob/master/bip-0134.mediawiki[BIP-134] |Flexible Transactions |Tom Zander |Standard |Draft -|[[bip-140]]https://github.com/bitcoin/bips/blob/master/bip-0140.mediawiki[BIP-140] |Normalized TXID |Christian Decker |Standard |Draft -|[[bip-141]]https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki[BIP-141] |Segregated Witness (Consensus layer) |Eric Lombrozo, Johnson Lau, Pieter Wuille |Standard |Draft -|[[bip-142]]https://github.com/bitcoin/bips/blob/master/bip-0142.mediawiki[BIP-142] |Address Format for Segregated Witness |Johnson Lau |Standard |Deferred -|[[bip-143]]https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki[BIP-143] |Transaction Signature Verification for Version 0 Witness Program |Johnson Lau, Pieter Wuille |Standard |Draft -|[[bip-144]]https://github.com/bitcoin/bips/blob/master/bip-0144.mediawiki[BIP-144] |Segregated Witness (Peer Services) |Eric Lombrozo, Pieter Wuille |Standard |Draft -|[[bip-145]]https://github.com/bitcoin/bips/blob/master/bip-0145.mediawiki[BIP-145] |getblocktemplate Updates for Segregated Witness |Luke Dashjr |Standard |Draft -|[[bip-146]]https://github.com/bitcoin/bips/blob/master/bip-0146.mediawiki[BIP-146] |Dealing with signature encoding malleability |Johnson Lau, Pieter Wuille |Standard |Draft -|[[bip-147]]https://github.com/bitcoin/bips/blob/master/bip-0147.mediawiki[BIP-147] |Dealing with dummy stack element malleability |Johnson Lau |Standard |Draft -|[[bip-148]]https://github.com/bitcoin/bips/blob/master/bip-0148.mediawiki[BIP-148] |Mandatory activation of segwit deployment |Shaolin Fry |Standard |Draft -|[[bip-150]]https://github.com/bitcoin/bips/blob/master/bip-0150.mediawiki[BIP-150] |Peer Authentication |Jonas Schnelli |Standard |Draft -|[[bip-151]]https://github.com/bitcoin/bips/blob/master/bip-0151.mediawiki[BIP-151] |Peer-to-Peer Communication Encryption |Jonas Schnelli |Standard |Draft -|[[bip-152]]https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki[BIP-152] |Compact Block Relay |Matt Corallo |Standard |Draft -|[[bip-171]]https://github.com/bitcoin/bips/blob/master/bip-0171.mediawiki[BIP-171] |Currency/exchange rate information API |Luke Dashjr |Standard |Draft -|[[bip-180]]https://github.com/bitcoin/bips/blob/master/bip-0180.mediawiki[BIP-180] |Block size/weight fraud proof |Luke Dashjr |Standard |Draft -|[[bip-199]]https://github.com/bitcoin/bips/blob/master/bip-0199.mediawiki[BIP-199] |Hashed Time-Locked Contract transactions |Sean Bowe, Daira Hopwood |Standard |Draft((("", startref="BIPsnap15"))) -|======================================================================= diff --git a/appdx-bitcore.asciidoc b/appdx-bitcore.asciidoc deleted file mode 100644 index 238556be..00000000 --- a/appdx-bitcore.asciidoc +++ /dev/null @@ -1,116 +0,0 @@ -[[appdx_bitcore]] -[appendix] - -== Bitcore - - -((("Bitcore", id="bitcore16")))Bitcore is a suite of tools provided by BitPay. Its goal is to provide easy-to-use tools for Bitcoin developers. Almost all of Bitcore's code is written in JavaScript. There are some modules written specifically for NodeJS. Finally, the "node" module of Bitcore includes Bitcoin Core's C++ code. Please see https://bitcore.io for more information. - -=== Bitcore's Feature List - -* Bitcoin full node (bitcore-node) -* Block explorer (insight) -* Block, transaction, and wallet utilities (bitcore-lib) -* Communicating directly with Bitcoin's P2P network (bitcore-p2p) -* Seed entropy mnemonic generation (bitcore-mnemonic) -* Payment protocol (bitcore-payment-protocol) -* Message verification and signing (bitcore-message) -* Elliptic curve Integrated Encryption Scheme (bitcore-ecies) -* Wallet service (bitcore-wallet-service) -* Wallet client (bitcore-wallet-client) -* Integrating services directly with Bitcoin Core (bitcore-node) - -=== Bitcore Library Examples - -==== Prerequisites - -* NodeJS >= 4.x - -If using NodeJS and the node REPL: - -[source,bash] ----- -$ npm install -g bitcore-lib bitcore-p2p ----- - -==== Wallet Examples using bitcore-lib - -Creating a new Bitcoin address with associated private key: - ----- -> bitcore = require('bitcore-lib') -> privateKey = new bitcore.PrivateKey() -> address = privateKey.toAddress().toString() ----- - -Creating a hierarchical deterministic private key and address: - ----- -> hdPrivateKey = bitcore.HDPrivateKey() -> hdPublicKey = bitcore.HDPublicKey(hdPrivateKey) -> hdAddress = new bitcore.Address(hdPublicKey.publicKey).toString() ----- - -Creating and signing a transaction from an UTXO: - ----- -> utxo = { - txId: txId, // transaction id containing an unspent output - outputIndex: outputIndex, // output index (e.g. 0) - address: addressOfUtxo, - script: bitcore.Script.buildPublicKeyHashOut(addressOfUtxo).toString(), - satoshis: satoshis // amount sent to the address -} -> fee = 3000 //set appropriately for conditions on the network -> tx = new bitcore.Transaction() - .from(utxo) - .to(address, 35000) - .fee(fee) - .enableRBF() - .sign(privateKeyOfUtxo) ----- - -Replace the last transaction in the mempool (replace-by-fee): - ----- -> rbfTx = new Transaction() - .from(utxo) - .to(address, 35000) - .fee(fee*2) - .enableRBF() - .sign(privateKeyOfUtxo); -> tx.serialize(); -> rbfTx.serialize(); ----- - -Broadcasting a transaction to the Bitcoin network -(note: broadcast valid transactions only; refer to https://bitnodes.21.co/nodes[] for peer hosts): - -1. Copy the code below into a file called _broadcast.js_. -2. The +tx+ and +rbfTx+ variables are the output of +tx.serialize()+ and +rbfTx.serialize()+, respectively. -3. In order to replace-by-fee, the peer must support bitcoind option +mempoolreplace+ and have it set to +1+. -4. Run the file node _broadcast.js_((("", startref="bitcore16"))): - ----- -var p2p = require('bitcore-p2p'); -var bitcore = require('bitcore-lib'); -var tx = new bitcore.Transaction('output from serialize function'); -var rbfTx = new bitcore.Transaction('output from serialize function'); -var host = 'ip address'; //use valid peer listening on tcp 8333 -var peer = new p2p.Peer({host: host}); -var messages = new p2p.Messages(); -peer.on('ready', function() { - var txs = [messages.Transaction(tx), messages.Transaction(rbfTx)]; - var index = 0; - var interval = setInterval(function() { - peer.sendMessage(txs[index++]); - console.log('tx: ' + index + ' sent'); - if (index === txs.length) { - clearInterval(interval); - console.log('disconnecting from peer: ' + host); - peer.disconnect(); - } - }, 2000); -}); -peer.connect(); ----- diff --git a/appdx-bx.asciidoc b/appdx-bx.asciidoc deleted file mode 100644 index b4fd2a55..00000000 --- a/appdx-bx.asciidoc +++ /dev/null @@ -1,181 +0,0 @@ -[[appdx_bx]] -[appendix] - -== Bitcoin Explorer (bx) Commands - -((("Bitcoin Explorer (bx) commands", id="BX18_1", range="startofrange")))Bitcoin Explorer (bx) is a command-line tool that offers a variety of commands for key management and transaction construction. It is part of the libbitcoin bitcoin library. - ----- -Usage: bx COMMAND [--help] - -Info: The bx commands are: - -address-decode -address-embed -address-encode -address-validate -base16-decode -base16-encode -base58-decode -base58-encode -base58check-decode -base58check-encode -base64-decode -base64-encode -bitcoin160 -bitcoin256 -btc-to-satoshi -ec-add -ec-add-secrets -ec-multiply -ec-multiply-secrets -ec-new -ec-to-address -ec-to-public -ec-to-wif -fetch-balance -fetch-header -fetch-height -fetch-history -fetch-stealth -fetch-tx -fetch-tx-index -hd-new -hd-private -hd-public -hd-to-address -hd-to-ec -hd-to-public -hd-to-wif -help -input-set -input-sign -input-validate -message-sign -message-validate -mnemonic-decode -mnemonic-encode -ripemd160 -satoshi-to-btc -script-decode -script-encode -script-to-address -seed -send-tx -send-tx-node -send-tx-p2p -settings -sha160 -sha256 -sha512 -stealth-decode -stealth-encode -stealth-public -stealth-secret -stealth-shared -tx-decode -tx-encode -uri-decode -uri-encode -validate-tx -watch-address -wif-to-ec -wif-to-public -wrap-decode -wrap-encode ----- - -For more information, see the https://github.com/libbitcoin/libbitcoin-explorer[Bitcoin Explorer homepage] and https://github.com/libbitcoin/libbitcoin-explorer/wiki[Bitcoin Explorer user documentation]. - -=== Examples of bx Command Use - -Let's look at some examples of using Bitcoin Explorer commands to experiment with keys and addresses. - -Generate a random "seed" value using the +seed+ command, which uses the operating system's random number generator. Pass the seed to the +ec-new+ command to generate a new private key. We save the standard output into the file _private_key_: - ----- -$ bx seed | bx ec-new > private_key -$ cat private_key -73096ed11ab9f1db6135857958ece7d73ea7c30862145bcc4bbc7649075de474 ----- - -Now, generate the public key from that private key using the +ec-to-public+ command. We pass the _private_key_ file into the standard input and save the standard output of the command into a new file _public_key_: - ----- -$ bx ec-to-public < private_key > public_key -$ cat public_key -02fca46a6006a62dfdd2dbb2149359d0d97a04f430f12a7626dd409256c12be500 ----- - -We can reformat the +public_key+ as an address using the +ec-to-address+ command. We pass the _public_key_ into standard input: - ----- -$ bx ec-to-address < public_key -17re1S4Q8ZHyCP8Kw7xQad1Lr6XUzWUnkG ----- - -Keys generated in this manner produce a type-0 nondeterministic wallet. That means that each key is generated from an independent seed. Bitcoin Explorer commands can also generate keys deterministically, in accordance with BIP-32. In this case, a "master" key is created from a seed and then extended deterministically to produce a tree of subkeys, resulting in a type-2 deterministic wallet. - -First, we use the +seed+ and +hd-new+ commands to generate a master key that will be used as the basis to derive a hierarchy of keys: - ----- -$ bx seed > seed -$ cat seed -eb68ee9f3df6bd4441a9feadec179ff1 - -$ bx hd-new < seed > master -$ cat master -xprv9s21ZrQH143K2BEhMYpNQoUvAgiEjArAVaZaCTgsaGe6LsAnwubeiTcDzd23mAoyizm9cApe51gNfLMkBqkYoWWMCRwzfuJk8RwF1SVEpAQ ----- - -We now use the +hd-private+ command to generate a hardened "account" key and a sequence of two private keys within the account: - ----- -$ bx hd-private --hard < master > account -$ cat account -xprv9vkDLt81dTKjwHB8fsVB5QK8cGnzveChzSrtCfvu3aMWvQaThp59ueufuyQ8Qi3qpjk4aKsbmbfxwcgS8PYbgoR2NWHeLyvg4DhoEE68A1n - -$ bx hd-private --index 0 < account -xprv9xHfb6w1vX9xgZyPNXVgAhPxSsEkeRcPHEUV5iJcVEsuUEACvR3NRY3fpGhcnBiDbvG4LgndirDsia1e9F3DWPkX7Tp1V1u97HKG1FJwUpU - -$ bx hd-private --index 1 < account -xprv9xHfb6w1vX9xjc8XbN4GN86jzNAZ6xHEqYxzbLB4fzHFd6VqCLPGRZFsdjsuMVERadbgDbziCRJru9n6tzEWrASVpEdrZrFidt1RDfn4yA3 ----- - -Next, we use the +hd-public+ command to generate the corresponding sequence of two public keys: - ----- -$ bx hd-public --index 0 < account -xpub6BH1zcTuktiFu43rUZ2gXqLgzu5F3tLEeTQ5t6iE3aQtM2VMTxMcyLN9fYHiGhGpQe9QQYmqL2eYPFJ3vezHz5wzaSW4FiGrseNDR4LKqTy - -$ bx hd-public --index 1 < account -xpub6BH1zcTuktiFx6CzhPbGjG3UYQ13WR16CmtbPiagEKpEVtpyjshWyMaMV1cn7nUPUkgQHPVXJVqsrA8xWbGQDhohEcDFTEYMvYzwRD7Juf8 ----- - -The public keys can also be derived from their corresponding private keys using the +hd-to-public+ command: - ----- -$ bx hd-private --index 0 < account | bx hd-to-public -xpub6BH1zcTuktiFu43rUZ2gXqLgzu5F3tLEeTQ5t6iE3aQtM2VMTxMcyLN9fYHiGhGpQe9QQYmqL2eYPFJ3vezHz5wzaSW4FiGrseNDR4LKqTy - -$ bx hd-private --index 1 < account | bx hd-to-public -xpub6BH1zcTuktiFx6CzhPbGjG3UYQ13WR16CmtbPiagEKpEVtpyjshWyMaMV1cn7nUPUkgQHPVXJVqsrA8xWbGQDhohEcDFTEYMvYzwRD7Juf8 ----- - -We can generate a practically limitless number of keys in a deterministic chain, all derived from a single seed. This technique is used in many wallet applications to generate keys that can be backed up and restored with a single seed value. This is easier than having to back up the wallet with all its randomly generated keys every time a new key is created. - -The seed can be encoded using the +mnemonic-encode+ command: - ----- -$ bx hd-mnemonic < seed > words -adore repeat vision worst especially veil inch woman cast recall dwell appreciate ----- - -The seed can then be decoded using the +mnemonic-decode+ command: - ----- -$ bx mnemonic-decode < words -eb68ee9f3df6bd4441a9feadec179ff1 ----- - -Mnemonic encoding can make the seed easier to record and even remember.(((range="endofrange", startref="BX18_1"))) diff --git a/appdx-pycoin.asciidoc b/appdx-pycoin.asciidoc deleted file mode 100644 index 0b0dfdf2..00000000 --- a/appdx-pycoin.asciidoc +++ /dev/null @@ -1,424 +0,0 @@ -[[appdx-pycoin]] -[appendix] -== pycoin, ku, and tx - - -((("pycoin library")))The Python library https://github.com/richardkiss/pycoin[+pycoin+], originally written and maintained by Richard Kiss, is a Python-based library that supports manipulation of bitcoin keys and transactions, even supporting the scripting language enough to properly deal with nonstandard transactions. - -The pycoin library supports both Python 2 (2.7.x) and Python 3 (3.3 and later) and comes with some handy command-line utilities, +ku+ and +tx+. - -=== Key Utility (KU) - -((("key utility (ku)", id="keyutil17")))The command-line utility +ku+ ("key utility") is a Swiss Army knife for manipulating keys. It supports BIP-32 keys, WIF, and addresses (bitcoin and alt coins). Following are some examples. - -Create a BIP-32 key using the default entropy sources of GPG and _/dev/random_: - - ----- -$ ku create - -input : create -network : Bitcoin -wallet key : xprv9s21ZrQH143K3LU5ctPZTBnb9kTjA5Su9DcWHvXJemiJBsY7VqXUG7hipgdWaU - m2nhnzdvxJf5KJo9vjP2nABX65c5sFsWsV8oXcbpehtJi -public version : xpub661MyMwAqRbcFpYYiuvZpKjKhnJDZYAkWSY76JvvD7FH4fsG3Nqiov2CfxzxY8 - DGcpfT56AMFeo8M8KPkFMfLUtvwjwb6WPv8rY65L2q8Hz -tree depth : 0 -fingerprint : 9d9c6092 -parent f'print : 00000000 -child index : 0 -chain code : 80574fb260edaa4905bc86c9a47d30c697c50047ed466c0d4a5167f6821e8f3c -private key : yes -secret exponent : 112471538590155650688604752840386134637231974546906847202389294096567806844862 - hex : f8a8a28b28a916e1043cc0aca52033a18a13cab1638d544006469bc171fddfbe -wif : L5Z54xi6qJusQT42JHA44mfPVZGjyb4XBRWfxAzUWwRiGx1kV4sP - uncompressed : 5KhoEavGNNH4GHKoy2Ptu4KfdNp4r56L5B5un8FP6RZnbsz5Nmb -public pair x : 76460638240546478364843397478278468101877117767873462127021560368290114016034 -public pair y : 59807879657469774102040120298272207730921291736633247737077406753676825777701 - x as hex : a90b3008792432060fa04365941e09a8e4adf928bdbdb9dad41131274e379322 - y as hex : 843a0f6ed9c0eb1962c74533795406914fe3f1957c5238951f4fe245a4fcd625 -y parity : odd -key pair as sec : 03a90b3008792432060fa04365941e09a8e4adf928bdbdb9dad41131274e379322 - uncompressed : 04a90b3008792432060fa04365941e09a8e4adf928bdbdb9dad41131274e379322 - 843a0f6ed9c0eb1962c74533795406914fe3f1957c5238951f4fe245a4fcd625 -hash160 : 9d9c609247174ae323acfc96c852753fe3c8819d - uncompressed : 8870d869800c9b91ce1eb460f4c60540f87c15d7 -Bitcoin address : 1FNNRQ5fSv1wBi5gyfVBs2rkNheMGt86sp - uncompressed : 1DSS5isnH4FsVaLVjeVXewVSpfqktdiQAM ----- - -Create a BIP-32 key from a passphrase: - -[WARNING] -==== -The passphrase in this example is way too easy to guess. -==== - ----- -$ ku P:foo - -input : P:foo -network : Bitcoin -wallet key : xprv9s21ZrQH143K31AgNK5pyVvW23gHnkBq2wh5aEk6g1s496M8ZMjxncCKZKgb5j - ZoY5eSJMJ2Vbyvi2hbmQnCuHBujZ2WXGTux1X2k9Krdtq -public version : xpub661MyMwAqRbcFVF9ULcqLdsEa5WnCCugQAcgNd9iEMQ31tgH6u4DLQWoQayvtS - VYFvXz2vPPpbXE1qpjoUFidhjFj82pVShWu9curWmb2zy -tree depth : 0 -fingerprint : 5d353a2e -parent f'print : 00000000 -child index : 0 -chain code : 5eeb1023fd6dd1ae52a005ce0e73420821e1d90e08be980a85e9111fd7646bbc -private key : yes -secret exponent : 65825730547097305716057160437970790220123864299761908948746835886007793998275 - hex : 91880b0e3017ba586b735fe7d04f1790f3c46b818a2151fb2def5f14dd2fd9c3 -wif : L26c3H6jEPVSqAr1usXUp9qtQJw6NHgApq6Ls4ncyqtsvcq2MwKH - uncompressed : 5JvNzA5vXDoKYJdw8SwwLHxUxaWvn9mDea6k1vRPCX7KLUVWa7W -public pair x : 81821982719381104061777349269130419024493616650993589394553404347774393168191 -public pair y : 58994218069605424278320703250689780154785099509277691723126325051200459038290 - x as hex : b4e599dfa44555a4ed38bcfff0071d5af676a86abf123c5b4b4e8e67a0b0b13f - y as hex : 826d8b4d3010aea16ff4c1c1d3ae68541d9a04df54a2c48cc241c2983544de52 -y parity : even -key pair as sec : 02b4e599dfa44555a4ed38bcfff0071d5af676a86abf123c5b4b4e8e67a0b0b13f - uncompressed : 04b4e599dfa44555a4ed38bcfff0071d5af676a86abf123c5b4b4e8e67a0b0b13f - 826d8b4d3010aea16ff4c1c1d3ae68541d9a04df54a2c48cc241c2983544de52 -hash160 : 5d353a2ecdb262477172852d57a3f11de0c19286 - uncompressed : e5bd3a7e6cb62b4c820e51200fb1c148d79e67da -Bitcoin address : 19Vqc8uLTfUonmxUEZac7fz1M5c5ZZbAii - uncompressed : 1MwkRkogzBRMehBntgcq2aJhXCXStJTXHT ----- - - -Get info as JSON: - - ----- -$ ku P:foo -P -j ----- -[source,json] ----- -{ - "y_parity": "even", - "public_pair_y_hex": "826d8b4d3010aea16ff4c1c1d3ae68541d9a04df54a2c48cc241c2983544de52", - "private_key": "no", - "parent_fingerprint": "00000000", - "tree_depth": "0", - "network": "Bitcoin", - "btc_address_uncompressed": "1MwkRkogzBRMehBntgcq2aJhXCXStJTXHT", - "key_pair_as_sec_uncompressed": "04b4e599dfa44555a4ed38bcfff0071d5af676a86abf123c5b4b4e8e67a0b0b13f826d8b4d3010aea16ff4c1c1d3ae68541d9a04df54a2c48cc241c2983544de52", - "public_pair_x_hex": "b4e599dfa44555a4ed38bcfff0071d5af676a86abf123c5b4b4e8e67a0b0b13f", - "wallet_key": "xpub661MyMwAqRbcFVF9ULcqLdsEa5WnCCugQAcgNd9iEMQ31tgH6u4DLQWoQayvtSVYFvXz2vPPpbXE1qpjoUFidhjFj82pVShWu9curWmb2zy", - "chain_code": "5eeb1023fd6dd1ae52a005ce0e73420821e1d90e08be980a85e9111fd7646bbc", - "child_index": "0", - "hash160_uncompressed": "e5bd3a7e6cb62b4c820e51200fb1c148d79e67da", - "btc_address": "19Vqc8uLTfUonmxUEZac7fz1M5c5ZZbAii", - "fingerprint": "5d353a2e", - "hash160": "5d353a2ecdb262477172852d57a3f11de0c19286", - "input": "P:foo", - "public_pair_x": "81821982719381104061777349269130419024493616650993589394553404347774393168191", - "public_pair_y": "58994218069605424278320703250689780154785099509277691723126325051200459038290", - "key_pair_as_sec": "02b4e599dfa44555a4ed38bcfff0071d5af676a86abf123c5b4b4e8e67a0b0b13f" -} ----- - -Public BIP32 key: - ----- -$ ku -w -P P:foo -xpub661MyMwAqRbcFVF9ULcqLdsEa5WnCCugQAcgNd9iEMQ31tgH6u4DLQWoQayvtSVYFvXz2vPPpbXE1qpjoUFidhjFj82pVShWu9curWmb2zy ----- - -Generate a subkey: - ----- -$ ku -w -s3/2 P:foo -xprv9wTErTSkjVyJa1v4cUTFMFkWMe5eu8ErbQcs9xajnsUzCBT7ykHAwdrxvG3g3f6BFk7ms5hHBvmbdutNmyg6iogWKxx6mefEw4M8EroLgKj ----- - -Hardened subkey: - ----- -$ ku -w -s3/2H P:foo -xprv9wTErTSu5AWGkDeUPmqBcbZWX1xq85ZNX9iQRQW9DXwygFp7iRGJo79dsVctcsCHsnZ3XU3DhsuaGZbDh8iDkBN45k67UKsJUXM1JfRCdn1 ----- - -WIF: - ----- -$ ku -W P:foo -L26c3H6jEPVSqAr1usXUp9qtQJw6NHgApq6Ls4ncyqtsvcq2MwKH ----- - -Address: - - ----- -$ ku -a P:foo -19Vqc8uLTfUonmxUEZac7fz1M5c5ZZbAii ----- - - - -Generate a bunch of subkeys: - - ----- -$ ku P:foo -s 0/0-5 -w -xprv9xWkBDfyBXmZjBG9EiXBpy67KK72fphUp9utJokEBFtjsjiuKUUDF5V3TU8U8cDzytqYnSekc8bYuJS8G3bhXxKWB89Ggn2dzLcoJsuEdRK -xprv9xWkBDfyBXmZnzKf3bAGifK593gT7WJZPnYAmvc77gUQVej5QHckc5Adtwxa28ACmANi9XhCrRvtFqQcUxt8rUgFz3souMiDdWxJDZnQxzx -xprv9xWkBDfyBXmZqdXA8y4SWqfBdy71gSW9sjx9JpCiJEiBwSMQyRxan6srXUPBtj3PTxQFkZJAiwoUpmvtrxKZu4zfsnr3pqyy2vthpkwuoVq -xprv9xWkBDfyBXmZsA85GyWj9uYPyoQv826YAadKWMaaEosNrFBKgj2TqWuiWY3zuqxYGpHfv9cnGj5P7e8EskpzKL1Y8Gk9aX6QbryA5raK73p -xprv9xWkBDfyBXmZv2q3N66hhZ8DAcEnQDnXML1J62krJAcf7Xb1HJwuW2VMJQrCofY2jtFXdiEY8UsRNJfqK6DAdyZXoMvtaLHyWQx3FS4A9zw -xprv9xWkBDfyBXmZw4jEYXUHYc9fT25k9irP87n2RqfJ5bqbjKdT84Mm7Wtc2xmzFuKg7iYf7XFHKkSsaYKWKJbR54bnyAD9GzjUYbAYTtN4ruo ----- - -[role="pagebreak-before"] -Generate the corresponding addresses: - - ----- -$ ku P:foo -s 0/0-5 -a -1MrjE78H1R1rqdFrmkjdHnPUdLCJALbv3x -1AnYyVEcuqeoVzH96zj1eYKwoWfwte2pxu -1GXr1kZfxE1FcK6ZRD5sqqqs5YfvuzA1Lb -116AXZc4bDVQrqmcinzu4aaPdrYqvuiBEK -1Cz2rTLjRM6pMnxPNrRKp9ZSvRtj5dDUML -1WstdwPnU6HEUPme1DQayN9nm6j7nDVEM ----- - - -Generate the corresponding WIFs: - - ----- -$ ku P:foo -s 0/0-5 -W -L5a4iE5k9gcJKGqX3FWmxzBYQc29PvZ6pgBaePLVqT5YByEnBomx -Kyjgne6GZwPGB6G6kJEhoPbmyjMP7D5d3zRbHVjwcq4iQXD9QqKQ -L4B3ygQxK6zH2NQGxLDee2H9v4Lvwg14cLJW7QwWPzCtKHdWMaQz -L2L2PZdorybUqkPjrmhem4Ax5EJvP7ijmxbNoQKnmTDMrqemY8UF -L2oD6vA4TUyqPF8QG4vhUFSgwCyuuvFZ3v8SKHYFDwkbM765Nrfd -KzChTbc3kZFxUSJ3Kt54cxsogeFAD9CCM4zGB22si8nfKcThQn8C ----- - - - -Check that it works by choosing a BIP32 string (the one corresponding to subkey 0/3): - - - ----- -$ ku -W xprv9xWkBDfyBXmZsA85GyWj9uYPyoQv826YAadKWMaaEosNrFBKgj2TqWuiWY3zuqxYGpHfv9cnGj5P7e8EskpzKL1Y8Gk9aX6QbryA5raK73p -L2L2PZdorybUqkPjrmhem4Ax5EJvP7ijmxbNoQKnmTDMrqemY8UF -$ ku -a xprv9xWkBDfyBXmZsA85GyWj9uYPyoQv826YAadKWMaaEosNrFBKgj2TqWuiWY3zuqxYGpHfv9cnGj5P7e8EskpzKL1Y8Gk9aX6QbryA5raK73p -116AXZc4bDVQrqmcinzu4aaPdrYqvuiBEK ----- - - -Yep, looks familiar. - -From secret exponent: - - ----- -$ ku 1 - -input : 1 -network : Bitcoin -secret exponent : 1 - hex : 1 -wif : KwDiBf89QgGbjEhKnhXJuH7LrciVrZi3qYjgd9M7rFU73sVHnoWn - uncompressed : 5HpHagT65TZzG1PH3CSu63k8DbpvD8s5ip4nEB3kEsreAnchuDf -public pair x : 55066263022277343669578718895168534326250603453777594175500187360389116729240 -public pair y : 32670510020758816978083085130507043184471273380659243275938904335757337482424 - x as hex : 79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - y as hex : 483ada7726a3c4655da4fbfc0e1108a8fd17b448a68554199c47d08ffb10d4b8 -y parity : even -key pair as sec : 0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - uncompressed : 0479be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - 483ada7726a3c4655da4fbfc0e1108a8fd17b448a68554199c47d08ffb10d4b8 -hash160 : 751e76e8199196d454941c45d1b3a323f1433bd6 - uncompressed : 91b24bf9f5288532960ac687abb035127b1d28a5 -Bitcoin address : 1BgGZ9tcN4rm9KBzDn7KprQz87SZ26SAMH - uncompressed : 1EHNa6Q4Jz2uvNExL497mE43ikXhwF6kZm ----- - - -Litecoin version: - - ----- -$ ku -nL 1 - -input : 1 -network : Litecoin -secret exponent : 1 - hex : 1 -wif : T33ydQRKp4FCW5LCLLUB7deioUMoveiwekdwUwyfRDeGZm76aUjV - uncompressed : 6u823ozcyt2rjPH8Z2ErsSXJB5PPQwK7VVTwwN4mxLBFrao69XQ -public pair x : 55066263022277343669578718895168534326250603453777594175500187360389116729240 -public pair y : 32670510020758816978083085130507043184471273380659243275938904335757337482424 - x as hex : 79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - y as hex : 483ada7726a3c4655da4fbfc0e1108a8fd17b448a68554199c47d08ffb10d4b8 -y parity : even -key pair as sec : 0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - uncompressed : 0479be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - 483ada7726a3c4655da4fbfc0e1108a8fd17b448a68554199c47d08ffb10d4b8 -hash160 : 751e76e8199196d454941c45d1b3a323f1433bd6 - uncompressed : 91b24bf9f5288532960ac687abb035127b1d28a5 -Litecoin address : LVuDpNCSSj6pQ7t9Pv6d6sUkLKoqDEVUnJ - uncompressed : LYWKqJhtPeGyBAw7WC8R3F7ovxtzAiubdM ----- - - -Dogecoin WIF: - - ----- -$ ku -nD -W 1 -QNcdLVw8fHkixm6NNyN6nVwxKek4u7qrioRbQmjxac5TVoTtZuot ----- - - -From public pair (on Testnet): - - ----- -$ ku -nT 55066263022277343669578718895168534326250603453777594175500187360389116729240,even - -input : 550662630222773436695787188951685343262506034537775941755001873603 - 89116729240,even -network : Bitcoin testnet -public pair x : 55066263022277343669578718895168534326250603453777594175500187360389116729240 -public pair y : 32670510020758816978083085130507043184471273380659243275938904335757337482424 - x as hex : 79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - y as hex : 483ada7726a3c4655da4fbfc0e1108a8fd17b448a68554199c47d08ffb10d4b8 -y parity : even -key pair as sec : 0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - uncompressed : 0479be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 - 483ada7726a3c4655da4fbfc0e1108a8fd17b448a68554199c47d08ffb10d4b8 -hash160 : 751e76e8199196d454941c45d1b3a323f1433bd6 - uncompressed : 91b24bf9f5288532960ac687abb035127b1d28a5 -Bitcoin testnet address : mrCDrCybB6J1vRfbwM5hemdJz73FwDBC8r - uncompressed : mtoKs9V381UAhUia3d7Vb9GNak8Qvmcsme ----- - - -From hash160: - - ----- -$ ku 751e76e8199196d454941c45d1b3a323f1433bd6 - -input : 751e76e8199196d454941c45d1b3a323f1433bd6 -network : Bitcoin -hash160 : 751e76e8199196d454941c45d1b3a323f1433bd6 -Bitcoin address : 1BgGZ9tcN4rm9KBzDn7KprQz87SZ26SAMH ----- - - -((("", startref="pycoin library")))As a Dogecoin address: - - ----- -$ ku -nD 751e76e8199196d454941c45d1b3a323f1433bd6 - -input : 751e76e8199196d454941c45d1b3a323f1433bd6 -network : Dogecoin -hash160 : 751e76e8199196d454941c45d1b3a323f1433bd6 -Dogecoin address : DFpN6QqFfUm3gKNaxN6tNcab1FArL9cZLE ----- - -==== Transaction Utility (TX) - -((("transaction utility (TX)", id="TX17"))) -The command-line utility +tx+ will display transactions in human-readable form, fetch base transactions from pycoin's transaction cache or from web services (blockchain.info, blockcypher.com, blockr.io and chain.so are currently supported), merge transactions, add or delete inputs or outputs, and sign transactions. - -Following are some examples. - - -View the famous "pizza" transaction: - - ----- -$ tx 49d2adb6e476fa46d8357babf78b1b501fd39e177ac7833124b3f67b17c40c2a -warning: consider setting environment variable PYCOIN_CACHE_DIR=~/.pycoin_cache to cache transactions fetched via web services -warning: no service providers found for get_tx; consider setting environment variable PYCOIN_BTC_PROVIDERS -usage: tx [-h] [-t TRANSACTION_VERSION] [-l LOCK_TIME] [-n NETWORK] [-a] - [-i address] [-f path-to-private-keys] [-g GPG_ARGUMENT] - [--remove-tx-in tx_in_index_to_delete] - [--remove-tx-out tx_out_index_to_delete] [-F transaction-fee] [-u] - [-b BITCOIND_URL] [-o path-to-output-file] - argument [argument ...] -tx: error: can't find Tx with id 49d2adb6e476fa46d8357babf78b1b501fd39e177ac7833124b3f67b17c40c2a ----- - - -Oops! We don't have web services set up. Let's do that now: - - -[source,bash] ----- -$ PYCOIN_CACHE_DIR=~/.pycoin_cache -$ PYCOIN_BTC_PROVIDERS="block.io blockchain.info blockexplorer.com" -$ export PYCOIN_CACHE_DIR PYCOIN_BTC_PROVIDERS ----- - - -It's not done automatically so a command-line tool won't leak potentially private information about what transactions you're interested in to a third-party website. If you don't care, you could put these lines into your _.profile_. - -Let's try again: - ----- -$ tx 49d2adb6e476fa46d8357babf78b1b501fd39e177ac7833124b3f67b17c40c2a -Version: 1 tx hash 49d2adb6e476fa46d8357babf78b1b501fd39e177ac7833124b3f67b17c40c2a 159 bytes -TxIn count: 1; TxOut count: 1 -Lock time: 0 (valid anytime) -Input: - 0: (unknown) from 1e133f7de73ac7d074e2746a3d6717dfc99ecaa8e9f9fade2cb8b0b20a5e0441:0 -Output: - 0: 1CZDM6oTttND6WPdt3D6bydo7DYKzd9Qik receives 10000000.00000 mBTC -Total output 10000000.00000 mBTC -including unspents in hex dump since transaction not fully signed -010000000141045e0ab2b0b82cdefaf9e9a8ca9ec9df17673d6a74e274d0c73ae77d3f131e000000004a493046022100a7f26eda874931999c90f87f01ff1ffc76bcd058fe16137e0e63fdb6a35c2d78022100a61e9199238eb73f07c8f209504c84b80f03e30ed8169edd44f80ed17ddf451901ffffffff010010a5d4e80000001976a9147ec1003336542cae8bded8909cdd6b5e48ba0ab688ac00000000 - -** can't validate transaction as source transactions missing ----- - -The final line appears because to validate the transactions' signatures, you technically need the source transactions. So let's add +-a+ to augment the transactions with source information: - ----- -$ tx -a 49d2adb6e476fa46d8357babf78b1b501fd39e177ac7833124b3f67b17c40c2a -warning: transaction fees recommendations casually calculated and estimates may be incorrect -warning: transaction fee lower than (casually calculated) expected value of 0.1 mBTC, transaction might not propagate -Version: 1 tx hash 49d2adb6e476fa46d8357babf78b1b501fd39e177ac7833124b3f67b17c40c2a 159 bytes -TxIn count: 1; TxOut count: 1 -Lock time: 0 (valid anytime) -Input: - 0: 17WFx2GQZUmh6Up2NDNCEDk3deYomdNCfk from 1e133f7de73ac7d074e2746a3d6717dfc99ecaa8e9f9fade2cb8b0b20a5e0441:0 10000000.00000 mBTC sig ok -Output: - 0: 1CZDM6oTttND6WPdt3D6bydo7DYKzd9Qik receives 10000000.00000 mBTC -Total input 10000000.00000 mBTC -Total output 10000000.00000 mBTC -Total fees 0.00000 mBTC - -010000000141045e0ab2b0b82cdefaf9e9a8ca9ec9df17673d6a74e274d0c73ae77d3f131e000000004a493046022100a7f26eda874931999c90f87f01ff1ffc76bcd058fe16137e0e63fdb6a35c2d78022100a61e9199238eb73f07c8f209504c84b80f03e30ed8169edd44f80ed17ddf451901ffffffff010010a5d4e80000001976a9147ec1003336542cae8bded8909cdd6b5e48ba0ab688ac00000000 - -all incoming transaction values validated ----- - -((("", startref="TX17")))Now, let's look at unspent outputs for a specific address (UTXO). In block #1, we see a coinbase transaction to +12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX+. Let's use +fetch_unspent+ to find all coins in this address: - ----- -$ fetch_unspent 12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX -a3a6f902a51a2cbebede144e48a88c05e608c2cce28024041a5b9874013a1e2a/0/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/333000 -cea36d008badf5c7866894b191d3239de9582d89b6b452b596f1f1b76347f8cb/31/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/10000 -065ef6b1463f552f675622a5d1fd2c08d6324b4402049f68e767a719e2049e8d/86/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/10000 -a66dddd42f9f2491d3c336ce5527d45cc5c2163aaed3158f81dc054447f447a2/0/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/10000 -ffd901679de65d4398de90cefe68d2c3ef073c41f7e8dbec2fb5cd75fe71dfe7/0/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/100 -d658ab87cc053b8dbcfd4aa2717fd23cc3edfe90ec75351fadd6a0f7993b461d/5/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/911 -36ebe0ca3237002acb12e1474a3859bde0ac84b419ec4ae373e63363ebef731c/1/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/100000 -fd87f9adebb17f4ebb1673da76ff48ad29e64b7afa02fda0f2c14e43d220fe24/0/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/1 -dfdf0b375a987f17056e5e919ee6eadd87dad36c09c4016d4a03cea15e5c05e3/1/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/1337 -cb2679bfd0a557b2dc0d8a6116822f3fcbe281ca3f3e18d3855aa7ea378fa373/0/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/1337 -d6be34ccf6edddc3cf69842dce99fe503bf632ba2c2adb0f95c63f6706ae0c52/1/76a914119b098e2e980a229e139a9ed01a469e518e6f2688ac/2000000 - 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098/0/410496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52da7589379515d4e0a604f8141781e62294721166bf621e73a82cbf2342c858eeac/5000000000 ----- diff --git a/appdx-scriptops.asciidoc b/appdx-scriptops.asciidoc deleted file mode 100644 index 2ad1ef56..00000000 --- a/appdx-scriptops.asciidoc +++ /dev/null @@ -1,200 +0,0 @@ -[[tx_script_ops]] -[appendix] -== Transaction Script Language Operators, Constants, and Symbols - -[NOTE] -==== -Tables and descriptions sourced from https://en.bitcoin.it/wiki/Script[]. -==== - -<> shows operators for pushing values onto the stack.((("transactions", "scripts and Script language", id="TRlang14")))((("scripting", "Script language operators, constants, and symbols", id="SCRlang14"))) - -[[tx_script_ops_table_pushdata]] -.Push value onto stack -[options="header"] -|======= -| Symbol | Value (hex) | Description -| OP_0 or OP_FALSE | 0x00 | An empty array is pushed onto the stack -| 1 to 75 | 0x01 to 0x4b | Push the next N bytes onto the stack, where N is 1 to 75 bytes -| OP_PUSHDATA1 | 0x4c | The next script byte contains N, push the following N bytes onto the stack -| OP_PUSHDATA2 | 0x4d | The next two script bytes contain N, push the following N bytes onto the stack -| OP_PUSHDATA4 | 0x4e | The next four script bytes contain N, push the following N bytes onto the stack -| OP_1NEGATE | 0x4f | Push the value "–1" onto the stack -| OP_RESERVED | 0x50 | Halt—Invalid transaction unless found in an unexecuted OP_IF clause -| OP_1 or OP_TRUE| 0x51 | Push the value "1" onto the stack -| OP_2 to OP_16 | 0x52 to 0x60 | For OP_N, push the value "N" onto the stack, e.g., OP_2 pushes "2" -|======= - -[role="pagebreak-before"] -<> shows conditional flow control operators. - -[[tx_script_ops_table_control]] -.Conditional flow control -[options="header"] -|======= -| Symbol | Value (hex) | Description -| OP_NOP | 0x61 | Do nothing -| OP_VER | 0x62 | Halt—Invalid transaction unless found in an unexecuted OP_IF clause -| OP_IF | 0x63 | Execute the statements following if top of stack is not 0 -| OP_NOTIF | 0x64 | Execute the statements following if top of stack is 0 -| OP_VERIF | 0x65 | Halt—Invalid transaction -| OP_VERNOTIF | 0x66 | Halt—Invalid transaction -| OP_ELSE | 0x67 | Execute only if the previous statements were not executed -| OP_ENDIF | 0x68 | End the OP_IF, OP_NOTIF, OP_ELSE block -| OP_VERIFY | 0x69 | Check the top of the stack, halt and invalidate transaction if not TRUE -| OP_RETURN | 0x6a | Halt and invalidate transaction -|======= - -<> shows operators used for timelocks. - -[[tx_script_ops_table_timelock]] -.Timelock operations -[options="header"] -|======= -| Symbol | Value (hex) | Description -| OP_CHECKLOCKTIMEVERIFY (previously OP_NOP2) | 0xb1 | Marks transaction as invalid if the top stack item is greater than the transaction's nLockTime field, otherwise script evaluation continues as though an OP_NOP was executed. Transaction is also invalid if 1. the stack is empty; or 2. the top stack item is negative; or 3. the top stack item is greater than or equal to 500000000 while the transaction's nLockTime field is less than 500000000, or vice versa; or 4. the input's nSequence field is equal to 0xffffffff. The precise semantics are described in BIP-65 -| OP_CHECKSEQUENCEVERIFY (previously OP_NOP3) | 0xb2 | Marks transaction as invalid if the relative lock time of the input (enforced by BIP 0068 with nSequence) is not equal to or longer than the value of the top stack item. The precise semantics are described in BIP-112| -|======= - -<> shows operators used to manipulate the stack. - -[[tx_script_ops_table_stack]] -.Stack operations -[options="header"] -|======= -| Symbol | Value (hex) | Description -| OP_TOALTSTACK | 0x6b | Pop top item from stack and push to alternative stack -| OP_FROMALTSTACK | 0x6c | Pop top item from alternative stack and push to stack -| OP_2DROP | 0x6d | Pop top two stack items -| OP_2DUP | 0x6e | Duplicate top two stack items -| OP_3DUP | 0x6f | Duplicate top three stack items -| OP_2OVER | 0x70 | Copy the third and fourth items in the stack to the top -| OP_2ROT | 0x71 | Move the fifth and sixth items in the stack to the top -| OP_2SWAP | 0x72 | Swap the two top pairs of items in the stack -| OP_IFDUP | 0x73 | Duplicate the top item in the stack if it is not 0 -| OP_DEPTH | 0x74 | Count the items on the stack and push the resulting count -| OP_DROP | 0x75 | Pop the top item in the stack -| OP_DUP | 0x76 | Duplicate the top item in the stack -| OP_NIP | 0x77 | Pop the second item in the stack -| OP_OVER | 0x78 | Copy the second item in the stack and push it onto the top -| OP_PICK | 0x79 | Pop value N from top, then copy the Nth item to the top of the stack -| OP_ROLL | 0x7a | Pop value N from top, then move the Nth item to the top of the stack -| OP_ROT | 0x7b | Rotate the top three items in the stack -| OP_SWAP | 0x7c | Swap the top two items on the stack -| OP_TUCK | 0x7d | Copy the top item and insert it between the top and second item -|======= - -<> shows string operators. - -[[tx_script_ops_table_splice]] -.String splice operations -[options="header"] -|======= -| Symbol | Value (hex) | Description -| _OP_CAT_ | 0x7e | Disabled (concatenates top two items) -| _OP_SUBSTR_ | 0x7f | Disabled (returns substring) -| _OP_LEFT_ | 0x80 | Disabled (returns left substring) -| _OP_RIGHT_ | 0x81 | Disabled (returns right substring) -| OP_SIZE | 0x82 | Calculate string length of top item and push the result -|======= - -<> shows binary arithmetic and boolean logic operators. - -[[tx_script_ops_table_binmath]] -.Binary arithmetic and conditionals -[options="header"] -|======= -| Symbol | Value (hex) | Description -| _OP_INVERT_ | 0x83 | Disabled (Flip the bits of the top item) -| _OP_AND_ | 0x84 | Disabled (Boolean AND of two top items) -| _OP_OR_ | 0x85 | Disabled (Boolean OR of two top items) -| _OP_XOR_ | 0x86 | Disabled (Boolean XOR of two top items) -| OP_EQUAL | 0x87 | Push TRUE (1) if top two items are exactly equal, push FALSE (0) otherwise -| OP_EQUALVERIFY | 0x88 | Same as OP_EQUAL, but run OP_VERIFY after to halt if not TRUE -| OP_RESERVED1 | 0x89 | Halt—Invalid transaction unless found in an unexecuted OP_IF clause -| OP_RESERVED2 | 0x8a | Halt—Invalid transaction unless found in an unexecuted OP_IF clause -|======= - -[role="pagebreak-before"] -<> shows numeric (arithmetic) operators. - -[[tx_script_ops_table_numbers]] -.Numeric operators -[options="header"] -|======= -| Symbol | Value (hex) | Description -| OP_1ADD | 0x8b | Add 1 to the top item -| OP_1SUB | 0x8c | Subtract 1 from the top item -| _OP_2MUL_ | 0x8d | Disabled (multiply top item by 2) -| _OP_2DIV_ | 0x8e | Disabled (divide top item by 2) -| OP_NEGATE | 0x8f | Flip the sign of top item -| OP_ABS | 0x90 | Change the sign of the top item to positive -| OP_NOT | 0x91 | If top item is 0 or 1 Boolean flip it, otherwise return 0 -| OP_0NOTEQUAL | 0x92 | If top item is 0 return 0, otherwise return 1 -| OP_ADD | 0x93 | Pop top two items, add them and push result -| OP_SUB | 0x94 | Pop top two items, subtract first from second, push result -| _OP_MUL_ | 0x95 | Disabled (multiply top two items) -| _OP_DIV_ | 0x96 | Disabled (divide second item by first item) -| _OP_MOD_ | 0x97 | Disabled (remainder divide second item by first item) -| _OP_LSHIFT_ | 0x98 | Disabled (shift second item left by first item number of bits) -| _OP_RSHIFT_ | 0x99 | Disabled (shift second item right by first item number of bits) -| OP_BOOLAND | 0x9a | Boolean AND of top two items -| OP_BOOLOR | 0x9b | Boolean OR of top two items -| OP_NUMEQUAL | 0x9c | Return TRUE if top two items are equal numbers -| OP_NUMEQUALVERIFY | 0x9d | Same as NUMEQUAL, then OP_VERIFY to halt if not TRUE -| OP_NUMNOTEQUAL | 0x9e | Return TRUE if top two items are not equal numbers -| OP_LESSTHAN | 0x9f | Return TRUE if second item is less than top item -| OP_GREATERTHAN | 0xa0 | Return TRUE if second item is greater than top item -| OP_LESSTHANOREQUAL | 0xa1 | Return TRUE if second item is less than or equal to top item -| OP_GREATERTHANOREQUAL | 0xa2 | Return TRUE if second item is great than or equal to top item -| OP_MIN | 0xa3 | Return the smaller of the two top items -| OP_MAX | 0xa4 | Return the larger of the two top items -| OP_WITHIN | 0xa5 | Return TRUE if the third item is between the second item (or equal) and first item -|======= - -[role="pagebreak-before"] -<> shows cryptographic function operators. - -[[tx_script_ops_table_crypto]] -.Cryptographic and hashing operations -[options="header"] -|======= -| Symbol | Value (hex) | Description -| OP_RIPEMD160 | 0xa6 | Return RIPEMD160 hash of top item -| OP_SHA1 | 0xa7 | Return SHA1 hash of top item -| OP_SHA256 | 0xa8 | Return SHA256 hash of top item -| OP_HASH160 | 0xa9 | Return RIPEMD160(SHA256(x)) hash of top item -| OP_HASH256 | 0xaa | Return SHA256(SHA256(x)) hash of top item -| OP_CODESEPARATOR | 0xab | Mark the beginning of signature-checked data -| OP_CHECKSIG | 0xac | Pop a public key and signature and validate the signature for the transaction's hashed data, return TRUE if matching -| OP_CHECKSIGVERIFY | 0xad | Same as CHECKSIG, then OP_VERIFY to halt if not TRUE -| OP_CHECKMULTISIG | 0xae | Run CHECKSIG for each pair of signature and public key provided. All must match. Bug in implementation pops an extra value, prefix with OP_0 as workaround -| OP_CHECKMULTISIGVERIFY | 0xaf | Same as CHECKMULTISIG, then OP_VERIFY to halt if not TRUE -|======= - -<> shows nonoperator symbols. - -[[tx_script_ops_table_nop]] -.Nonoperators -[options="header"] -|======= -| Symbol | Value (hex) | Description -| OP_NOP1 to OP_NOP10 | 0xb0 to 0xb9 | Does nothing, ignored -|======= - - -<> shows operator codes reserved for use by the internal script parser.((("", startref="TRlang14")))((("", startref="SCRlang14"))) - -[[tx_script_ops_table_internal]] -.Reserved OP codes for internal use by the parser -[options="header"] -|======= -| Symbol | Value (hex) | Description -| OP_SMALLDATA | 0xf9 | Represents small data field -| OP_SMALLINTEGER | 0xfa | Represents small integer data field -| OP_PUBKEYS | 0xfb | Represents public key fields -| OP_PUBKEYHASH | 0xfd | Represents a public key hash field -| OP_PUBKEY | 0xfe | Represents a public key field -| OP_INVALIDOPCODE | 0xff | Represents any OP code not currently assigned -|======= - diff --git a/atlas.json b/atlas.json index 7d416659..6b61972e 100644 --- a/atlas.json +++ b/atlas.json @@ -2,30 +2,29 @@ "branch": "master", "files": [ "cover.html", + "praise.html", "titlepage.html", "copyright.html", "dedication.html", "toc.html", "preface.asciidoc", - "glossary.asciidoc", - "ch01.asciidoc", - "ch02.asciidoc", - "ch03.asciidoc", - "ch04.asciidoc", - "ch05.asciidoc", - "ch06.asciidoc", - "ch07.asciidoc", - "ch08.asciidoc", - "ch09.asciidoc", - "ch10.asciidoc", - "ch11.asciidoc", - "ch12.asciidoc", - "appdx-bitcoinwhitepaper.asciidoc", - "appdx-scriptops.asciidoc", - "appdx-bips.asciidoc", - "appdx-bitcore.asciidoc", - "appdx-pycoin.asciidoc", - "appdx-bx.asciidoc", + "ch01_intro.adoc", + "ch02_overview.adoc", + "ch03_bitcoin-core.adoc", + "ch04_keys.adoc", + "ch05_wallets.adoc", + "ch06_transactions.adoc", + "ch07_authorization-authentication.adoc", + "ch08_signatures.adoc", + "ch09_fees.adoc", + "ch10_network.adoc", + "ch11_blockchain.adoc", + "ch12_mining.adoc", + "ch13_security.adoc", + "ch14_applications.adoc", + "appa_whitepaper.adoc", + "appb_errata.adoc", + "appc_bips.adoc", "ix.html", "author_bio.html", "colo.html" @@ -35,7 +34,7 @@ "version": "print", "toc": true, "index": true, - "antennahouse_version": "AHFormatterV62_64-MR4", + "antennahouse_version": "AHFormatterV71_64-MR2", "syntaxhighlighting": true, "show_comments": false, "color_count": "1", @@ -47,7 +46,8 @@ "epubcheck": true, "syntaxhighlighting": true, "show_comments": false, - "downsample_images": false + "downsample_images": true, + "mathmlreplacement": false }, "mobi": { "index": true, @@ -66,8 +66,9 @@ }, "theme": "oreillymedia/animal_theme_sass", "title": "Mastering Bitcoin", - "print_isbn13": "9781491954386", + "print_isbn13": "9781098150099", "templating": false, "lang": "en", - "accent_color": "cmyk(26%, 89%, 0%, 0%)" + "accent_color": "", + "preprocessing": "none" } \ No newline at end of file diff --git a/author_bio.html b/author_bio.html index 212dd998..a2161696 100644 --- a/author_bio.html +++ b/author_bio.html @@ -1,10 +1,13 @@ -
-

About the Author

-

Andreas M. Antonopoulos is a noted technologist and serial entrepreneur who has become one of the most well-known and well-respected figures in bitcoin. As an engaging public speaker, teacher, and writer, Andreas makes complex subjects accessible and easy to understand. As an advisor, he helps startups recognize, evaluate, and navigate security and business risks.

+
+

About the Authors

-

Andreas grew up with the internet, starting his first company, an early BBS and proto-ISP, as a teenager in his home in Greece. He earned degrees in computer science, data communications, and distributed systems from University College London (UCL)—recently ranked among the world’s top 10 universities. After moving to the United States, Andreas cofounded and managed a successful technology research company, and in that role advised dozens of Fortune 500 company executives on networking, security, data centers, and cloud computing. More than 200 of his articles on security, cloud computing, and data centers have been published in print and syndicated worldwide. He holds two patents in networking and security.

+

Andreas M. Antonopoulos is a noted technologist and serial entrepreneur who has become one of the most well-known and well-respected figures in Bitcoin. As an engaging public speaker, teacher, and writer, Andreas makes complex subjects accessible and easy to understand. As an advisor, he helps startups recognize, evaluate, and navigate security and business risks.

-

In 1990, Andreas started teaching various IT topics in private, professional, and academic environments. He honed his speaking skills in front of audiences ranging in size from five executives in a boardroom to thousands of people in large conferences. With more than 400 speaking engagements under his belt he is considered a world-class and charismatic public speaker and teacher. In 2014, he was appointed as a teaching fellow with the University of Nicosia, the first university in the world to offer a masters degree in digital currency. In this role, he helped develop the curriculum and cotaught the Introduction to Digital Currencies course, offered as a massive open online course (MOOC) through the university.

+

Andreas grew up with the internet, starting his first company, an early BBS and proto-ISP, as a teenager in his home in Greece. He earned degrees in computer science, data communications, and distributed systems from University College London (UCL)—recently ranked among the world’s top 10 universities. After moving to the United States, Andreas cofounded and managed a successful technology research company, and in that role advised dozens of Fortune 500 company executives on networking, security, data centers, and cloud computing. More than two hundred of his articles on security, cloud computing, and data centers have been published in print and syndicated worldwide. He holds two patents in networking and security.

-

As a bitcoin entrepreneur, Andreas has founded a number of bitcoin businesses and launched several community open source projects. He serves as an advisor to several bitcoin and cryptocurrency companies. He is a widely published author of articles and blog posts on bitcoin, a permanent host on the popular Let’s Talk Bitcoin podcast, and a frequent speaker at technology and security conferences worldwide.

+

In 1990, Andreas started teaching various IT topics in private, professional, and academic environments. He honed his speaking skills in front of audiences ranging in size from five executives in a boardroom to thousands of people in large conferences. With more than four hundred speaking engagements under his belt, he is considered a world-class and charismatic public speaker and teacher. In 2014, he was appointed as a teaching fellow with the University of Nicosia, the first university in the world to offer a master's degree in digital currency. In this role, he helped develop the curriculum and cotaught the "Introduction to Digital Currencies" course, offered as a massive open online course (MOOC) through the university.

+ +

As a Bitcoin entrepreneur, Andreas has founded a number of Bitcoin businesses and launched several community open source projects. He serves as an advisor to several Bitcoin and cryptocurrency companies. He is a widely published author of articles and blog posts on Bitcoin, a permanent host on the popular Let’s Talk Bitcoin podcast, and a frequent speaker at technology and security conferences worldwide.

+ +

David A. Harding is a technical writer focused on creating documentation for open source software. He is the coauthor of the Bitcoin Optech weekly newsletter (2018–2023), 21.co Bitcoin Computer tutorials (2015–2017), and Bitcoin.org developer documentation (2014–2015). He is also a Brink.dev grant committee member (2022–2023) and former board member (2020–2022).

diff --git a/book.adoc b/book.adoc new file mode 100644 index 00000000..5cbdcd9e --- /dev/null +++ b/book.adoc @@ -0,0 +1,51 @@ += Mastering Bitcoin + +include::preface.asciidoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::intro.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::overview.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::bitcoin-core.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::keys.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::wallets.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::transactions.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::authorization-authentication.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::signatures.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::fees.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::network.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::blockchain.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::mining.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::security.adoc[] + +//IMAGE_REPORT:NEW_CHAPTER +include::applications.adoc[] + +include::whitepaper.adoc[] + +include::errata.adoc[] + +include::bips.adoc[] diff --git a/book.asciidoc b/book.asciidoc deleted file mode 100644 index ef27e03e..00000000 --- a/book.asciidoc +++ /dev/null @@ -1,45 +0,0 @@ -= Mastering Bitcoin - -include::preface.asciidoc[] - -include::glossary.asciidoc[] - -include::ch01.asciidoc[] - -include::ch02.asciidoc[] - -include::ch03.asciidoc[] - -include::ch04.asciidoc[] - -include::ch05.asciidoc[] - -include::ch06.asciidoc[] - -include::ch07.asciidoc[] - -include::ch08.asciidoc[] - -include::ch09.asciidoc[] - -include::ch10.asciidoc[] - -include::ch11.asciidoc[] - -include::ch12.asciidoc[] - -include::appdx-bitcoinwhitepaper.asciidoc[] - -include::appdx-scriptops.asciidoc[] - -include::appdx-bips.asciidoc[] - -include::appdx-bitcore.asciidoc[] - -include::appdx-pycoin.asciidoc[] - -include::appdx-bx.asciidoc[] - -include::ix.html[] - -include::colo.html[] diff --git a/ch01.asciidoc b/ch01.asciidoc deleted file mode 100644 index e569cb68..00000000 --- a/ch01.asciidoc +++ /dev/null @@ -1,246 +0,0 @@ -[role="pagenumrestart"] -[[ch01_intro_what_is_bitcoin]] -== Introduction - -=== What Is Bitcoin? - -((("bitcoin", "defined", id="GSdefine01")))Bitcoin is a collection of concepts and technologies that form the basis of a digital money ecosystem. Units of currency called bitcoin are used to store and transmit value among participants in the Bitcoin network. Bitcoin users communicate with each other using the Bitcoin protocol primarily via the internet, although other transport networks can also be used. The Bitcoin protocol stack, available as open source software, can be run on a wide range of computing devices, including laptops and smartphones, making the technology easily accessible. - -[TIP] -==== -Notice how the unit of currency is called "bitcoin" with a small _b_, and the system is called "Bitcoin", with a capital _B_. -==== - -Users can transfer bitcoin over the network to do just about anything that can be done with conventional currencies, including buying and selling goods, sending money to people or organizations, or extending credit. Bitcoin can be purchased, sold, and exchanged for other currencies at specialized currency exchanges. Bitcoin in a sense is the perfect form of money for the internet because it is fast, secure, and borderless. - -Unlike traditional currencies, bitcoin is entirely virtual. There are no physical coins or even digital coins per se. The coins are implied in transactions that transfer value from sender to recipient. Users of Bitcoin own keys that allow them to prove ownership of bitcoin in the Bitcoin network. With these keys, they can sign transactions to unlock the value and spend it by transferring it to a new owner. Keys are often stored in a digital wallet on each user’s computer or smartphone. Possession of the key that can sign a transaction is the only prerequisite to spending bitcoin, putting the control entirely in the hands of each user. - -Bitcoin is a distributed, peer-to-peer system. As such, there is no "central" server or point of control. Bitcoins, i.e. units of bitcoin, are created through a process called "mining," which involves competing to find solutions to a mathematical problem while processing Bitcoin transactions. Any participant in the Bitcoin network (i.e., anyone using a device running the full Bitcoin protocol stack) may operate as a miner, using their computer's processing power to verify and record transactions. Every 10 minutes, on average, a Bitcoin miner can validate the transactions of the past 10 minutes and is rewarded with brand new bitcoin. Essentially, Bitcoin mining decentralizes the currency-issuance and clearing functions of a central bank and replaces the need for any central bank. - -The Bitcoin protocol includes built-in algorithms that regulate the mining function across the network. The difficulty of the processing task that miners must perform is adjusted dynamically so that, on average, someone succeeds every 10 minutes regardless of how many miners (and how much processing) are competing at any moment. The protocol also halves the rate at which new bitcoin is created every 4 years, and limits the total number of bitcoin that will be created to a fixed total just below 21 million coins. The result is that the number of bitcoin in circulation closely follows an easily predictable curve that approaches 21 million by the year 2140. Due to bitcoin's diminishing rate of issuance, over the long term, the Bitcoin currency is deflationary. Furthermore, bitcoin cannot be inflated by "printing" new money above and beyond the expected issuance rate. - -Behind the scenes, Bitcoin is also the name of the protocol, a peer-to-peer network, and a distributed computing innovation. The bitcoin currency is really only the first application of this invention. Bitcoin represents the culmination of decades of research in cryptography and distributed systems and includes four key innovations brought together in a unique and powerful combination. Bitcoin consists of: - -* A decentralized peer-to-peer network (the Bitcoin protocol) -* A public transaction ledger (the blockchain) -* ((("mining and consensus", "consensus rules", "satisfying")))A set of rules for independent transaction validation and currency issuance (consensus rules) -* A mechanism for reaching global decentralized consensus on the valid blockchain (Proof-of-Work algorithm) - -As a developer, I see Bitcoin as akin to the internet of money, a network for propagating value and securing the ownership of digital assets via distributed computation. There's a lot more to Bitcoin than first meets the eye. - -In this chapter we'll get started by explaining some of the main concepts and terms, getting the necessary software, and using Bitcoin for simple transactions. In the following chapters, we'll start unwrapping the layers of technology that make Bitcoin possible and examine the inner workings of the Bitcoin network and protocol.((("", startref="GSdefine01"))) - -[role="pagebreak-before less_space"] -.Digital Currencies Before Bitcoin -**** - -((("digital currencies", "prior to bitcoin")))The emergence of viable digital money is closely linked to developments in cryptography. This is not surprising when one considers the fundamental challenges involved with using bits to represent value that can be exchanged for goods and services. Three basic questions for anyone accepting digital money are: - -1. Can I trust that the money is authentic and not counterfeit? -2. Can I trust that the digital money can only be spent once (known as the “double-spend” problem)? -3. Can I be sure that no one else can claim this money belongs to them and not me? - -Issuers of paper money are constantly battling the counterfeiting problem by using increasingly sophisticated papers and printing technology. Physical money addresses the double-spend issue easily because the same paper note cannot be in two places at once. Of course, conventional money is also often stored and transmitted digitally. In these cases, the counterfeiting and double-spend issues are handled by clearing all electronic transactions through central authorities that have a global view of the currency in circulation. For digital money, which cannot take advantage of esoteric inks or holographic strips, cryptography provides the basis for trusting the legitimacy of a user’s claim to value. Specifically, cryptographic digital signatures enable a user to sign a digital asset or transaction proving the ownership of that asset. With the appropriate architecture, digital signatures also can be used to address the double-spend issue. - -When cryptography started becoming more broadly available and understood in the late 1980s, many researchers began trying to use cryptography to build digital currencies. These early digital currency projects issued digital money, usually backed by a national currency or precious metal such as gold. - -((("decentralized systems", "vs. centralized", secondary-sortas="centralized")))Although these earlier digital currencies worked, they were centralized and, as a result, were easy to attack by governments and hackers. Early digital currencies used a central clearinghouse to settle all transactions at regular intervals, just like a traditional banking system. Unfortunately, in most cases, these nascent digital currencies were targeted by worried governments and eventually litigated out of existence. Some failed in spectacular crashes when the parent company liquidated abruptly. To be robust against intervention by antagonists, whether legitimate governments or criminal elements, a _decentralized_ digital currency was needed to avoid a single point of attack. Bitcoin is such a system, decentralized by design, and free of any central authority or point of control that can be attacked or corrupted. - -**** - -=== History of Bitcoin - -((("Nakamoto, Satoshi")))((("distributed computing")))((("bitcoin", "history of")))Bitcoin was invented in 2008 with the publication of a paper titled "Bitcoin: A Peer-to-Peer Electronic Cash System,"footnote:["Bitcoin: A Peer-to-Peer Electronic Cash System," Satoshi Nakamoto (https://bitcoin.org/bitcoin.pdf).] written under the alias of Satoshi Nakamoto (see <>). Nakamoto combined several prior inventions such as b-money and HashCash to create a completely decentralized electronic cash system that does not rely on a central authority for currency issuance or settlement and validation of transactions. ((("Proof-of-Work algorithm")))((("decentralized systems", "consensus in")))((("mining and consensus", "Proof-of-Work algorithm")))The key innovation was to use a distributed computation system (called a "Proof-of-Work" algorithm) to conduct a global "election" every 10 minutes, allowing the decentralized network to arrive at _consensus_ about the state of transactions. ((("double-spend problem")))((("spending bitcoin", "double-spend problem")))This elegantly solves the issue of double-spend where a single currency unit can be spent twice. Previously, the double-spend problem was a weakness of digital currency and was addressed by clearing all transactions through a central clearinghouse. - -The Bitcoin network started in 2009, based on a reference implementation published by Nakamoto and since revised by many other programmers. The implementation of the Proof-of-Work algorithm (mining) that provides security and resilience for Bitcoin has increased in power exponentially, and now exceeds the combined processing power of the world's top supercomputers. Bitcoin's total market value has at times exceeded $1 trillion US dollars, depending on the bitcoin-to-dollar exchange rate. The largest transaction processed so far by the network was $1.1 billion US dollars, transmitted instantly and processed for a fee of only $0.68. - -Satoshi Nakamoto withdrew from the public in April 2011, leaving the responsibility of developing the code and network to a thriving group of volunteers. The identity of the person or people behind Bitcoin is still unknown. ((("open source licenses")))However, neither Satoshi Nakamoto nor anyone else exerts individual control over the Bitcoin system, which operates based on fully transparent mathematical principles, open source code, and consensus among participants. The invention itself is groundbreaking and has already spawned new science in the fields of distributed computing, economics, and econometrics. - - -.A Solution to a Distributed Computing Problem -**** -((("Byzantine Generals' Problem")))Satoshi Nakamoto's invention is also a practical and novel solution to a problem in distributed computing, known as the "Byzantine Generals' Problem." Briefly, the problem consists of trying to agree on a course of action or the state of a system by exchanging information over an unreliable and potentially compromised network. ((("central trusted authority")))Satoshi Nakamoto's solution, which uses the concept of Proof-of-Work to achieve consensus _without a central trusted authority_, represents a breakthrough in distributed computing and has wide applicability beyond currency. It can be used to achieve consensus on decentralized networks to prove the fairness of elections, lotteries, asset registries, digital notarization, and more. -**** - - -[[user-stories]] -=== Bitcoin Uses, Users, and Their Stories - -((("bitcoin", "use cases", id="GSuses01")))Bitcoin is an innovation in the ancient technology of money. At its core, money simply facilitates the exchange of value between people. Therefore, in order to fully understand Bitcoin and its uses, we'll examine it from the perspective of people using it. Each of the people and their stories, as listed here, illustrates one or more specific use cases. We'll be seeing them throughout the book: - -North American low-value retail:: -((("use cases", "retail sales")))Alice lives in Northern California's Bay Area. She has heard about Bitcoin from her techie friends and wants to start using it. We will follow her story as she learns about Bitcoin, acquires some, and then spends some of her bitcoin to buy a cup of coffee at Bob's Cafe in Palo Alto. This story will introduce us to the software, the exchanges, and basic transactions from the perspective of a retail consumer. - -North American high-value retail:: -Carol is an art gallery owner in San Francisco. She sells expensive paintings for Bitcoin. This story will introduce the risks of a "51%" consensus attack for retailers of high-value items. - -Offshore contract services:: -((("offshore contract services")))((("use cases", "offshore contract services")))Bob, the cafe owner in Palo Alto, is building a new website. He has contracted with an Indian web developer, Gopesh, who lives in Bangalore, India. Gopesh has agreed to be paid in bitcoin. This story will examine the use of Bitcoin for outsourcing, contract services, and international wire transfers. - -Web store:: -((("use cases", "web store")))Gabriel is an enterprising young teenager in Rio de Janeiro, running a small web store that sells Bitcoin-branded t-shirts, coffee mugs, and stickers. Gabriel is too young to have a bank account, but his parents are encouraging his entrepreneurial spirit. - -Charitable donations:: -((("charitable donations")))((("use cases", "charitable donations")))Eugenia is the director of a children's charity in the Philippines. Recently she has discovered Bitcoin and wants to use it to reach a whole new group of foreign and domestic donors to fundraise for her charity. She's also investigating ways to use Bitcoin to distribute funds quickly to areas of need. This story will show the use of Bitcoin for global fundraising across currencies and borders and the use of an open ledger for transparency in charitable organizations. - -Import/export:: -((("use cases", "import/export")))Mohammed is an electronics importer in Dubai. He's trying to use Bitcoin to buy electronics from the United States and China for import into the UAE to accelerate the process of payments for imports. This story will show how Bitcoin can be used for large business-to-business international payments tied to physical goods. - -Mining for bitcoin:: -((("use cases", "mining for bitcoin")))Jing is a computer engineering student in Shanghai. He has built a "mining" rig to mine for bitcoin using his engineering skills to supplement his income. This story will examine the "industrial" base of Bitcoin: the specialized equipment used to secure the Bitcoin network and issue new currency. - -Each of these stories is based on the real people and real industries currently using Bitcoin to create new markets, new industries, and innovative solutions to global economic issues.((("", startref="GSuses01"))) - -=== Getting Started - -((("getting started", "wallet selection", id="GSwallet01")))((("wallets", "selecting", id="Wselect01")))((("bitcoin", "getting started", id="BCbasic01")))Bitcoin is a protocol that can be accessed using a client application that speaks the protocol. A "Bitcoin wallet" is the most common user interface to the Bitcoin system, just like a web browser is the most common user interface for the HTTP protocol. There are many implementations and brands of Bitcoin wallets, just like there are many brands of web browsers (e.g., Chrome, Safari, Firefox, and Internet Explorer). And just like we all have our favorite browsers (Mozilla Firefox, Yay!) and our villains (Internet Explorer, Yuck!), Bitcoin wallets vary in quality, performance, security, privacy, and reliability. There is also a reference implementation of the Bitcoin protocol that includes a wallet, known as the "Satoshi Client" or "Bitcoin Core," which is derived from the original implementation written by Satoshi Nakamoto. - -==== Choosing a Bitcoin Wallet - -((("security", "wallet selection")))Bitcoin wallets are one of the most actively developed applications in the Bitcoin ecosystem. There is intense competition, and while a new wallet is probably being developed right now, several wallets from last year are no longer actively maintained. Many wallets focus on specific platforms or specific uses and some are more suitable for beginners while others are filled with features for advanced users. Choosing a wallet is highly subjective and depends on the use and user expertise. Therefore it would be pointless to recommend a specific brand or wallet. However, we can categorize Bitcoin wallets according to their platform and function and provide some clarity about all the different types of wallets that exist. Better yet, moving keys or seeds between Bitcoin wallets is relatively easy, so it is worth trying out several different wallets until you find one that fits your needs. - -[role="pagebreak-before"] -Bitcoin wallets can be categorized as follows, according to the platform: - -Desktop wallet:: A desktop wallet was the first type of Bitcoin wallet created as a reference implementation and many users run desktop wallets for the features, autonomy, and control they offer. Running on general-use operating systems such as Windows and Mac OS has certain security disadvantages, however, as these platforms are often insecure and poorly configured. - -Mobile wallet:: A mobile wallet is the most common type of Bitcoin wallet. Running on smart-phone operating systems such as Apple iOS and Android, these wallets are often a great choice for new users. Many are designed for simplicity and ease-of-use, but there are also fully featured mobile wallets for power users. - -Web wallet:: Web wallets are accessed through a web browser and store the user's wallet on a server owned by a third party. This is similar to webmail in that it relies entirely on a third-party server. Some of these services operate using client-side code running in the user's browser, which keeps control of the Bitcoin keys in the hands of the user. Most, however, present a compromise by taking control of the Bitcoin keys from users in exchange for ease-of-use. It is inadvisable to store large amounts of bitcoin on third-party systems. - -Hardware wallet:: Hardware wallets are devices that operate a secure self-contained Bitcoin wallet on special-purpose hardware. They usually connect to a desktop or mobile device via USB cable or near-field-communication (NFC), and are operated with a web browser or accompanying software. By handling all Bitcoin-related operations on the specialized hardware, these wallets are considered very secure and suitable for storing large amounts of bitcoin. - -Another way to categorize bitcoin wallets is by their degree of autonomy and how they interact with the Bitcoin network: - -Full-node client:: ((("full-node clients")))A full client, or "full node," is a client that stores the entire history of Bitcoin transactions (every transaction by every user, ever), manages users' wallets, and can initiate transactions directly on the Bitcoin network. A full node handles all aspects of the protocol and can independently validate the entire blockchain and any transaction. A full-node client consumes substantial computer resources (e.g., more than 125 GB of disk, 2 GB of RAM) but offers complete autonomy and independent transaction verification. - -Lightweight client:: ((("lightweight clients")))((("simplified-payment-verification (SPV)")))A lightweight client, also known as a simplified-payment-verification (SPV) client, connects to Bitcoin full nodes (mentioned previously) for access to the Bitcoin transaction information, but stores the user wallet locally and independently creates, validates, and transmits transactions. Lightweight clients interact directly with the Bitcoin network, without an intermediary. - -Third-party API client:: ((("third-party API clients")))A third-party API client is one that interacts with Bitcoin through a third-party system of application programming interfaces (APIs), rather than by connecting to the Bitcoin network directly. The wallet may be stored by the user or by third-party servers, but all transactions go through a third party. - -Combining these categorizations, many Bitcoin wallets fall into a few groups, with the three most common being desktop full client, mobile lightweight wallet, and web third-party wallet. The lines between different categories are often blurry, as many wallets run on multiple platforms and can interact with the network in different ways. - -For the purposes of this book, we will be demonstrating the use of a variety of downloadable Bitcoin clients, from the reference implementation (Bitcoin Core) to mobile and web wallets. Some of the examples will require the use of Bitcoin Core, which, in addition to being a full client, also exposes APIs to the wallet, network, and transaction services. If you are planning to explore the programmatic interfaces into the Bitcoin system, you will need to run Bitcoin Core, or one of the alternative clients.((("", startref="GSwallet01")))((("", startref="Wselect01"))) - -==== Quick Start - -((("getting started", "quick start example", id="GSquick01")))((("wallets", "quick start example", id="Wquick01")))((("use cases", "buying coffee", id="aliceone")))Alice, who we introduced in <>, is not a technical user and only recently heard about Bitcoin from her friend Joe. While at a party, Joe is once again enthusiastically explaining Bitcoin to all around him and is offering a demonstration. Intrigued, Alice asks how she can get started with Bitcoin. Joe says that a mobile wallet is best for new users and he recommends a few of his favorite wallets. Alice downloads "Bluewallet" (available for iOS and Android) and installs it on her phone. - -When Alice runs her wallet application for the first time, she chooses the option to create a new Bitcoin wallet, and takes a moment **away from Joe and all other parties** to write down a secret mnemonic phrase _in order_ on a piece of paper. As explained by the mobile wallet and by Joe earlier, the mnemonic phrase allows Alice to restore her wallet in case she loses her mobile device and grants her access to her funds on another device. After creating her wallet and securing her mnemonic phrase, Alice can tap on her wallet to see her bitcoin amount, transaction history, as well as two buttons that allow her to either _receive_ or _send_ bitcoin, shown in <>. - -==== Mnemonic Words - -A modern Bitcoin wallet will provide a _mnemonic phrase_ (also sometimes called a "seed" or "seed phrase") for Alice to back up. The mnemonic phrase consists of 12-24 English words, selected randomly by the software, and used as the basis for the keys that are generated by the wallet. The mnemonic phrase can be used by Alice to restore all the transactions and funds in her wallet in the case of an event such as a lost mobile device, a software bug, or memory corruption. - -[TIP] -==== -The correct term for these backup words is "mnemonic phrase". We avoid the use of the term "seed" to refer to a mnemonic phrase, because even though its use is common it is incorrect. -==== - -[[mnemonic-storage]] -==== Storing the Mnemonic Safely - -Alice needs to be careful to store the mnemonic phrase in a way that balances the need to prevent theft and accidental loss. If she doesn't protect it enough, her mnemonic will be at risk of being stolen. If she protects it too much, her mnemonic will be at risk of being permanently lost. The recommended way to properly balance these risks is to write two copies of the mnemonic phrase on paper, with each of the words numbered as the order matters. - -Once Alice has recorded the mnemonic phrase, she should plan to store each copy in a separate secure location such as a locked desk drawer or a fireproof safe. - -[WARNING] -==== -Never attempt a "DIY" security scheme that deviates in any way from the best practice recommendation in <>. Do not cut your mnemonic in half, make screenshots, store on USB drives, email or cloud drives, encrypt it, or try any other non-standard method. You will tip the balance in such a way as to risk permanent loss or theft. Many people have lost funds, not from theft but because they tried a non-standard solution without having the expertise to balance the risks involved. The best practice recommendation is carefully balanced by experts and suitable for the vast majority of users. -==== -**** -[[bluewallet-welcome]] -.Alice uses the Receive screen on her Bluewallet mobile Bitcoin wallet, and displays her address to Bob in a QR code format -image::images/mbc2_0101.png["BluewalletWelcome"] - -((("addresses", "bitcoin wallet quick start example")))((("QR codes", "bitcoin wallet quick start example")))((("addresses", see="also keys and addresses"))) The main wallet view displays the bitcoin amount, transaction history, and _Receive_ and _Send_ buttons. In addition, many wallets feature the ability to purchase Bitcoin directly through an exchange or similar service where you can offer fiat money in return for cryptocurrency, which is done by <> and selling to the wallet user at or above this price. The _Buy Bitcoin_ button would allow Alice to purchase Bitcoin in this fashion. - -Alice is now ready to start using her new bitcoin wallet. ((("", startref="GSquick01")))((("", startref="Wquick01"))) Her wallet application randomly generated a private key (described in more detail in <>) which will be used to derive Bitcoin addresses that direct to her wallet. At this point, her Bitcoin addresses are not known to the Bitcoin network or "registered" with any part of the Bitcoin system. Her Bitcoin addresses are simply random numbers that correspond to her private key that she can use to control access to the funds. The addresses are generated independently by her wallet without reference or registration with any service. In fact, in most wallets, there is no association between a Bitcoin address and any externally identifiable information including the user's identity. Until the moment an address is referenced as the recipient of value in a transaction posted on the bitcoin ledger, the Bitcoin address is simply part of the vast number of possible addresses that are valid in bitcoin. Only once an address has been associated with a transaction does it become part of the known addresses in the network. - -Alice uses the _Receive_ button, which displays a QR code along with a Bitcoin address. The QR code is the square with a pattern of black and white dots, serving as a form of barcode that contains the same information in a format that can be scanned by Joe's smartphone camera. Next to the wallet's QR code is the Bitcoin address it encodes, and Alice may choose to manually send her address to Joe by copying it onto her clipboard with a tap. Of note, when receiving funds to a new mobile wallet for the first time, many wallets will often re-verify that you have indeed secured your mnemonic phrase. This can range from a simple prompt to requiring the user to manually re-enter the phrase. - -[TIP] -==== -((("addresses", "security of")))((("security", "Bitcoin addresses")))Bitcoin addresses start with 1, 3, or bc1. Like email addresses, they can be shared with other bitcoin users who can use them to send bitcoin directly to your wallet. There is nothing sensitive, from a security perspective, about the Bitcoin address. It can be posted anywhere without risking the security of the account. Unlike email addresses, you can create new addresses as often as you like, all of which will direct funds to your wallet. In fact, many modern wallets automatically create a new address for every transaction to maximize privacy. A wallet is simply a collection of addresses and the keys that unlock the funds within. -==== - - -[[getting_first_bitcoin]] -==== Getting Her First Bitcoin - -There are several ways Alice can acquire bitcoin: - -* She can exchange some of her national currency (e.g. USD) at a cryptocurrency exchange -* She can buy some from a friend, or an acquaintance from a Bitcoin Meetup, in exchange for cash -* She can find a _Bitcoin ATM_ in her area, which acts as a vending machine, selling bitcoin for cash -* She can offer her skills or a product she sells and accepts payment in bitcoin -* She can ask her employer or clients to pay her in bitcoin - -All of these methods have varying degrees of difficulty, and many will involve paying a fee. Some financial institutions will also require Alice to provide identification documents to comply with local banking regulations/anti-money laundering (AML) practices, a process which is known as Know Your Customer (KYC). However, with all these methods, Alice will be able to receive bitcoin. - -[TIP] -==== -((("privacy, maintaining")))((("security", "maintaining privacy")))((("digital currencies", "currency exchanges")))((("currency exchanges")))((("digital currencies", "benefits of bitcoin")))((("bitcoin", "benefits of")))One of the advantages of bitcoin over other payment systems is that, when used correctly, it affords users much more privacy. Acquiring, holding, and spending bitcoin does not require you to divulge sensitive and personally identifiable information to third parties. However, where bitcoin touches traditional systems, such as currency exchanges, national and international regulations often apply. In order to exchange bitcoin for your national currency, you will often be required to provide proof of identity and banking information. Users should be aware that once a Bitcoin address is attached to an identity, all associated bitcoin transactions are also easy to identify and track. This is one reason many users choose to maintain dedicated exchange accounts unlinked to their wallets. -==== - -Alice was introduced to bitcoin by a friend so she has an easy way to acquire her first bitcoin. Next, we will look at how she buys bitcoin from her friend Joe and how Joe sends the bitcoin to her wallet. - -[[bitcoin_price]] -==== Finding the Current Price of Bitcoin - -((("getting started", "exchange rates")))((("exchange rates", "determining")))Before Alice can buy bitcoin from Joe, they have to agree on the _exchange rate_ between bitcoin and US dollars. This brings up a common question for those new to bitcoin: "Who sets the bitcoin price?" The short answer is that the price is set by markets. - -((("exchange rates", "floating")))((("floating exchange rate")))Bitcoin, like most other currencies, has a _floating exchange rate_. That means that the value of bitcoin vis-a-vis any other currency fluctuates according to supply and demand in the various markets where it is traded. For example, the "price" of bitcoin in US dollars is calculated in each market based on the most recent trade of bitcoin and US dollars. As such, the price tends to fluctuate minutely several times per second. A pricing service will aggregate the prices from several markets and calculate a volume-weighted average representing the broad market exchange rate of a currency pair (e.g., BTC/USD). - -There are hundreds of applications and websites that can provide the current market rate. Here are some of the most popular: - -https://bitcoinaverage.com/[Bitcoin Average]:: ((("BitcoinAverage")))A site that provides a simple view of the volume-weighted-average for each currency. -https://coincap.io/[CoinCap]:: A service listing the market capitalization and exchange rates of hundreds of crypto-currencies, including bitcoin. -https://bit.ly/cmebrr[Chicago Mercantile Exchange Bitcoin Reference Rate]:: A reference rate that can be used for institutional and contractual reference, provided as part of investment data feeds by the CME. - -In addition to these various sites and applications, most bitcoin wallets will automatically convert amounts between bitcoin and other currencies. Joe will use his wallet to convert the price automatically before sending bitcoin to Alice. - -[[sending_receiving]] -==== Sending and Receiving Bitcoin - -((("getting started", "sending and receiving bitcoin", id="GSsend01")))((("spending bitcoin", "bitcoin wallet quick start example")))((("spending bitcoin", see="also transactions")))Alice has decided to exchange $10 US dollars for bitcoin, so as not to risk too much money on this new technology. She gives Joe $10 in cash, opens her Bluewallet mobile wallet application, and selects Receive. This displays a QR code with Alice's first Bitcoin address. - -Joe then selects Send on his Bluewallet smartphone wallet and is presented with a screen containing the following inputs: - -* The amount to send, in bitcoin (BTC) or his local currency (USD) -* A destination Bitcoin address -* A transaction note (description) -* A transaction fee - -In the input field for the Bitcoin address, there is a small _Scan_ button. This allows Joe to scan the QR code with his smartphone camera so that he doesn't have to type in Alice's Bitcoin address, which is quite long and difficult to type. Joe taps the _Scan_ button and it activates the smartphone camera, scanning the QR code displayed on Alice's smartphone. - -Joe now has Alice's Bitcoin address set as the recipient. Joe enters the amount as $10 US dollars and his wallet converts it by accessing the most recent exchange rate from an online service. The exchange rate at the time is $100 US dollars per bitcoin, so $10 US dollars is worth 0.10 bitcoin (BTC), or 100 millibitcoin (mBTC) as shown in the screenshot from Joe's wallet (see <>). - -In the transaction note/description input, Joe enters "Alice". He can use this field to add some information regarding his transaction for future reference. This function is for his record keeping only. The transaction note will be stored in his wallet and only Joe will be able to see it. It will not be sent to Alice, nor stored on the blockchain. - -He also selects a transaction fee for his transaction. The higher the transaction fee, the faster his transaction will be confirmed (included in a block by a miner). He selects the minimum transaction fee possible at that time (0 satoshis/byte). - -[TIP] -==== -The price of bitcoin has changed a lot over time, and an incredible amount since the first edition of this book was written. As of March 2021, a person would need approximately $54,000 USD to purchase one whole bitcoin. Many examples in this book reference real-life past transactions, when the price of bitcoin was much lower and transactions with zero fees were still possible. Think about how generous of a friend Joe would have been if he had made the same agreement with Alice today! -==== - -[[bluewallet-mobile-send]] -[role="smallereighty"] -.Bluewallet mobile bitcoin wallet send screen -image::images/mbc2_0102.png["bluewallet mobile send screen"] - -Using Bluewallet, Joe carefully checks to make sure he has entered the correct amount, because he is about to transmit money and mistakes are irreversible. For simplicity sake, we will assume that Joe does not pay any transaction fees. The purpose and setting of transaction fees are covered in subsequent chapters. After double-checking the address and amount, he presses _Send_ to transmit the transaction. Joe's mobile bitcoin wallet constructs a transaction that assigns 0.10 BTC to the address provided by Alice, sourcing the funds from Joe's wallet and signing the transaction with Joe's private keys. This tells the Bitcoin network that Joe has authorized a transfer of value to Alice's new address. As the transaction is transmitted via the peer-to-peer protocol, it quickly propagates across the Bitcoin network. In less than a second, most of the well-connected nodes in the network receive the transaction and see Alice's address for the first time. - -Meanwhile, Alice's wallet is constantly "listening" to published transactions on the Bitcoin network, looking for any that match the addresses it contains. A few seconds after Joe's wallet transmits the transaction, Alice's wallet will indicate that it is receiving 0.10 BTC. - -[TIP] -==== -Each bitcoin can be subdivided into 100 million units, each called a "satoshi" (singular) or "satoshis" (plural). Named for bitcoin's creator, the Satoshi is the smallest unit of bitcoin, equivalent to 0.00000001 BTC. -==== - -[[confirmations]] -.Confirmations -((("getting started", "confirmations")))((("confirmations", "bitcoin wallet quick start example")))((("confirmations", see="also mining and consensus; transactions")))((("clearing", seealso="confirmations")))At first, Alice's wallet will show the transaction from Joe as "Unconfirmed." This means that the transaction has been propagated to the network but has not yet been recorded in the bitcoin transaction ledger, known as the blockchain. To be confirmed, a transaction must be included in a block and added to the blockchain, which happens every 10 minutes, on average. In traditional financial terms this is known as _clearing_. This book will cover in detail the propagation, validation, and clearing (or confirmation) of bitcoin transactions. - -Alice is now the proud owner of 0.10 BTC that she can spend. In the next chapter we will look at her first purchase with bitcoin, and examine the underlying transaction and propagation technologies in more detail.((("", startref="BCbasic01")))((("use cases", "buying coffee", startref="aliceone"))) diff --git a/ch01_intro.adoc b/ch01_intro.adoc new file mode 100644 index 00000000..f937a240 --- /dev/null +++ b/ch01_intro.adoc @@ -0,0 +1,505 @@ +[role="pagenumrestart"] +[[ch01_intro_what_is_bitcoin]] +== Introduction + +Bitcoin((("Bitcoin", "operational overview", id="bitcoin-operational-overview"))) is a collection of concepts and technologies that form the basis of a digital money ecosystem. Units of currency called bitcoin are used to store and transmit value among participants in the Bitcoin network. Bitcoin users communicate with each other using the Bitcoin protocol primarily via the internet, although other transport networks can also be used. The Bitcoin protocol stack, available as open source software, can be run on a wide range of computing devices, including laptops and smartphones, making the technology easily accessible. + +[TIP] +==== +In this book, the ((("bitcoins", "defined")))unit of currency is called "bitcoin" with a small _b_, +and the system is called "Bitcoin," with a capital _B_. +==== + +Users can transfer bitcoin over the network to do just about anything +that can be done with conventional currencies, including buying and selling +goods, sending money to people or organizations, or extending credit. Bitcoin +can be purchased, sold, and exchanged for other currencies at +specialized currency exchanges. Bitcoin is arguably the perfect form +of money for the internet because it is fast, secure, and borderless. + +Unlike traditional currencies, the bitcoin currency is entirely virtual. There are no +physical coins or even individual digital coins. The coins are implied in +transactions that transfer value from spender to receiver. Users of +Bitcoin control keys that allow them to prove ownership of bitcoin in the +Bitcoin network. With these keys, they can sign transactions to unlock +the value and spend it by transferring it to a new owner. Keys are often +stored in a digital wallet on each user’s computer or smartphone. +Possession of the key that can sign a transaction is the only +prerequisite to spending bitcoin, putting the control entirely in the +hands of each user. + +Bitcoin is a distributed, peer-to-peer system. As such, there is no +central server or point of control. Units of bitcoin +are created through a process((("bitcoins", "mining")))((("mining"))) called "mining," which involves repeatedly +performing a computational task that references a list of recent Bitcoin +transactions. Any participant in the Bitcoin network may operate as a +miner, using their computing devices to help secure +transactions. Every 10 minutes, on average, one Bitcoin miner can add security to +past transactions and is rewarded with both brand new +bitcoins and the fees paid by recent transactions. Essentially, Bitcoin +mining decentralizes the currency-issuance +and clearing functions of a central bank and replaces the need for any +central bank. + +//-- Math for following paragraph -- +//total_btc = 0 +//for i in range(0, 10_000_000): +// total_btc += (50 / (2**int(i/210000)) ) +// if total_btc / 21e6 > 0.99: +// print(i) +// break + +The Bitcoin protocol includes built-in algorithms that regulate the +mining function across the network. The difficulty of the computational +task that miners must perform is adjusted dynamically so that, on +average, someone succeeds every 10 minutes regardless of how many miners +(and how much processing) are competing at any moment. The protocol also +periodically decreases the number of new bitcoins that are created, +limiting the total number of bitcoins that will ever be created to a fixed total +just below 21 million coins. The result is that the number of bitcoins in +circulation closely follows an easily predictable curve where half of +the remaining coins are added to circulation every four years. At +approximately block 1,411,200, which is expected to be produced around +the year 2035, 99% of all bitcoins +that will ever exist will have been issued. Due to Bitcoin's +diminishing rate of issuance, over the long term, the Bitcoin currency +is deflationary. Furthermore, nobody can force you to accept +any bitcoins that were created beyond the +expected issuance rate. + +Behind the scenes, Bitcoin is also the name of the protocol, a peer-to-peer network, and a distributed computing innovation. Bitcoin builds on decades of research in cryptography and distributed systems and includes at least four key innovations brought together in a unique and powerful combination. Bitcoin consists of: + +* A decentralized peer-to-peer network (the Bitcoin protocol) +* A public transaction journal (the blockchain) +* A set of rules for independent transaction validation and currency issuance (consensus rules) +* A mechanism for reaching global decentralized consensus on the valid blockchain (proof-of-work algorithm) + +As a developer, I see Bitcoin as akin to the internet of money, a network for propagating value and securing the ownership of digital assets via distributed computation. There's a lot more to Bitcoin than first meets ((("Bitcoin", "operational overview", startref="bitcoin-operational-overview")))the eye. + +In this chapter we'll get started by explaining some of the main concepts and terms, getting the necessary software, and using Bitcoin for simple transactions. In the following chapters, we'll start unwrapping the layers of technology that make Bitcoin possible and examine the inner workings of the Bitcoin network and protocol. + +.Digital Currencies Before Bitcoin +**** + +The ((("digital currencies, history of")))((("history", "of digital currencies", secondary-sortas="digital currencies")))((("cryptography")))emergence of viable digital money is closely linked to developments in cryptography. This is not surprising when one considers the fundamental challenges involved with using bits to represent value that can be exchanged for goods and services. Three basic questions for anyone accepting digital money are: + +* Can I trust that the money is authentic and not counterfeit? +* Can I trust that the digital money can only be spent once (known as the “double-spend” problem)? +* Can I be sure that no one else can claim this money belongs to them and not me? + +Issuers of paper money are constantly battling the counterfeiting problem by using increasingly sophisticated papers and printing technology. Physical money addresses the double-spend issue easily because the same paper note cannot be in two places at once. Of course, conventional money is also often stored and transmitted digitally. In these cases, the counterfeiting and double-spend issues are handled by clearing all electronic transactions through central authorities that have a global view of the currency in circulation. For digital money, which cannot take advantage of esoteric inks or holographic strips, cryptography provides the basis for trusting the legitimacy of a user’s claim to value. Specifically, cryptographic digital signatures enable a user to sign a digital asset or transaction proving the ownership of that asset. With the appropriate architecture, digital signatures also can be used to address the double-spend issue. + +When cryptography started becoming more broadly available and understood in the late 1980s, many researchers began trying to use cryptography to build digital currencies. These early digital currency projects issued digital money, usually backed by a national currency or precious metal such as gold. + +Although these earlier digital currencies worked, they were centralized and, as a result, were easy to attack by governments and hackers. Early digital currencies used a central clearinghouse to settle all transactions at regular intervals, just like a traditional banking system. Unfortunately, in most cases these nascent digital currencies were targeted by worried governments and eventually litigated out of existence. Some failed in spectacular crashes when the parent company liquidated abruptly. To be robust against intervention by antagonists, whether legitimate governments or criminal elements, a _decentralized_ digital currency was needed to avoid a single point of attack. Bitcoin is such a system, decentralized by design, and free of any central authority or point of control that can be attacked or corrupted. + +**** + +=== History of Bitcoin + +Bitcoin was((("Bitcoin", "history of")))((("history", "of Bitcoin", secondary-sortas="Bitcoin")))((("Nakamoto, Satoshi"))) first described in 2008 with the publication of a +paper titled "Bitcoin: A Peer-to-Peer Electronic Cash +System,"footnote:[https://oreil.ly/KUaBM["Bitcoin: A Peer-to-Peer Electronic Cash System"], +Satoshi Nakamoto.] written under the +alias of Satoshi Nakamoto (see <>). Nakamoto +combined several prior inventions such as digital signatures and Hashcash to create +a completely decentralized electronic cash system that does not rely on +a central authority for currency issuance or settlement and validation +of transactions. A key innovation was to use a distributed computation +system (called a "proof-of-work" algorithm) to conduct a global +lottery every 10 minutes on average, allowing the decentralized network to +arrive at _consensus_ about the state of transactions. This +elegantly solves the issue of double-spend where a single currency unit +can be spent twice. Previously, the double-spend problem was a weakness +of digital currency and was addressed by clearing all transactions +through a central clearinghouse. + +The Bitcoin network started in 2009, based on a reference implementation +published by Nakamoto and since revised by many other programmers. The +number and power of machines running the((("proof-of-work algorithm", seealso="mining"))) proof-of-work algorithm +(mining) that provides security and resilience for Bitcoin have +increased exponentially, and their combined computational power now +exceeds the combined number of computing operations of the +world's top supercomputers. + + +Satoshi Nakamoto withdrew from the public in April 2011, leaving the responsibility of developing the code and network to a thriving group of volunteers. The identity of the person or people behind Bitcoin is still unknown. However, neither Satoshi Nakamoto nor anyone else exerts individual control over the Bitcoin system, which operates based on fully transparent mathematical principles, open source code, and consensus among participants. The invention itself is groundbreaking and has already spawned new science in the fields of distributed computing, economics, and econometrics. + +.A Solution to a Distributed Computing Problem +**** +Satoshi Nakamoto's invention ((("Byzantine Generals' Problem")))((("distributed computing problem")))is +also a practical and novel solution to a problem in distributed +computing, known as the "Byzantine Generals' Problem." Briefly, the +problem consists of trying to get multiple participants without a leader +to agree on a course of action by exchanging information over an +unreliable and potentially compromised network. Satoshi Nakamoto's solution, which uses the concept of +proof of work to achieve consensus _without a central trusted +authority_, represents a breakthrough in distributed computing. +**** + +=== Getting Started + +Bitcoin is((("Bitcoin", "wallets", see="wallets")))((("wallets", "explained"))) a protocol that can be accessed using an +application that speaks the protocol. A "Bitcoin wallet" is the +most common user interface to the Bitcoin system, just like a web +browser is the most common user interface for the HTTP protocol. There +are many implementations and brands of Bitcoin wallets, just like there +are many brands of web browsers (e.g., Chrome, Safari, and Firefox). +And just like we all have our favorite browsers, +Bitcoin wallets vary in quality, performance, security, privacy, and +reliability. There is also a reference implementation of the Bitcoin +protocol that includes a wallet, known as "Bitcoin Core," which is +derived from the original implementation written by Satoshi Nakamoto. + +==== Choosing a Bitcoin Wallet + +Bitcoin wallets ((("wallets", "choosing")))((("selecting", "wallets")))((("choosing", see="selecting")))are one of the most actively developed applications in the Bitcoin ecosystem. There is intense competition, and while a new wallet is probably being developed right now, several wallets from last year are no longer actively maintained. Many wallets focus on specific platforms or specific uses and some are more suitable for beginners while others are filled with features for advanced users. Choosing a wallet is highly subjective and depends on the use and user expertise. Therefore, it would be pointless to recommend a specific brand or wallet. However, we can categorize Bitcoin wallets according to their platform and function and provide some clarity about all the different types of wallets that exist. It is worth trying out several different wallets until you find one that fits your needs. + +===== Types of Bitcoin wallets +Bitcoin wallets ((("wallets", "types of", id="wallet-type")))can be categorized as follows, according to the platform: + +Desktop wallet:: A ((("desktop wallets")))desktop wallet was the first type of Bitcoin wallet created as a reference implementation. Many users run desktop wallets for the features, autonomy, and control they offer. Running on general-use operating systems such as Windows and macOS has certain security disadvantages, however, as these platforms are often insecure and poorly configured. + +Mobile wallet:: A ((("mobile wallets")))mobile wallet is the most common type of Bitcoin +wallet. Running on smart-phone operating systems such as Apple iOS and +Android, these wallets are often a great choice for new users. Many are +designed for simplicity and ease-of-use, but there are also fully +featured mobile wallets for power users. To avoid downloading and +storing large amounts of data, most mobile wallets retrieve information +from remote servers, reducing your privacy by disclosing to third +parties information about your Bitcoin addresses and balances. + +Web wallet:: Web wallets ((("web wallets")))are accessed through a web browser and store +the user's wallet on a server owned by a third party. This is similar to +webmail in that it relies entirely on a third-party server. Some of +these services operate using client-side code running in the user's +browser, which keeps control of the Bitcoin keys in the hands of the +user, although the user's dependence on the server still compromises +their privacy. Most, however, take control of the Bitcoin keys from +users in exchange for ease-of-use. It is inadvisable +to store large amounts of bitcoin on third-party systems. + +Hardware signing devices:: Hardware signing devices((("hardware signing devices"))) are devices that can +store keys and sign transactions using special-purpose hardware and +firmware. They usually +connect to a desktop, mobile, or web wallet via USB cable, +near-field-communication (NFC), or a camera with QR codes. By handling +all Bitcoin-related operations on the specialized hardware, these +wallets are less vulnerable to many types of attacks. Hardware signing +devices are sometimes called "hardware wallets", but they need to be +paired with a full-featured wallet to send and receive transactions, and +the security and privacy offered by that paired wallet plays a critical +role in how much security and privacy the user obtains when using the +hardware signing device. + +===== Full node versus Lightweight +Another way to categorize Bitcoin wallets is by their degree of autonomy and how they interact with the Bitcoin network: + +Full node:: A full node ((("full nodes")))is a program that validates the +entire history of Bitcoin transactions (every transaction by every user, ever). Optionally, full nodes can also store previously validated +transactions and serve data to other Bitcoin programs, either on the +same computer or over the internet. A full node uses substantial +computer resources--about the same as watching an hour-long streaming +video for each day of Bitcoin transactions--but the full node offers +complete autonomy to its users. + +Lightweight client:: +A lightweight client,((("lightweight clients")))((("simplified-payment-verification (SPV) clients")))((("SPV (simplified-payment-verification) clients"))) also known as a simplified-payment-verification (SPV) client, +connects to a full node or other remote server for receiving and sending +Bitcoin transaction information, but stores the user wallet locally, +partially validates the transactions it receives, and independently +creates outgoing transactions. + +Third-party API client:: A third-party +API client ((("third-party API clients")))is one that interacts with Bitcoin through a third-party +system of APIs rather than by +connecting to the Bitcoin network directly. The wallet may be stored by +the user or by third-party servers, but the client trusts the remote +server to provide it with accurate information and protect its ((("wallets", "types of", startref="wallet-type")))privacy. + +[TIP] +==== +Bitcoin ((("Bitcoin", "as peer-to-peer network", secondary-sortas="peer-to-peer network")))((("peer-to-peer networks, Bitcoin as")))((("peers")))((("clients")))is a peer-to-peer (P2P) network. Full nodes are the _peers:_ +each peer individually validates every confirmed transaction and can +provide data to its user with complete authority. Lightweight wallets +and other software are _clients:_ each client depends on one or more peers +to provide it with valid data. Bitcoin clients can perform secondary +validation on some of the data they receive and make connections to +multiple peers to reduce their dependence on the integrity of a single +peer, but the security of a client ultimately relies on the integrity of +its peers. +==== + +===== Who controls the keys + +A very((("wallets", "key control")))((("keys, control of")))((("bitcoins", "key control"))) important additional consideration is _who controls the keys_. As +we will see in subsequent chapters, access to bitcoins is +controlled by "private keys," which are like very long PINs. If +you are the only one to have control over these private +keys, you are in control of your bitcoins. Conversely, if you do not have +control, then your bitcoins are managed by a third-party who +ultimately controls your funds on your behalf. Key management software falls into two +important categories based on control: _wallets_, where you +control the keys, and the funds and accounts with custodians where some +third-party controls the keys. To emphasize this point, I (Andreas) +coined the phrase: _Your keys, your coins. Not your keys, not your coins_. + +Combining these categorizations, many Bitcoin wallets fall into a few +groups, with the three most common being desktop full node +(you control the keys), mobile lightweight wallet (you control the keys), and web-based +accounts with third parties (you don't control the keys). The lines between different categories +are sometimes blurry, as software runs on multiple platforms and can +interact with the network in different ways. + +==== Quick Start + +Alice is not a +technical user and only recently heard about Bitcoin from her friend +Joe. While at a party, Joe is enthusiastically explaining +Bitcoin to everyone around him and is offering a demonstration. Intrigued, +Alice asks how she can get started with Bitcoin. Joe says that a mobile +wallet is best for new users and he recommends a few of his favorite +wallets. Alice downloads one of Joe's recommendations +and installs it on her phone. + +When Alice runs her wallet application for the first time, she chooses +the option to create a new Bitcoin wallet. Because the wallet she has +chosen is a ((("wallets", "noncustodial")))((("noncustodial wallets")))noncustodial wallet, Alice (and only Alice) will be in +control of her keys. Therefore, she bears responsibility for backing +them up, since losing the keys means she loses access to her bitcoins. To +facilitate this, her wallet produces a _recovery code_ that can be used +to restore her wallet. + +[[recovery_code_intro]] +==== Recovery Codes + +Most ((("wallets", "recovery codes", id="wallet-recovery")))((("recovery codes", id="recovery-code")))((("backing up", "recovery codes", see="recovery codes")))modern noncustodial Bitcoin wallets will provide a recovery +code for their user +to back up. The recovery code usually consists of numbers, letters, or words +selected randomly by the software, and is used as the basis for the keys +that are generated by the wallet. See <> for +examples. + +++++ + + + + + + + + + + + + + + + + + + + + + + +
Sample recovery codes
WalletRecovery code

BlueWallet

(1) media (2) suspect (3) effort (4) dish (5) album (6) shaft (7) price (8) junk (9) pizza (10) situate (11) oyster (12) rib

Electrum

nephew dog crane clever quantum crazy purse traffic repeat fruit old clutch

Muun

LAFV TZUN V27E NU4D WPF4 BRJ4 ELLP BNFL

+++++ + +[TIP] +==== +A recovery code is ((("mnemonic phrases", see="recovery codes")))((("seed phrases", see="recovery codes")))sometimes called a "mnemonic" or "mnemonic phrase," +which implies you should memorize the phrase, but writing the phrase +down on paper takes less work and tends to be more reliable than most +people's memories. Another alternative name is "seed phrase" because +it provides the input ("seed") to the function that generates all of +a wallet's keys. +==== + +If something happens to Alice's wallet, she can download a new copy of +her wallet software and enter this recovery code to rebuild the wallet +database of all the onchain transactions she's ever sent or received. +However, recovering from the recovery code will not by itself restore any additional +data Alice entered into her wallet, such as the labels she associated +with particular addresses or transactions. Although losing access to +that metadata isn't as important as losing access to money, it can +still be important in its own way. Imagine you need to review an old +bank or credit card statement and the name of every entity you paid (or +who paid you) has been blanked out. To prevent losing metadata, many +wallets provide an additional backup feature beyond recovery codes. + +For some wallets, that additional backup feature is even more important +today than it used to be. Many Bitcoin payments ((("offchain technology")))are now made using +_offchain_ technology, where not every payment is stored in the public blockchain. This reduces user's costs and improves privacy, among other +benefits, but it means that a mechanism like recovery codes that depends on +onchain data can't guarantee recovery of all of a user's bitcoins. For +applications with offchain support, it's important to make frequent +backups of the wallet database. + +Of note, when receiving funds to a new mobile wallet for the first time, +many wallets will often re-verify that you have securely backed-up your +recovery code. This can range from a simple prompt to requiring the +user to manually re-enter the code. + +[WARNING] +==== +Although many legitimate wallets will prompt you to re-enter +your recovery code, there are also many malware applications that mimic the +design of a wallet, insist you enter your recovery code, and then +relay any entered code to the malware developer so they can steal +your funds. This is the equivalent of phishing websites that try to +trick you into giving them your bank passphrase. For most wallet +applications, the only times they will ask for your recovery code are during +the initial set up (before you have received any bitcoins) and during +recovery (after you lost access to your original wallet). If the application +asks for your recovery code any other time, consult with an expert to +ensure you aren't ((("wallets", "recovery codes", startref="wallet-recovery")))((("recovery codes", startref="recovery-code")))being phished. +==== + +==== Bitcoin Addresses + +Alice is ((("addresses", "explained")))now ready to start using her new Bitcoin wallet. Her wallet application randomly generated a private key (described in more detail in <>) that will be used to derive Bitcoin addresses that direct to her wallet. At this point, her Bitcoin addresses are not known to the Bitcoin network or "registered" with any part of the Bitcoin system. Her Bitcoin addresses are simply numbers that correspond to her private key that she can use to control access to the funds. The addresses are generated independently by her wallet without reference or registration with any service. + +[TIP] +==== +There +are a variety ((("invoices")))of Bitcoin addresses and invoice formats. Addresses and +invoices can be shared with other Bitcoin users +who can use them to send bitcoins directly to your wallet. You can share +an address or invoice with other people without worrying about the +security of your bitcoins. Unlike a bank account number, nobody who +learns one of your Bitcoin addresses can withdraw money from your wallet--you +must initiate all spends. However, if you give two people the same +address, they will be able to see how many bitcoins the other person sent +you. If you post your address publicly, everyone will be able to see +how much bitcoin other people sent to that address. To protect your privacy, you +should generate a new invoice with a new address each time you request a +payment. +==== + +==== Receiving Bitcoin + +Alice((("bitcoins", "receiving")))((("receiving bitcoins"))) uses the _Receive_ button, which displays a QR code, shown in <>. + +[role="width-50"] +[[wallet_receive]] +.Alice uses the Receive screen on her mobile Bitcoin wallet and displays her address in a QR code format. +image::images/mbc3_0101.png["Wallet receive screen with QR code displayed. Image derived from Bitcoin Design Guide CC-BY"] + +The QR code is the square with a pattern of black and white dots, serving as a form of barcode that contains the same information in a format that can be scanned by Joe's smartphone camera. + +[WARNING] +==== +Any funds sent to the addresses in this book will be lost. If you want +to test sending bitcoins, please consider donating it to a +bitcoin-accepting charity. +==== + +[[getting_first_bitcoin]] +==== Getting Your First Bitcoin + +The((("bitcoins", "acquiring", id="bitcoin-acquire")))((("acquiring bitcoins", id="acquire-bitcoin"))) first task for new users is to acquire some bitcoin. + +Bitcoin transactions are irreversible. Most electronic payment networks such as credit cards, debit cards, PayPal, and bank account transfers are reversible. For someone selling bitcoin, this difference introduces a very high risk that the buyer will reverse the electronic payment after they have received bitcoin, in effect defrauding the seller. To mitigate this risk, companies accepting traditional electronic payments in return for bitcoin usually require buyers to undergo identity verification and credit-worthiness checks, which may take several days or weeks. As a new user, this means you cannot buy bitcoin instantly with a credit card. With a bit of patience and creative thinking, however, you won't need to. + + +Here are some methods for acquiring bitcoin as a new user: + +* Find a friend who has bitcoins and buy some from him or her directly. Many Bitcoin users start this way. This method is the least complicated. One way to meet people with bitcoins is to attend a local Bitcoin meetup listed at pass:[Meetup.com]. +* Earn bitcoin by selling a product or service for bitcoin. If you are a programmer, sell your programming skills. If you're a hairdresser, cut hair for bitcoins. +* Use a Bitcoin ATM in your city. A Bitcoin ATM is a machine that accepts cash and sends bitcoins to your smartphone Bitcoin wallet. +* Use((("bitcoins", "currency exchanges")))((("currency exchanges"))) a Bitcoin currency exchange linked to your bank account. Many countries now have currency exchanges that offer a market for buyers and sellers to swap bitcoins with local currency. Exchange-rate listing services, such as https://bitcoinaverage.com[BitcoinAverage], often show a list of Bitcoin exchanges for each currency. + +[TIP] +==== +One of the advantages of +Bitcoin over other payment systems is that, when used correctly, it +affords users much more privacy. Acquiring, holding, and spending +bitcoin does not require you to divulge sensitive and personally +identifiable information to third parties. However, where bitcoin +touches traditional systems, such as currency exchanges, national and +international regulations often apply. In order to exchange bitcoin for +your national currency, you will often be required to provide proof of +identity and banking information. Users should be aware that once a +Bitcoin address is attached to an identity, other associated Bitcoin +transactions may also become easy to identify and track--including +transactions made earlier. This is one reason +many users choose to maintain dedicated exchange accounts independent from +their wallets. +==== + +Alice was introduced to Bitcoin by a friend, so she has an easy way to acquire her first bitcoins. Next, we will look at how she buys bitcoins from her friend Joe and how Joe sends the bitcoins to her ((("bitcoins", "acquiring", startref="bitcoin-acquire")))((("acquiring bitcoins", startref="acquire-bitcoin")))wallet. + +[[bitcoin_price]] +==== Finding the Current Price of Bitcoin + +Before ((("bitcoins", "exchange rate", id="bitcoin-exchange-rate")))((("exchange rate", id="exchange-rate")))((("current price of bitcoins", id="current-price")))Alice can buy bitcoin from Joe, they have to agree on the _exchange rate_ between bitcoin and US dollars. This brings up a common question for those new to Bitcoin: "Who sets the price +of bitcoins?" The short answer is that the price is set by markets. + +Bitcoin, like most other currencies, has a _floating exchange rate_. That means that the value of bitcoin fluctuates according to supply and demand in the various markets where it is traded. For example, the "price" of bitcoin in US dollars is calculated in each market based on the most recent trade of bitcoins and US dollars. As such, the price tends to fluctuate minutely several times per second. A pricing service will aggregate the prices from several markets and calculate a volume-weighted average representing the broad market exchange rate of a currency pair (e.g., BTC/USD). + +There are hundreds of applications and websites that can provide the current market rate. Here are some of the most popular: + +https://bitcoinaverage.com[Bitcoin Average]:: A site that provides a simple view of the volume-weighted average for each currency. +https://coincap.io[CoinCap]:: A service listing the market capitalization and exchange rates of hundreds of cryptocurrencies, including bitcoins. +https://oreil.ly/ACieC[Chicago Mercantile Exchange Bitcoin Reference Rate]:: A reference rate that can be used for institutional and contractual reference, provided as part of investment data feeds by the CME. + +In addition to these various sites and applications, some Bitcoin +wallets will automatically convert amounts between bitcoin and other currencies. + +[[sending_receiving]] +==== Sending and Receiving Bitcoin + +Alice ((("bitcoins", "spending", id="bitcoin-send")))((("spending bitcoins", id="send-bitcoin")))((("bitcoins", "receiving", id="bitcoin-receive")))((("receiving bitcoins", id="receive-bitcoin")))has +decided to buy 0.001 bitcoins. After she and Joe check the exchange rate, +she gives Joe an appropriate amount of cash, opens her mobile wallet +application, and selects Receive. This +displays a QR code with Alice's first Bitcoin address. + +Joe then selects Send on his smartphone wallet and opens the QR code +scanner. This allows Joe to scan the barcode with his smartphone camera +so that he doesn't have to type in Alice's Bitcoin address, which is +quite long. + +Joe now has Alice's Bitcoin address set as the recipient. Joe enters the amount as 0.001 bitcoins (BTC); see <>. Some wallets may +show the amount in a different denomination: 0.001 BTC is 1 millibitcoin +(mBTC) or 100,000 satoshis (sats). + +Some wallets may also suggest Joe enter a label for this transaction; if +so, Joe enters "Alice". Weeks or months from now, this will help Joe +remember why he sent these 0.001 bitcoins. Some wallets may also prompt +Joe about fees. Depending on the wallet and how the transaction is +being sent, the wallet may ask Joe to either enter a transaction fee rate or +prompt him with a suggested fee (or fee rate). The higher the transaction fee, the +faster the transaction will be confirmed (see <>). + +[[wallet-send]] +.Bitcoin wallet send screen. +image::images/mbc3_0102.png["Wallet send screen. Image derived from Bitcoin Design Guide CC-BY"] + +Joe then carefully checks to make sure he has entered the correct +amount, because he is about to transmit money and mistakes will soon become +irreversible. After double-checking the address and amount, he presses +Send to transmit the transaction. Joe's mobile Bitcoin wallet constructs +a transaction that assigns 0.001 BTC to the address provided by Alice, +sourcing the funds from Joe's wallet, and signing the transaction with +Joe's private keys. This tells the Bitcoin network that Joe has +authorized a transfer of value to Alice's new address. As the +transaction is transmitted via the peer-to-peer protocol, it quickly +propagates across the Bitcoin network. After just a few seconds, most of +the well-connected nodes in the network receive the transaction and see +Alice's address for the first time. + +Meanwhile, Alice's wallet is constantly "listening" for new +transactions on the Bitcoin network, looking for any that match the +addresses it contains. A few seconds after Joe's wallet transmits the +transaction, Alice's wallet will indicate that it is receiving +0.001 BTC. + +[[confirmations]] +.Confirmations +**** +At first, ((("bitcoins", "clearing transactions")))((("clearing transactions")))((("transactions", "clearing")))((("confirmations")))Alice's address will show the transaction from Joe as "Unconfirmed." This means that the transaction has been propagated to the network but has not yet been recorded in the Bitcoin transaction journal, known as the blockchain. To be confirmed, a transaction must be included in a block and added to the blockchain, which happens every 10 minutes, on average. In traditional financial terms this is known as _clearing_. For more details on propagation, validation, and clearing (confirmation) of bitcoin transactions, see <>. +**** + +Alice is now the proud owner of 0.001 BTC that she can spend. Over the next few days, Alice buys more bitcoin using an ATM and((("bitcoins", "spending", startref="bitcoin-send")))((("spending bitcoins", startref="send-bitcoin")))((("bitcoins", "receiving", startref="bitcoin-receive")))((("receiving bitcoins", startref="receive-bitcoin"))) an exchange. In the next chapter we will look at her first purchase with Bitcoin, and examine the underlying transaction and propagation technologies in more detail. diff --git a/ch02.asciidoc b/ch02.asciidoc deleted file mode 100644 index 1b14f1d4..00000000 --- a/ch02.asciidoc +++ /dev/null @@ -1,288 +0,0 @@ -[[ch02_bitcoin_overview]] -== How Bitcoin Works - -=== Transactions, Blocks, Mining, and the Blockchain - -((("bitcoin", "overview of", id="BCover02")))((("central trusted authority")))((("decentralized systems", "bitcoin overview", id="DCSover02")))The Bitcoin system, unlike traditional banking and payment systems, is based on decentralized trust. Instead of a central trusted authority, in bitcoin, trust is achieved as an emergent property from the interactions of different participants in the Bitcoin system. In this chapter, we will examine bitcoin from a high level by tracking a single transaction through the Bitcoin system and watch as it becomes "trusted" and accepted by the bitcoin mechanism of distributed consensus and is finally recorded on the blockchain, the distributed ledger of all transactions. Subsequent chapters will delve into the technology behind transactions, the network, and mining. - -==== Bitcoin Overview - -In the overview diagram shown in <>, we see that the Bitcoin system consists of users with wallets containing keys, transactions that are propagated across the network, and miners who produce (through competitive computation) the consensus blockchain, which is the authoritative ledger of all transactions. - - -((("blockchain explorer sites")))Each example in this chapter is based on an actual transaction made on the Bitcoin network, simulating the interactions between the users (Joe, Alice, Bob, and Gopesh) by sending funds from one wallet to another. While tracking a transaction through the Bitcoin network to the blockchain, we will use a _blockchain explorer_ site to visualize each step. A blockchain explorer is a web application that operates as a bitcoin search engine, in that it allows you to search for addresses, transactions, and blocks and see the relationships and flows between them. - -[[bitcoin-overview]] -.Bitcoin overview -image::images/mbc2_0201.png["Bitcoin Overview"] - -((("Bitcoin Block Explorer")))((("BlockCypher Explorer")))((("Blockchain.com")))((("BitPay Insight")))Popular blockchain explorers include: - -* https://live.blockcypher.com[BlockCypher Explorer] -* https://blockchain.com[Blockchain.com] -* https://insight.bitpay.com[BitPay Insight] -* https://blockstream.info[Blockstream Explorer] - -Each of these has a search function that can take a Bitcoin address, transaction hash, block number, or block hash and retrieve corresponding information from the Bitcoin network. With each transaction or block example, we will provide a URL so you can look it up yourself and study it in detail. - - -[[cup_of_coffee]] -==== Buying a Cup of Coffee - -((("use cases", "buying coffee", id="UCcoffee02")))Alice, introduced in the previous chapter, is a new user who has just acquired her first bitcoin. In <>, Alice met with her friend Joe to exchange some cash for bitcoin. The transaction created by Joe funded Alice's wallet with 0.10 BTC. Now Alice will make her first retail transaction, buying a cup of coffee at Bob's coffee shop in Palo Alto, California. - -((("exchange rates", "determining")))Bob's Cafe recently started accepting bitcoin payments by adding a bitcoin option to its point-of-sale system. The prices at Bob's Cafe are listed in the local currency (US dollars), but at the register, customers have the option of paying in either dollars or bitcoin. Alice places her order for a cup of coffee and Bob enters it into the register, as he does for all transactions. The point-of-sale system automatically converts the total price from US dollars to bitcoin at the prevailing market rate and displays the price in both currencies: - ----- -Total: -$1.50 USD -0.015 BTC ----- - - -((("millibits")))Bob says, "That's one-dollar-fifty, or fifteen millibits." - -((("payment requests")))((("QR codes", "payment requests")))Bob's point-of-sale system will also automatically create a special QR code containing a _payment request_ (see <>). - -Unlike a QR code that simply contains a destination Bitcoin address, a payment request is a QR-encoded URL that contains a destination address, a payment amount, and a generic description such as "Bob's Cafe." This allows a bitcoin wallet application to prefill the information used to send the payment while showing a human-readable description to the user. You can scan the QR code with a bitcoin wallet application to see what Alice would see. - - -[[payment-request-QR]] -.Payment request QR code -image::images/mbc2_0202.png["payment-request"] - -[TIP] -==== -((("QR codes", "warnings and cautions")))((("transactions", "warnings and cautions")))((("warnings and cautions", "avoid sending money to addresses appearing in book")))Try to scan this with your wallet to see the address and amount but DO NOT SEND MONEY. -==== -[[payment-request-URL]] -.The payment request QR code encodes the following URL, defined in BIP-21: ----- -bitcoin:1GdK9UzpHBzqzX2A9JFP3Di4weBwqgmoQA? -amount=0.015& -label=Bob%27s%20Cafe& -message=Purchase%20at%20Bob%27s%20Cafe - -Components of the URL - -A Bitcoin address: "1GdK9UzpHBzqzX2A9JFP3Di4weBwqgmoQA" -The payment amount: "0.015" -A label for the recipient address: "Bob's Cafe" -A description for the payment: "Purchase at Bob's Cafe" ----- - -Alice uses her smartphone to scan the barcode on display. Her smartphone shows a payment of +0.0150 BTC+ to +Bob's Cafe+ and she selects Send to authorize the payment. Within a few seconds (about the same amount of time as a credit card authorization), Bob sees the transaction on the register, completing the transaction. - -In the following sections, we will examine this transaction in more detail. We'll see how Alice's wallet constructed it, how it was propagated across the network, how it was verified, and finally, how Bob can spend that amount in subsequent transactions. - -[NOTE] -==== -((("fractional values")))((("milli-bitcoin")))((("satoshis")))The Bitcoin network can transact in fractional values, e.g., from millibitcoin (1/1000th of a bitcoin) down to 1/100,000,000th of a bitcoin, which is known as a satoshi. Throughout this book, we’ll use the term “bitcoin” to refer to any quantity of bitcoin currency, from the smallest unit (1 satoshi) to the total number (21,000,000) of all bitcoin that will ever be mined. -==== - -You can examine Alice's transaction to Bob's Cafe on the blockchain using a block explorer site (<>): - -[[view_alice_transaction]] -.View Alice's transaction on https://www.blockchain.com/btc/tx/0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2[blockchain.com] -==== ----- -https://www.blockchain.com/btc/tx/0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2 ----- -==== - -=== Bitcoin Transactions - -((("transactions", "defined")))In simple terms, a transaction tells the network that the owner of some bitcoin value has authorized the transfer of that value to another owner. The new owner can now spend the bitcoin by creating another transaction that authorizes the transfer to another owner, and so on, in a chain of ownership. - -==== Transaction Inputs and Outputs - -((("transactions", "overview of", id="Tover02")))((("outputs and inputs", "basics of")))Transactions are like lines in a double-entry bookkeeping ledger. Each transaction contains one or more "inputs," which are like debits against a bitcoin account. On the other side of the transaction, there are one or more "outputs," which are like credits added to a bitcoin account. ((("fees", "transaction fees")))The inputs and outputs (debits and credits) do not necessarily add up to the same amount. Instead, outputs add up to slightly less than inputs and the difference represents an implied _transaction fee_, which is a small payment collected by the miner who includes the transaction in the ledger. A bitcoin transaction is shown as a bookkeeping ledger entry in <>. - -The transaction also contains proof of ownership for each amount of bitcoin (inputs) whose value is being spent, in the form of a digital signature from the owner, which can be independently validated by anyone. ((("spending bitcoin", "defined")))In bitcoin terms, "spending" is signing a transaction that transfers value from a previous transaction over to a new owner identified by a Bitcoin address. - -[[transaction-double-entry]] -.Transaction as double-entry bookkeeping -image::images/mbc2_0203.png["Transaction Double-Entry"] - -==== Transaction Chains - -((("chain of transactions")))Alice's payment to Bob's Cafe uses a previous transaction's output as its input. In the previous chapter, Alice received bitcoin from her friend Joe in return for cash. That transaction created a bitcoin value locked by Alice's key. Her new transaction to Bob's Cafe references the previous transaction as an input and creates new outputs to pay for the cup of coffee and receive change. The transactions form a chain, where the inputs from the latest transaction correspond to outputs from previous transactions. Alice's key provides the signature that unlocks those previous transaction outputs, thereby proving to the Bitcoin network that she owns the funds. She attaches the payment for coffee to Bob's address, thereby "encumbering" that output with the requirement that Bob produces a signature in order to spend that amount. This represents a transfer of value between Alice and Bob. This chain of transactions, from Joe to Alice to Bob, is illustrated in <>. - -[[blockchain-mnemonic]] -.A chain of transactions, where the output of one transaction is the input of the next transaction -image::images/mbc2_0204.png["Transaction chain"] - -==== Making Change - -((("change, making")))((("change addresses")))((("addresses", "change addresses")))Many bitcoin transactions will include outputs that reference both an address of the new owner and an address of the current owner, called the _change_ address. This is because transaction inputs, like currency notes, cannot be divided. If you purchase a $5 US dollar item in a store but use a $20 US dollar bill to pay for the item, you expect to receive $15 US dollars in change. The same concept applies to bitcoin transaction inputs. If you purchased an item that costs 5 bitcoin but only had a 20 bitcoin input to use, your wallet would create a single transaction that sends two outputs, one output of 5 bitcoin to the store owner and one output of 15 bitcoin back to yourself as change (less any applicable transaction fee). Importantly, the change address does not have to be the same address as that of the input and for privacy reasons is often a new address from the owner's wallet. - -Different wallets may use different strategies when aggregating inputs to make a payment requested by the user. They might aggregate many small inputs, or use one that is equal to or larger than the desired payment. Unless the wallet can aggregate inputs in such a way to exactly match the desired payment plus transaction fees, the wallet will need to generate some change. This is very similar to how people handle cash. If you always use the largest bill in your pocket, you will end up with a pocket full of loose change. If you only use the loose change, you'll always have only big bills. People subconsciously find a balance between these two extremes, and bitcoin wallet developers strive to program this balance. - -((("transactions", "defined")))((("outputs and inputs", "defined")))((("inputs", see="outputs and inputs")))In summary, _transactions_ move value from _transaction inputs_ to _transaction outputs_. An input is a reference to a previous transaction's output, showing where the value is coming from. A transaction usually includes an output that directs a specific value to a new owner's Bitcoin address and a change output back to the original owner. Outputs from one transaction can be used as inputs in a new transaction, thus creating a chain of ownership as the value is moved from owner to owner (see <>). - -==== Common Transaction Forms - -The most common form of transaction is a simple payment from one address to another, which often includes some "change" returned to the original owner. This type of transaction has one input and two outputs and is shown in <>. - -[[transaction-common]] -.Most common transaction -image::images/mbc2_0205.png["Common Transaction"] - -Another common form of transaction is one that aggregates several inputs into a single output (see <>). This represents the real-world equivalent of exchanging a pile of coins and currency notes for a single larger note. Transactions like these are sometimes generated by wallet applications to clean up lots of smaller amounts that were received as change for payments. - -[[transaction-aggregating]] -.Transaction aggregating funds -image::images/mbc2_0206.png["Aggregating Transaction"] - -Finally, another transaction form that is seen often on the bitcoin ledger is a batched transaction, which distributes one input to multiple outputs representing multiple recipients, a technique called "transaction batching" (see <>). Since this type of transaction is useful for saving in transaction fees, it is commonly used by commercial entities to distribute funds, such as when a company is processing payroll payments to multiple employees or when a bitcoin exchange is processing multiple customers' withdrawals in a single transaction.((("", startref="Tover02"))) - -[[transaction-distributing]] -.Transaction distributing funds -image::images/mbc2_0207.png["Distributing Transaction"] - -=== Constructing a Transaction - -((("transactions", "constructing", id="Tconstruct02")))((("wallets", "constructing transactions")))Alice's wallet application contains all the logic for selecting appropriate inputs and outputs to build a transaction to Alice's specification. Alice only needs to specify a destination and an amount, and the rest happens in the wallet application without her seeing the details. Importantly, a wallet application can construct transactions even if it is completely offline. Like writing a check at home and later sending it to the bank in an envelope, the transaction does not need to be constructed and signed while connected to the Bitcoin network. - -==== Getting the Right Inputs - -((("outputs and inputs", "locating and tracking inputs")))Alice's wallet application will first have to find inputs that can pay the amount she wants to send to Bob. Most wallets keep track of all the available outputs belonging to addresses in the wallet. Therefore, Alice's wallet would contain a copy of the transaction output from Joe's transaction, which was created in exchange for cash (see <>). A bitcoin wallet application that runs as a full-node client actually contains a copy of every unspent output from every transaction in the blockchain. This allows a wallet to construct transaction inputs as well as quickly verify incoming transactions as having correct inputs. However, because a full-node client takes up a lot of disk space, most user wallets run "lightweight" clients that track only the user's own unspent outputs. - -If the wallet application does not maintain a copy of unspent transaction outputs, it can query the Bitcoin network to retrieve this information using a variety of APIs available by different providers or by asking a full-node using an application programming interface (API) call. <> shows an API request, constructed as an HTTP GET command to a specific URL. This URL will return all the unspent transaction outputs for an address, giving any application the information it needs to construct transaction inputs for spending. We use the simple command-line HTTP client _cURL_ to retrieve the response. - -[[example_2-2]] -.Look up all the unspent outputs for Alice's Bitcoin address -==== -[source,bash] ----- -$ curl https://blockchain.info/unspent?active=1Cdid9KFAaatwczBwBttQcwXYCpvK8h7FK ----- -==== - -[source,json] ----- -{ - - "unspent_outputs":[ - - { - "tx_hash":"186f9f998a5...2836dd734d2804fe65fa35779", - "tx_index":104810202, - "tx_output_n": 0, - "script":"76a9147f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a888ac", - "value": 10000000, - "value_hex": "00989680", - "confirmations":0 - } - - ] -} ----- - - -The response in <> shows one unspent output (one that has not been redeemed yet) under the ownership of Alice's address +1Cdid9KFAaatwczBwBttQcwXYCpvK8h7FK+. The response includes the reference to the transaction in which this unspent output is contained (the payment from Joe) and its value in satoshis, at 10 million, equivalent to 0.10 bitcoin. With this information, Alice's wallet application can construct a transaction to transfer that value to new owner addresses. - -[TIP] -==== -View the https://www.blockchain.com/btc/tx/7957a35fe64f80d234d76d83a2a8f1a0d8149a41d81de548f0a65a8a999f6f18[transaction from Joe to Alice]. -==== - -As you can see, Alice's wallet contains enough bitcoin in a single unspent output to pay for the cup of coffee. Had this not been the case, Alice's wallet application might have to "rummage" through a pile of smaller unspent outputs, like picking coins from a purse until it could find enough to pay for the coffee. In both cases, there might be a need to get some change back, which we will see in the next section, as the wallet application creates the transaction outputs (payments). - - -==== Creating the Outputs - -((("outputs and inputs", "creating outputs")))A transaction output is created in the form of a script that creates an encumbrance on the value and can only be redeemed by the introduction of a solution to the script. In simpler terms, Alice's transaction output will contain a script that says something like, "This output is payable to whoever can present a signature from the key corresponding to Bob's address." Because only Bob has the wallet with the keys corresponding to that address, only Bob's wallet can present such a signature to redeem this output. Alice will therefore "encumber" the output value with a demand for a signature from Bob. - -This transaction will also include a second output, because Alice's funds are in the form of a 0.10 BTC output, too much money for the 0.015 BTC cup of coffee. Alice will need 0.085 BTC in change. Alice's change payment is created by Alice's wallet as an output in the very same transaction as the payment to Bob. Essentially, Alice's wallet breaks her funds into two payments: one to Bob and one back to herself. She can then use (spend) the change output in a subsequent transaction. - -Finally, for the transaction to be processed by the network in a timely fashion, Alice's wallet application will add a small fee. This is not explicit in the transaction; it is implied by the difference between inputs and outputs. If instead of taking 0.085 in change, Alice creates only 0.0845 as the second output, there will be 0.0005 BTC (half a millibitcoin) left over. The input's 0.10 BTC is not fully spent with the two outputs, because they will add up to less than 0.10. The resulting difference is the _transaction fee_ that is collected by the miner as a fee for validating and including the transaction in a block to be recorded on the blockchain. - -The resulting transaction can be seen using a blockchain explorer web application, as shown in <>. - -[[transaction-alice]] -[role="smallerseventyfive"] -.Alice's transaction to Bob's Cafe -image::images/mbc2_0208.png["Alice Coffee Transaction"] - -[[transaction-alice-url]] -[TIP] -==== -View the https://www.blockchain.com/btc/tx/0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2[transaction from Alice to Bob's Cafe]. -==== - -==== Adding the Transaction to the Ledger - -The transaction created by Alice's wallet application is 258 bytes long and contains everything necessary to confirm ownership of the funds and assign new owners. Now, the transaction must be transmitted to the Bitcoin network where it will become part of the blockchain. In the next section we will see how a transaction becomes part of a new block and how the block is "mined." Finally, we will see how the new block, once added to the blockchain, is increasingly trusted by the network as more blocks are added. - -===== Transmitting the transaction - -((("propagation", "process of")))Because the transaction contains all the information necessary to process, it does not matter how or where it is transmitted to the Bitcoin network. The Bitcoin network is a peer-to-peer network, with each Bitcoin client participating by connecting to several other Bitcoin clients. The purpose of the Bitcoin network is to propagate transactions and blocks to all participants. - -===== How it propagates - -((("Bitcoin nodes", "defined")))((("nodes", see="Bitcoin nodes")))Any system, such as a server, desktop application, or wallet, that participates in the Bitcoin network by "speaking" the Bitcoin protocol is called a _Bitcoin node_. Alice's wallet application can send the new transaction to any Bitcoin node it is connected to over any type of connection: wired, WiFi, mobile, etc. Her bitcoin wallet does not have to be connected to Bob's bitcoin wallet directly and she does not have to use the internet connection offered by the cafe, though both those options are possible, too. ((("propagation", "flooding technique")))((("flooding technique")))Any Bitcoin node that receives a valid transaction it has not seen before will immediately forward it to all other nodes to which it is connected, a propagation technique known as _flooding_. Thus, the transaction rapidly propagates out across the peer-to-peer network, reaching a large percentage of the nodes within a few seconds. - -===== Bob's view - -If Bob's bitcoin wallet application is directly connected to Alice's wallet application, Bob's wallet application might be the first node to receive the transaction. However, even if Alice's wallet sends the transaction through other nodes, it will reach Bob's wallet within a few seconds. Bob's wallet will immediately identify Alice's transaction as an incoming payment because it contains outputs redeemable by Bob's keys. Bob's wallet application can also independently verify that the transaction is well formed, uses previously unspent outputs, and contains sufficient transaction fees to be included in the next block. At this point Bob can assume, with little risk, that the transaction will shortly be included in a block and confirmed. - -[TIP] -==== -((("confirmations", "of small-value transactions", secondary-sortas="small-value transactions")))A common misconception about bitcoin transactions is that they must be "confirmed" by waiting 10 minutes for a new block, or up to 60 minutes for a full six confirmations. Although confirmations ensure the transaction has been accepted by the whole network, such a delay is unnecessary for small-value items such as a cup of coffee. A merchant may accept a valid small-value transaction with no confirmations, with no more risk than a credit card payment made without an ID or a signature, as merchants routinely accept today.((("", startref="Tconstruct02"))) -==== - -=== Bitcoin Mining - -((("mining and consensus", "overview of", id="MACover02")))((("blockchain (the)", "overview of mining", id="BToverview02")))Alice's transaction is now propagated on the Bitcoin network. It does not become part of the _blockchain_ until it is verified and included in a block by a process called _mining_. See <> for a detailed explanation. - -The Bitcoin system of trust is based on computation. Transactions are bundled into _blocks_, which require an enormous amount of computation to prove, but only a small amount of computation to verify as proven. The mining process serves two purposes in bitcoin: - -* ((("mining and consensus", "consensus rules", "security provided by")))((("consensus", see="mining and consensus")))Mining nodes validate all transactions by reference to bitcoin's _consensus rules_. Therefore, mining provides security for bitcoin transactions by rejecting invalid or malformed transactions. -* Mining creates new bitcoin in each block, almost like a central bank printing new money. The amount of bitcoin created per block is limited and diminishes with time, following a fixed issuance schedule. - - -Mining achieves a fine balance between cost and reward. Mining uses electricity to solve a mathematical problem. A successful miner will collect a _reward_ in the form of new bitcoin and transaction fees. However, the reward will only be collected if the miner has correctly validated all the transactions, to the satisfaction of the rules of _consensus_. This delicate balance provides security for bitcoin without a central authority. - -A good way to describe mining is like a giant competitive game of sudoku that resets every time someone finds a solution and whose difficulty automatically adjusts so that it takes approximately 10 minutes to find a solution. Imagine a giant sudoku puzzle, several thousand rows and columns in size. If I show you a completed puzzle you can verify it quite quickly. However, if the puzzle has a few squares filled and the rest are empty, it takes a lot of work to solve! The difficulty of the sudoku can be adjusted by changing its size (more or fewer rows and columns), but it can still be verified quite easily even if it is very large. The "puzzle" used in bitcoin is based on a cryptographic hash and exhibits similar characteristics: it is asymmetrically hard to solve but easy to verify, and its difficulty can be adjusted. - -((("mining and consensus", "mining farms and pools")))In <>, we introduced ((("use cases", "mining for bitcoin")))Jing, an entrepreneur in Shanghai. Jing runs a _mining farm_, which is a business that runs thousands of specialized mining computers, competing for the reward. Every 10 minutes or so, Jing's mining computers compete against thousands of similar systems in a global race to find a solution to a block of transactions. ((("Proof-of-Work algorithm")))((("mining and consensus", "Proof-of-Work algorithm")))Finding such a solution, the so-called _Proof-of-Work_ (PoW), requires quadrillions of hashing operations per second across the entire Bitcoin network. The algorithm for Proof-of-Work involves repeatedly hashing the header of the block and a random number with the SHA256 cryptographic algorithm until a solution matching a predetermined pattern emerges. The first miner to find such a solution wins the round of competition and publishes that block into the blockchain. - -Jing started mining in 2010 using a very fast desktop computer to find a suitable Proof-of-Work for new blocks. As more miners started joining the Bitcoin network, the difficulty of the problem increased rapidly. Soon, Jing and other miners upgraded to more specialized hardware, with high-end dedicated graphical processing units (GPUs), often used in gaming desktops or consoles. At the time of this writing, the difficulty is so high that it is profitable only to mine with ((("application-specific integrated circuits (ASIC)")))application-specific integrated circuits (ASIC), essentially hundreds of mining algorithms printed in hardware, running in parallel on a single silicon chip. ((("mining pools", "defined")))Jing's company also participates in a _mining pool_, which much like a lottery pool allows several participants to share their efforts and rewards. Jing's company now runs a warehouse containing thousands of ASIC miners to mine for bitcoin 24 hours a day. The company pays its electricity costs by selling the bitcoin it is able to generate from mining, creating some income from the profits. - -=== Mining Transactions in Blocks - -((("blocks", "mining transactions in")))New transactions are constantly flowing into the network from user wallets and other applications. As these are seen by the Bitcoin network nodes, they get added to a temporary pool of unverified transactions maintained by each node. As miners construct a new block, they add unverified transactions from this pool to the new block and then attempt to prove the validity of that new block, with the mining algorithm (Proof-of-Work). The process of mining is explained in detail in <>. - -Transactions are added to the new block, prioritized by the highest-fee transactions first and a few other criteria. Each miner starts the process of mining a new block of transactions as soon as they receive the previous block from the network, knowing they have lost that previous round of competition. They immediately create a new block, fill it with transactions and the fingerprint of the previous block, and start calculating the Proof-of-Work for the new block. Each miner includes a special transaction in their block, one that pays their own Bitcoin address the block reward (currently 6.25 newly created bitcoin) plus the sum of transaction fees from all the transactions included in the block. If they find a solution that makes that block valid, they "win" this reward because their successful block is added to the global blockchain and the reward transaction they included becomes spendable. ((("mining pools", "operation of")))Jing, who participates in a mining pool, has set up his software to create new blocks that assign the reward to a pool address. From there, a share of the reward is distributed to Jing and other miners in proportion to the amount of work they contributed in the last round. - -((("candidate blocks")))((("blocks", "candidate blocks")))Alice's transaction was picked up by the network and included in the pool of unverified transactions. Once validated by the mining software it was included in a new block, called a _candidate block_, generated by Jing's mining pool. All the miners participating in that mining pool immediately start computing Proof-of-Work for the candidate block. Approximately five minutes after the transaction was first transmitted by Alice's wallet, one of Jing's ASIC miners found a solution for the candidate block and announced it to the network. Once other miners validated the winning block they started the race to generate the next block. - -Jing's winning block became part of the blockchain as block #277316, containing 419 transactions, including Alice's transaction. The block containing Alice's transaction is counted as one "confirmation" of that transaction. - -[TIP] -==== -You can see the block that includes https://www.blockchain.com/btc/block/277316[Alice's transaction]. -==== - -((("confirmations", "role in transactions")))Approximately 20 minutes later, a new block, #277317, is mined by another miner. Because this new block is built on top of block #277316 that contained Alice's transaction, it added even more computation to the blockchain, thereby strengthening the trust in those transactions. Each block mined on top of the one containing the transaction counts as an additional confirmation for Alice's transaction. As the blocks pile on top of each other, it becomes exponentially harder to reverse the transaction, thereby making it more and more trusted by the network. - -((("genesis block")))((("blocks", "genesis block")))((("blockchain (the)", "genesis block")))In the diagram in <>, we can see block #277316, which contains Alice's transaction. Below it are 277,316 blocks (including block #0), linked to each other in a chain of blocks (blockchain) all the way back to block #0, known as the _genesis block_. Over time, as the "height" in blocks increases, so does the computation difficulty for each block and the chain as a whole. The blocks mined after the one that contains Alice's transaction act as further assurance, as they pile on more computation in a longer and longer chain. By convention, any block with more than six confirmations is considered irrevocable, because it would require an immense amount of computation to invalidate and recalculate six blocks. We will examine the process of mining and the way it builds trust in more detail in <>.((("", startref="BToverview02")))((("", startref="MACover02"))) - -[[block-alice1]] -.Alice's transaction included in block #277316 -image::images/mbc2_0209.png["Alice's transaction included in a block"] - -=== Spending the Transaction - -((("spending bitcoin", "simplified-payment-verification (SPV)")))((("simplified-payment-verification (SPV)")))Now that Alice's transaction has been embedded in the blockchain as part of a block, it is part of the distributed ledger of bitcoin and visible to all bitcoin applications. Each Bitcoin client can independently verify the transaction as valid and spendable. Full-node clients can track the source of the funds from the moment the bitcoin were first generated in a block, incrementally from transaction to transaction, until they reach Bob's address. Lightweight clients can do what is called a simplified payment verification (see <>) by confirming that the transaction is in the blockchain and has several blocks mined after it, thus providing assurance that the miners accepted it as valid. - -Bob can now spend the output from this and other transactions. For example, Bob can pay a contractor or supplier by transferring value from Alice's coffee cup payment to these new owners. Most likely, Bob's bitcoin software will aggregate many small payments into a larger payment, perhaps concentrating all the day's bitcoin revenue into a single transaction. This would aggregate the various payments into a single output (and a single address). For a diagram of an aggregating transaction, see <>. - -As Bob spends the payments received from Alice and other customers, he extends the chain of transactions. Let's assume that Bob pays his web designer Gopesh((("use cases", "offshore contract services"))) in Bangalore for a new web page. Now the chain of transactions will look like <>. - -[[block-alice2]] -.Alice's transaction as part of a transaction chain from Joe to Gopesh, where the output of one transaction is spent as the input of the next transaction -image::images/mbc2_0210.png["Alice's transaction as part of a transaction chain"] - -In this chapter, we saw how transactions build a chain that moves value from owner to owner. We also tracked Alice's transaction, from the moment it was created in her wallet, through the Bitcoin network and to the miners who recorded it on the blockchain. In the rest of this book, we will examine the specific technologies behind wallets, addresses, signatures, transactions, the network, and finally mining.((("", startref="BCover02")))((("", startref="DCSover02"))) ((("", startref="UCcoffee02"))) diff --git a/ch02_overview.adoc b/ch02_overview.adoc new file mode 100644 index 00000000..d658831f --- /dev/null +++ b/ch02_overview.adoc @@ -0,0 +1,629 @@ +[[ch02_bitcoin_overview]] +== How Bitcoin Works + +The Bitcoin system, unlike traditional banking and +payment systems, does not require trust in third parties. Instead of a central +trusted authority, in Bitcoin, each user can use software running on +their own computer to verify the correct operation of every +aspect of the Bitcoin system. +In this chapter, we will examine Bitcoin from a high level by tracking a +single transaction through the Bitcoin system and watch as it +is recorded on the blockchain, the distributed journal of all +transactions. Subsequent chapters will delve into the technology behind +transactions, the network, and ((("Bitcoin", "operational overview", id="bitcoin-operational-overview-ch2")))((("blockchain explorers", id="blockchain-explorers")))mining. + +=== Bitcoin Overview + +++++ +

+The Bitcoin system consists of users with wallets containing keys, +transactions that are propagated across the network, and miners who +produce (through competitive computation) the consensus blockchain, +which is the authoritative journal of all transactions. +

+ +

+Each example in this chapter is based +on an actual transaction made on the Bitcoin network, simulating the +interactions between several users by sending +funds from one wallet to another. While tracking a transaction through +the Bitcoin network to the blockchain, we will use a blockchain +explorer site to visualize each step. A blockchain explorer is a web +application that operates as a Bitcoin search engine, in that it allows +you to search for addresses, transactions, and blocks and see the +relationships and flows between them. +

+++++ + +Popular blockchain explorers include the following: + +* https://blockstream.info[Blockstream Explorer] +* https://mempool.space[Mempool.Space] +* https://live.blockcypher.com[BlockCypher Explorer] + +Each of these has a search function that can take a Bitcoin address, +transaction hash, block number, or block hash and retrieve corresponding +information from the Bitcoin network. With each transaction or block +example, we will provide a URL so you can look it up yourself and study +it in detail. + +[[block-explorer-privacy]] +.Block Explorer Privacy Warning +[WARNING] +==== +Searching information((("privacy", "blockchain explorers"))) on a block explorer may disclose to its operator +that you're interested in that information, allowing them to associate +it with your IP address, browser details, past searches, or other +identifiable information. If you look up the transactions in this book, +the operator of the block explorer might guess that you're learning +about Bitcoin, which shouldn't be a problem. But if you look up your +own transactions, the operator may be able to guess how many bitcoins +you've received, spent, and ((("Bitcoin", "operational overview", startref="bitcoin-operational-overview-ch2")))((("blockchain explorers", startref="blockchain-explorers")))currently own. +==== + +[[spending_bitcoin]] +=== Buying from an Online Store + +Alice, ((("transactions", "spending bitcoins", id="transaction-spend")))((("bitcoins", "spending", id="bitcoin-spend")))((("spending bitcoins", id="spend-bitcoin")))introduced in the previous chapter, is a new user who has just +acquired her first bitcoins. In <>, Alice met with +her friend Joe to exchange some cash for bitcoins. Since then, Alice has +bought additional bitcoins. Now Alice will make +her first spending transaction, buying access to a premium podcast episode from Bob's online store. + +Bob's web store recently started accepting bitcoin payments by adding a +Bitcoin option to its website. The prices at Bob's store are listed in +the local currency (US dollars), but at checkout, customers have the +option of paying in either dollars or bitcoin. + +Alice finds the podcast episode she wants to buy and proceeds to the checkout page. At checkout, +Alice is offered the option to pay with bitcoin in addition to the +usual options. The checkout cart displays the price in US dollars and +also in bitcoin (BTC), at Bitcoin's prevailing exchange rate. + +Bob's +ecommerce system will automatically create a QR code((("invoices")))((("QR codes"))) containing an +_invoice_ (<>). + +//// +TODO: Replace QR code with test-BTC address +//// + +[[invoice-QR]] +.Invoice QR code. +image::images/mbc3_0201.png["payment-request"] + +Unlike a QR code that simply contains a destination Bitcoin address, this +invoice is a QR-encoded URI that contains a destination address, +a payment amount, and a description. +This allows a Bitcoin wallet application to prefill the +information used to send the payment while showing a human-readable +description to the user. You can scan the QR code with a bitcoin wallet +application to see what Alice would see: + + +[[invoice-URI]] +.The invoice QR code encodes the following URI, defined in BIP21: +---- +bitcoin:bc1qk2g6u8p4qm2s2lh3gts5cpt2mrv5skcuu7u3e4?amount=0.01577764& +label=Bob%27s%20Store& +message=Purchase%20at%20Bob%27s%20Store + +Components of the URI + +A Bitcoin address: "bc1qk2g6u8p4qm2s2lh3gts5cpt2mrv5skcuu7u3e4" +The payment amount: "0.01577764" +A label for the recipient address: "Bob's Store" +A description for the payment: "Purchase at Bob's Store" +---- + +[TIP] +==== +Try to scan this with your wallet to see +the address and amount but DO NOT SEND MONEY. +==== + +Alice uses her smartphone to scan the barcode on display. Her smartphone +shows a payment for the correct amount to +Bob's Store+ and she selects Send to +authorize the payment. Within a few seconds (about the same amount of +time as a credit card authorization), Bob sees the transaction on the +register. + +[NOTE] +==== +The +Bitcoin network((("bitcoins", "fractional values")))((("fractional values of bitcoins")))((("satoshis")))((("millibitcoins"))) can transact in fractional values, e.g., from +millibitcoin (1/1000th of a bitcoin) down to 1/100,000,000th of a +bitcoin, which is known as a satoshi. This book uses the same +pluralization rules used for dollars and other traditional currencies +when talking about amounts greater than one bitcoin and when using +decimal notation, such as "10 bitcoins" or "0.001 bitcoins." The same +rules also apply to other bitcoin bookkeeping units, such as +millibitcoins and satoshis. +==== + +You can use a block explorer to examine blockchain data, such as the payment made to Bob in Alice's((("transactions", "spending bitcoins", startref="transaction-spend")))((("bitcoins", "spending", startref="bitcoin-spend")))((("spending bitcoins", startref="spend-bitcoin"))) https://oreil.ly/hAeyh[transaction]. + +In the following sections, we will examine this transaction in more +detail. We'll see how Alice's wallet constructed it, how it was +propagated across the network, how it was verified, and finally, how Bob +can spend that amount in subsequent transactions. + +=== Bitcoin Transactions + +In ((("transactions", "defined")))((("bitcoins", "transactions", see="transactions")))simple terms, a transaction tells the +network that the owner of certain bitcoins has authorized the transfer +of that value to another owner. The new owner can now spend the bitcoin +by creating another transaction that authorizes the transfer to another +owner, and so on, in a chain of ownership. + +==== Transaction Inputs and Outputs + +Transactions((("transactions", "inputs", id="transaction-input-ch2")))((("transactions", "outputs", id="transaction-output-ch2")))((("inputs", id="input")))((("outputs", id="output"))) are like lines in a double-entry +bookkeeping ledger. Each transaction contains one or more _inputs_, +which spend funds. On the other side of +the transaction, there are one or more _outputs_, which receive funds. +The inputs +and outputs do not necessarily add up to the same +amount. Instead, outputs add up to slightly less than inputs and the +difference represents an ((("transaction fees")))implied _transaction fee_, which is a small +payment collected by the miner who includes the transaction in the +blockchain. A Bitcoin transaction is shown as a bookkeeping ledger entry in +<>. + +The transaction also contains proof of ownership for each amount of +bitcoins (inputs) whose value is being spent, in the form of a digital +signature from the owner, which can be independently validated by +anyone. In Bitcoin terms, spending +is signing a transaction that transfers value from a previous +transaction over to a new owner identified by a Bitcoin address. + +[[transaction-double-entry]] +.Transaction as double-entry bookkeeping. +image::images/mbc3_0202.png["Transaction Double-Entry"] + +==== Transaction Chains + +Alice's ((("transaction chains", id="transaction-chains")))payment to Bob's Store uses a +previous transaction's output as its input. In the previous chapter, +Alice received bitcoins from her friend Joe in return for cash. +We've labeled that as _Transaction 1_ (Tx1) in <>. + +Tx1 sent 0.001 bitcoins (100,000 satoshis) to an output locked by +Alice's key. Her new transaction to Bob's Store (Tx2) references the +previous output as an input. In the illustration, we show that +reference using an arrow and by labeling the input as "Tx1:0". In an +actual transaction, the reference is the 32-byte transaction identifier +(txid) for the transaction where Alice received the money from Joe. The +":0" indicates the position of the output where Alice received the +money; in this case, the first position (position 0). + +As shown, actual Bitcoin transactions don't +explicitly include the value of their input. To determine the value of +an input, software needs to use the input's reference to find the +previous transaction output being spent. + +Alice's Tx2 contains two new outputs, one paying 75,000 satoshis for the +podcast and another paying 20,000 satoshis back to Alice to receive +change. + +//// +@startditaa + Transaction 1 Tx2 Tx3 + Inputs Outputs In Out In Out + +-------+---------+ +-------+--------+ +-------+--------+ + | | | | | cDDD | | | | +<--+ Tx0꞉0 | 100,000 |<--+ Tx1꞉0 | 20,000 | +-+ Tx2꞉1 | 67,000 | + | | | | | | | | | | + +-------+---------+ +-------+--------+ | +-------+--------+ + | | cDDD | | | | | | | | + | | 500,000 | | | 75,000 |<-+ | | | + | | | | | | | | | + +-------+---------+ +-------+--------+ +-------+--------+ + Fee꞉ (unknown) Fee꞉ 5,000 Fee꞉ 8,000 +@enddittaa +//// + +[[transaction-chain]] +.A chain of transactions, where the output of one transaction is the input of the next transaction. +image::images/mbc3_0203.png["Transaction chain"] + +[TIP] +==== +Serialized Bitcoin transactions--the data format that software uses for +sending transactions--encodes the value to transfer using an integer +of the smallest defined onchain unit of value. When Bitcoin was first +created, this unit didn't have a name and some developers simply called +it the _base unit._ Later many users began calling this unit a +_satoshi_ (sat) in honor of Bitcoin's creator. In <> +and some other illustrations in this book, we use satoshi values because +that's what the protocol itself ((("satoshis")))uses. +==== + +==== Making Change + +In addition((("transactions", "change output", id="transaction-change-output")))((("change output", id="change-output")))((("outputs", "change output", id="output-change"))) to one or more outputs that pay the receiver of +bitcoins, many transactions will also include an output that pays the +spender of the bitcoins, called a _change_ output. +This is because transaction inputs, +like currency notes, cannot be partly spent. If you purchase a $5 US item in a store but use a $20 bill to pay for the item, you +expect to receive $15 in change. The same concept applies to +Bitcoin transaction inputs. If you purchased an item that costs 5 +bitcoins but only had an input worth 20 bitcoins to use, you would send one +output of 5 bitcoins to the store owner and one output of 15 bitcoins back +to yourself as change (not counting your transaction fee). + +At the level of the Bitcoin protocol, there is no difference between a +change output (and the address it pays, called a _change address_) and a +payment output. + +Importantly, the change address does not have to be the +same address as that of the input and, for privacy reasons, is often a new +address from the owner's wallet. In ideal circumstances, the two +different uses of outputs both use never-before-seen addresses and +otherwise look identical, preventing any third party from determining +which outputs are change and which are payments. However, for +illustration purposes, we've added shading to the change outputs in +<>. + +Not every transaction has a change output. Those that don't are ((("changeless transactions")))((("transactions", "changeless")))called +_changeless transactions_, and they can have only a single output. +Changeless transactions are only a practical option if the amount being +spent is roughly the same as the amount available in the transaction +inputs minus the anticipated transaction fee. In <>, +we see Bob creating Tx3 as a changeless transaction that spends the +output he received in Tx2. + +==== Coin Selection + +Different wallets((("transactions", "coin selection")))((("coin selection in transactions")))((("selecting", "coins in transactions"))) use different strategies when choosing which +inputs to use in a payment, called _coin selection_. + +They might aggregate many small +inputs, or use one that is equal to or larger than the desired payment. +Unless the wallet can aggregate inputs in such a way to exactly match +the desired payment plus transaction fees, the wallet will need to +generate some change. This is very similar to how people handle cash. If +you always use the largest bill in your pocket, you will end up with a +pocket full of loose change. If you only use the loose change, you'll +often have only big bills. People subconsciously find a balance between +these two extremes, and Bitcoin wallet developers strive to program this +balance. + +==== Common Transaction Forms + +A very ((("transactions", "common types", id="transaction-common-ch2")))common form of transaction is a simple payment. This type of +transaction has one input and two outputs and is shown in +<>. + +[[transaction-common]] +.Most common transaction. +image::images/mbc3_0204.png["Common Transaction"] + +Another common form of transaction ((("consolidation transactions")))is a _consolidation transaction_, which spends several inputs +into a single output (<>). This represents +the real-world equivalent of exchanging a pile of coins and currency +notes for a single larger note. Transactions like these are sometimes +generated by wallets and businesses to clean up lots of smaller amounts. + +[[transaction-consolidating]] +.Consolidation transaction aggregating funds. +image::images/mbc3_0205.png["Aggregating Transaction"] + +Finally, another transaction form that is seen often on the +blockchain ((("payment batching")))is _payment batching_, which pays to multiple outputs +representing multiple recipients (<>). +This type of transaction is sometimes used by commercial entities to +distribute funds, such as when processing payroll payments to multiple employees. + +[[transaction-distributing]] +.Batch transaction distributing funds. +image::images/mbc3_0206.png["Distributing Transaction"] + +=== Constructing a Transaction + +Alice's wallet((("transactions", "constructing", id="transaction-construct"))) application contains all +the logic for selecting inputs and generating outputs to build a +transaction to Alice's specification. Alice only needs to choose a +destination, amount, and transaction fee, and the rest happens in the wallet +application without her seeing the details. Importantly, if a wallet +already knows what inputs it controls, it can construct transactions +even if it is completely offline. +Like writing a check at home and later sending it to the bank in an +envelope, the transaction does not need to be constructed and signed +while connected to the Bitcoin network. + +==== Getting the Right Inputs + +Alice's wallet +application ((("inputs", "constructing transactions")))will first have to find inputs that can pay the amount she +wants to send to Bob. Most wallets keep track of all the available +outputs belonging to addresses in the wallet. Therefore, Alice's wallet +would contain a copy of the transaction output from Joe's transaction, +which was created in exchange for cash (see <>). +A Bitcoin wallet application that runs on a full node actually +contains a copy of every confirmed transaction's ((("UTXOs (unspent transaction outputs)")))unspent outputs, called +_unspent transaction outputs_ (UTXOs). +However, because full nodes use more resources, many +user wallets run lightweight clients that track only the user's own +UTXOs. + +In this case, this single +UTXO is sufficient to pay for the podcast. Had this not been the case, +Alice's wallet application might have to combine several +smaller UTXOs, like picking coins from a purse, until it could +find enough to pay for the podcast. In both cases, there might be a need +to get some change back, which we will see in the next section, as the +wallet application creates the transaction outputs (payments). + + +==== Creating the Outputs + +A transaction output((("outputs", "constructing transactions"))) is +created with a +script that says something like, "This output is paid to whoever can +present a signature from the key corresponding to Bob's public address." +Because only Bob has the wallet with the keys corresponding to that +address, only Bob's wallet can present such a signature to later spend this +output. Alice will therefore _encumber_ the output value with a demand +for a signature from Bob. + +This transaction will also include a second output ((("change output")))because Alice's +funds contain more money than the cost of the +podcast. Alice's change +output is created in the very same +transaction as the payment to Bob. Essentially, Alice's wallet breaks +her funds into two outputs: one to Bob and one back to herself. She can +then spend the change output in a subsequent transaction. + +Finally, for the transaction to be processed by the network in a((("transaction fees"))) timely +fashion, Alice's wallet application will add a small fee. The fee is not +explicitly stated in the transaction; it is implied by the difference in value between +inputs and outputs. This transaction fee is collected by the +miner as a fee for including the transaction in a block +that gets recorded on the blockchain. + +[[transaction-alice-url]] +[TIP] +==== +View the https://oreil.ly/GwBq1[transaction from Alice to Bob's Store]. +==== + +==== Adding the Transaction to the Blockchain + +The ((("transactions", "adding to blockchain", id="transaction-add-blockchain")))((("blockchain", "adding transactions to", id="blockchain-add-transaction")))transaction created by Alice's wallet application +contains everything necessary to confirm ownership of the funds and +assign new owners. Now, the transaction must be transmitted to the +Bitcoin network where it will become part of the blockchain. In the next +section we will see how a transaction becomes part of a new block and +how the block is mined. Finally, we will see how the new block, once +added to the blockchain, is increasingly trusted by the network as more +blocks are added. + +===== Transmitting the transaction + +Because the transaction contains all +the information necessary for it to be processed, it does not matter how or where it +is transmitted to the Bitcoin network. The Bitcoin network is a +peer-to-peer network, with each Bitcoin peer participating by +connecting to several other Bitcoin peers. The purpose of the Bitcoin +network is to propagate transactions and blocks to all participants. + +===== How it propagates + + +Peers in ((("Bitcoin", "as peer-to-peer network", secondary-sortas="peer-to-peer network")))((("peer-to-peer networks, Bitcoin as")))the Bitcoin peer-to-peer network are programs that have both +the software logic and the data necessary for them to fully verify the +correctness of a new transaction. The connections between peers are +often visualized as edges (lines) in a graph, with the peers themselves +being the nodes (dots). For that reason, Bitcoin peers are commonly +called "full verification nodes," ((("full nodes")))or _full nodes_ for short. + +Alice's wallet application can send the new +transaction to any Bitcoin node over any type of +connection: wired, WiFi, mobile, etc. It can also send the transaction +to another program (such as a block explorer) that will relay it to a +node. Her Bitcoin wallet does not have +to be connected to Bob's Bitcoin wallet directly and she does not have +to use the internet connection offered by Bob, though both those +options are possible too. Any Bitcoin node that receives a +valid transaction it has not seen before will forward it to +all other nodes to which it is connected, a propagation technique known +((("gossiping")))as _gossiping_. Thus, the transaction rapidly propagates out across the +peer-to-peer network, reaching a large percentage of the nodes within a +few seconds. + +===== Bob's view + +If Bob's Bitcoin wallet application is directly connected to Alice's +wallet application, Bob's wallet application might be the first to +receive the transaction. However, even if Alice's wallet sends the +transaction through other nodes, it will reach Bob's wallet within a few +seconds. Bob's wallet will immediately identify Alice's transaction as +an incoming payment because it contains an output redeemable by Bob's +keys. Bob's wallet application can also independently verify that the +transaction is well formed. If Bob is using his own full node, his +wallet can further verify Alice's transaction only spends((("transactions", "constructing", startref="transaction-construct")))((("transactions", "adding to blockchain", startref="transaction-add-blockchain")))((("blockchain", "adding transactions to", startref="blockchain-add-transaction"))) valid UTXOs. + +=== Bitcoin Mining + +Alice's transaction((("mining", "operational overview", id="mining-operational-overview"))) is now propagated on the Bitcoin +network. It does not become part of the _blockchain_ until it is +included in a block by a process called _mining_ and that block has been +validated by full nodes. See +<> for a detailed explanation. + +Bitcoin's system of counterfeit protection is based on computation. +Transactions are bundled((("transactions", "in blocks", secondary-sortas="blocks")))((("blocks"))) into _blocks_. Blocks have a very small header +that must be formed in a very specific way, requiring an enormous +amount of computation to get right--but only a small amount of +computation to verify as correct. +The mining process serves two purposes in Bitcoin: + +[role="less_space pagebreak-before"] +* Miners can only +receive honest income from creating blocks that follow all of((("consensus rules"))) Bitcoin's +_consensus rules_. Therefore, miners are normally incentivized to +only include valid transactions in their blocks and the blocks they +build upon. This allows users to optionally make a trust-based +assumption that any transaction in a block is a valid transaction. + +* Mining currently creates new bitcoins in each block, almost like a central bank +printing new money. The amount of bitcoin created per block is limited +and diminishes with time, following a fixed issuance schedule. + + +Mining achieves a fine balance between cost and reward. Mining uses +electricity to solve a computational problem. A successful miner will +collect ((("rewards")))a _reward_ in the form of new bitcoins and transaction fees. +However, the reward will only be collected if the miner has only +included valid transactions, with the Bitcoin protocol's rules for +_consensus_ determining what is valid. This delicate balance provides +security for Bitcoin without a central authority. + +Mining is designed to be a decentralized lottery. Each miner can create +their own lottery ticket by creating ((("candidate blocks")))a _candidate block_ that includes +the new transactions they want to mine plus some additional data fields. +The miner inputs their candidate into a specially designed algorithm that +scrambles((("hash functions"))) (or "hashes") the data, producing output that looks nothing +like the input data. This _hash_ function will always produce the same +output for the same input--but nobody can predict what the output will +look like for a new input, even if it is only slightly different from a +previous input. If the output of the hash function matches a template +determined by the Bitcoin protocol, the miner wins the lottery and +Bitcoin users will accept the block with its transactions as a +valid block. If the output doesn't match the template, the miner makes +a small change to their candidate block and tries again. As of this +writing, the number of candidate blocks miners need to try before finding +a winning combination is about 168 billion trillion. That's also how +many times the hash function needs to be run. + +However, once a winning combination has been found, anyone can verify +the block is valid by running the hash function just once. That makes a +valid block something that requires an incredible amount of work to +create but only a trivial amount of work to verify. The simple +verification process is able to probabalistically prove the work was +done, so the data necessary to generate that proof--in this case, the +block--is called((("proof-of-work algorithm"))) _proof of work (PoW)_. + +Transactions are added to the new block, prioritized by the highest fee rate +transactions first and a few other criteria. Each miner starts the +process of mining a new candidate block of transactions as soon as they receive the +previous block from the network, knowing that some other miner won that +iteration of the lottery. They immediately create a new candidate block +with a commitment to the previous block, fill it with transactions, and start +calculating the PoW for the candidate block. Each miner includes a +special transaction in their candidate blocks, one that pays their own Bitcoin address +the block reward plus the sum of +transaction fees from all the transactions included in the candidate block. If they +find a solution that makes the candidate into a valid block, they receive this reward +after their successful block is added to the global blockchain and the +reward transaction they included becomes spendable. Miners who participate in a mining pool have set up their +software to create candidate blocks that assign the reward to a pool address. +From there, a share of the reward is distributed to members of the pool +miners in proportion to the amount of work they contributed. + +Alice's +transaction was picked up by the network and included in the pool of +unverified transactions. Once validated by a full node, it was +included in a candidate block. +Approximately five minutes after the transaction was first transmitted +by Alice's wallet, a miner finds a solution for the +block and announces it to the network. After each other miner +validates the winning block, they start a new lottery to generate the next +block. + +The winning block containing Alice's transaction became part of the +blockchain. The block containing Alice's transaction is counted as ((("confirmations")))((("clearing transactions")))((("transactions", "clearing")))one +_confirmation_ of that transaction. After the block containing Alice's +transaction has propagated through the network, creating an alternative +block with a different version of Alice's transaction (such as a +transaction that doesn't pay Bob) would require performing the same +amount of work as it will take all Bitcoin miners to create an entirely +new block. When there are multiple alternative blocks to choose from, +Bitcoin full nodes choose the chain of valid blocks with the most total +PoW, called the _best blockchain_. For the entire network to +accept an alternative block, an additional new block would need to be +mined on top of the alternative. + +That means miners have a choice. They can work with Alice on an +alternative to the transaction where she pays Bob, perhaps with +Alice paying miners a share of the money she previously paid Bob. This +dishonest behavior will require they expend the effort required to +create two new blocks. Instead, miners who behave honestly can create a +single new block and receive all of the fees from the transactions +they include in it, plus the block subsidy. Normally, the high cost of +dishonestly creating two blocks for a small additional payment is much +less profitable than honestly creating a new block, making it unlikely +that a confirmed transaction will be deliberately changed. For Bob, this +means that he can begin to believe that the payment from Alice can be +relied upon. + +[TIP] +==== +You can see the block that includes +https://oreil.ly/7v_lH[Alice's transaction]. +==== + +Approximately 19 minutes +after the block containing Alice's transaction is broadcast, a new block +is mined by another miner. Because this +new block is built on top of the block that contained Alice's +transaction (giving Alice's transaction two confirmations), Alice's +transaction can now only be changed if two alternative blocks are +mined--plus a new block built on top of them--for a total of three +blocks that would need to be mined for Alice to take back the money she +sent Bob. Each block mined on top of the one containing Alice's +transaction counts as an additional confirmation. As the blocks pile on +top of each other, it becomes harder to reverse the transaction, thereby +giving Bob more and more confidence that Alice's payment is secure. + +In <>, we can see the block that contains Alice's transaction. Below it are +hundreds of thousands of blocks, linked to each other in a chain of +blocks (blockchain) all the way back to block #0, known as((("genesis block"))) the _genesis +block_. Over time, as the "height" of new blocks increases, so does the +computation difficulty for the chain as a whole. +By convention, any block with more than six confirmations +is considered very hard to change, because it would require an immense amount of +computation to recalculate six blocks (plus one new block). We will examine +the process of mining and the way it builds confidence in more ((("mining", "operational overview", startref="mining-operational-overview")))detail in +<>. + +[[block-alice1]] +.Alice's transaction included in a block. +image::images/mbc3_0207.png["Alice's transaction included in a block"] + +[role="less_space pagebreak-before"] +=== Spending the Transaction + +Now((("transactions", "spending bitcoins", id="transaction-spend2")))((("bitcoins", "spending", id="bitcoin-spend2")))((("spending bitcoins", id="spend-bitcoin2"))) that Alice's +transaction has been embedded in the blockchain as part of a block, it +is visible to all Bitcoin +applications. Each Bitcoin full node can independently verify the +transaction as valid and spendable. Full nodes validate every transfer +of the funds from the moment the bitcoins were first generated in +a block through each subsequent transaction until they reach +Bob's address. Lightweight clients can partially verify payments +by confirming that the +transaction is in the blockchain and has several blocks mined after it, +thus providing assurance that the miners expended significant effort +committing to it (see <>). + +Bob can now spend the output from this and other transactions. For +example, Bob can pay a contractor or supplier by transferring value from +Alice's podcast payment to these new owners. +As Bob spends the payments received from Alice and other customers, he +extends the chain of transactions. Let's assume that Bob pays his web +designer Gopesh +for a new website page. Now the chain of transactions will +look like <>. + +[[block-alice2]] +.Alice's transaction as part of a transaction chain from Joe to Gopesh. +image::images/mbc3_0208.png["Alice's transaction as part of a transaction chain"] + +In this chapter, we saw how transactions build a chain that moves value +from owner to owner. We also tracked Alice's transaction from the +moment it was created in her wallet, through the Bitcoin network, and to +the miners who recorded it on the blockchain. In the rest of this book, +we will examine the specific technologies behind wallets, addresses, +signatures, transactions, the network, and finally, mining. diff --git a/ch03.asciidoc b/ch03.asciidoc deleted file mode 100644 index 1f41cdf7..00000000 --- a/ch03.asciidoc +++ /dev/null @@ -1,800 +0,0 @@ -[[ch03_bitcoin_client]] -== Bitcoin Core: The Reference Implementation - -((("open source licenses")))((("Nakamoto, Satoshi")))Bitcoin is an _open source_ project and the source code is available under an open (MIT) license, free to download and use for any purpose. Open source means more than simply free to use. It also means that bitcoin is developed by an open community of volunteers. At first, that community consisted of only Satoshi Nakamoto. By 2016, bitcoin's source code had more than 400 contributors with about a dozen developers working on the code almost full-time and several dozen more on a part-time basis. Anyone can contribute to the code—including you! - - -((("bitcoin whitepaper")))((("Satoshi client")))((("reference implementation", see="Bitcoin Core")))((("Bitcoin Core", "reference implementation")))When bitcoin was created by Satoshi Nakamoto, the software was actually completed before the whitepaper reproduced in <> was written. Satoshi wanted to make sure it worked before writing about it. That first implementation, then simply known as "Bitcoin" or "Satoshi client," has been heavily modified and improved. It has evolved into what is known as _Bitcoin Core_, to differentiate it from other compatible implementations. Bitcoin Core is the _reference implementation_ of the Bitcoin system, meaning that it is the authoritative reference on how each part of the technology should be implemented. Bitcoin Core implements all aspects of bitcoin, including wallets, a transaction and block validation engine, and a full network node in the peer-to-peer Bitcoin network. - -[WARNING] -==== -((("wallets", "best practices for")))((("bitcoin improvement proposals", "Mnemonic Code Words (BIP-39)")))((("bitcoin improvement proposals", "Hierarchical Deterministic Wallets (BIP-32/BIP-44)")))Even though Bitcoin Core includes a reference implementation of a wallet, this is not intended to be used as a production wallet for users or for applications. Application developers are advised to build wallets using modern standards such as BIP-39 and BIP-32 (see <> and <>). BIP stands for _Bitcoin Improvement Proposal_. -==== - -<> shows the architecture of Bitcoin Core.((("Bitcoin Core", "architecture"))) - -[[bitcoin_core_architecture]] -.Bitcoin Core architecture (Source: Eric Lombrozo) -image::images/mbc2_0301.png["Bitcoin Core Architecture"] - - -=== Bitcoin Development Environment - -((("development environment", "setup", see="Bitcoin Core")))If you're a developer, you will want to set up a development environment with all the tools, libraries, and support software for writing bitcoin applications. In this highly technical chapter, we'll walk through that process step-by-step. If the material becomes too dense (and you're not actually setting up a development environment) feel free to skip to the next chapter, which is less technical. - -[[compiling_core]] -=== Compiling Bitcoin Core from the Source Code - -((("Bitcoin Core", "compiling from source code", id="BCsource03")))((("Bitcoin Core", "compiling from source code", "downloading")))((("code examples, obtaining and using")))Bitcoin Core's source code can be downloaded as an archive or by cloning the authoritative source repository from GitHub. ((("Bitcoin Core downloads")))On the https://bitcoincore.org/bin/[Bitcoin Core download page], select the most recent version and download the compressed archive of the source code, e.g., +bitcoin-0.15.0.2.tar.gz+. ((("GitHub bitcoin page")))Alternatively, use the git command line to create a local copy of the source code from the https://github.com/bitcoin/bitcoin[GitHub bitcoin page]. - -[TIP] -==== -((("$ symbol")))((("shell commands")))((("terminal applications")))In many of the examples in this chapter we will be using the operating system's command-line interface (also known as a "shell"), accessed via a "terminal" application. The shell will display a prompt; you type a command; and the shell responds with some text and a new prompt for your next command. The prompt may look different on your system, but in the following examples it is denoted by a +$+ symbol. In the examples, when you see text after a +$+ symbol, don't type the +$+ symbol but type the command immediately following it, then press Enter to execute the command. In the examples, the lines below each command are the operating system's responses to that command. When you see the next +$+ prefix, you'll know it's a new command and you should repeat the process. -==== - -((("cloning source code")))((("source code, cloning", seealso="Bitcoin Core")))In this example, we are using the +git+ command to create a local copy ("clone") of the source code: - ----- -$ git clone https://github.com/bitcoin/bitcoin.git -Cloning into 'bitcoin'... -remote: Counting objects: 102071, done. -remote: Compressing objects: 100% (10/10), done. -Receiving objects: 100% (102071/102071), 86.38 MiB | 730.00 KiB/s, done. -remote: Total 102071 (delta 4), reused 5 (delta 1), pack-reused 102060 -Resolving deltas: 100% (76168/76168), done. -Checking connectivity... done. -$ ----- - -[TIP] -==== -((("distributed version control systems")))Git is the most widely used distributed version control system, an essential part of any software developer's toolkit. You may need to install the +git+ command, or a graphical user interface for git, on your operating system if you do not have it already. -==== - -When the git cloning operation has completed, you will have a complete local copy of the source code repository in the directory _bitcoin_. Change to this directory by typing ++**cd bitcoin**++ at the prompt: - ----- -$ cd bitcoin ----- - -==== Selecting a Bitcoin Core Release -((("Bitcoin Core", "compiling from source code", "version selection")))By default, the local copy will be synchronized with the most recent code, which might be an unstable or beta version of bitcoin. Before compiling the code, select a specific version by checking out a release _tag_. This will synchronize the local copy with a specific snapshot of the code repository identified by a keyword tag. Tags are used by the developers to mark specific releases of the code by version number. First, to find the available tags, we use the +git tag+ command: - ----- -$ git tag -v0.1.5 -v0.1.6test1 -v0.10.0 -... -v0.11.2 -v0.11.2rc1 -v0.12.0rc1 -v0.12.0rc2 -... ----- - -The list of tags shows all the released versions of bitcoin. By convention, _release candidates_, which are intended for testing, have the suffix "rc." Stable releases that can be run on production systems have no suffix. From the preceding list, select the highest version release, which at the time of writing was v0.15.0. To synchronize the local code with this version, use the +git checkout+ command: - ----- -$ git checkout v0.15.0 -HEAD is now at 3751912... Merge #11295: doc: Old fee_estimates.dat are discarded by 0.15.0 ----- - -You can confirm you have the desired version "checked out" by issuing the command +git status+: - ----- -$ git status -HEAD detached at v0.15.0 -nothing to commit, working directory clean ----- - -==== Configuring the Bitcoin Core Build - -((("Bitcoin Core", "compiling from source code", "build configuration")))((("documentation")))((("build documentation", seealso="Bitcoin Core")))The source code includes documentation, which can be found in a number of files. Review the main documentation located in _README.md_ in the _bitcoin_ directory by typing ++**more README.md**++ at the prompt and using the spacebar to progress to the next page. In this chapter, we will build the command-line Bitcoin client, also known as +bitcoind+ on Linux. Review the instructions for compiling the +bitcoind+ command-line client on your platform by typing ++**more doc/build-unix.md**++. Alternative instructions for macOS and Windows can be found in the _doc_ directory, as _build-osx.md_ or _build-windows.md_, respectively. - -Carefully review the build prerequisites, which are in the first part of the build documentation. These are libraries that must be present on your system before you can begin to compile bitcoin. If these prerequisites are missing, the build process will fail with an error. If this happens because you missed a prerequisite, you can install it and then resume the build process from where you left off. Assuming the prerequisites are installed, you start the build process by generating a set of build scripts using the _autogen.sh_ script. - ----- -$ ./autogen.sh -... -glibtoolize: copying file 'build-aux/m4/libtool.m4' -glibtoolize: copying file 'build-aux/m4/ltoptions.m4' -glibtoolize: copying file 'build-aux/m4/ltsugar.m4' -glibtoolize: copying file 'build-aux/m4/ltversion.m4' -... -configure.ac:10: installing 'build-aux/compile' -configure.ac:5: installing 'build-aux/config.guess' -configure.ac:5: installing 'build-aux/config.sub' -configure.ac:9: installing 'build-aux/install-sh' -configure.ac:9: installing 'build-aux/missing' -Makefile.am: installing 'build-aux/depcomp' -... ----- - -The _autogen.sh_ script creates a set of automatic configuration scripts that will interrogate your system to discover the correct settings and ensure you have all the necessary libraries to compile the code. The most important of these is the +configure+ script that offers a number of different options to customize the build process. Type ++**./configure --help**++ to see the various options: - ----- -$ ./configure --help -`configure' configures Bitcoin Core 0.15.0 to adapt to many kinds of systems. - -Usage: ./configure [OPTION]... [VAR=VALUE]... - -... -Optional Features: - --disable-option-checking ignore unrecognized --enable/--with options - --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) - --enable-FEATURE[=ARG] include FEATURE [ARG=yes] - - --enable-wallet enable wallet (default is yes) - - --with-gui[=no|qt4|qt5|auto] -... ----- - -The +configure+ script allows you to enable or disable certain features of +bitcoind+ through the use of the +--enable-FEATURE+ and +--disable-FEATURE+ flags, where pass:[FEATURE] is replaced by the feature name, as listed in the help output. In this chapter, we will build the +bitcoind+ client with all the default features. We won't be using the configuration flags, but you should review them to understand what optional features are part of the client. If you are in an academic setting, computer lab restrictions may require you to install applications in your home directory (e.g., using +--prefix=$HOME+). - -Here are some useful options that override the default behavior of the configure script: - -++++ -
-
--prefix=$HOME
-

This overrides the default installation location (which is /usr/local/) for the resulting executable. Use $HOME to put everything in your home directory, or a different path.

- -
--disable-wallet
-

This is used to disable the reference wallet implementation.

- -
--with-incompatible-bdb
-

If you are building a wallet, allow the use of an incompatible version of the Berkeley DB library.

- -
--with-gui=no
-

Don't build the graphical user interface, which requires the Qt library. This builds server and command-line bitcoin only.

-
-++++ - -Next, run the +configure+ script to automatically discover all the necessary libraries and create a customized build script for your system: - ----- -$ ./configure -checking build system type... x86_64-unknown-linux-gnu -checking host system type... x86_64-unknown-linux-gnu -checking for a BSD-compatible install... /usr/bin/install -c -checking whether build environment is sane... yes -checking for a thread-safe mkdir -p... /bin/mkdir -p -checking for gawk... gawk -checking whether make sets $(MAKE)... yes -... -[many pages of configuration tests follow] -... -$ ----- - - -If all went well, the +configure+ command will end by creating the customized build scripts that will allow us to compile +bitcoind+. If there are any missing libraries or errors, the +configure+ command will terminate with an error instead of creating the build scripts. If an error occurs, it is most likely because of a missing or incompatible library. Review the build documentation again and make sure you install the missing prerequisites. Then run +configure+ again and see if that fixes the error. - -==== Building the Bitcoin Core Executables - -((("Bitcoin Core", "compiling from source code", "core executables")))((("core executables", seealso="Bitcoin Core")))Next, you will compile the source code, a process that can take up to an hour to complete, depending on the speed of your CPU and available memory. During the compilation process you should see output every few seconds or every few minutes, or an error if something goes wrong. If an error occurs, or the compilation process is interrupted, it can be resumed any time by typing +make+ again. Type ++**make**++ to start compiling the executable application: - ----- -$ make -Making all in src - CXX crypto/libbitcoinconsensus_la-hmac_sha512.lo - CXX crypto/libbitcoinconsensus_la-ripemd160.lo - CXX crypto/libbitcoinconsensus_la-sha1.lo - CXX crypto/libbitcoinconsensus_la-sha256.lo - CXX crypto/libbitcoinconsensus_la-sha512.lo - CXX libbitcoinconsensus_la-hash.lo - CXX primitives/libbitcoinconsensus_la-transaction.lo - CXX libbitcoinconsensus_la-pubkey.lo - CXX script/libbitcoinconsensus_la-bitcoinconsensus.lo - CXX script/libbitcoinconsensus_la-interpreter.lo - -[... many more compilation messages follow ...] - -$ ----- - -On a fast system with more than one CPU, you might want to set the number of parallel compile jobs. For instance, +make -j 2+ will use two cores if they are available. If all goes well, Bitcoin Core is now compiled. You should run the unit test suite with +make check+ to ensure the linked libraries are not broken in obvious ways. The final step is to install the various executables on your system using the +make install+ command. You may be prompted for your user password, because this step requires administrative privileges: - ----- -$ make check && sudo make install -Password: -Making install in src - ../build-aux/install-sh -c -d '/usr/local/lib' -libtool: install: /usr/bin/install -c bitcoind /usr/local/bin/bitcoind -libtool: install: /usr/bin/install -c bitcoin-cli /usr/local/bin/bitcoin-cli -libtool: install: /usr/bin/install -c bitcoin-tx /usr/local/bin/bitcoin-tx -... -$ ----- - -((("", startref="BCsource03")))The default installation of +bitcoind+ puts it in _/usr/local/bin_. You can confirm that Bitcoin Core is correctly installed by asking the system for the path of the executables, as follows: - ----- -$ which bitcoind -/usr/local/bin/bitcoind - -$ which bitcoin-cli -/usr/local/bin/bitcoin-cli ----- - -=== Running a Bitcoin Core Node - -((("Bitcoin Core", "running core nodes", id="BCnode03")))((("Bitcoin nodes", "running core nodes", id="BNcore03")))Bitcoin's peer-to-peer network is composed of network "nodes," run mostly by volunteers and some of the businesses that build bitcoin applications. Those running Bitcoin nodes have a direct and authoritative view of the Bitcoin blockchain, with a local copy of all the transactions, independently validated by their own system. By running a node, you don't have to rely on any third party to validate a transaction. Moreover, by running a Bitcoin node you contribute to the Bitcoin network by making it more robust. - -Running a node, however, requires a permanently connected system with enough resources to process all bitcoin transactions. Depending on whether you choose to index all transactions and keep a full copy of the blockchain, you may also need a lot of disk space and RAM. As of early 2021, a full-index node needs 2 GB of RAM and a minimum of 360 GB of disk space (see https://www.blockchain.com/charts/blocks-size[]). Bitcoin nodes also transmit and receive bitcoin transactions and blocks, consuming internet bandwidth. If your internet connection is limited, has a low data cap, or is metered (charged by the gigabit), you should probably not run a Bitcoin node on it, or run it in a way that constrains its bandwidth (see <>). - -[TIP] -==== -((("warnings and cautions", "core node resource requirements")))((("resource requirements")))Bitcoin Core keeps a full copy of the blockchain by default, with every transaction that has ever occurred on the Bitcoin network since its inception in 2009. This dataset is dozens of gigabytes in size and is downloaded incrementally over several days or weeks, depending on the speed of your CPU and internet connection. Bitcoin Core will not be able to process transactions or update account balances until the full blockchain dataset is downloaded. Make sure you have enough disk space, bandwidth, and time to complete the initial synchronization. You can configure Bitcoin Core to reduce the size of the blockchain by discarding old blocks (see <>), but it will still download the entire dataset before discarding data. -==== - -Despite these resource requirements, thousands of volunteers run Bitcoin nodes. Some are running on systems as simple as a Raspberry Pi (a $35 USD computer the size of a pack of cards). Many volunteers also run Bitcoin nodes on rented servers, usually some variant of Linux. A _Virtual Private Server_ (VPS) or _Cloud Computing Server_ instance can be used to run a Bitcoin node. Such servers can be rented for $25 to $50 USD per month from a variety of providers. - -Why would you want to run a node? Here are some of the most common reasons: - -* If you are developing bitcoin software and need to rely on a Bitcoin node for programmable (API) access to the network and blockchain. - -* If you are building applications that must validate transactions according to bitcoin's consensus rules. Typically, bitcoin software companies run several nodes. - -* If you want to support bitcoin. Running a node makes the network more robust and able to serve more wallets, more users, and more transactions. - -* If you do not want to rely on any third party to process or validate your transactions. - -If you're reading this book and interested in developing bitcoin software, you should be running your own node. - -==== Configuring the Bitcoin Core Node - -((("Bitcoin Core", "running core nodes", "configuring")))((("warnings and cautions", "password creation")))((("passwords", "creating")))((("security", "passwords")))Bitcoin Core will look for a configuration file in its data directory on every start. In this section we will examine the various configuration options and set up a configuration file. To locate the configuration file, run +bitcoind -printtoconsole+ in your terminal and look for the first couple of lines. - ----- -$ bitcoind -printtoconsole -Bitcoin version v0.15.0 -Using the 'standard' SHA256 implementation -Using data directory /home/ubuntu/.bitcoin/ -Using config file /home/ubuntu/.bitcoin/bitcoin.conf -... -[a lot more debug output] -... ----- - -You can hit Ctrl-C to shut down the node once you determine the location of the config file. Usually the configuration file is inside the _.bitcoin_ data directory under your user's home directory. It is not created automatically, but you can create a starter config file by copying and pasting from the <<#full_index_node>> example, below. You can create or modify the configuration file in your preferred editor. - -Bitcoin Core offers more than 100 configuration options that modify the behavior of the network node, the storage of the blockchain, and many other aspects of its operation. To see a listing of these options, run +bitcoind --help+: - ----- -$ bitcoind --help -Bitcoin Core Daemon version v0.15.0 - -Usage: - bitcoind [options] Start Bitcoin Core Daemon - -Options: - - -? - Print this help message and exit - - -version - Print version and exit - - -alertnotify= - Execute command when a relevant alert is received or we see a really - long fork (%s in cmd is replaced by message) -... -[many more options] -... - - -rpcthreads= - Set the number of threads to service RPC calls (default: 4) ----- - -((("configuration options", seealso="Bitcoin Core")))Here are some of the most important options that you can set in the configuration file, or as command-line parameters to +bitcoind+: - -alertnotify:: Run a specified command or script to send emergency alerts to the owner of this node, usually by email. - -conf:: An alternative location for the configuration file. This only makes sense as a command-line parameter to +bitcoind+, as it can't be inside the configuration file it refers to. - -datadir:: Select the directory and filesystem in which to put all the blockchain data. By default this is the _.bitcoin_ subdirectory of your home directory. Make sure this filesystem has several gigabytes of free space. - -prune:: Reduce the disk space requirements to this many megabytes, by deleting old blocks. Use this on a resource-constrained node that can't fit the full blockchain. - -txindex:: Maintain an index of all transactions. This means a complete copy of the blockchain that allows you to programmatically retrieve any transaction by ID. - -dbcache:: The size of the UTXO cache. The default is 450 MiB. Increase this on high-end hardware and reduce the size on low-end hardware to save memory at the expense of slow disk IO. - -maxconnections:: Set the maximum number of nodes from which to accept connections. Reducing this from the default will reduce your bandwidth consumption. Use if you have a data cap or pay by the gigabyte. - -maxmempool:: Limit the transaction memory pool to this many megabytes. Use it to reduce memory use on memory-constrained nodes. - -maxreceivebuffer/maxsendbuffer:: Limit per-connection memory buffer to this many multiples of 1000 bytes. Use on memory-constrained nodes. - -minrelaytxfee:: Set the minimum fee rate for transaction you will relay. Below this value, the transaction is treated nonstandard, rejected from the transaction pool and not relayed. - - -[[txindex]] -.Transaction Database Index and txindex Option -**** -((("Bitcoin Core", "running core nodes", "database options")))((("transactions", "database configuration options")))((("txindex option")))((("full indexing option")))By default, Bitcoin Core builds a database containing _only_ the transactions related to the user's wallet. If you want to be able to access _any_ transaction with commands like +getrawtransaction+ (see <>), you need to configure Bitcoin Core to build a complete transaction index, which can be achieved with the +txindex+ option. Set +txindex=1+ in the Bitcoin Core configuration file. If you don't set this option at first and later set it to full indexing, you need to restart +bitcoind+ with the +-reindex+ option and wait for it to rebuild the index. -**** - -<> shows how you might combine the preceding options, with a fully indexed node, running as an API backend for a bitcoin application. - -[[full_index_node]] -.Sample configuration of a full-index node -==== ----- -alertnotify=myemailscript.sh "Alert: %s" -datadir=/lotsofspace/bitcoin -txindex=1 ----- -==== - -<> shows a resource-constrained node running on a smaller server. - -[[constrained_resources]] -.Sample configuration of a resource-constrained system -==== ----- -alertnotify=myemailscript.sh "Alert: %s" -maxconnections=15 -prune=5000 -dbcache=150 -maxmempool=150 -maxreceivebuffer=2500 -maxsendbuffer=500 ----- -==== - -Once you've edited the configuration file and set the options that best represent your needs, you can test +bitcoind+ with this configuration. Run Bitcoin Core with the option +printtoconsole+ to run in the foreground with output to the console: - ----- -$ bitcoind -printtoconsole - -Bitcoin version v0.15.0 -InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1 -Assuming ancestors of block 0000000000000000003b9ce759c2a087d52abc4266f8f4ebd6d768b89defa50a have valid signatures. -Using the 'standard' SHA256 implementation -Default data directory /home/ubuntu/.bitcoin -Using data directory /lotsofspace/.bitcoin -Using config file /home/ubuntu/.bitcoin/bitcoin.conf -Using at most 125 automatic connections (1048576 file descriptors available) -Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements -Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements -Using 2 threads for script verification -HTTP: creating work queue of depth 16 -No rpcpassword set - using random cookie authentication -Generated RPC authentication cookie /lotsofspace/.bitcoin/.cookie -HTTP: starting 4 worker threads -init message: Verifying wallet(s)... -Using BerkeleyDB version Berkeley DB 4.8.30: (April 9, 2010) -Using wallet wallet.dat -CDBEnv::Open: LogDir=/lotsofspace/.bitcoin/database ErrorFile=/lotsofspace/.bitcoin/db.log -scheduler thread start -Cache configuration: -* Using 250.0MiB for block index database -* Using 8.0MiB for chain state database -* Using 1742.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space) -init message: Loading block index... -Opening LevelDB in /lotsofspace/.bitcoin/blocks/index -Opened LevelDB successfully - -[... more startup messages ...] ----- - -You can hit Ctrl-C to interrupt the process once you are satisfied that it is loading the correct settings and running as you expect. - -To run Bitcoin Core in the background as a process, start it with the +daemon+ option, as +bitcoind -daemon+. - -To monitor the progress and runtime status of your Bitcoin node, use the command +bitcoin-cli getblockchaininfo+: - ----- -$ bitcoin-cli getblockchaininfo ----- - -[source,json] ----- -{ - "chain": "main", - "blocks": 0, - "headers": 83999, - "bestblockhash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f", - "difficulty": 1, - "mediantime": 1231006505, - "verificationprogress": 3.783041623201835e-09, - "chainwork": "0000000000000000000000000000000000000000000000000000000100010001", - "pruned": false, - [...] -} ----- - -This shows a node with a blockchain height of 0 blocks and 83999 headers. The node currently fetches the block headers of the best chain and afterward continues to download the full blocks. - -Once you are happy with the configuration options you have selected, you should add bitcoin to the startup scripts in your operating system, so that it runs continuously and restarts when the operating system restarts. You will find a number of example startup scripts for various operating systems in bitcoin's source directory under _contrib/init_ and a _README.md_ file showing which system uses which script.((("", startref="BCnode03")))((("", startref="BNcore03"))) - -=== Bitcoin Core Application Programming Interface (API) - -((("Bitcoin Core", "Bitcoin Core API", id="BCapi03")))The Bitcoin Core client implements a JSON-RPC interface that can also be accessed using the command-line helper +bitcoin-cli+. The command line allows us to experiment interactively with the capabilities that are also available programmatically via the API. ((("Bitcoin Core", "Bitcoin Core API", "RPC commands")))To start, invoke the +help+ command to see a list of the available bitcoin RPC commands: - -[[bitcoind_commands]] - ----- -$ bitcoin-cli help -addmultisigaddress nrequired ["key",...] ( "account" ) -addnode "node" "add|remove|onetry" -backupwallet "destination" -createmultisig nrequired ["key",...] -createrawtransaction [{"txid":"id","vout":n},...] {"address":amount,...} -decoderawtransaction "hexstring" -... -... -verifymessage "bitcoinaddress" "signature" "message" -walletlock -walletpassphrase "passphrase" timeout -walletpassphrasechange "oldpassphrase" "newpassphrase" ----- - -Each of these commands may take a number of parameters. To get additional help, a detailed description, and information on the parameters, add the command name after +help+. For example, to see help on the +getblockhash+ RPC command: - ----- -$ bitcoin-cli help getblockhash -getblockhash height - -Returns hash of block in best-block-chain at height provided. - -Arguments: -1. height (numeric, required) The height index - -Result: -"hash" (string) The block hash - -Examples: -> bitcoin-cli getblockhash 1000 -> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getblockhash", "params": [1000] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/ ----- - -At the end of the help information you will see two examples of the RPC command, using the +bitcoin-cli+ helper or the HTTP client +curl+. These examples demonstrate how you might call the command. Copy the first example and see the result: - ----- -$ bitcoin-cli getblockhash 1000 -00000000c937983704a73af28acdec37b049d214adbda81d7e2a3dd146f6ed09 ----- - -The result is a block hash, which is described in more detail in the following chapters. But for now, this command should return the same result on your system, demonstrating that your Bitcoin Core node is running, is accepting commands, and has information about block 1000 to return to you. - -In the next sections we will demonstrate some very useful RPC commands and their expected output. - -==== Getting Information on the Bitcoin Core Client Status - -((("Bitcoin Core", "Bitcoin Core API", "status information")))Bitcoin Core provides status reports on different modules through the JSON-RPC interface. The most important commands include +getblockchaininfo+, +getmempoolinfo+, +getnetworkinfo+ and +getwalletinfo+. - -Bitcoin's +getblockchaininfo+ RPC command was introduced earlier. The +getnetworkinfo+ command displays basic information about the status of the Bitcoin network node. Use +bitcoin-cli+ to run it: - ----- -$ bitcoin-cli getnetworkinfo ----- -[source,json] ----- -{ - "version": 150000, - "subversion": "/Satoshi:0.15.0/", - "protocolversion": 70015, - "localservices": "000000000000000d", - "localrelay": true, - "timeoffset": 0, - "networkactive": true, - "connections": 8, - "networks": [ - ... - detailed information about all networks (ipv4, ipv6 or onion) - ... - ], - "relayfee": 0.00001000, - "incrementalfee": 0.00001000, - "localaddresses": [ - ], - "warnings": "" -} - ----- - -The data is returned in JavaScript Object Notation (JSON), a format that can easily be "consumed" by all programming languages but is also quite human-readable. Among this data we see the version numbers for the bitcoin software client (150000) and Bitcoin protocol (70015). We see the current number of connections (8) and various information about the Bitcoin network and the settings related to this client. - -[TIP] -==== -It will take some time, perhaps more than a day, for the +bitcoind+ client to "catch up" to the current blockchain height as it downloads blocks from other Bitcoin clients. You can check its progress using +getblockchaininfo+ to see the number of known blocks. -==== - -[[exploring_and_decoding_transanctions]] -==== Exploring and Decoding Transactions - -((("Bitcoin Core", "Bitcoin Core API", "exploring and decoding transactions")))((("transactions", "exploring with Bitcoin Core API")))Commands: +getrawtransaction+, +decoderawtransaction+ - - - -In <>, ((("use cases", "buying coffee", id="alicethree")))Alice bought a cup of coffee from Bob's Cafe. Her transaction was recorded on the blockchain with transaction ID (+txid+) +0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2+. Let's use the API to retrieve and examine that transaction by passing the transaction ID as a parameter: - -++++ -
-$ bitcoin-cli getrawtransaction 0627052b6f28912f2703066a912ea577f2ce4da4caa5a↵
-5fbd8a57286c345c2f2
-
-0100000001186f9f998a5aa6f048e51dd8419a14d8a0f1a8a2836dd734d2804fe65fa35779000↵
-000008b483045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4↵
-ae24cb02204b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813014↵
-10484ecc0d46f1918b30928fa0e4ed99f16a0fb4fde0735e7ade8416ab9fe423cc54123363767↵
-89d172787ec3457eee41c04f4938de5cc17b4a10fa336a8d752adfffffffff0260e3160000000↵
-0001976a914ab68025513c3dbd2f7b92a94e0581f5d50f654e788acd0ef8000000000001976a9↵
-147f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a888ac00000000
-
-++++ - - -[TIP] -==== -((("transaction IDs (txd)")))((("malleability")))A transaction ID is not authoritative until a transaction has been confirmed. Absence of a transaction hash in the blockchain does not mean the transaction was not processed. This is known as "transaction malleability," because transaction hashes can be modified prior to confirmation in a block. After confirmation, the +txid+ is immutable and authoritative. -==== - -The command +getrawtransaction+ returns a serialized transaction in hexadecimal notation. To decode that, we use the +decoderawtransaction+ command, passing the hex data as a parameter. You can copy the hex returned by +getrawtransaction+ and paste it as a parameter to +decoderawtransaction+: - -++++ -
-$ bitcoin-cli decoderawtransaction 0100000001186f9f998a5aa6f048e51dd8419a14d8↵
-a0f1a8a2836dd734d2804fe65fa35779000000008b483045022100884d142d86652a3f47ba474↵
-6ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039ff08df09cbe9f6addac960298↵
-cad530a863ea8f53982c09db8f6e381301410484ecc0d46f1918b30928fa0e4ed99f16a0fb4fd↵
-e0735e7ade8416ab9fe423cc5412336376789d172787ec3457eee41c04f4938de5cc17b4a10fa↵
-336a8d752adfffffffff0260e31600000000001976a914ab68025513c3dbd2f7b92a94e0581f5↵
-d50f654e788acd0ef8000000000001976a9147f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a8↵
-88ac00000000
-
-
-++++ - -++++ -
-{
-  "txid": "0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2",
-  "size": 258,
-  "version": 1,
-  "locktime": 0,
-  "vin": [
-    {
-      "txid": "7957a35fe64f80d234d76d83a2...8149a41d81de548f0a65a8a999f6f18",
-      "vout": 0,
-      "scriptSig": {
-        "asm":"3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1decc...",
-        "hex":"483045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1de..."
-      },
-      "sequence": 4294967295
-    }
-  ],
-  "vout": [
-    {
-      "value": 0.01500000,
-      "n": 0,
-      "scriptPubKey": {
-        "asm": "OP_DUP OP_HASH160 ab68...5f654e7 OP_EQUALVERIFY OP_CHECKSIG",
-        "hex": "76a914ab68025513c3dbd2f7b92a94e0581f5d50f654e788ac",
-        "reqSigs": 1,
-        "type": "pubkeyhash",
-        "addresses": [
-          "1GdK9UzpHBzqzX2A9JFP3Di4weBwqgmoQA"
-        ]
-      }
-    },
-    {
-      "value": 0.08450000,
-      "n": 1,
-      "scriptPubKey": {
-        "asm": "OP_DUP OP_HASH160 7f9b1a...025a8 OP_EQUALVERIFY OP_CHECKSIG",
-        "hex": "76a9147f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a888ac",
-        "reqSigs": 1,
-        "type": "pubkeyhash",
-        "addresses": [
-          "1Cdid9KFAaatwczBwBttQcwXYCpvK8h7FK"
-        ]
-      }
-    }
-  ]
-}
-
-
-++++ - -The transaction decode shows all the components of this transaction, including the transaction inputs and outputs. In this case we see that the transaction that credited our new address with 15 millibits used one input and generated two outputs. The input to this transaction was the output from a previously confirmed transaction (shown as the vin +txid+ starting with +7957a35fe+). The two outputs correspond to the 15 millibit credit and an output with change back to the sender. - -We can further explore the blockchain by examining the previous transaction referenced by its +txid+ in this transaction using the same commands (e.g., +getrawtransaction+). Jumping from transaction to transaction we can follow a chain of transactions back as the coins are transmitted from owner address to owner address. - - - -==== Exploring Blocks - -((("Bitcoin Core", "Bitcoin Core API", "exploring blocks")))((("blocks", "exploring with Bitcoin Core API")))Commands: +getblock+, +getblockhash+ - -((("blocks", "block height")))((("blocks", "block hash")))Exploring blocks is similar to exploring transactions. However, blocks can be referenced either by the block _height_ or by the block _hash_. First, let's find a block by its height. In <>, we saw that Alice's transaction was included in block 277316. - -We use the +getblockhash+ command, which takes the block height as the parameter and returns the block hash for that block: - -++++ -
-$ bitcoin-cli getblockhash 277316
-0000000000000001b6b9a13b095e96db41c4a928b97ef2d944a9b31b2cc7bdc4
-
-++++ - -Now that we know which block Alice's transaction was included in, we can query that block. We use the +getblock+ command with the block hash as the parameter: - -++++ -
-$ bitcoin-cli getblock 0000000000000001b6b9a13b095e96db41c4a928b97ef2d944a9b3↵
-1b2cc7bdc4
-
-++++ - -++++ -
-{
-  "hash": "0000000000000001b6b9a13b095e96db41c4a928b97ef2d944a9b31b2cc7bdc4",
-  "confirmations": 37371,
-  "size": 218629,
-  "height": 277316,
-  "version": 2,
-  "merkleroot": "c91c008c26e50763e9f548bb8b2fc323735f73577effbc55502c51eb4cc7cf2e",
-  "tx": [
-    "d5ada064c6417ca25c4308bd158c34b77e1c0eca2a73cda16c737e7424afba2f",
-    "b268b45c59b39d759614757718b9918caf0ba9d97c56f3b91956ff877c503fbe",
-    "04905ff987ddd4cfe603b03cfb7ca50ee81d89d1f8f5f265c38f763eea4a21fd",
-    "32467aab5d04f51940075055c2f20bbd1195727c961431bf0aff8443f9710f81",
-    "561c5216944e21fa29dd12aaa1a45e3397f9c0d888359cb05e1f79fe73da37bd",
-[... hundreds of transactions ...]
-    "78b300b2a1d2d9449b58db7bc71c3884d6e0579617e0da4991b9734cef7ab23a",
-    "6c87130ec283ab4c2c493b190c20de4b28ff3caf72d16ffa1ce3e96f2069aca9",
-    "6f423dbc3636ef193fd8898dfdf7621dcade1bbe509e963ffbff91f696d81a62",
-    "802ba8b2adabc5796a9471f25b02ae6aeee2439c679a5c33c4bbcee97e081196",
-    "eaaf6a048588d9ad4d1c092539bd571dd8af30635c152a3b0e8b611e67d1a1af",
-    "e67abc6bd5e2cac169821afc51b207127f42b92a841e976f9b752157879ba8bd",
-    "d38985a6a1bfd35037cb7776b2dc86797abbb7a06630f5d03df2785d50d5a2ac",
-    "45ea0a3f6016d2bb90ab92c34a7aac9767671a8a84b9bcce6c019e60197c134b",
-    "c098445d748ced5f178ef2ff96f2758cbec9eb32cb0fc65db313bcac1d3bc98f"
-  ],
-  "time": 1388185914,
-  "mediantime": 1388183675,
-  "nonce": 924591752,
-  "bits": "1903a30c",
-  "difficulty": 1180923195.258026,
-  "chainwork": "000000000000000000000000000000000000000000000934695e92aaf53afa1a",
-  "previousblockhash": "0000000000000002a7bbd25a417c0374cc55261021e8a9ca74442b01284f0569",
-  "nextblockhash": "000000000000000010236c269dd6ed714dd5db39d36b33959079d78dfd431ba7"
-}
-
-++++ - -The block contains 419 transactions and the 64th transaction listed (+0627052b...+) is Alice's coffee payment. The +height+ entry tells us this is the 277316th block in the blockchain. - -==== Using Bitcoin Core's Programmatic Interface - -((("Bitcoin Core", "Bitcoin Core API", "using programmatic interface")))((("programmatic interface", id="progint03")))The +bitcoin-cli+ helper is very useful for exploring the Bitcoin Core API and testing functions. But the whole point of an application programming interface is to access functions programmatically. In this section we will demonstrate accessing Bitcoin Core from another program. - -Bitcoin Core's API is a JSON-RPC interface. JSON stands for JavaScript Object Notation and it is a very convenient way to present data that both humans and programs can easily read. RPC stands for Remote Procedure Call, which means that we are calling procedures (functions) that are remote (on the Bitcoin Core node) via a network protocol. In this case, the network protocol is HTTP, or HTTPS (for encrypted connections). - -When we used the +bitcoin-cli+ command to get help on a command, it showed us an example of using +curl+, the versatile command-line HTTP client to construct one of these JSON-RPC calls: - ----- -$ curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getblockchaininfo", "params": [] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/ ----- - -This command shows that +curl+ submits an HTTP request to the local host (127.0.0.1), connecting to the default bitcoin port (8332), and submitting a +jsonrpc+ request for the +getblockchaininfo+ method using +text/plain+ encoding. - -You might notice that curl will ask for credentials to be sent along with the request. Bitcoin Core will create a random password on each start and place it in the data directory under the name +.cookie+. The +bitcoin-cli+ helper can read this password file given the data directory. Similarly, you can copy the password and pass it to curl (or any higher level Bitcoin Core RPC wrappers). Alternatively, you can create a static password with the helper script provided in _./share/rpcauth/rpcauth.py_ in Bitcoin Core's source directory. - -If you're implementing a JSON-RPC call in your own program, you can use a generic HTTP library to construct the call, similar to what is shown in the preceding +curl+ example. - -However, there are libraries in most every programming language that "wrap" the Bitcoin Core API in a way that makes this a lot simpler. We will use the +python-bitcoinlib+ library to simplify API access. Remember, this requires you to have a running Bitcoin Core instance, which will be used to make JSON-RPC calls. - -The Python script in <> makes a simple +getblockchaininfo+ call and prints the +blocks+ parameter from the data returned by Bitcoin Core (full node required). - -[[rpc_example]] -.Running getblockchaininfo via Bitcoin Core's JSON-RPC API -==== -[source,python] ----- -include::code/rpc_example.py[] ----- -==== - -Running it gives us the following result: - ----- -$ python rpc_example.py -394075 ----- - -It tells us that our local Bitcoin Core node has 394075 blocks in its blockchain. Not a spectacular result, but it demonstrates the basic use of the library as a simplified interface to Bitcoin Core's JSON-RPC API. - -Next, let's use the +getrawtransaction+ and +decodetransaction+ calls to retrieve the details of Alice's coffee payment. In <>, we retrieve Alice's transaction and list the transaction's outputs. For each output, we show the recipient address and value. As a reminder, Alice's transaction had one output paying Bob's Cafe and one output for change back to Alice. - -[[rpc_transaction]] -.Retrieving a transaction and iterating its outputs -==== -[source,python] ----- -include::code/rpc_transaction.py[] ----- -==== - -Running this code, we get: - ----- -$ python rpc_transaction.py -([u'1GdK9UzpHBzqzX2A9JFP3Di4weBwqgmoQA'], Decimal('0.01500000')) -([u'1Cdid9KFAaatwczBwBttQcwXYCpvK8h7FK'], Decimal('0.08450000')) ----- - -Both of the preceding examples are rather simple. You don't really need a program to run them; you could just as easily use the +bitcoin-cli+ helper. The next example, however, requires several hundred RPC calls and more clearly demonstrates the use of a programmatic interface. - -In <>, we first retrieve block 277316, then retrieve each of the 419 transactions within by reference to each transaction ID. Next, we iterate through each of the transaction's outputs and add up the value.((("", startref="alicethree"))) - -[[rpc_block]] -.Retrieving a block and adding all the transaction outputs -==== -[source,python] ----- -include::code/rpc_block.py[] ----- -==== - -Running this code, we get: - ----- -$ python rpc_block.py - -('Total value in block: ', Decimal('10322.07722534')) ----- - -Our example code calculates that the total value transacted in this block is 10,322.07722534 BTC (including 25 BTC reward and 0.0909 BTC in fees). Compare that to the amount reported by a block explorer site by searching for the block hash or height. Some block explorers report the total value excluding the reward and excluding the fees. See if you can spot the difference.((("", startref="BCapi03")))((("", startref="progint03"))) - -[[alt_libraries]] -=== Alternative Clients, Libraries, and Toolkits - -((("Bitcoin Core", "alternatives to", id="BCalt03")))((("clients, libraries, and toolkits", id="clients03")))((("libraries, clients, and toolkits", id="librar03")))((("toolkits, libraries, and clients", id="toolkit03")))((("third-party API clients", id="thirdpart03")))There are many alternative clients, libraries, toolkits, and even full-node implementations in the bitcoin ecosystem. These are implemented in a variety of programming languages, offering programmers native interfaces in their preferred language. - -The following sections list some of the best libraries, clients, and toolkits, organized by programming languages. - -==== C/C++ -https://github.com/bitcoin/bitcoin[Bitcoin Core] :: The reference implementation of bitcoin -https://github.com/libbitcoin/libbitcoin-system[libbitcoin]:: Cross-platform C++ development toolkit, node, and consensus library -https://github.com/libbitcoin/libbitcoin-explorer[bitcoin explorer]:: Libbitcoin's command-line tool -https://github.com/jgarzik/picocoin[picocoin]:: A C language lightweight client library for bitcoin by Jeff Garzik - -==== JavaScript -https://bcoin.io/[bcoin]:: A modular and scalable full-node implementation with API -https://bitcore.io/[Bitcore] :: Full node, API, and library by Bitpay -https://github.com/bitcoinjs/bitcoinjs-lib[BitcoinJS] :: A pure JavaScript Bitcoin library for node.js and browsers - -==== Java -https://bitcoinj.github.io[bitcoinj]:: A Java full-node client library - -==== PHP -https://github.com/bit-wasp/bitcoin-php[bitwasp/bitcoin]:: A PHP bitcoin library, and related projects - -==== Python -https://github.com/petertodd/python-bitcoinlib[python-bitcoinlib]:: A Python bitcoin library, consensus library, and node by Peter Todd -https://github.com/richardkiss/pycoin[pycoin]:: A Python bitcoin library by Richard Kiss -https://github.com/primal100/pybitcointools[pybitcointools]:: An archived fork of Python bitcoin library by Vitalik Buterin - -==== Ruby -https://github.com/sinisterchipmunk/bitcoin-client[bitcoin-client]:: A Ruby library wrapper for the JSON-RPC API - -==== Go -https://github.com/btcsuite/btcd[btcd]:: A Go language full-node Bitcoin client - -==== Rust -https://github.com/rust-bitcoin/rust-bitcoin[rust-bitcoin]:: Rust bitcoin library for serialization, parsing, and API calls - -==== C# -https://github.com/MetacoSA/NBitcoin[NBitcoin]:: Comprehensive bitcoin library for the .NET framework - -==== Objective-C -https://github.com/oleganza/CoreBitcoin[CoreBitcoin]:: Bitcoin toolkit for ObjC and Swift - -Many more libraries exist in a variety of other programming languages and more are created all the time.((("", startref="BCalt03")))((("", startref="clients03")))((("", startref="thirdpart03")))((("", startref="toolkit03")))((("", startref="librar03"))) diff --git a/ch03_bitcoin-core.adoc b/ch03_bitcoin-core.adoc new file mode 100644 index 00000000..08d6c1e8 --- /dev/null +++ b/ch03_bitcoin-core.adoc @@ -0,0 +1,1176 @@ +[[ch03_bitcoin_client]] +== Bitcoin Core: The Reference Implementation + +People only accept money in exchange for their valuable goods and +services if they believe that they'll be able to spend that money later. +Money that is counterfeit or unexpectedly debased may not be spendable +later, so every person accepting bitcoins has a strong incentive to +verify the integrity of the bitcoins they receive. The Bitcoin system +was designed so that it's possible for software running entirely on your +local computer to perfectly prevent counterfeiting, debasement, and +several other critical problems. Software which provides that function +is called a _full verification node_ because it verifies every confirmed +Bitcoin transaction against every rule in the system. Full ((("full nodes")))verification +nodes, _full nodes_ for short, may also provide tools and data for +understanding how Bitcoin works and what is currently happening in the +network. + +In this chapter, we'll install Bitcoin Core, the implementation that +most full node operators have used since the beginning of the Bitcoin +network. We'll then inspect blocks, transactions, and other data from +your node, data which is authoritative--not because some powerful entity +designated it as such but because your node independently verified it. +Throughout the rest of this book, we'll continue using Bitcoin Core to +create and examine data related to the blockchain and network. + +=== From Bitcoin to Bitcoin Core + +Bitcoin ((("Bitcoin Core", "explained", id="bitcoin-core-explain")))is an _open +source_ project and the source code is available under an open (MIT) +license, free to download and use for any purpose. More than just being +open source, Bitcoin is developed by +an open community of volunteers. At first, that community consisted of +only ((("Nakamoto, Satoshi")))Satoshi Nakamoto. By 2023, Bitcoin's source code had more than 1,000 +contributors with about a dozen developers working on the code almost +full time and several dozen more on a part-time basis. Anyone can +contribute to the code—including you! + +When Bitcoin was created by Satoshi Nakamoto, the +software was mostly completed before publication of the whitepaper (reproduced in +<>). Satoshi wanted to make sure the +implementation worked before publishing a paper about it. That first implementation, then simply +known as "Bitcoin," has been heavily modified and +improved. It has evolved into what is known as _Bitcoin Core_, to +differentiate it from other implementations. Bitcoin Core is +the _reference implementation_ of the Bitcoin system, meaning that it +provides a reference for how each part of the technology should be +implemented. Bitcoin Core implements all aspects of Bitcoin, including +wallets, a transaction and block validation engine, tools for block construction, and all modern parts +of Bitcoin peer-to-peer communication. + +<> shows the architecture of Bitcoin +Core. + +[[bitcoin_core_architecture]] +.Bitcoin Core architecture (Source: Eric Lombrozo). +image::images/mbc3_0301.png["Bitcoin Core Architecture"] + +Although Bitcoin Core serves as a reference implementation for many +major parts of the system, the Bitcoin whitepaper describes several +early parts of the system. Most major parts of the system since 2011 +have been documented in a set of +https://oreil.ly/BCXAQ[Bitcoin Improvement +Proposals (BIPs)]. Throughout((("BIPs (Bitcoin Improvement Proposals)")))((("Bitcoin Improvement Proposals", see="BIPs"))) this book, we refer to BIP specifications +by their number; for example, BIP9 describes a mechanism used for +several major upgrades to((("Bitcoin Core", "explained", startref="bitcoin-core-explain"))) Bitcoin. + +=== Bitcoin Development Environment + +If you're a +developer, you will want to set up a development environment with all +the tools, libraries, and support software for writing Bitcoin +applications. In this highly technical chapter, we'll walk through that +process step by step. If the material becomes too dense (and you're not +actually setting up a development environment) feel free to skip to the +next chapter, which is less technical. + +[[compiling_core]] +=== Compiling Bitcoin Core from the Source Code + +Bitcoin Core's +source code((("Bitcoin Core", "compiling from source code", id="bitcoin-core-compile")))((("compiling Bitcoin Core from source code", id="compile-bitcoin-core")))((("source code, compiling Bitcoin Core", id="source-code-compile"))) can be downloaded as an archive or by cloning the +source repository from GitHub. On the https://oreil.ly/hN9g1[Bitcoin Core download +page], select the most recent version and download the compressed +archive of the source code. Alternatively, use the Git command line to create a +local copy of the source code from the +https://oreil.ly/BdOwl[GitHub Bitcoin page]. + +[TIP] +==== +In +many of the examples in this chapter, we will be using the operating +system's command-line interface (also known as a "shell"), accessed via +a "terminal" application. The shell will display a prompt, you type a +command, and the shell responds with some text and a new prompt for your +next command. The prompt may look different on your system, but in the +following examples, it is denoted by a +$+ symbol. In the examples, when +you see text after a +$+ symbol, don't type the +$+ symbol but type the +command immediately following it, then press Enter to execute the +command. In the examples, the lines below each command are the operating +system's responses to that command. When you see the next +$+ prefix, +you'll know it's a new command and you should repeat the process. +==== + +Here, we use the +git+ command to create a +local copy ("clone") of the source code: + +---- +$ git clone https://github.com/bitcoin/bitcoin.git +Cloning into 'bitcoin'... +remote: Enumerating objects: 245912, done. +remote: Counting objects: 100% (3/3), done. +remote: Compressing objects: 100% (2/2), done. +remote: Total 245912 (delta 1), reused 2 (delta 1), pack-reused 245909 +Receiving objects: 100% (245912/245912), 217.74 MiB | 13.05 MiB/s, done. +Resolving deltas: 100% (175649/175649), done. +---- + +[TIP] +==== +Git is the most widely used +distributed version control system, an essential part of any software +developer's toolkit. You may need to install the +git+ command, or a +graphical user interface for Git, on your operating system if you do not +have it already. +==== + +When the Git cloning operation has completed, you will have a complete +local copy of the source code repository in the directory _bitcoin_. +Change to this directory using the +cd+ command: + +---- +$ cd bitcoin +---- + +==== Selecting a Bitcoin Core Release + +By default,((("Bitcoin Core", "compiling from source code", "selecting release version", id="bitcoin-core-compile-release")))((("compiling Bitcoin Core from source code", "selecting release version", id="compile-bitcoin-core-release")))((("source code, compiling Bitcoin Core", "selecting release version", id="source-code-compile-release")))((("release version (Bitcoin Core), selecting", id="release-select")))((("selecting", "release version (Bitcoin Core)", id="select-release"))) the local copy will be synchronized with the +most recent code, which might be an unstable or beta version of Bitcoin. +Before compiling the code, select a specific version by checking out a +release _tag_. This will synchronize the local copy with a specific +snapshot of the code repository identified by a keyword tag. Tags are +used by the developers to mark specific releases of the code by version +number. First, to find the available tags, we use the +git tag+ command: + +---- +$ git tag +v0.1.5 +v0.1.6test1 +v0.10.0 +... +v0.11.2 +v0.11.2rc1 +v0.12.0rc1 +v0.12.0rc2 +... +---- + +The list of tags shows all the released versions of Bitcoin. By +convention, _release candidates_, which ((("release candidates")))are intended for testing, have +the suffix "rc." Stable releases that can be run on production systems +have no suffix. From the preceding list, select the highest version +release, which at the time of writing was v24.0.1. To synchronize the +local code with this version, use the +git checkout+ command: + +---- +$ git checkout v24.0.1 +Note: switching to 'v24.0.1'. + +You are in 'detached HEAD' state. You can look around, make experimental +changes and commit them, and you can discard any commits you make in this +state without impacting any branches by switching back to a branch. + +HEAD is now at b3f866a8d Merge bitcoin/bitcoin#26647: 24.0.1 final changes +---- + +You can confirm you have the desired version "checked out" by((("Bitcoin Core", "compiling from source code", "selecting release version", startref="bitcoin-core-compile-release")))((("compiling Bitcoin Core from source code", "selecting release version", startref="compile-bitcoin-core-release")))((("source code, compiling Bitcoin Core", "selecting release version", startref="source-code-compile-release")))((("release version (Bitcoin Core), selecting", startref="release-select")))((("selecting", "release version (Bitcoin Core)", startref="select-release"))) issuing +the command +git status+: + +---- +HEAD detached at v24.0.1 +nothing to commit, working tree clean +---- + +==== Configuring the Bitcoin Core Build + +The((("Bitcoin Core", "compiling from source code", "configuring build", id="bitcoin-core-compile-configure")))((("compiling Bitcoin Core from source code", "configuring build", id="compile-bitcoin-core-configure")))((("source code, compiling Bitcoin Core", "configuring build", id="source-code-compile-configure")))((("configuring", "Bitcoin Core build", id="configure-build"))) source code includes documentation, which +can be found in a number of files. Review the main documentation located +in _README.md_ in the _bitcoin_ directory. +In this chapter, we will build the Bitcoin Core daemon +(server), also known as +bitcoind+ on Linux (a Unix-like system). Review the instructions for +compiling the +bitcoind+ command-line client on your platform by reading +_doc/build-unix.md_. Alternative instructions can be found in +the _doc_ directory; for example, _build-windows.md_ for Windows +instructions. As of this writing, instructions are available for +Android, FreeBSD, NetBSD, OpenBSD, macOS (OSX), Unix, and Windows. + +Carefully review the build prerequisites, which are in the first part of +the build documentation. These are libraries that must be present on +your system before you can begin to compile Bitcoin. If these +prerequisites are missing, the build process will fail with an error. If +this happens because you missed a prerequisite, you can install it and +then resume the build process from where you left off. Assuming the +prerequisites are installed, you start the build process by generating a +set of build scripts using the _autogen.sh_ script: + +---- +$ ./autogen.sh +libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'. +libtoolize: copying file 'build-aux/ltmain.sh' +libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'build-aux/m4'. + ... +configure.ac:58: installing 'build-aux/missing' +src/Makefile.am: installing 'build-aux/depcomp' +parallel-tests: installing 'build-aux/test-driver' +---- + +The _autogen.sh_ script creates a set of automatic configuration scripts +that will interrogate your system to discover the correct settings and +ensure you have all the necessary libraries to compile the code. The +most important of these is the +configure+ script that offers a number +of different options to customize the build process. Use the ++--help+ flag to see the various options: + +---- +$ ./configure --help +`configure' configures Bitcoin Core 24.0.1 to adapt to many kinds of systems. + +Usage: ./configure [OPTION]... [VAR=VALUE]... + +... +Optional Features: + --disable-option-checking ignore unrecognized --enable/--with options + --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) + --enable-FEATURE[=ARG] include FEATURE [ARG=yes] + --enable-silent-rules less verbose build output (undo: "make V=1") + --disable-silent-rules verbose build output (undo: "make V=0") +... +---- + +The +configure+ script allows you to enable or disable certain features +of +bitcoind+ through the use of the +--enable-FEATURE+ and ++--disable-FEATURE+ flags, where pass:[FEATURE] is replaced by the +feature name, as listed in the help output. In this chapter, we will +build the +bitcoind+ client with all the default features. We won't be +using the configuration flags, but you should review them to understand +what optional features are part of the client. If you are in an academic +setting, computer lab restrictions may require you to install +applications in your home directory (e.g., using +--prefix=$HOME+). + +Here are some useful options that override the default behavior of the ++configure+ script: + +++++ +
+
--prefix=$HOME
+

This overrides the default installation location (which is /usr/local/) for the resulting executable. Use $HOME to put everything in your home directory, or a different path.

+ +
--disable-wallet
+

This is used to disable the reference wallet implementation.

+ +
--with-incompatible-bdb
+

If you are building a wallet, allow the use of an incompatible version of the Berkeley DB library.

+ +
--with-gui=no
+

Don't build the graphical user interface, which requires the Qt library. This builds server and command-line Bitcoin Core only.

+
+++++ + +Next, run the +configure+ script to automatically discover all the necessary libraries and create a customized build script for your system: + +---- +$ ./configure +checking for pkg-config... /usr/bin/pkg-config +checking pkg-config is at least version 0.9.0... yes +checking build system type... x86_64-pc-linux-gnu +checking host system type... x86_64-pc-linux-gnu +checking for a BSD-compatible install... /usr/bin/install -c +... +[many pages of configuration tests follow] +... +---- + +If all went well, the +configure+ command will end by creating the +customized build scripts that will allow us to compile +bitcoind+. If +there are any missing libraries or errors, the +configure+ command will +terminate with an error instead of creating the build scripts. If an +error occurs, it is most likely because of a missing or incompatible +library. Review the build documentation again and make sure you install +the missing prerequisites. Then run +configure+ again and see if that +fixes the((("Bitcoin Core", "compiling from source code", "configuring build", startref="bitcoin-core-compile-configure")))((("compiling Bitcoin Core from source code", "configuring build", startref="compile-bitcoin-core-configure")))((("source code, compiling Bitcoin Core", "configuring build", startref="source-code-compile-configure")))((("configuring", "Bitcoin Core build", startref="configure-build"))) error. + +==== Building the Bitcoin Core Executables + +Next, you +will ((("Bitcoin Core", "compiling from source code", "building executables")))((("compiling Bitcoin Core from source code", "building executables")))((("source code, compiling Bitcoin Core", "building executables")))((("executables (Bitcoin Core), building")))compile the source code, a process that can take up to an hour to +complete, depending on the speed of your CPU and available memory. +If an error +occurs, or the compilation process is interrupted, it can be resumed any +time by typing +make+ again. Type *+make+* to start compiling the +executable application: + +---- +$ make +Making all in src + CXX bitcoind-bitcoind.o + CXX libbitcoin_node_a-addrdb.o + CXX libbitcoin_node_a-addrman.o + CXX libbitcoin_node_a-banman.o + CXX libbitcoin_node_a-blockencodings.o + CXX libbitcoin_node_a-blockfilter.o +[... many more compilation messages follow ...] +---- + +On a fast system with more than one CPU, you might want to set the +number of parallel compile jobs. For instance, +make -j 2+ will use two +cores if they are available. If all goes well, Bitcoin Core is now +compiled. You should run the unit test suite with +make check+ to ensure +the linked libraries are not broken in obvious ways. The final step is +to install the various executables on your system using the +make +install+ command. You may be prompted for your user password because +this step requires administrative privileges: + +---- +$ make check && sudo make install +Password: +Making install in src + ../build-aux/install-sh -c -d '/usr/local/lib' +libtool: install: /usr/bin/install -c bitcoind /usr/local/bin/bitcoind +libtool: install: /usr/bin/install -c bitcoin-cli /usr/local/bin/bitcoin-cli +libtool: install: /usr/bin/install -c bitcoin-tx /usr/local/bin/bitcoin-tx +... +---- + +The default installation of +bitcoind+ +puts it in _/usr/local/bin_. You can confirm that Bitcoin Core is +correctly installed by asking the system for the path of the +executables, ((("Bitcoin Core", "compiling from source code", startref="bitcoin-core-compile")))((("compiling Bitcoin Core from source code", startref="compile-bitcoin-core")))((("source code, compiling Bitcoin Core", startref="source-code-compile")))as follows: + +---- +$ which bitcoind +/usr/local/bin/bitcoind + +$ which bitcoin-cli +/usr/local/bin/bitcoin-cli +---- + +=== Running a Bitcoin Core Node + +Bitcoin's ((("Bitcoin Core", "nodes", "running", id="bitcoin-core-nodes-running")))((("nodes", "running", id="nodes-running")))((("running nodes", id="running-nodes")))peer-to-peer +network is composed of network "nodes," run mostly by individuals and +some of the businesses that provide Bitcoin services. Those running +Bitcoin nodes have a direct and authoritative view of the Bitcoin +blockchain, with a local copy of all the spendable bitcoins +independently validated by their own system. By running a node, you +don't have to rely on any third party to validate a transaction. +Additionally, by using a Bitcoin node to fully validate the transactions +you receive to your wallet, you contribute to the Bitcoin network and +help make it more robust. + +Running a node, however, requires downloading and processing over 500 GB +of data initially and about 400 MB of Bitcoin transactions per day. +These figures are for 2023 and will likely increase over time. If you +shut down your node or get disconnected from the internet for several +days, your node will need to download the data that it missed. For +example, if you close Bitcoin Core for 10 days, you will need to +download approximately 4 GB the next time you start it. + +Depending on whether you choose to index all transactions and keep a +full copy of the blockchain, you may also need a lot of disk space--at +least 1 TB if you plan to run Bitcoin Core for several years. By +default, Bitcoin nodes also transmit transactions and blocks to other +nodes (called "peers"), consuming upload internet bandwidth. If your +internet connection is limited, has a low data cap, or is metered +(charged by the gigabit), you should probably not run a Bitcoin node on +it, or run it in a way that constrains its bandwidth (see +<>). You may connect your node instead to an +alternative network, such as a free satellite data provider like +https://oreil.ly/cIwf3[Blockstream Satellite]. + +[TIP] +==== +Bitcoin Core keeps a full +copy of the blockchain by default, with nearly every transaction that has ever +been confirmed on the Bitcoin network since its inception in 2009. This +dataset is hundreds of gigabytes in size and is downloaded incrementally +over several hours or days, depending on the speed of your CPU and +internet connection. Bitcoin Core will not be able to process +transactions or update account balances until the full blockchain +dataset is downloaded. Make sure you have enough disk space, bandwidth, +and time to complete the initial synchronization. You can configure +Bitcoin Core to reduce the size of the blockchain by discarding old +blocks, but it will still download the +entire dataset. +==== + +Despite these resource requirements, thousands of people run Bitcoin +nodes. Some are running on systems as simple as a Raspberry Pi (a $35 +USD computer the size of a pack of cards). + +[role="less_space pagebreak-before"] +Why would you want to run a node? Here are some of the most common +reasons: + +- You do not want to rely on any third party to validate the + transactions you receive. + +- You do not want to disclose to third parties which transactions belong + to your wallet. + +- You are developing Bitcoin software and need to rely on a Bitcoin + node for programmable (API) access to the network and blockchain. + +- You are building applications that must validate transactions + according to Bitcoin's consensus rules. Typically, Bitcoin software + companies run several nodes. + +- You want to support Bitcoin. Running a node that you use to + validate the transactions you receive to your wallet makes the network + more robust. + +If you're reading this book and interested in strong security, superior +privacy, or developing Bitcoin software, you should be running your ((("Bitcoin Core", "nodes", "running", startref="bitcoin-core-nodes-running")))((("nodes", "running", startref="nodes-running")))((("running nodes", startref="running-nodes")))own +node. + +[[bitcoincorenode_config]] +=== Configuring the Bitcoin Core Node + +Bitcoin Core will((("Bitcoin Core", "nodes", "configuring", id="bitcoin-core-nodes-configure")))((("nodes", "configuring", id="nodes-configure")))((("configuring", "nodes", id="configure-nodes"))) look for a +configuration file in its data directory on every start. In this section +we will examine the various configuration options and set up a +configuration file. To locate the configuration file, run +bitcoind +-printtoconsole+ in your terminal and look for the first couple of +lines: + +---- +$ bitcoind -printtoconsole +2023-01-28T03:21:42Z Bitcoin Core version v24.0.1 +2023-01-28T03:21:42Z Using the 'x86_shani(1way,2way)' SHA256 implementation +2023-01-28T03:21:42Z Using RdSeed as an additional entropy source +2023-01-28T03:21:42Z Using RdRand as an additional entropy source +2023-01-28T03:21:42Z Default data directory /home/harding/.bitcoin +2023-01-28T03:21:42Z Using data directory /home/harding/.bitcoin +2023-01-28T03:21:42Z Config file: /home/harding/.bitcoin/bitcoin.conf +... +[a lot more debug output] +... +---- + +You can hit Ctrl-C to shut down the node once you determine the location +of the config file. Usually the configuration file is inside the +_.bitcoin_ data directory under your user's home directory. Open the +configuration file in your preferred editor. + +Bitcoin Core offers more than 100 configuration options that modify the +behavior of the network node, the storage of the blockchain, and many +other aspects of its operation. To see a listing of these options, run ++bitcoind --help+: + +---- +$ bitcoind --help +Bitcoin Core version v24.0.1 + +Usage: bitcoind [options] Start Bitcoin Core + +Options: + + -? + Print this help message and exit + + -alertnotify= + Execute command when an alert is raised (%s in cmd is replaced by + message) +... +[many more options] +---- + +Here are some of +the most important options that you can set in the configuration file, +or as command-line parameters to +bitcoind+: + +++alertnotify++:: Run ((("alertnotify option (bitcoind option)")))a specified command or script to send emergency alerts +to the owner of this node. + +++conf++:: An ((("conf option (bitcoind option)")))alternative location for the configuration file. This only +makes sense as a command-line parameter to +bitcoind+, as it can't be +inside the configuration file it refers to. + +++datadir++:: Select((("datadir option (bitcoind option)"))) the directory and filesystem in which to put all the +blockchain data. By default this is the _.bitcoin_ subdirectory of your +home directory. Depending on your configuration, this can use from about 10 +GB to almost 1 TB as of this writing, with the maximum size +expected to increase by several hundred gigabytes per year. + +++prune++:: Reduce the((("prune option (bitcoind option)"))) blockchain disk space requirements to this many megabytes by +deleting old blocks. Use this on a resource-constrained node that can't +fit the full blockchain. Other parts of the system will use other disk +space that can't currently be pruned, so you will still need at least +the minimum amount of space mentioned in the +datadir+ option. + +++txindex++:: Maintain ((("txindex option (bitcoind option)")))an index of all transactions. This allows you to +programmatically retrieve any transaction by its ID provided that the +block containing that transaction hasn't been pruned. + +[role="less_space pagebreak-before"] +++dbcache++:: The size((("dbcache option (bitcoind option)"))) of the UTXO cache. The default is 450 mebibytes (MiB). Increase +this size on high-end hardware to read and write from your disk less +often, or reduce the size on low-end hardware to save memory at the +expense of using your disk more frequently. + +++blocksonly++:: Minimize ((("blocksonly option (bitcoind option)")))your bandwidth usage by only accepting blocks of +confirmed transactions from your peers instead of relaying unconfirmed +transactions. + +++maxmempool++:: Limit ((("maxmempool option (bitcoind option)")))the transaction memory pool to this many megabytes. +Use it to reduce memory use on memory-constrained nodes. + +[[txindex]] +.Transaction Database Index and txindex Option +**** +By default, +Bitcoin Core builds((("txindex option (bitcoind option)")))((("transactions", "building complete index"))) a database containing _only_ the transactions +related to the user's wallet. If you want to be able to access _any_ +transaction with commands like +getrawtransaction+ (see +<>), you need to configure Bitcoin +Core to build a complete transaction index, which can be achieved with +the +txindex+ option. Set +txindex=1+ in the Bitcoin Core configuration +file. If you don't set this option at first and later set it to full +indexing, you need to +wait for it to rebuild the index. +**** + +<> shows how you might combine the preceding options +with a fully indexed node, running as an API backend for a bitcoin +application. + +[[full_index_node]] +.Sample configuration of a full-index node +==== +---- +alertnotify=myemailscript.sh "Alert: %s" +datadir=/lotsofspace/bitcoin +txindex=1 +---- +==== + +<> shows a resource-constrained node running on a +smaller server. + +[[constrained_resources]] +.Sample configuration of a resource-constrained system +==== +---- +alertnotify=myemailscript.sh "Alert: %s" +blocksonly=1 +prune=5000 +dbcache=150 +maxmempool=150 +---- +==== + +After you've edited the configuration file and set the options that best +represent your needs, you can test +bitcoind+ with this configuration. +Run Bitcoin Core with the option +printtoconsole+ to run in the +foreground with output to the console: + +---- +$ bitcoind -printtoconsole +2023-01-28T03:43:39Z Bitcoin Core version v24.0.1 +2023-01-28T03:43:39Z Using the 'x86_shani(1way,2way)' SHA256 implementation +2023-01-28T03:43:39Z Using RdSeed as an additional entropy source +2023-01-28T03:43:39Z Using RdRand as an additional entropy source +2023-01-28T03:43:39Z Default data directory /home/harding/.bitcoin +2023-01-28T03:43:39Z Using data directory /lotsofspace/bitcoin +2023-01-28T03:43:39Z Config file: /home/harding/.bitcoin/bitcoin.conf +2023-01-28T03:43:39Z Config file arg: [main] blockfilterindex="1" +2023-01-28T03:43:39Z Config file arg: [main] maxuploadtarget="1000" +2023-01-28T03:43:39Z Config file arg: [main] txindex="1" +2023-01-28T03:43:39Z Setting file arg: wallet = ["msig0"] +2023-01-28T03:43:39Z Command-line arg: printtoconsole="" +2023-01-28T03:43:39Z Using at most 125 automatic connections +2023-01-28T03:43:39Z Using 16 MiB out of 16 MiB requested for signature cache +2023-01-28T03:43:39Z Using 16 MiB out of 16 MiB requested for script execution +2023-01-28T03:43:39Z Script verification uses 3 additional threads +2023-01-28T03:43:39Z scheduler thread start +2023-01-28T03:43:39Z [http] creating work queue of depth 16 +2023-01-28T03:43:39Z Using random cookie authentication. +2023-01-28T03:43:39Z Generated RPC cookie /lotsofspace/bitcoin/.cookie +2023-01-28T03:43:39Z [http] starting 4 worker threads +2023-01-28T03:43:39Z Using wallet directory /lotsofspace/bitcoin/wallets +2023-01-28T03:43:39Z init message: Verifying wallet(s)… +2023-01-28T03:43:39Z Using BerkeleyDB version Berkeley DB 4.8.30 +2023-01-28T03:43:39Z Using /16 prefix for IP bucketing +2023-01-28T03:43:39Z init message: Loading P2P addresses… +2023-01-28T03:43:39Z Loaded 63866 addresses from peers.dat 114ms +[... more startup messages ...] +---- + +You can hit Ctrl-C to interrupt the process once you are satisfied that +it is loading the correct settings and running as you expect. + +To run Bitcoin Core in the background as a process, start it with the ++daemon+ option, as +bitcoind -daemon+. + +To monitor the progress and runtime status of your Bitcoin node, start +it in daemon mode and then use the command +bitcoin-cli getblockchaininfo+: + +---- +$ bitcoin-cli getblockchaininfo +---- + +[source,json] +---- +{ + "chain": "main", + "blocks": 0, + "headers": 83999, + "bestblockhash": "[...]19d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f", + "difficulty": 1, + "time": 1673379796, + "mediantime": 1231006505, + "verificationprogress": 3.783041623201835e-09, + "initialblockdownload": true, + "chainwork": "[...]000000000000000000000000000000000000000000000100010001", + "size_on_disk": 89087, + "pruned": false, + "warnings": "" +} +---- + +This shows a node with a blockchain height of 0 blocks and 83,999 +headers. The node first fetches the block headers from its peers in +order to find the blockchain with the most proof of work and +afterward continues to download the full blocks, validating them as it +goes. + +Once you are happy with the configuration options you have selected, you +should add Bitcoin Core to the startup scripts in your operating system, so +that it runs continuously and restarts when the operating system +restarts. You will find a number of example startup scripts for various +operating systems in Bitcoin Core's source directory under _contrib/init_ and +a _README.md_ file showing which system uses ((("Bitcoin Core", "nodes", "configuring", startref="bitcoin-core-nodes-configure")))((("nodes", "configuring", startref="nodes-configure")))((("configuring", "nodes", startref="configure-nodes")))which script. + +=== Bitcoin Core API + +Bitcoin Core +implements((("Bitcoin Core", "command-line interface", "help command", id="bitcoin-core-command-help")))((("command-line interface (Bitcoin Core)", "help command", id="command-help")))((("help command (Bitcoin Core)", id="help")))((("RPC commands", see="command-line interface (Bitcoin Core)")))((("bitcoin-cli command", see="command-line interface (Bitcoin Core)"))) a JSON-RPC interface that can also be accessed using +the command-line helper +bitcoin-cli+. The command line allows us to +experiment interactively with the capabilities that are also available +programmatically via the API. To start, invoke the +help+ command to see a list of +the available Bitcoin Core RPC commands: + +[[bitcoind_commands]] + +---- +$ bitcoin-cli help ++== Blockchain == +getbestblockhash +getblock "blockhash" ( verbosity ) +getblockchaininfo +... +walletpassphrase "passphrase" timeout +walletpassphrasechange "oldpassphrase" "newpassphrase" +walletprocesspsbt "psbt" ( sign "sighashtype" bip32derivs finalize ) +---- + +Each of these commands may take a number of parameters. To get +additional help, a detailed description, and information on the +parameters, add the command name after +help+. For example, to see help +on the +getblockhash+ RPC command: + +---- +$ bitcoin-cli help getblockhash +getblockhash height + +Returns hash of block in best-block-chain at height provided. + +Arguments: +1. height (numeric, required) The height index + +Result: +"hex" (string) The block hash + +Examples: +> bitcoin-cli getblockhash 1000 +> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id": "curltest", + "method": "getblockhash", + "params": [1000]}' -H 'content-type: text/plain;' http://127.0.0.1:8332/ +---- + +At the end of the help information you will see two examples of the RPC +command, using the +bitcoin-cli+ helper or the HTTP client +curl+. These +examples demonstrate how you might call the command. Copy the first +example and see the result: + +---- +$ bitcoin-cli getblockhash 1000 +00000000c937983704a73af28acdec37b049d214adbda81d7e2a3dd146f6ed09 +---- + +The result is a block hash, which is described in more detail in the +following chapters. But for now, this command should return the same +result on your system, demonstrating that your Bitcoin Core node is +running, is accepting commands, and has information about block 1,000 to +return to you. + +In the next sections we will demonstrate some very useful RPC commands +and their expected ((("Bitcoin Core", "command-line interface", "help command", startref="bitcoin-core-command-help")))((("command-line interface (Bitcoin Core)", "help command", startref="command-help")))((("help command (Bitcoin Core)", startref="help")))output. + +==== Getting Information on Bitcoin Core's Status + +Bitcoin +Core provides((("Bitcoin Core", "command-line interface", "status information", id="bitcoin-core-command-status")))((("command-line interface (Bitcoin Core)", "status information", id="command-status")))((("status information (Bitcoin Core)", id="status-bitcoin-core"))) status reports on different modules through the JSON-RPC +interface. The most important commands include +getblockchaininfo+, ++getmempoo⁠l​info+, +getnetworkinfo+, and +getwalletinfo+. + +Bitcoin's +getblockchaininfo+ RPC command was introduced earlier. The ++getnetwor⁠k​info+ command displays basic information about the status of +the Bitcoin network node. Use +bitcoin-cli+ to run it: + +---- +$ bitcoin-cli getnetworkinfo +---- +[source,json] +---- +{ + "version": 240001, + "subversion": "/Satoshi:24.0.1/", + "protocolversion": 70016, + "localservices": "0000000000000409", + "localservicesnames": [ + "NETWORK", + "WITNESS", + "NETWORK_LIMITED" + ], + "localrelay": true, + "timeoffset": -1, + "networkactive": true, + "connections": 10, + "connections_in": 0, + "connections_out": 10, + "networks": [ + "...detailed information about all networks..." + ], + "relayfee": 0.00001000, + "incrementalfee": 0.00001000, + "localaddresses": [ + ], + "warnings": "" +} +---- + +The data is returned in JavaScript Object Notation (JSON), a format that +can easily be "consumed" by all programming languages but is also quite +human-readable. Among this data we see the version numbers for the +Bitcoin Core software and Bitcoin protocol. We see +the current number of connections and various information about the +Bitcoin network and the settings related to this node. + +[TIP] +==== +It will take some time, perhaps more than a day, for +bitcoind+ +to catch up to the current blockchain height as it downloads blocks from +other Bitcoin nodes and validates every transaction in those +blocks--almost a billion transactions as of this writing. You can check +its progress using ++getblockchaininfo+ to see the number of known blocks. The examples in +the rest of this chapter assume you're at least at block 775,072. +Because the security of Bitcoin transactions depends on blocks, some of +the information in the following examples will change slightly depending +on how many blocks your ((("Bitcoin Core", "command-line interface", "status information", startref="bitcoin-core-command-status")))((("command-line interface (Bitcoin Core)", "status information", startref="command-status")))((("status information (Bitcoin Core)", startref="status-bitcoin-core")))node has. +==== + +[[exploring_and_decoding_transactions]] +==== Exploring and Decoding Transactions + +In <>, Alice ((("Bitcoin Core", "command-line interface", "exploring/decoding transactions", id="bitcoin-core-command-transaction")))((("command-line interface (Bitcoin Core)", "exploring/decoding transactions", id="command-transaction")))((("transactions", "exploring/decoding", id="transactions-explore-decode")))((("exploring", "transactions", id="explore-transaction")))((("decoding", "transactions", id="decode-transaction")))made a purchase from Bob's store. Her +transaction was recorded on the blockchain. +Let's use the API to retrieve and examine that transaction by passing +the transaction ID (txid) as a parameter: + +[[alice_tx_serialized]] +.Alice's serialized transaction +---- +$ bitcoin-cli getrawtransaction 466200308696215bbc949d5141a49a41\ +38ecdfdfaa2a8029c1f9bcecd1f96177 + +01000000000101eb3ae38f27191aa5f3850dc9cad00492b88b72404f9da13569 +8679268041c54a0100000000ffffffff02204e0000000000002251203b41daba +4c9ace578369740f15e5ec880c28279ee7f51b07dca69c7061e07068f8240100 +000000001600147752c165ea7be772b2c0acb7f4d6047ae6f4768e0141cf5efe +2d8ef13ed0af21d4f4cb82422d6252d70324f6f4576b727b7d918e521c00b51b +e739df2f899c49dc267c0ad280aca6dab0d2fa2b42a45182fc83e81713010000 +0000 +---- + +[TIP] +==== +A transaction ID (txid) +is ((("transaction IDs (txid)")))not authoritative. Absence of a txid in the blockchain does not mean +the transaction was not processed. This is known as "transaction +malleability," because transactions can be modified prior to +confirmation in a block, changing their txids. After a transaction is +included in a block, its txid cannot change unless there is a blockchain +reorganization where that block is removed from the best blockchain. +Reorganizations are rare after a transaction has several confirmations. +==== + +The command +getrawtransaction+ returns a serialized transaction in +hexadecimal notation. To decode that, we use the +decoderawtransaction+ +command, passing the hex data as a parameter. You can copy the hex +returned by +getrawtransaction+ and paste it as a parameter to ++decoderawtransaction+: + +++++ +
+$ bitcoin-cli decoderawtransaction 01000000000101eb3ae38f27191aa5f3850dc9cad0\
+0492b88b72404f9da135698679268041c54a0100000000ffffffff02204e00000000000022512\
+03b41daba4c9ace578369740f15e5ec880c28279ee7f51b07dca69c7061e07068f82401000000\
+00001600147752c165ea7be772b2c0acb7f4d6047ae6f4768e0141cf5efe2d8ef13ed0af21d4f\
+4cb82422d6252d70324f6f4576b727b7d918e521c00b51be739df2f899c49dc267c0ad280aca6\
+dab0d2fa2b42a45182fc83e817130100000000
+
+++++ + +++++ +
+{
+  "txid": "466200308696215bbc949d5141a49a4138ecdfdfaa2a8029c1f9bcecd1f96177",
+  "hash": "f7cdbc7cf8b910d35cc69962e791138624e4eae7901010a6da4c02e7d238cdac",
+  "version": 1,
+  "size": 194,
+  "vsize": 143,
+  "weight": 569,
+  "locktime": 0,
+  "vin": [
+    {
+      "txid": "4ac541802679866935a19d4f40728bb89204d0cac90d85f3a51a19...aeb",
+      "vout": 1,
+      "scriptSig": {
+        "asm": "",
+        "hex": ""
+      },
+      "txinwitness": [
+        "cf5efe2d8ef13ed0af21d4f4cb82422d6252d70324f6f4576b727b7d918e5...301"
+      ],
+      "sequence": 4294967295
+    }
+  ],
+  "vout": [
+    {
+      "value": 0.00020000,
+      "n": 0,
+      "scriptPubKey": {
+        "asm": "1 3b41daba4c9ace578369740f15e5ec880c28279ee7f51b07dca...068",
+        "desc": "rawtr(3b41daba4c9ace578369740f15e5ec880c28279ee7f51b...6ev",
+        "hex": "51203b41daba4c9ace578369740f15e5ec880c28279ee7f51b07d...068",
+        "address": "bc1p8dqa4wjvnt890qmfws83te0v3qxzsfu7ul63kp7u56w8q...5qn",
+        "type": "witness_v1_taproot"
+      }
+    },
+    {
+      "value": 0.00075000,
+      "n": 1,
+      "scriptPubKey": {
+        "asm": "0 7752c165ea7be772b2c0acb7f4d6047ae6f4768e",
+        "desc": "addr(bc1qwafvze0200nh9vkq4jmlf4sy0tn0ga5w0zpkpg)#qq404gts",
+        "hex": "00147752c165ea7be772b2c0acb7f4d6047ae6f4768e",
+        "address": "bc1qwafvze0200nh9vkq4jmlf4sy0tn0ga5w0zpkpg",
+        "type": "witness_v0_keyhash"
+      }
+    }
+  ]
+}
+
+++++ + +The transaction decode shows all the components of this transaction, +including the transaction inputs and outputs. In this case we see that +the transaction used one input and generated two outputs. The input to +this transaction was the output from a previously confirmed transaction +(shown as the input +txid+). The two outputs correspond to the payment to +Bob and the change back to Alice. + +We can further explore the blockchain by examining the previous +transaction referenced by its +txid+ in this transaction using the same +commands (e.g., [.keep-together]#+getrawtransaction+).# Jumping from transaction to +transaction, we can follow a chain of transactions back as the coins are +transmitted from one owner to the((("Bitcoin Core", "command-line interface", "exploring/decoding transactions", startref="bitcoin-core-command-transaction")))((("command-line interface (Bitcoin Core)", "exploring/decoding transactions", startref="command-transaction")))((("transactions", "exploring/decoding", startref="transactions-explore-decode")))((("exploring", "transactions", startref="explore-transaction")))((("decoding", "transactions", startref="decode-transaction"))) next. + +==== Exploring Blocks + +Exploring +blocks is((("Bitcoin Core", "command-line interface", "exploring blocks", id="bitcoin-core-command-blocks")))((("command-line interface (Bitcoin Core)", "exploring blocks", id="command-blocks")))((("blocks", "exploring", id="blocks-explore")))((("exploring", "blocks", id="explore-blocks"))) similar to exploring transactions. However, blocks can be +referenced either by the block _height_ or by the block _hash_. First, +let's find a block by its height. +We use the +getblockhash+ command, which takes the block height as the +parameter and returns the block _header hash_ for that block: + +++++ +
+$ bitcoin-cli getblockhash 123456
+0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca
+
+++++ + +Now that we know the header hash for our chosen block, we can +query that block. We use the +getblock+ command with the block hash as +the parameter: + +++++ +
+$ bitcoin-cli getblock 0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682f\
+f6cd83c3ca
+
+++++ + +++++ +
+{
+  "hash": "0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca",
+  "confirmations": 651742,
+  "height": 123456,
+  "version": 1,
+  "versionHex": "00000001",
+  "merkleroot": "0e60651a9934e8f0decd1c[...]48fca0cd1c84a21ddfde95033762d86c",
+  "time": 1305200806,
+  "mediantime": 1305197900,
+  "nonce": 2436437219,
+  "bits": "1a6a93b3",
+  "difficulty": 157416.4018436489,
+  "chainwork": "[...]00000000000000000000000000000000000000541788211ac227bc",
+  "nTx": 13,
+  "previousblockhash": "[...]60bc96a44724fd72daf9b92cf8ad00510b5224c6253ac40095",
+  "nextblockhash": "[...]00129f5f02be247070bf7334d3753e4ddee502780c2acaecec6d66",
+  "strippedsize": 4179,
+  "size": 4179,
+  "weight": 16716,
+  "tx": [
+    "5b75086dafeede555fc8f9a810d8b10df57c46f9f176ccc3dd8d2fa20edd685b",
+    "e3d0425ab346dd5b76f44c222a4bb5d16640a4247050ef82462ab17e229c83b4",
+    "137d247eca8b99dee58e1e9232014183a5c5a9e338001a0109df32794cdcc92e",
+    "5fd167f7b8c417e59106ef5acfe181b09d71b8353a61a55a2f01aa266af5412d",
+    "60925f1948b71f429d514ead7ae7391e0edf965bf5a60331398dae24c6964774",
+    "d4d5fc1529487527e9873256934dfb1e4cdcb39f4c0509577ca19bfad6c5d28f",
+    "7b29d65e5018c56a33652085dbb13f2df39a1a9942bfe1f7e78e97919a6bdea2",
+    "0b89e120efd0a4674c127a76ff5f7590ca304e6a064fbc51adffbd7ce3a3deef",
+    "603f2044da9656084174cfb5812feaf510f862d3addcf70cacce3dc55dab446e",
+    "9a4ed892b43a4df916a7a1213b78e83cd83f5695f635d535c94b2b65ffb144d3",
+    "dda726e3dad9504dce5098dfab5064ecd4a7650bfe854bb2606da3152b60e427",
+    "e46ea8b4d68719b65ead930f07f1f3804cb3701014f8e6d76c4bdbc390893b94",
+    "864a102aeedf53dd9b2baab4eeb898c5083fde6141113e0606b664c41fe15e1f"
+  ]
+}
+
+++++ + +The +confirmations+ entry tells us the _depth_ of this block--how many +blocks have been built on top of it, indicating the difficulty of +changing any of the transactions in this block. The +height+ tells us +how many blocks preceeded this block. We see the block's version, the +time it was created (according to its miner), the median time of the 11 +blocks that precede this block (a time measurement that's harder for +miners to manipulate), and the size of the block in three different +measurements (its legacy stripped size, its full size, and its size in +weight units). We also see some fields used for security and +proof of work (merkle root, nonce, bits, difficulty, and chainwork); +we'll examine those ((("Bitcoin Core", "command-line interface", "exploring blocks", startref="bitcoin-core-command-blocks")))((("command-line interface (Bitcoin Core)", "exploring blocks", startref="command-blocks")))((("blocks", "exploring", startref="blocks-explore")))((("exploring", "blocks", startref="explore-blocks")))in detail in <>. + +==== Using Bitcoin Core's Programmatic Interface + +The ++bitcoin-cli+ helper is ((("Bitcoin Core", "command-line interface", "API access", id="bitcoin-core-command-api")))((("command-line interface (Bitcoin Core)", "API access", id="command-api")))((("API access", id="api")))very useful for exploring the Bitcoin Core API +and testing functions. But the whole point of an API is to access functions programmatically. In this section we +will demonstrate accessing Bitcoin Core from another program. + +Bitcoin Core's API is a JSON-RPC interface. JSON is a very convenient way to present data that +both humans and programs can easily read. RPC stands for remote +[.keep-together]#procedure# call, which means that we are calling procedures (functions) +that are remote (on the Bitcoin Core node) via a network protocol. In +this case, the network protocol is HTTP. + +When we used the +bitcoin-cli+ command to get help on a command, it +showed us an example of using +curl+, the versatile command-line HTTP +client to construct one of these JSON-RPC calls: + +---- +$ curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", + "method": "getblockchaininfo", + "params": [] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/ +---- + +This command shows that +curl+ submits an HTTP request to the local host +(127.0.0.1), connecting to the default Bitcoin RPC port (8332), and +submitting a +jsonrpc+ request for the +getblockchaininfo+ method using ++text/plain+ encoding. + +You might notice that +curl+ will ask for credentials to be sent along +with the request. Bitcoin Core will create a random password on each +start and place it in the data directory under the name +.cookie+. The ++bitcoin-cli+ helper can read this password file given the data +directory. Similarly, you can copy the password and pass it to +curl+ (or +any higher-level Bitcoin Core RPC wrappers), ((("Bitcoin Core", "authentication")))((("authentication")))as seen in <>. + +[[cookie_auth]] +.Using cookie-based authentication with Bitcoin Core +==== +---- +$ cat .bitcoin/.cookie + __cookie__:17c9b71cef21b893e1a019f4bc071950c7942f49796ed061b274031b17b19cd0 + +$ curl + --user __cookie__:17c9b71cef21b893e1a019f4bc071950c7942f49796ed061b274031b17b19cd0 + --data-binary '{"jsonrpc": "1.0", "id":"curltest", + "method": "getblockchaininfo", + "params": [] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/ + +{"result":{"chain":"main","blocks":799278,"headers":799278, +"bestblockhash":"000000000000000000018387c50988ec705a95d6f765b206b6629971e6978879", +"difficulty":53911173001054.59,"time":1689703111,"mediantime":1689701260, +"verificationprogress":0.9999979206082515,"initialblockdownload":false, +"chainwork":"00000000000000000000000000000000000000004f3e111bf32bcb47f9dfad5b", +"size_on_disk":563894577967,"pruned":false,"warnings":""},"error":null, +"id":"curltest"} +---- +==== + +Alternatively, you can +create a static password with the helper script provided in +[.keep-together]#_./share/rpcauth/rpcauth.py_# in Bitcoin Core's source directory. + +If you're implementing a JSON-RPC call in your own program, you can use +a generic HTTP library to construct the call, similar to what is shown +in the preceding +curl+ example. + +However, there ((("Bitcoin Core", "wrapper libraries")))((("wrapper libraries")))are libraries in most popular programming languages that +"wrap" the Bitcoin Core API in a way that makes this a lot simpler. We +will use the +python-bitcoinlib+ library to simplify API access. +This library is not part of the Bitcoin Core project and needs to be +installed the usual way you install Python libraries. +Remember, this requires you to have a running Bitcoin Core instance, +which will be used to make JSON-RPC calls. + +The Python script in <> makes a simple +getblockchaininfo+ +call and prints the +block+ parameter from the data returned by Bitcoin +Core. + +[[rpc_example]] +.Running +getblockchaininfo+ via Bitcoin Core's JSON-RPC API +==== +[source,python] +---- +include::code/rpc_example.py[] +---- +==== + +Running it gives us the following result: + +---- +$ python rpc_example.py +773973 +---- + +It tells us how many blocks our local Bitcoin Core node has in its +blockchain. Not a spectacular result, but it demonstrates the basic use +of the library as a simplified interface to Bitcoin Core's JSON-RPC API. + +Next, let's((("Bitcoin Core", "command-line interface", "exploring/decoding transactions", id="bitcoin-core-command-transaction2")))((("command-line interface (Bitcoin Core)", "exploring/decoding transactions", id="command-transaction2")))((("transactions", "exploring/decoding", id="transactions-explore-decode2")))((("exploring", "transactions", id="explore-transaction2")))((("decoding", "transactions", id="decode-transaction2"))) use the +getrawtransaction+ and +decodetransaction+ calls to +retrieve the details of Alice's payment to Bob. In <>, +we retrieve Alice's transaction and list the transaction's outputs. For +each output, we show the recipient address and value. As a reminder, +Alice's transaction had one output paying Bob and one output for +change back to Alice. + +[[rpc_transaction]] +.Retrieving a transaction and iterating its outputs +==== +[source,python] +---- +include::code/rpc_transaction.py[] +---- +==== + +Running this code, we get: + +---- +$ python rpc_transaction.py +bc1p8dqa4wjvnt890qmfws83te0v3qxzsfu7ul63kp7u56w8qc0qwp5qv995qn 0.00020000 +bc1qwafvze0200nh9vkq4jmlf4sy0tn0ga5w0zpkpg 0.00075000 +---- + +Both of the ((("Bitcoin Core", "command-line interface", "exploring/decoding transactions", startref="bitcoin-core-command-transaction2")))((("command-line interface (Bitcoin Core)", "exploring/decoding transactions", startref="command-transaction2")))((("transactions", "exploring/decoding", startref="transactions-explore-decode2")))((("exploring", "transactions", startref="explore-transaction2")))((("decoding", "transactions", startref="decode-transaction2")))preceding examples are rather simple. You don't really need +a program to run them; you could just as easily use the +bitcoin-cli+ +helper. The next example, however, requires several hundred RPC calls +and more clearly demonstrates the use of a programmatic interface. + +In <>, we first ((("Bitcoin Core", "command-line interface", "exploring blocks", id="bitcoin-core-command-blocks2")))((("command-line interface (Bitcoin Core)", "exploring blocks", id="command-blocks2")))((("blocks", "exploring", id="blocks-explore2")))((("exploring", "blocks", id="explore-blocks2")))retrieve a block, then retrieve each of +the transactions within it by reference to each transaction ID. Next, +we iterate through each of the transaction's outputs and add up the +value. + +[[rpc_block]] +.Retrieving a block and adding all the transaction outputs +==== +[source,python] +---- +include::code/rpc_block.py[] +---- +==== + +Running this code, we get: + +---- +$ python rpc_block.py + +Total value in block: 10322.07722534 +---- + +Our example code calculates that the total value transacted in this +block is 10,322.07722534 BTC (including 25 BTC reward and 0.0909 BTC in +fees). Compare that to the amount reported by a block explorer site by +searching for the block hash or height. Some block explorers report the +total value excluding the reward and excluding the fees. See if you can +spot the ((("Bitcoin Core", "command-line interface", "API access", startref="bitcoin-core-command-api")))((("command-line interface (Bitcoin Core)", "API access", startref="command-api")))((("API access", startref="api")))((("Bitcoin Core", "command-line interface", "exploring blocks", startref="bitcoin-core-command-blocks2")))((("command-line interface (Bitcoin Core)", "exploring blocks", startref="command-blocks2")))((("blocks", "exploring", startref="blocks-explore2")))((("exploring", "blocks", startref="explore-blocks2")))difference. + +[[alt_libraries]] +=== Alternative Clients, Libraries, and Toolkits + +There +are many alternative clients, libraries, toolkits, and even full-node +implementations in the Bitcoin ecosystem. These are implemented in a +variety of programming languages, offering programmers native interfaces +in their preferred language. + +The following sections list some of the best libraries, clients, and +toolkits, organized by programming languages. + +==== C/C++ +https://oreil.ly/BdOwl[Bitcoin Core] :: The reference((("C/C++ toolkits")))((("Bitcoin Core"))) implementation of Bitcoin + +==== JavaScript +https://bcoin.io[bcoin]:: A modular((("JavaScript toolkits")))((("bcoin")))((("Bitcore"))) and scalable full-node implementation with API +https://bitcore.io[Bitcore] :: Full node, API, and library by Bitpay +https://oreil.ly/4iqf2[BitcoinJS] :: A pure JavaScript Bitcoin library for node.js and browsers + +==== Java +https://bitcoinj.github.io[bitcoinj]:: A ((("Java toolkits")))((("bitcoinj")))Java full-node client library + +==== Python +https://oreil.ly/xn_rg[python-bitcoinlib]:: A ((("Python toolkits")))((("python-bitcoinlib")))((("pycoin")))Python bitcoin library, consensus library, and node by Peter Todd +https://oreil.ly/wcpXP[pycoin]:: A Python bitcoin library by Richard Kiss + +==== Go +https://oreil.ly/h5MEI[btcd]:: A ((("Go toolkits")))((("btcd")))Go language, full-node Bitcoin client + +==== Rust +https://oreil.ly/me6gf[rust-bitcoin]:: Rust ((("Rust toolkits")))((("rust-bitcoin")))bitcoin library for serialization, parsing, and API calls + +==== Scala +https://bitcoin-s.org[bitcoin-s]:: A ((("Scala toolkits")))((("bitcoin-s")))Bitcoin implementation in Scala + +==== C# +https://oreil.ly/Qfjgq[NBitcoin]:: Comprehensive((("C# toolkits")))((("NBitcoin"))) bitcoin library for the .NET framework + +Many more libraries exist in a variety of other programming languages, +and more are created all the time. + +If you followed the instructions in this chapter, you now have Bitcoin +Core running and have begun exploring the network and blockchain using +your own full node. From now on you can independently use software you +control—on a computer you control—to verify that any bitcoins you receive +follow every rule in the Bitcoin system without having to trust any +outside authority. In the coming chapters, we'll learn more about the +rules of the system and how your node and your wallet use them to secure +your money, protect your privacy, and make spending and receiving +[.keep-together]#convenient.# diff --git a/ch04.asciidoc b/ch04.asciidoc deleted file mode 100644 index ca0d1088..00000000 --- a/ch04.asciidoc +++ /dev/null @@ -1,801 +0,0 @@ -[[ch04_keys_addresses]] -== Keys, Addresses - -((("cryptography", "defined")))((("cryptography", see="also keys and addresses")))You may have heard that bitcoin is based on _cryptography_, which is a branch of mathematics used extensively in computer security. Cryptography means "secret writing" in Greek, but the science of cryptography encompasses more than just secret writing, which is referred to as encryption. Cryptography can also be used to prove knowledge of a secret without revealing that secret (digital signature), or prove the authenticity of data (digital fingerprint). These types of cryptographic proofs are the mathematical tools critical to bitcoin and used extensively in bitcoin applications. ((("encryption")))((("encryption", see="also keys and addresses")))Ironically, encryption is not an important part of bitcoin, as its communications and transaction data are not encrypted and do not need to be encrypted to protect the funds. In this chapter we will introduce some of the cryptography used in bitcoin to control ownership of funds, in the form of keys, addresses, and wallets. - -=== Introduction - -((("digital keys", see="keys and addresses")))((("keys and addresses", "overview of", id="KAover04")))((("digital signatures", "purpose of")))Ownership of bitcoin is established through _digital keys_, _Bitcoin addresses_, and _digital signatures_. The digital keys are not actually stored in the network, but are instead created and stored by users in a file, or simple database, called a _wallet_. The digital keys in a user's wallet are completely independent of the Bitcoin protocol and can be generated and managed by the user's wallet software without reference to the blockchain or access to the internet. Keys enable many of the interesting properties of bitcoin, including decentralized trust and control, ownership attestation, and the cryptographic-proof security model. - -Most bitcoin transactions require a valid digital signature to be included in the blockchain, which can only be generated with a secret key; therefore, anyone with a copy of that key has control of the bitcoin. ((("witnesses")))The digital signature used to spend funds is also referred to as a _witness_, a term used in cryptography. The witness data in a bitcoin transaction testifies to the true ownership of the funds being spent. - -((("public and private keys", "key pairs")))((("public and private keys", see="also keys and addresses")))Keys come in pairs consisting of a private (secret) key and a public key. Think of the public key as similar to a bank account number and the private key as similar to the secret PIN, or signature on a check, that provides control over the account. These digital keys are very rarely seen by the users of bitcoin. For the most part, they are stored inside the wallet file and managed by the bitcoin wallet software. - -In the payment portion of a bitcoin transaction, the recipient's public key is represented by its digital fingerprint, called a _Bitcoin address_, which is used in the same way as the beneficiary name on a check (i.e., "Pay to the order of"). In most cases, a Bitcoin address is generated from and corresponds to a public key. However, not all Bitcoin addresses represent public keys; they can also represent other beneficiaries such as scripts, as we will see later in this chapter. This way, Bitcoin addresses abstract the recipient of funds, making transaction destinations flexible, similar to paper checks: a single payment instrument that can be used to pay into people's accounts, pay into company accounts, pay for bills, or pay to cash. The Bitcoin address is the only representation of the keys that users will routinely see, because this is the part they need to share with the world. - -First, we will introduce cryptography and explain the mathematics used in bitcoin. Next, we will look at how keys are generated, stored, and managed. We will review the various encoding formats used to represent private and public keys, addresses, and script addresses. Finally, we will look at advanced use of keys and addresses: vanity, multisignature, and script addresses and paper wallets. - -==== Public Key Cryptography and Cryptocurrency - -((("keys and addresses", "overview of", "public key cryptography")))((("digital currencies", "cryptocurrency")))Public key cryptography was invented in the 1970s and is a mathematical foundation for computer and information security. - -Since the invention of public key cryptography, several suitable mathematical functions, such as prime number exponentiation and elliptic curve multiplication, have been discovered. These mathematical functions are practically irreversible, meaning that they are easy to calculate in one direction and infeasible to calculate in the opposite direction. Based on these mathematical functions, cryptography enables the creation of digital secrets and unforgeable digital signatures. Bitcoin uses elliptic curve multiplication as the basis for its cryptography. - -In bitcoin, we use public key cryptography to create a key pair that controls access to bitcoin. The key pair consists of a private key and--derived from it--a unique public key. The public key is used to receive funds, and the private key is used to sign transactions to spend the funds. - -There is a mathematical relationship between the public and the private key that allows the private key to be used to generate signatures on messages. These signatures can be validated against the public key without revealing the private key. - -When spending bitcoin, the current bitcoin owner presents her public key and a signature (different each time, but created from the same private key) in a transaction to spend those bitcoin. Through the presentation of the public key and signature, everyone in the Bitcoin network can verify and accept the transaction as valid, confirming that the person transferring the bitcoin owned them at the time of the transfer. - -[TIP] -==== -((("keys and addresses", "overview of", "key pairs")))In most wallet implementations, the private and public keys are stored together as a _key pair_ for convenience. However, the public key can be calculated from the private key, so storing only the private key is also possible. -==== - -[[private_public_keys]] -==== Private and Public Keys - -((("keys and addresses", "overview of", "private and public key pairs")))((("elliptic curve cryptography")))((("cryptography", "elliptic curve cryptography")))A bitcoin wallet contains a collection of key pairs, each consisting of a private key and a public key. The private key (k) is a number, usually picked at random. From the private key, we use elliptic curve multiplication, a one-way cryptographic function, to generate a public key (K). From the public key (K), we use a one-way cryptographic hash function to generate a Bitcoin address (A). In this section, we will start with generating the private key, look at the elliptic curve math that is used to turn that into a public key, and finally, generate a Bitcoin address from the public key. The relationship between private key, public key, and Bitcoin address is shown in <>. - -[[k_to_K_to_A]] -.Private key, public key, and Bitcoin address -image::images/mbc2_0401.png["privk_to_pubK_to_addressA"] - -.Why Use Asymmetric Cryptography (Public/Private Keys)? -**** -((("cryptography", "asymmetric")))((("digital signatures", "asymmetric cryptography and")))((("asymmetric cryptography")))Why is asymmetric cryptography used in bitcoin? It's not used to "encrypt" (make secret) the transactions. Rather, the useful property of asymmetric cryptography is the ability to generate _digital signatures_. A private key can be applied to the digital fingerprint of a transaction to produce a numerical signature. This signature can only be produced by someone with knowledge of the private key. However, anyone with access to the public key and the transaction fingerprint can use them to _verify_ the signature. This useful property of asymmetric cryptography makes it possible for anyone to verify every signature on every transaction, while ensuring that only the owners of private keys can produce valid signatures. -**** - -[[private_keys]] -==== Private Keys - -((("keys and addresses", "overview of", "private key generation")))((("warnings and cautions", "private key protection")))A private key is simply a number, picked at random. Ownership and control over the private key is the root of user control over all funds associated with the corresponding Bitcoin address. The private key is used to create signatures that are required to spend bitcoin by proving ownership of funds used in a transaction. The private key must remain secret at all times, because revealing it to third parties is equivalent to giving them control over the bitcoin secured by that key. The private key must also be backed up and protected from accidental loss, because if it's lost it cannot be recovered and the funds secured by it are forever lost, too. - -[TIP] -==== -The bitcoin private key is just a number. You can pick your private keys randomly using just a coin, pencil, and paper: toss a coin 256 times and you have the binary digits of a random private key you can use in a bitcoin wallet. The public key can then be generated from the private key. -==== - -===== Generating a private key from a random number - -The first and most important step in generating keys is to find a secure source of entropy, or randomness. Creating a bitcoin key is essentially the same as "Pick a number between 1 and 2^256^." The exact method you use to pick that number does not matter as long as it is not predictable or repeatable. Bitcoin software uses the underlying operating system's random number generators to produce 256 bits of entropy (randomness). Usually, the OS random number generator is initialized by a human source of randomness, which is why you may be asked to wiggle your mouse around for a few seconds. - -More precisely, the private key can be any number between +0+ and +n - 1+ inclusive, where n is a constant (n = 1.1578 * 10^77^, slightly less than 2^256^) defined as the order of the elliptic curve used in bitcoin (see <>). To create such a key, we randomly pick a 256-bit number and check that it is less than +n+. In programming terms, this is usually achieved by feeding a larger string of random bits, collected from a cryptographically secure source of randomness, into the SHA256 hash algorithm, which will conveniently produce a 256-bit number. If the result is less than +n+, we have a suitable private key. Otherwise, we simply try again with another random number. - -[WARNING] -==== -((("random numbers", "random number generation")))((("entropy", "random number generation")))Do not write your own code to create a random number or use a "simple" random number generator offered by your programming language. Use a cryptographically secure pseudorandom number generator (CSPRNG) with a seed from a source of sufficient entropy. Study the documentation of the random number generator library you choose to make sure it is cryptographically secure. Correct implementation of the CSPRNG is critical to the security of the keys. -==== - -The following is a randomly generated private key (k) shown in hexadecimal format (256 bits shown as 64 hexadecimal digits, each 4 bits): - ----- -1E99423A4ED27608A15A2616A2B0E9E52CED330AC530EDCC32C8FFC6A526AEDD ----- - - -[TIP] -==== -The size of bitcoin's private key space, (2^256^) is an unfathomably large number. It is approximately 10^77^ in decimal. For comparison, the visible universe is estimated to contain 10^80^ atoms. -==== - -((("dumpprivkey command")))To generate a new key with the Bitcoin Core client (see <>), use the +getnewaddress+ command. For security reasons it displays the address only, not the private key. To ask +bitcoind+ to expose the private key, use the +dumpprivkey+ command. The +dumpprivkey+ command shows the private key in a Base58 checksum-encoded format called the _Wallet Import Format_ (WIF), which we will examine in more detail in <>. Here's an example of generating and displaying a private key using these two commands: - ----- -$ bitcoin-cli getnewaddress -1J7mdg5rbQyUHENYdx39WVWK7fsLpEoXZy -$ bitcoin-cli dumpprivkey 1J7mdg5rbQyUHENYdx39WVWK7fsLpEoXZy -KxFC1jmwwCoACiCAWZ3eXa96mBM6tb3TYzGmf6YwgdGWZgawvrtJ ----- - -The +dumpprivkey+ command opens the wallet and extracts the private key that was generated by the +getnewaddress+ command. It is not possible for +bitcoind+ to know the private key from the address unless they are both stored in the wallet. - -[TIP] -===================================================================== -The +dumpprivkey+ command does not generate a private key from an address, as this is impossible. The command simply reveals the private key that is already known to the wallet and which was generated by the +getnewaddress+ command. -===================================================================== - -[role="pagebreak-before"] -You can also use the Bitcoin Explorer command-line tool (see <>) to generate and display private keys with the commands +seed+, +ec-new+, and +ec-to-wif+: - ----- -$ bx seed | bx ec-new | bx ec-to-wif -5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn ----- - -[[pubkey]] -==== Public Keys - -((("keys and addresses", "overview of", "public key calculation")))((("generator point")))The public key is calculated from the private key using elliptic curve multiplication, which is irreversible: _K_ = _k_ * _G_, where _k_ is the private key, _G_ is a constant point called the _generator point_, and _K_ is the resulting public key. The reverse operation, known as "finding the discrete logarithm"—calculating _k_ if you know __K__—is as difficult as trying all possible values of _k_, i.e., a brute-force search. Before we demonstrate how to generate a public key from a private key, let's look at elliptic curve cryptography in a bit more detail. - -[TIP] -==== -Elliptic curve multiplication is a type of function that cryptographers call a "one-way" function: it is easy to do in one direction (multiplication) and impossible to do in the reverse direction ("division", or finding the discrete logarithm). The owner of the private key can easily create the public key and then share it with the world knowing that no one can reverse the function and calculate the private key from the public key. This mathematical trick becomes the basis for unforgeable and secure digital signatures that prove ownership of bitcoin funds. -==== - -[[elliptic_curve]] -==== Elliptic Curve Cryptography Explained - -((("keys and addresses", "overview of", "elliptic curve cryptography")))((("elliptic curve cryptography", id="eliptic04")))((("cryptography", "elliptic curve cryptography", id="Celliptic04")))Elliptic curve cryptography is a type of asymmetric or public key cryptography based on the discrete logarithm problem as expressed by addition and multiplication on the points of an elliptic curve. - -<> is an example of an elliptic curve, similar to that used by bitcoin. - -[[ecc-curve]] -[role="smallerthirty"] -.An elliptic curve -image::images/mbc2_0402.png["ecc-curve"] - -Bitcoin uses a specific elliptic curve and set of mathematical constants, as defined in a standard called +secp256k1+, established by the National Institute of Standards and Technology (NIST). The +secp256k1+ curve is defined by the following function, which produces an elliptic curve: - -[latexmath] -++++ -\begin{equation} -{y^2 = (x^3 + 7)}~\text{over}~(\mathbb{F}_p) -\end{equation} -++++ - -or - -[latexmath] -++++ -\begin{equation} -{y^2 \mod p = (x^3 + 7) \mod p} -\end{equation} -++++ - -The _mod p_ (modulo prime number p) indicates that this curve is over a finite field of prime order _p_, also written as latexmath:[\( \mathbb{F}_p \)], where p = 2^256^ – 2^32^ – 2^9^ – 2^8^ – 2^7^ – 2^6^ – 2^4^ – 1, a very large prime number. - -Because this curve is defined over a finite field of prime order instead of over the real numbers, it looks like a pattern of dots scattered in two dimensions, which makes it difficult to visualize. However, the math is identical to that of an elliptic curve over real numbers. As an example, <> shows the same elliptic curve over a much smaller finite field of prime order 17, showing a pattern of dots on a grid. The +secp256k1+ bitcoin elliptic curve can be thought of as a much more complex pattern of dots on a unfathomably large grid. - -[[ecc-over-F17-math]] -[role="smallersixty"] -.Elliptic curve cryptography: visualizing an elliptic curve over F(p), with p=17 -image::images/mbc2_0403.png["ecc-over-F17-math"] - -So, for example, the following is a point P with coordinates (x,y) that is a point on the +secp256k1+ curve: - ----- -P = (55066263022277343669578718895168534326250603453777594175500187360389116729240, 32670510020758816978083085130507043184471273380659243275938904335757337482424) ----- - -<> shows how you can check this yourself using Python: - -[[example_4_1]] -.Using Python to confirm that this point is on the elliptic curve -==== -[source, pycon] ----- -Python 3.4.0 (default, Mar 30 2014, 19:23:13) -[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin -Type "help", "copyright", "credits" or "license" for more information. ->>> p = 115792089237316195423570985008687907853269984665640564039457584007908834671663 ->>> x = 55066263022277343669578718895168534326250603453777594175500187360389116729240 ->>> y = 32670510020758816978083085130507043184471273380659243275938904335757337482424 ->>> (x ** 3 + 7 - y**2) % p -0 ----- -==== - -In elliptic curve math, there is a point called the "point at infinity," which roughly corresponds to the role of zero in addition. On computers, it's sometimes represented by x = y = 0 (which doesn't satisfy the elliptic curve equation, but it's an easy separate case that can be checked). - -There is also a pass:[+] operator, called "addition," which has some properties similar to the traditional addition of real numbers that gradeschool children learn. Given two points P~1~ and P~2~ on the elliptic curve, there is a third point P~3~ = P~1~ + P~2~, also on the elliptic curve. - -Geometrically, this third point P~3~ is calculated by drawing a line between P~1~ and P~2~. This line will intersect the elliptic curve in exactly one additional place. Call this point P~3~' = (x, y). Then reflect in the x-axis to get P~3~ = (x, –y). - -There are a couple of special cases that explain the need for the "point at infinity." - -If P~1~ and P~2~ are the same point, the line "between" P~1~ and P~2~ should extend to be the tangent on the curve at this point P~1~. This tangent will intersect the curve in exactly one new point. You can use techniques from calculus to determine the slope of the tangent line. These techniques curiously work, even though we are restricting our interest to points on the curve with two integer coordinates! - -In some cases (i.e., if P~1~ and P~2~ have the same x values but different y values), the line between P~1~ and P~2~ will be exactly vertical, in which case P~3~ = "point at infinity." - -If P~1~ is the "point at infinity," then P~1~ + P~2~ = P~2~. Similarly, if P~2~ is the point at infinity, then P~1~ + P~2~ = P~1~. This shows how the point at infinity plays the role of zero. - -It turns out that pass:[+] is associative, which means that (A pass:[+] B) pass:[+] C = A pass:[+] (B pass:[+] C). That means we can write A pass:[+] B pass:[+] C without parentheses and without ambiguity. - -Now that we have defined addition, we can define multiplication in the standard way that extends addition. For a point P on the elliptic curve, if k is a whole number, then kP = P + P + P + ... + P (k times). Note that k is sometimes confusingly called an "exponent" in this case.((("", startref="eliptic04")))((("", startref="Celliptic04"))) - -[[public_key_derivation]] -==== Generating a Public Key - -((("keys and addresses", "overview of", "public key generation")))((("generator point")))Starting with a private key in the form of a randomly generated number _k_, we multiply it by a predetermined point on the curve called the _generator point_ _G_ to produce another point somewhere else on the curve, which is the corresponding public key _K_. The generator point is specified as part of the +secp256k1+ standard and is always the same for all keys in bitcoin: - -[latexmath] -++++ -\begin{equation} -{K = k * G} -\end{equation} -++++ - -where _k_ is the private key, _G_ is the generator point, and _K_ is the resulting public key, a point on the curve. Because the generator point is always the same for all bitcoin users, a private key _k_ multiplied with _G_ will always result in the same public key _K_. The relationship between _k_ and _K_ is fixed, but can only be calculated in one direction, from _k_ to _K_. That's why a Bitcoin address (derived from _K_) can be shared with anyone and does not reveal the user's private key (_k_). - -[TIP] -==== -A private key can be converted into a public key, but a public key cannot be converted back into a private key because the math only works one way. -==== - -Implementing the elliptic curve multiplication, we take the private key _k_ generated previously and multiply it with the generator point G to find the public key _K_: - ----- -K = 1E99423A4ED27608A15A2616A2B0E9E52CED330AC530EDCC32C8FFC6A526AEDD * G ----- - -Public key _K_ is defined as a point +K = (x,y)+: - ----- -K = (x, y) - -where, - -x = F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A -y = 07CF33DA18BD734C600B96A72BBC4749D5141C90EC8AC328AE52DDFE2E505BDB ----- - -To visualize multiplication of a point with an integer, we will use the simpler elliptic curve over real numbers—remember, the math is the same. Our goal is to find the multiple _kG_ of the generator point _G_, which is the same as adding _G_ to itself, _k_ times in a row. In elliptic curves, adding a point to itself is the equivalent of drawing a tangent line on the point and finding where it intersects the curve again, then reflecting that point on the x-axis. - -<> shows the process for deriving _G_, _2G_, _4G_, and _8G_ as a geometric operation on the curve. - -[TIP] -==== -((("secp256k1 optimized C library")))Bitcoin uses the https://github.com/bitcoin-core/secp256k1[secp256k1 optimized C library] to do the elliptic curve math.((("", startref="KAover04"))) -==== - -[[ecc_illustrated]] -.Elliptic curve cryptography: visualizing the multiplication of a point G by an integer k on an elliptic curve -image::images/mbc2_0404.png["ecc_illustrated"] - -=== Bitcoin Addresses - -((("keys and addresses", "Bitcoin addresses", id="KAaddress04")))A Bitcoin address is a string of digits and characters that can be shared with anyone who wants to send you money. Addresses produced from public keys consist of a string of numbers and letters, beginning with the digit "1". Here's an example of a Bitcoin address: - ----- -1J7mdg5rbQyUHENYdx39WVWK7fsLpEoXZy ----- - - -The Bitcoin address is what appears most commonly in a transaction as the "recipient" of the funds. If we compare a bitcoin transaction to a paper check, the Bitcoin address is the beneficiary, which is what we write on the line after "Pay to the order of." On a paper check, that beneficiary can sometimes be the name of a bank account holder, but can also include corporations, institutions, or even cash. Because paper checks do not need to specify an account, but rather use an abstract name as the recipient of funds, they are very flexible payment instruments. Bitcoin transactions use a similar abstraction, the Bitcoin address, to make them very flexible. A Bitcoin address can represent the owner of a private/public key pair, or it can represent something else, such as a payment script, as we will see in <>. For now, let's examine the simple case, a Bitcoin address that represents, and is derived from, a public key. - -((("addresses", "algorithms used to create")))The Bitcoin address is derived from the public key through the use of one-way cryptographic hashing. A "hashing algorithm" or simply "hash algorithm" is a one-way function that produces a fingerprint or "hash" of an arbitrary-sized input. Cryptographic hash functions are used extensively in bitcoin: in Bitcoin addresses, in script addresses, and in the mining Proof-of-Work algorithm. The algorithms used to make a Bitcoin address from a public key are the Secure Hash Algorithm (SHA) and the RACE Integrity Primitives Evaluation Message Digest (RIPEMD), specifically SHA256 and RIPEMD160. - -Starting with the public key _K_, we compute the SHA256 hash and then compute the RIPEMD160 hash of the result, producing a 160-bit (20-byte) number: - -[latexmath] -++++ -\begin{equation} -{A = RIPEMD160(SHA256(K))} -\end{equation} -++++ - -where _K_ is the public key and _A_ is the resulting Bitcoin address. - - -[TIP] -==== -A Bitcoin address is _not_ the same as a public key. Bitcoin addresses are derived from a public key using a one-way function. -==== - -Bitcoin addresses are almost always encoded as "Base58Check" (see <>), which uses 58 characters (a Base58 number system) and a checksum to help human readability, avoid ambiguity, and protect against errors in address transcription and entry. Base58Check is also used in many other ways in bitcoin, whenever there is a need for a user to read and correctly transcribe a number, such as a Bitcoin address, a private key, an encrypted key, or a script hash. In the next section we will examine the mechanics of Base58Check encoding and decoding and the resulting representations. <> illustrates the conversion of a public key into a Bitcoin address. - -[[pubkey_to_address]] -.Public key to Bitcoin address: conversion of a public key into a Bitcoin address -image::images/mbc2_0405.png["pubkey_to_address"] - -[[base58]] -==== Base58 and Base58Check Encoding - -((("keys and addresses", "Bitcoin addresses", "Base58 and Base58check encoding")))((("Base58 and Base58check encoding", id="base5804")))((("addresses", "Base58 and Base58check encoding", id="Abase5804")))In order to represent long numbers in a compact way, using fewer symbols, many computer systems use mixed-alphanumeric representations with a base (or radix) higher than 10. For example, whereas the traditional decimal system uses the 10 numerals 0 through 9, the hexadecimal system uses 16, with the letters A through F as the six additional symbols. A number represented in hexadecimal format is shorter than the equivalent decimal representation. Even more compact, Base64 representation uses 26 lowercase letters, 26 capital letters, 10 numerals, and 2 more characters such as “`+`” and "/" to transmit binary data over text-based media such as email. Base64 is most commonly used to add binary attachments to email. Base58 is a text-based binary-encoding format developed for use in bitcoin and used in many other cryptocurrencies. It offers a balance between compact representation, readability, and error detection and prevention. Base58 is a subset of Base64, using upper- and lowercase letters and numbers, but omitting some characters that are frequently mistaken for one another and can appear identical when displayed in certain fonts. Specifically, Base58 is Base64 without the 0 (number zero), O (capital o), l (lower L), I (capital i), and the symbols “`+`” and "/". Or, more simply, it is a set of lowercase and capital letters and numbers without the four (0, O, l, I) just mentioned. <> shows the full Base58 alphabet. - -[[base58alphabet]] -.Bitcoin's Base58 alphabet -==== ----- -123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz ----- -==== - - -To add extra security against typos or transcription errors, Base58Check is a Base58 encoding format, frequently used in bitcoin, which has a built-in error-checking code. The checksum is an additional four bytes added to the end of the data that is being encoded. The checksum is derived from the hash of the encoded data and can therefore be used to detect and prevent transcription and typing errors. When presented with Base58Check code, the decoding software will calculate the checksum of the data and compare it to the checksum included in the code. If the two do not match, an error has been introduced and the Base58Check data is invalid. This prevents a mistyped Bitcoin address from being accepted by the wallet software as a valid destination, an error that would otherwise result in loss of funds. - -To convert data (a number) into a Base58Check format, we first add a prefix to the data, called the "version byte," which serves to easily identify the type of data that is encoded. For example, in the case of a Bitcoin address the prefix is zero (0x00 in hex), whereas the prefix used when encoding a private key is 128 (0x80 in hex). A list of common version prefixes is shown in <>. - -Next, we compute the "double-SHA" checksum, meaning we apply the SHA256 hash-algorithm twice on the previous result (prefix and data): - ----- -checksum = SHA256(SHA256(prefix+data)) ----- - -From the resulting 32-byte hash (hash-of-a-hash), we take only the first four bytes. These four bytes serve as the error-checking code, or checksum. The checksum is concatenated (appended) to the end. - -The result is composed of three items: a prefix, the data, and a checksum. This result is encoded using the Base58 alphabet described previously. <> illustrates the Base58Check encoding process. - -[[base58check_encoding]] -.Base58Check encoding: a Base58, versioned, and checksummed format for unambiguously encoding bitcoin data -image::images/mbc2_0406.png["Base58CheckEncoding"] - -In bitcoin, most of the data presented to the user is Base58Check-encoded to make it compact, easy to read, and easy to detect errors. The version prefix in Base58Check encoding is used to create easily distinguishable formats, which when encoded in Base58 contain specific characters at the beginning of the Base58Check-encoded payload. These characters make it easy for humans to identify the type of data that is encoded and how to use it. This is what differentiates, for example, a Base58Check-encoded Bitcoin address that starts with a 1 from a Base58Check-encoded private key WIF that starts with a 5. Some example version prefixes and the resulting Base58 characters are shown in <>. - -[[base58check_versions]] -.Base58Check version prefix and encoded result examples -[options="header"] -|======= -|Type| Version prefix (hex)| Base58 result prefix -| Bitcoin Address | 0x00 | 1 -| Pay-to-Script-Hash Address | 0x05 | 3 -| Bitcoin Testnet Address | 0x6F | m or n -| Private Key WIF | 0x80 | 5, K, or L -| BIP-38 Encrypted Private Key | 0x0142 | 6P -| BIP-32 Extended Public Key | 0x0488B21E | xpub -|======= - -==== Key Formats - -((("keys and addresses", "Bitcoin addresses", "key formats")))Both private and public keys can be represented in a number of different formats. These representations all encode the same number, even though they look different. These formats are primarily used to make it easy for people to read and transcribe keys without introducing errors. - -[[priv_formats]] -===== Private key formats - -((("public and private keys", "private key formats")))The private key can be represented in a number of different formats, all of which correspond to the same 256-bit number. <> shows three common formats used to represent private keys. Different formats are used in different circumstances. Hexadecimal and raw binary formats are used internally in software and rarely shown to users. The WIF is used for import/export of keys between wallets and often used in QR code (barcode) representations of private keys. - -[[table_4-2]] -.Private key representations (encoding formats) -[options="header"] -|======= -|Type|Prefix|Description -| Raw | None | 32 bytes -| Hex | None | 64 hexadecimal digits -| WIF | 5 | Base58Check encoding: Base58 with version prefix of 0x80 and 4-byte checksum -| WIF-compressed | K or L | As above, with added suffix 0x01 before encoding -|======= - -<> shows the private key generated in these three formats. - -[[table_4-3]] -.Example: Same key, different formats -[options="header"] -|======= -|Format | Private key -| Hex | 1e99423a4ed27608a15a2616a2b0e9e52ced330ac530edcc32c8ffc6a526aedd -| WIF | 5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn -| WIF-compressed | KxFC1jmwwCoACiCAWZ3eXa96mBM6tb3TYzGmf6YwgdGWZgawvrtJ -|======= - -All of these representations are different ways of showing the same number, the same private key. They look different, but any one format can easily be converted to any other format. Note that the "raw binary" is not shown in <> as any encoding for display here would, by definition, not be raw binary data. - -We use the +wif-to-ec+ command from Bitcoin Explorer (see <>) to show that both WIF keys represent the same private key: - ----- -$ bx wif-to-ec 5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn -1e99423a4ed27608a15a2616a2b0e9e52ced330ac530edcc32c8ffc6a526aedd - -$ bx wif-to-ec KxFC1jmwwCoACiCAWZ3eXa96mBM6tb3TYzGmf6YwgdGWZgawvrtJ -1e99423a4ed27608a15a2616a2b0e9e52ced330ac530edcc32c8ffc6a526aedd ----- - -===== Decode from Base58Check - -The Bitcoin Explorer commands (see <>) make it easy to write shell scripts and command-line "pipes" that manipulate bitcoin keys, addresses, and transactions. You can use Bitcoin Explorer to decode the Base58Check format on the command line. - -We use the +base58check-decode+ command to decode the uncompressed key: - ----- -$ bx base58check-decode 5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn -wrapper -{ - checksum 4286807748 - payload 1e99423a4ed27608a15a2616a2b0e9e52ced330ac530edcc32c8ffc6a526aedd - version 128 -} ----- - -The result contains the key as payload, the WIF version prefix 128, and a checksum. - -Notice that the "payload" of the compressed key is appended with the suffix +01+, signalling that the derived public key is to be compressed: - ----- -$ bx base58check-decode KxFC1jmwwCoACiCAWZ3eXa96mBM6tb3TYzGmf6YwgdGWZgawvrtJ -wrapper -{ - checksum 2339607926 - payload 1e99423a4ed27608a15a2616a2b0e9e52ced330ac530edcc32c8ffc6a526aedd01 - version 128 -} ----- - -===== Encode from hex to Base58Check - -To encode into Base58Check (the opposite of the previous command), we use the +base58check-encode+ command from Bitcoin Explorer (see <>) and provide the hex private key, followed by the WIF version prefix 128: - ----- -bx base58check-encode 1e99423a4ed27608a15a2616a2b0e9e52ced330ac530edcc32c8ffc6a526aedd --version 128 -5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn ----- - -===== Encode from hex (compressed key) to Base58Check - -To encode into Base58Check as a "compressed" private key (see <>), we append the suffix +01+ to the hex key and then encode as in the preceding section: - ----- -$ bx base58check-encode 1e99423a4ed27608a15a2616a2b0e9e52ced330ac530edcc32c8ffc6a526aedd01 --version 128 -KxFC1jmwwCoACiCAWZ3eXa96mBM6tb3TYzGmf6YwgdGWZgawvrtJ ----- - -The resulting WIF-compressed format starts with a "K." This denotes that the private key within has a suffix of "01" and will be used to produce compressed public keys only (see <>). - -===== Public key formats - -((("public and private keys", "public key formats")))Public keys are also presented in different ways, usually as either _compressed_ or _uncompressed_ public keys. - -As we saw previously, the public key is a point on the elliptic curve consisting of a pair of coordinates +(x,y)+. It is usually presented with the prefix +04+ followed by two 256-bit numbers: one for the _x_ coordinate of the point, the other for the _y_ coordinate. The prefix +04+ is used to distinguish uncompressed public keys from compressed public keys that begin with a +02+ or a +03+. - -Here's the public key generated by the private key we created earlier, shown as the coordinates +x+ and +y+: - ----- -x = F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A -y = 07CF33DA18BD734C600B96A72BBC4749D5141C90EC8AC328AE52DDFE2E505BDB ----- - -Here's the same public key shown as a 520-bit number (130 hex digits) with the prefix +04+ followed by +x+ and then +y+ coordinates, as +04 x y+: - -++++ -
-K = 04F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A↵
-07CF33DA18BD734C600B96A72BBC4749D5141C90EC8AC328AE52DDFE2E505BDB
-
-++++ - -[[comp_pub]] -===== Compressed public keys - -((("public and private keys", "compressed public keys")))Compressed public keys were introduced to bitcoin to reduce the size of transactions and conserve disk space on nodes that store the Bitcoin blockchain database. Most transactions include the public key, which is required to validate the owner's credentials and spend the bitcoin. Each public key requires 520 bits (prefix + x + y), which when multiplied by several hundred transactions per block, or tens of thousands of transactions per day, adds a significant amount of data to the blockchain. - -As we saw in the section <>, a public key is a point (x,y) on an elliptic curve. Because the curve expresses a mathematical function, a point on the curve represents a solution to the equation and, therefore, if we know the _x_ coordinate we can calculate the _y_ coordinate by solving the equation y^2^ mod p = (x^3^ + 7) mod p. That allows us to store only the _x_ coordinate of the public key point, omitting the _y_ coordinate and reducing the size of the key and the space required to store it by 256 bits. An almost 50% reduction in size in every transaction adds up to a lot of data saved over time! - -Whereas uncompressed public keys have a prefix of +04+, compressed public keys start with either a +02+ or a +03+ prefix. Let's look at why there are two possible prefixes: because the left side of the equation is __y__^2^, the solution for _y_ is a square root, which can have a positive or negative value. Visually, this means that the resulting _y_ coordinate can be above or below the x-axis. As you can see from the graph of the elliptic curve in <>, the curve is symmetric, meaning it is reflected like a mirror by the x-axis. So, while we can omit the _y_ coordinate we have to store the _sign_ of _y_ (positive or negative); or in other words, we have to remember if it was above or below the x-axis because each of those options represents a different point and a different public key. When calculating the elliptic curve in binary arithmetic on the finite field of prime order p, the _y_ coordinate is either even or odd, which corresponds to the positive/negative sign as explained earlier. Therefore, to distinguish between the two possible values of _y_, we store a compressed public key with the prefix +02+ if the _y_ is even, and +03+ if it is odd, allowing the software to correctly deduce the _y_ coordinate from the _x_ coordinate and uncompress the public key to the full coordinates of the point. Public key compression is illustrated in <>. - -Here's the same public key generated previously, shown as a compressed public key stored in 264 bits (66 hex digits) with the prefix +03+ indicating the _y_ coordinate is odd: - ----- -K = 03F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A ----- - -This compressed public key corresponds to the same private key, meaning it is generated from the same private key. However, it looks different from the uncompressed public key. More importantly, if we convert this compressed public key to a Bitcoin address using the double-hash function (+RIPEMD160(SHA256(K))+) it will produce a _different_ Bitcoin address. This can be confusing, because it means that a single private key can produce a public key expressed in two different formats (compressed and uncompressed) that produce two different Bitcoin addresses. However, the private key is identical for both Bitcoin addresses. - -[[pubkey_compression]] -[role="smallerseventy"] -.Public key compression -image::images/mbc2_0407.png["pubkey_compression"] - -Compressed public keys are gradually becoming the default across Bitcoin clients, which is having a significant impact on reducing the size of transactions and therefore the blockchain. However, not all clients support compressed public keys yet. Newer clients that support compressed public keys have to account for transactions from older clients that do not support compressed public keys. This is especially important when a wallet application is importing private keys from another bitcoin wallet application, because the new wallet needs to scan the blockchain to find transactions corresponding to these imported keys. Which Bitcoin addresses should the bitcoin wallet scan for? The Bitcoin addresses produced by uncompressed public keys, or the Bitcoin addresses produced by compressed public keys? Both are valid Bitcoin addresses, and can be signed for by the private key, but they are different addresses! - -To resolve this issue, when private keys are exported from a wallet, the WIF that is used to represent them is implemented differently in newer bitcoin wallets, to indicate that these private keys have been used to produce _compressed_ public keys and therefore _compressed_ Bitcoin addresses. This allows the importing wallet to distinguish between private keys originating from older or newer wallets and search the blockchain for transactions with Bitcoin addresses corresponding to the uncompressed, or the compressed, public keys, respectively. Let's look at how this works in more detail, in the next section. - -[[comp_priv]] -===== Compressed private keys - -((("public and private keys", "compressed private keys")))Ironically, the term "compressed private key" is a misnomer, because when a private key is exported as WIF-compressed it is actually one byte _longer_ than an "uncompressed" private key. That is because the private key has an added one-byte suffix (shown as 01 in hex in <>), which signifies that the private key is from a newer wallet and should only be used to produce compressed public keys. Private keys are not themselves compressed and cannot be compressed. The term "compressed private key" really means "private key from which only compressed public keys should be derived," whereas "uncompressed private key" really means "private key from which only uncompressed public keys should be derived." You should only refer to the export format as "WIF-compressed" or "WIF" and not refer to the private key itself as "compressed" to avoid further confusion. - -<> shows the same key, encoded in WIF and WIF-compressed formats. - -[[table_4-4]] -.Example: Same key, different formats -[options="header"] -|======= -|Format | Private key -| Hex | 1E99423A4ED27608A15A2616A2B0E9E52CED330AC530EDCC32C8FFC6A526AEDD -| WIF | 5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn -| Hex-compressed | 1E99423A4ED27608A15A2616A2B0E9E52CED330AC530EDCC32C8FFC6A526AEDD01 -| WIF-compressed | KxFC1jmwwCoACiCAWZ3eXa96mBM6tb3TYzGmf6YwgdGWZgawvrtJ -|======= - -Notice that the hex-compressed private key format has one extra byte at the end (01 in hex). While the Base58Check version prefix is the same (0x80) for both WIF and WIF-compressed formats, the addition of one byte on the end of the number causes the first character of the Base58 encoding to change from a 5 to either a _K_ or _L_. Think of this as the Base58 equivalent of the decimal encoding difference between the number 100 and the number 99. While 100 is one digit longer than 99, it also has a prefix of 1 instead of a prefix of 9. As the length changes, it affects the prefix. In Base58, the prefix 5 changes to a _K_ or _L_ as the length of the number increases by one byte. - -Remember, these formats are _not_ used interchangeably. In a newer wallet that implements compressed public keys, the private keys will only ever be exported as WIF-compressed (with a _K_ or _L_ prefix). If the wallet is an older implementation and does not use compressed public keys, the private keys will only ever be exported as WIF (with a 5 prefix). The goal here is to signal to the wallet importing these private keys whether it must search the blockchain for compressed or uncompressed public keys and addresses. - -If a bitcoin wallet is able to implement compressed public keys, it will use those in all transactions. The private keys in the wallet will be used to derive the public key points on the curve, which will be compressed. The compressed public keys will be used to produce Bitcoin addresses and those will be used in transactions. When exporting private keys from a new wallet that implements compressed public keys, the WIF is modified, with the addition of a one-byte suffix +01+ to the private key. The resulting Base58Check-encoded private key is called a "compressed WIF" and starts with the letter _K_ or _L_, instead of starting with "5" as is the case with WIF-encoded (noncompressed) keys from older wallets. - - -[TIP] -==== -"Compressed private keys" is a misnomer! They are not compressed; rather, WIF-compressed signifies that the keys should only be used to derive compressed public keys and their corresponding Bitcoin addresses. Ironically, a "WIF-compressed" encoded private key is one byte longer because it has the added +01+ suffix to distinguish it from an "uncompressed" one.((("", startref="KAaddress04"))) -==== - -=== Implementing Keys and Addresses in Cpass:[++] - -Let's look at the complete process of creating a Bitcoin address, from a private key, to a public key (a point on the elliptic curve), to a double-hashed address, and finally, the Base58Check encoding. The C++ code in <> shows the complete step-by-step process, from private key to Base58Check-encoded Bitcoin address. The code example uses the libbitcoin library introduced in <> for some helper functions. - -[[addr_example]] -.Creating a Base58Check-encoded Bitcoin address from a private key -==== -[role="c_less_space"] -[source, cpp] ----- -include::code/addr.cpp[] ----- -==== - -The code uses a predefined private key to produce the same Bitcoin address every time it is run, as shown in <>.((("", startref="base5804")))((("", startref="Abase5804"))) - -[[addr_example_run]] -.Compiling and running the addr code -==== -[source,bash] ----- -# Compile the addr.cpp code -$ g++ -o addr addr.cpp -std=c++11 $(pkg-config --cflags --libs libbitcoin) -# Run the addr executable -$ ./addr -Public key: 0202a406624211f2abbdc68da3df929f938c3399dd79fac1b51b0e4ad1d26a47aa -Address: 1PRTTaJesdNovgne6Ehcdu1fpEdX7913CK ----- -==== - -[TIP] -==== -The code in <> produces a Bitcoin address (+1PRTT...+) from a _compressed_ public key (see <>). If you used the uncompressed public key instead, it would produce a different Bitcoin address (+14K1y...+). -==== - -=== Implementing Keys and Addresses in Python - -((("keys and addresses", "implementing in Python", id="KApython04")))((("pybitcointools")))The most comprehensive bitcoin library in Python is https://github.com/vbuterin/pybitcointools[pybitcointools] by Vitalik Buterin. In <>, we use the pybitcointools library (imported as "bitcoin") to generate and display keys and addresses in various formats. - -[[key-to-address_script]] -.Key and address generation and formatting with the pybitcointools library -==== -[source,python] ----- -include::code/key-to-address-ecc-example.py[] ----- -==== - -<> shows the output from running this code. - -[[key-to-address_script_run]] -.Running key-to-address-ecc-example.py -==== -++++ -
-$ python key-to-address-ecc-example.py
-Private Key (hex) is:
- 3aba4162c7251c891207b747840551a71939b0de081f85c4e44cf7c13e41daa6
-Private Key (decimal) is:
- 26563230048437957592232553826663696440606756685920117476832299673293013768870
-Private Key (WIF) is:
- 5JG9hT3beGTJuUAmCQEmNaxAuMacCTfXuw1R3FCXig23RQHMr4K
-Private Key Compressed (hex) is:
- 3aba4162c7251c891207b747840551a71939b0de081f85c4e44cf7c13e41daa601
-Private Key (WIF-Compressed) is:
- KyBsPXxTuVD82av65KZkrGrWi5qLMah5SdNq6uftawDbgKa2wv6S
-Public Key (x,y) coordinates is:
- (41637322786646325214887832269588396900663353932545912953362782457239403430124L,
- 16388935128781238405526710466724741593761085120864331449066658622400339362166L)
-Public Key (hex) is:
- 045c0de3b9c8ab18dd04e3511243ec2952002dbfadc864b9628910169d9b9b00ec↵
-243bcefdd4347074d44bd7356d6a53c495737dd96295e2a9374bf5f02ebfc176
-Compressed Public Key (hex) is:
- 025c0de3b9c8ab18dd04e3511243ec2952002dbfadc864b9628910169d9b9b00ec
-Bitcoin Address (b58check) is:
- 1thMirt546nngXqyPEz532S8fLwbozud8
-Compressed Bitcoin Address (b58check) is:
- 14cxpo3MBCYYWCgF74SWTdcmxipnGUsPw3
-
-++++ -==== - - -<> is another example, using the Python ECDSA library for the elliptic curve math and without using any specialized bitcoin libraries. - -[[ec_math]] -.A script demonstrating elliptic curve math used for bitcoin keys -==== -[source, python] ----- -include::code/ec-math.py[] ----- -==== - -<> shows the output produced by running this script. - -[WARNING] -==== -<> ((("random numbers", "os.urandom", see="entropy")))((("entropy", "os.urandom", see="random numbers")))((("random numbers", "random number generation")))((("entropy", "random number generation")))uses +os.urandom+, which reflects a cryptographically secure random number generator (CSRNG) provided by the underlying operating system. Caution: Depending on the OS, +os.urandom+ may _not_ be implemented with sufficient security or seeded properly and may _not_ be appropriate for generating production-quality bitcoin keys.((("", startref="KApython04"))) -==== - -[[ec_math_run]] -.Installing the Python ECDSA library and running the ec_math.py script -==== ----- -# Install Python PIP package manager -$ sudo apt-get install python-pip -# Install the Python ECDSA library -$ sudo pip install ecdsa -# Run the script -$ python ec-math.py -Secret: 38090835015954358862481132628887443905906204995912378278060168703580660294000 -EC point: (70048853531867179489857750497606966272382583471322935454624595540007269312627, 105262206478686743191060800263479589329920209527285803935736021686045542353380) -BTC public key: 029ade3effb0a67d5c8609850d797366af428f4a0d5194cb221d807770a1522873 ----- -==== - -=== Advanced Keys and Addresses - -((("keys and addresses", "advanced forms", id="KAadvanced04")))In the following sections we will look at advanced forms of keys and addresses, such as encrypted private keys, script and multisignature addresses, vanity addresses, and paper wallets. - -[[p2sh_addresses]] -==== Pay-to-Script Hash (P2SH) and Multisig Addresses - -((("keys and addresses", "advanced forms", "pay-to-script hash and multisig addresses")))((("Pay-to-Script-Hash (P2SH)", "multisig addresses and")))((("multisig addresses")))((("addresses", "multisig addresses")))As we know, traditional Bitcoin addresses begin with the number “1” and are derived from the public key, which is derived from the private key. Although anyone can send bitcoin to a “1” address, that bitcoin can only be spent by presenting the corresponding private key signature and public key. - -((("bitcoin improvement proposals", "Pay to Script Hash (BIP-16)")))Bitcoin addresses that begin with the number “3” are pay-to-script hash (P2SH) addresses, sometimes erroneously called multisignature or multisig addresses. They designate the beneficiary of a bitcoin transaction as the hash of a script, instead of the owner of a public key. The feature was introduced in January 2012 with BIP-16 (see <>), and is being widely adopted because it provides the opportunity to add functionality to the address itself. Unlike transactions that "send" funds to traditional “1” Bitcoin addresses, also known as a pay-to-public-key-hash (P2PKH), funds sent to “3” addresses require something more than the presentation of one public key and one private key signature as proof of ownership. The requirements are designated at the time the address is created, within the script, and all inputs to this address will be encumbered with the same requirements. - -A P2SH address is created from a transaction script, which defines who can spend a transaction output (for more details, see <>). Encoding a P2SH address involves using the same double-hash function as used during creation of a Bitcoin address, only applied on the script instead of the public key: - ----- -script hash = RIPEMD160(SHA256(script)) ----- - -The resulting "script hash" is encoded with Base58Check with a version prefix of 5, which results in an encoded address starting with a +3+. An example of a P2SH address is +3F6i6kwkevjR7AsAd4te2YB2zZyASEm1HM+, which can be derived using the Bitcoin Explorer commands +script-encode+, +sha256+, +ripemd160+, and +base58check-encode+ (see <>) as follows: - ----- -$ echo \ -'DUP HASH160 [89abcdefabbaabbaabbaabbaabbaabbaabbaabba] EQUALVERIFY CHECKSIG' > script -$ bx script-encode < script | bx sha256 | bx ripemd160 \ -| bx base58check-encode --version 5 -3F6i6kwkevjR7AsAd4te2YB2zZyASEm1HM ----- - -[TIP] -==== -P2SH is not necessarily the same as a multisignature standard transaction. A P2SH address _most often_ represents a multi-signature script, but it might also represent a script encoding other types of transactions. -==== - -===== Multisignature addresses and P2SH - -Currently, the most common implementation of the P2SH function is the multi-signature address script. As the name implies, the underlying script requires a minimum number of signatures to prove ownership and therefore spend funds. The bitcoin multi-signature feature is designed to require M signatures (also known as the “threshold”) from a total of N keys, known as an M-of-N multisig, where M is equal to or less than N. For example, Bob the coffee shop owner from <> could use a multisignature address requiring 1-of-2 signatures from a key belonging to him and a key belonging to his spouse, ensuring either of them could sign to spend a transaction output locked to this address. This would be similar to a “joint account” as implemented in traditional banking where either spouse can spend with a single signature. Or Gopesh,((("use cases", "offshore contract services"))) the web designer paid by Bob to create a website, might have a 2-of-3 multisignature address for his business that ensures that no funds can be spent unless at least two of the business partners sign a transaction. - -We will explore how to create transactions that spend funds from P2SH (and multi-signature) addresses in <>. - -==== Vanity Addresses - -((("keys and addresses", "advanced forms", "vanity addresses")))((("vanity addresses", id="vanity04")))((("addresses", "vanity addresses", id="Avanity04")))Vanity addresses are valid Bitcoin addresses that contain human-readable messages. For example, +1LoveBPzzD72PUXLzCkYAtGFYmK5vYNR33+ is a valid address that contains the letters forming the word "Love" as the first four Base58 letters. Vanity addresses require generating and testing billions of candidate private keys, until a Bitcoin address with the desired pattern is found. Although there are some optimizations in the vanity generation algorithm, the process essentially involves picking a private key at random, deriving the public key, deriving the Bitcoin address, and checking to see if it matches the desired vanity pattern, repeating billions of times until a match is found. - -Once a vanity address matching the desired pattern is found, the private key from which it was derived can be used by the owner to spend bitcoin in exactly the same way as any other address. Vanity addresses are no less or more secure than any other address. They depend on the same Elliptic Curve Cryptography (ECC) and SHA as any other address. You can no more easily find the private key of an address starting with a vanity pattern than you can of any other address. - -In <>, we introduced Eugenia, a children's charity director operating in the Philippines. Let's say that Eugenia is organizing a bitcoin fundraising drive and wants to use a vanity Bitcoin address to publicize the fundraising. Eugenia will create a vanity address that starts with "1Kids" to promote the children's charity fundraiser. Let's see how this vanity address will be created and what it means for the security of Eugenia's charity.((("use cases", "charitable donations", startref="eugeniafour"))) - -===== Generating vanity addresses - -It's important to realize that a Bitcoin address is simply a number represented by symbols in the Base58 alphabet. The search for a pattern like "1Kids" can be seen as searching for an address in the range from +1Kids11111111111111111111111111111+ to +1Kidszzzzzzzzzzzzzzzzzzzzzzzzzzzzz+. There are approximately 58^29^ (approximately 1.4 * 10^51^) addresses in that range, all starting with "1Kids." <> shows the range of addresses that have the prefix 1Kids. - -[[table_4-11]] -.The range of vanity addresses starting with "1Kids" -|======= -| *From* | +1Kids11111111111111111111111111111+ -| | +1Kids11111111111111111111111111112+ -| | +1Kids11111111111111111111111111113+ -| | +...+ -| *To* | +1Kidszzzzzzzzzzzzzzzzzzzzzzzzzzzzz+ -|======= - -Let's look at the pattern "1Kids" as a number and see how frequently we might find this pattern in a Bitcoin address (see <>). An average desktop computer PC, without any specialized hardware, can search approximately 100,000 keys per second. - -[[table_4-12]] -.The frequency of a vanity pattern (1KidsCharity) and average search time on a desktop PC -[options="header"] -|======= -| Length | Pattern | Frequency | Average search time -| 1 | 1K | 1 in 58 keys | < 1 milliseconds -| 2 | 1Ki| 1 in 3,364 | 50 milliseconds -| 3 | 1Kid | 1 in 195,000 | < 2 seconds -| 4 | 1Kids | 1 in 11 million | 1 minute -| 5 | 1KidsC | 1 in 656 million | 1 hour -| 6 | 1KidsCh | 1 in 38 billion | 2 days -| 7 | 1KidsCha | 1 in 2.2 trillion | 3–4 months -| 8 | 1KidsChar | 1 in 128 trillion | 13–18 years -| 9 | 1KidsChari | 1 in 7 quadrillion | 800 years -| 10 | 1KidsCharit | 1 in 400 quadrillion | 46,000 years -| 11 | 1KidsCharity | 1 in 23 quintillion | 2.5 million years -|======= - - -As you can see, Eugenia won't be creating the vanity address "1KidsCharity" anytime soon, even if she had access to several thousand computers. Each additional character increases the difficulty by a factor of 58. Patterns with more than seven characters are usually found by specialized hardware, such as custom-built desktops with multiple GPUs. These are often repurposed bitcoin mining "rigs" that are no longer profitable for bitcoin mining but can be used to find vanity addresses. Vanity searches on GPU systems are many orders of magnitude faster than on a general-purpose CPU. - -Another way to find a vanity address is to outsource the work to a pool of vanity miners, such as the pool at https://vanitypool.appspot.com[Vanity Pool]. A pool of this type is a service that allows those with GPU hardware to earn bitcoin searching for vanity addresses for others. For a small payment (0.01 bitcoin or approximately $5 at the time of this writing), Eugenia can outsource the search for a seven-character pattern vanity address and get results in a few hours instead of having to run a CPU search for months. - -Generating a vanity address is a brute-force exercise: try a random key, check the resulting address to see if it matches the desired pattern, repeat until successful. <> shows an example of a "vanity miner," a program designed to find vanity addresses, written in C++. The example uses the libbitcoin library, which we introduced in <>. - -[[vanity_miner_code]] -.Vanity address miner -==== -[source,cpp] ----- -include::code/vanity-miner.cpp[] ----- -==== - -[NOTE] -==== -<> uses +std::random_device+. Depending on the implementation it may reflect a CSRNG provided by the underlying operating system. In the case of a Unix-like operating system such as Linux, it draws from +/dev/urandom+. The random number generator used here is for demonstration purposes, and it is _not_ appropriate for generating production-quality bitcoin keys as it is not implemented with sufficient security. -==== - -The example code must be compiled using a pass:[C++] compiler and linked against the libbitcoin library (which must be first installed on that system). To run the example, run the ++vanity-miner++ executable with no parameters (see <>) and it will attempt to find a vanity address starting with "1kid." - -[[vanity_miner_run]] -.Compiling and running the vanity-miner example -==== -[source,bash] ----- -# Compile the code with g++ -$ g++ -o vanity-miner vanity-miner.cpp $(pkg-config --cflags --libs libbitcoin) -# Run the example -$ ./vanity-miner -Found vanity address! 1KiDzkG4MxmovZryZRj8tK81oQRhbZ46YT -Secret: 57cc268a05f83a23ac9d930bc8565bac4e277055f4794cbd1a39e5e71c038f3f -# Run it again for a different result -$ ./vanity-miner -Found vanity address! 1Kidxr3wsmMzzouwXibKfwTYs5Pau8TUFn -Secret: 7f65bbbbe6d8caae74a0c6a0d2d7b5c6663d71b60337299a1a2cf34c04b2a623 -# Use "time" to see how long it takes to find a result -$ time ./vanity-miner -Found vanity address! 1KidPWhKgGRQWD5PP5TAnGfDyfWp5yceXM -Secret: 2a802e7a53d8aa237cd059377b616d2bfcfa4b0140bc85fa008f2d3d4b225349 - -real 0m8.868s -user 0m8.828s -sys 0m0.035s ----- -==== - -The example code will take a few seconds to find a match for the three-character pattern "kid," as we can see when we use the +time+ Unix command to measure the execution time. Change the +search+ pattern in the source code and see how much longer it takes for four- or five-character patterns! - -===== Vanity address security - -((("security", "vanity addresses")))Vanity addresses can be used to enhance _and_ to defeat security measures; they are truly a double-edged sword. Used to improve security, a distinctive address makes it harder for adversaries to substitute their own address and fool your customers into paying them instead of you. Unfortunately, vanity addresses also make it possible for anyone to create an address that _resembles_ any random address, or even another vanity address, thereby fooling your customers. - -Eugenia could advertise a randomly generated address (e.g., +1J7mdg5rbQyUHENYdx39WVWK7fsLpEoXZy+) to which people can send their donations. Or, she could generate a vanity address that starts with 1Kids, to make it more distinctive. - -In both cases, one of the risks of using a single fixed address (rather than a separate dynamic address per donor) is that a thief might be able to infiltrate your website and replace it with his own address, thereby diverting donations to himself. If you have advertised your donation address in a number of different places, your users may visually inspect the address before making a payment to ensure it is the same one they saw on your website, on your email, and on your flyer. In the case of a random address like +1J7mdg5rbQyUHENYdx39WVWK7fsLpEoXZy+, the average user will perhaps inspect the first few characters "1J7mdg" and be satisfied that the address matches. Using a vanity address generator, someone with the intent to steal by substituting a similar-looking address can quickly generate addresses that match the first few characters, as shown in <>. - -[[table_4-13]] -.Generating vanity addresses to match a random address -|======= -| *Original Random Address* | 1J7mdg5rbQyUHENYdx39WVWK7fsLpEoXZy -| *Vanity (4-character match)* | 1J7md1QqU4LpctBetHS2ZoyLV5d6dShhEy -| *Vanity (5-character match)* | 1J7mdgYqyNd4ya3UEcq31Q7sqRMXw2XZ6n -| *Vanity (6-character match)* | 1J7mdg5WxGENmwyJP9xuGhG5KRzu99BBCX -|======= - -So does a vanity address increase security? If Eugenia generates the vanity address +1Kids33q44erFfpeXrmDSz7zEqG2FesZEN+, users are likely to look at the vanity pattern word _and a few characters beyond_, for example noticing the "1Kids33" part of the address. That would force an attacker to generate a vanity address matching at least six characters (two more), expending an effort that is 3,364 times (58 × 58) higher than the effort Eugenia expended for her 4-character vanity. Essentially, the effort Eugenia expends (or pays a vanity pool for) "pushes" the attacker into having to produce a longer pattern vanity. If Eugenia pays a pool to generate an 8-character vanity address, the attacker would be pushed into the realm of 10 characters, which is infeasible on a personal computer and expensive even with a custom vanity-mining rig or vanity pool. What is affordable for Eugenia becomes unaffordable for the attacker, especially if the potential reward of fraud is not high enough to cover the cost of the vanity address generation.((("", startref="Avanity04")))((("", startref="vanity04")))((("", startref="eugeniafour"))) - -[[paper_wallets]] -==== Paper Wallets - -((("keys and addresses", "advanced forms", "paper wallets")))((("paper wallets", id="paperw04")))((("wallets", "types of", "paper wallets", id="Wpaper04")))Paper wallets are bitcoin private keys printed on paper. Often the paper wallet also includes the corresponding Bitcoin address for convenience, but this is not necessary because it can be derived from the private key. - -[WARNING] -==== -Paper wallets are an OBSOLETE technology and are dangerous for most users. There are many subtle pitfalls involved in generating them, not least of which the possibility that the generating code is compromised with a "back door". Hundreds of bitcoin have been stolen this way. Paper wallets are shown here for informational purposes only and should not be used for storing bitcoin. Use a BIP-39 mnemonic phrase to backup your keys. Use a hardware wallet to store keys and sign transactions. DO NOT USE PAPER WALLETS. -==== - -Paper wallets come in many shapes, sizes, and designs, but at a very basic level are just a key and an address printed on paper. <> shows the simplest form of a paper wallet. - -[[table_4-14]] -.Simplest form of a paper wallet—a printout of the Bitcoin address and private key -[options="header"] -|======================= -|Public address|Private key (WIF) -|1424C2F4bC9JidNjjTUZCbUxv6Sa1Mt62x|5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn -|======================= - -Paper wallets come in many designs and sizes, with many different features. <> shows a sample paper wallet. - -[[paper_wallet_simple]] -.An example of a simple paper wallet -image::images/mbc2_0408.png[] - - -Some are intended to be given as gifts and have seasonal themes, such as Christmas and New Year's themes. Others are designed for storage in a bank vault or safe with the private key hidden in some way, either with opaque scratch-off stickers, or folded and sealed with tamper-proof adhesive foil. - -Other designs feature additional copies of the key and address, in the form of detachable stubs similar to ticket stubs, allowing you to store multiple copies to protect against fire, flood, or other natural disasters.((("", startref="KAadvanced04")))((("", startref="Wpaper04")))((("", startref="paperw04"))) - -[[paper_wallet_spw]] -.An example of a paper wallet with additional copies of the keys on a backup "stub" -image::images/mbc2_0412.png[] diff --git a/ch04_keys.adoc b/ch04_keys.adoc new file mode 100644 index 00000000..be604b31 --- /dev/null +++ b/ch04_keys.adoc @@ -0,0 +1,1889 @@ +[[ch04_keys_addresses]] +== Keys and Addresses + +Alice wants to pay Bob, but the thousands of Bitcoin full nodes who +will verify her transaction don't know who Alice or Bob are--and we want +to keep it that way to protect their privacy. Alice needs to +communicate that Bob should receive some of her bitcoins without tying +any aspect of that transaction to Bob's real-world identity or to other +Bitcoin payments that Bob receives. The method Alice uses must ensure +that only Bob can further spend the bitcoins he receives. + +The original Bitcoin paper describes a very simple scheme for achieving +those goals, shown in <>. + +[[pay-to-pure-pubkey]] +.Transaction chain from original Bitcoin paper. +image::images/mbc3_aain01.png["Transaction chain from original Bitcoin paper"] + +A receiver like Bob +accepts bitcoins to a public key in a transaction that is signed by the +spender (like Alice). The bitcoins that Alice is spending had been +previously received to one of her public keys, and she uses the +corresponding private key to generate her signature. Full nodes can +verify that Alice's signature commits to the output of a hash function +that itself commits to Bob's public key and other transaction details. + +We'll examine public keys, private keys, signatures, and hash functions +in this chapter, and then use all of them together to describe +the addresses used by modern Bitcoin software. + +=== Public Key Cryptography + +Public ((("public key cryptography", id="pub-key")))key +cryptography was invented in the 1970s and is a mathematical foundation +for modern computer and information security. + +Since the invention of public key cryptography, several suitable +mathematical functions, such as prime number exponentiation and elliptic +curve multiplication, have been discovered. These mathematical functions +are easy to calculate in +one direction and infeasible to calculate in the opposite direction +using the computers and algorithms available today. +Based on these mathematical functions, cryptography enables the creation +of unforgeable digital signatures. Bitcoin uses +elliptic curve addition and multiplication as the basis for its cryptography. + +In Bitcoin, we can use public key cryptography to create a ((("key pairs", id="key-pair")))((("public keys", "purpose of")))((("private keys", "purpose of")))key pair that +controls access to bitcoins. The key pair consists of a private key +and a public key derived from the private key. The public key is used to +receive funds, and the private key is used to sign transactions to spend +the funds. + +There is a mathematical relationship between the public and the private +key that allows the private key to be used to generate signatures on +messages. These signatures can be validated against the public key without +revealing the private key. + +[TIP] +==== +In some wallet +implementations, the private and public keys are stored together as a +_key pair_ for convenience. However, the public key can be calculated +from the private key, so storing only the private key is also possible. +==== + +A Bitcoin wallet contains a collection of key +pairs, each consisting of a private key and a public key. The private +key (_k_) is a number, usually derived from a number picked at random. +From the private key, we +use elliptic curve multiplication, a one-way cryptographic function, to +generate a public key (_K_). + +[role="less_space pagebreak-before"] +.Why Use Asymmetric Cryptography (Public/Private Keys)? +**** +Why is ((("asymmetric cryptography", see="public key cryptography")))((("public key cryptography", "purpose in Bitcoin")))((("digital signatures")))asymmetric +cryptography used in Bitcoin? It's not used to "encrypt" (make secret) +the transactions. Rather, a useful property of asymmetric cryptography +is the ability to generate _digital signatures_. A private key can be +applied to a transaction to produce a +numerical signature. This signature can only be produced by someone with +knowledge of the private key. However, anyone with access to the public +key and the transaction can use them to _verify_ the +signature. This useful property of asymmetric cryptography makes it +possible for anyone to verify every signature on every transaction, +while ensuring that only the owners of private keys can produce valid +signatures. +**** + +[[private_keys]] +==== Private Keys + +A +private ((("private keys", "generating", id="private-key-generate")))key is simply a number, picked at random. Control +over the private key is the root of user control over all funds +associated with the corresponding Bitcoin public key. The private key is +used to create signatures that are used to spend bitcoins by proving +control of funds used in a transaction. The private key must remain +secret at all times because revealing it to third parties is equivalent +to giving them control over the bitcoins secured by that key. The private +key must also be backed up and protected from accidental loss because +if it's lost, it cannot be recovered and the funds secured by it are +forever lost too. + +[TIP] +==== +A Bitcoin private key is just a number. You can pick your private keys +randomly using just a coin, pencil, and paper: toss a coin 256 times and +you have the binary digits of a random private key you can use in a +Bitcoin wallet. The public key can then be generated from the private +key. Be careful, though, as any process that's less than completely +random can significantly reduce the security of your private key and the +bitcoins it controls. +==== + +The first and most important step in generating keys is to find a secure +source of randomness (which computer scientists ((("entropy")))call _entropy_). Creating a Bitcoin key is almost +the same as "Pick a number between 1 and 2^256^." The exact method you +use to pick that number does not matter as long as it is not predictable +or repeatable. Bitcoin software uses cryptographically secure random +number generators to produce 256 bits of entropy. + +[role="less_space pagebreak-before"] +More precisely, the private key can be any number between 0 and _n_ - +1 inclusive, where _n_ is a constant (_n_ = 1.1578 × 10^77^, slightly less +than 2^256^) defined as the order of the elliptic curve used in Bitcoin +(see <>). To create such a key, we randomly pick a +256-bit number and check that it is less than _n_. In programming terms, +this is usually achieved by feeding a larger string of random bits, +collected from a cryptographically secure source of randomness, into the +SHA256 hash algorithm, which will conveniently produce a 256-bit value +that can be interpreted as a number. +If the result is less than _n_, we have a suitable private key. +Otherwise, we simply try again with another random number. + +[WARNING] +==== +Do not write your own code to create a random +number or use a "simple" random number generator offered by your +programming language. Use a cryptographically secure pseudorandom number +generator (CSPRNG) with a seed from a source of sufficient entropy. +Study the documentation of the random number generator library you +choose to make sure it is cryptographically secure. Correct +implementation of the CSPRNG is critical to the security of the keys. +==== + +The following is a randomly generated private key (_k_) shown in +hexadecimal format (256 bits shown as 64 hexadecimal digits, each 4 +bits): + +---- +1E99423A4ED27608A15A2616A2B0E9E52CED330AC530EDCC32C8FFC6A526AEDD +---- + +[TIP] +==== +The size of Bitcoin's private key space (2^256^) is an unfathomably +large number. It is approximately 10^77^ in decimal. For comparison, the +visible universe is estimated to((("private keys", "generating", startref="private-key-generate"))) contain 10^80^ atoms. +==== + +[[elliptic_curve]] +==== Elliptic Curve Cryptography Explained + +Elliptic curve cryptography (ECC) is((("public key cryptography", "elliptic curve cryptography as", id="pub-key-ecc")))((("elliptic curve cryptography (ECC)", id="ecc"))) a type of asymmetric +or public key cryptography based on the discrete logarithm problem as +expressed by addition and multiplication on the points of an elliptic +curve. + +<> is an example of an elliptic curve, similar to that used +by Bitcoin. + +[[ecc-curve]] +[role="width-50"] +.An elliptic curve. +image::images/mbc3_0402.png["ecc-curve"] + +Bitcoin uses a specific elliptic curve and set of mathematical +constants, as defined in a standard called +secp256k1+, established by +the National Institute of Standards and Technology (NIST). The ++secp256k1+ curve is defined by the following function, which produces +an elliptic curve: + +[latexmath] +++++ +\begin{equation} +{y^2 = (x^3 + 7)}~\text{over}~(\mathbb{F}_p) +\end{equation} +++++ + +or + +[latexmath] +++++ +\begin{equation} +{y^2 \mod p = (x^3 + 7) \mod p} +\end{equation} +++++ + +The _mod p_ (modulo prime number _p_) indicates that this curve is over a +finite field of prime order _p_, also written as latexmath:[\( +\mathbb{F}_p\)], where _p_ = 2^256^ – 2^32^ – 2^9^ – 2^8^ – 2^7^ – 2^6^ – +2^4^ – 1, a very large prime number. + +Because this curve is defined over a finite field of prime order instead +of over the real numbers, it looks like a pattern of dots scattered in +two dimensions, which makes it difficult to visualize. However, the math +is identical to that of an elliptic curve over real numbers. As an +example, <> shows the same elliptic curve over a much +smaller finite field of prime order 17, showing a pattern of dots on a +grid. The +secp256k1+ Bitcoin elliptic curve can be thought of as a much +more complex pattern of dots on a unfathomably large grid. + +[[ecc-over-F17-math]] +.Elliptic curve cryptography: visualizing an elliptic curve over F(p), with p=17. +image::images/mbc3_0403.png["ecc-over-F17-math"] + +So, for example, the following is a point P with coordinates (x, y) that +is a point on the +secp256k1+ curve: + +[source, python] +---- +P = +(55066263022277343669578718895168534326250603453777594175500187360389116729240, +32670510020758816978083085130507043184471273380659243275938904335757337482424) +---- + +<> shows how you can check this yourself using Python. + +[[example_4_1]] +.Using Python to confirm that this point is on the elliptic curve +==== +[source, pycon] +---- +Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +> p = 115792089237316195423570985008687907853269984665640564039457584007908834671663 +> x = 55066263022277343669578718895168534326250603453777594175500187360389116729240 +> y = 32670510020758816978083085130507043184471273380659243275938904335757337482424 +> (x ** 3 + 7 - y**2) % p +0 +---- +==== + +[role="less_space pagebreak-before"] +In elliptic curve math, there is a point called the "point at infinity," +which roughly corresponds to the role of zero in addition. On computers, +it's sometimes represented by x = y = 0 (which doesn't satisfy the +elliptic curve equation, but it's an easy separate case that can be +checked). + +There is also a pass:[+] operator, called "addition," which has some +properties similar to the traditional addition of real numbers that +gradeschool children learn. Given two points P~1~ and P~2~ on the +elliptic curve, there is a third point P~3~ = P~1~ + P~2~, also on the +elliptic curve. + +Geometrically, this third point P~3~ is calculated by drawing a line +between P~1~ and P~2~. This line will intersect the elliptic curve in +exactly one additional place. Call this point P~3~' = (x, y). Then +reflect in the x-axis to get P~3~ = (x, –y). + +There are a couple of special cases that explain the need for the "point +at infinity." + +If P~1~ and P~2~ are the same point, the line "between" P~1~ and P~2~ +should extend to be the tangent on the curve at this point P~1~. This +tangent will intersect the curve in exactly one new point. You can use +techniques from calculus to determine the slope of the tangent line. +These techniques curiously work, even though we are restricting our +interest to points on the curve with two integer coordinates! + +In some cases (i.e., if P~1~ and P~2~ have the same x values but +different y values), the tangent line will be exactly vertical, in which +case P~3~ = "point at infinity." + +If P~1~ is the "point at infinity," then P~1~ + P~2~ = P~2~. Similarly, +if P~2~ is the point at infinity, then P~1~ + P~2~ = P~1~. This shows +how the point at infinity plays the role of zero. + +It turns out that pass:[+] is associative, which means that (A pass:[+] +B) pass:[+] C = A pass:[+] (B pass:[+] C). That means we can write A +pass:[+] B pass:[+] C without parentheses and without ambiguity. + +Now that we have defined addition, we can define multiplication in the +standard way that extends addition. For a point P on the elliptic curve, +if k is a whole number, then kP = P + P + P + ... + P (k times). Note +that k is sometimes confusingly called an "exponent" in ((("public key cryptography", "elliptic curve cryptography as", startref="pub-key-ecc")))((("elliptic curve cryptography (ECC)", startref="ecc")))this case. + +[[public_key_derivation]] +==== Public Keys + +The ((("public keys", "generating", id="public-key-generate")))((("elliptic curve multiplication", id="elliptic-multiply")))public key is calculated from +the private key using elliptic curve multiplication, which is +irreversible: _K_ = _k_ × _G_, where _k_ is the private key, _G_ is a +constant point called the _generator point_, and _K_ is the resulting +public key. The reverse operation, known as "finding the discrete +logarithm"—calculating _k_ if you know __K__—is as difficult as trying +all possible values of _k_ (i.e., a brute-force search). Before we +demonstrate how to generate a public key from a private key, let's look +at elliptic curve cryptography in a bit more detail. + +[TIP] +==== +Elliptic curve multiplication is a type of function that cryptographers +call a "trap door" function: it is easy to do in one direction +(multiplication) and impossible to do in the reverse direction +(division). Someone with a private key can easily create the public +key and then share it with the world knowing that no one can reverse the +function and calculate the private key from the public key. This +mathematical trick becomes the basis for unforgeable and secure digital +signatures that prove control over bitcoin funds. +==== + +Starting with a private key in the +form of a randomly generated number _k_, we multiply it by a +predetermined point on the curve called the _generator point_ _G_ to +produce another point somewhere else on the curve, which is the +corresponding public key _K_. The generator point is specified as part +of the +secp256k1+ standard and is always the same for all keys in +bitcoin: + +[latexmath] +++++ +\begin{equation} +{K = k \times G} +\end{equation} +++++ + +where _k_ is the private key, _G_ is the generator point, and _K_ is the +resulting public key, a point on the curve. Because the generator point +is always the same for all Bitcoin users, a private key _k_ multiplied +with _G_ will always result in the same public key _K_. The relationship +between _k_ and _K_ is fixed but can only be calculated in one +direction, from _k_ to _K_. That's why a Bitcoin public key (_K_) can be +shared with anyone and does not reveal the user's private key (_k_). + +[TIP] +==== +A private key can be converted into a public key, but a public key +cannot be converted back into a private key because the math only works +one way. +==== + +Implementing the elliptic curve multiplication, we take the private key +_k_ generated previously and multiply it with the generator point _G_ to +find the public key _K_: + +[source, python] +---- +K = 1E99423A4ED27608A15A2616A2B0E9E52CED330AC530EDCC32C8FFC6A526AEDD × G +---- + +Public key _K_ is defined as a point _K_ = (_x_, _y_): + +[latexmath] +++++ +\begin{equation} +K = (x, y) +\end{equation} +++++ + + +where, + +---- +x = F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A +y = 07CF33DA18BD734C600B96A72BBC4749D5141C90EC8AC328AE52DDFE2E505BDB +---- + + + +To visualize multiplication of a point with an integer, we will use the +simpler elliptic curve over real numbers—remember, the math is +the same. Our goal is to find the multiple _kG_ of the generator point +_G_, which is the same as adding _G_ to itself, _k_ times in a row. In +elliptic curves, adding a point to itself is the equivalent of drawing a +tangent line on the point and finding where it intersects the curve +again, then reflecting that point on the x-axis. + +<> shows the process for deriving _G_, _2G_, _4G_, as a +geometric operation on the curve. + +[TIP] +==== +Many Bitcoin implementations use +the https://oreil.ly/wD60m[libsecp256k1 cryptographic +library] to do the elliptic curve((("public keys", "generating", startref="public-key-generate")))((("elliptic curve multiplication", startref="elliptic-multiply"))) math. +==== + +[[ecc_illustrated]] +.Elliptic curve cryptography: visualizing the multiplication of a point G by an integer k on an elliptic curve. +image::images/mbc3_0404.png["ecc_illustrated"] + +=== Output and Input Scripts + +Although((("public key cryptography", "input/output scripts", id="pub-key-input-output")))((("input scripts", id="input-script")))((("output scripts", id="output-script")))((("scripts", "input/output", id="script-input-output"))) the illustration from the original Bitcoin paper, <>, +shows public keys (pubkeys) and signatures (sigs) being used directly, +the first version of Bitcoin instead had payments sent to a field called +_output script_ and had spends of those bitcoins authorized by a field called _input script_. +These fields allow additional operations to be performed in addition to +(or instead of) verifying that a signature corresponds to a public key. +For example, an output script can contain two public keys and require two +corresponding signatures be placed in the spending input script. + +Later, in <>, we'll learn about scripts in detail. For now, +all we need to understand is that bitcoins are received to an +output script that acts like a public key, and bitcoin spending is +authorized by an input script that acts like a ((("public key cryptography", "input/output scripts", startref="pub-key-input-output")))((("input scripts", startref="input-script")))((("output scripts", startref="output-script")))((("scripts", "input/output", startref="script-input-output")))signature. + +[[p2pk]] +=== IP Addresses: The Original Address for Bitcoin (P2PK) + +We've ((("public key cryptography", "IP address payments and", id="pub-key-ipaddress")))((("IP addresses for Bitcoin payments", id="ipaddress-payment")))((("payments", "via IP addresses", id="payment-ipaddress")))((("P2PK (pay to public key)", id="p2pk-ch4")))((("addresses", "P2PK (pay to public key)", id="address-p2pk-ch4")))established that Alice can pay Bob by assigning some of her +bitcoins to one of Bob's public keys. But how does Alice get one of +Bob's public keys? Bob could just give her a copy, but let's look again +at the public key we worked with in <>. Notice +that it's quite long. Imagine Bob trying to read that to Alice over the +phone: + +---- +x = F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A +y = 07CF33DA18BD734C600B96A72BBC4749D5141C90EC8AC328AE52DDFE2E505BDB +---- + + +Instead of direct public key entry, the earliest version of Bitcoin +software allowed a spender to enter the receiver's IP address, as shown in <>. This +feature was later removed--there are many problems +with using IP addresses--but a quick description of it will help us +better understand why certain features may have been added to the +Bitcoin protocol. + +[[bitcoin_01_send]] +.Early send screen for Bitcoin via https://oreil.ly/IDV1a[The Internet Archive]. +image::images/mbc3_0405.png["Early Bitcoin send screen"] + +If Alice entered Bob's IP address in Bitcoin 0.1, her full node would +establish a connection with his full node and receive a new public key +from Bob's wallet that his node had never previously given anyone. This +being a new public key was important to ensure that different +transactions paying Bob couldn't be connected together by someone +looking at the blockchain and noticing that all of the transactions paid +the same public key. + +Using the public key her node received from Bob's node, Alice's wallet +would construct a transaction output paying a very simple output script: + +---- + OP_CHECKSIG +---- + +Bob would later be able to spend that output with an input script consisting +entirely of his signature: + +---- + +---- + +To figure out what an output and input script are doing, you can +combine them together (input script first) and then note that each piece of +data (shown in angle brackets) is placed at the top of a list of items, +called a stack. When an operation code (opcode) is encountered, it uses +items from the stack, starting with the topmost items. Let's look at +how that works by beginning with the combined script: + +---- + OP_CHECKSIG +---- + +For this script, Bob's signature is put on the stack, then Bob's public +key is placed on top of it. The +OP_CHECKSIG+ operation consumes two +elements, starting with the public key and followed by the signature, +removing them from the stack. It verifies the signature corresponds to +the public key and also commits to (signs) the various fields in the +transaction. If the signature is correct, +OP_CHECKSIG+ replaces itself +on the stack with the value 1; if the signature was not correct, it +replaces itself with a 0. If there's a nonzero item on top of the stack at the +end of evaluation, the script passes. If all scripts in a transaction +pass, and all of the other details about the transaction are valid, then +full nodes will consider the transaction to be valid. + +In short, the preceding script uses the same public key and signature +described in the original paper but adds in the complexity of two script +fields and an opcode. That seems like extra work here, but we'll begin +to see the benefits when we look at the following section. + +This type of output is known today as _pay to public key_, or _P2PK_ for +short. It was never widely used for payments, and no widely used +program has supported IP address payments for almost((("public key cryptography", "IP address payments and", startref="pub-key-ipaddress")))((("IP addresses for Bitcoin payments", startref="ipaddress-payment")))((("payments", "via IP addresses", startref="payment-ipaddress")))((("P2PK (pay to public key)", startref="p2pk-ch4")))((("addresses", "P2PK (pay to public key)", startref="address-p2pk-ch4"))) a decade. + +[[addresses_for_p2pkh]] +=== Legacy Addresses for P2PKH + +Entering ((("public key cryptography", "hash functions and", id="pub-key-hash")))((("hash functions", "Bitcoin payments and", id="hash-payment")))((("payments", "with hash functions", secondary-sortas="hash functions", id="payment-hash")))((("P2PKH (pay to public key hash)", id="p2pkh-legacy")))((("addresses", "P2PKH (pay to public key hash)", id="address-p2pkh-legacy")))the IP address of the person you want to pay has a number of +advantages, but it also has a number of downsides. One particular +downside is that the receiver needs their wallet to be online at their +IP address, and it needs to be accessible from the outside world. For +a lot of people, that isn't an option. They turn their computers off at +night, their laptops go to sleep, they're behind firewalls, or they're +using Network Address Translation (NAT). + +This brings us back to the problem of receivers like Bob having to give +spenders like Alice a long public key. The shortest version of Bitcoin +public keys known to the developers of early Bitcoin were 65 bytes, the +equivalent of 130 characters when written in hexadecimal. However, Bitcoin +already contains several data structures much larger than 65 bytes +that need to be securely referenced in other parts of Bitcoin using the +smallest amount of data that was secure. + +Bitcoin accomplishes that with a _hash function_, a function that takes +a potentially large amount of data, scrambles it (hashes it), and outputs a +fixed amount of data. A cryptographic hash function will always produce +the same output when given the same input, and a secure function will +also make it impractical for somebody to choose a different input that +produces a previously-seen output. That makes the ((("commitments", id="commitment")))output a _commitment_ +to the input. It's a promise that, in practice, only input _x_ will +produce output _X_. + +For example, imagine I want to ask you a question and also give you my +answer in a form that you can't read immediately. Let's say the +question is, "in what year did Satoshi Nakamoto start working on +Bitcoin?" I'll give you a commitment to my answer in the form of +output from the((("SHA256 hash function"))) SHA256 hash function, the function most commonly used in +Bitcoin: + +---- +94d7a772612c8f2f2ec609d41f5bd3d04a5aa1dfe3582f04af517d396a302e4e +---- + +Later, after you tell me your guess to the answer of the question, I can +reveal my answer and prove to you that my answer, as input to the hash +function, produces exactly the same output I gave you earlier: + +---- +$ echo "2007. He said about a year and a half before Oct 2008" | sha256sum +94d7a772612c8f2f2ec609d41f5bd3d04a5aa1dfe3582f04af517d396a302e4e +---- + +Now imagine that we ask Bob the question, "what is your public key?" Bob +can use a hash function to give us a cryptographically secure commitment +to his public key. If he later reveals his key, and we verify it +produces the same commitment he previously gave us, we can be sure it +was the exact same key that was used to create that earlier commitment. + +The SHA256 hash function is considered to be very secure and produces +256 bits (32 bytes) of output, less than half the size of original +Bitcoin public keys. However, there are other slightly less secure hash +functions that produce smaller output, such as the ((("RIPEMD-160 hash function")))RIPEMD-160 hash +function whose output is 160 bits (20 bytes). For reasons Satoshi +Nakamoto never stated, the original version of Bitcoin made commitments +to public keys by first hashing the key with SHA256 and then hashing +that output with RIPEMD-160; this produced a 20-byte commitment to the +public key. + +[role="less_space pagebreak-before"] +We can look at that algorithmically. +Starting with the public key _K_, we compute the SHA256 hash and then +compute the RIPEMD-160 hash of the result, producing a 160-bit (20-byte) +number: + +[latexmath] +++++ +\begin{equation} +{A = RIPEMD160(SHA256(K))} +\end{equation} +++++ + +where _K_ is the public key and _A_ is the resulting commitment. + +Now that we understand how to make a commitment to a public key, we need +to figure out how to use it in a transaction. Consider the following +output script: + +---- +OP_DUP OP_HASH160 OP_EQUAL OP_CHECKSIG +---- + +And also the following input script: + +---- + +---- + +Together, they form the following script: + +---- + OP_DUP OP_HASH160 OP_EQUALVERIFY OP_CHECKSIG +---- + +As we did in <>, we start putting items on the stack. Bob's +signature goes on first; his public key is then placed on top of the +stack. The +OP_DUP+ operation duplicates the top item, so the top and +second-to-top item on the stack are now both Bob's public key. The ++OP_HASH160+ operation consumes (removes) the top public key and +replaces it with the result of hashing it with +RIPEMD160(SHA256(K))+, +so now the top of the stack is a hash of Bob's public key. Next, the +commitment to Bob's public key is added to the top of the stack. The ++OP_EQUALVERIFY+ operation consumes the top two items and verifies that +they are equal; that should be the case if the public key Bob provided +in the input script is the same public key used to create the commitment in +the output script that Alice paid. If +OP_EQUALVERIFY+ fails, the whole +script fails. Finally, we're left with a stack containing just Bob's +signature and his public key; the +OP_CHECKSIG+ opcode verifies they +correspond with each other and that the signature commits to the +transaction. + +Although this process of paying to a public key hash (_P2PKH_) may seem +convoluted, it allows Alice's payment to +Bob to contain only a 20 byte commitment to his public key instead of +the key itself, which would've been 65 bytes in the original version of +Bitcoin. That's a lot less data for Bob to have to communicate to +Alice. + +However, we haven't yet discussed how Bob gets those 20 bytes from his +Bitcoin wallet to Alice's wallet. There are commonly used encodings for +byte values, such as hexadecimal, but any mistake made in copying a +commitment would result in the bitcoins being sent to an unspendable +output, causing them to be lost forever. In the next section, we'll +look at compact encoding and reliable ((("public key cryptography", "hash functions and", startref="pub-key-hash")))((("hash functions", "Bitcoin payments and", startref="hash-payment")))((("payments", "with hash functions", secondary-sortas="hash functions", startref="payment-hash")))((("P2PKH (pay to public key hash)", startref="p2pkh-legacy")))((("addresses", "P2PKH (pay to public key hash)", startref="address-p2pkh-legacy")))((("commitments", startref="commitment")))checksums. + +[[base58]] +=== Base58check Encoding + +In order((("public key cryptography", "base58check encoding", id="pub-key-base58")))((("base58check encoding", id="base58-ch4")))((("encoding", "base58check", id="encode-base58"))) to represent long numbers in a compact way, +using fewer symbols, many computer systems use mixed-alphanumeric +representations with a base (or radix) higher than 10. For example, +whereas the traditional decimal system uses 10 numerals, 0 through 9, +the hexadecimal system uses 16, with the letters A through F as the six +additional symbols. A number represented in hexadecimal format is +shorter than the equivalent decimal representation. Even more compact, +base64 representation ((("base64 encoding")))uses 26 lowercase letters, 26 capital letters, 10 +numerals, and 2 more characters such as "+" and "/" to +transmit binary data over text-based media such as email. + +Base58 is a similar encoding to +base64, using upper- and lowercase letters and numbers, +but omitting some characters that are frequently mistaken for one +another and can appear identical when displayed in certain fonts. +Specifically, base58 is base64 without the 0 (number zero), O (capital +o), l (lower L), I (capital i), and the symbols "+" and +"/." Or, more simply, it is a set of lowercase and capital letters and +numbers without the four (0, O, l, I) just mentioned. <> +shows the full base58 alphabet. + +[[base58alphabet]] +.Bitcoin's base58 alphabet +==== +---- +123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz +---- +==== + +To add extra security against typos or transcription errors, base58check +includes((("checksums"))) a _checksum_ encoded in the base58 alphabet. The checksum is an +additional four bytes +added to the end of the data that is being encoded. The checksum is +derived from the hash of the encoded data and can therefore be used to +detect transcription and typing errors. When presented with +base58check code, the decoding software will calculate the checksum of +the data and compare it to the checksum included in the code. If the two +do not match, an error has been introduced and the base58check data is +invalid. This prevents a mistyped Bitcoin address from being accepted by +the wallet software as a valid destination, an error that would +otherwise result in loss of funds. + +To convert data (a number) into a base58check format, we first add a +prefix to the data, called ((("version prefixes", id="version-prefix")))the "version byte," which serves to easily +identify the type of data that is encoded. For example, the prefix zero +(0x00 in hex) indicates that the data should be used as the commitment (hash) in +a legacy P2PKH output script. A list of common version prefixes is shown +in <>. + +Next, we compute the "double-SHA" checksum, meaning we apply the SHA256 +hash-algorithm twice on the previous result (the prefix concatenated +with the data): + +---- +checksum = SHA256(SHA256(prefix||data)) +---- + +From the resulting 32-byte hash (hash-of-a-hash), we take only the first +four bytes. These four bytes serve as the error-checking code, or +checksum. The checksum is appended to the end. + +The result is composed of three items: a prefix, the data, and a +checksum. This result is encoded using the base58 alphabet described +previously. <> illustrates the base58check +encoding process. + +[[base58check_encoding]] +.Base58check encoding: a base58, versioned, and checksummed format for unambiguously encoding bitcoin data. +image::images/mbc3_0406.png["Base58checkEncoding"] + +++++ +

+In Bitcoin, other data besides public key commitments are presented to the user in +base58check encoding to make that data compact, easy to read, and easy to detect +errors. The version prefix in base58check encoding is used to create +easily distinguishable formats, which when encoded in base58 contain +specific characters at the beginning of the base58check-encoded payload. +These characters make it easy for humans to identify the type of data +that is encoded and how to use it. This is what differentiates, for +example, a base58check-encoded Bitcoin address that starts with a 1 from +a base58check-encoded private key wallet import format (WIF) that starts with a 5. Some example +version prefixes and the resulting base58 characters are shown in +#base58check_versions. +

+++++ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Base58check version prefix and encoded result examples
TypeVersion prefix (hex)Base58 result prefix

Address for pay to public key hash (P2PKH)

0x00

1

Address for pay to script hash (P2SH)

0x05

3

Testnet Address for P2PKH

0x6F

m or n

Testnet Address for P2SH

0xC4

2

Private Key WIF

0x80

5, K, or L

BIP32 Extended Public Key

0x0488B21E

xpub

+++++ + +Combining public keys, hash-based commitments, and base58check +encoding, <> illustrates the conversion of a public key +into a Bitcoin ((("public key cryptography", "base58check encoding", startref="pub-key-base58")))((("base58check encoding", startref="base58-ch4")))((("encoding", "base58check", startref="encode-base58")))((("version prefixes", startref="version-prefix")))address. + +[[pubkey_to_address]] +.Public key to Bitcoin address: conversion of a public key to a Bitcoin address. +image::images/mbc3_0407.png["pubkey_to_address"] + +[[comp_pub]] +=== Compressed Public Keys + +//https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2011-November/000778.html + + +When ((("public key cryptography", "compressed public keys", id="pub-key-compress")))((("compressed public keys", id="compress-pub-key")))((("uncompressed public keys", id="uncompress-pub-key")))Bitcoin was first authored, its developers only knew how to create +65-byte public keys. However, a later developer became aware of an +alternative encoding for public keys that used only 33 bytes and which +was backward compatible with all Bitcoin full nodes at the time, +so there was no need to change the Bitcoin protocol. Those 33-byte +public keys are known as _compressed public keys_, and the original 65-byte keys are known as _uncompressed public keys_. Using smaller public keys +results in smaller transactions, allowing more payments to be made in the same +block. + +As we saw in the section <>, a public key is a point [.keep-together]#(x, y)# on an +elliptic curve. Because the curve expresses a mathematical function, a +point on the curve represents a solution to the equation and, therefore, +if we know the _x_ coordinate, we can calculate the _y_ coordinate by +solving the equation [.keep-together]#y^2^ mod p = (x^3^ + 7) mod p.# That allows us to +store only the _x_ coordinate of the public key point, omitting the _y_ +coordinate and reducing the size of the key and the space required to +store it by 256 bits. An almost 50% reduction in size in every +transaction adds up to a lot of data saved over time! + +Here is the public key generated by the private key we created in +<>: + +---- +x = F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A +y = 07CF33DA18BD734C600B96A72BBC4749D5141C90EC8AC328AE52DDFE2E505BDB +---- + +Here's the same public key shown as a 520-bit number (130 hex digits) +with the prefix +04+ followed by +x+ and then +y+ coordinates, as +04 x +y+: + +++++ +
+K = 04F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A\
+    07CF33DA18BD734C600B96A72BBC4749D5141C90EC8AC328AE52DDFE2E505BDB
+
+++++ + +Whereas uncompressed public keys have a prefix of +04+, compressed +public keys start with either a +02+ or a +03+ prefix. Let's look at why +there are two possible prefixes: because the left side of the equation +is __y__^2^, the solution for _y_ is a square root, which can have a +positive or negative value. Visually, this means that the resulting _y_ +coordinate can be above or below the x-axis. As you can see from the +graph of the elliptic curve in <>, the curve is symmetric, +meaning it is reflected like a mirror by the x-axis. So, while we can +omit the _y_ coordinate, we have to store the _sign_ of _y_ (positive or +negative); in other words, we have to remember if it was above or +below the x-axis because each of those options represents a different +point and a different public key. When calculating the elliptic curve in +binary arithmetic on the finite field of prime order p, the _y_ +coordinate is either even or odd, which corresponds to the +positive/negative sign as explained earlier. Therefore, to distinguish +between the two possible values of _y_, we store a compressed public key +with the prefix +02+ if the _y_ is even, and +03+ if it is odd, allowing +the software to correctly deduce the _y_ coordinate from the _x_ +coordinate and uncompress the public key to the full coordinates of the +point. Public key compression is illustrated in <>. + + +[[pubkey_compression]] +.Public key compression. +image::images/mbc3_0408.png["pubkey_compression"] + + +Here's the same public key generated in <>, shown as a compressed +public key stored in 264 bits (66 hex digits) with the prefix +03+ +indicating the _y_ coordinate is odd: + +---- +K = 03F028892BAD7ED57D2FB57BF33081D5CFCF6F9ED3D3D7F159C2E2FFF579DC341A +---- + +This compressed public key corresponds to the same private key, meaning +it is generated from the same private key. However, it looks different +from the uncompressed public key. More importantly, if we convert this +compressed public key to a commitment using the HASH160 +function (+RIPEMD160(SHA256(K))+), it will produce a _different_ +commitment than the uncompressed public key, leading to a different +address. This can be confusing because it means that a single private +key can produce a public key expressed in two different formats +(compressed and uncompressed) that produce two different Bitcoin +addresses. However, the private key is identical for both Bitcoin +addresses. + + + +Compressed public keys are now the default in almost all Bitcoin +software and were required when using certain new features added +in later protocol upgrades. + +However, some software still needs to support uncompressed public keys, +such as a wallet application importing private keys from an older +wallet. When the new wallet scans the blockchain for old P2PKH outputs +and inputs, it needs to know whether to scan the 65-byte keys (and +commitments to those keys) or 33-byte keys (and their commitments). Failure +to scan for the correct type can lead to the user not being able to +spend their full balance. To resolve this issue, when private keys are +exported from a wallet, the WIF that is used to +represent them is implemented slightly differently in newer Bitcoin +wallets to indicate that these private keys have been used to produce((("public key cryptography", "compressed public keys", startref="pub-key-compress")))((("compressed public keys", startref="compress-pub-key")))((("uncompressed public keys", startref="uncompress-pub-key"))) +compressed public keys. + +[[addresses_for_p2sh]] +=== Legacy Pay to Script Hash (P2SH) + +As we've ((("public key cryptography", "hash functions and", id="pub-key-hash2")))((("hash functions", "Bitcoin payments and", id="hash-payment2")))((("payments", "with hash functions", secondary-sortas="hash functions", id="payment-hash2")))((("P2SH (pay to script hash)", id="p2sh-ch4")))((("addresses", "P2SH (pay to script hash)", id="address-p2sh-ch4")))seen in preceding sections, someone receiving bitcoins (like +Bob) can require that payments to him contain certain constraints in their +output script. Bob will need to fulfill those constraints using an +input script when he spends those bitcoins. In <>, the constraint +was simply that the input script needed to provide an appropriate +signature. In <>, an appropriate public key also needed to be +provided. + +++++ +

+For a spender (like Alice) to place the constraints Bob wants +in the output script she uses to pay him, Bob needs to communicate those +constraints to her. This is similar to the problem of Bob needing to +communicate his public key to her. Like that problem, where +public keys can be fairly large, the constraints Bob uses can also be +quite large—potentially thousands of bytes. That's not only thousands +of bytes that need to be communicated to Alice, but thousands of bytes +for which she needs to pay transaction fees every time she wants to spend money to Bob. However, the solution of using hash functions to create +small commitments to large amounts of data also applies here. +

+++++ + +The BIP16 upgrade to the Bitcoin protocol in 2012 allows an +output script to ((("redeem scripts", id="redeem-script")))commit to a _redemption script_ (_redeem script_). When +Bob spends his bitcoins, his input script needs to provide a redeem script +that matches the commitment and also any data necessary to satisfy the +redeem script (such as signatures). Let's start by imagining Bob wants +to require two signatures to spend his bitcoins, one signature from his +desktop wallet and one from a hardware signing device. He puts those +conditions into a redeem script: + +---- + OP_CHECKSIGVERIFY OP_CHECKSIG +---- + +He then creates a commitment to the redeem script using the same +HASH160 mechanism used for P2PKH commitments, +RIPEMD160(SHA256(script))+. +That commitment is placed into the output script using a special +template: + +---- +OP_HASH160 OP_EQUAL +---- + +[WARNING] +==== +When using pay to script hash (P2SH), you must use the specific P2SH template +with no extra data or conditions in the output script. If the +output script is not exactly +OP_HASH160 <20 bytes> OP_EQUAL+, the +redeem script will not be used and any bitcoins may either be unspendable +or spendable by anyone (meaning anyone can take them). +==== + +When Bob goes to spend the payment he received to the commitment for his +script, he uses an input script that includes the redeem script, with it +serialized as a single data element. He also provides the signatures +he needs to satisfy the redeem script, putting them in the order that +they will be consumed by the opcodes: + +---- + +---- + +When Bitcoin full nodes receive Bob's spend, they'll verify that the +serialized redeem script will hash to the same value as the commitment. +Then they'll replace it on the stack with its deserialized value: + +---- + OP_CHECKSIGVERIFY OP_CHECKSIG +---- + +The script is executed and, if it passes and all of the other +transaction details are correct, the transaction is valid. + +Addresses for P2SH are also created with +base58check. The version prefix is set to 5, which results in an +encoded address starting with a +3+. An example of a P2SH address is ++3F6i6kwkevjR7AsAd4te2YB2zZyASEm1HM+. + +[TIP] +==== +P2SH is not necessarily the same as a multisignature +transaction. A P2SH address _most often_ represents a multisignature +script, but it might also represent a script encoding other types of +transactions. +==== + +P2PKH and P2SH are the only two script templates used with base58check +encoding. They are now known as legacy addresses and have become less +common over time. +Legacy addresses were supplanted by the bech32 family of ((("redeem scripts", startref="redeem-script")))addresses. + +[[p2sh_collision_attacks]] +.P2SH Collision Attacks +**** +All addresses ((("collision attacks", id="collision")))based on hash functions are theoretically vulnerable to an +attacker independently finding the same input that produced the hash +function output (commitment). In the case of Bitcoin, if they find the +input the same way the original user did, they'll know the user's private +key and be able to spend that user's bitcoins. The chance of an attacker +independently generating the input for an existing commitment is +proportional to the strength of the hash algorithm. For a secure +160-bit algorithm like HASH160, the probability is 1-in-2^160^. This ((("preimage attacks")))is +a _preimage attack_. + +An attacker can also try to generate two different inputs (e.g., redeem +scripts) that produce the same commitment. For addresses created +entirely by a single party, the chance of an attacker generating a +different input for an existing commitment is also about 1-in-2^160^ for +the HASH160 algorithm. This is((("second preimage attacks"))) a _second preimage attack_. + +However, this changes when an attacker is able to influence the original input +value. For example, an attacker participates in the creation of a +multisignature script where they don't need to submit their public key until after they learn all of the other partys' public keys. +In that case, the strength of hash algorithm is reduced to its square +root. For HASH160, the probability becomes 1-in-2^80^. This is a +_collision attack_. + +// bits80=$( echo '2^80' | bc ) +// seconds_per_hour="$(( 60 * 60))" +// bitcoin-cli getmininginfo | jq "(.networkhashps / $bits80 * $seconds_per_hour)" +// 0.8899382363032076 + +To put those numbers in context, as of early 2023, all Bitcoin miners +combined execute about 2^80^ hash functions every hour. They run a +different hash function than HASH160, so their existing hardware can't +create collision attacks for it, but the existence of the Bitcoin +network proves that collision attacks against 160-bit functions like +HASH160 are ((("HASH160")))practical. Bitcoin miners have spent the equivalent of +billions of US dollars on special hardware, so creating a collision +attack wouldn't be cheap, but there are organizations that expect to +receive billions of dollars in bitcoins to addresses generated by +processes involving multiple parties, which could make the attack +profitable. + +There are well-established cryptographic protocols for preventing +collision attacks, but a simple solution that doesn't require any +special knowledge on the part of wallet developers is to simply use +a stronger hash function. Later upgrades to Bitcoin made that possible, +and newer Bitcoin addresses provide at least 128 bits of collision +resistance. To perform 2^128^ hash operations would take all current +Bitcoin miners about 32 billion years. + +Although we do not believe there is any immediate threat to anyone +creating new P2SH addresses, we recommend all new wallets use newer +types of addresses to eliminate address collision attacks((("public key cryptography", "hash functions and", startref="pub-key-hash2")))((("hash functions", "Bitcoin payments and", startref="hash-payment2")))((("payments", "with hash functions", secondary-sortas="hash functions", startref="payment-hash2")))((("P2SH (pay to script hash)", startref="p2sh-ch4")))((("addresses", "P2SH (pay to script hash)", startref="address-p2sh-ch4"))) as a concern. +**** + +=== Bech32 Addresses + +In 2017, the ((("public key cryptography", "bech32 addresses", "advantages of", id="pub-key-bech32-adv")))((("addresses", "bech32", "advantages of", id="address-bech32-adv")))((("bech32 addresses", "advantages of", id="bech32-adv")))Bitcoin protocol was upgraded. When the upgrade is used, +it prevents transaction +identifiers (txids) from being changed without the consent of a spending +user (or a quorum of signers when multiple signatures are required). +The upgrade, ((("segregated witness (segwit)", id="segwit-bech32")))called _segregated witness_ (or _segwit_ for short), also +provided additional capacity for transaction data in blocks and several +other benefits. However, users wanting direct access to segwit's +benefits had to accept payments to new output scripts. + +As mentioned in <>, one of the advantages of the P2SH output type +was that a spender (such as Alice) didn't need to know the details of +the script the receiver (such as Bob) used. The segwit upgrade was +designed to use this mechanism, allowing users to +immediately begin accessing many of the new benefits by using a P2SH +address. But for Bob to gain access to all of the benefits, he would +need Alice's wallet to pay him using a different type of script. That +would require Alice's wallet to upgrade to support the new scripts. + +At first, Bitcoin developers proposed BIP142, which would continue using +base58check with a new version byte, similar to the P2SH upgrade. But +getting all wallets to upgrade to new scripts with a new base58check +version was expected to require almost as much work as getting them to +upgrade to an entirely new address format, so several Bitcoin +contributors set out to design the best possible address format. They +identified several problems((("public key cryptography", "base58check encoding")))((("base58check encoding")))((("encoding", "base58check"))) with base58check: + +- Its mixed-case presentation made it inconvenient to read aloud or + transcribe. Try reading one of the legacy addresses in this chapter + to a friend who you have transcribe it. Notice how you have to prefix + every letter with the words "uppercase" and "lowercase." Also, note + when you review their writing that the uppercase and lowercase + versions of some letters can look similar in many people's + handwriting. + +- It can detect errors, but it can't help users correct those errors. + For example, if you accidentally transpose two characters when manually + entering an address, your wallet will almost certainly warn that a + mistake exists, but it won't help you figure out where the error is + located. It might take you several frustrating minutes to eventually + discover the mistake. + +- A mixed-case alphabet also requires extra space to encode in QR codes, + which are commonly used to share addresses and invoices + between wallets. That extra space means QR codes need to be larger at + the same resolution or they become harder to scan quickly. + +[role="less_space pagebreak-before"] +- It requires every spender wallet upgrade to support new protocol + features like P2SH and segwit. Although the upgrades themselves might + not require much code, experience shows that many wallet authors are + busy with other work and can sometimes delay upgrading for years. + This adversely affects everyone who wants to use the new features. + +The developers working on an address format for segwit found solutions +for each of these problems in a new address format called +bech32 (pronounced with a soft "ch", as in "besh thirty-two"). The +"bech" stands for BCH, the initials of the three individuals who +discovered the cyclic code in 1959 and 1960 upon which bech32 is based. +The "32" stands for the number of characters in the bech32 alphabet +(similar to the 58 in base58check): + +- Bech32 uses only numbers and a single case of letters (preferably + rendered in lowercase). Despite its alphabet being almost half the + size of the base58check alphabet, a bech32 address for a pay to witness public key hash (P2WPKH) script + is only slightly longer than a legacy address for an equivalent P2PKH + script. + +- Bech32 can both detect and help correct errors. In an address of an + expected length, it is mathematically guaranteed to detect any error + affecting four characters or less; that's more reliable than + base58check. For longer errors, it will fail to detect them less than + one time in a billion, which is roughly the same reliability as + base58check. Even better, for an address typed with just a few + errors, it can tell the user where those errors occurred, allowing them to + quickly correct minor transcription mistakes. See <> + for an example of an address entered with errors. ++ +[[bech32_typo_detection]] +.Bech32 typo detection +==== +Address: + bc1p9nh05ha8wrljf7ru236awpass:[n]4t2x0d5ctkkywmpass:[v]9sclnm4t0av2vgs4k3au7 + +Detected errors shown in bold and underlined. Generated using the +https://oreil.ly/paWIx[bech32 address decoder demo]. +==== + +- Bech32 is preferably written with only lowercase characters, but those + lowercase characters can be replaced with uppercase characters before + encoding an address in a QR code. This allows the use of a special QR + encoding mode that uses less space. Notice the difference in size and + complexity of the two QR codes for the same address in + <>. ++ +[[bech32_qrcode_uc_lc]] +.The same bech32 address QR encoded in lowercase and uppercase. +image::images/mbc3_0409.png["The same bech32 address QR encoded in lowercase and uppercase"] + +- Bech32 takes advantage of an upgrade mechanism designed as part of + segwit to make it possible for spender wallets to be able to pay + output types that aren't in use yet. The goal was to allow developers + to build a wallet today that allows spending to a bech32 address + and have that wallet remain able to spend to bech32 addresses for + users of new features added in future protocol upgrades. It was + hoped that we might never again need to go through the system-wide + upgrade cycles necessary to allow people to fully use P2SH and((("public key cryptography", "bech32 addresses", "advantages of", startref="pub-key-bech32-adv")))((("addresses", "bech32", "advantages of", startref="address-bech32-adv")))((("bech32 addresses", "advantages of", startref="bech32-adv")))((("segregated witness (segwit)", startref="segwit-bech32"))) segwit. + +==== Problems with Bech32 Addresses + +Bech32 addresses((("public key cryptography", "bech32 addresses", "problems with", id="pub-key-bech32-prob")))((("addresses", "bech32", "problems with", id="address-bech32-prob")))((("bech32 addresses", "problems with", id="bech32-prob"))) would have been a success in every area except for one +problem. The mathematical guarantees about their ability to detect +errors only apply if the length of the address you enter into a wallet +is the same length of the original address. If you add or remove any +characters during transcription, the guarantee doesn't apply and your +wallet may spend funds to a wrong address. However, even without the +guarantee, it was thought that it would be very unlikely that a user adding +or removing characters would produce a string with a valid checksum, ensuring +users' funds were safe. + +Unfortunately, the choice for one of the constants in the bech32 +algorithm just happened to make it very easy to add or remove the letter +"q" in the penultimate position of an address that ends with the letter +"p." In those cases, you can also add or remove the letter "q" multiple +times. This will be caught by the checksum some of the time, but it +will be missed far more often than the one-in-a-billion expectations for +bech32's substitution errors. For an example, see <>. + +[role="less_space pagebreak-before"] +[[bech32_length_extension_example]] +.Extending the length of bech32 address without invalidating its checksum +==== +---- +Intended bech32 address: +bc1pqqqsq9txsqp + +Incorrect addresses with a valid checksum: +bc1pqqqsq9txsqqqqp +bc1pqqqsq9txsqqqqqqp +bc1pqqqsq9txsqqqqqqqqp +bc1pqqqsq9txsqqqqqqqqqp +bc1pqqqsq9txsqqqqqqqqqqqp +---- +==== +//from segwit_addr import * +// +//for foo in range(0,1000): +// addr = encode('bc', 1, foo.to_bytes(3,'big')) +// print(foo, addr) + + + +For the initial version of segwit (version 0), this wasn't a practical +concern. Only two valid lengths were defined for v0 segwit outputs: 22 +bytes and 34 bytes. Those correspond to bech32 addresses that are 42 characters +or 62 characters long, so someone would need to add or remove the letter "q" +from the penultimate position of a bech32 address 20 times in order to +send money to an invalid address without a wallet being able to detect +it. However, it would become a problem for users in the future if +a segwit-based upgrade were ever to be ((("public key cryptography", "bech32 addresses", "problems with", startref="pub-key-bech32-prob")))((("addresses", "bech32", "problems with", startref="address-bech32-prob")))((("bech32 addresses", "problems with", startref="bech32-prob")))implemented. + +==== Bech32m + +Although((("public key cryptography", "bech32 addresses", "bech32m", id="pub-key-bech32-bech32m")))((("bech32 addresses", "bech32m", id="bech32-bech32m")))((("addresses", "bech32m", id="address-bech32m")))((("bech32m addresses", id="bech32m"))) bech32 worked well for segwit v0, developers didn't want to +unnecessarily constrain output sizes in later versions of segwit. +Without constraints, adding or removing a single "q" in a bech32 address +could result in a user accidentally sending their money to an +output that was either unspendable or spendable by anyone (allowing +those bitcoins to be taken by anyone). Developers exhaustively analyzed the bech32 +problem and found that changing a single constant in their algorithm +would eliminate the problem, ensuring that any insertion or deletion of +up to five characters will only fail to be detected less often than one +time in a billion. + +//https://gist.github.com/sipa/a9845b37c1b298a7301c33a04090b2eb + +The version of bech32 with a single different constant is known as +bech32 modified (bech32m). All of the characters in bech32 and bech32m +addresses for the same underlying data will be identical except for the +last six (the checksum). That means a wallet will need to know which +version is in use in order to validate the checksum, but both address +types contain an internal version byte that makes determining that easy. + + + +To work with both bech32 and((("encoding", "bech32m addresses", id="encode-bech32m")))((("decoding", "addresses", see="addresses"))) bech32m, we'll look at the encoding and parsing rules for +bech32m Bitcoin addresses since they encompass the ability to parse +bech32 addresses and are the current recommended address format for +Bitcoin wallets. + +Bech32m addresses start with a human readable part (HRP). There are +rules in BIP173 for creating your own HRPs, but for Bitcoin you only +need to know about the HRPs already chosen, shown in +<>. + +++++ + + + + + + + + + + + + + + + + + + +
Bech32 HRPs for Bitcoin
HRPsNetwork

bc

Bitcoin mainnet

tb

Bitcoin testnet

+++++ + +The HRP is followed by a separator, the number "1." Earlier proposals +for a protocol separator used a colon but some operating systems and +applications that allow a user to double-click a word to highlight +it for copy and pasting won't extend the highlighting to and past a +colon. A number ensured double-click highlighting would work with any +program that supports bech32m strings in general (which include other +numbers). The number "1" was chosen because bech32 strings don't +otherwise use it in order to prevent accidental transliteration between +the number "1" and the lowercase letter "l." + +The other part of a bech32m address is called the "data part." There +are three elements to this part: + +Witness version:: + A single byte that encodes as a single character + in a bech32m Bitcoin address immediately following the separator. + This letter represents the segwit version. The letter "q" is the + encoding of "0" for segwit v0, the initial version of segwit where + bech32 addresses were introduced. The letter "p" is the encoding of + "1" for segwit v1 (also called taproot) where bech32m began to be + used. There are seventeen possible versions of segwit and it's + required for Bitcoin that the first byte of a bech32m data part decode + to the number 0 through 16 (inclusive). + +Witness program:: + From 2 to 40 bytes. For segwit v0, this witness program + must be either 20 or 32 bytes; no other length is valid. For segwit + v1, the only defined length as of this writing is 32 bytes but other + lengths may be defined later. + +Checksum:: + Exactly 6 characters. This is created using a BCH code, a type of + error correction code (although for Bitcoin addresses, we'll see later + that it's essential to use the checksum only for error detection--not + correction). +//TODO + +Let's illustrate these rules by walking through an example of creating +bech32 and bech32m addresses. For all of the following examples, we'll use the +https://oreil.ly/gpTT6[bech32m reference code +for Python]. + +We'll start by generating four output scripts, one for each of the +different segwit outputs in use at the time of publication, plus one for +a future segwit version that doesn't yet have a defined meaning. The +scripts are listed in <>. + +// bc1q9d3xa5gg45q2j39m9y32xzvygcgay4rgc6aaee +// 2b626ed108ad00a944bb2922a309844611d25468 +// +// bc1qvj9r9egtd7mu2gemy28kpf4zefq4ssqzdzzycj7zjhk4arpavfhsct5a3p +// 648a32e50b6fb7c5233b228f60a6a2ca4158400268844c4bc295ed5e8c3d626f +// +// bc1p9nh05ha8wrljf7ru236awm4t2x0d5ctkkywmu9sclnm4t0av2vgs4k3au7 +// 2ceefa5fa770ff24f87c5475d76eab519eda6176b11dbe1618fcf755bfac5311 +// +// bc1sqqqqkfw08p +// O_16 OP_PUSH2 0000 + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + +
Scripts for different types of segwit outputs
Output typeExample script

P2WPKH

OP_0 2b626ed108ad00a944bb2922a309844611d25468

P2WSH

OP_0 648a32e50b6fb7c5233b228f60a6a2ca4158400268844c4bc295ed5e8c3d626f

P2TR

OP_1 2ceefa5fa770ff24f87c5475d76eab519eda6176b11dbe1618fcf755bfac5311

Future Example

OP_16 0000

+++++ + + +For the P2WPKH output, the witness program contains a commitment constructed in exactly the same +way as the commitment for a P2PKH output seen in <>. A public key is passed into a SHA256 hash +function. The resultant 32-byte digest is then passed into a RIPEMD-160 +hash function. The digest of that function (the commitment) is placed +in the witness program. + +For the pay to witness script hash (P2WSH) output, we don't use the P2SH algorithm. Instead we take +the script, pass it into a SHA256 hash function, and use the 32-byte +digest of that function in the witness program. For P2SH, the SHA256 +digest was hashed again with RIPEMD-160, but that may not be secure in +some cases; for details, see <>. A result of +using SHA256 without RIPEMD-160 is that P2WSH commitments are 32 bytes +(256 bits) instead of 20 bytes (160 bits). + +For the pay-to-taproot (P2TR) output, the witness program is a point on +the secp256k1 curve. It may be a simple public key, but in most cases +it should be a public key that commits to some additional data. We'll +learn more about that commitment in <>. + +++++ +

+For the example of a future segwit version, we simply use the highest +possible segwit version number (16) and the smallest allowed witness +program (2 bytes) with a null value.

+++++ + +Now that we know the version number and the witness program, we can +convert each of them into a bech32 address. Let's use the bech32m reference +library for Python to quickly generate those addresses, and then take a +deeper look at what's happening: + +---- +$ github="https://raw.githubusercontent.com" +$ wget $github/sipa/bech32/master/ref/python/segwit_addr.py + +$ python +>>> from segwit_addr import * +>>> from binascii import unhexlify + +>>> help(encode) +encode(hrp, witver, witprog) + Encode a segwit address. + +>>> encode('bc', 0, unhexlify('2b626ed108ad00a944bb2922a309844611d25468')) +'bc1q9d3xa5gg45q2j39m9y32xzvygcgay4rgc6aaee' +>>> encode('bc', 0, +unhexlify('648a32e50b6fb7c5233b228f60a6a2ca4158400268844c4bc295ed5e8c3d626f')) +'bc1qvj9r9egtd7mu2gemy28kpf4zefq4ssqzdzzycj7zjhk4arpavfhsct5a3p' +>>> encode('bc', 1, +unhexlify('2ceefa5fa770ff24f87c5475d76eab519eda6176b11dbe1618fcf755bfac5311')) +'bc1p9nh05ha8wrljf7ru236awm4t2x0d5ctkkywmu9sclnm4t0av2vgs4k3au7' +>>> encode('bc', 16, unhexlify('0000')) +'bc1sqqqqkfw08p' +---- + +If we open the file __segwit_addr.py__ and look at what the code is doing, +the first thing we will notice +is the sole difference between bech32 (used for segwit v0) and bech32m +(used for later segwit versions) is the constant: + +---- +BECH32_CONSTANT = 1 +BECH32M_CONSTANT = 0x2bc830a3 +---- + +Next we notice the code that produces the checksum. In the final step of the +checksum, the appropriate constant is merged into the value using an xor +operation. That single value is the only difference between bech32 and +bech32m. + +With the checksum created, each 5-bit character in the data part +(including the witness version, witness program, and checksum) is +converted to alphanumeric characters. + +For decoding back into an output script, we work in reverse. First let's +use the reference library to decode two of our addresses: + +---- +>>> help(decode) +decode(hrp, addr) + Decode a segwit address. + +>>> _ = decode("bc", "bc1q9d3xa5gg45q2j39m9y32xzvygcgay4rgc6aaee") +>>> _[0], bytes(_[1]).hex() +(0, '2b626ed108ad00a944bb2922a309844611d25468') +>>> _ = decode("bc", + "bc1p9nh05ha8wrljf7ru236awm4t2x0d5ctkkywmu9sclnm4t0av2vgs4k3au7") +>>> _[0], bytes(_[1]).hex() +(1, '2ceefa5fa770ff24f87c5475d76eab519eda6176b11dbe1618fcf755bfac5311') +---- + +We get back both the witness version and the witness program. Those can +be inserted into the template for our output script: + +---- + +---- + +For example: + +---- +OP_0 2b626ed108ad00a944bb2922a309844611d25468 +OP_1 2ceefa5fa770ff24f87c5475d76eab519eda6176b11dbe1618fcf755bfac5311 +---- + +[WARNING] +==== +One +possible mistake here to be aware of is that a witness version of `0` is +for `OP_0`, which uses the byte 0x00--but a witness version of `1` uses +`OP_1`, which is byte 0x51. Witness versions `2` through `16` use 0x52 +through 0x60, respectively. +==== + +When implementing bech32m encoding or decoding, we very strongly +recommend that you use the test vectors provided in BIP350. We also ask +that you ensure your code passes the test vectors related to paying future segwit +versions that haven't been defined yet. This will help make your +software usable for many years to come even if you aren't able to add +support for new Bitcoin features as soon as they become ((("public key cryptography", "bech32 addresses", "bech32m", startref="pub-key-bech32-bech32m")))((("bech32 addresses", "bech32m", startref="bech32-bech32m")))((("addresses", "bech32m", startref="address-bech32m")))((("bech32m addresses", startref="bech32m")))((("encoding", "bech32m addresses", startref="encode-bech32m")))available. + +[[priv_formats]] +==== Private Key Formats + +The ((("private keys", "formats", id="private-key-format")))private key +can be represented in a number of different formats, all of which +correspond to the same 256-bit number. <> shows several common +formats used to represent private keys. Different formats are used in +different circumstances. Hexadecimal and raw binary formats are used +internally in software and rarely shown to users. The WIF is used for +import/export of keys between wallets and often used in QR code +(barcode) representations of private keys. + +.Modern Relevancy of Private Key Formats +**** +Early Bitcoin ((("wallets", "private key formats")))wallet software generated one or more independent private +keys when a new user wallet was initialized. When the initial set of +keys had all been used, the wallet might generate additional private +keys. Individual private keys could be exported or imported. Any time +new private keys were generated or imported, a new backup of the wallet +needed to be created. + +Later Bitcoin wallets began using deterministic wallets where all +private keys are generated from a single seed value. These wallets only +ever need to be backed up once for typical onchain use. However, if a +user exports a single private key from one of these wallets and an +attacker acquires that key plus some nonprivate data about the wallet, +they can potentially derive any private key in the wallet--allowing the +attacker to steal all of the wallet funds. Additionally, keys cannot be +imported into deterministic wallets. This means almost no modern +wallets support the ability to export or import an individual key. The +information in this section is mainly of interest to anyone needing +compatibility with early Bitcoin wallets. + +See <> for more information. + +**** + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + +
Private key representations (encoding formats)
TypePrefixDescription

Hex

None

64 hexadecimal digits

WIF

5

Base58check encoding: base58 with version prefix of 128 and 32-bit checksum

WIF-compressed

K or L

As above, with added suffix 0x01 before encoding

+++++ + +<> shows the private key generated in several different formats. + +++++ + + + + + + + + + + + + + + + + + + + + + + +
Example: Same key, different formats
FormatPrivate key

Hex

1e99423a4ed27608a15a2616a2b0e9e52ced330ac530edcc32c8ffc6a526aedd

WIF

5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn

WIF-compressed

KxFC1jmwwCoACiCAWZ3eXa96mBM6tb3TYzGmf6YwgdGWZgawvrtJ

+++++ + +All of these representations are different ways of showing the same +number, the same private key. They look different, but any one format +can easily be converted to any other((("private keys", "formats", startref="private-key-format"))) format. + +[[comp_priv]] +==== Compressed Private Keys + +The commonly((("private keys", "compressed", id="private-key-compress")))((("compressed private keys", id="compress-private-key"))) used term "compressed private key" is a misnomer, because when a private +key is exported as WIF-compressed, it is actually one byte _longer_ than +an "uncompressed" private key. That is because the private key has an +added one-byte suffix (shown as 01 in hex in <>), which +signifies that the private key is from a newer wallet and should only be +used to produce compressed public keys. Private keys are not themselves +compressed and cannot be compressed. The term _compressed private key_ +really means "private key from which only compressed public keys should +be derived," whereas _uncompressed private key_ really means "private +key from which only uncompressed public keys should be derived." You +should only refer to the export format as "WIF-compressed" or "WIF" and +not refer to the private key itself as "compressed" to avoid further +confusion + +<> shows the same key, encoded in WIF and WIF-compressed formats. + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + +
Example: Same key, different formats
FormatPrivate key

Hex

1E99423A4ED27608A15A2616A2B0E9E52CED330AC530EDCC32C8FFC6A526AEDD

WIF

5J3mBbAH58CpQ3Y5RNJpUKPE62SQ5tfcvU2JpbnkeyhfsYB1Jcn

Hex-compressed

1E99423A4ED27608A15A2616A2B0E9E52CED330AC530EDCC32C8FFC6A526AEDD01

WIF-compressed

KxFC1jmwwCoACiCAWZ3eXa96mBM6tb3TYzGmf6YwgdGWZgawvrtJ

+++++ + +Notice that the hex-compressed private key format has one extra byte at +the end (01 in hex). While the base58 encoding version prefix is the +same (0x80) for both WIF and WIF-compressed formats, the addition of one +byte on the end of the number causes the first character of the base58 +encoding to change from a 5 to either a _K_ or _L_. Think of this as the +base58 equivalent of the decimal encoding difference between the number +100 and the number 99. While 100 is one digit longer than 99, it also +has a prefix of 1 instead of a prefix of 9. As the length changes, it +affects the prefix. In base58, the prefix 5 changes to a _K_ or _L_ as +the length of the number increases by one byte. + +Remember, these formats are _not_ used interchangeably. In a newer +wallet that implements compressed public keys, the private keys will +only ever be exported as WIF-compressed (with a _K_ or _L_ prefix). If +the wallet is an older implementation and does not use compressed public +keys, the private keys will only ever be exported as WIF (with a 5 +prefix). The goal here is to signal to the wallet importing these +private keys whether it must search the blockchain for compressed or +uncompressed public keys and addresses. + +If a Bitcoin wallet is able to implement compressed public keys, it will +use those in all transactions. The private keys in the wallet will be +used to derive the public key points on the curve, which will be +compressed. The compressed public keys will be used to produce Bitcoin +addresses and those will be used in transactions. When exporting private +keys from a new wallet that implements compressed public keys, the WIF +is modified, with the addition of a one-byte suffix +01+ to the private +key. The resulting base58check-encoded private key is called a +"compressed WIF" and starts with the letter _K_ or _L_ instead of +starting with "5," as is the case with WIF-encoded (uncompressed) keys +from((("private keys", "compressed", startref="private-key-compress")))((("compressed private keys", startref="compress-private-key"))) older wallets. + +=== Advanced Keys and Addresses + +In the +following sections we will look at advanced forms of keys and addresses, +such as vanity addresses and paper wallets. + +==== Vanity Addresses + +Vanity((("public key cryptography", "vanity addresses", id="pub-key-vanity")))((("vanity addresses", id="vanity-addr")))((("addresses", "vanity", id="address-vanity"))) addresses are valid Bitcoin +addresses that contain human-readable messages. For example, ++1LoveBPzzD72PUXLzCkYAtGFYmK5vYNR33+ is a valid address that contains +the letters forming the word "Love" as the first four base58 letters. +Vanity addresses require generating and testing billions of candidate +private keys until a Bitcoin address with the desired pattern is found. +Although there are some optimizations in the vanity generation +algorithm, the process essentially involves picking a private key at +random, deriving the public key, deriving the Bitcoin address, and +checking to see if it matches the desired vanity pattern, repeating +billions of times until a match is found. + +Once a vanity address matching the desired pattern is found, the private +key from which it was derived can be used by the owner to spend bitcoins +in exactly the same way as any other address. Vanity addresses are no +less or more secure than any other address. They depend on the same +elliptic curve cryptography (ECC) and secure hash algorithm (SHA) as any other address. You can +no more easily find the private key of an address starting with a vanity +pattern than you can any other address. + +Eugenia is a children's +charity director operating in the Philippines. Let's say that Eugenia is +organizing a fundraising drive and wants to use a vanity Bitcoin +address to publicize the fundraising. Eugenia will create a vanity +address that starts with "1Kids" to promote the children's charity +fundraiser. Let's see how this vanity address will be created and what +it means for the security of Eugenia's charity. + +===== Generating vanity addresses + +It's important to realize that a Bitcoin address is simply a number +represented by symbols in the base58 alphabet. The search for a pattern +like "1Kids" can be seen as searching for an address in the range from ++1Kids11111111111111111111111111111+ to ++1Kidszzzzzzzzzzzzzzzzzzzzzzzzzzzzz+. There are approximately 58^29^ +(approximately 1.4 × 10^51^) addresses in that range, all starting with +"1Kids." <> shows the range of addresses that have the +prefix 1Kids. + +++++ + + + + + + + + + + + + + + + + + + + + + +
The range of vanity addresses starting with “1Kids”

From

1Kids11111111111111111111111111111

+

1Kids11111111111111111111111111112

+

1Kids11111111111111111111111111113

+

…​

To

1Kidszzzzzzzzzzzzzzzzzzzzzzzzzzzzz

+++++ + +Let's look at the pattern "1Kids" as a number and see how frequently we +might find this pattern in a Bitcoin address (see <>). An +average desktop computer PC, without any specialized hardware, can +search approximately 100,000 keys per second. + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
The frequency of a vanity pattern (1KidsCharity) and average search time on a desktop PC
LengthPatternFrequencyAverage search time

1

1K

1 in 58 keys

< 1 milliseconds

2

1Ki

1 in 3,364

50 milliseconds

3

1Kid

1 in 195,000

< 2 seconds

4

1Kids

1 in 11 million

1 minute

5

1KidsC

1 in 656 million

1 hour

6

1KidsCh

1 in 38 billion

2 days

7

1KidsCha

1 in 2.2 trillion

3–4 months

8

1KidsChar

1 in 128 trillion

13–18 years

9

1KidsChari

1 in 7 quadrillion

800 years

10

1KidsCharit

1 in 400 quadrillion

46,000 years

11

1KidsCharity

1 in 23 quintillion

2.5 million years

+++++ + +As you can see, Eugenia won't be creating the vanity address +"1KidsCharity" anytime soon, even if she had access to several thousand +computers. Each additional character increases the difficulty by a +factor of 58. Patterns with more than seven characters are usually found +by specialized hardware, such as custom-built desktops with multiple +graphics processing units (GPUs). +Vanity searches on GPU systems are many orders of magnitude +faster than on a general-purpose CPU. + +Another((("vanity pools"))) way to find a vanity address is to outsource the work to a pool +of vanity miners. A https://oreil.ly/99K81[vanity pool] is a service that +allows those with fast hardware to earn bitcoin searching for vanity +addresses for others. For a fee, Eugenia can outsource the search for a +seven-character pattern vanity address and get results in a few hours +instead of having to run a CPU search for months. + +Generating a vanity address is a brute-force exercise: try a random key, +check the resulting address to see if it matches the desired pattern, +repeat until successful. + +===== Vanity address security and privacy + +Vanity addresses((("privacy", "vanity addresses", id="privacy-vanity"))) were popular in the +early years of Bitcoin but have almost entirely disappeared from use as +of 2023. There are two likely causes for this trend: + +Deterministic wallets:: As we saw in <>, it's possible to +back up every key in most modern wallets by simply writing down a few +words or characters. This is achieved by deriving every key in the +wallet from those words or characters using a deterministic algorithm. +It's not possible to use vanity addresses with a deterministic wallet +unless the user backs up additional data for every vanity address they +create. More practically, most wallets using deterministic key +generation simply don't allow importing a private key or key tweak from +a vanity generator. + +Avoiding address reuse:: Using a vanity address to receive multiple +payments to the same address creates a link between all of those +payments. This might be acceptable to Eugenia if her nonprofit needs +to report its income and expenditures to a tax authority anyway. +However, it also reduces the privacy of people who either pay Eugenia or +receive payments from her. For example, Alice may want to donate +anonymously and Bob may not want his other customers to know that he +gives discount pricing to Eugenia. + +// https://github.com/MakisChristou/vanitybech + +We don't expect to see many vanity addresses in +the future unless the preceding problems are((("addresses", "vanity", startref="address-vanity")))((("vanity addresses", startref="vanity-addr")))((("public key cryptography", "vanity addresses", startref="pub-key-vanity"))) solved. + +[[paper_wallets]] +==== Paper Wallets + +Paper wallets((("public key cryptography", "paper wallets", id="pub-key-paper")))((("paper wallets", id="paper-wallet")))((("wallets", "paper", id="wallet-paper"))) are private keys printed on paper. +Often the paper wallet also includes the corresponding Bitcoin address +for convenience, but this is not necessary because it can be derived +from the private key. + +[WARNING] +==== +Paper wallets are an OBSOLETE technology and are dangerous for most +users. There are many subtle pitfalls involved in generating them, not least of which is the possibility that the generating code is compromised +with a "back door." Many bitcoins have been stolen this way. Paper +wallets are shown here for informational purposes only and should not be +used for storing bitcoin. Use a recovery code to back up your +keys, possibly with a hardware signing device to store keys and sign transactions. DO NOT +USE PAPER [.keep-together]#WALLETS.# +==== + + +Paper wallets come in many designs and sizes, with many different +features. <> shows a sample paper wallet. + +[[paper_wallet_simple]] +.An example of a simple paper wallet. +image::images/mbc3_0410.png[] + +Some are intended to be given as gifts and have seasonal themes, such as +Christmas and New Year's. Others are designed for storage in a +bank vault or safe with the private key hidden in some way, either with +opaque scratch-off stickers or folded and sealed with tamper-proof +adhesive foil. Other designs feature additional copies of the key and +address, in the form of detachable stubs similar to ticket stubs, +allowing you to store multiple copies to protect against fire, flood, or +other natural disasters. + +From the original public-key focused design of Bitcoin to modern addresses +and scripts like bech32m and pay to taproot--and even addresses for +future Bitcoin upgrades--you've learned how the Bitcoin protocol allows +spenders to identify the wallets that should receive their payments. +But when it's actually your wallet receiving the payments, you're going +to want the assurance that you'll still have access to that money even +if something happens to your wallet data. In the next chapter, we'll +look at how Bitcoin wallets are designed to protect their funds ((("public key cryptography", "paper wallets", startref="pub-key-paper")))((("paper wallets", startref="paper-wallet")))((("wallets", "paper", startref="wallet-paper")))from a +variety of threats. diff --git a/ch05.asciidoc b/ch05.asciidoc deleted file mode 100644 index ddfaa029..00000000 --- a/ch05.asciidoc +++ /dev/null @@ -1,500 +0,0 @@ -[[ch05_wallets]] -== Wallets - -((("wallets", "defined")))The word "wallet" is used to describe a few different things in bitcoin. - -At a high level, a wallet is an application that serves as the primary user interface. The wallet controls access to a user's money, managing keys and addresses, tracking the balance, and creating and signing transactions. - -More narrowly, from a programmer's perspective, the word "wallet" refers to the data structure used to store and manage a user's keys. - -In this chapter we will look at the second meaning, where wallets are containers for private keys, usually implemented as structured files or simple databases. - -=== Wallet Technology Overview - -In this section we summarize the various technologies used to construct user-friendly, secure, and flexible bitcoin wallets. - -((("wallets", "contents of")))A common misconception about bitcoin is that bitcoin wallets contain bitcoin. In fact, the wallet contains only keys. The "coins" are recorded in the blockchain on the Bitcoin network. Users control the coins on the network by signing transactions with the keys in their wallets. ((("keychains")))In a sense, a bitcoin wallet is a _keychain_. - -[TIP] -==== -Bitcoin wallets contain keys, not coins. Each user has a wallet containing keys. Wallets are really keychains containing pairs of private/public keys (see <>). Users sign transactions with the keys, thereby proving they own the transaction outputs (their coins). The coins are stored on the blockchain in the form of transaction outputs (often noted as vout or txout). -==== - -((("wallets", "types of", "primary distinctions")))There are two primary types of wallets, distinguished by whether the keys they contain are related to each other or not. - -((("JBOK wallets", seealso="wallets")))((("wallets", "types of", "JBOK wallets")))((("nondeterministic wallets", seealso="wallets")))The first type is a _nondeterministic wallet_, where each key is independently generated from a random number. The keys are not related to each other. This type of wallet is also known as a JBOK wallet from the phrase "Just a Bunch Of Keys." - -((("deterministic wallets", seealso="wallets")))The second type of wallet is a _deterministic wallet_, where all the keys are derived from a single master key, known as the _seed_. All the keys in this type of wallet are related to each other and can be generated again if one has the original seed. ((("key derivation methods")))There are a number of different _key derivation_ methods used in deterministic wallets. ((("hierarchical deterministic (HD) wallets", seealso="wallets")))The most commonly used derivation method uses a tree-like structure and is known as a _hierarchical deterministic_ or _HD_ wallet. - -((("mnemonic code words")))Deterministic wallets are initialized from a random sequence (entropy). To make these easier to use, random sequences are encoded as English words, also known as _mnemonic code words_. - -The next few sections introduce each of these technologies at a high level. - -[[random_wallet]] -==== Nondeterministic (Random) Wallets - -((("wallets", "types of", "nondeterministic (random) wallets")))In the first bitcoin wallet (now called Bitcoin Core), wallets were collections of randomly generated private keys. For example, the original Bitcoin Core client pregenerates 100 random private keys when first started and generates more keys as needed, using each key only once. Such wallets are being replaced with deterministic wallets because they are cumbersome to manage, back up, and import. The disadvantage of random keys is that if you generate many of them you must keep copies of all of them, meaning that the wallet must be backed up frequently. Each key must be backed up, or the funds it controls are irrevocably lost if the wallet becomes inaccessible. This conflicts directly with the principle of avoiding address reuse, by using each Bitcoin address for only one transaction. Address reuse reduces privacy by associating multiple transactions and addresses with each other. A Type-0 nondeterministic wallet is a poor choice of wallet, especially if you want to avoid address reuse because it means managing many keys, which creates the need for frequent backups. Although the Bitcoin Core client includes a Type-0 wallet, using this wallet is discouraged by developers of Bitcoin Core. <> shows a nondeterministic wallet, containing a loose collection of random keys. - -[TIP] -==== -The use of nondeterministic wallets is discouraged for anything other than simple tests. They are simply too cumbersome to back up and use. Instead, use an industry-standard–based _HD wallet_ with a _mnemonic_ random sequence (entropy, or "initial seed") for backup. -==== - -[[Type0_wallet]] -[role="smallersixty"] -.Type-0 nondeterministic (random) wallet: a collection of randomly generated keys -image::images/mbc2_0501.png["Non-Deterministic Wallet"] - -==== Deterministic (Seeded) Wallets - -((("wallets", "types of", "deterministic (seeded) wallets")))Deterministic, or "seeded," wallets are wallets that contain private keys that are all derived from a common seed, through the use of a one-way hash function. The seed is a randomly generated number that is combined with other data, such as an index number or "chain code" (see <>) to derive the private keys. In a deterministic wallet, the seed is sufficient to recover all the derived keys, and therefore a single backup at creation time is sufficient. The seed is also sufficient for a wallet export or import, allowing for easy migration of all the user's keys between different wallet implementations. <> shows a logical diagram of a deterministic wallet. - -[[Type1_wallet]] -[role="smallersixty"] -.Type-1 deterministic (seeded) wallet: a deterministic sequence of keys derived from a seed -image::images/mbc2_0502.png["Deterministic Wallet"] - -[[hd_wallets]] -==== HD Wallets (BIP-32/BIP-44) - -((("wallets", "types of", "hierarchical deterministic (HD) wallets")))((("hierarchical deterministic (HD) wallets")))((("bitcoin improvement proposals", "Hierarchical Deterministic Wallets (BIP-32/BIP-44)")))Deterministic wallets were developed to make it easy to derive many keys from a single "seed". The most advanced form of deterministic wallets is the HD wallet defined by the BIP-32 standard. HD wallets contain keys derived in a tree structure, such that a parent key can derive a sequence of children keys, each of which can derive a sequence of grandchildren keys, and so on, to an infinite depth. This tree structure is illustrated in <>. - -[[Type2_wallet]] -.Type-2 HD wallet: a tree of keys generated from a single seed -image::images/mbc2_0503.png["HD wallet"] - -HD wallets offer two major advantages over random (nondeterministic) keys. First, the tree structure can be used to express additional organizational meaning, such as when a specific branch of subkeys is used to receive incoming payments and a different branch is used to receive change from outgoing payments. Branches of keys can also be used in corporate settings, allocating different branches to departments, subsidiaries, specific functions, or accounting categories. - -The second advantage of HD wallets is that users can create a sequence of public keys without having access to the corresponding private keys. This allows HD wallets to be used on an insecure server or in a receive-only capacity, issuing a different public key for each transaction. The public keys do not need to be preloaded or derived in advance, yet the server doesn't have the private keys that can spend the funds. - -==== Seeds and Mnemonic Codes (BIP-39) - -((("wallets", "technology of", "seeds and mnemonic codes")))((("mnemonic code words")))((("bitcoin improvement proposals", "Mnemonic Code Words (BIP-39)")))HD wallets are a very powerful mechanism for managing many keys and addresses. They are even more useful if they are combined with a standardized way of creating seeds from a sequence of English words that are easy to transcribe, export, and import across wallets. This is known as a _mnemonic_ and the standard is defined by BIP-39. Today, most bitcoin wallets (as well as wallets for other cryptocurrencies) use this standard and can import and export seeds for backup and recovery using interoperable mnemonics. - -Let's look at this from a practical perspective. Which of the following seeds is easier to transcribe, record on paper, read without error, export, and import into another wallet? - -.A seed for a deterministic wallet, in hex ----- -0C1E24E5917779D297E14D45F14E1A1A ----- - -.A seed for a deterministic wallet, from a 12-word mnemonic ----- -army van defense carry jealous true -garbage claim echo media make crunch ----- - -==== Wallet Best Practices - -((("wallets", "best practices for")))((("bitcoin improvement proposals", "Multipurpose HD Wallet Structure (BIP-43)")))As bitcoin wallet technology has matured, certain common industry standards have emerged that make bitcoin wallets broadly interoperable, easy to use, secure, and flexible. These common standards are: - -* Mnemonic code words, based on BIP-39 -* HD wallets, based on BIP-32 -* Multipurpose HD wallet structure, based on BIP-43 -* Multicurrency and multiaccount wallets, based on BIP-44 - -These standards may change or may become obsolete by future developments, but for now they form a set of interlocking technologies that have become the de facto wallet standard for bitcoin. - -The standards have been adopted by a broad range of software and hardware bitcoin wallets, making all these wallets interoperable. A user can export a mnemonic generated on one of these wallets and import it in another wallet, recovering all transactions, keys, and addresses. - -((("hardware wallets")))((("hardware wallets", see="also wallets")))Some example of software wallets supporting these standards include (listed alphabetically) Bluewallet, Breadwallet, Copay, and Multibit HD. Examples of hardware wallets supporting these standards include (listed alphabetically) KeepKey, Ledger, and Trezor. - -The following sections examine each of these technologies in detail. - -[TIP] -==== -If you are implementing a bitcoin wallet, it should be built as a HD wallet, with a seed derived from, and encoded as, a mnemonic code for backup, following the BIP-32, BIP-39, BIP-43, and BIP-44 standards, as described in the following sections. -==== - -==== Using a Bitcoin Wallet - -((("wallets", "using bitcoin wallets")))In <> we introduced Gabriel, ((("use cases", "web store", id="gabrielfive")))an enterprising young teenager in Rio de Janeiro, who is running a simple web store that sells bitcoin-branded t-shirts, coffee mugs, and stickers. - -((("wallets", "types of", "hardware wallets")))Gabriel uses a Trezor bitcoin hardware wallet (<>) to securely manage his bitcoin. The Trezor is a simple USB device with two buttons that stores keys (in the form of an HD wallet) and signs transactions. Trezor wallets implement all the industry standards discussed in this chapter, so Gabriel is not reliant on any proprietary technology or single vendor solution. - -[[a_trezor_device]] -.A Trezor device: a bitcoin HD wallet in hardware -image::images/mbc2_0504.png[alt] - -When Gabriel used the Trezor for the first time, the device generated a random sequence (entropy), the associated mnemonic and derived a seed from a built-in hardware random number generator. During this initialization phase, the wallet displayed a numbered sequence of words, one by one, on the screen (see <>). - -[[trezor_mnemonic_display]] -.Trezor displaying one of the mnemonic words -image::images/mbc2_0505.png["Trezor wallet display of mnemonic word"] - -By writing down this mnemonic, Gabriel created a backup (see <>) that can be used for recovery in the case of loss or damage to the Trezor device. This mnemonic can be used for recovery in a new Trezor or in any one of the many compatible software or hardware wallets. Note that the sequence of words is important, so mnemonic paper backups have numbered spaces for each word. Gabriel had to carefully record each word in the numbered space to preserve the correct sequence. - -[[mnemonic_paper_backup]] -.Gabriel's paper backup of the mnemonic -[cols="<1,^50,<1,^50", width="80%"] -|=== -|*1.*| _army_ |*7.*| _garbage_ -|*2.*| _van_ |*8.*| _claim_ -|*3.*| _defense_ |*9.*| _echo_ -|*4.*| _carry_ |*10.*| _media_ -|*5.*| _jealous_ |*11.*| _make_ -|*6.*| _true_ |*12.*| _crunch_ -|=== - -[NOTE] -==== -A 12-word mnemonic is shown in <>, for simplicity. In fact, most hardware wallets generate a more secure 24-word mnemonic. The mnemonic is used in exactly the same way, regardless of length. -==== - -For the first implementation of his web store, Gabriel uses a single Bitcoin address, generated on his Trezor device. This single address is used by all customers for all orders. As we will see, this approach has some drawbacks and can be improved upon with an HD wallet.((("", startref="gabrielfive"))) - -=== Wallet Technology Details - -Let's now examine each of the important industry standards that are used by many bitcoin wallets in detail. - -[[mnemonic_code_words]] -==== Mnemonic Code Words (BIP-39) - -((("wallets", "technology of", "mnemonic code words")))((("mnemonic code words", id="mnemonic05")))((("bitcoin improvement proposals", "Mnemonic Code Words (BIP-39)", id="BIP3905")))Mnemonic code words are word sequences that represent (encode) a random number used as a seed to derive a deterministic wallet. The sequence of words is sufficient to re-create the seed and from there re-create the wallet and all the derived keys. A wallet application that implements deterministic wallets with mnemonic words will show the user a sequence of 12 to 24 words when first creating a wallet. That sequence of words is the wallet backup and can be used to recover and re-create all the keys in the same or any compatible wallet application. Mnemonic words make it easier for users to back up wallets because they are easy to read and correctly transcribe, as compared to a random sequence of numbers. - -[TIP] -==== -((("brainwallets")))Mnemonic words are often confused with "brainwallets." They are not the same. The primary difference is that a brainwallet consists of words chosen by the user, whereas mnemonic words are created randomly by the wallet and presented to the user. This important difference makes mnemonic words much more secure, because humans are very poor sources of randomness. -==== - -Mnemonic codes are defined in BIP-39 (see <>). Note that BIP-39 is one implementation of a mnemonic code standard. ((("Electrum wallet", seealso="wallets")))There is a different standard, with a different set of words, used by the Electrum wallet and predating BIP-39. BIP-39 was proposed by the company behind the Trezor hardware wallet and is incompatible with Electrum's implementation. However, BIP-39 has now achieved broad industry support across dozens of interoperable implementations and should be considered the de facto industry standard. - -BIP-39 defines the creation of a mnemonic code and seed, which we describe here in nine steps. For clarity, the process is split into two parts: steps 1 through 6 are shown in <> and steps 7 through 9 are shown in <>. - -[[generating_mnemonic_words]] -===== Generating mnemonic words - -Mnemonic words are generated automatically by the wallet using the standardized process defined in BIP-39. The wallet starts from a source of entropy, adds a checksum, and then maps the entropy to a word list: - -1. Create a random sequence (entropy) of 128 to 256 bits. -2. Create a checksum of the random sequence by taking the first (entropy-length/32) bits of its SHA256 hash. -3. Add the checksum to the end of the random sequence. -4. Split the result into 11-bit length segments. -5. Map each 11-bit value to a word from the predefined dictionary of 2048 words. -6. The mnemonic code is the sequence of words. - -<> shows how entropy is used to generate mnemonic words. - -[[generating_entropy_and_encoding]] -[role="smallerseventy"] -.Generating entropy and encoding as mnemonic words -image::images/mbc2_0506.png["Generating entropy and encoding as mnemonic words"] - -<> shows the relationship between the size of the entropy data and the length of mnemonic codes in words. - -[[table_4-5]] -.Mnemonic codes: entropy and word length -[options="header"] -|======= -|Entropy (bits) | Checksum (bits) | Entropy *+* checksum (bits) | Mnemonic length (words) -| 128 | 4 | 132 | 12 -| 160 | 5 | 165 | 15 -| 192 | 6 | 198 | 18 -| 224 | 7 | 231 | 21 -| 256 | 8 | 264 | 24 -|======= - -[[mnemonic_to_seed]] -===== From mnemonic to seed - -((("key-stretching function")))((("PBKDF2 function")))The mnemonic words represent entropy with a length of 128 to 256 bits. The entropy is then used to derive a longer (512-bit) seed through the use of the key-stretching function PBKDF2. The seed produced is then used to build a deterministic wallet and derive its keys. - -((("salts")))((("passphrases")))The key-stretching function takes two parameters: the mnemonic and a _salt_. The purpose of a salt in a key-stretching function is to make it difficult to build a lookup table enabling a brute-force attack. In the BIP-39 standard, the salt has another purpose—it allows the introduction of a passphrase that serves as an additional security factor protecting the seed, as we will describe in more detail in <>. - -The process described in steps 7 through 9 continues from the process described previously in <>: - -++++ -
    -
  1. The first parameter to the PBKDF2 key-stretching function is the mnemonic produced from step 6.
    2. -
  2. The second parameter to the PBKDF2 key-stretching function is a salt. The salt is composed of the string constant "mnemonic" concatenated with an optional user-supplied passphrase string.
    3. -
  3. PBKDF2 stretches the mnemonic and salt parameters using 2048 rounds of hashing with the HMAC-SHA512 algorithm, producing a 512-bit value as its final output. That 512-bit value is the seed.
    4. -
-++++ - -<> shows how a mnemonic is used to generate a seed. - -[[fig_5_7]] -.From mnemonic to seed -image::images/mbc2_0507.png["From mnemonic to seed"] - -[TIP] -==== -The key-stretching function, with its 2048 rounds of hashing, is a very effective protection against brute-force attacks against the mnemonic or the passphrase. It makes it extremely costly (in computation) to try more than a few thousand passphrase and mnemonic combinations, while the number of possible derived seeds is vast (2^512^). -==== - -Tables pass:[#mnemonic_128_no_pass], pass:[#mnemonic_128_w_pass], and pass:[#mnemonic_256_no_pass] show some examples of mnemonic codes and the seeds they produce (either with or without a passphrase). - -[[mnemonic_128_no_pass]] -.128-bit entropy mnemonic code, no passphrase, resulting seed -[cols="h,"] -|======= -| *Entropy input (128 bits)*| +0c1e24e5917779d297e14d45f14e1a1a+ -| *Mnemonic (12 words)* | +army van defense carry jealous true garbage claim echo media make crunch+ -| *Passphrase*| (none) -| *Seed (512 bits)* | +5b56c417303faa3fcba7e57400e120a0ca83ec5a4fc9ffba757fbe63fbd77a89a1a3be4c67196f57c39+ -+a88b76373733891bfaba16ed27a813ceed498804c0570+ -|======= - -[[mnemonic_128_w_pass]] -.128-bit entropy mnemonic code, with passphrase, resulting seed -[cols="h,"] -|======= -| *Entropy input (128 bits)*| +0c1e24e5917779d297e14d45f14e1a1a+ -| *Mnemonic (12 words)* | +army van defense carry jealous true garbage claim echo media make crunch+ -| *Passphrase*| SuperDuperSecret -| *Seed (512 bits)* | +3b5df16df2157104cfdd22830162a5e170c0161653e3afe6c88defeefb0818c793dbb28ab3ab091897d0+ -+715861dc8a18358f80b79d49acf64142ae57037d1d54+ -|======= - - -[[mnemonic_256_no_pass]] -.256-bit entropy mnemonic code, no passphrase, resulting seed -[cols="h,"] -|======= -| *Entropy input (256 bits)* | +2041546864449caff939d32d574753fe684d3c947c3346713dd8423e74abcf8c+ -| *Mnemonic (24 words)* | +cake apple borrow silk endorse fitness top denial coil riot stay wolf -luggage oxygen faint major edit measure invite love trap field dilemma oblige+ -| *Passphrase*| (none) -| *Seed (512 bits)* | +3269bce2674acbd188d4f120072b13b088a0ecf87c6e4cae41657a0bb78f5315b33b3a04356e53d062e5+ -+5f1e0deaa082df8d487381379df848a6ad7e98798404+ -|======= - -[TIP] -==== -Many wallets do not allow for the creation of wallets with more than a 12 word mnemonic phrase. You will notice from the tables above that despite the unique lengths of entropy input, the seed size remains the same (512 bits). From a security perspective, the amount of entropy actually used for the production of HD wallets is roughly 128 bits, which equals 12 words. Providing more than 12 words produces additional entropy which is unnecessary, and this _unused_ entropy is not used for the derivation of the seed in the way that one might initially suspect. From a usability perspective, 12 words is also easier to write down, back up, and store. -==== - -[[mnemonic_passphrase]] -===== Optional passphrase in BIP-39 - -((("passphrases")))The BIP-39 standard allows the use of an optional passphrase in the derivation of the seed. If no passphrase is used, the mnemonic is stretched with a salt consisting of the constant string +"mnemonic"+, producing a specific 512-bit seed from any given mnemonic. If a passphrase is used, the stretching function produces a _different_ seed from that same mnemonic. In fact, given a single mnemonic, every possible passphrase leads to a different seed. Essentially, there is no "wrong" passphrase. All passphrases are valid and they all lead to different seeds, forming a vast set of possible uninitialized wallets. The set of possible wallets is so large (2^512^) that there is no practical possibility of brute-forcing or accidentally guessing one that is in use. - -[TIP] -==== -There are no "wrong" passphrases in BIP-39. Every passphrase leads to some wallet, which unless previously used will be empty. -==== - -The optional passphrase creates two important features: - -* A second factor (something memorized) that makes a mnemonic useless on its own, protecting mnemonic backups from compromise by a thief. - -* A form of plausible deniability or "duress wallet," where a chosen passphrase leads to a wallet with a small amount of funds used to distract an attacker from the "real" wallet that contains the majority of funds. - -However, it is important to note that the use of a passphrase also introduces the risk of loss: - -* If the wallet owner is incapacitated or dead and no one else knows the passphrase, the mnemonic code is useless and all the funds stored in the wallet are lost forever. - -* Conversely, if the owner backs up the passphrase in the same place as the mnemonic code, it defeats the purpose of a second factor. - -While passphrases are very useful, they should only be used in combination with a carefully planned process for backup and recovery, considering the possibility of surviving the owner and allowing his or her family to recover the cryptocurrency estate. - -===== Working with mnemonic codes - -BIP-39 is implemented as a library in many different programming languages: - -https://github.com/trezor/python-mnemonic[python-mnemonic]:: The reference implementation of the standard by the SatoshiLabs team that proposed BIP-39, in Python - -https://github.com/bitcoinjs/bip39[bitcoinjs/bip39]:: An implementation of BIP-39, as part of the popular bitcoinJS framework, in JavaScript - -https://github.com/libbitcoin/libbitcoin/blob/master/src/wallet/mnemonic.cpp[libbitcoin/mnemonic]:: An implementation of BIP-39, as part of the popular Libbitcoin framework, in pass:[C++] - -==== Creating an HD Wallet from the Seed - -((("wallets", "technology of", "creating HD wallets from root seed")))((("root seeds")))((("hierarchical deterministic (HD) wallets")))HD wallets are created from a single _root seed_, which is a 128-, 256-, or 512-bit random number. Most commonly, this seed is generated from a _mnemonic_ as detailed in the previous section. - -Every key in the HD wallet is deterministically derived from this root seed, which makes it possible to re-create the entire HD wallet from that seed in any compatible HD wallet. This makes it easy to back up, restore, export, and import HD wallets containing thousands or even millions of keys by simply transferring only the mnemonic that the root seed is derived from. - -The process of creating the master keys and master chain code for an HD wallet is shown in <>. - -[[HDWalletFromSeed]] -.Creating master keys and chain code from a root seed -image::images/mbc2_0509.png["HDWalletFromRootSeed"] - -The root seed is input into the HMAC-SHA512 algorithm and the resulting hash is used to create a _master private key_ (m) and a _master chain code_ (c). - -The master private key (m) then generates a corresponding master public key (M) using the normal elliptic curve multiplication process +m * G+ that we saw in <>. - -The chain code (c) is used to introduce entropy in the function that creates child keys from parent keys, as we will see in the next section. - -===== Private child key derivation - -((("child key derivation (CKD)")))((("public and private keys", "child key derivation (CKD)")))HD wallets use a _child key derivation_ (CKD) function to derive child keys from parent keys. - -The child key derivation functions are based on a one-way hash function that combines: - -* A parent private or public key (ECDSA compressed key) -* A seed called a chain code (256 bits) -* An index number (32 bits) - -The chain code is used to introduce deterministic random data to the process, so that knowing the index and a child key is not sufficient to derive other child keys. Knowing a child key does not make it possible to find its siblings, unless you also have the chain code. The initial chain code seed (at the root of the tree) is made from the seed, while subsequent child chain codes are derived from each parent chain code. - -These three items (parent key, chain code, and index) are combined and hashed to generate children keys, as follows. - -The parent public key, chain code, and the index number are combined and hashed with the HMAC-SHA512 algorithm to produce a 512-bit hash. This 512-bit hash is split into two 256-bit halves. The right-half 256 bits of the hash output become the chain code for the child. The left-half 256 bits of the hash are added to the parent key to produce the child private key. In <>, we see this illustrated with the index set to 0 to produce the "zero" (first by index) child of the parent. - -[[CKDpriv]] -.Extending a parent private key to create a child private key -image::images/mbc2_0510.png["ChildPrivateDerivation"] - -Changing the index allows us to extend the parent and create the other children in the sequence, e.g., Child 0, Child 1, Child 2, etc. Each parent key can have 2,147,483,647 (2^31^) children (2^31^ is half of the entire 2^32^ range available because the other half is reserved for a special type of derivation we will talk about later in this chapter). - -Repeating the process one level down the tree, each child can in turn become a parent and create its own children, in an infinite number of generations. - -===== Using derived child keys - -Child private keys are indistinguishable from nondeterministic (random) keys. Because the derivation function is a one-way function, the child key cannot be used to find the parent key. The child key also cannot be used to find any siblings. If you have the n~th~ child, you cannot find its siblings, such as the n–1 child or the n+1 child, or any other children that are part of the sequence. Only the parent key and chain code can derive all the children. Without the child chain code, the child key cannot be used to derive any grandchildren either. You need both the child private key and the child chain code to start a new branch and derive grandchildren. - -So what can the child private key be used for on its own? It can be used to make a public key and a Bitcoin address. Then, it can be used to sign transactions to spend anything paid to that address. - -[TIP] -==== -A child private key, the corresponding public key, and the Bitcoin address are all indistinguishable from keys and addresses created randomly. The fact that they are part of a sequence is not visible outside of the HD wallet function that created them. Once created, they operate exactly as "normal" keys. -==== - -===== Extended keys - -((("public and private keys", "extended keys")))((("extended keys")))As we saw earlier, the key derivation function can be used to create children at any level of the tree, based on the three inputs: a key, a chain code, and the index of the desired child. The two essential ingredients are the key and chain code, and combined these are called an _extended key_. The term "extended key" could also be thought of as "extensible key" because such a key can be used to derive children. - -Extended keys are stored and represented simply as the concatenation of the 256-bit key and 256-bit chain code into a 512-bit sequence. There are two types of extended keys. An extended private key is the combination of a private key and chain code and can be used to derive child private keys (and from them, child public keys). An extended public key is a public key and chain code, which can be used to create child public keys (_public only_), as described in <>. - -Think of an extended key as the root of a branch in the tree structure of the HD wallet. With the root of the branch, you can derive the rest of the branch. The extended private key can create a complete branch, whereas the extended public key can _only_ create a branch of public keys. - -[TIP] -==== -An extended key consists of a private or public key and chain code. An extended key can create children, generating its own branch in the tree structure. Sharing an extended key gives access to the entire branch. -==== - -Extended keys are encoded using Base58Check, to easily export and import between different BIP-32–compatible wallets. The Base58Check coding for extended keys uses a special version number that results in the prefix "xprv" and "xpub" when encoded in Base58 characters to make them easily recognizable. Because the extended key is 512 or 513 bits, it is also much longer than other Base58Check-encoded strings we have seen previously. - -Here's an example of an extended _private_ key, encoded in Base58Check: - ----- -xprv9tyUQV64JT5qs3RSTJkXCWKMyUgoQp7F3hA1xzG6ZGu6u6Q9VMNjGr67Lctvy5P8oyaYAL9CAWrUE9i6GoNMKUga5biW6Hx4tws2six3b9c ----- - -Here's the corresponding extended _public_ key, encoded in Base58Check: - ----- -xpub67xpozcx8pe95XVuZLHXZeG6XWXHpGq6Qv5cmNfi7cS5mtjJ2tgypeQbBs2UAR6KECeeMVKZBPLrtJunSDMstweyLXhRgPxdp14sk9tJPW9 ----- - -[[public__child_key_derivation]] -===== Public child key derivation - -((("public and private keys", "public child key derivation")))As mentioned previously, a very useful characteristic of HD wallets is the ability to derive public child keys from public parent keys, _without_ having the private keys. This gives us two ways to derive a child public key: either from the child private key, or directly from the parent public key. - -An extended public key can be used, therefore, to derive all of the _public_ keys (and only the public keys) in that branch of the HD wallet structure. - -This shortcut can be used to create very secure public key–only deployments where a server or application has a copy of an extended public key and no private keys whatsoever. That kind of deployment can produce an infinite number of public keys and Bitcoin addresses, but cannot spend any of the money sent to those addresses. Meanwhile, on another, more secure server, the extended private key can derive all the corresponding private keys to sign transactions and spend the money. - -One common application of this solution is to install an extended public key on a web server that serves an ecommerce application. The web server can use the public key derivation function to create a new Bitcoin address for every transaction (e.g., for a customer shopping cart). The web server will not have any private keys that would be vulnerable to theft. Without HD wallets, the only way to do this is to generate thousands of Bitcoin addresses on a separate secure server and then preload them on the ecommerce server. That approach is cumbersome and requires constant maintenance to ensure that the ecommerce server doesn't "run out" of addresses. - -((("cold storage")))((("storage", "cold storage")))((("hardware wallets")))Another common application of this solution is for cold-storage or hardware wallets. In that scenario, the extended private key can be stored on a paper wallet or hardware device (such as a Trezor hardware wallet), while the extended public key can be kept online. The user can create "receive" addresses at will, while the private keys are safely stored offline. To spend the funds, the user can use the extended private key on an offline signing Bitcoin client or sign transactions on the hardware wallet device (e.g., Trezor). <> illustrates the mechanism for extending a parent public key to derive child public keys. - -[[CKDpub]] -.Extending a parent public key to create a child public key -image::images/mbc2_0511.png["ChildPublicDerivation"] - -===== Hardened child key derivation - -((("public and private keys", "hardened child key derivation")))((("hardened derivation")))The ability to derive a branch of public keys from an xpub is very useful, but it comes with a potential risk. Access to an xpub does not give access to child private keys. However, because the xpub contains the chain code, if a child private key is known, or somehow leaked, it can be used with the chain code to derive all the other child private keys. A single leaked child private key, together with a parent chain code, reveals all the private keys of all the children. Worse, the child private key together with a parent chain code can be used to deduce the parent private key. - -To counter this risk, HD wallets use an alternative derivation function called _hardened derivation_, which "breaks" the relationship between parent public key and child chain code. The hardened derivation function uses the parent private key to derive the child chain code, instead of the parent public key. This creates a "firewall" in the parent/child sequence, with a chain code that cannot be used to compromise a parent or sibling private key. The hardened derivation function looks almost identical to the normal child private key derivation, except that the parent private key is used as input to the hash function, instead of the parent public key, as shown in the diagram in <>. - -[[CKDprime]] -.Hardened derivation of a child key; omits the parent public key -image::images/mbc2_0513.png["ChildHardPrivateDerivation"] - -[role="pagebreak-before"] -When the hardened private derivation function is used, the resulting child private key and chain code are completely different from what would result from the normal derivation function. The resulting "branch" of keys can be used to produce extended public keys that are not vulnerable, because the chain code they contain cannot be exploited to reveal any private keys. Hardened derivation is therefore used to create a "gap" in the tree above the level where extended public keys are used. - -In simple terms, if you want to use the convenience of an xpub to derive branches of public keys, without exposing yourself to the risk of a leaked chain code, you should derive it from a hardened parent key, rather than a normal (non-hardened) parent key. As a best practice, the level-1 children of the master keys are always derived through the hardened derivation, to prevent compromise of the master keys. - -===== Index numbers for normal and hardened derivation - -The index number used in the derivation function is a 32-bit integer. To easily distinguish between keys derived through the normal derivation function versus keys derived through hardened derivation, this index number is split into two ranges. Index numbers between 0 and 2^31^–1 (0x0 to 0x7FFFFFFF) are used _only_ for normal derivation. Index numbers between 2^31^ and 2^32^–1 (0x80000000 to 0xFFFFFFFF) are used _only_ for hardened derivation. Therefore, if the index number is less than 2^31^, the child is normal, whereas if the index number is equal or above 2^31^, the child is hardened. - -To make the index number easier to read and display, the index number for hardened children is displayed starting from zero, but with a prime symbol. The first normal child key is therefore displayed as 0, whereas the first hardened child (index 0x80000000) is displayed as 0++'++. In sequence then, the second hardened key would have index 0x80000001 and would be displayed as 1++'++, and so on. When you see an HD wallet index i++'++, that means 2^31^+i. - -===== HD wallet key identifier (path) - -((("hierarchical deterministic (HD) wallets")))Keys in an HD wallet are identified using a "path" naming convention, with each level of the tree separated by a slash (/) character (see <>). Private keys derived from the master private key start with "m." Public keys derived from the master public key start with "M." Therefore, the first child private key of the master private key is m/0. The first child public key is M/0. The second grandchild of the first child is m/0/1, and so on. - -The "ancestry" of a key is read from right to left, until you reach the master key from which it was derived. For example, identifier m/x/y/z describes the private key that is the z-th child of the parent private key m/x/y, which is the y-th child of the parent private key m/x, which is the x-th child of the parent master private key m. - -[[table_4-8]] -.HD wallet path examples -[options="header"] -|======= -|HD path | Key described -| m/0 | The first (0) child private key from the master private key (m) -| m/0/0 | The first (0) child private key from the first child (m/0) -| m/0'/0 | The first (0) normal child from the first _hardened_ child (m/0') -| m/1/0 | The first (0) child private key from the second child (m/1) -| M/23/17/0/0 | The first (0) child public key from the first child (M/23/17/0) from the 18th child (M/23/17) from the 24th child (M/23) -|======= - -===== Navigating the HD wallet tree structure - -The HD wallet tree structure offers tremendous flexibility. Each parent extended key can have 4 billion children: 2 billion normal children and 2 billion hardened children. Each of those children can have another 4 billion children, and so on. The tree can be as deep as you want, with an infinite number of generations. With all that flexibility, however, it becomes quite difficult to navigate this infinite tree. It is especially difficult to transfer HD wallets between implementations, because the possibilities for internal organization into branches and subbranches are endless. - -Two BIPs offer a solution to this complexity by creating some proposed standards for the structure of HD wallet trees. BIP-43 proposes the use of the first hardened child index as a special identifier that signifies the "purpose" of the tree structure. Based on BIP-43, an HD wallet should use only one level-1 branch of the tree, with the index number identifying the structure and namespace of the rest of the tree by defining its purpose. For example, an HD wallet using only branch m/i++'++/ is intended to signify a specific purpose and that purpose is identified by index number "i." - -Extending that specification, BIP-44 proposes a multiaccount structure as "purpose" number +44'+ under BIP-43. All HD wallets following the BIP-44 structure are identified by the fact that they only used one branch of the tree: m/44'/. - -BIP-44 specifies the structure as consisting of five predefined tree levels: - ------ -m / purpose' / coin_type' / account' / change / address_index ------ - -The first-level "purpose" is always set to +44'+. The second-level "coin_type" specifies the type of cryptocurrency coin, allowing for multicurrency HD wallets where each currency has its own subtree under the second level. There are three currencies defined for now: Bitcoin is m/44'/0', Bitcoin Testnet is m/44++'++/1++'++, and Litecoin is m/44++'++/2++'++. - -The third level of the tree is "account," which allows users to subdivide their wallets into separate logical subaccounts, for accounting or organizational purposes. For example, an HD wallet might contain two bitcoin "accounts": m/44++'++/0++'++/0++'++ and m/44++'++/0++'++/1++'++. Each account is the root of its own subtree. - -((("keys and addresses", see="also public and private keys")))On the fourth level, "change," an HD wallet has two subtrees, one for creating receiving addresses and one for creating change addresses. Note that whereas the previous levels used hardened derivation, this level uses normal derivation. This is to allow this level of the tree to export extended public keys for use in a nonsecured environment. Usable addresses are derived by the HD wallet as children of the fourth level, making the fifth level of the tree the "address_index." For example, the third receiving address for bitcoin payments in the primary account would be M/44++'++/0++'++/0++'++/0/2. <> shows a few more examples. - -[[table_4-9]] -.BIP-44 HD wallet structure examples -[options="header"] -|======= -|HD path | Key described -| M/44++'++/0++'++/0++'++/0/2 | The third receiving public key for the primary bitcoin account -| M/44++'++/0++'++/3++'++/1/14 | The fifteenth change-address public key for the fourth bitcoin account -| m/44++'++/2++'++/0++'++/0/1 | The second private key in the Litecoin main account, for signing transactions -|======= - -==== Using an Extended Public Key on a Web Store - -((("wallets", "technology of", "using extended public keys on web stores")))Let's see how HD wallets are used by continuing our story with Gabriel's web store.((("use cases", "web store", id="gabrielfivetwo"))) - -Gabriel first set up his web store as a hobby, based on a simple hosted Wordpress page. His store was quite basic with only a few pages and an order form with a single Bitcoin address. - -Gabriel used the first Bitcoin address generated by his Trezor device as the main Bitcoin address for his store. This way, all incoming payments would be paid to an address controlled by his Trezor hardware wallet. - -Customers would submit an order using the form and send payment to Gabriel's published Bitcoin address, triggering an email with the order details for Gabriel to process. With just a few orders each week, this system worked well enough. - -However, the little web store became quite successful and attracted many orders from the local community. Soon, Gabriel was overwhelmed. With all the orders paying the same address, it became difficult to correctly match orders and transactions, especially when multiple orders for the same amount came in close together. - -Gabriel's HD wallet offers a much better solution through the ability to derive public child keys without knowing the private keys. Gabriel can load an extended public key (xpub) on his website, which can be used to derive a unique address for every customer order. Gabriel can spend the funds from his Trezor, but the xpub loaded on the website can only generate addresses and receive funds. This feature of HD wallets is a great security feature. Gabriel's website does not contain any private keys and therefore does not need high levels of security. - -To export the xpub, Gabriel uses the Trezor Suite desktop app in conjunction with the Trezor hardware wallet. The Trezor device must be plugged in for the public keys to be exported. Note that hardware wallets will never export private keys—those always remain on the device. <> shows what Gabriel sees in Trezor Suite when exporting the xpub. - -[[export_xpub]] -.Exporting an xpub from a Trezor hardware wallet -image::images/mbc2_0512.png["Exporting the xpub from the Trezor"] - -Gabriel copies the xpub to his web store's bitcoin shop software. He uses _BTCPay Server_, which is an open source web-store for a variety of web hosting and content platforms. BTCPay Server uses the xpub to generate a unique address for every purchase. ((("", startref="gabrielfivetwo"))) - -===== Account Discovery and Management - -Gabriel's business is flourishing. He has provided his extended public key (xpub) to _BTCPay Server_, which is generating unique addresses for customers to his website. Every time a customer to Gabriel's website clicks on the "Checkout" button with a specified payment modality (in this case, bitcoin), _BTCPay Server_ generates a new address for that customer. More specifically, _BTCPay Server_ iterates on the _address_index_ tree to create a new address to display to the customer, as defined by BIP-44. If the customer decides to switch payment methods or abandon the transaction entirely, this Bitcoin address goes unused and will not be used for another customer right away. - -At a single moment in time, Gabriel's website may have a large volume of outstanding addresses for customers making purchases, some of which may go unused and eventually expire. Once these addresses expire, _BTCPay Server_ will go back to reuse these addresses to fill the gap in _address_index_, but it becomes clear how there can be gaps between the _address_index_ leaves of the hierarchical deterministic tree where the money is actually located. - -Let's say that Gabriel is interested in viewing his total amount of bitcoin earned on a watch-only wallet (one that allows you to view transaction history, but not spend funds) that is separate from BTCPay Server but also conforms to the BIP-44 standard. How should this separate wallet go about searching for funds in this vast hierarchical tree, and when should it stop looking? Most wallets will typically follow an iterative process that utilizes a predefined limit, known as the _gap limit_. If, while searching for used addresses, the wallet doesn't find used addresses in a row beyond this limit number, it will stop searching the address chain. The default gap limit is typically set to 20. This is detailed in [[bip-44]]https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki[BIP-44]. - -[TIP] -==== -Gap limits explain the phenomenon whereby the importing of a wallet may show an incorrect or zero balance. The funds are not lost, but rather, the wallet importing function has not traversed enough leaves to fully detect funds. Many wallets allow this default gap limit to be changed, and Gabriel may need to increase this limit to allow his wallet to fully import his transaction history. -==== diff --git a/ch05_wallets.adoc b/ch05_wallets.adoc new file mode 100644 index 00000000..7a2ddc54 --- /dev/null +++ b/ch05_wallets.adoc @@ -0,0 +1,1572 @@ +//FIXME:reduce difficulty / we ramp up too quick Lesmes feedback +[[ch05_wallets]] +== Wallet Recovery + +Creating pairs of private and public keys is a crucial part of allowing +Bitcoin wallets to receive and spend bitcoins. But losing access to a +private key can make it impossible for anyone to ever spend the bitcoins +received to the corresponding public key. Wallet and protocol +developers over the years have worked to design systems that allow users +to recover access to their bitcoins after a problem without compromising +security the rest of the time. + +In((("wallets", "key generation", "independent", id="wallet-keygen-independent")))((("key generation", "independent", id="keygen-independent")))((("independent key generation", id="independent-keygen")))((("public key cryptography", "wallet recovery key generation", see="key generation"))) this chapter, we'll examine some of the different methods employed by +wallets to prevent the loss of data from becoming a loss of money. +Some solutions have almost no downsides and are universally adopted by +modern wallets. We'll simply recommend those solutions as best +practices. Other solutions have both advantages and disadvantages, +leading different wallet authors to make different trade-offs. +In those cases, we'll describe the various options available. + +=== Independent Key Generation + +++++ +

+Wallets for physical cash hold that cash, +so it's unsurprising that many people mistakenly believe that +Bitcoin wallets contain bitcoins. In fact, what many people call a +Bitcoin wallet—which we call a wallet database to distinguish it +from wallet applications—contains only keys. Those keys are associated +with bitcoins recorded on the blockchain. By proving to Bitcoin full nodes that you +control the keys, you can spend the associated bitcoins. +

+++++ + +Simple wallet databases contain both the public keys to which bitcoins +are received and the private keys that allow creating the signatures +necessary to authorize spending those bitcoins. Other wallets' databases +may contain only public keys, or only some of the private keys necessary +to authorize a spending transaction. Their wallet applications produce +the necessary signatures by working with external tools, such as +hardware signing devices or other wallets in a multisignature scheme. + +It's possible for a wallet application to independently generate each of +the wallet keys it later plans to use, as illustrated in +<>. All early Bitcoin wallet applications did +this, but it required users to back up the wallet database each time they +generated and distributed new keys, which could be as often as each time +they generated a new address to receive a new payment. Failure to back +up the wallet database on time would lead to the user losing access to +any funds received to keys that had not been backed up. + +For each independently generated key, the user would need to back up +about 32 bytes, plus overhead. Some users and wallet applications tried +to minimize the amount of data that needed to be backed up +by only using a single key. Although that can be secure, it severely +reduces the privacy of that user and all of the people with whom they +transact. People who valued their privacy and those of their peers +created new key pairs for each transaction, producing wallet databases +that could only reasonably be backed up using digital media. + +[[Type0_wallet]] +[role="width-60"] +.Nondeterministic key generation: a collection of independently generated keys stored in a wallet database. +image::images/mbc3_0501.png["Non-Deterministic Wallet"] + +Modern wallet applications don't independently generate keys but instead +derive them from a single random seed using a ((("wallets", "key generation", "independent", startref="wallet-keygen-independent")))((("key generation", "independent", startref="keygen-independent")))((("independent key generation", startref="independent-keygen")))repeatable (deterministic) +algorithm. + +==== Deterministic Key Generation + +A hash function((("wallets", "key generation", "deterministic", id="wallet-keygen-determine")))((("key generation", "deterministic", id="keygen-determine")))((("deterministic key generation", id="determine-keygen")))((("hash functions", "deterministic key generation", id="hash-determine"))) will always produce the same output when given the same +input, but if the input is changed even slightly, the output will be +different. If the function is cryptographically secure, nobody should +be able to predict the new output--not even if they know the new input. + +This allows us to take one random value and transform it into a +practically unlimited number of seemingly random values. Even more +useful, later using the same hash function with the same ((("seeds")))input +(called a _seed_) will produce the same seemingly random values: + +---- +# Collect some entropy (randomness) +$ dd if=/dev/random count=1 status=none | sha256sum +f1cc3bc03ef51cb43ee7844460fa5049e779e7425a6349c8e89dfbb0fd97bb73 - + +# Set our seed to the random value +$ seed=f1cc3bc03ef51cb43ee7844460fa5049e779e7425a6349c8e89dfbb0fd97bb73 + +# Deterministically generate derived values +$ for i in {0..2} ; do echo "$seed + $i" | sha256sum ; done +50b18e0bd9508310b8f699bad425efdf67d668cb2462b909fdb6b9bd2437beb3 - +a965dbcd901a9e3d66af11759e64a58d0ed5c6863e901dfda43adcd5f8c744f3 - +19580c97eb9048599f069472744e51ab2213f687d4720b0efc5bb344d624c3aa - +---- + +If we use the derived values as our private keys, we can later generate +exactly those same private keys by using our seed value with the +algorithm we used before. A user of deterministic key generation can +back up every key in their wallet by simply recording their seed and +a reference to the deterministic algorithm they used. For example, even +if Alice has a million bitcoins received to a million different +addresses, all she needs to back up in order to later recover access to +those bitcoins is: + +---- +f1cc 3bc0 3ef5 1cb4 3ee7 8444 60fa 5049 +e779 e742 5a63 49c8 e89d fbb0 fd97 bb73 +---- + +A logical diagram of basic sequential deterministic key generation is +shown in <>. However, modern wallet applications have a +more clever way of accomplishing this that allows public keys to be +derived separately from their corresponding private keys, making it +possible to store private keys more securely than ((("wallets", "key generation", "deterministic", startref="wallet-keygen-determine")))((("key generation", "deterministic", startref="keygen-determine")))((("deterministic key generation", startref="determine-keygen")))((("hash functions", "deterministic key generation", startref="hash-determine")))public keys. + +[[Type1_wallet]] +[role="width-70"] +.Deterministic key generation: a deterministic sequence of keys derived from a seed for a wallet database. +image::images/mbc3_0502.png["Deterministic Wallet"] + + +[[public_child_key_derivation]] +==== Public Child Key Derivation + +In <>, we learned((("wallets", "key generation", "public child key derivation", id="wallet-keygen-public-child")))((("key generation", "public child key derivation", id="keygen-public-child")))((("public child key derivation", id="public-child-derive")))((("child key pair derivation", id="child-key-pair"))) how to create a public key from a private key +using elliptic curve cryptography (ECC). Although operations on an +elliptic curve are not intuitive, they are analogous to the addition, +subtraction, and multiplication operations used in regular +arithmetic. In other words, it's possible to add or subtract from a +public key, or to multiply it. Consider the operation we used in +<> for +generating a public key (_K_) from a private key (_k_) using the generator +point (_G_): + +[latexmath] +++++ +\begin{equation} +{K = k \times G} +\end{equation} +++++ + +It's possible to create a derived key pair, called a child key pair, by +simply adding the same value to both sides of the equation: + +//// +[latexmath] +++++ +\begin{equation} +K + (123 \times G) =\!\!\!= (k + 123) \times G +\end{equation} +++++ +//// + +++++ +
+ + + K + + + ( + 123 + × + G + ) + == + ( + k + + + 123 + ) + × + G + + +
+++++ + + +[TIP] +==== +In equations throughout this book, we use a single equals sign for +operations such as _K_ = _k_ × _G_ where the value of a variable is +calculated. We use a double equals sign to show both sides of an +equation are equivalent, or that an operation should return false (not +true) if the two sides aren't equivalent. +==== + +An interesting consequence of this is that adding 123 to the public +key can be done using entirely public information. For example, Alice +generates public key _K_ and gives it to Bob. Bob doesn't know the +private key, but he does know the global constant _G_, so he can add any +value to the public key to produce a derived public child key. If he +then tells Alice the value he added to the public key, she can add the +same value to the private key, producing a derived private child key +that corresponds to the public child key Bob created. + +In other words, it's possible to create child public keys even if you +don't know anything about the parent private key. The value added to a +public key is ((("key tweaks")))known as a _key tweak._ If a deterministic algorithm is +used for generating the key tweaks, then it's possible for someone +who doesn't know the private key to create an essentially unlimited +sequence of public child keys from a single public parent key. The +person who controls the private parent key can then use the same key +tweaks to create all the corresponding private child keys. + +This technique is commonly used to separate wallet application +frontends (which don't require private keys) from signing operations +(which do require private keys). For example, Alice's frontend +distributes her public keys to people wanting to pay her. Later, when +she wants to spend the received money, she can provide the key tweaks +she used((("hardware signing devices"))) to a _hardware signing device_ (sometimes confusingly called a +_hardware wallet_) that securely stores her original private key. The +hardware signer uses the tweaks to derive the necessary child private +keys and uses them to sign the transactions, returning the signed +transactions to the less-secure frontend for broadcast to the Bitcoin +network. + +Public child key derivation can produce a linear sequence of keys +similar to the previously seen <>, but modern wallet +applications use one more trick to provide a tree of keys instead a +single sequence, as described in the((("wallets", "key generation", "public child key derivation", startref="wallet-keygen-public-child")))((("key generation", "public child key derivation", startref="keygen-public-child")))((("public child key derivation", startref="public-child-derive")))((("child key pair derivation", startref="child-key-pair"))) following section. + +[[hd_wallets]] +==== Hierarchical Deterministic (HD) Key Generation (BIP32) + +Every ((("wallets", "key generation", "HD (hierarchical deterministic)")))((("key generation", "HD (hierarchical deterministic)")))((("HD (hierarchical deterministic) key generation")))((("BIP32 HD (hierarchical deterministic) key generation", primary-sortas="BIP032")))modern Bitcoin wallet of which we're aware uses hierarchical +deterministic (HD) key generation by default. This standard, defined in +BIP32, uses deterministic key generation and optional public child key +derivation with an algorithm that produces a tree of keys. +In this tree, any key can be the parent of a sequence of child keys, and +any of those child keys can be a parent for another sequence of +child keys (grandchildren of the original key). There's no arbitrary +limit on the depth of the tree. This tree structure is illustrated in +<>. + +[[Type2_wallet]] +.HD wallet: a tree of keys generated from a single seed. +image::images/mbc3_0503.png["HD wallet"] + +The tree structure can be used to express additional +organizational meaning, such as when a specific branch of subkeys is +used to receive incoming payments and a different branch is used to +receive change from outgoing payments. Branches of keys can also be used +in corporate settings, allocating different branches to departments, +subsidiaries, specific functions, or accounting categories. + +We'll provide a detailed exploration of HD wallets in <>. + +==== Seeds and Recovery Codes + +HD wallets((("wallets", "recovery codes", id="wallet-recovery2")))((("recovery codes", id="recovery-code2"))) are a very powerful mechanism for managing many +keys all derived from a single seed. If your wallet database +is ever corrupted or lost, you can regenerate all of the private keys +for your wallet using your ((("seeds")))original seed. But, if someone else gets +your seed, they can also generate all of the private keys, allowing them +to steal all of the bitcoins from a single-sig wallet and reduce the +security of bitcoins in multisignature wallets. In this section, we'll +look at several _recovery codes_, which are intended to make backups +easier and safer. + +Although seeds are large random numbers, usually 128 to 256 bits, most +recovery codes use human-language words. A large part of the motivation +for using words was to make a recovery code easy to remember. For +example, consider the recovery code encoded using both hexadecimal and +words in <>. + +[[hex_seed_vs_recovery_words]] +.A seed encoded in hex and in English words +==== +---- +Hex-encoded: +0C1E 24E5 9177 79D2 97E1 4D45 F14E 1A1A + +Word-encoded: +army van defense carry jealous true +garbage claim echo media make crunch +---- +==== + +There may be cases where ((("memorizing recovery codes")))remembering a recovery code is a powerful +feature, such as when you are unable to transport physical belongings +(like a recovery code written on paper) without them being seized or +inspected by an outside party that might steal your bitcoins. However, +most of the time, relying on memory alone is dangerous: + +- If you forget your recovery code and lose access to your original + wallet database, your bitcoins are lost to you forever. + +- If you die or suffer a severe injury, and your heirs don't have access + to your original wallet database, they won't be able to inherit your + bitcoins. + +- If someone thinks you have a recovery code memorized that will give + them access to bitcoins, they may attempt to coerce you into + disclosing that code. As of this writing, Bitcoin contributor Jameson + Lopp has + https://oreil.ly/aw5XM[documented] + over 100 physical attacks against suspected owners of bitcoin and + other digital assets, including at least three deaths and numerous + occasions where someone was tortured, held hostage, or had their + family threatened. + +[TIP] +==== +Even if you use a type of recovery code that was designed for easy +memorization, we very strongly encourage you to consider writing it down. +==== + +Several ((("wallets", "recovery codes", "types of", id="wallet-recovery-type")))((("recovery codes", "types of", id="recovery-code-type")))different types of recovery codes are in wide use as of this +writing: + +BIP39:: + The most ((("BIP39 recovery codes", primary-sortas="BIP039")))popular method for generating recovery codes for the + past decade, BIP39 involves generating a random sequence of bytes, + adding a checksum to it, and encoding the data into a series of 12 to + 24 words (which may be localized to a user's native language). The + words (plus an optional passphrase) are run through a _key-stretching + function_, and the output is used as a seed. BIP39 recovery codes have + several shortcomings, which later schemes attempt to address. + +Electrum v2:: + Used in((("Electrum v2 recovery codes"))) the Electrum wallet (version 2.0 and above), this word-based + recovery code has several advantages over BIP39. It doesn't rely on a + global word list that must be implemented by every version of every + compatible program, plus its recovery codes include a version number that + improves reliability and efficiency. Like BIP39, it supports an optional + passphrase (which Electrum calls a _seed extension_) and uses the same + key-stretching function. + +Aezeed:: + Used in ((("Aezeed recovery codes")))the LND wallet, this is another word-based recovery code that + offers improvements over BIP39. It includes two version numbers: one + is internal and eliminates several issues with upgrading wallet + applications (like Electrum v2's version number); the other version + number is external, which can be incremented to change the underlying + cryptographic properties of the recovery code. + It also includes a _wallet birthday_ + in the recovery code, a reference to the date when the user created + the wallet database. This allows a restoration process to find all of + the funds associated with a wallet without scanning the entire + blockchain, which is especially useful for privacy-focused lightweight clients. + It includes support for changing the passphrase or changing other + aspects of the recovery code without needing to move funds to a new + seed--the user need only back up a new recovery code. One + disadvantage compared to Electrum v2 is that, like BIP39, it depends + on both the backup and the recovery software supporting the same + word list. + +[role="less_space pagebreak-before"] +Muun:: + Used in ((("Muun recovery codes")))the Muun wallet, which defaults to requiring spending + transactions be signed by multiple keys, this is a nonword code that + must be accompanied by additional information (which Muun currently + provides in a PDF). This recovery code is unrelated to the seed and + is instead used to decrypt the private keys [.keep-together]#contained# in the PDF. + Although this is unwieldy compared to the BIP39, Electrum v2, and + Aezeed recovery codes, it provides support for new technologies and + standards that are becoming more common in new wallets, such as + Lightning Network (LN) support, output script descriptors, and miniscript. + +SLIP39:: + A successor ((("SLIP39 recovery codes")))to BIP39 with some of the same authors, SLIP39 allows + a single seed to be distributed using multiple recovery codes that can + be stored in different places (or by different people). When you + create the recovery codes, you can specify how many will be required + to recover the seed. For example, you create five recovery codes but + only require three of them to recover the seed. SLIP39 provides + support for an optional passphrase, depends on a global word list, and + doesn't directly provide versioning. + +[NOTE] +==== +A new system((("Codex32 recovery codes"))) for distributing recovery codes with similarities to SLIP39 +was proposed during the writing of this book. Codex32 allows creating +and validating recovery codes with nothing except printed instructions, +scissors, a precision knife, brass fasteners, and a pen--plus privacy +and a few hours of spare time. Alternatively, those who trust computers can create recovery codes +instantly using software on a digital device. You can create up to 31 +recovery codes to be stored in different places, specifying how many of +them will be required in order to recover the seed. As a new proposal, +details about Codex32 may change significantly before this book is +published, so we encourage ((("wallets", "recovery codes", "types of", startref="wallet-recovery-type")))((("recovery codes", "types of", startref="recovery-code-type")))any readers interested in distributed +recovery codes to investigate its https://oreil.ly/Xx_Zq[current +status]. +==== + +.Recovery Code Passphrases +**** +The BIP39, ((("wallets", "recovery codes", "passphrases", id="wallet-recovery-passphrase")))((("recovery codes", "passphrases", id="recovery-code-passphrase")))((("passphrases (for recovery codes)", id="passphrase")))Electrum v2, Aezeed, and SLIP39 schemes may all be used with an +optional passphrase. If the only place you keep this passphrase is in +your memory, it has the same advantages and disadvantages as memorizing +your recovery code. However, there's a further set of trade-offs +specific to the way the passphrase is used by the recovery code. + +Three of the schemes (BIP39, Electrum v2, and SLIP39) do not include the optional passphrase in the +checksum they use to protect against data entry mistakes. Every +passphrase (including not using a passphrase) will result in producing a +seed for a BIP32 tree of keys, but they won't be the same trees. +Different passphrases will result in different keys. That can be a +positive or a negative, depending on your perspective: + +- On the positive, if someone obtains your recovery code (but not your + passphrase), they will see a valid BIP32 tree of keys. + If you prepared for that contingency and sent some bitcoins to the + nonpassphrase tree, they will steal that money. Although having some + of your bitcoins stolen is normally a bad thing, it can also provide + you with a warning that your recovery code has been compromised, + allowing you to investigate and take corrective measures. + The ability to create multiple passphrases for the same recovery code + that all look valid is a type of _plausible deniability_. + +- On the negative, if you're coerced to give an attacker a recovery + code (with or without a passphrase) and it doesn't yield the amount of + bitcoins they expected, they may continue trying to coerce you until + you give them a different passphrase with access to more bitcoins. + Designing for plausible deniability means there's no way to prove to + an attacker that you've revealed all of your information, so they may + continue trying to coerce you even after you've given them all of + your bitcoins. + +- An additional negative is the reduced amount of error detection. If + you enter a slightly wrong passphrase when restoring from a backup, + your wallet can't warn you about the mistake. If you were expecting + a balance, you will know something is wrong when your wallet + application shows you a zero balance for the regenerated key tree. + However, novice users may think their money was permanently lost and do + something foolish, such as give up and throw away their recovery code. + Or, if you were actually expecting a zero balance, you might use the + wallet application for years after your mistake until the next time + you restore with the correct passphrase and see a zero balance. + Unless you can figure out what typo you previously made, your funds + are gone. + +Unlike the other schemes, the Aezeed seed encryption scheme +authenticates its optional passphrase and will return an error if you +provide an incorrect value. This eliminates plausible deniability, adds +error detection, and makes it possible to prove that the passphrase has been +revealed. + +Many users and developers disagree on which approach is better, with +some strongly in favor of plausible deniability and others preferring the +increased safety that error detection gives novice users and those under +duress. We suspect the debate will continue for as long as recovery +codes continue to be widely ((("wallets", "recovery codes", startref="wallet-recovery2")))((("recovery codes", startref="recovery-code2")))((("passphrases (for recovery codes)", startref="passphrase")))((("wallets", "recovery codes", "passphrases", startref="wallet-recovery-passphrase")))((("recovery codes", "passphrases", startref="recovery-code-passphrase")))used. +**** + +==== Backing Up Nonkey Data + +The ((("wallets", "nonkey data, backing up", id="wallet-nonkey-backups")))((("nonkey data, backing up", id="nonkey-backups")))((("backing up", "nonkey data", id="backup-nonkey")))most important data in a wallet database is its private keys. If +you lose access to the private keys, you lose the ability to spend your +bitcoins. Deterministic key derivation and recovery codes provide a +reasonably robust solution for backing up and recovering your keys and +the bitcoins they control. However, it's important to consider that +many wallet databases store more than +just keys--they also store user-provided information about every +transaction they sent or received. + +For example, when Bob creates a new address as part of sending an +invoice to Alice, he ((("labels, backing up", id="label-backup")))adds a _label_ to the address he generates +so that he can distinguish her payment +from other payments he receives. When Alice pays Bob's address, she +labels the transaction as paying Bob for the same reason. Some wallets +also add other useful information to transactions, such as the current +exchange rate, which can be useful for calculating taxes in some +jurisdictions. These labels are stored entirely within their own +wallets--not shared with the network--protecting their privacy +and keeping unnecessary personal data out of the blockchain. For +an example, see <>. + +++++ + + + + + + + + + + + + + + + + + + + + + +
Alice’s transaction history with each transaction labeled
DateLabelBTC

2023-01-01

Bought bitcoins from Joe

+0.00100

2023-01-02

Paid Bob for podcast

−0.00075

+++++ + +However, because address and transaction labels are stored only in each +user's wallet database and because they aren't deterministic, they can't +be restored by using just a recovery code. If the only recovery is +seed-based, then all the user will see is a list of approximate +transaction times and bitcoin amounts. This can make it quite difficult +to figure out how you used your money in the past. Imagine reviewing a +bank or credit card statement from a year ago that had the date and +amount of every transaction listed but a blank entry for the +"description" field. + +Wallets should provide their users with a convenient way to back up +label data. That seems obvious, but there are a number of +widely used wallet applications that make it easy to create and use +recovery codes but that provide no way to back up or restore label +data. + +Additionally, it may be useful for wallet applications to provide a +standardized format to export labels so that they can be used in other +applications (e.g., accounting software). A standard for that format is +proposed in BIP329. + +Wallet applications implementing additional protocols beyond basic +Bitcoin support may also need or want to store other data. For example, +as of 2023, an increasing number of applications have added support for +sending and receiving transactions over the Lightning Network (LN). +Although the LN protocol provides a method to recover +funds in the event of a data loss, called _static channel backups_, it +can't guarantee results. If the node your wallet connects to realizes +you've lost data, it may be able to steal bitcoins from you. If it +loses its wallet database at the same time you lose your database, and +neither of you has an adequate backup, you'll both lose funds. + +Again, this means users and wallet applications need to do more than just back up a +recovery code. + +One solution implemented by a few wallet applications is to frequently +and automatically create complete backups of their wallet database +encrypted by one of the keys derived from their seed. Bitcoin keys must +be unguessable and modern encryption algorithms are considered very +secure, so nobody should be able to open the encrypted backup except +someone who can generate the seed. This makes it safe to store the backup on +untrusted computers such as cloud hosting services or even random +network peers. + +Later, if the original wallet database is lost, the user can enter their +recovery code into the wallet application to restore their seed. The +application can then retrieve the latest backup file, regenerate the +encryption key, decrypt the backup, and restore all of the user's labels +and additional ((("wallets", "nonkey data, backing up", startref="wallet-nonkey-backups")))((("nonkey data, backing up", startref="nonkey-backups")))((("backing up", "nonkey data", startref="backup-nonkey")))protocol data. + +==== Backing Up Key Derivation Paths + +In a ((("wallets", "key generation", "backing up derivation paths", id="wallet-keygen-backups")))((("key generation", "backing up derivation paths", id="keygen-backups")))((("backing up", "key derivation paths", id="backup-key-derive")))BIP32 tree of keys, there are approximately four billion first-level +keys; each of those keys can have its own four billion children, with +those children each potentially having four billion children of their +own, and so on. It's not possible for a wallet application to generate +even a small fraction of every possible key in a BIP32 tree, which means +that recovering from data loss requires knowing more than just the +recovery code, the algorithm for obtaining your seed (e.g., BIP39), and +the deterministic key derivation algorithm +(e.g., BIP32)—it also requires knowing what paths in the tree of keys +your wallet application used for generating the specific keys it distributed. + +Two solutions to this problem have been adopted. The first is using +standard paths. Every time there's a change related to the addresses +that wallet applications might want to generate, someone creates a BIP +defining what key derivation path to use. For example, BIP44 defines +`m/44'/0'/0'` as the path to use for keys in P2PKH scripts (a +legacy address). A wallet application implementing this standard uses +the keys in that path both when it is first started and after a +restoration from a recovery code. We call((("implicit paths", id="implicit-path"))) this solution _implicit +paths_. Several popular implicit paths defined by BIPs are shown in <> + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Implicit script paths defined by various BIPs
StandardScriptBIP32 path

BIP44

P2PKH

m/44'/0'/0'

BIP49

Nested P2WPKH

m/49'/1'/0'

BIP84

P2WPKH

m/84'/0'/0'

BIP86

P2TR Single-key

m/86'/0'/0'

+++++ + +[role="less_space pagebreak-before"] +The second solution is to back up the path information with the recovery +code, making it clear which path is used with which scripts. We ((("explicit paths", id="explicit-path")))call +this _explicit paths_. + +The advantage of implicit paths is that users don't need to keep a record +of what paths they use. If the user enters their recovery code into the +same wallet application they previously used, of the same version or +higher, it will automatically regenerate keys for the same paths it +previously used. + +The disadvantage of implicit scripts is their inflexibility. When a +recovery code is entered, a wallet application must generate the keys +for every path it supports and it must scan the blockchain for +transactions involving those keys, otherwise it might not find all of a +user's transactions. This is wasteful in wallets that support many +features each with their own path if the user only tried a few of those +features. + +For implicit path recovery codes that don't include a version number, +such as BIP39 and SLIP39, a new version of a wallet application that drops support +for an older path can't warn users during the restore process that some +of their funds may not be found. The same problem happens in reverse if +a user enters their recovery code into older software: it won't find +newer paths to which the user may have received funds. Recovery codes +that include version information, such as Electrum v2 and Aezeed, can +detect that a user is entering an older or newer recovery code and +direct them to appropriate resources. + +The final consequence of implicit paths is that they can only include +information that is either universal (such as a standardized path) or +derived from the seed (such as keys). Important nondeterministic +information that's specific to a certain user can't be restored using +a recovery code. For example, Alice, Bob, and Carol receive funds that +can only be spent with signatures from two out of three of them. Although +Alice only needs either Bob's or Carol's signature to spend, she needs +both of their public keys in order to find their joint funds on the +blockchain. That means each of them must back up the public keys for +all three of them. As multisignature and other advanced scripts become +more common on Bitcoin, the inflexibility of implicit paths becomes more +significant. + +The advantage of explicit paths is that they can describe exactly what +keys should be used with what scripts. There's no need to support +outdated scripts, no problems with backward or forward compatibility, +and any extra information (like the public keys of other users) can be +included directly. Their disadvantage is that they require users to back +up additional information along with their recovery code. The +additional information usually can't compromise a user's security, so it +doesn't require as much protection as the recovery code, although it can +reduce their privacy and does require some protection. + +[role="less_space pagebreak-before"] +Almost all wallet applications that use explicit paths as of this +writing use the _output script descriptors_ standard (called +_descriptors_ for short) as specified in BIPs 380, 381, 382, 383, 384, +385, 386, and 389. Descriptors +describe a script and the keys (or key paths) to be used with it. +A few example descriptors are shown in <>. + +++++ + + + + + + + + + + + + + + + + + + + + + + +
Sample descriptors from Bitcoin Core documentation (with elision)
DescriptorExplanation

pkh(02c6…​9ee5)

P2PKH script for the provided public key

sh(multi(2,022f…​2a01,03ac…​ccbe))

P2SH multisignature requiring two signatures corresponding to these two keys

pkh([d34db33f/44'/0'/0']xpub6ERA…​RcEL/1/*)

P2PKH scripts for the BIP32 d34db33f with the extended public key (xpub) at the path M/44'/0'/0', which is xpub6ERA…​RcEL, using the keys at M/1/* of that xpub

+++++ + + +It has long been the trend for wallet applications designed only for +single signature scripts to use implicit paths. Wallet applications +designed for multiple signatures or other advanced scripts are +increasingly adopting support for explicit paths using descriptors. +Applications that do both will usually conform to the standards for +implicit paths and also provide ((("wallets", "key generation", "backing up derivation paths", startref="wallet-keygen-backups")))((("key generation", "backing up derivation paths", startref="keygen-backups")))((("backing up", "key derivation paths", startref="backup-key-derive")))((("implicit paths", startref="implicit-path")))((("explicit paths", startref="explicit-path")))descriptors. + +=== A Wallet Technology Stack in Detail + +Developers of modern wallets can choose from a variety of different +technologies to help users create and use backups--and new solutions +appear every year. Instead of going into detail about each of the +options we described earlier in this chapter, we'll focus the rest of +this chapter on the stack of technologies we think is most widely +used in wallets as of early 2023: + +- BIP39 recovery codes +- BIP32 HD key derivation +- BIP44-style implicit paths + +All of these standards have been around since 2014 or earlier, and +you'll have no problem finding additional resources for using them. +However, if you're feeling bold, we do encourage you to investigate more +modern standards that may provide additional features or safety. + +[[recovery_code_words]] +==== BIP39 Recovery Codes + +BIP39 ((("wallets", "recovery codes", id="wallet-recovery-bip39")))((("recovery codes", id="recovery-code-bip39")))((("BIP39 recovery codes", primary-sortas="BIP039", id="bip39-recovery")))recovery codes are word +sequences that represent (encode) a random number used as a seed to +derive a deterministic wallet. The sequence of words is sufficient to +re-create the seed and from there, re-create all the +derived keys. A wallet application that implements deterministic wallets +with a BIP39 recovery code will show the user a sequence of 12 to 24 words when +first creating a wallet. That sequence of words is the wallet backup and +can be used to recover and re-create all the keys in the same or any +compatible wallet application. Recovery codes make it easier for users +to back up because they are easy to read and correctly +transcribe. + +[TIP] +==== +Recovery codes((("brainwallets"))) are often confused with +"brainwallets." They are not the same. The primary difference is that a +brainwallet consists of words chosen by the user, whereas recovery codes +are created randomly by the wallet and presented to the user. This +important difference makes recovery codes much more secure because +humans are very poor sources of randomness. +==== + +Note that BIP39 is one implementation of a recovery code standard. +BIP39 was proposed by the company behind the Trezor hardware wallet and +is compatible with many other wallets applications, although certainly +not all. + +BIP39 defines the creation of a recovery code and seed, which we +describe here in nine steps. For clarity, the process is split into two +parts: steps 1 through 6 are shown in <> and +steps 7 through 9 are shown in <>. + +[[generating_recovery_words]] +===== Generating a recovery code + +Recovery((("wallets", "recovery codes", "generating", id="wallet-recovery-bip39-generate")))((("recovery codes", "generating", id="recovery-code-bip39-generate")))((("BIP39 recovery codes", "generating", primary-sortas="BIP039", id="bip39-recovery-generate")))((("entropy", "recovery code generation", id="entropy-recovery-generate"))) codes are generated automatically by the wallet application using the +standardized process defined in BIP39. The wallet starts from a source +of entropy, adds a checksum, and then maps the entropy to a word list: + +1. Create a random sequence (entropy) of 128 to 256 bits. + +2. Create a checksum of the random sequence by taking the first +(entropy-length/32) bits of its SHA256 hash. + +3. Add the checksum to the end of the random sequence. + +4. Split the result into 11-bit length segments. + +5. Map each 11-bit value to a word from the predefined dictionary of +2,048 words. + +6. The recovery code is the sequence of words. + +<> shows how entropy is used to +generate a BIP39 recovery code. + +[[generating_entropy_and_encoding]] +.Generating entropy and encoding as a recovery code. +image::images/mbc3_0504.png["Generating entropy and encoding as a recovery code"] + +<> shows the relationship between the size of the entropy +data and the length of recovery code in((("wallets", "recovery codes", "generating", startref="wallet-recovery-bip39-generate")))((("recovery codes", "generating", startref="recovery-code-bip39-generate")))((("BIP39 recovery codes", "generating", primary-sortas="BIP039", startref="bip39-recovery-generate")))((("entropy", "recovery code generation", startref="entropy-recovery-generate"))) words. + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BIP39: entropy and word length
Entropy (bits)Checksum (bits)Entropy + checksum (bits)Recovery code words

128

4

132

12

160

5

165

15

192

6

198

18

224

7

231

21

256

8

264

24

+++++ + +[role="less_space pagebreak-before"] +[[recovery_to_seed]] +===== From recovery code to seed + +The ((("wallets", "recovery codes", "seed generation", id="wallet-recovery-bip39-seed")))((("recovery codes", "seed generation", id="recovery-code-bip39-seed")))((("BIP39 recovery codes", "seed generation", primary-sortas="BIP039", id="bip39-recovery-seed")))((("entropy", "seed generation", id="entropy-seed-generate")))((("seeds", "generating", id="seed-generate")))((("key-stretching functions", id="key-stretch")))recovery code +represents entropy with a length of 128 to 256 bits. The entropy is then +used to derive a longer (512-bit) seed through the use of the +https://oreil.ly/6lwbd[key-stretching function PBKDF2]. The seed produced is then used to build +a deterministic wallet and derive its keys. + +The key-stretching function takes two +parameters: the entropy and((("salt"))) a _salt_. The purpose of a salt in a +key-stretching function is to make it difficult to build a lookup table +enabling a brute-force attack. In the BIP39 standard, the salt has +another purpose--it allows the introduction of a passphrase that +serves as an additional security factor protecting the seed, as we will +describe in more detail in <>. + +[TIP] +==== +The key-stretching function, with its 2,048 rounds of hashing, makes it +slightly harder to brute-force attack the recovery code using software. +Special-purpose hardware is not significantly affected. For an attacker +who needs to guess a user's entire recovery code, the length of the code +(128 bits at a minimum) provides more than sufficient security. But for +cases where an attacker might learn a small part of the user's code, +key-stretching adds some security by slowing down how fast an attacker +can check different recovery code combinations. BIP39's parameters were +considered weak by modern standards even when it was first published +almost a decade ago, although that's likely a consequence of being +designed for compatibility with hardware signing devices with low-powered +CPUs. Some alternatives to BIP39 use stronger key-stretching +parameters, such as Aezeed's 32,768 rounds of hashing using the more +complex Scrypt algorithm, although they may not be as convenient to run +on hardware signing devices. +==== + +The process described in steps 7 through 9 continues from the process +described previously in <>: + +++++ +
    +
  1. The first parameter to the PBKDF2 key-stretching function is the + entropy produced from step 6.
    2. + +
  2. The second parameter to the PBKDF2 key-stretching function is a + salt. The salt is composed of the string constant + "mnemonic" concatenated with an optional user-supplied + passphrase string.
    3. + +
  3. PBKDF2 stretches the recovery code and salt parameters using 2,048 + rounds of hashing with the HMAC-SHA512 algorithm, producing a 512-bit + value as its final output. That 512-bit value is the seed.
    4. +
+++++ + +<> shows how a recovery code is used to generate a seed. + +[[fig_5_7]] +.From recovery code to seed. +image::images/mbc3_0505.png["From recovery code to seed"] + + +Tables pass:[#bip39_128_no_pass], +pass:[#bip39_128_w_pass], and +pass:[#bip39_256_no_pass] show +some examples of recovery codes and the seeds ((("key-stretching functions", startref="key-stretch")))they produce. + +++++ + + + + + + + + + + + + + + + + + + + + +
128-bit entropy BIP39 recovery code, no passphrase, resulting seed

Entropy input (128 bits)

0c1e24e5917779d297e14d45f14e1a1a

Recovery Code (12 words)

army van defense carry jealous true garbage claim echo media make crunch

Passphrase

(none)

Seed (512 bits)

5b56c417303faa3fcba7e57400e120a0ca83ec5a4fc9ffba757fbe63fbd77a89a1a3be4 +c67196f57c39a88b76373733891bfaba16ed27a813ceed498804c0570

+ + + + + + + + + + + + + + + + + + + + + +
128-bit entropy BIP39 recovery code, with passphrase, resulting seed

Entropy input (128 bits)

0c1e24e5917779d297e14d45f14e1a1a

Recovery Code (12 words)

army van defense carry jealous true garbage claim echo media make crunch

Passphrase

SuperDuperSecret

Seed (512 bits)

3b5df16df2157104cfdd22830162a5e170c0161653e3afe6c88defeefb0818c793dbb28 +ab3ab091897d0715861dc8a18358f80b79d49acf64142ae57037d1d54

+ + + + + + + + + + + + + + + + + + + + +
256-bit entropy BIP39 recovery code, no passphrase, resulting seed

Entropy input (256 bits)

2041546864449caff939d32d574753fe684d3c947c3346713dd8423e74abcf8c

Recovery Code (24 words)

cake apple borrow silk endorse fitness top denial coil riot stay wolf +luggage oxygen faint major edit measure invite love trap field dilemma oblige

Passphrase

(none)

Seed (512 bits)

3269bce2674acbd188d4f120072b13b088a0ecf87c6e4cae41657a0bb78f5315b33b3 +a04356e53d062e55f1e0deaa082df8d487381379df848a6ad7e98798404

+++++ + + +.How Much Entropy Do You Need? +**** +BIP32 allows seeds to be from 128 to 512 bits. BIP39 accepts from 128 +to 256 bits of entropy; Electrum v2 accepts 132 bits of entropy; Aezeed +accepts 128 bits of entropy; SLIP39 accepts either 128 or 256 bits. The +variation in these numbers makes it unclear how much entropy is needed +for safety. We'll try to demystify that. + +BIP32 extended private keys consist of a 256-bit key and a 256-bit chain +code, for a total of 512 bits. That means there's a maximum of 2^512^ +different possible extended private keys. If you start with more than +512 bits of entropy, you'll still get an extended private key containing +512 bits of entropy--so there's no point in using more than 512 bits +even if any of the standards we mentioned allowed that. + +However, even though there are 2^512^ different extended private keys, +there are only (slightly less than) 2^256^ regular private keys--and its +those private keys that actually secure your bitcoins. That means, if +you use more than 256 bits of entropy for your seed, you still get private keys +containing only 256 bits of entropy. There may be future +Bitcoin-related protocols where extra entropy in the extended keys +provides extra security, but that's not currently the case. + +The security strength of a Bitcoin public key is 128 bits. An attacker +with a classical computer (the only kind which can be used for a +practical attack as of this writing) would need to perform about 2^128^ +operations on Bitcoin's elliptic curve in order to find a private key +for another user's public key. The implication of a security strength +of 128 bits is that there's no apparent benefit to using more than 128 +bits of entropy (although you need to ensure your generated private +keys are selected uniformly from within the entire 2^256^ range of +private keys). + +There is one extra benefit of greater entropy: if a fixed percentage of +your recovery code (but not the whole code) is seen by an attacker, the +greater the entropy, the harder it will be for them to figure out part +of the code they didn't see. For example, if an attacker sees half of a +128-bit code (64 bits), it's plausible that they'll be able to brute +force the remaining 64 bits. If they see half of a 256-bit code (128 +bits), it's not plausible that they can brute force the other half. We +don't recommend relying on this defense--either keep your recovery codes +very safe or use a method like SLIP39 that lets you distribute your +recovery code across multiple locations without relying on the safety of +any individual code. + +As of 2023, most modern wallets generate 128 bits of entropy for their +recovery codes (or a value near 128, such as((("wallets", "recovery codes", "seed generation", startref="wallet-recovery-bip39-seed")))((("recovery codes", "seed generation", startref="recovery-code-bip39-seed")))((("BIP39 recovery codes", "seed generation", primary-sortas="BIP039", startref="bip39-recovery-seed")))((("entropy", "seed generation", startref="entropy-seed-generate")))((("seeds", "generating", startref="seed-generate"))) Electrum v2's 132 bits). +**** + +[[recovery_passphrase]] +===== Optional passphrase in BIP39 + +The((("wallets", "recovery codes", "passphrases", id="wallet-recovery-bip39-passphrase")))((("recovery codes", "passphrases", id="recovery-code-bip39-passphrase")))((("BIP39 recovery codes", "passphrases", primary-sortas="BIP039", id="bip39-recovery-passphrase")))((("passphrases (for recovery codes)", id="passphrase-optional"))) BIP39 standard allows the use of an optional +passphrase in the derivation of the seed. If no passphrase is used, the +recovery code is stretched with a salt consisting of the constant string ++"mnemonic"+, producing a specific 512-bit seed from any given recovery code. +If a passphrase is used, the stretching function produces a _different_ +seed from that same recovery code. In fact, given a single recovery code, every +possible passphrase leads to a different seed. Essentially, there is no +"wrong" passphrase. All passphrases are valid and they all lead to +different seeds, forming a vast set of possible uninitialized wallets. +The set of possible wallets is so large (2^512^) that there is no +practical possibility of brute-forcing or accidentally guessing one that +is in use. + +[TIP] +==== +There are no "wrong" passphrases in BIP39. Every passphrase leads to +some wallet, which unless previously used will be empty. +==== + +The optional passphrase creates two important features: + +- A second factor (something memorized) that makes a recovery code useless on + its own, protecting recovery codes from compromise by a casual thief. For + protection from a tech-savvy thief, you will need to use a very strong + passphrase. + +- A form of plausible deniability or "duress wallet," where a chosen + passphrase leads to a wallet with a small amount of funds used to + distract an attacker from the "real" wallet that contains the majority + of funds. + +It's important to note that the use of a passphrase also introduces the risk of loss: + +* If the wallet owner is incapacitated or dead and no one else knows the passphrase, the seed is useless and all the funds stored in the wallet are lost forever. + +* Conversely, if the owner backs up the passphrase in the same place as the seed, it defeats the purpose of a second((("wallets", "recovery codes", startref="wallet-recovery-bip39")))((("recovery codes", startref="recovery-code-bip39")))((("BIP39 recovery codes", primary-sortas="BIP039", startref="bip39-recovery")))((("wallets", "recovery codes", "passphrases", startref="wallet-recovery-bip39-passphrase")))((("recovery codes", "passphrases", startref="recovery-code-bip39-passphrase")))((("BIP39 recovery codes", "passphrases", primary-sortas="BIP039", startref="bip39-recovery-passphrase")))((("passphrases (for recovery codes)", startref="passphrase-optional"))) factor. + +++++ +

+While passphrases are very useful, they should only be used in +combination with a carefully planned process for backup and recovery, +considering the possibility of surviving the owner and allowing his or +her family to recover the cryptocurrency estate. +

+++++ + +[[hd_wallet_details]] +==== Creating an HD Wallet from the Seed + +HD wallets ((("wallets", "key generation", "HD (hierarchical deterministic)", id="wallet-keygen-hd")))((("key generation", "HD (hierarchical deterministic)", id="keygen-hd")))((("HD (hierarchical deterministic) key generation", id="hd-keygen")))((("BIP32 HD (hierarchical deterministic) key generation", primary-sortas="BIP032", id="bip32")))((("seeds", "HD wallet creation", id="seed-hdwallet")))are created from a ((("root seeds")))single _root seed_, which is a +128-, 256-, or 512-bit random number. Most commonly, this seed is +generated by or decrypted from a recovery code as detailed in the previous section. + +Every key in the HD wallet is deterministically derived from this root +seed, which makes it possible to re-create the entire HD wallet from +that seed in any compatible HD wallet. This makes it easy to back up, +restore, export, and import HD wallets containing thousands or even +millions of keys by simply transferring only the recovery code that the root +seed is derived from. The process of creating the master keys and master chain code for an HD +wallet is shown in <>. + +[[HDWalletFromSeed]] +.Creating master keys and chain code from a root seed. +image::images/mbc3_0506.png["HDWalletFromRootSeed"] + +The root seed is input into the HMAC-SHA512 algorithm and the resulting +hash is used to create a _master private key_ (_m_) and a _master chain +code_ (_c_). + +The master private key (_m_) then generates a corresponding master public +key (_M_) using the normal elliptic curve multiplication process _m_ × _G_ +that we saw in <>. + +The master chain code (_c_) is used to introduce entropy in the function that +creates child keys from parent keys, as we will see in the next section. + +===== Private child key derivation + +HD wallets ((("key generation", "HD (hierarchical deterministic)", "private child key derivation", id="keygen-hd-private-child")))((("HD (hierarchical deterministic) key generation", "private child key derivation", id="hd-keygen-private-child")))((("private child key derivation", id="private-child")))((("child key pair derivation", "private keys", id="child-key-pair-private")))use a _child key derivation_ (CKD) +function to derive child keys from parent keys. + +The child key derivation functions are based on a one-way hash function +that [.keep-together]#combines:# + +* A parent private or public key (uncompressed key) +* A seed called a chain code (256 bits) +* An index number (32 bits) + +The chain code is used to introduce deterministic random data to the +process, so that knowing the index and a child key is not sufficient to +derive other child keys. Knowing a child key does not make it possible +to find its siblings unless you also have the chain code. The initial +chain code seed (at the root of the tree) is made from the seed, while +subsequent child chain codes are derived from each parent chain code. + +These three items (parent key, chain code, and index) are combined and +hashed to generate children keys, as follows. + +The parent public key, chain code, and the index number are combined and +hashed with the HMAC-SHA512 algorithm to produce a 512-bit hash. This +512-bit hash is split into two 256-bit halves. The right-half 256 bits +of the hash output become the chain code for the child. The left-half +256 bits of the hash are added to the parent private key to produce the +child private key. In <>, we see this illustrated with the +index set to 0 to produce the "zero" (first by index) child of the +parent. + +[[CKDpriv]] +.Extending a parent private key to create a child private key. +image::images/mbc3_0507.png["ChildPrivateDerivation"] + +Changing the index allows us to extend the parent and create the other +children in the sequence (e.g., Child 0, Child 1, Child 2, etc.). Each +parent key can have 2,147,483,647 (2^31^) children (2^31^ is half of the +entire 2^32^ range available because the other half is reserved for a +special type of derivation we will talk about later in this chapter). + +Repeating the process one level down the tree, each child can in turn +become a parent and create its own children, in an infinite number of +generations. + +===== Using derived child keys + +Child private keys are indistinguishable from nondeterministic (random) +keys. Because the derivation function is a one-way function, the child +key cannot be used to find the parent key. The child key also cannot be +used to find any siblings. If you have the n^th^ child, you cannot find +its siblings, such as the n–1 child or the n+1 child, or any +other children that are part of the sequence. Only the parent key and +chain code can derive all the children. Without the child chain code, +the child key cannot be used to derive any grandchildren either. You +need both the child private key and the child chain code to start a new +branch and derive grandchildren. + +So what can the child private key be used for on its own? It can be used +to make a public key and a Bitcoin address. Then, it can be used to sign +transactions to spend anything paid to that address. + +[TIP] +==== +A child private key, the corresponding public key, and the Bitcoin +address are all indistinguishable from keys and addresses created +randomly. The fact that they are part of a sequence is not visible +outside of the HD wallet function that created them. Once created, they +operate exactly((("key generation", "HD (hierarchical deterministic)", "private child key derivation", startref="keygen-hd-private-child")))((("HD (hierarchical deterministic) key generation", "private child key derivation", startref="hd-keygen-private-child")))((("private child key derivation", startref="private-child")))((("child key pair derivation", "private keys", startref="child-key-pair-private"))) as "normal" keys. +==== + +===== Extended keys + +As +we saw ((("key generation", "HD (hierarchical deterministic)", "extended keys", id="keygen-hd-extend")))((("HD (hierarchical deterministic) key generation", "extended keys", "explained", id="hd-keygen-extend")))((("extended keys", "explained", id="extend-key")))earlier, the key derivation function can be used to create +children at any level of the tree, based on the three inputs: a key, a +chain code, and the index of the desired child. The two essential +ingredients are the key and chain code, and combined these are called an +_extended key_. The term "extended key" could also be thought of as +"extensible key" because such a key can be used to derive children. + +Extended keys are stored and represented simply as the concatenation of +the key and chain code. There +are two types of extended keys. An extended private key is the +combination of a private key and chain code and can be used to derive +child private keys (and from them, child public keys). An extended +public key is a public key and chain code, which can be used to create +child public keys (_public only_), as described in +<>. + +Think of an extended key as the root of a branch in the tree structure +of the HD wallet. With the root of the branch, you can derive the rest +of the branch. The extended private key can create a complete branch, +whereas the extended public key can _only_ create a branch of public +keys. + +Extended keys are encoded using base58check, to easily export and import +between different BIP32-compatible wallets. The base58check +coding for extended keys uses a special version number that results in +the prefix "xprv" and "xpub" when encoded in base58 characters to make +them easily recognizable. Because the extended key contains many more +bytes than regular addresses, +it is also much longer than other base58check-encoded strings we have +seen previously. + +Here's an example of an extended _private_ key, encoded in base58check: + +---- +xprv9tyUQV64JT5qs3RSTJkXCWKMyUgoQp7F3hA1xzG6ZGu6u6Q9VMNjGr67Lctvy5P8oyaYAL9CA +WrUE9i6GoNMKUga5biW6Hx4tws2six3b9c +---- + +Here's the corresponding extended _public_ key, encoded in base58check: + +---- +xpub67xpozcx8pe95XVuZLHXZeG6XWXHpGq6Qv5cmNfi7cS5mtjJ2tgypeQbBs2UAR6KECeeMVKZBP +LrtJunSDMstweyLXhRgPxdp14sk9tJPW9 +---- + +[role="less_space pagebreak-before"] +[[public__child_key_derivation]] +===== Public child key derivation + +As +mentioned previously, ((("key generation", "HD (hierarchical deterministic)", "public child key derivation", id="keygen-hd-public-child")))((("HD (hierarchical deterministic) key generation", "public child key derivation", id="hd-keygen-public-child")))((("public child key derivation", id="public-child")))((("child key pair derivation", "public keys", id="child-key-pair-public")))a very useful characteristic of HD wallets is the +ability to derive public child keys from public parent keys _without_ +having the private keys. This gives us two ways to derive a child public +key: either from the child private key or directly from the parent +public key. + +An extended public key can be used, therefore, to derive all of the +_public_ keys (and only the public keys) in that branch of the HD wallet +structure. + +This shortcut can be used to create public key–only +deployments where a server or application has a copy of an extended +public key and no private keys whatsoever. That kind of deployment can +produce an infinite number of public keys and Bitcoin addresses but +cannot spend any of the money sent to those addresses. Meanwhile, on +another, more secure server, the extended private key can derive all the +corresponding private keys to sign transactions and spend the money. + +One common application of this solution is to install an extended public +key on a web server that serves an ecommerce application. The web server +can use the public key derivation function to create a new Bitcoin +address for every transaction (e.g., for a customer shopping cart). The +web server will not have any private keys that would be vulnerable to +theft. Without HD wallets, the only way to do this is to generate +thousands of Bitcoin addresses on a separate secure server and then +preload them on the ecommerce server. That approach is cumbersome and +requires constant maintenance to ensure that the ecommerce server +doesn't "run out" of keys. + +.Mind the Gap +**** +An ((("gap limit", id="gap-limit")))extended public key can generate approximately 4 billion direct +child keys, far more than any store or application should ever need. +However, it would also take a wallet application an unreasonable amount +of time to generate all 4 billion keys and scan the blockchain for +transactions involving those keys. For that reason, most wallets only +generate a few keys at a time, scan for payments involving those keys, +and generate additional keys in the sequence as the previous keys are used. +For example, Alice's wallet generates 100 keys. When it sees a payment +to the first key, it generates the 101st key. + +Sometimes a wallet application will distribute a key to someone who +later decides not to pay, creating a gap in the key chain. That's fine as +long as the wallet has already generated keys after the gap so that it +finds later payments and continues generating more keys. The maximum +number of unused keys in a row that can fail to receive a payment +without causing problems is called the _gap limit_. + +When a wallet application has distributed all of the keys up to its gap +limit and none of those keys have received a payment, it has three +options about how to handle future requests for new keys: + +[role="less_space pagebreak-before"] +1. It can refuse the requests, preventing it from receiving any further +payments. This is obviously an unpalatable option, although it's the +simplest to implement. + +2. It can generate new keys beyond its gap limit. This ensures that +every person requesting to pay gets a unique key, preventing address +reuse and improving privacy. However, if the wallet needs to be +restored from a recovery code, or if the wallet owner is using other +software loaded with the same extended public key, those other wallets +won't see any payments received after the extended gap. + +3. It can distribute keys it previously distributed, ensuring a smooth +recovery but potentially reducing the privacy of the wallet owner and +the people with whom they transact. + +Open source production systems for online merchants, such as BTCPay +Server, attempt to dodge this problem by using very large gap limits and +limiting the rate at which they generate invoices. Other solutions have +been proposed, such as +asking the spender's wallet to construct (but not broadcast) a +transaction paying a possibly reused address before they receive a fresh +address for the actual transaction. However, these other solutions have +not been used in production as of this((("gap limit", startref="gap-limit"))) writing. +**** + +Another common application of this solution is for +cold-storage or hardware signing devices. In that scenario, the extended +private key can be stored on a paper wallet or hardware device, while +the extended public key can be kept online. The +user can create "receive" addresses at will, while the private keys are +safely stored offline. To spend the funds, the user can use the extended +private key on an offline software wallet application or +the hardware signing device. <> illustrates the +mechanism for extending a parent public key to derive child ((("wallets", "key generation", "HD (hierarchical deterministic)", startref="wallet-keygen-hd")))((("key generation", "HD (hierarchical deterministic)", startref="keygen-hd")))((("HD (hierarchical deterministic) key generation", startref="hd-keygen")))((("BIP32 HD (hierarchical deterministic) key generation", primary-sortas="BIP032", startref="bip32")))((("seeds", "HD wallet creation", startref="seed-hdwallet")))((("key generation", "HD (hierarchical deterministic)", "public child key derivation", startref="keygen-hd-public-child")))((("HD (hierarchical deterministic) key generation", "public child key derivation", startref="hd-keygen-public-child")))((("public child key derivation", startref="public-child")))((("child key pair derivation", "public keys", startref="child-key-pair-public")))public keys. + +[[CKDpub]] +.Extending a parent public key to create a child public key. +image::images/mbc3_0508.png["ChildPublicDerivation"] + +==== Using an Extended Public Key on a Web Store + +Let's see ((("key generation", "HD (hierarchical deterministic)", "extended keys", id="keygen-hd-extend-webstore")))((("HD (hierarchical deterministic) key generation", "extended keys", "web store example", id="hd-keygen-extend-webstore")))((("extended keys", "web store example", id="extend-key-webstore")))((("web store example (extended keys)", id="webstore-extend-key")))how HD wallets are used by looking at +Gabriel's web store. + +Gabriel first set up his web store as a hobby, based on a simple hosted +WordPress page. His store was quite basic with only a few pages and an +order form with a single Bitcoin address. + +Gabriel used the first Bitcoin address generated by his regular wallet as +the main Bitcoin address for his store. +Customers would submit an order using the form and send payment to +Gabriel's published Bitcoin address, triggering an email with the order +details for Gabriel to process. With just a few orders each week, this +system worked well enough, even though it weakened the privacy of +Gabriel, his clients, and the people he paid. + +However, the little web store became quite successful and attracted many +orders from the local community. Soon, Gabriel was overwhelmed. With all +the orders paying the same address, it became difficult to correctly +match orders and transactions, especially when multiple orders for the +same amount came in close together. + +The only metadata that is chosen by the receiver of a typical Bitcoin +transaction are the amount and payment address. There's no subject +or message field that can be used to hold a unique identifier invoice number. + +Gabriel's HD wallet offers a much better solution through the ability to +derive public child keys without knowing the private keys. Gabriel can +load an extended public key (xpub) on his website, which can be used to +derive a unique address for every customer order. The unique address +immediately improves privacy and also gives each order a unique +identifier that can be used for tracking which invoices have been paid. + +Using the HD wallet allows Gabriel to spend the +funds from his personal wallet application, but the xpub loaded on the website can only +generate addresses and receive funds. This feature of HD wallets is a +great security feature. Gabriel's website does not contain any private +keys and therefore any hack of it can only steal the funds Gabriel would +have received in the future, not any funds he received in the past. + +To export the xpub from his Trezor hardware signing device, Gabriel uses +the web-based Trezor wallet application. The Trezor device must be plugged in +for the public keys to be exported. Note that most hardware signing devices will +never export private keys--those always remain on the device. + +Gabriel copies the xpub to his web store's Bitcoin payment processing +software, such as the widely used open source BTCPay Server. + +===== Hardened child key derivation + +The ((("private child key derivation", "hardened derivation", id="private-child-harden")))((("child key pair derivation", "hardened derivation", id="child-key-pair-harden")))((("hardened child key derivation", id="harden-child-key")))ability to derive a branch +of public keys from an xpub is very useful, but it comes with a +potential risk. Access to an xpub does not give access to child private +keys. However, because the xpub contains the chain code, if a child +private key is known, or somehow leaked, it can be used with the chain +code to derive all the other child private keys. A single leaked child +private key, together with a parent chain code, reveals all the private +keys of all the children. Worse, the child private key together with a +parent chain code can be used to deduce the parent private key. + +To counter this risk, HD wallets provide an alternative derivation function +called _hardened derivation_, which breaks the relationship between +parent public key and child chain code. The hardened derivation function +uses the parent private key to derive the child chain code, instead of +the parent public key. This creates a "firewall" in the parent/child +sequence, with a chain code that cannot be used to compromise a parent +or sibling private key. The hardened derivation function looks almost +identical to the normal child private key derivation, except that the +parent private key is used as input to the hash function, instead of the +parent public key, as shown in the diagram in <>. + +[[CKDprime]] +.Hardened derivation of a child key; omits the parent public key. +image::images/mbc3_0509.png["ChildHardPrivateDerivation"] + + +When the hardened private derivation function is used, the resulting +child private key and chain code are completely different from what +would result from the normal derivation function. The resulting "branch" +of keys can be used to produce extended public keys that are not +vulnerable because the chain code they contain cannot be exploited to +reveal any private keys for their siblings or parents. Hardened derivation is therefore used to create +a "gap" in the tree above the level where extended public keys are used. + +In simple terms, if you want to use the convenience of an xpub to derive +branches of public keys, without exposing yourself to the risk of a +leaked chain code, you should derive it from a hardened parent rather +than a normal parent. As a best practice, the level-1 children of the +master keys are always derived through the hardened derivation to +prevent compromise of the master keys. + +===== Index numbers for normal and hardened derivation + +The index number ((("index numbers for hardened derivation")))used in the derivation function is a 32-bit integer. To +easily distinguish between keys created through the normal derivation +function versus keys derived through hardened derivation, this index +number is split into two ranges. Index numbers between 0 and +2^31^ – 1 (0x0 to 0x7FFFFFFF) are used _only_ for normal +derivation. Index numbers between 2^31^ and 2^32^ – 1 (0x80000000 +to 0xFFFFFFFF) are used _only_ for hardened derivation. Therefore, if +the index number is less than 2^31^, the child is normal, whereas if the +index number is equal or above 2^31^, the child is hardened. + +To make the index number easier to read and display, the index number +for hardened children is displayed starting from zero, but with a prime +symbol. The first normal child key is therefore displayed as 0, whereas +the first hardened child (index 0x80000000) is displayed as 0++'++. +In a sequence then, the second hardened key would have index 0x80000001 +and would be displayed as 1++'++, and so on. When you see an HD +wallet index i++'++, that means 2^31^+i. In regular ASCII text, the +prime symbol is substituted with either a single apostrophe or the +letter _h_. For situations, such as in output script descriptors, where +text may be used in a shell or other context where a single apostrophe +has special meaning, using the letter _h_ is ((("private child key derivation", "hardened derivation", startref="private-child-harden")))((("child key pair derivation", "hardened derivation", startref="child-key-pair-harden")))((("hardened child key derivation", startref="harden-child-key")))recommended. + +===== HD wallet key identifier (path) + +Keys in ((("path references in HD wallets")))((("key generation", "HD (hierarchical deterministic)", "path references")))((("HD (hierarchical deterministic) key generation", "path references")))an HD wallet are +identified using a "path" naming convention, with each level of the tree +separated by a slash (/) character (see <>). Private keys +derived from the master private key start with "m." Public keys derived +from the master public key start with "M." Therefore, the first child +private key of the master private key is m/0. The first child public key +is M/0. The second grandchild of the first child is m/0/1, and so on. + +The "ancestry" of a key is read from right to left, until you reach the +master key from which it was derived. For example, identifier m/x/y/z +describes the key that is the z-th child of key m/x/y, which is the y-th +child of key m/x, which is the x-th child of m. + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
HD wallet path examples
HD pathKey described

m/0

The first (0) child private key from the master private key (m)

m/0/0

The first grandchild private key from the first child (m/0)

m/0'/0

The first normal grandchild private key from the first hardened child (m/0')

m/1/0

The first grandchild private key from the second child (m/1)

M/23/17/0/0

The first great-great-grandchild public key from the first great-grandchild from the 18th grandchild from the 24th child

+++++ + +===== Navigating the HD wallet tree structure + +The((("tree structure in HD wallets", id="tree-hd-wallet")))((("key generation", "HD (hierarchical deterministic)", "tree structure", id="keygen-hd-tree")))((("HD (hierarchical deterministic) key generation", "tree structure", id="hd-keygen-tree"))) HD wallet tree structure offers tremendous flexibility. Each parent +extended key can have 4 billion children: 2 billion normal children and +2 billion hardened children. Each of those children can have another 4 +billion children, and so on. The tree can be as deep as you want, with +an infinite number of generations. With all that flexibility, however, +it becomes quite difficult to navigate this infinite tree. It is +especially difficult to transfer HD wallets between implementations +because the possibilities for internal organization into branches and +subbranches are endless. + +Two BIPs offer a solution to this complexity by creating some proposed +standards for the structure of HD wallet trees. ((("BIP43 HD wallet tree structure", primary-sortas="BIP043")))BIP43 proposes the use +of the first hardened child index as a special identifier that signifies +the "purpose" of the tree structure. Based on BIP43, an HD wallet +should use only one level-1 branch of the tree, with the index number +identifying the structure and namespace of the rest of the tree by +defining its purpose. For example, an HD wallet using only branch +m/i++'++/ is intended to signify a specific purpose, and that +purpose is identified by index number "i." + +Extending that specification, ((("BIP44 HD wallet tree structure", primary-sortas="BIP044", id="bip44")))BIP44 proposes a multiaccount structure +as "purpose" number +44'+ under BIP43. All HD wallets following the +BIP44 structure are identified by the fact that they only used one +branch of the tree: m/44++'++/. + +BIP44 specifies the structure as consisting of five predefined tree levels: + +----- +m / purpose' / coin_type' / account' / change / address_index +----- + + +The first-level "purpose" is always set to ++44'++. The second-level +"coin_type" specifies the type of cryptocurrency coin, allowing for +multicurrency HD wallets where each currency has its own subtree under +the second level. Bitcoin is m/44++'++/0++'++ and Bitcoin Testnet is m/44++'++/1++'++. + +The third level of the tree is "account," which allows users to +subdivide their wallets into separate logical subaccounts for +accounting or organizational purposes. For example, an HD wallet might +contain two Bitcoin "accounts": m/44++'++/0++'++/0++'++ +and m/44++'++/0++'++/1++'++. Each account is the root of +its own subtree. + +On the +fourth level, "change," an HD wallet has two subtrees, one for creating +receiving addresses and one for creating change addresses. Note that +whereas the previous levels used hardened derivation, this level uses +normal derivation. This is to allow this level of the tree to export +extended public keys for use in a nonsecured environment. Usable +addresses are derived by the HD wallet as children of the fourth level, +making the fifth level of the tree the "address_index." For example, the +third receiving address for payments in the primary account +would be M/44++'++/0++'++/0++'++/0/2. <> shows +a few more examples. + +++++ + + + + + + + + + + + + + + + + + + + + + + +
BIP44 HD wallet structure examples
HD pathKey described

M/44'/0'/0'/0/2

The third receiving public key for the primary Bitcoin account

M/44'/0'/3'/1/14

The fifteenth change-address public key for the fourth Bitcoin account

m/44'/2'/0'/0/1

The second private key in the Litecoin main account, for signing transactions

+++++ + +Many people ((("BIP44 HD wallet tree structure", primary-sortas="BIP044", startref="bip44")))focus on securing their bitcoins against theft and other +attacks, but one of the leading causes of lost bitcoins--perhaps _the_ +leading cause--is data loss. If the keys and other essential data +required to spend your bitcoins is lost, those bitcoins will forever be +unspendable. Nobody can get them back for you. In this chapter, we +looked at the systems that modern wallet applications use to help you +prevent losing that data. Remember, however, that it's up to you to +actually use the systems available to make good backups and ((("key generation", "HD (hierarchical deterministic)", "extended keys", startref="keygen-hd-extend-webstore")))((("HD (hierarchical deterministic) key generation", "extended keys", "web store example", startref="hd-keygen-extend-webstore")))((("extended keys", "web store example", startref="extend-key-webstore")))((("web store example (extended keys)", startref="webstore-extend-key")))((("tree structure in HD wallets", startref="tree-hd-wallet")))((("key generation", "HD (hierarchical deterministic)", "tree structure", startref="keygen-hd-tree")))((("HD (hierarchical deterministic) key generation", "tree structure", startref="hd-keygen-tree")))regularly +test them. diff --git a/ch06.asciidoc b/ch06.asciidoc deleted file mode 100644 index f3d82f95..00000000 --- a/ch06.asciidoc +++ /dev/null @@ -1,759 +0,0 @@ -[[ch06]] -[[transactions]] -== Transactions - -[[ch06_intro]] -=== Introduction - -((("transactions", "defined")))((("warnings and cautions", see="also security")))Transactions are the most important part of the Bitcoin system. Everything else in bitcoin is designed to ensure that transactions can be created, propagated on the network, validated, and finally added to the global ledger of transactions (the blockchain). Transactions are data structures that encode the transfer of value between participants in the Bitcoin system. Each transaction is a public entry in bitcoin's blockchain, the global double-entry bookkeeping ledger. - -In this chapter we will examine all the various forms of transactions, what they contain, how to create them, how they are verified, and how they become part of the permanent record of all transactions. When we use the term "wallet" in this chapter, we are referring to the software that constructs transactions, not just the database of keys. - -[[tx_structure]] -=== Transactions in Detail - -((("use cases", "buying coffee", id="alicesix")))In <>, we looked at the transaction Alice used to pay for coffee at Bob's coffee shop using a block explorer (<>). - -The block explorer application shows a transaction from Alice's "address" to Bob's "address." This is a much simplified view of what is contained in a transaction. In fact, as we will see in this chapter, much of the information shown is constructed by the block explorer and is not actually in the transaction. - -[[alices_transactions_to_bobs_cafe]] -.Alice's transaction to Bob's Cafe -image::images/mbc2_0208.png["Alice Coffee Transaction"] - -[[transactions_behind_the_scenes]] -==== Transactions—Behind the Scenes - -((("transactions", "behind the scenes details of")))Behind the scenes, an actual transaction looks very different from a transaction provided by a typical block explorer. In fact, most of the high-level constructs we see in the various bitcoin application user interfaces _do not actually exist_ in the Bitcoin system. - -We can use Bitcoin Core's command-line interface (+getrawtransaction+ and +decoderawtransaction+) to retrieve Alice's "raw" transaction, decode it, and see what it contains. The result looks like this: - -[[alice_tx]] -.Alice's transaction decoded -[source,json] ----- -{ - "version": 1, - "locktime": 0, - "vin": [ - { - "txid": "7957a35fe64f80d234d76d83a2a8f1a0d8149a41d81de548f0a65a8a999f6f18", - "vout": 0, - "scriptSig" : "3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813[ALL] 0484ecc0d46f1918b30928fa0e4ed99f16a0fb4fde0735e7ade8416ab9fe423cc5412336376789d172787ec3457eee41c04f4938de5cc17b4a10fa336a8d752adf", - "sequence": 4294967295 - } - ], - "vout": [ - { - "value": 0.01500000, - "scriptPubKey": "OP_DUP OP_HASH160 ab68025513c3dbd2f7b92a94e0581f5d50f654e7 OP_EQUALVERIFY OP_CHECKSIG" - }, - { - "value": 0.08450000, - "scriptPubKey": "OP_DUP OP_HASH160 7f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a8 OP_EQUALVERIFY OP_CHECKSIG", - } - ] -} ----- - -You may notice a few things about this transaction, mostly the things that are missing! Where is Alice's address? Where is Bob's address? Where is the 0.1 input "sent" by Alice? In bitcoin, there are no coins, no senders, no recipients, no balances, no accounts, and no addresses. All those things are constructed at a higher level for the benefit of the user, to make things easier to understand. - -You may also notice a lot of strange and indecipherable fields and hexadecimal strings. Don't worry, we will explain each field shown here in detail in this chapter. - -[[tx_inputs_outputs]] -=== Transaction Outputs and Inputs - -((("transactions", "outputs and inputs", id="Tout06")))((("outputs and inputs", "outputs defined")))((("unspent transaction outputs (UTXO)")))((("UTXO sets")))((("transactions", "outputs and inputs", "output characteristics")))((("outputs and inputs", "output characteristics")))The fundamental building block of a bitcoin transaction is a _transaction output_. Transaction outputs are indivisible chunks of bitcoin currency, recorded on the blockchain, and recognized as valid by the entire network. Bitcoin full nodes track all available and spendable outputs, known as _unspent transaction outputs_, or _UTXO_. The collection of all UTXO is known as the _UTXO set_ and currently numbers in the millions of UTXO. The UTXO set grows as new UTXO is created and shrinks when UTXO is consumed. Every transaction represents a change (state transition) in the UTXO set. - -((("balances")))When we say that a user's wallet has "received" bitcoin, what we mean is that the wallet has detected on the blockchain an UTXO that can be spent with one of the keys controlled by that wallet. Thus, a user's bitcoin "balance" is the sum of all UTXO that user's wallet can spend and which may be scattered among hundreds of transactions and hundreds of blocks. The concept of a balance is created by the wallet application. The wallet calculates the user's balance by scanning the blockchain and aggregating the value of any UTXO the wallet can spend with the keys it controls. Most wallets maintain a database or use a database service to store a quick reference set of all the UTXO they can spend with the keys they control. - -((("spent transaction outputs (STXO)")))<> displays the blockchain in three different moments, as the transaction chain from Joe to Gopesh is being built. Notice how each transaction spends an UTXO that was created in a previous transaction, turning it into a _spent transaction output_, or _STXO_. Since transaction #1 (from Joe to Alice) spends a single UTXO (from Joe) and creates a single UTXO (to Alice), it doesn't modify the UTXO set size. On the other hand, transactions #2 and #3 both create change outputs to the sender, spending a single UTXO and creating two UTXO (the payment and the change output). Therefore, each of them increases the UTXO set size by 1. - -[[utxo-stxo]] -.Transaction chain from Joe to Gopesh being built on the blockchain -image::images/mbc2_0609.png["Transaction chain from Joe to Gopesh being built on the blockchain"] - -((("satoshis")))A transaction output can have an arbitrary (integer) value denominated as a multiple of satoshis. Just as dollars can be divided down to two decimal places as cents, bitcoin can be divided down to eight decimal places as satoshis. Although an output can have any arbitrary value, once created it is indivisible. This is an important characteristic of outputs that needs to be emphasized: outputs are _discrete_ and _indivisible_ units of value, denominated in integer satoshis. An unspent output can only be consumed in its entirety by a transaction. - -((("change, making")))If an UTXO is larger than the desired value of a transaction, it must still be consumed in its entirety and change must be generated in the transaction. In other words, if you have an UTXO worth 20 bitcoin and want to pay only 1 bitcoin, your transaction must consume the entire 20-bitcoin UTXO and produce two outputs: one paying 1 bitcoin to your desired recipient and another paying 19 bitcoin in change back to your wallet. As a result of the indivisible nature of transaction outputs, most bitcoin transactions will have to generate change. - -Imagine a shopper buying a $1.50 beverage, reaching into her wallet and trying to find a combination of coins and bank notes to cover the $1.50 cost. The shopper will choose exact change if available e.g. a dollar bill and two quarters (a quarter is $0.25), or a combination of smaller denominations (six quarters), or if necessary, a larger unit such as a $5 note. If she hands too much money, say $5, to the shop owner, she will expect $3.50 change, which she will return to her wallet and have available for future transactions. - -Similarly, a bitcoin transaction must be created from a user's UTXO in whatever denominations that user has available. Users cannot cut an UTXO in half any more than they can cut a dollar bill in half and use it as currency. The user's wallet application will typically select from the user's available UTXO to compose an amount greater than or equal to the desired transaction amount. - -As with real life, the bitcoin application can use several strategies to satisfy the purchase amount: combining several smaller units, finding exact change, or using a single unit larger than the transaction value and making change. All of this complex assembly of spendable UTXO is done by the user's wallet automatically and is invisible to users. It is only relevant if you are programmatically constructing raw transactions from UTXO. - -A transaction consumes previously recorded unspent transaction outputs and creates new transaction outputs that can be consumed by a future transaction. This way, chunks of bitcoin value move forward from owner to owner in a chain of transactions consuming and creating UTXO. - -((("transactions", "coinbase transactions")))((("coinbase transactions")))((("mining and consensus", "coinbase transactions")))The exception to the output and input chain is a special type of transaction called the _coinbase_ transaction, which is the first transaction in each block. This transaction is placed there by the "winning" miner and creates brand-new bitcoin payable to that miner as a reward for mining. This special coinbase transaction does not consume UTXO; instead, it has a special type of input called the "coinbase." This is how bitcoin's money supply is created during the mining process, as we will see in <>. - -[TIP] -==== -What comes first? Inputs or outputs, the chicken or the egg? Strictly speaking, outputs come first because coinbase transactions, which generate new bitcoin, have no inputs and create outputs from nothing. -==== - -[[tx_outs]] -==== Transaction Outputs - -((("transactions", "outputs and inputs", "output components")))((("outputs and inputs", "output parts")))Every bitcoin transaction creates outputs, which are recorded on the bitcoin ledger. Almost all of these outputs, with one exception (see <>) create spendable chunks of bitcoin called UTXO, which are then recognized by the whole network and available for the owner to spend in a future transaction. - -UTXO are tracked by every full-node Bitcoin client in the UTXO set. New transactions consume (spend) one or more of these outputs from the UTXO set. - -Transaction outputs consist of two parts: - -* An amount of bitcoin, denominated in _satoshis_, the smallest bitcoin unit -* A cryptographic puzzle that determines the conditions required to spend the output - -((("locking scripts")))((("scripting", "locking scripts")))((("witnesses")))((("scriptPubKey")))The cryptographic puzzle is also known as a _locking script_, a _witness script_, or a +scriptPubKey+. - -The transaction scripting language, used in the locking script mentioned previously, is discussed in detail in <>. - -Now, let's look at Alice's transaction (shown previously in <>) and see if we can identify the outputs. In the JSON encoding, the outputs are in an array (list) named +vout+: - -[source,json] ----- -"vout": [ - { - "value": 0.01500000, - "scriptPubKey": "OP_DUP OP_HASH160 ab68025513c3dbd2f7b92a94e0581f5d50f654e7 OP_EQUALVERIFY - OP_CHECKSIG" - }, - { - "value": 0.08450000, - "scriptPubKey": "OP_DUP OP_HASH160 7f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a8 OP_EQUALVERIFY OP_CHECKSIG", - } -] ----- - -As you can see, the transaction contains two outputs. Each output is defined by a value and a cryptographic puzzle. In the encoding shown by Bitcoin Core, the value is shown in bitcoin, but in the transaction itself it is recorded as an integer denominated in satoshis. The second part of each output is the cryptographic puzzle that sets the conditions for spending. Bitcoin Core shows this as +scriptPubKey+ and shows us a human-readable representation of the script. - -The topic of locking and unlocking UTXO will be discussed later, in <>. The scripting language that is used for the script in +scriptPubKey+ is discussed in <>. But before we delve into those topics, we need to understand the overall structure of transaction inputs and outputs. - -===== Transaction serialization—outputs - -((("transactions", "outputs and inputs", "structure of")))((("outputs and inputs", "structure of")))((("serialization", "outputs")))When transactions are transmitted over the network or exchanged between applications, they are _serialized_. Serialization is the process of converting the internal representation of a data structure into a format that can be transmitted one byte at a time, also known as a byte stream. Serialization is most commonly used for encoding data structures for transmission over a network or for storage in a file. The serialization format of a transaction output is shown in <>. - -[[tx_out_structure]] -.Transaction output serialization -[options="header"] -|======= -|Size| Field | Description -| 8 bytes (little-endian) | Amount | Bitcoin value in satoshis (10^-8^ bitcoin) -| 1–9 bytes (VarInt) | Locking-Script Size | Locking-Script length in bytes, to follow -| Variable | Locking-Script | A script defining the conditions needed to spend the output -|======= - -Most bitcoin libraries and frameworks do not store transactions internally as byte-streams, as that would require complex parsing every time you needed to access a single field. For convenience and readability, bitcoin libraries store transactions internally in data structures (usually object-oriented structures). - -((("deserialization")))((("parsing")))((("transactions", "parsing")))The process of converting from the byte-stream representation of a transaction to a library's internal representation data structure is called _deserialization_ or _transaction parsing_. The process of converting back to a byte-stream for transmission over the network, for hashing, or for storage on disk is called _serialization_. Most bitcoin libraries have built-in functions for transaction serialization and deserialization. - -See if you can manually decode Alice's transaction from the serialized hexadecimal form, finding some of the elements we saw previously. The section containing the two outputs is highlighted in <> to help you: - -[[example_6_1]] -.Alice's transaction, serialized and presented in hexadecimal notation -==== -+0100000001186f9f998a5aa6f048e51dd8419a14d8a0f1a8a2836dd73+ -+4d2804fe65fa35779000000008b483045022100884d142d86652a3f47+ -+ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039+ -+ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813+ -+01410484ecc0d46f1918b30928fa0e4ed99f16a0fb4fde0735e7ade84+ -+16ab9fe423cc5412336376789d172787ec3457eee41c04f4938de5cc1+ -+7b4a10fa336a8d752adfffffffff02+*+60e31600000000001976a914ab6+* -*+8025513c3dbd2f7b92a94e0581f5d50f654e788acd0ef800000000000+* -*+1976a9147f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a888ac+* -+00000000+ -==== - -Here are some hints: - -* There are two outputs in the highlighted section, each serialized as shown in <>. -* The value of 0.015 bitcoin is 1,500,000 satoshis. That's +16 e3 60+ in hexadecimal. -* In the serialized transaction, the value +16 e3 60+ is encoded in little-endian (least-significant-byte-first) byte order, so it looks like +60 e3 16+. -* The +scriptPubKey+ length is 25 bytes, which is +19+ in hexadecimal. - -[[tx_inputs]] -==== Transaction Inputs - -((("transactions", "outputs and inputs", "input components")))((("outputs and inputs", "input components")))((("unspent transaction outputs (UTXO)")))((("UTXO sets")))Transaction inputs identify (by reference) which UTXO will be consumed and provide proof of ownership through an unlocking script. - -To build a transaction, a wallet selects from the UTXO it controls, UTXO with enough value to make the requested payment. Sometimes one UTXO is enough, other times more than one is needed. For each UTXO that will be consumed to make this payment, the wallet creates one input pointing to the UTXO and unlocks it with an unlocking script. - -Let's look at the components of an input in greater detail. The first part of an input is a pointer to an UTXO by reference to the transaction hash and an output index, which identifies the specific UTXO in that transaction. The second part is an unlocking script, which the wallet constructs in order to satisfy the spending conditions set in the UTXO. Most often, the unlocking script is a digital signature and public key proving ownership of the bitcoin. However, not all unlocking scripts contain signatures. The third part is a sequence number, which will be discussed later. - -Consider our example in <>. The transaction inputs are an array (list) called +vin+: - -[[vin]] -.The transaction inputs in Alice's transaction -[source,json] ----- -"vin": [ - { - "txid": "7957a35fe64f80d234d76d83a2a8f1a0d8149a41d81de548f0a65a8a999f6f18", - "vout": 0, - "scriptSig" : "3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813[ALL] 0484ecc0d46f1918b30928fa0e4ed99f16a0fb4fde0735e7ade8416ab9fe423cc5412336376789d172787ec3457eee41c04f4938de5cc17b4a10fa336a8d752adf", - "sequence": 4294967295 - } -] ----- - -As you can see, there is only one input in the list (because one UTXO contained sufficient value to make this payment). The input contains four elements: - -* A ((("transaction IDs (txd)")))transaction ID, referencing the transaction that contains the UTXO being spent -* An output index (+vout+), identifying which UTXO from that transaction is referenced (first one is zero) -* A +scriptSig+, which satisfies the conditions placed on the UTXO, unlocking it for spending -* A sequence number (to be discussed later) - -In Alice's transaction, the input points to the transaction ID: - ----- -7957a35fe64f80d234d76d83a2a8f1a0d8149a41d81de548f0a65a8a999f6f18 ----- - -and output index +0+ (i.e., the first UTXO created by that transaction). The unlocking script is constructed by Alice's wallet by first retrieving the referenced UTXO, examining its locking script, and then using it to build the necessary unlocking script to satisfy it. - -Looking just at the input you may have noticed that we don't know anything about this UTXO, other than a reference to the parent transaction containing it. We don't know its value (amount in satoshi), and we don't know the locking script that sets the conditions for spending it. To find this information, we must retrieve the referenced UTXO by retrieving the parent transaction that contains it. Notice that because the value of the input is not explicitly stated, we must also use the referenced UTXO in order to calculate the fees that will be paid in this transaction (see <>). - -It's not just Alice's wallet that needs to retrieve UTXO referenced in the inputs. Once this transaction is broadcast to the network, every validating node will also need to retrieve the UTXO referenced in the transaction inputs in order to validate the transaction. - -Transactions on their own seem incomplete because they lack context. They reference UTXO in their inputs but without retrieving that UTXO we cannot know the value of the inputs or their locking conditions. When writing bitcoin software, anytime you decode a transaction with the intent of validating it or counting the fees or checking the unlocking script, your code will first have to retrieve the referenced UTXO from the blockchain in order to build the context implied but not present in the UTXO references of the inputs. For example, to calculate the amount paid in fees, you must know the sum of the values of inputs and outputs. But without retrieving the UTXO referenced in the inputs, you do not know their value. So a seemingly simple operation like counting fees in a single transaction in fact involves multiple steps and data from multiple transactions. - -We can use the same sequence of commands with Bitcoin Core as we used when retrieving Alice's transaction (+getrawtransaction+ and +decoderawtransaction+). With that we can get the UTXO referenced in the input from Alice's transaction and take a look: - -[[alice_input_tx]] -.UTXO from the previous transaction, referenced in the input from Alice's transaction -[source,json] ----- -"vout": [ - { - "value": 0.10000000, - "scriptPubKey": "OP_DUP OP_HASH160 7f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a8 OP_EQUALVERIFY OP_CHECKSIG" - } - ] ----- - -We see that this UTXO has a value of 0.1 BTC and that it has a locking script (+scriptPubKey+) that contains "OP_DUP OP_HASH160...". - -[TIP] -==== -To fully understand Alice's transaction we had to retrieve the previous transaction referenced as input. A function that retrieves previous transactions and unspent transaction outputs is very common and exists in almost every bitcoin library and API. -==== - -===== Transaction serialization—inputs - -((("serialization", "inputs")))((("transactions", "outputs and inputs", "input serialization")))((("outputs and inputs", "input serialization")))When transactions are serialized for transmission on the network, their inputs are encoded into a byte stream as shown in <>. - -[[tx_in_structure]] -.Transaction input serialization -[options="header"] -|======= -|Size| Field | Description -| 32 bytes | Transaction Hash | Pointer to the transaction containing the UTXO to be spent -| 4 bytes | Output Index | The index number of the UTXO to be spent; first one is 0 -| 1–9 bytes (VarInt) | Unlocking-Script Size | Unlocking-Script length in bytes, to follow -| Variable | Unlocking-Script | A script that fulfills the conditions of the UTXO locking script -| 4 bytes | Sequence Number | Used for locktime or disabled (0xFFFFFFFF) -|======= - -As with the outputs, let's see if we can find the inputs from Alice's transaction in the serialized format. First, the inputs decoded: - -[source,json] ----- -"vin": [ - { - "txid": "7957a35fe64f80d234d76d83a2a8f1a0d8149a41d81de548f0a65a8a999f6f18", - "vout": 0, - "scriptSig" : "3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813[ALL] 0484ecc0d46f1918b30928fa0e4ed99f16a0fb4fde0735e7ade8416ab9fe423cc5412336376789d172787ec3457eee41c04f4938de5cc17b4a10fa336a8d752adf", - "sequence": 4294967295 - } -], ----- - -Now, let's see if we can identify these fields in the serialized hex encoding in <>: - -[[example_6_2]] -.Alice's transaction, serialized and presented in hexadecimal notation -==== -+0100000001+*+186f9f998a5aa6f048e51dd8419a14d8a0f1a8a2836dd73+* -*+4d2804fe65fa35779000000008b483045022100884d142d86652a3f47+* -*+ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039+* -*+ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813+* -*+01410484ecc0d46f1918b30928fa0e4ed99f16a0fb4fde0735e7ade84+* -*+16ab9fe423cc5412336376789d172787ec3457eee41c04f4938de5cc1+* -*+7b4a10fa336a8d752adfffffffff+*+0260e31600000000001976a914ab6+ -+8025513c3dbd2f7b92a94e0581f5d50f654e788acd0ef800000000000+ -+1976a9147f9b1a7fb68d60c536c2fd8aeaa53a8f3cc025a888ac00000+ -+000+ -==== - -Hints: - -* The transaction ID is serialized in reversed byte order, so it starts with (hex) +18+ and ends with +79+ -* The output index is a 4-byte group of zeros, easy to identify -* The length of the +scriptSig+ is 139 bytes, or +8b+ in hex -* The sequence number is set to +FFFFFFFF+, again easy to identify((("", startref="alicesix"))) - -ScriptSig is a specific type of unlocking script that when serialized for transmission on the network, inputs are encoded into a byte stream as shown in <>. The serialization of the signature field is detailed in <>. The signature field also includes a Signature Hash Type (SIGHASH), which is detailed in <>. - -[[scriptsig_in_structure]] -.ScriptSig input serialization -[options="header"] -|======= -|Size| Field | Description -| 1–9 bytes (VarInt) | Signature Size | Signature length in bytes, to follow -| Variable | Signature | A signature that is produced by the user’s wallet from his or her private key, which includes a SIGHASH -| 1–9 bytes (VarInt) | Public Key Size | Public key length in bytes, to follow -| Variable | Public Key | The public key, unhashed -|======= - -[[tx_fees]] -==== Transaction Fees - -((("transactions", "outputs and inputs", "transaction fees")))((("fees", "transaction fees")))((("mining and consensus", "rewards and fees")))Most transactions include transaction fees, which compensate the bitcoin miners for securing the network. Fees also serve as a security mechanism themselves, by making it economically infeasible for attackers to flood the network with transactions. Mining and the fees and rewards collected by miners are discussed in more detail in <>. - -This section examines how transaction fees are included in a typical transaction. Most wallets calculate and include transaction fees automatically. However, if you are constructing transactions programmatically, or using a command-line interface, you must manually account for and include these fees. - -Transaction fees serve as an incentive to include (mine) a transaction into the next block and also as a disincentive against abuse of the system by imposing a small cost on every transaction. Transaction fees are collected by the miner who mines the block that records the transaction on the blockchain. - -Transaction fees are calculated based on the size of the transaction in kilobytes, not the value of the transaction in bitcoin. Overall, transaction fees are set based on market forces within the Bitcoin network. Miners prioritize transactions based on many different criteria, including fees, and might even process transactions for free under certain circumstances. Transaction fees affect the processing priority, meaning that a transaction with sufficient fees is likely to be included in the next block mined, whereas a transaction with insufficient or no fees might be delayed, processed on a best-effort basis after a few blocks, or not processed at all. Transaction fees are not mandatory, and transactions without fees might be processed eventually; however, including transaction fees encourages priority processing. - -Over time, the way transaction fees are calculated and the effect they have on transaction prioritization has evolved. At first, transaction fees were fixed and constant across the network. Gradually, the fee structure relaxed and may be influenced by market forces, based on network capacity and transaction volume. Since at least the beginning of 2016, capacity limits in bitcoin have created competition between transactions, resulting in higher fees and effectively making free transactions a thing of the past. Zero fee or very low fee transactions rarely get mined and sometimes will not even be propagated across the network. - -((("fees", "fee relay policies")))((("minrelaytxfee option")))In Bitcoin Core, fee relay policies are set by the +minrelaytxfee+ option. The current default +minrelaytxfee+ is 0.00001 bitcoin or a hundredth of a millibitcoin per kilobyte. Therefore, by default, transactions with a fee less than 0.00001 bitcoin are treated as free and are only relayed if there is space in the mempool; otherwise, they are dropped. Bitcoin nodes can override the default fee relay policy by adjusting the value of +minrelaytxfee+. - -((("dynamic fees")))((("fees", "dynamic fees")))Any bitcoin service that creates transactions, including wallets, exchanges, retail applications, etc., _must_ implement dynamic fees. Dynamic fees can be implemented through a third-party fee estimation service or with a built-in fee estimation algorithm. If you're unsure, begin with a third-party service and as you gain experience design and implement your own algorithm if you wish to remove the third-party dependency. - -Fee estimation algorithms calculate the appropriate fee, based on capacity and the fees offered by "competing" transactions. These algorithms range from simplistic (average or median fee in the last block) to sophisticated (statistical analysis). They estimate the necessary fee (in satoshis per byte) that will give a transaction a high probability of being selected and included within a certain number of blocks. Most services offer users the option of choosing high, medium, or low priority fees. High priority means users pay higher fees but the transaction is likely to be included in the next block. Medium and low priority means users pay lower transaction fees but the transactions may take much longer to confirm. - -((("bitcoinfees (third-party service)")))Many wallet applications use third-party services for fee calculations. One popular service is https://bitcoinfees.earn.com/[_https://bitcoinfees.earn.com/_], which provides an API and a visual chart showing the fee in satoshi/byte for different priorities. - -[TIP] -==== -((("static fees")))((("fees", "static fees")))Static fees are no longer viable on the Bitcoin network. Wallets that set static fees will produce a poor user experience as transactions will often get "stuck" and remain unconfirmed. Users who don't understand bitcoin transactions and fees are dismayed by "stuck" transactions because they think they've lost their money. -==== - -The chart in <> shows the real-time estimate of fees in 10 satoshi/byte increments and the expected confirmation time (in minutes and number of blocks) for transactions with fees in each range. For each fee range (e.g., 61–70 satoshi/byte), two horizontal bars show the number of unconfirmed transactions (1405) and total number of transactions in the past 24 hours (102,975), with fees in that range. Based on the graph, the recommended high-priority fee at this time was 80 satoshi/byte, a fee likely to result in the transaction being mined in the very next block (zero block delay). For perspective, the median transaction size is 226 bytes, so the recommended fee for this transaction size would be 18,080 satoshis (0.00018080 BTC). - -The fee estimation data can be retrieved via a simple HTTP REST API, at https://bitcoinfees.earn.com/api/v1/fees/recommended[https://bitcoinfees.earn.com/api/v1/fees/recommended]. For example, on the command line using the +curl+ command: - -.Using the fee estimation API ----- -$ curl https://bitcoinfees.earn.com/api/v1/fees/recommended - -{"fastestFee":80,"halfHourFee":80,"hourFee":60} ----- - -The API returns a JSON object with the current fee estimate for fastest confirmation (+fastestFee+), confirmation within three blocks (+halfHourFee+) and six blocks (+hourFee+), in satoshi per byte. - -[[bitcoinfeesearncom]] -.Fee estimation service bitcoinfees.earn.com -image::images/mbc2_0602.png[Fee Estimation Service bitcoinfees.earn.com] - -==== Adding Fees to Transactions - -The data structure of transactions does not have a field for fees. Instead, fees are implied as the difference between the sum of inputs and the sum of outputs. Any excess amount that remains after all outputs have been deducted from all inputs is the fee that is collected by the miners: - -[[tx_fee_equation]] -.Transaction fees are implied, as the excess of inputs minus outputs: ----- -Fees = Sum(Inputs) – Sum(Outputs) ----- - -This is a somewhat confusing element of transactions and an important point to understand, because if you are constructing your own transactions you must ensure you do not inadvertently include a very large fee by underspending the inputs. That means that you must account for all inputs, if necessary by creating change, or you will end up giving the miners a very big tip! - -For example, if you consume a 20-bitcoin UTXO to make a 1-bitcoin payment, you must include a 19-bitcoin change output back to your wallet. Otherwise, the 19-bitcoin "leftover" will be counted as a transaction fee and will be collected by the miner who mines your transaction in a block. Although you will receive priority processing and make a miner very happy, this is probably not what you intended. - -[WARNING] -==== -((("warnings and cautions", "change outputs")))If you forget to add a change output in a manually constructed transaction, you will be paying the change as a transaction fee. Saying "Keep the change!" to the miner might not be what you really intended. -==== - -((("use cases", "buying coffee")))Let's see how this works in practice, by looking at Alice's coffee purchase again. Alice wants to spend 0.015 bitcoin to pay for coffee. To ensure this transaction is processed promptly, she will want to include a transaction fee, say 0.0005. That will mean that the total cost of the transaction will be 0.0155. Her wallet must therefore source a set of UTXO that adds up to 0.0155 bitcoin or more and, if necessary, create change. Let's say her wallet has a 0.1-bitcoin UTXO available. It will therefore need to consume this UTXO, create one output to Bob's Cafe for 0.015, and a second output with 0.0845 bitcoin in change back to her own wallet, leaving 0.0005 bitcoin unallocated, as an implicit fee for the transaction. - -((("use cases", "charitable donations")))((("charitable donations")))Now let's look at a different scenario. Eugenia, our children's charity director in the Philippines, has completed a fundraiser to purchase schoolbooks for the children. She received several thousand small donations from people all around the world, totaling 50 bitcoin, so her wallet is full of very small payments (UTXO). Now she wants to purchase hundreds of schoolbooks from a local publisher, paying in bitcoin. - -As Eugenia's wallet application tries to construct a single larger payment transaction, it must source from the available UTXO set, which is composed of many smaller amounts. That means that the resulting transaction will source from more than a hundred small-value UTXO as inputs and only one output, paying the book publisher. A transaction with that many inputs will be larger than one kilobyte, perhaps several kilobytes in size. As a result, it will require a much higher fee than the median-sized transaction. - -Eugenia's wallet application will calculate the appropriate fee by measuring the size of the transaction and multiplying that by the per-kilobyte fee. Many wallets will overpay fees for larger transactions to ensure the transaction is processed promptly. The higher fee is not because Eugenia is spending more money, but because her transaction is more complex and larger in size--the fee is independent of the transaction's bitcoin value.((("", startref="Tout06"))) - -[[tx_script]] -[role="pagebreak-before less_space_h1"] -=== Transaction Scripts and Script Language - -((("transactions", "scripts and Script language", id="Tsript06")))((("scripting", "transactions and", id="Stransact06")))The bitcoin transaction script language, called _Script_, is a Forth-like reverse-polish notation stack-based execution language. If that sounds like gibberish, you probably haven't studied 1960s programming languages, but that's ok—we will explain it all in this chapter. Both the locking script placed on an UTXO and the unlocking script are written in this scripting language. When a transaction is validated, the unlocking script in each input is executed alongside the corresponding locking script to see if it satisfies the spending condition. - -Script is a very simple language that was designed to be limited in scope and executable on a range of hardware, perhaps as simple as an embedded device. It requires minimal processing and cannot do many of the fancy things modern programming languages can do. For its use in validating programmable money, this is a deliberate security feature. - -((("Pay-to-Public-Key-Hash (P2PKH)")))Today, most transactions processed through the Bitcoin network have the form "Payment to Bob's Bitcoin address" and are based on a script called a Pay-to-Public-Key-Hash script. However, bitcoin transactions are not limited to the "Payment to Bob's Bitcoin address" script. In fact, locking scripts can be written to express a vast variety of complex conditions. In order to understand these more complex scripts, we must first understand the basics of transaction scripts and script language. - -In this section, we will demonstrate the basic components of the bitcoin transaction scripting language and show how it can be used to express simple conditions for spending and how those conditions can be satisfied by unlocking scripts. - -[TIP] -==== -((("programmable money")))Bitcoin transaction validation is not based on a static pattern, but instead is achieved through the execution of a scripting language. This language allows for a nearly infinite variety of conditions to be expressed. This is how bitcoin gets the power of "programmable money". -==== - - -==== Turing Incompleteness - -((("Turing incompleteness")))The bitcoin transaction script language contains many operators, but is deliberately limited in one important way--there are no loops or complex flow control capabilities other than conditional flow control. This ensures that the language is not _Turing Complete_, meaning that scripts have limited complexity and predictable execution times. Script is not a general-purpose language. ((("denial-of-service attacks")))((("denial-of-service attacks", see="also security")))((("security", "denial-of-service attacks")))These limitations ensure that the language cannot be used to create an infinite loop or other form of "logic bomb" that could be embedded in a transaction in a way that causes a denial-of-service attack against the Bitcoin network. Remember, every transaction is validated by every full node on the Bitcoin network. A limited language prevents the transaction validation mechanism from being used as a vulnerability. - -==== Stateless Verification - -((("stateless verification")))The bitcoin transaction script language is stateless, in that there is no state prior to execution of the script, or state saved after execution of the script. Therefore, all the information needed to execute a script is contained within the script. A script will predictably execute the same way on any system. If your system verifies a script, you can be sure that every other system in the Bitcoin network will also verify the script, meaning that a valid transaction is valid for everyone and everyone knows this. This predictability of outcomes is an essential benefit of the Bitcoin system. - -[[tx_lock_unlock]] -==== Script Construction (Lock + Unlock) - -Bitcoin's transaction validation engine relies on two types of scripts to validate transactions: a locking script and an unlocking script. - -((("locking scripts")))((("unlocking scripts")))((("scripting", "locking scripts")))A locking script is a spending condition placed on an output: it specifies the conditions that must be met to spend the output in the future. ((("scriptPubKey")))Historically, the locking script was called a _scriptPubKey_, because it usually contained a public key or Bitcoin address (public key hash). In this book we refer to it as a "locking script" to acknowledge the much broader range of possibilities of this scripting technology. In most bitcoin applications, what we refer to as a locking script will appear in the source code as +scriptPubKey+. ((("witnesses")))((("cryptographic puzzles")))You will also see the locking script referred to as a _witness script_ (see <>) or more generally as a _cryptographic puzzle_. These terms all mean the same thing, at different levels of abstraction. - -An unlocking script is a script that "solves," or satisfies, the conditions placed on an output by a locking script and allows the output to be spent. Unlocking scripts are part of every transaction input. Most of the time they contain a digital signature produced by the user's wallet from his or her private key. ((("scriptSig")))Historically, the unlocking script was called _scriptSig_, because it usually contained a digital signature. In most bitcoin applications, the source code refers to the unlocking script as +scriptSig+. You will also see the unlocking script referred to as a _witness_ (see <>). In this book, we refer to it as an "unlocking script" to acknowledge the much broader range of locking script requirements, because not all unlocking scripts must contain signatures. - -Every bitcoin validating node will validate transactions by executing the locking and unlocking scripts together. Each input contains an unlocking script and refers to a previously existing UTXO. The validation software will copy the unlocking script, retrieve the UTXO referenced by the input, and copy the locking script from that UTXO. The unlocking and locking script are then executed in sequence. The input is valid if the unlocking script satisfies the locking script conditions (see <>). All the inputs are validated independently, as part of the overall validation of the transaction. - -Note that the UTXO is permanently recorded in the blockchain, and therefore is invariable and is unaffected by failed attempts to spend it by reference in a new transaction. Only a valid transaction that correctly satisfies the conditions of the output results in the output being considered as "spent" and removed from the set of unspent transaction outputs (UTXO set). - -<> is an example of the unlocking and locking scripts for the most common type of bitcoin transaction (a payment to a public key hash), showing the combined script resulting from the concatenation of the unlocking and locking scripts prior to script validation. - -[[scriptSig_and_scriptPubKey]] -.Combining scriptSig and scriptPubKey to evaluate a transaction script -image::images/mbc2_0603.png["scriptSig_and_scriptPubKey"] - -===== The script execution stack - -Bitcoin's scripting language is called a stack-based language because it uses a data structure called a _stack_. A stack is a very simple data structure that can be visualized as a stack of cards. A stack allows two operations: push and pop. Push adds an item on top of the stack. Pop removes the top item from the stack. Operations on a stack can only act on the topmost item on the stack. A stack data structure is also called a Last-In-First-Out, or "LIFO" queue. - -The scripting language executes the script by processing each item from left to right. Numbers (data constants) are pushed onto the stack. Operators push or pop one or more parameters from the stack, act on them, and might push a result onto the stack. For example, +OP_ADD+ will pop two items from the stack, add them, and push the resulting sum onto the stack. - -Conditional operators evaluate a condition, producing a boolean result of TRUE or FALSE. For example, +OP_EQUAL+ pops two items from the stack and pushes TRUE (TRUE is represented by the number 1) if they are equal or FALSE (represented by zero) if they are not equal. Bitcoin transaction scripts usually contain a conditional operator, so that they can produce the TRUE result that signifies a valid transaction. - -===== A simple script - -Now let's apply what we've learned about scripts and stacks to some simple examples. - -In <>, the script +2 3 OP_ADD 5 OP_EQUAL+ demonstrates the arithmetic addition operator +OP_ADD+, adding two numbers and putting the result on the stack, followed by the conditional operator +OP_EQUAL+, which checks that the resulting sum is equal to +5+. For brevity, the +OP_+ prefix is omitted in the step-by-step example. For more details on the available script operators and functions, see <>. - -Although most locking scripts refer to a public key hash (essentially, a Bitcoin address), thereby requiring proof of ownership to spend the funds, the script does not have to be that complex. Any combination of locking and unlocking scripts that results in a TRUE value is valid. The simple arithmetic we used as an example of the scripting language is also a valid locking script that can be used to lock a transaction output. - -Use part of the arithmetic example script as the locking script: - ----- -3 OP_ADD 5 OP_EQUAL ----- - -which can be satisfied by a transaction containing an input with the unlocking script: - ----- -2 ----- - -The validation software combines the locking and unlocking scripts and the resulting script is: - ----- -2 3 OP_ADD 5 OP_EQUAL ----- - -As we saw in the step-by-step example in <>, when this script is executed, the result is +OP_TRUE+, making the transaction valid. Not only is this a valid transaction output locking script, but the resulting UTXO could be spent by anyone with the arithmetic skills to know that the number 2 satisfies the script. - -[TIP] -==== -((("transactions", "valid and invalid")))Transactions are valid if the top result on the stack is +TRUE+ (noted as ++{0x01}++), any other nonzero value, not +OP_0+, or if the stack is empty after script execution. Transactions are invalid if the top value on the stack is +FALSE+ (a zero-length empty value, noted as ++{}++) or if script execution is halted explicitly by an operator, such as +OP_VERIFY+, +OP_RETURN+, or a conditional terminator such as +OP_ENDIF+. See <> for details. -==== - -[[simplemath_script]] -.Bitcoin's script validation doing simple math -image::images/mbc2_0604.png["TxScriptSimpleMathExample"] - -[role="pagebreak-before"] -The following is a slightly more complex script, which calculates ++2 + 7 - 3 + 1++. Notice that when the script contains several operators in a row, the stack allows the results of one operator to be acted upon by the next operator: - ----- -2 7 OP_ADD 3 OP_SUB 1 OP_ADD 7 OP_EQUAL ----- - -Try validating the preceding script yourself using pencil and paper. When the script execution ends, you should be left with the value +TRUE+ on the stack. - -[[script_exec]] -===== Separate execution of unlocking and locking scripts - -((("security", "locking and unlocking scripts")))In the original Bitcoin client, the unlocking and locking scripts were concatenated and executed in sequence. For security reasons, this was changed in 2010, because of a vulnerability that allowed a malformed unlocking script to push data onto the stack and corrupt the locking script. In the current implementation, the scripts are executed separately with the stack transferred between the two executions, as described next. - -First, the unlocking script is executed, using the stack execution engine. If the unlocking script is executed without errors (e.g., it has no "dangling" pointers left over), the main stack is copied and the locking script is executed. If the result of executing the locking script with the stack data copied from the unlocking script is "TRUE," the unlocking script has succeeded in resolving the conditions imposed by the locking script and, therefore, the input is a valid authorization to spend the UTXO. If any result other than "TRUE" remains after execution of the combined script, the input is invalid because it has failed to satisfy the spending conditions placed on the UTXO. - - -[[p2pkh]] -==== Pay-to-Public-Key-Hash (P2PKH) - -((("Pay-to-Public-Key-Hash (P2PKH)")))The vast majority of transactions processed on the Bitcoin network spend outputs locked with a Pay-to-Public-Key-Hash or "P2PKH" script. These outputs contain a locking script that locks the output to a public key hash, more commonly known as a Bitcoin address. An output locked by a P2PKH script can be unlocked (spent) by presenting a public key and a digital signature created by the corresponding private key (see <>). - -((("use cases", "buying coffee")))For example, let's look at Alice's payment to Bob's Cafe again. Alice made a payment of 0.015 bitcoin to the cafe's Bitcoin address. That transaction output would have a locking script of the form: - ----- -OP_DUP OP_HASH160 OP_EQUALVERIFY OP_CHECKSIG ----- - -The +Cafe Public Key Hash+ is equivalent to the Bitcoin address of the cafe, without the Base58Check encoding. Most applications would show the _public key hash_ in hexadecimal encoding and not the familiar Bitcoin address Base58Check format that begins with a "1." - -The preceding locking script can be satisfied with an unlocking script of the form: - ----- - ----- - -The two scripts together would form the following combined validation script: - ----- - OP_DUP OP_HASH160 - OP_EQUALVERIFY OP_CHECKSIG ----- - -When executed, this combined script will evaluate to TRUE if, and only if, the unlocking script matches the conditions set by the locking script. In other words, the result will be TRUE if the unlocking script has a valid signature from the cafe's private key that corresponds to the public key hash set as an encumbrance. - -Figures pass:[#P2PubKHash1] and pass:[#P2PubKHash2] show (in two parts) a step-by-step execution of the combined script, which will prove this is a valid transaction.((("", startref="Tsript06")))((("", startref="Stransact06"))) - -[[P2PubKHash1]] -.Evaluating a script for a P2PKH transaction (part 1 of 2) -image::images/mbc2_0605.png["Tx_Script_P2PubKeyHash_1"] - -[[P2PubKHash2]] -.Evaluating a script for a P2PKH transaction (part 2 of 2) -image::images/mbc2_0606.png["Tx_Script_P2PubKeyHash_2"] - -[[digital_sigs]] -=== Digital Signatures (ECDSA) - -((("transactions", "digital signatures and", id="Tdigsig06")))So far, we have not delved into any detail about "digital signatures." In this section we look at how digital signatures work and how they can present proof of ownership of a private key without revealing that private key. - -((("digital signatures", "algorithm used")))((("Elliptic Curve Digital Signature Algorithm (ECDSA)")))The digital signature algorithm used in bitcoin is the _Elliptic Curve Digital Signature Algorithm_, or _ECDSA_. ECDSA is the algorithm used for digital signatures based on elliptic curve private/public key pairs, as described in <>. ECDSA is used by the script functions +OP_CHECKSIG+, +OP_CHECKSIGVERIFY+, +OP_CHECKMULTISIG+, and +OP_CHECKMULTISIGVERIFY+. Any time you see those in a locking script, the unlocking script must contain an ECDSA signature. - -((("digital signatures", "purposes of")))A digital signature serves three purposes in bitcoin. First, the signature proves that the owner of the private key, who is by implication the owner of the funds, has _authorized_ the spending of those funds. Secondly, the proof of authorization is _undeniable_ (nonrepudiation). Thirdly, the signature proves that the transaction (or specific parts of the transaction) have not and _cannot be modified_ by anyone after it has been signed. - -Note that each transaction input is signed independently. This is critical, as neither the signatures nor the inputs have to belong to or be applied by the same "owners." In fact, a specific transaction scheme called "CoinJoin" uses this fact to create multi-party transactions for privacy. - -[NOTE] -==== -Each transaction input and any signature it may contain is _completely_ independent of any other input or signature. Multiple parties can collaborate to construct transactions and sign only one input each. -==== - -[[digital_signature_definition]] -.Wikipedia's Definition of a "Digital Signature" -**** -((("digital signatures", "defined")))A digital signature is a mathematical scheme for demonstrating the authenticity of a digital message or documents. A valid digital signature gives a recipient reason to believe that the message was created by a known sender (authentication), that the sender cannot deny having sent the message (nonrepudiation), and that the message was not altered in transit (integrity). - -_Source: https://en.wikipedia.org/wiki/Digital_signature_ -**** - -==== How Digital Signatures Work - -((("digital signatures", "how they work")))A digital signature is a _mathematical scheme_ that consists of two parts. The first part is an algorithm for creating a signature, using a private key (the signing key), from a message (the transaction). The second part is an algorithm that allows anyone to verify the signature, given also the message and a public key. - -===== Creating a digital signature - -In bitcoin's implementation of the ECDSA algorithm, the "message" being signed is the transaction, or more accurately a hash of a specific subset of the data in the transaction (see <>). The signing key is the user's private key. The result is the signature: - -latexmath:[\(Sig = F_{sig}(F_{hash}(m), dA)\)] - -where: - -* _dA_ is the signing private key -* _m_ is the transaction (or parts of it) -* _F_~_hash_~ is the hashing function -* _F_~_sig_~ is the signing algorithm -* _Sig_ is the resulting signature - -More details on the mathematics of ECDSA can be found in <>. - -The function _F_~_sig_~ produces a signature +Sig+ that is composed of two values, commonly referred to as +R+ and +S+: - ----- -Sig = (R, S) ----- - -((("Distinguished Encoding Rules (DER)")))Now that the two values +R+ and +S+ have been calculated, they are serialized into a byte-stream using an international standard encoding scheme called the _Distinguished Encoding Rules_, or _DER_. - -[[seralization_of_signatures_der]] -===== Serialization of signatures (DER) - -Let's look at the transaction Alice ((("use cases", "buying coffee", id="alicesixtwo")))created again. In the transaction input there is an unlocking script that contains the following DER-encoded signature from Alice's wallet: - ----- -3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e381301 ----- - -That signature is a serialized byte-stream of the +R+ and +S+ values produced by Alice's wallet to prove she owns the private key authorized to spend that output. The serialization format consists of nine elements as follows: - -* +0x30+—indicating the start of a DER sequence -* +0x45+—the length of the sequence (69 bytes) - * +0x02+—an integer value follows - * +0x21+—the length of the integer (33 bytes) - * +R+—++00884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb++ - * +0x02+—another integer follows - * +0x20+—the length of the integer (32 bytes) - * +S+—++4b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813++ -* A suffix (+0x01+) indicating the type of hash used (+SIGHASH_ALL+) - -See if you can decode Alice's serialized (DER-encoded) signature using this list. The important numbers are +R+ and +S+; the rest of the data is part of the DER encoding scheme. - -==== Verifying the Signature - -((("digital signatures", "verifying")))To verify the signature, one must have the signature (+R+ and +S+), the serialized transaction, and the public key (that corresponds to the private key used to create the signature). Essentially, verification of a signature means "Only the owner of the private key that generated this public key could have produced this signature on this transaction." - -The signature verification algorithm takes the message (a hash of the transaction or parts of it), the signer's public key and the signature (+R+ and +S+ values), and returns TRUE if the signature is valid for this message and public key. - -[[sighash_types]] -==== Signature Hash Types (SIGHASH) - -((("digital signatures", "signature hash types")))((("commitment")))Digital signatures are applied to messages, which in the case of bitcoin, are the transactions themselves. The signature implies a _commitment_ by the signer to specific transaction data. In the simplest form, the signature applies to the entire transaction, thereby committing all the inputs, outputs, and other transaction fields. However, a signature can commit to only a subset of the data in a transaction, which is useful for a number of scenarios as we will see in this section. - -((("SIGHASH flags")))Bitcoin signatures have a way of indicating which part of a transaction's data is included in the hash signed by the private key using a +SIGHASH+ flag. The +SIGHASH+ flag is a single byte that is appended to the signature. Every signature has a +SIGHASH+ flag and the flag can be different from input to input. A transaction with three signed inputs may have three signatures with different +SIGHASH+ flags, each signature signing (committing) different parts of the transaction. - -Remember, each input may contain a signature in its unlocking script. As a result, a transaction that contains several inputs may have signatures with different +SIGHASH+ flags that commit different parts of the transaction in each of the inputs. Note also that bitcoin transactions may contain inputs from different "owners," who may sign only one input in a partially constructed (and invalid) transaction, collaborating with others to gather all the necessary signatures to make a valid transaction. Many of the +SIGHASH+ flag types only make sense if you think of multiple participants collaborating outside the Bitcoin network and updating a partially signed transaction. - -[role="pagebreak-before"] -There are three +SIGHASH+ flags: +ALL+, +NONE+, and +SINGLE+, as shown in <>. - -[[sighash_types_and_their]] -.SIGHASH types and their meanings -[options="header"] -|======================= -|+SIGHASH+ flag| Value | Description -| +ALL+ | 0x01 | Signature applies to all inputs and outputs -| +NONE+ | 0x02 | Signature applies to all inputs, none of the outputs -| +SINGLE+ | 0x03 | Signature applies to all inputs but only the one output with the same index number as the signed input -|======================= - -In addition, there is a modifier flag +SIGHASH_ANYONECANPAY+, which can be combined with each of the preceding flags. When +ANYONECANPAY+ is set, only one input is signed, leaving the rest (and their sequence numbers) open for modification. The +ANYONECANPAY+ has the value +0x80+ and is applied by bitwise OR, resulting in the combined flags as shown in <>. - -[[sighash_types_with_modifiers]] -.SIGHASH types with modifiers and their meanings -[options="header"] -|======================= -|SIGHASH flag| Value | Description -| ALL\|ANYONECANPAY | 0x81 | Signature applies to one input and all outputs -| NONE\|ANYONECANPAY | 0x82 | Signature applies to one input, none of the outputs -| SINGLE\|ANYONECANPAY | 0x83 | Signature applies to one input and the output with the same index number -|======================= - -These flag combinations are summarized in <>. - -[[sighash_combinations]] -.Summary of different sighash combinations -image::images/sighash_combinations.png["Summary of different SIGHASH flag combinations"] - -The way +SIGHASH+ flags are applied during signing and verification is that a copy of the transaction is made and certain fields within are truncated (set to zero length and emptied). The resulting transaction is serialized. The +SIGHASH+ flag is added to the end of the serialized transaction and the result is hashed. The hash itself is the "message" that is signed. Depending on which +SIGHASH+ flag is used, different parts of the transaction are truncated. The resulting hash depends on different subsets of the data in the transaction. By including the +SIGHASH+ as the last step before hashing, the signature commits the +SIGHASH+ type as well, so it can't be changed (e.g., by a miner). - -[NOTE] -==== -All +SIGHASH+ types sign the transaction +nLocktime+ field (see <>). In addition, the +SIGHASH+ type itself is appended to the transaction before it is signed, so that it can't be modified once signed. -==== - -In the example of Alice's transaction (see the list in <>), we saw that the last part of the DER-encoded signature was +01+, which is the +SIGHASH_ALL+ flag. This locks the transaction data, so Alice's signature is committing the state of all inputs and outputs. This is the most common signature form. - -Let's look at some of the other +SIGHASH+ types and how they can be used in practice: - -+ALL|ANYONECANPAY+ :: ((("charitable donations")))((("use cases", "charitable donations")))This construction can be used to make a "crowdfunding”-style transaction. Someone attempting to raise funds can construct a transaction with a single output. The single output pays the "goal" amount to the fundraiser. Such a transaction is obviously not valid, as it has no inputs. However, others can now amend it by adding an input of their own, as a donation. They sign their own input with +ALL|ANYONECANPAY+. Unless enough inputs are gathered to reach the value of the output, the transaction is invalid. Each donation is a "pledge," which cannot be collected by the fundraiser until the entire goal amount is raised. - -+NONE+ :: This construction can be used to create a "bearer check" or "blank check" of a specific amount. It commits to the input, but allows the output locking script to be changed. Anyone can write their own Bitcoin address into the output locking script and redeem the transaction. However, the output value itself is locked by the signature. - -+NONE|ANYONECANPAY+ :: This construction can be used to build a "dust collector." Users who have tiny UTXO in their wallets can't spend these because the cost in fees exceeds the value of the dust. With this type of signature, the dust UTXO can be donated for anyone to aggregate and spend whenever they want. - -((("Bitmask Sighash Modes")))There are some proposals to modify or expand the +SIGHASH+ system. One such proposal is _Bitmask Sighash Modes_ by Blockstream's Glenn Willen, as part of the Elements project. This aims to create a flexible replacement for +SIGHASH+ types that allows "arbitrary, miner-rewritable bitmasks of inputs and outputs" that can express "more complex contractual precommitment schemes, such as signed offers with change in a distributed asset exchange." - -[NOTE] -==== -You will not see +SIGHASH+ flags presented as an option in a user's wallet application. With few exceptions, wallets construct P2PKH scripts and sign with +SIGHASH_ALL+ flags. To use a different +SIGHASH+ flag, you would have to write software to construct and sign transactions. More importantly, +SIGHASH+ flags can be used by special-purpose bitcoin applications that enable novel uses. -==== - -[[ecdsa_math]] -==== ECDSA Math - -((("Elliptic Curve Digital Signature Algorithm (ECDSA)")))As mentioned previously, signatures are created by a mathematical function _F_~_sig_~ that produces a signature composed of two values _R_ and _S_. In this section we look at the function _F_~_sig_~ in more detail. - -((("public and private keys", "key pairs", "ephemeral")))The signature algorithm first generates an _ephemeral_ (temporary) private public key pair. This temporary key pair is used in the calculation of the _R_ and _S_ values, after a transformation involving the signing private key and the transaction hash. - -The temporary key pair is based on a random number _k_, which is used as the temporary private key. From _k_, we generate the corresponding temporary public key _P_ (calculated as _P = k*G_, in the same way bitcoin public keys are derived; see <>). The _R_ value of the digital signature is then the x coordinate of the ephemeral public key _P_. - -From there, the algorithm calculates the _S_ value of the signature, such that: - -_S_ = __k__^-1^ (__Hash__(__m__) + __dA__ * __R__) _mod n_ - -where: - -* _k_ is the ephemeral private key -* _R_ is the x coordinate of the ephemeral public key -* _dA_ is the signing private key -* _m_ is the transaction data -* _n_ is the prime order of the elliptic curve - -Verification is the inverse of the signature generation function, using the _R_, _S_ values and the public key to calculate a value _P_, which is a point on the elliptic curve (the ephemeral public key used in signature creation): - -_P_ = __S__^-1^ * __Hash__(__m__) * _G_ + __S__^-1^ * _R_ * _Qa_ - -where: - -* _R_ and _S_ are the signature values -* _Qa_ is Alice's public key -* _m_ is the transaction data that was signed -* _G_ is the elliptic curve generator point - -If the x coordinate of the calculated point _P_ is equal to _R_, then the verifier can conclude that the signature is valid. - -Note that in verifying the signature, the private key is neither known nor revealed. - -[TIP] -==== -ECDSA is necessarily a fairly complicated piece of math; a full explanation is beyond the scope of this book. A number of great guides online take you through it step by step: search for "ECDSA explained" or try this one: https://bit.ly/2r0HhGB[]. -==== - -==== The Importance of Randomness in Signatures - -((("digital signatures", "randomness in")))As we saw in <>, the signature generation algorithm uses a random key _k_, as the basis for an ephemeral private/public key pair. The value of _k_ is not important, _as long as it is random_. If the same value _k_ is used to produce two signatures on different messages (transactions), then the signing _private key_ can be calculated by anyone. Reuse of the same value for _k_ in a signature algorithm leads to exposure of the private key! - -[WARNING] -==== -((("warnings and cautions", "digital signatures")))If the same value _k_ is used in the signing algorithm on two different transactions, the private key can be calculated and exposed to the world! -==== - -This is not just a theoretical possibility. We have seen this issue lead to exposure of private keys in a few different implementations of transaction-signing algorithms in bitcoin. People have had funds stolen because of inadvertent reuse of a _k_ value. The most common reason for reuse of a _k_ value is an improperly initialized random-number generator. - -((("random numbers", "random number generation")))((("entropy", "random number generation")))((("deterministic initialization")))To avoid this vulnerability, the industry best practice is to not generate _k_ with a random-number generator seeded with entropy, but instead to use a deterministic-random process seeded with the transaction data itself. This ensures that each transaction produces a different _k_. The industry-standard algorithm for deterministic initialization of _k_ is defined in https://tools.ietf.org/html/rfc6979[RFC 6979], published by the Internet Engineering Task Force. - -If you are implementing an algorithm to sign transactions in bitcoin, you _must_ use RFC 6979 or a similarly deterministic-random algorithm to ensure you generate a different _k_ for each transaction.((("", startref="Tdigsig06"))) - -=== Bitcoin Addresses, Balances, and Other Abstractions - -((("transactions", "higher-level abstractions", id="Thigher06")))We began this chapter with the discovery that transactions look very different "behind the scenes" than how they are presented in wallets, blockchain explorers, and other user-facing applications. Many of the simplistic and familiar concepts from the earlier chapters, such as Bitcoin addresses and balances, seem to be absent from the transaction structure. We saw that transactions don't contain Bitcoin addresses, per se, but instead operate through scripts that lock and unlock discrete values of bitcoin. Balances are not present anywhere in this system and yet every wallet application prominently displays the balance of the user's wallet. - -Now that we have explored what is actually included in a bitcoin transaction, we can examine how the higher-level abstractions are derived from the seemingly primitive components of the transaction. - -Let's look again at how Alice's transaction was presented on a popular block explorer (<>). - -[[alice_transaction_to_bobs_cafe]] -.Alice's transaction to Bob's Cafe -image::images/mbc2_0208.png["Alice Coffee Transaction"] - -On the left side of the transaction, the blockchain explorer shows Alice's Bitcoin address as the "sender." In fact, this information is not in the transaction itself. When the blockchain explorer references the transaction it also references the previous transaction associated with the input and extracts the first output from that older transaction. Within that output is a locking script that locks the UTXO to Alice's public key hash (a P2PKH script). The blockchain explorer extracted the public key hash and encoded it using Base58Check encoding to produce and display the Bitcoin address that represents that public key. - -Similarly, on the right side, the blockchain explorer shows the two outputs; the first to Bob's Bitcoin address and the second to Alice's Bitcoin address (as change). Once again, to create these Bitcoin addresses, the blockchain explorer extracted the locking script from each output, recognized it as a P2PKH script, and extracted the public-key-hash from within. Finally, the blockchain explorer reencoded each public key hash with Base58Check to produce and display the Bitcoin addresses. - -If you were to click on Bob's Bitcoin address, the blockchain explorer would show you the view in <>. - -[[the_balance_of_bobs_bitcoin_address]] -.The balance of Bob's Bitcoin address -image::images/mbc2_0608.png["The balance of Bob's Bitcoin address"] - -The blockchain explorer displays the balance of Bob's Bitcoin address. But nowhere in the Bitcoin system is there a concept of a "balance." Rather, the values displayed here are constructed by the blockchain explorer as follows. - -To construct the "Total Received" amount, the blockchain explorer first will decode the Base58Check encoding of the Bitcoin address to retrieve the 160-bit hash of Bob's public key that is encoded within the address. Then, the blockchain explorer will search through the database of transactions, looking for outputs with P2PKH locking scripts that contain Bob's public key hash. By summing up the value of all the outputs, the blockchain explorer can produce the total value received. - -Constructing the current balance (displayed as "Final Balance") requires a bit more work. The blockchain explorer keeps a separate database of the outputs that are currently unspent, the UTXO set. To maintain this database, the blockchain explorer must monitor the Bitcoin network, add newly created UTXO, and remove spent UTXO, in real time, as they appear in unconfirmed transactions. This is a complicated process that depends on keeping track of transactions as they propagate, as well as maintaining consensus with the Bitcoin network to ensure that the correct chain is followed. Sometimes, the blockchain explorer goes out of sync and its perspective of the UTXO set is incomplete or incorrect. - -From the UTXO set, the blockchain explorer sums up the value of all unspent outputs referencing Bob's public key hash and produces the "Final Balance" number shown to the user. - -In order to produce this one image, with these two "balances," the blockchain explorer has to index and search through dozens, hundreds, or even hundreds of thousands of transactions. - -In summary, the information presented to users through wallet applications, blockchain explorers, and other bitcoin user interfaces is often composed of higher-level abstractions that are derived by searching many different transactions, inspecting their content, and manipulating the data contained within them. By presenting this simplistic view of bitcoin transactions that resemble bank checks from one sender to one recipient, these applications have to abstract a lot of underlying details. They mostly focus on the common types of transactions: P2PKH with SIGHASH_ALL signatures on every input. Thus, while bitcoin applications can present more than 80% of all transactions in an easy-to-read manner, they are sometimes stumped by transactions that deviate from the norm. Transactions that contain more complex locking scripts, or different SIGHASH flags, or many inputs and outputs, demonstrate the simplicity and weakness of these abstractions. - -Every day, hundreds of transactions that do not contain P2PKH outputs are confirmed on the blockchain. The blockchain explorers often present these with red warning messages saying they cannot decode an address. - -As we will see in the next chapter, these are not necessarily strange transactions. They are transactions that contain more complex locking scripts than the common P2PKH. We will learn how to decode and understand more complex scripts and the applications they support next.((("", startref="Thigher06")))((("", startref="alicesixtwo"))) diff --git a/ch06_transactions.adoc b/ch06_transactions.adoc new file mode 100644 index 00000000..db25c8f3 --- /dev/null +++ b/ch06_transactions.adoc @@ -0,0 +1,1287 @@ +[[c_transactions]] +== Transactions + +The way we typically transfer physical cash has little resemblance to +the way we transfer bitcoins. Physical cash is a bearer token. Alice +pays Bob by handing him some number of tokens, such as dollar bills. +By comparison, bitcoins don't exist either physically or as digital +data--Alice can't hand Bob some bitcoins or send them by email. + +Instead, consider how Alice might transfer control over a parcel of land +to Bob. She can't physically pick up the land and hand it to Bob. +Rather there exists some sort of record (usually maintained by a local +government) that describes the land Alice owns. Alice transfers that +land to Bob by convincing the government to update the record to say +that Bob now owns the land. + +Bitcoin works in a similar way. There exists a database on every +Bitcoin full node that says that Alice controls some number of +bitcoins. Alice pays Bob by convincing full nodes to update their +database to say that some of Alice's bitcoins are now controlled by Bob. +The data that Alice uses to convince full nodes to update their +databases is called a _transaction_. This is done without directly +using either Alice's or Bob's identities, as we'll see in +<>. + +In this chapter we'll deconstruct a Bitcoin transaction and examine each +of its parts to see how they facilitate the transfer of value in a way +that's highly expressive and amazingly reliable. + +[[tx_structure]] +=== A Serialized Bitcoin Transaction + +In <>, we ((("transactions", "serialized", id="transaction-serialize")))((("serialized transactions", id="serial-transactions")))((("Bitcoin Core", "serialized transactions", id="bitcoin-core-serial-transaction")))used Bitcoin Core with +the txindex option enabled to retrieve a copy of Alice's payment to Bob. +Let's retrieve the transaction containing that payment again, as shown in <>. + +[[alice_tx_serialized_reprint]] +.Alice's serialized transaction +==== +---- +$ bitcoin-cli getrawtransaction 466200308696215bbc949d5141a49a41\ +38ecdfdfaa2a8029c1f9bcecd1f96177 + +01000000000101eb3ae38f27191aa5f3850dc9cad00492b88b72404f9da13569 +8679268041c54a0100000000ffffffff02204e0000000000002251203b41daba +4c9ace578369740f15e5ec880c28279ee7f51b07dca69c7061e07068f8240100 +000000001600147752c165ea7be772b2c0acb7f4d6047ae6f4768e0141cf5efe +2d8ef13ed0af21d4f4cb82422d6252d70324f6f4576b727b7d918e521c00b51b +e739df2f899c49dc267c0ad280aca6dab0d2fa2b42a45182fc83e81713010000 +0000 +---- +==== + + +Bitcoin Core's serialization format is special because it's the format +used to make commitments to transactions and to relay them across +Bitcoin's P2P network, but otherwise programs can use +a different format as long as they transmit all of the +same data. However, Bitcoin Core's format is reasonably compact for the +data it transmits and simple to parse, so many other Bitcoin programs +use this format. + +[TIP] +==== +The only ((("partially signed bitcoin transaction (PSBT) format")))((("PSBT (partially signed bitcoin transaction) format")))other widely used transaction serialization format that +we're aware of is the partially signed bitcoin transaction (PSBT) format +documented in BIPs 174 and 370 (with extensions documented in other +BIPs). PSBT allows an untrusted program to produce a transaction +template that can be verified and updated by trusted programs (such as +hardware signing devices) that have the necessary private keys or +other sensitive data to fill in the template. To accomplish this, PSBT +allows storing a significant amount of metadata about a transaction, +making it much less compact than the standard serialization format. +This book does not go into detail about PSBT, but we strongly recommend +it to developers of wallets that plan to support signing +with multiple keys. +==== + +The transaction displayed in hexadecimal in <> is +replicated as a byte map in <>. Note that it takes +64 hexadecimal characters to display 32 bytes. This map shows only the +top-level fields. We'll examine each of them in the order they appear +in the transaction and describe any additional fields that they((("transactions", "serialized", startref="transaction-serialize")))((("serialized transactions", startref="serial-transactions")))((("Bitcoin Core", "serialized transactions", startref="bitcoin-core-serial-transaction"))) contain. + +[[alice_tx_byte_map]] +.A byte map of Alice's transaction. +image::images/mbc3_0601.png["A byte map of Alice's transaction"] + +[[version]] +=== Version + +The first ((("transactions", "version of", id="transactions-version")))((("version (of transactions)", id="version-transactions")))four bytes of a serialized Bitcoin transaction are its +version. The original version of Bitcoin transactions was version 1 +(0x01000000). All transactions in Bitcoin must follow +the rules of version 1 transactions, with many of those rules being +described throughout this book. + +Version 2 Bitcoin transactions were introduced in the BIP68 soft fork +change to Bitcoin's consensus rules. BIP68 places additional +constraints on the sequence field, but those constraints only apply to +transactions with version 2 or higher. Version 1 transactions are +unaffected. BIP112, which was part of the same soft fork as BIP68, +upgraded an opcode (++OP_CHECKSEQUENCEVERIFY++), which will now fail if it is +evaluated as part of a transaction with a version less than 2. Beyond +those two changes, version 2 transactions are identical to version 1 +transactions. + +[role="less_space pagebreak-before"] +.Protecting Presigned Transactions +**** +The last step ((("transactions", "presigned", id="transaction-presign")))((("presigned transactions", id="presign-transaction")))before broadcasting a transaction to the network for +inclusion in the blockchain is to sign it. However, it's possible to +sign a transaction without broadcasting it immediately. You can save +that presigned transaction for months or years in the belief that it +can be added to the blockchain later when you do broadcast it. In the +interim, you may even lose access to the private key (or keys) necessary +to sign an alternative transaction spending the funds. This isn't +hypothetical: several protocols built on Bitcoin, including Lightning +Network, depend on presigned [.keep-together]#transactions.# + +This creates a challenge for protocol developers when they assist users +in upgrading the Bitcoin consensus protocol. Adding new +constraints--such as BIP68 did to the sequence field--may invalidate +some presigned transactions. If there's no way to create a new +signature for an equivalent transaction, then the money being spent in +the presigned transaction is permanently lost. + +This problem is solved by reserving some transaction features for +upgrades, such as version numbers. Anyone creating presigned +transactions prior to BIP68 should have been using version 1 +transactions, so only applying BIP68's additional constraints on +sequence to transactions v2 or higher should not invalidate any +presigned [.keep-together]#transactions.# + +If you implement a protocol that uses presigned transactions, ensure +that it doesn't use any features that are reserved for future upgrades. +Bitcoin Core's default transaction relay policy does not allow the use +of reserved features. You can test whether a transaction complies with +that policy by using Bitcoin Core's +testmempoolaccept+ RPC on Bitcoin +mainnet. +**** + +As of this writing, a proposal to begin using version 3 transactions is +being widely considered. That proposal does not seek to change the +consensus rules but only the policy that Bitcoin full nodes use to relay +transactions. Under the proposal, version 3 transactions would be +subject to additional constraints in order to prevent certain denial of +service (DoS) attacks that we'll discuss((("transactions", "version of", startref="transactions-version")))((("version (of transactions)", startref="version-transactions"))) further in <>. + +=== Extended Marker and Flag + +The next two fields((("transactions", "extended serialization format")))((("extended serialization format")))((("BIP144 extended serialization format"))) of the example serialized transaction were added as +part of the segregated witness (segwit) soft fork change to Bitcoin's +consensus rules. The rules were changed according to BIPs 141 and 143, +but the _extended serialization format_ is defined in BIP144. + +[role="less_space pagebreak-before"] +If the transaction includes a witness structure (which we'll describe in +<>), the marker must be zero (0x00) and the flag must be +nonzero. In the current P2P protocol, the flag should always be one +(0x01); alternative flags are reserved for later protocol upgrades. + +If the transaction doesn't need a witness stack, the marker and flag must not +be present. This is compatible with the original version of Bitcoin's +transaction serialization format, now called _legacy serialization_. +For details, see <>. + +In ((("transactions", "legacy serialization")))((("legacy serialization")))legacy serialization, the marker byte would have been interpreted as +the number of inputs (zero). A transaction can't have zero inputs, so +the marker signals to modern programs that extended serialization is +being used. The flag field provides a similar signal and also +simplifies the process of updating the serialization format in the +future. + +[[inputs]] +=== Inputs + +The((("transactions", "inputs", id="transaction-input")))((("inputs", id="input-transaction"))) inputs field contains several other fields, so let's start by showing a +map of those bytes in <>. + +[[alice_tx_input_map]] +.Map of bytes in the inputs field of Alice's transaction. +image::images/mbc3_0602.png["map of bytes in the inputs field of Alice's transaction"] + +==== Length of Transaction Input List + +The ((("transactions", "inputs", "length of list", id="transaction-input-length")))((("inputs", "length of list", id="input-transaction-length")))transaction input list starts with an integer indicating the number of inputs +in the transaction. The minimum value is one. There's no explicit +maximum value, but restrictions on the maximum size of a transaction +effectively limit transactions to a few thousand inputs. The number is +encoded as a compactSize unsigned integer. + +[role="less_space pagebreak-before"] +.CompactSize Unsigned Integers +**** +Unsigned integers((("compactSize unsigned integers", id="compactsize")))((("unsigned integers", id="unsigned"))) in Bitcoin that often have low values, but which may +sometimes have high values, are usually encoded using the compactSize +data type. CompactSize is a version of a variable-length integer, so +it's sometimes called var_int or varint (see, for example, documentation +for BIPs 37 and 144). + + +[WARNING] +==== +Several varieties of variable length integers are used +in different programs, including in different Bitcoin programs. For +example, Bitcoin Core serializes its UTXO database using a data type it +calls +VarInts+, which is different from compactSize. Additionally, the +nBits field in a Bitcoin block header is encoded using a custom data +type known as +Compact+, which is unrelated to compactSize. When +talking about the variable length integers used in Bitcoin transaction +serialization and other parts of the Bitcoin P2P protocol, we will +always use the full name compactSize. +==== + +For numbers from 0 to 252, compactSize unsigned integers are identical +to the C-language data type +uint8_t+, which is probably the native +encoding familiar to any programmer. For other numbers up to +0xffffffffffffffff, a byte is prefixed to the number to indicate its +length—but otherwise the numbers look like regular C-language encoded +unsigned integers: + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ValueBytes usedFormat

≥ 0 && ≤ 252 (0xfc)

1

uint8_t

≥ 253 && ≤ 0xffff

3

0xfd followed by the number as uint16_t

≥ 0x10000 && ≤ 0xffffffff

5

0xfe followed by the number as uint32_t

≥ 0x100000000 && ≤ 0xffffffffffffffff

9

0xff followed by the number as uint64_t

+++++ +**** + +Each input in a transaction must contain three fields: an _outpoint_ field, a length-prefixed _input script_ field, and a _sequence_ + +We'll look at each of those fields in the following sections. Some +inputs also include a witness stack, but this is serialized at the end of a +transaction so we'll ((("transactions", "inputs", "length of list", startref="transaction-input-length")))((("inputs", "length of list", startref="input-transaction-length")))examine it later. + + +[[outpoints]] +==== Outpoint + +A Bitcoin ((("transactions", "inputs", "outpoint field", id="transaction-input-outpoint")))((("inputs", "outpoint field", id="input-transaction-outpoint")))((("outpoint field (transaction inputs)", id="outpoint")))transaction is a request for full nodes to update their +database of coin ownership information. For Alice to transfer control +of some of her bitcoins to Bob, she first needs to tell full nodes how +to find the previous transfer where she received those bitcoins. Since +control over bitcoins is assigned in transaction outputs, Alice _points_ +to the previous _output_ using an _outpoint_ field. Each input must +contain a single outpoint. + +The outpoint contains a 32-byte txid for the +transaction where Alice received the bitcoins she now wants to spend. +This txid is in Bitcoin's internal byte order for hashes; see +<>. + +Because transactions may contain multiple outputs, Alice also needs to +identify which particular output from that transaction to use, ((("output indexes")))called +its _output index_. Output indexes are 4-byte unsigned +integers starting from zero. + +When a full node encounters an outpoint, it uses that information to try +to find the referenced output. Full nodes are only required to look at earlier +transactions in the blockchain. For example, Alice's transaction is +included in block 774,958. A full node verifying her transaction +only looks for the previous output referenced by her outpoint in that +block and previous blocks, not any later blocks. Within block 774,958, +they will only look at transactions placed in the block prior to Alice's +transaction, as determined by the order of leaves in the block's merkle +tree (see <>). + +Upon finding the previous output, the full node obtains several critical +pieces of information from it: + +- The amount of bitcoins assigned to that previous output. All of those + bitcoins will be transferred in this transaction. In the example + transaction, the value of the previous output was 100,000 satoshis. + +- The authorization conditions for that previous output. These are the + conditions that must be fulfilled in order to spend the bitcoins + assigned to that previous output. + +- For confirmed transactions, the height of the block that confirmed it + and the median time past (MTP) for that block. This is required for + relative timelocks (described in <>) and outputs + of coinbase transactions (described in <>). + +- Proof that the previous output exists in the blockchain (or as a known + unconfirmed transaction) and that no other transaction has spent it. + One of Bitcoin's consensus rules forbids any output from being spent + more than once within a valid blockchain. This is the((("double spending")))((("conflicting transactions"))) rule against + _double spending_: Alice can't use the same previous output to pay + both Bob and Carol in separate transactions. Two transactions that each try to spend the + same previous output are called _conflicting transactions_ because + only one of them can be included in a valid blockchain. + +Different approaches to tracking previous outputs have been tried by +different full node implementations at various times. Bitcoin Core +currently uses the solution believed to be most effective at retaining +all necessary information while minimizing disk space: it keeps a +database that stores every UTXO and +essential metadata about it (like its confirmation block height). Each +time a new block of transactions arrives, all of the outputs they spend +are removed from the UTXO database and all of the outputs they create +are added to the((("transactions", "inputs", "outpoint field", startref="transaction-input-outpoint")))((("inputs", "outpoint field", startref="input-transaction-outpoint")))((("outpoint field (transaction inputs)", startref="outpoint"))) database. + +[[internal_and_display_order]] +.Internal and Display Byte Orders +**** +Bitcoin uses ((("hash functions", "digests")))((("digests")))((("internal byte order")))((("display byte order")))the output of hash functions, called _digests_, in various +ways. Digests provide unique identifiers for blocks and transactions; +they're used in commitments for addresses, blocks, transactions, +signatures, and more; and digests are iterated upon in Bitcoin's +proof-of-work function. In some cases, hash digests are displayed to +users in one byte order but are used internally in a different byte +order, creating confusion. For example, consider the previous output +txid from the outpoint in our example transaction: + +---- +eb3ae38f27191aa5f3850dc9cad00492b88b72404f9da135698679268041c54a +---- + +If we try using that txid to retrieve that transaction using +Bitcoin Core, we get an error and must reverse its byte order: + +---- +$ bitcoin-cli getrawtransaction \ + eb3ae38f27191aa5f3850dc9cad00492b88b72404f9da135698679268041c54a +error code: -5 +error message: +No such mempool or blockchain transaction. +Use gettransaction for wallet transactions. + +$ echo eb3ae38f27191aa5f3850dc9cad00492b88b72404f9da135698679268041c54a \ + | fold -w2 | tac | tr -d "\n" +4ac541802679866935a19d4f40728bb89204d0cac90d85f3a51a19278fe33aeb + +$ bitcoin-cli getrawtransaction \ + 4ac541802679866935a19d4f40728bb89204d0cac90d85f3a51a19278fe33aeb +02000000000101c25ae90c9f3d40cc1fc509ecfd54b06e35450702... +---- + +This odd behavior is probably an unintentional consequence of a +https://oreil.ly/01JH2[design +decision in early Bitcoin software]. As a practical matter, it means +developers of Bitcoin software need to remember to reverse the order of +bytes in transaction and block identifiers they show to users. + +In this book, we use the term _internal byte order_ for the data that +appears within transactions and blocks. We use _display byte order_ for +the form displayed to users. Another set of common terms is +_little-endian byte order_ for the internal version and _big-endian byte +order_ for the display version. +**** + +==== Input Script + +The ((("transactions", "inputs", "input script")))((("inputs", "input script")))((("input scripts")))input script field is a remnant of the legacy transaction format. Our +example transaction input spends a native segwit output that doesn't +require any data in the input script, so the length prefix for the +input script is set to zero (0x00). + +For an example of a length-prefixed input script that spends a legacy +output, we use one from an arbitrary transaction in the most recent +block as of this writing: + +---- +6b483045022100a6cc4e8cd0847951a71fad3bc9b14f24d44ba59d19094e0a8c +fa2580bb664b020220366060ea8203d766722ed0a02d1599b99d3c95b97dab8e +41d3e4d3fe33a5706201210369e03e2c91f0badec46c9c903d9e9edae67c167b +9ef9b550356ee791c9a40896 +---- + +The length prefix is a compactSize unsigned integer indicating the +length of the serialized input script field. In this case, it's a single +byte (0x6b) indicating the input script is 107 bytes. We'll cover parsing +and using scripts in detail in <>. + +[[sequence]] +==== Sequence + +The ((("transactions", "inputs", "sequence field", id="transaction-input-sequence")))((("inputs", "sequence field", id="input-transaction-sequence")))((("sequence field (transaction inputs)", id="sequence-field")))final four bytes of an input are its _sequence_ number. +The use and meaning of this field has changed over time. + +[[original_tx_replacement]] +===== Original sequence-based transaction replacement + +The ((("sequence-based transaction replacement", id="sequence-replace")))sequence field was originally intended to allow creation of +multiple versions of the same transaction, with later versions replacing +earlier versions as candidates for confirmation. The sequence number +tracked the version of the transaction. + +For example, imagine Alice and Bob want to bet on a game of cards. They +start by each signing a transaction that deposits some money into an +output with a script that requires signatures from both of them to ((("multisignature scripts")))((("setup transactions")))spend, a +_multisignature_ script (_multisig_ for short). This is called the +_setup transaction_. They then create a transaction that spends that +output: + +- The first version of the transaction, with nSequence 0 (0x00000000), + pays Alice and Bob back the money they initially deposited. This is + called a _refund transaction_. Neither of them broadcasts the refund + transaction at this time. They only need it if there's a problem. + +- Alice wins the first round of the card game, so the second version of + the transaction, with sequence 1, increases the amount of money paid + to Alice and decreases Bob's share. They both sign the updated + transaction. Again, they don't need to broadcast this version of the + transaction unless there's a problem. + +- Bob wins the second round, so the sequence is incremented to 2, + Alice's share is decreased, and Bob's share is increased. They again + sign but don't broadcast. + +- After many more rounds where the sequence is incremented, the + funds redistributed, and the resulting transaction is signed but not + broadcast, they decide to finalize the transaction. Creating a + transaction with the final balance of funds, they set sequence to its + maximum value (0xffffffff), finalizing the transaction. They broadcast + this version of the transaction, it's relayed across the network, and + eventually confirmed by miners. + +We can see the replacement rules for sequence at work if we consider +alternative scenarios: + +- Imagine that Alice broadcasts the final transaction, with a sequence of + 0xffffffff, and then Bob broadcasts one of the earlier transactions + where his balance was higher. Because Bob's version of the + transaction has a lower sequence number, full nodes using the original + Bitcoin code won't relay it to miners, and miners who also used the + original code won't mine it. + +- In another scenario, imagine that Bob broadcasts an earlier version of + the transaction a few seconds before Alice broadcasts the final + version. Nodes will relay Bob's version and miners will attempt to + mine it, but when Alice's version with its higher sequence number + arrives, nodes will also relay it and miners using the original + Bitcoin code will try to mine it instead of Bob's version. Unless Bob + got lucky and a block was discovered before Alice's version arrived, + it's Alice's version of the transaction that will get confirmed. + +This type of protocol is what we now ((("payment channels")))call a _payment channel_. +Bitcoin's creator, in an email attributed to him, called((("high-frequency transactions"))) these +_high-frequency transactions_ and described a number of features added to +the protocol to support them. We'll learn about several of those other +features later and also discover how modern versions of payment channels +are increasingly being used in Bitcoin today. + +There were a few problems with purely sequence-based payment channels. +The first was that the rules for replacing a lower-sequence transaction +with a higher-sequence transaction were only a matter of software +policy. There was no direct incentive for miners to prefer one version +of the transaction over any other. The second problem was that the +first person to send their transaction might get lucky and have it +confirmed even if it wasn't the highest-sequence transaction. A +security protocol that fails a few percent of the time due to bad luck +isn't a very effective protocol. + +The third problem was that it was possible to replace one version of a +transaction with a different version an unlimited number of +times. Each replacement would consume the bandwidth of all the relaying full nodes +on the network. For example, as of this writing, there are about 50,000 +relaying full nodes; an attacker creating 1,000 replacement transactions +per minute at 200 bytes each would use about 20 KB of their +personal bandwidth but about 10 GB of full node network bandwidth +every minute. Except for the cost of their 20 KB/minute bandwidth and +the occasional fee when a transaction got confirmed, the attacker wouldn't +need to pay any costs for the enormous burden they placed on full node +operators. + +To eliminate the risk of this attack, the original type of +sequence-based transaction replacement was disabled in an early version +of the Bitcoin software. For several years, Bitcoin full nodes would +not allow an unconfirmed transaction containing a particular input (as +indicated by its outpoint) to be replaced by a different transaction +containing the same input. However, that situation didn't ((("sequence-based transaction replacement", startref="sequence-replace")))last forever. + +[[sequence-bip125]] +===== Opt-in transaction replacement signaling + +After the((("opt-in transaction replacement")))((("replace by fee (RBF)")))((("transaction fees", "opt-in transaction replacement"))) original sequence-based transaction replacement was disabled +due to the potential for abuse, a solution was proposed: programming +Bitcoin Core and other relaying full node software to allow a +transaction that paid a higher transaction fee rate to replace a +conflicting transaction that paid a lower fee rate. This is called +_replace by fee_, or _RBF_ for short. Some users and businesses +objected to adding support for transaction replacement back into Bitcoin +Core, so a compromise was reached that once again used the sequence +field in support of replacement. + +As documented in BIP125, an unconfirmed transaction with any input that +has a sequence set to a value below 0xfffffffe (i.e., at least 2 below +the maximum value) signals to the network that its signer wants it to be +replaceable by a conflicting transaction paying a higher fee rate. +Bitcoin Core allowed those unconfirmed transactions to be replaced and +continued to disallow other transactions from being replaced. This +allowed users and businesses that objected to replacement to simply +ignore unconfirmed transactions containing the BIP125 signal until they +became confirmed. + +There's more to modern transaction replacement policies than fee rates +and sequence signals, which we'll see in <>. + +[[relative_timelocks]] +===== Sequence as a consensus-enforced relative timelock + +In <>, we learned ((("relative timelocks", id="relative-timelock")))that the BIP68 soft fork added +a new constraint to transactions with version numbers 2 or higher. That +constraint applies to the sequence field. + +Transaction inputs with sequence values less than 2^31^ are +interpreted as having a relative timelock. Such a transaction may only +be included in the blockchain once the previous output (referenced by the +outpoint) has aged by the relative timelock amount. For example, a +transaction with one input with a relative timelock of 30 blocks can +only be confirmed in a block with at least 29 blocks between it and the +block containing the output being spent on the same blockchain. +Since sequence is a per-input field, a transaction may contain any +number of timelocked inputs, all of which must have sufficiently aged +for the transaction to be valid. A disable flag allows a transaction to +include both inputs with a relative timelock (sequence < 2^31^) and +inputs without a relative timelock (sequence ≥ 2^31^). + +The sequence value is specified in either blocks or seconds. +A type-flag +is used to differentiate between values counting blocks and values +counting time in seconds. The type-flag is set in the 23rd +least-significant bit (i.e., value 1<<22). If the type-flag is set, then +the sequence value is interpreted as a multiple of 512 seconds. If +the type-flag is not set, the sequence value is interpreted as a +number of blocks. + + +When interpreting sequence as a relative timelock, only the 16 least +significant bits are considered. Once the flags (bits 32 and 23) are +evaluated, the sequence value is usually "masked" with a 16-bit mask +(e.g., +sequence+ & 0x0000FFFF). The multiple of 512 seconds is +roughly equal to the average amount of time between blocks, so the +maximum relative timelock in both blocks and seconds from 16 bits +(2^16^) is a bit more than one year. + +<> shows the binary layout of the sequence value, +as defined by BIP68. + +[[bip_68_def_of_nseq]] +.BIP68 definition of sequence encoding (Source: BIP68). +image::images/mbc3_0603.png["BIP68 definition of sequence encoding"] + +Note that any transaction that sets a relative timelock using sequence +also sends the signal for opt-in replace by fee ((("transactions", "inputs", startref="transaction-input")))((("inputs", startref="input-transaction")))((("transactions", "inputs", "sequence field", startref="transaction-input-sequence")))((("inputs", "sequence field", startref="input-transaction-sequence")))((("sequence field (transaction inputs)", startref="sequence-field")))as described in +<>. + +=== Outputs + +The ((("transactions", "outputs", id="transaction-output")))((("outputs", id="output-transaction")))outputs field of a transaction contains several fields related to +specific outputs. Just as we did with the inputs field, we'll start by +looking at the specific bytes of the outputs field from the example +transaction where Alice pays Bob, displayed as +a map of those bytes in <>. + +[[output-byte-map]] +.A byte map of the outputs field from Alice's transaction. +image::images/mbc3_0604.png["A byte map of the outputs field from Alice's transaction"] + +==== Outputs Count + +Identical((("transactions", "outputs", "count")))((("outputs", "count"))) to the start of the inputs section of a transaction, the outputs +field begins with a count indicating the number of outputs in this +transaction. It's a compactSize integer and must be greater than zero. + +The example transaction has two outputs. + +==== Amount + +The first((("transactions", "outputs", "amount field", id="transaction-output-amount")))((("outputs", "amount field", id="output-transaction-amount")))((("amount field (transaction outputs)", id="amount-field"))) field of a specific output is its _amount_, also called +"value" in Bitcoin Core. This is an 8-byte signed integer indicating +the number of satoshis to transfer. A satoshi is the smallest unit of +bitcoin that can be represented in an onchain Bitcoin transaction. +There are 100 million satoshis in a bitcoin. + +Bitcoin's consensus rules allow an output to have a value as small as +zero and as large as 21 million bitcoins (2.1 quadrillion satoshis). + +//TODO:describe early integer overflow problem + +[[uneconomical_outputs]] +===== Uneconomical outputs and disallowed dust + +Despite not ((("uneconomical outputs", id="uneconomical")))((("dust policies", id="dust")))having any value, a zero-value output can be spent under +the same rules as any other output. However, spending an output (using +it as the input in a transaction) increases the size of a transaction, +which increases the amount of fee that needs to be paid. If the value +of the output is less than the cost of the additional fee, then it doesn't +make economic sense to spend the output. Such outputs are known as +_uneconomical outputs_. + +A zero-value output is always an uneconomical output; it wouldn't +contribute any value to a transaction spending it even if the +transaction's fee rate was zero. However, many other outputs with low +values can be uneconomical as well, even unintentionally. For example, +at a typical fee rate on the network today, an output might add more +value to a transaction than it costs to spend--but tomorrow, fee rates +might rise and make the output uneconomical. + +The need for full nodes to keep track of all UTXOs, as described in <>, means that every UTXO makes it +slightly harder to run a full node. For UTXOs containing significant +value, there's an incentive to eventually spend them, so they aren't a +problem. But there's no incentive for the person controlling an +uneconomical UTXO to ever spend it, potentially making it a perpetual +burden on operators of full nodes. Because Bitcoin's decentralization +depends on many people being willing to run full nodes, several full +node implementations such as Bitcoin Core discourage the creation of +uneconomical outputs using policies that affect the relay and mining of +unconfirmed transactions. + +[role="less_space pagebreak-before"] +The policies against relaying or mining transactions creating new +uneconomical outputs are called _dust_ policies, based on a metaphorical +comparison between outputs with very small values and particles with +very small size. Bitcoin Core's dust policy is complicated and contains +several arbitrary numbers, so many programs we're aware of simply +assume outputs with less than 546 satoshis are dust and will not be +relayed or mined by default. There are occasionally proposals to lower +dust limits, and counterproposals to raise them, so we encourage +developers using presigned transactions or multiparty protocols to +check whether the policy has changed since publication of this book. + +[TIP] +==== +Since Bitcoin's inception, every full node has needed to keep a copy of +every UTXO, but that might not always be +the case. Several developers have been working on((("Utreexo"))) Utreexo, a project +that allows full nodes to store a commitment to the set of UTXOs rather +than the data itself. A minimal commitment might be only a kilobyte or +two in size--compare that to the over five gigabytes Bitcoin Core stores +as of this writing. + +However, Utreexo will still require some nodes to store all UTXO data, +especially nodes serving miners and other operations that need to +quickly validate new blocks. That means uneconomical outputs can still +be a problem for full nodes even in a possible future where most nodes +use Utreexo. +==== + +Bitcoin Core's policy rules about dust do have one exception: output +scripts starting with +OP_RETURN+, called _data carrier outputs_, +can have a value of zero. The +OP_RETURN+ opcode causes the script to +immediately fail no matter what comes after it, so these outputs can +never be spent. That means full nodes don't need to keep track of them, +a feature Bitcoin Core takes advantage of to allow users to store small +amounts of arbitrary data in the blockchain without increasing the size +of its UTXO database. Since the outputs are unspendable, they aren't +uneconomical--any satoshis assigned to them become +permanently unspendable--so allowing the amount to be zero ensures +satoshis aren't being ((("transactions", "outputs", "amount field", startref="transaction-output-amount")))((("outputs", "amount field", startref="output-transaction-amount")))((("amount field (transaction outputs)", startref="amount-field")))((("uneconomical outputs", startref="uneconomical")))((("dust policies", startref="dust")))destroyed. + +==== Output Scripts + +The ((("transactions", "outputs", "output scripts", id="transaction-output-script")))((("outputs", "output scripts", id="output-transaction-script")))((("output scripts", id="output-script2")))output amount is followed by a compactSize integer indicating the +length of the _output script_, the script that contains the +conditions that will need to be fulfilled in order to spend the +bitcoins. According to Bitcoin's +consensus rules, the minimum size of an output script is zero. + +The consensus maximum allowed size of an output script varies depending on +when it's being checked. There's no explicit limit on the size of an +output script in the output of a transaction, but a later transaction can +only spend a previous output with a script of 10,000 bytes or +smaller. Implicitly, an output script can be almost as large as the +transaction containing it, and a transaction can be almost as large as +the block containing it. + +[[anyone-can-spend]] +[TIP] +==== +An output script with zero length can be spent by an input script containing +++OP_TRUE++. Anyone can create that input script, which means anyone +can spend an empty output script. There are an essentially unlimited +number of scripts that anyone can spend, and they are known to Bitcoin +protocol developers as _anyone can spends_. Upgrades to Bitcoin's +script language often take an existing anyone-can-spend script and add +new constraints to it, making it only spendable under the new +conditions. Application developers should never need to use an +anyone-can-spend script, but if you do, we highly recommend that you +loudly announce your plans to Bitcoin users and developers so that +future upgrades don't accidentally interfere with your system. +==== + +Bitcoin Core's policy for relaying and mining transactions effectively +limits output scripts to just a few templates, ((("standard transaction outputs")))called _standard +transaction outputs_. This was originally implemented after the +discovery of several early bugs in Bitcoin related to the Script +language and is retained in modern Bitcoin Core to support +anyone-can-spend upgrades and to encourage the best practice of placing +script conditions in P2SH redeem scripts, segwit v0 witness scripts, and +segwit v1 (taproot) leaf scripts. + +We'll look at each of the current standard transaction templates and +learn how to ((("transactions", "outputs", startref="transaction-output")))((("outputs", startref="output-transaction")))((("transactions", "outputs", "output scripts", startref="transaction-output-script")))((("outputs", "output scripts", startref="output-transaction-script")))((("output scripts", startref="output-script2")))parse scripts in <>. + +[[witness_structure]] +=== Witness Structure + +In court,((("transactions", "witnesses", id="transaction-witness")))((("witnesses", id="witness"))) a witness is someone who testifies that they saw something +important happen. Human witnesses aren't always reliable, so courts +have various processes for interrogating witnesses to (ideally) only +accept evidence from those who are reliable. + +Imagine what a witness would look like for a math problem. For example, +if the important problem was _x + 2 == 4_ and someone claimed they +witnessed the solution, what would we ask them? We'd want a +mathematical proof that showed a value that could be summed with two to +equal four. We could even omit the need for a person and just use the +proposed value for _x_ as our witness. If we were told that the witness +was _two_, then we could fill in the equation, check that it was correct, and +decide that the important problem had been solved. + +When spending bitcoins, the important problem we want to solve is +determining whether the spend was authorized by the person or people who +control those bitcoins. The thousands of full nodes that enforce +Bitcoin's consensus rules can't interrogate human witnesses, but they can +accept _witnesses_ that consist entirely of data for solving math +problems. For example, a witness of _2_ will allow spending bitcoins +protected by the following script: + +---- +2 OP_ADD 4 OP_EQUAL +---- + +Obviously, allowing your bitcoins to be spent by anyone who can solve a +simple equation wouldn't be secure. As we'll see in <>, an +unforgeable digital signature scheme uses an equation that can only be +solved by someone in possession of certain data they're able to +keep secret. They're able to reference that secret data using a public +identifier. That public identifier is ((("public keys")))((("digital signatures")))((("signatures", see="digital signatures")))called a _public key_ and a +solution to the equation is called a _signature_. + +The following script contains a public key and an opcode that requires +a corresponding signature commit to the data in the spending transaction. Like +the number _2_ in our simple example, the signature is our witness: + +---- + OP_CHECKSIG +---- + +Witnesses, the values used to solve the math problems that protect +bitcoins, need to be included in the transactions where they're used in +order for full nodes to verify them. In the legacy transaction format +used for all early Bitcoin transactions, signatures and other data are +placed in the input script field. However, when developers started to +implement contract protocols on Bitcoin, such as we saw in +<>, they discovered several significant +problems with placing witnesses in the input script field. + +==== Circular Dependencies + +Many ((("transactions", "witnesses", "circular dependencies", id="transaction-witness-circular")))((("witnesses", "circular dependencies", id="witness-circular")))((("circular dependencies", id="circular")))contract protocols for Bitcoin involve a series of transactions +that are signed out of order. For example, Alice and Bob want to +deposit funds into a script that can only be spent with signatures from +both of them, but they each also want to get their money back if the +other person becomes unresponsive. A simple solution is to sign +transactions out of order: + +- Tx~0~ pays money from Alice and money from Bob into an output with a + script that requires signatures from both Alice and Bob to spend. + +- Tx~1~ spends the previous output to two outputs, one refunding Alice + her money and one refunding Bob his money (minus a small amount for + transaction fees). + +- If Alice and Bob sign Tx~1~ before they sign Tx~0~, then they're both + guaranteed to be able to get a refund at any time. The protocol + doesn't require either of them to trust the other, making ((("trustless protocols")))it a _trustless + protocol_. + +A problem with this construction in the legacy transaction format is +that every field, including the input script field that contains +signatures, is used to derive a [.keep-together]#transaction's# identifier (txid). The +txid for Tx~0~ is part of the input's outpoint in Tx~1~. That means +there's no way for Alice and Bob to construct Tx~1~ until both +signatures for Tx~0~ are known--but if they know the signatures for +Tx~0~, one of them can broadcast that transaction before signing the +refund transaction, eliminating the guarantee of a refund. This is a +_circular dependency_. + +==== Third-Party Transaction Malleability + +A more ((("transactions", "witnesses", "third-party transaction malleability", id="transaction-witness-third-party")))((("witnesses", "third-party transaction malleability", id="witness-circular-third-party")))((("third-party transaction malleability", id="third-party")))complex series of transactions can sometimes eliminate a circular +dependency, but many protocols will then encounter a new concern: it's +often possible to solve the same script in different ways. For example, +consider our simple script from <>: + +---- +2 OP_ADD 4 OP_EQUAL +---- + +We can make this script pass by providing the value _2_ in an input script, +but there are several ways to put that value on the stack in Bitcoin. +Here are just a few: + +---- +OP_2 +OP_PUSH1 0x02 +OP_PUSH2 0x0002 +OP_PUSH3 0x000002 +... +OP_PUSHDATA1 0x0102 +OP_PUSHDATA1 0x020002 +... +OP_PUSHDATA2 0x000102 +OP_PUSHDATA2 0x00020002 +... +OP_PUSHDATA4 0x0000000102 +OP_PUSHDATA4 0x000000020002 +... +---- + +Each alternative encoding of the number _2_ in an input script will produce +a slightly different transaction with a completely different txid. Each +different version of the transaction spends the same inputs (outpoints) +as every other version of the transaction, making them all _conflict_ +with each other. Only one version of a set of conflicting transactions +can be contained within a valid blockchain. + +Imagine Alice creates one version of the transaction with +OP_2+ in the +input script and an output that pays Bob. Bob then immediately spends that +output to Carol. Anyone on the network can replace +OP_2+ with ++OP_PUSH1 0x02+, creating a conflict with Alice's original version. If +that conflicting transaction is confirmed, then there's no way to +include Alice's original version in the same blockchain, which means +there's no way for Bob's transaction to spend its output. +Bob's payment to Carol has been made invalid even though neither Alice, +Bob, nor Carol did anything wrong. Someone not involved in the +transaction (a third party) was able to change (mutate) Alice's +transaction, a problem called _unwanted third-party transaction +malleability_. + +[TIP] +==== +There are cases when people want their transactions to be malleable and +Bitcoin provides several features to support that, most notably the +signature hashes (sighash) we'll learn about in <>. For +example, Alice can use a sighash to allow Bob to help her pay some +transaction fees. This mutates Alice's transaction but only in a way +that Alice wants. For that reason, we will occasionally prefix the +word _unwanted_ to the term _transaction malleability_. Even when we +and other Bitcoin technical writers use the shorter term, we're almost +certainly talking about the unwanted variant of((("transactions", "witnesses", "third-party transaction malleability", startref="transaction-witness-third-party")))((("witnesses", "third-party transaction malleability", startref="witness-circular-third-party")))((("third-party transaction malleability", startref="third-party"))) malleability. +==== + +==== Second-Party Transaction Malleability + +When the ((("transactions", "witnesses", "second-party transaction malleability", id="transaction-witness-second-party")))((("witnesses", "second-party transaction malleability", id="witness-circular-second-party")))((("second-party transaction malleability", id="second-party")))legacy transaction format was the only transaction format, +developers worked on proposals to minimize third-party malleability, +such as BIP62. However, even if they were able to entirely eliminate +third-party malleability, users of contract protocols faced another problem: +if they required a signature from someone else involved in the protocol, +that person could generate alternative signatures and change the txid. + +For example, Alice and Bob have deposited their money into a script +requiring a signature from both of them to spend. They've also created +a refund transaction that allows each of them to get their money back at +any time. Alice decides she wants to spend just some of the +money, so she cooperates with Bob to create a chain of transactions: + +- Tx~0~ includes signatures from both Alice and Bob, spending its + bitcoins to two outputs. The first output spends some of Alice's + money; the second output returns the remainder of the bitcoins back to + the script requiring Alice and [.keep-together]#Bob's signatures.# Before signing this + transaction, they create a new refund transaction, Tx~1~. + +- Tx~1~ spends the second output of Tx~0~ to two new outputs, one to + Alice for her share of the joint funds, and one to Bob for his share. + Alice and Bob both sign this transaction before they sign Tx~0~. + +There's no circular dependency here and, if we ignore third-party +transaction malleability, this looks like it should provide us with a +trustless protocol. However, it's a property of Bitcoin signatures that +the signer has to choose a large random number when creating their +signature. Choosing a different random number will produce a different +signature even if everything being signed stays the same. It's sort of +like how, if you provide a handwritten signature for two copies of the +same contract, each of those physical signatures will look slightly +different. + +This mutability of signatures means that, if Alice tries to broadcast +Tx~0~ (which contains Bob's signature), Bob can generate an alternative +signature to create a conflicting transaction with a different txid. If +Bob's alternative version of Tx~0~ gets confirmed, then Alice can't use +the presigned version of Tx~1~ to claim her refund. This type of +mutation ((("transactions", "witnesses", "second-party transaction malleability", startref="transaction-witness-second-party")))((("witnesses", "second-party transaction malleability", startref="witness-circular-second-party")))((("second-party transaction malleability", startref="second-party")))is called _unwanted second-party transaction malleability_. + +[[segwit]] +==== Segregated Witness + +As ((("transactions", "witnesses", "segregated witness", id="transaction-witness-segwit")))((("witnesses", "segregated witness", id="witness-segwit")))((("segregated witness (segwit)", id="segregated-witness-segwit")))early as https://oreil.ly/---bp[2011], +protocol developers knew how to solve the problems of circular +dependence, third-party malleability, and second-party malleability. The +idea was to avoid including the input script in the calculation that +produces a transaction's txid. Recall that an abstract name for the data +held by an input script is a _witness_. The idea of separating the rest of +the data in a transaction from its witness for the purpose of generating +a txid is called _segregated witness_ (segwit). + +The obvious method for implementing segwit requires a +change to Bitcoin's consensus rules that would not be compatible with +older full nodes, also ((("hard forks")))((("forks", "hard forks")))called +a _hard fork_. Hard forks come with a lot of challenges, as we'll +discuss further in <>. + +An alternative approach to segwit was described in late 2015. This +would use a backward-compatible change to the consensus rules, ((("soft forks")))((("forks", "soft forks")))called a +_soft fork_. Backward compatible means that full nodes implementing +the change must not accept any blocks that full nodes without the change +would consider invalid. As long as they obey that rule, newer full +nodes can reject blocks that older full nodes would accept, giving them +the ability to enforce new consensus rules (but only if the newer full +nodes represent the economic consensus among Bitcoin users--we'll +explore the details of upgrading Bitcoin's consensus rules in +<>). + +The soft fork segwit approach is based on anyone-can-spend +output scripts. A script that starts with any of the numbers 0 to 16 +and followed by 2 to 40 bytes of data is defined as a segwit +output script template. The number indicates its version (e.g., 0 is +segwit version 0, or _segwit v0_). The data is called a _witness +program_. It's also possible to wrap the segwit template in a P2SH +commitment, but we won't deal with that in this chapter. + +From the perspective of old nodes, these output script templates can be +spent with an empty input script. From the perspective of a new node that +is aware of the new segwit rules, any payment to a segwit output script +template must only be spent with an empty input script. Notice the +difference here: old nodes _allow_ an empty input script; new nodes +_require_ an empty input script. + +[role="less_space pagebreak-before"] +An empty input script keeps witnesses from affecting the txid, eliminating +circular dependencies, third-party transaction malleability, and +second-party transaction malleability. But, with no ability to put +data in an input script, users of segwit output script templates need a +new field. That field is called the _witness structure_. + +The introduction of witness programs and the witness structure complicates Bitcoin, +but it follows an existing trend of increasing abstraction. Recall from +<> that the original Bitcoin whitepaper describes a system +where bitcoins were received to public keys (pubkeys) and spent with +signatures (sigs). The public key defined who was _authorized_ to spend +the bitcoins (whoever controlled the corresponding private key) and the +signature provided _authentication_ that the spending transaction came +from someone who controlled the private key. To make that system more +flexible, the initial release of Bitcoin introduced scripts that allow +bitcoins to be received to output scripts and spent with input scripts. +Later experience with contract protocols inspired allowing bitcoins to +be received to witness programs and spent with the witness structure. The terms and fields used in different versions of Bitcoin((("transactions", "witnesses", "segregated witness", startref="transaction-witness-segwit")))((("witnesses", "segregated witness", startref="witness-segwit")))((("segregated witness (segwit)", startref="segregated-witness-segwit"))) are shown in <>. + +++++ + + + + + + + + + + + + + + + + + + + + + + + + +
Terms used for authorization and authentication data in different parts of Bitcoin
+

Authorization

Authentication

Whitepaper

Public key

Signature

Original (Legacy)

Output script

Input script

Segwit

Witness program

Witness structure

+++++ + +==== Witness Structure Serialization + +Similar to((("transactions", "witnesses", "count", id="transaction-witness-count")))((("witnesses", "count", id="witness-count"))) the inputs and outputs fields, the witness structure contains +other fields, so we'll start with a map of those bytes from +Alice's transaction in <>. + +[[alice_tx_witness_map]] +.A byte map of the witness structure from Alice's transaction. +image::images/mbc3_0605.png["A byte map of the witness from Alice's transaction"] + +Unlike the inputs and outputs fields, the overall witness structure doesn't +start with any indication of the total number of witness stacks it contains. +Instead, this is implied by the inputs field--there's one witness +stack for every input in a transaction. + +The witness structure for a particular input does start with a count of the +number of elements they contain. Those elements are ((("witness items")))called _witness +items_. We'll explore them in detail in +<>, but for now we need to know that +each witness item is prefixed by a compactSize integer indicating its +size. + +Legacy inputs don't contain any witness items, so their witness stack +consists entirely of a count of zero (0x00). + +Alice's transaction contains one input and one ((("transactions", "witnesses", startref="transaction-witness")))((("witnesses", startref="witness")))((("transactions", "witnesses", "count", startref="transaction-witness-count")))((("witnesses", "count", startref="witness-count")))witness item. + +[[lock_time]] +=== Lock Time + +The ((("transactions", "lock time")))((("lock time")))final field in a serialized transaction is its lock time. This +field was part of Bitcoin's original serialization format, but it was +initially only enforced by Bitcoin's policy for choosing which +transactions to mine. Bitcoin's earliest known soft fork added a rule +that, starting at block height 31,000, forbid the inclusion of a +transaction in a block unless it satisfies one of the following rules: + +- The transaction indicates that it should be eligible for inclusion in + any block by setting its lock time to 0. + +- The transaction indicates that it wants to restrict which blocks it + can be included in by setting its lock time to a value less than + 500,000,000. In this case, the transaction can only be included in a + block that has a height equal to the lock time or higher. For + example, a transaction with a lock time of 123,456 can be included in + block 123,456 or any later block. + +- The transaction indicates that it wants to restrict when it can be + included in the blockchain by setting its lock time to a value of + 500,000,000 or greater. In this case, the field is parsed as epoch + time (the number of seconds since 1970-01-01T00:00 UTC) and the + transaction can only be included in a ((("median time past (MTP)")))((("MTP (median time past)")))block with a _median time past_ + (MTP) greater than the lock time. MTP is normally about an hour or + two behind the current time. The rules for MTP are described in + <>. + +[[coinbase_transactions]] +=== Coinbase Transactions + +The first ((("transactions", "coinbase", id="transaction-coinbase2")))((("coinbase transactions", id="coinbase-transaction")))((("generation transactions")))transaction in each block is a special case. Most older +documentation calls this a _generation transaction_, but most newer +documentation calls it a _coinbase transaction_ (not to be confused with +transactions created by the company named "Coinbase"). + +Coinbase transactions are created by the miner of the block that +includes them and gives the miner the option to claim any fees paid by +transactions in that block. Additionally, up until block 6,720,000, +miners are allowed to claim a subsidy consisting of bitcoins that have +never previously been circulated, called ((("block subsidy")))((("block reward")))the _block subsidy_. The total +amount a miner can claim for a block--the combination of fees and +subsidy--is called the _block reward_. + +Some of the special rules for coinbase transactions include: + +- They may only have one input. + +- The single input must have an outpoint with a null txid (consisting entirely + of zeros) and a maximal output index (0xffffffff). This prevents the + coinbase transaction from referencing a previous transaction output, + which would (at the very least) be confusing given that the coinbase + transaction pays out fees and subsidy. + +- The field that would contain an input script in a normal transaction is + called a _coinbase_. It's this field that gives the coinbase + transaction its name. The coinbase field must be at least two bytes + and not longer than 100 bytes. This script is not executed but legacy + transaction limits on the number of signature-checking operations + (sigops) do apply to it, so any arbitrary data placed in it should be + prefixed by a data-pushing opcode. Since a 2013 soft fork defined in + BIP34, the first few bytes of this field must follow additional rules + we'll describe in <>. + +- The sum of the outputs must not exceed the value of the fees collected + from all the transactions in that block plus the subsidy. The subsidy + started at 50 BTC per block and halves every 210,000 blocks + (approximately every four years). Subsidy values are rounded down to the + nearest satoshi. + +- Since the 2017 segwit soft fork documented in BIP141, any block that contains + a transaction spending a segwit output must contain an output to the + coinbase transaction that commits to all of the transactions in the + block (including their witnesses). We'll explore this commitment in + <>. + +A coinbase transaction can have any other outputs that would be valid in +a normal transaction. However, a transaction spending one of those +outputs cannot be included in any block until after the coinbase +transaction has received 100 confirmations. This is called the +_maturity rule_, and ((("maturity rule")))coinbase transaction outputs that don't yet have +100 confirmations are called _immature_. + +//TODO:stretch goal to describe the reason for the maturity rule and, +//by extension the reason for no expiring timelocks + +Most Bitcoin software doesn't need to deal with coinbase transactions, +but their special nature does mean they can occasionally be the cause of +unusual problems in software that's not designed to expect((("transactions", "coinbase", startref="transaction-coinbase2")))((("coinbase transactions", startref="coinbase-transaction"))) them. + +// Useful content deleted +// - no input amount in transactions +// - no balances in transactions +// - UTXO model theory? +// Coin selection +// Change +// Inability for lightweight clients to get old UTXOs + +=== Weight and Vbytes + +Each ((("transactions", "weights", id="transactions-weight")))((("weights (of transactions)", id="weights")))((("vbytes", id="vbytes")))Bitcoin block is limited in the amount of transaction data it can +contain, so most Bitcoin software needs to be able to measure the +transactions it creates or processes. The modern unit of measurement +for Bitcoin is called _weight_. An alternative version of weight is +_vbytes_, where four units of weight equal one vbyte, providing an easy +comparison to the original _byte_ measurement unit used in legacy +Bitcoin blocks. + +Blocks are limited to 4 million weight. The block header takes up 240 +weight. An additional field, the transaction count, uses either 4 or +12 weight. All of the remaining weight may be used for transaction +data. + +To calculate the weight of a particular field in a transaction, the size +of that serialized field in bytes is multiplied by a factor. To +calculate the weight of a transaction, sum together the weights of all +of its fields. The factors for each of the fields in a transaction are +shown in <>. To provide an example, we also calculate +the weight of each field in this chapter's example transaction from +Alice to Bob. + +The factors, and the fields to which they are applied, were chosen to +reduce the weight used when spending a UTXO. This helps discourage the +creation of uneconomical outputs as described in +<>. + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Weight factors for all fields in a Bitcoin transaction

Field

Factor

Weight in Alice’s Tx

Version

4

16

Marker & Flag

1

2

Inputs Count

4

4

Outpoint

4

144

Input script

4

4

Sequence

4

16

Outputs Count

4

4

Amount

4

64 (2 outputs)

Output script

4

232 (2 outputs with different scripts)

Witness Count

1

1

Witness items

1

66

Lock time

4

16

Total

N/A

569

+++++ + +[role="less_space pagebreak-before"] +We can verify our weight calculation by getting the total for Alice's +transaction from Bitcoin Core: + +---- +$ bitcoin-cli getrawtransaction 466200308696215bbc949d5141a49a41\ +38ecdfdfaa2a8029c1f9bcecd1f96177 2 | jq .weight +569 +---- + +Alice's transaction from <> at the beginning of +this chapter is shown represented in weight units in +<>. You can see the factor at work by comparing +the difference in size between the various fields in the ((("transactions", "weights", startref="transactions-weight")))((("weights (of transactions)", startref="weights")))((("vbytes", startref="vbytes")))two images. + +[[alice_tx_weight_map]] +.A byte map of Alice's transaction. +image::images/mbc3_0606.png["A weight map of Alice's transaction"] + +[[legacy_serialization]] +=== Legacy Serialization + +The ((("transactions", "legacy serialization")))((("legacy serialization")))serialization format described in this chapter is used for the +majority of new Bitcoin transactions as of the writing of this book, but +an older serialization format is still used for many transactions. That +older format, called _legacy serialization_, must be used on the Bitcoin +P2P network for any transaction with an empty witness structure (which is only +valid if the transaction doesn't spend any witness programs). + +Legacy serialization does not include the marker, flag, and witness structure +fields. + +++++ +

+In this chapter, we looked at each of the fields in a transaction and +discovered how they communicate to full nodes the details about the +bitcoins to be transferred between users. We only briefly looked at the +output script, input script, and witness structure that allow specifying and +satisfying conditions that restrict who can spend what bitcoins. +Understanding how to construct and use these conditions is essential to +ensuring that only Alice can spend her bitcoins, so they will be the +subject of the next chapter.

+++++ + +//TODO:text long section or full chapter about psbts diff --git a/ch07.asciidoc b/ch07.asciidoc deleted file mode 100644 index 543034c6..00000000 --- a/ch07.asciidoc +++ /dev/null @@ -1,1069 +0,0 @@ -[[ch07]] -[[adv_transactions]] -== Advanced Transactions and Scripting - -[[ch07_intro]] -=== Introduction - -In the previous chapter, we introduced the basic elements of bitcoin transactions and looked at the most common type of transaction script, the P2PKH script. In this chapter we will look at more advanced scripting and how we can use it to build transactions with complex conditions. - -First, we will look at _multisignature_ scripts. Next, we will examine the second most common transaction script, _Pay-to-Script-Hash_, which opens up a whole world of complex scripts. Then, we will examine new script operators that add a time dimension to bitcoin, through _timelocks_. Finally, we will look at _Segregated Witness_, an architectural change to the structure of transactions. - -[[multisig]] -=== Multisignature - -((("transactions", "advanced", "multisignature scripts")))((("transactions", "advanced", id="Tadv07")))((("scripting", "multisignature scripts", id="Smulti07")))((("multisignature scripts")))Multisignature scripts set a condition where N public keys are recorded in the script and at least M of those must provide signatures to unlock the funds. This is also known as an M-of-N scheme, where N is the total number of keys and M is the threshold of signatures required for validation. For example, a 2-of-3 multisignature is one where three public keys are listed as potential signers and at least two of those must be used to create signatures for a valid transaction to spend the funds. - -At this time, _standard_ multisignature scripts are limited to at most 3 listed public keys, meaning you can do anything from a 1-of-1 to a 3-of-3 multisignature or any combination within that range. The limitation to 3 listed keys might be lifted by the time this book is published, so check the +IsStandard()+ function to see what is currently accepted by the network. Note that the limit of 3 keys applies only to standard (also known as "bare") multisignature scripts, not to multisignature scripts wrapped in a Pay-to-Script-Hash (P2SH) script. P2SH multisignature scripts are limited to 15 keys, allowing for up to 15-of-15 multisignature. This limitation is also imposed by the +IsStandard()+ function. We will learn about P2SH in <>. - -The general form of a locking script setting an M-of-N multisignature condition is: - ----- -M ... N CHECKMULTISIG ----- - -where N is the total number of listed public keys and M is the threshold of required signatures to spend the output. - -A locking script setting a 2-of-3 multisignature condition looks like this: - ----- -2 3 CHECKMULTISIG ----- - -The preceding locking script can be satisfied with an unlocking script containing any combination of two signatures from the private keys corresponding to the three listed public keys: - ----- - ----- - -The two scripts together would form the combined validation script: - ----- - 2 3 CHECKMULTISIG ----- - -When executed, this combined script will evaluate to TRUE if, and only if, the unlocking script matches the conditions set by the locking script. In this case, the condition is whether the unlocking script has a valid signature from the two private keys that correspond to two of the three public keys set as an encumbrance. - -[[multisig_bug]] -===== A bug in CHECKMULTISIG execution - -((("scripting", "multisignature scripts", "CHECKMULTISIG bug")))((("CHECKMULTISIG bug workaround")))There is a bug in ++CHECKMULTISIG++'s execution that requires a slight workaround. When +CHECKMULTISIG+ executes, it should consume M+N+2 items on the stack as parameters. However, due to the bug, +CHECKMULTISIG+ will pop an extra value or one value more than expected. - -Let's look at this in greater detail using the previous validation example: - ----- - 2 3 CHECKMULTISIG ----- - -First, +CHECKMULTISIG+ pops the top item, which is +N+ (in this example "3"). Then it pops +N+ items, which are the public keys that can sign. In this example, public keys A, B, and C. Then, it pops one item, which is +M+, the quorum (how many signatures are needed). Here M = 2. At this point, +CHECKMULTISIG+ should pop the final +M+ items, which are the signatures, and see if they are valid. However, unfortunately, a bug in the implementation causes +CHECKMULTISIG+ to pop one more item (M+1 total) than it should. The extra item is disregarded when checking the signatures so it has no direct effect on +CHECKMULTISIG+ itself. However, an extra value must be present because if it is not present, when +CHECKMULTISIG+ attempts to pop on an empty stack, it will cause a stack error and script failure (marking the transaction as invalid). Because the extra item is disregarded it can be anything, but customarily +0+ is used. - -Because this bug became part of the consensus rules, it must now be replicated forever. Therefore the correct script validation would look like this: - ----- -0 2 3 CHECKMULTISIG ----- - -Thus the unlocking script actually used in multisig is not: - ----- - ----- - -but instead it is: - ----- -0 ----- - -From now on, if you see a multisig unlocking script, you should expect to see an extra +0+ in the beginning, whose only purpose is as a workaround to a bug that accidentally became a consensus rule.((("", startref="Smulti07"))) - -[[p2sh]] -=== Pay-to-Script-Hash (P2SH) - -((("transactions", "advanced", "Pay-to-Script-Hash")))((("scripting", "Pay-to-Script-Hash", id="Spay07")))Pay-to-Script-Hash (P2SH) was introduced in 2012 as a powerful new type of transaction that greatly simplifies the use of complex transaction scripts. To explain the need for P2SH, let's look at a practical example. - -((("use cases", "import/export", id="mohamseven")))((("scripting", "Pay-to-Script-Hash", "import/export example")))((("Pay-to-Script-Hash (P2SH)", "import/export example")))In <> we introduced Mohammed, an electronics importer based in Dubai. Mohammed's company uses bitcoin's multisignature feature extensively for its corporate accounts. Multisignature scripts are one of the most common uses of bitcoin's advanced scripting capabilities and are a very powerful feature. ((("accounts receivable (AR)")))Mohammed's company uses a multisignature script for all customer payments, known in accounting terms as "accounts receivable," or AR. With the multisignature scheme, any payments made by customers are locked in such a way that they require at least two signatures to release, from Mohammed and one of his partners or from his attorney who has a backup key. A multisignature scheme like that offers corporate governance controls and protects against theft, embezzlement, or loss. - -The resulting script is quite long and looks like this: - ----- -2 5 CHECKMULTISIG ----- - -Although multisignature scripts are a powerful feature, they are cumbersome to use. Given the preceding script, Mohammed would have to communicate this script to every customer prior to payment. Each customer would have to use special bitcoin wallet software with the ability to create custom transaction scripts, and each customer would have to understand how to create a transaction using custom scripts. Furthermore, the resulting transaction would be about five times larger than a simple payment transaction, because this script contains very long public keys. The burden of that extra-large transaction would be borne by the customer in the form of fees. Finally, a large transaction script like this would be carried in the UTXO set in RAM in every full node, until it was spent. All of these issues make using complex locking scripts difficult in practice. - -P2SH was developed to resolve these practical difficulties and to make the use of complex scripts as easy as a payment to a Bitcoin address. With P2SH payments, the complex locking script is replaced with its digital fingerprint, a cryptographic hash. When a transaction attempting to spend the UTXO is presented later, it must contain the script that matches the hash, in addition to the unlocking script. In simple terms, P2SH means "pay to a script matching this hash, a script that will be presented later when this output is spent." - -((("redeem scripts")))((("scripting", "redeem scripts")))In P2SH transactions, the locking script that is replaced by a hash is referred to as the _redeem script_ because it is presented to the system at redemption time rather than as a locking script. <> shows the script without P2SH and <> shows the same script encoded with P2SH. - -[[without_p2sh]] -.Complex script without P2SH -|======= -| Locking Script | 2 PubKey1 PubKey2 PubKey3 PubKey4 PubKey5 5 CHECKMULTISIG -| Unlocking Script | 0 Sig1 Sig2 -|======= - -[[with_p2sh]] -.Complex script as P2SH -|======= -| Redeem Script | 2 PubKey1 PubKey2 PubKey3 PubKey4 PubKey5 5 CHECKMULTISIG -| Locking Script | HASH160 <20-byte hash of redeem script> EQUAL -| Unlocking Script | 0 Sig1 Sig2 -|======= - -As you can see from the tables, with P2SH the complex script that details the conditions for spending the output (redeem script) is not presented in the locking script. Instead, only a hash of it is in the locking script and the redeem script itself is presented later, as part of the unlocking script when the output is spent. This shifts the burden in fees and complexity from the sender (who creates the transaction) to the recipient (who unlocks and spends the transaction). - -Let's look at Mohammed's company, the complex multisignature script, and the resulting P2SH scripts. - -First, the multisignature script that Mohammed's company uses for all incoming payments from customers: - ----- -2 5 CHECKMULTISIG ----- - -If the placeholders are replaced by actual public keys (shown here as 520-bit numbers starting with 04) you can see that this script becomes very long: - ----- -2 -04C16B8698A9ABF84250A7C3EA7EEDEF9897D1C8C6ADF47F06CF73370D74DCCA01CDCA79DCC5C395D7EEC6984D83F1F50C900A24DD47F569FD4193AF5DE762C58704A2192968D8655D6A935BEAF2CA23E3FB87A3495E7AF308EDF08DAC3C1FCBFC2C75B4B0F4D0B1B70CD2423657738C0C2B1D5CE65C97D78D0E34224858008E8B49047E63248B75DB7379BE9CDA8CE5751D16485F431E46117B9D0C1837C9D5737812F393DA7D4420D7E1A9162F0279CFC10F1E8E8F3020DECDBC3C0DD389D99779650421D65CBD7149B255382ED7F78E946580657EE6FDA162A187543A9D85BAAA93A4AB3A8F044DADA618D087227440645ABE8A35DA8C5B73997AD343BE5C2AFD94A5043752580AFA1ECED3C68D446BCAB69AC0BA7DF50D56231BE0AABF1FDEEC78A6A45E394BA29A1EDF518C022DD618DA774D207D137AAB59E0B000EB7ED238F4D800 5 CHECKMULTISIG ----- - -This entire script can instead be represented by a 20-byte cryptographic hash, by first applying the SHA256 hashing algorithm and then applying the RIPEMD160 algorithm on the result. - -We use +libbitcoin-explorer+ (+bx+) on the command-line to produce the script hash, as follows: - ----- -echo \ -2 \ -[04C16B8698A9ABF84250A7C3EA7EEDEF9897D1C8C6ADF47F06CF73370D74DCCA01CDCA79DCC5C395D7EEC6984D83F1F50C900A24DD47F569FD4193AF5DE762C587] \ -[04A2192968D8655D6A935BEAF2CA23E3FB87A3495E7AF308EDF08DAC3C1FCBFC2C75B4B0F4D0B1B70CD2423657738C0C2B1D5CE65C97D78D0E34224858008E8B49] \ -[047E63248B75DB7379BE9CDA8CE5751D16485F431E46117B9D0C1837C9D5737812F393DA7D4420D7E1A9162F0279CFC10F1E8E8F3020DECDBC3C0DD389D9977965] \ -[0421D65CBD7149B255382ED7F78E946580657EE6FDA162A187543A9D85BAAA93A4AB3A8F044DADA618D087227440645ABE8A35DA8C5B73997AD343BE5C2AFD94A5] \ -[043752580AFA1ECED3C68D446BCAB69AC0BA7DF50D56231BE0AABF1FDEEC78A6A45E394BA29A1EDF518C022DD618DA774D207D137AAB59E0B000EB7ED238F4D800] \ -5 CHECKMULTISIG \ -| bx script-encode | bx sha256 | bx ripemd160 -54c557e07dde5bb6cb791c7a540e0a4796f5e97e ----- - -The series of commands above first encodes Mohammed's multisig redeem script as a serialized hex-encoded bitcoin Script. The next +bx+ command calculates the SHA256 hash of that. The next +bx+ command hashes again with RIPEMD160, producing the final script-hash: - -The 20-byte hash of Mohammed's redeem script is: - ----- -54c557e07dde5bb6cb791c7a540e0a4796f5e97e ----- - -A P2SH transaction locks the output to this hash instead of the longer redeem script, using the locking script: - ----- -HASH160 54c557e07dde5bb6cb791c7a540e0a4796f5e97e EQUAL ----- - -which, as you can see, is much shorter. Instead of "pay to this 5-key multisignature script," the P2SH equivalent transaction is "pay to a script with this hash." A customer making a payment to Mohammed's company need only include this much shorter locking script in his payment. When Mohammed and his partners want to spend this UTXO, they must present the original redeem script (the one whose hash locked the UTXO) and the signatures necessary to unlock it, like this: - ----- - <2 PK1 PK2 PK3 PK4 PK5 5 CHECKMULTISIG> ----- - -The two scripts are combined in two stages. First, the redeem script is checked against the locking script to make sure the hash matches: - ----- -<2 PK1 PK2 PK3 PK4 PK5 5 CHECKMULTISIG> HASH160 EQUAL ----- -If the redeem script hash matches, the unlocking script is executed on its own, to unlock the redeem script: - ----- - 2 PK1 PK2 PK3 PK4 PK5 5 CHECKMULTISIG ----- - -Almost all the scripts described in this chapter can only be implemented as P2SH scripts. For example, a 2 of 5 standard multisignature locking script cannot be used directly in the locking script of an UTXO, as +IsStandard()+ would invalidate the transaction. To conform, a P2SH locking script can be used instead, as seen above. A transaction that then includes a P2SH unlocking script can be used to redeem this UTXO and will be valid so long as it does not contain more than 15 public keys. ((("", startref="mohamseven"))) - -[TIP] -==== -Remember, because of policy set forth by the +IsStandard()+ function at the time of this writing, standard multisignature scripts are limited to at most 3 listed public keys, while P2SH scripts are limited to at most 15 listed public keys. Standard multisignature scripts can invalidate transactions by way of their locking _or_ unlocking script, while P2SH scripts can invalidate transactions by way of their unlocking script _only_. This is because there is no way for +IsStandard()+ to tell if a hash of a redeem script in a locking script includes more signatures than the currently imposed size limitation, so it can only observe the unlocking scripts in transaction inputs. -==== - -==== P2SH Addresses - -((("scripting", "Pay-to-Script-Hash", "addresses")))((("Pay-to-Script-Hash (P2SH)", "addresses")))((("bitcoin improvement proposals", "Address Format for P2SH (BIP-13)")))Another important part of the P2SH feature is the ability to encode a script hash as an address, as defined in BIP-13. P2SH addresses are Base58Check encodings of the 20-byte hash of a script, just like Bitcoin addresses are Base58Check encodings of the 20-byte hash of a public key. P2SH addresses use the version prefix "5," which results in Base58Check-encoded addresses that start with a "3." - -For example, Mohammed's complex script, hashed and Base58Check-encoded as a P2SH address, becomes +39RF6JqABiHdYHkfChV6USGMe6Nsr66Gzw+. We can confirm that with the +bx+ command: - ----- -echo \ -'54c557e07dde5bb6cb791c7a540e0a4796f5e97e'\ - | bx address-encode -v 5 -39RF6JqABiHdYHkfChV6USGMe6Nsr66Gzw ----- - - -Now, Mohammed can give this "address" to his customers and they can use almost any bitcoin wallet to make a simple payment, as if it were a Bitcoin address. The 3 prefix gives them a hint that this is a special type of address, one corresponding to a script instead of a public key, but otherwise it works in exactly the same way as a payment to a Bitcoin address. - -P2SH addresses hide all of the complexity, so that the person making a payment does not see the script. - -==== Benefits of P2SH - -((("scripting", "Pay-to-Script-Hash", "benefits of")))((("Pay-to-Script-Hash (P2SH)", "benefits of")))The P2SH feature offers the following benefits compared to the direct use of complex scripts in locking outputs: - -* Complex scripts are replaced by shorter fingerprints in the transaction output, making the transaction smaller. -* Scripts can be coded as an address, so the sender and the sender's wallet don't need complex engineering to implement P2SH. -* P2SH shifts the burden of constructing the script to the recipient, not the sender. -* P2SH shifts the burden in data storage for the long script from the output (which additionally to being stored on the blockchain is in the UTXO set) to the input (only stored on the blockchain). -* P2SH shifts the burden in data storage for the long script from the present time (payment) to a future time (when it is spent). -* P2SH shifts the higher transaction fee costs of a long script from the sender to the recipient, who has to include the long redeem script to spend it. - -==== Redeem Script and Validation - -((("scripting", "Pay-to-Script-Hash", "redeem scripts and validation")))((("Pay-to-Script-Hash (P2SH)", "redeem scripts and validation")))((("redeem scripts")))((("validation")))Prior to version 0.9.2 of the Bitcoin Core client, Pay-to-Script-Hash was limited to the standard types of bitcoin transaction scripts, by the +IsStandard()+ function. That means that the redeem script presented in the spending transaction could only be one of the standard types: P2PK, P2PKH, or multisig. - -As of version 0.9.2 of the Bitcoin Core client, P2SH transactions can contain any valid script, making the P2SH standard much more flexible and allowing for experimentation with many novel and complex types of transactions. - -You are not able to put a P2SH inside a P2SH redeem script, because the P2SH specification is not recursive. Also, while it is technically possible to include +RETURN+ (see <>) in a redeem script, as nothing in the rules prevents you from doing so, it is of no practical use because executing +RETURN+ during validation will cause the transaction to be marked invalid. - -Note that because the redeem script is not presented to the network until you attempt to spend a P2SH output, if you lock an output with the hash of an invalid redeem script it will be processed regardless. The UTXO will be successfully locked. However, you will not be able to spend it because the spending transaction, which includes the redeem script, will not be accepted because it is an invalid script. This creates a risk, because you can lock bitcoin in a P2SH that cannot be spent later. The network will accept the P2SH locking script even if it corresponds to an invalid redeem script, because the script hash gives no indication of the script it represents.((("", startref="Spay07"))) - -[WARNING] -==== -((("warnings and cautions", "accidental bitcoin locking")))P2SH locking scripts contain the hash of a redeem script, which gives no clues as to the content of the redeem script itself. The P2SH transaction will be considered valid and accepted even if the redeem script is invalid. You might accidentally lock bitcoin in such a way that it cannot later be spent. -==== - - - -[[op_return]] -=== Data Recording Output (RETURN) - -((("transactions", "advanced", "data recording output")))((("scripting", "data recording output")))((("RETURN operator")))((("data recording (nonpayment data)")))((("nonpayment data")))((("blockchain (the)", "nonpayment data recording")))((("digital notary services")))Bitcoin's distributed and timestamped ledger, the blockchain, has potential uses far beyond payments. Many developers have tried to use the transaction scripting language to take advantage of the security and resilience of the system for applications such as digital notary services, stock certificates, and smart contracts. Early attempts to use bitcoin's script language for these purposes involved creating transaction outputs that recorded data on the blockchain; for example, to record a digital fingerprint of a file in such a way that anyone could establish proof-of-existence of that file on a specific date by reference to that transaction. - -((("blockchain bloat")))((("bloat")))((("unspent transaction outputs (UTXO)")))((("UTXO sets")))The use of bitcoin's blockchain to store data unrelated to bitcoin payments is a controversial subject. Many developers consider such use abusive and want to discourage it. Others view it as a demonstration of the powerful capabilities of blockchain technology and want to encourage such experimentation. Those who object to the inclusion of nonpayment data argue that it causes "blockchain bloat," burdening those running full Bitcoin nodes with carrying the cost of disk storage for data that the blockchain was not intended to carry. Moreover, such transactions create UTXO that cannot be spent, using the destination Bitcoin address as a freeform 20-byte field. Because the address is used for data, it doesn't correspond to a private key and the resulting UTXO can _never_ be spent; it's a fake payment. These transactions that can never be spent are therefore never removed from the UTXO set and cause the size of the UTXO database to forever increase, or "bloat." - -In version 0.9 of the Bitcoin Core client, a compromise was reached with the introduction of the +RETURN+ operator. +RETURN+ allows developers to add 80 bytes of nonpayment data to a transaction output. However, unlike the use of "fake" UTXO, the +RETURN+ operator creates an explicitly _provably unspendable_ output, which does not need to be stored in the UTXO set. +RETURN+ outputs are recorded on the blockchain, so they consume disk space and contribute to the increase in the blockchain's size, but they are not stored in the UTXO set and therefore do not bloat the UTXO memory pool and burden full nodes with the cost of more expensive RAM. - -+RETURN+ scripts look like this: - ----- -RETURN ----- - -((("Proof of Existence")))((("DOCPROOF prefix")))The data portion is limited to 80 bytes and most often represents a hash, such as the output from the SHA256 algorithm (32 bytes). Many applications put a prefix in front of the data to help identify the application. For example, the https://proofofexistence.com[Proof of Existence] digital notarization service uses the 8-byte prefix +DOCPROOF+, which is ASCII encoded as +44 4f 43 50 52 4f 4f 46+ in hexadecimal. - -Keep in mind that there is no "unlocking script" that corresponds to +RETURN+ that could possibly be used to "spend" a +RETURN+ output. The whole point of +RETURN+ is that you can't spend the money locked in that output, and therefore it does not need to be held in the UTXO set as potentially spendable—+RETURN+ is _provably unspendable_. +RETURN+ is usually an output with a zero bitcoin amount, because any bitcoin assigned to such an output is effectively lost forever. If a +RETURN+ is referenced as an input in a transaction, the script validation engine will halt the execution of the validation script and mark the transaction as invalid. The execution of +RETURN+ essentially causes the script to "RETURN" with a +FALSE+ and halt. Thus, if you accidentally reference a +RETURN+ output as an input in a transaction, that transaction is invalid. - -A standard transaction (one that conforms to the +IsStandard()+ checks) can have only one +RETURN+ output. However, a single +RETURN+ output can be combined in a transaction with outputs of any other type. - -Two new command-line options have been added in Bitcoin Core as of version 0.10. The option +datacarrier+ controls relay and mining of +RETURN+ transactions, with the default set to "1" to allow them. The option +datacarriersize+ takes a numeric argument specifying the maximum size in bytes of the +RETURN+ script, 83 bytes by default, which, allows for a maximum of 80 bytes of +RETURN+ data plus one byte of +RETURN+ opcode and two bytes of +PUSHDATA+ opcode. - -[NOTE] -==== -+RETURN+ was initially proposed with a limit of 80 bytes, but the limit was reduced to 40 bytes when the feature was released. In February 2015, in version 0.10 of Bitcoin Core, the limit was raised back to 80 bytes. Nodes may choose not to relay or mine +RETURN+, or only relay and mine +RETURN+ containing less than 80 bytes of data. -==== - -=== Timelocks - -((("transactions", "advanced", "timelocks")))((("scripting", "timelocks", id="Stimelock07")))((("nLocktime field")))((("scripting", "timelocks", "uses for")))((("timelocks", "uses for")))Timelocks are restrictions on transactions or outputs that only allow spending after a point in time. Bitcoin has had a transaction-level timelock feature from the beginning. It is implemented by the +nLocktime+ field in a transaction. Two new timelock features were introduced in late 2015 and mid-2016 that offer UTXO-level timelocks. These are +CHECKLOCKTIMEVERIFY+ and +CHECKSEQUENCEVERIFY+. - -Timelocks are useful for postdating transactions and locking funds to a date in the future. More importantly, timelocks extend bitcoin scripting into the dimension of time, opening the door for complex multistep smart contracts. - -[[transaction_locktime_nlocktime]] -==== Transaction Locktime (nLocktime) - -((("scripting", "timelocks", "nLocktime")))((("timelocks", "nLocktime")))From the beginning, bitcoin has had a transaction-level timelock feature. Transaction locktime is a transaction-level setting (a field in the transaction data structure) that defines the earliest time that a transaction is valid and can be relayed on the network or added to the blockchain. Locktime is also known as +nLocktime+ from the variable name used in the Bitcoin Core codebase. It is set to zero in most transactions to indicate immediate propagation and execution. If +nLocktime+ is nonzero and below 500 million, it is interpreted as a block height, meaning the transaction is not valid and is not relayed or included in the blockchain prior to the specified block height. If it is greater than or equal to 500 million, it is interpreted as a Unix Epoch timestamp (seconds since Jan-1-1970) and the transaction is not valid prior to the specified time. Transactions with +nLocktime+ specifying a future block or time must be held by the originating system and transmitted to the Bitcoin network only after they become valid. If a transaction is transmitted to the network before the specified +nLocktime+, the transaction will be rejected by the first node as invalid and will not be relayed to other nodes. The use of +nLocktime+ is equivalent to postdating a paper check. - -[[locktime_limitations]] -===== Transaction locktime limitations - -+nLocktime+ has the limitation that while it makes it possible to spend some outputs in the future, it does not make it impossible to spend them until that time. Let's explain that with the following example. - -((("use cases", "buying coffee", id="alicesseven")))Alice signs a transaction spending one of her outputs to Bob's address, and sets the transaction +nLocktime+ to 3 months in the future. Alice sends that transaction to Bob to hold. With this transaction Alice and Bob know that: - -* Bob cannot transmit the transaction to redeem the funds until 3 months have elapsed. -* Bob may transmit the transaction after 3 months. - -However: - -* Alice can create another transaction, double-spending the same inputs without a locktime. Thus, Alice can spend the same UTXO before the 3 months have elapsed. -* Bob has no guarantee that Alice won't do that. - -It is important to understand the limitations of transaction +nLocktime+. The only guarantee is that Bob will not be able to redeem it before 3 months have elapsed. There is no guarantee that Bob will get the funds. To achieve such a guarantee, the timelock restriction must be placed on the UTXO itself and be part of the locking script, rather than on the transaction. This is achieved by the next form of timelock, called Check Lock Time Verify. - -==== Check Lock Time Verify (CLTV) - -((("Check Lock Time Verify (CLTV)", id="cltv07")))((("timelocks", "Check Lock Time Verify (CLTV)")))((("scripting", "timelocks", "Check Lock Time Verify (CLTV)")))((("bitcoin improvement proposals", "CHECKLOCKTIMEVERIFY (BIP-65)")))In December 2015, a new form of timelock was introduced to bitcoin as a soft fork upgrade. Based on a specification in BIP-65, a new script operator called _CHECKLOCKTIMEVERIFY_ (_CLTV_) was added to the scripting language. +CLTV+ is a per-output timelock, rather than a per-transaction timelock as is the case with +nLocktime+. This allows for much greater flexibility in the way timelocks are applied. - -In simple terms, by adding the +CLTV+ opcode in the redeem script of an output it restricts the output, so that it can only be spent after the specified time has elapsed. - -[TIP] -==== -While +nLocktime+ is a transaction-level timelock, +CLTV+ is an output-based timelock. -==== - -+CLTV+ doesn't replace +nLocktime+, but rather restricts specific UTXO such that they can only be spent in a future transaction with +nLocktime+ set to a greater or equal value. - -The +CLTV+ opcode takes one parameter as input, expressed as a number in the same format as +nLocktime+ (either a block height or Unix epoch time). As indicated by the +VERIFY+ suffix, +CLTV+ is the type of opcode that halts execution of the script if the outcome is +FALSE+. If it results in TRUE, execution continues. - -In order to lock an output with +CLTV+, you insert it into the redeem script of the output in the transaction that creates the output. For example, if Alice is paying Bob's address, the output would normally contain a P2PKH script like this: - ----- -DUP HASH160 EQUALVERIFY CHECKSIG ----- - -To lock it to a time, say 3 months from now, the transaction would be a P2SH transaction with a redeem script like this: - ----- - CHECKLOCKTIMEVERIFY DROP DUP HASH160 EQUALVERIFY CHECKSIG ----- - -where ++ is a block height or time value estimated 3 months from the time the transaction is mined: current block height {plus} 12,960 (blocks) or current Unix epoch time {plus} 7,760,000 (seconds). For now, don't worry about the +DROP+ opcode that follows +CHECKLOCKTIMEVERIFY+; it will be explained shortly. - -When Bob tries to spend this UTXO, he constructs a transaction that references the UTXO as an input. He uses his signature and public key in the unlocking script of that input and sets the transaction +nLocktime+ to be equal to or greater than the timelock in the +CHECKLOCKTIMEVERIFY+ Alice set. Bob then broadcasts the transaction on the Bitcoin network. - -Bob's transaction is evaluated as follows. If the +CHECKLOCKTIMEVERIFY+ parameter Alice set is less than or equal to the spending transaction's +nLocktime+, script execution continues (acts as if a “no operation” or NOP opcode was executed). Otherwise, script execution halts and the transaction is deemed invalid. - -More precisely, +CHECKLOCKTIMEVERIFY+ fails and halts execution, marking the transaction invalid if (source: BIP-65): - -1. the stack is empty; or -2. the top item on the stack is less than 0; or -3. the timelock type (height versus timestamp) of the top stack item and the +nLocktime+ field are not the same; or -4. the top stack item is greater than the transaction's +nLocktime+ field; or -5. the +nSequence+ field of the input is 0xffffffff. - -[NOTE] -==== -+CLTV+ and +nLocktime+ use the same format to describe timelocks, either a block height or the time elapsed in seconds since Unix epoch. Critically, when used together, the format of +nLocktime+ must match that of +CLTV+ in the outputs—they must both reference either block height or time in seconds. -==== - -After execution, if +CLTV+ is satisfied, the time parameter that preceded it remains as the top item on the stack and may need to be dropped, with +DROP+, for correct execution of subsequent script opcodes. You will often see +CHECKLOCKTIMEVERIFY+ followed by +DROP+ in scripts for this reason. - -By using +nLocktime+ in conjunction with +CLTV+, the scenario described in <> changes. Alice can no longer spend the money (because it's locked with Bob's key) and Bob cannot spend it before the 3-month locktime has expired.((("", startref="alicesseven"))) - -By introducing timelock functionality directly into the scripting language, +CLTV+ allows us to develop some very interesting complex scripts.((("", startref="cltv07"))) - -The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki[BIP-65 (CHECKLOCKTIMEVERIFY)]. - -==== Relative Timelocks - -+nLocktime+ and +CLTV+ are ((("timelocks", "relative timelocks", id="Trelative07")))((("scripting", "timelocks", "relative timelocks")))((("relative timelocks", id="relativetime07")))both _absolute timelocks_ in that they specify an absolute point in time. The next two timelock features we will examine are _relative timelocks_ in that they specify, as a condition of spending an output, an elapsed time from the confirmation of the output in the blockchain. - -Relative timelocks are useful because they allow a chain of two or more interdependent transactions to be held off chain, while imposing a time constraint on one transaction that is dependent on the elapsed time from the confirmation of a previous transaction. In other words, the clock doesn't start counting until the UTXO is recorded on the blockchain. This functionality is especially useful in bidirectional state channels and Lightning Networks, as we will see in <>. - -Relative timelocks, like absolute timelocks, are implemented with both a transaction-level feature and a script-level opcode. The transaction-level relative timelock is implemented as a consensus rule on the value of +nSequence+, a transaction field that is set in every transaction input. Script-level relative timelocks are implemented with the +CHECKSEQUENCEVERIFY+ (CSV) opcode. - -((("bitcoin improvement proposals", "Relative lock-time using consensus-enforced sequence numbers (BIP-68)")))((("bitcoin improvement proposals", "CHECKSEQUENCEVERIFY (BIP-112)")))Relative timelocks are implemented according to the specifications in https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68, Relative lock-time using consensus-enforced sequence numbers] and https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[BIP-112, CHECKSEQUENCEVERIFY]. - -BIP-68 and BIP-112 were activated in May 2016 as a soft fork upgrade to the consensus rules. - -==== Relative Timelocks with nSequence - -((("nSequence field")))((("scripting", "timelocks", "relative timelocks with nSequence")))Relative timelocks can be set on each input of a transaction, by setting the +nSequence+ field in each input. - -===== Original meaning of nSequence - -The +nSequence+ field was originally intended (but never properly implemented) to allow modification of transactions in the mempool. In that use, a transaction containing inputs with +nSequence+ value below 2^32^ - 1 (0xFFFFFFFF) indicated a transaction that was not yet "finalized." Such a transaction would be held in the mempool until it was replaced by another transaction spending the same inputs with a higher +nSequence+ value. Once a transaction was received whose inputs had an +nSequence+ value of 0xFFFFFFFF it would be considered "finalized" and mined. - -The original meaning of +nSequence+ was never properly implemented and the value of +nSequence+ is customarily set to 0xFFFFFFFF in transactions that do not utilize timelocks. For transactions with +nLocktime+ or +CHECKLOCKTIMEVERIFY+, the +nSequence+ value must be set to less than 2^31^ for the timelock guards to have an effect, as explained below. - -===== nSequence as a consensus-enforced relative timelock - -Since the activation of BIP-68, new consensus rules apply for any transaction containing an input whose +nSequence+ value is less than 2^31^ (bit 1<<31 is not set). Programmatically, that means that if the most significant bit (bit 1<<31) is not set, it is a flag that means "relative locktime." Otherwise (bit 1<<31 set), the +nSequence+ value is reserved for other uses such as enabling +CHECKLOCKTIMEVERIFY+, +nLocktime+, Opt-In-Replace-By-Fee, and other future developments. - -Transaction inputs with +nSequence+ values less than 2^31^ are interpreted as having a relative timelock. Such a transaction is only valid once the input has aged by the relative timelock amount. For example, a transaction with one input with an +nSequence+ relative timelock of 30 blocks is only valid when at least 30 blocks have elapsed from the time the UTXO referenced in the input was mined. Since +nSequence+ is a per-input field, a transaction may contain any number of timelocked inputs, all of which must have sufficiently aged for the transaction to be valid. A transaction can include both timelocked inputs (+nSequence+ < 2^31^) and inputs without a relative timelock (+nSequence+ >= 2^31^). - -The +nSequence+ value is specified in either blocks or seconds, but in a slightly different format than we saw used in +nLocktime+. A type-flag is used to differentiate between values counting blocks and values counting time in seconds. The type-flag is set in the 23rd least-significant bit (i.e., value 1<<22). If the type-flag is set, then the +nSequence+ value is interpreted as a multiple of 512 seconds. If the type-flag is not set, the +nSequence+ value is interpreted as a number of blocks. - -When interpreting +nSequence+ as a relative timelock, only the 16 least significant bits are considered. Once the flags (bits 32 and 23) are evaluated, the +nSequence+ value is usually "masked" with a 16-bit mask (e.g., +nSequence+ & 0x0000FFFF). - -<> shows the binary layout of the +nSequence+ value, as defined by BIP-68. - -[[bip_68_def_of_nseq]] -.BIP-68 definition of nSequence encoding (Source: BIP-68) -image::images/mbc2_0701.png["BIP-68 definition of nSequence encoding"] - - -Relative timelocks based on consensus enforcement of the +nSequence+ value are defined in BIP-68. - -The standard is defined in https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki[BIP-68, Relative lock-time using consensus-enforced sequence numbers]. - -==== Relative Timelocks with CSV - -((("scripting", "timelocks", "relative timelocks with CHECKSEQUENCEVERIFY")))((("CHECKSEQUENCEVERIFY (CSV)")))Just like CLTV and +nLocktime+, there is a script opcode for relative timelocks that leverages the +nSequence+ value in scripts. That opcode is +CHECKSEQUENCEVERIFY+, commonly referred to as +CSV+ for short. - -The +CSV+ opcode when evaluated in an UTXO's redeem script allows spending only in a transaction whose input +nSequence+ value is greater than or equal to the +CSV+ parameter. Essentially, this restricts spending the UTXO until a certain number of blocks or seconds have elapsed relative to the time the UTXO was mined. - -As with CLTV, the value in +CSV+ must match the format in the corresponding +nSequence+ value. If +CSV+ is specified in terms of blocks, then so must +nSequence+. If +CSV+ is specified in terms of seconds, then so must +nSequence+. - -Relative timelocks with +CSV+ are especially useful when several (chained) transactions are created and signed, but not propagated, when they're kept "off-chain." A child transaction cannot be used until the parent transaction has been propagated, mined, and aged by the time specified in the relative timelock. One application of this use case can be seen in <> and <>.((("", startref="relativetime07")))((("", startref="Trelative07"))) - -+CSV+ is defined in detail in https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki[BIP-112, CHECKSEQUENCEVERIFY]. - - -==== Median-Time-Past - -((("scripting", "timelocks", "Median-Tme-Past")))((("Median-Tme-Past")))((("timelocks", "Median-Tme-Past")))As part of the activation of relative timelocks, there was also a change in the way "time" is calculated for timelocks (both absolute and relative). In bitcoin there is a subtle, but very significant, difference between wall time and consensus time. Bitcoin is a decentralized network, which means that each participant has his or her own perspective of time. Events on the network do not occur instantaneously everywhere. Network latency must be factored into the perspective of each node. Eventually everything is synchronized to create a common ledger. Bitcoin reaches consensus every 10 minutes about the state of the ledger as it existed in the _past_. - -The timestamps set in block headers are set by the miners. There is a certain degree of latitude allowed by the consensus rules to account for differences in clock accuracy between decentralized nodes. However, this creates an unfortunate incentive for miners to lie about the time in a block so as to earn extra fees by including timelocked transactions that are not yet mature. See the following section for more information. - -To remove the incentive to lie and strengthen the security of timelocks, a BIP was proposed and activated at the same time as the BIPs for relative timelocks. This is BIP-113, which defines a new consensus measurement of time called _Median-Time-Past_. - -Median-Time-Past is calculated by taking the timestamps of the last 11 blocks and finding the median. That median time then becomes consensus time and is used for all timelock calculations. By taking the midpoint from approximately two hours in the past, the influence of any one block's timestamp is reduced. By incorporating 11 blocks, no single miner can influence the timestamps in order to gain fees from transactions with a timelock that hasn't yet matured. - -Median-Time-Past changes the implementation of time calculations for +nLocktime+, +CLTV+, +nSequence+, and +CSV+. The consensus time calculated by Median-Time-Past is always approximately one hour behind wall clock time. If you create timelock transactions, you should account for it when estimating the desired value to encode in +nLocktime+, +nSequence+, +CLTV+, and +CSV+. - -Median-Time-Past is specified in https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki[BIP-113]. - -[[fee_sniping]] -==== Timelock Defense Against Fee Sniping - -((("scripting", "timelocks", "defense against fee-sniping")))((("timelocks", "defense against fee-sniping")))((("fees", "fee sniping")))((("security", "defense against fee-sniping")))((("sniping")))Fee-sniping is a theoretical attack scenario, where miners attempting to rewrite past blocks "snipe" higher-fee transactions from future blocks to maximize their profitability. - -For example, let's say the highest block in existence is block #100,000. If instead of attempting to mine block #100,001 to extend the chain, some miners attempt to remine #100,000. These miners can choose to include any valid transaction (that hasn't been mined yet) in their candidate block #100,000. They don't have to remine the block with the same transactions. In fact, they have the incentive to select the most profitable (highest fee per kB) transactions to include in their block. They can include any transactions that were in the "old" block #100,000, as well as any transactions from the current mempool. Essentially they have the option to pull transactions from the "present" into the rewritten "past" when they re-create block #100,000. - -Today, this attack is not very lucrative, because block reward is much higher than total fees per block. But at some point in the future, transaction fees will be the majority of the mining reward (or even the entirety of the mining reward). At that time, this scenario becomes inevitable. - -To prevent "fee sniping," when Bitcoin Core creates transactions, it uses +nLocktime+ to limit them to the "next block," by default. In our scenario, Bitcoin Core would set +nLocktime+ to 100,001 on any transaction it created. Under normal circumstances, this +nLocktime+ has no effect—the transactions could only be included in block #100,001 anyway; it's the next block. - -But under a blockchain fork/double-spend attack, the miners would not be able to pull high-fee transactions from the mempool, because all those transactions would be timelocked to block #100,001. They can only remine #100,000 with whatever transactions were valid at that time, essentially gaining no new fees. - -To achieve this, Bitcoin Core sets the +nLocktime+ on all new transactions to and sets the +nSequence+ on all the inputs to 0xFFFFFFFE to enable +nLocktime+.((("", startref="Stimelock07"))) - -=== Scripts with Flow Control (Conditional Clauses) - -((("transactions", "advanced", "flow control scripts")))((("scripting", "flow control scripts", id="Sflow07")))((("conditional clauses", id="condition07")))((("flow control", id="flow07")))One of the more powerful features of Bitcoin Script is flow control, also known as conditional clauses. You are probably familiar with flow control in various programming languages that use the construct +IF...THEN...ELSE+. Bitcoin conditional clauses look a bit different, but are essentially the same construct. - -At a basic level, bitcoin conditional opcodes allow us to construct a redeem script that has two ways of being unlocked, depending on a +TRUE+/+FALSE+ outcome of evaluating a logical condition. For example, if x is +TRUE+, the redeem script is A and the ELSE redeem script is B. - -Additionally, bitcoin conditional expressions can be "nested" indefinitely, meaning that a conditional clause can contain another within it, which contains another, etc. Bitcoin Script flow control can be used to construct very complex scripts with hundreds or even thousands of possible execution paths. There is no limit to nesting, but consensus rules impose a limit on the maximum size, in bytes, of a script. - -Bitcoin implements flow control using the +IF+, +ELSE+, +ENDIF+, and +NOTIF+ opcodes. Additionally, conditional expressions can contain boolean operators such as +BOOLAND+, +BOOLOR+, and +NOT+. - -At first glance, you may find the bitcoin's flow control scripts confusing. That is because Bitcoin Script is a stack language. The same way that +1 {plus} 1+ looks "backward" when expressed as +1 1 ADD+, flow control clauses in bitcoin also look "backward." - -In most traditional (procedural) programming languages, flow control looks like this: - -.Pseudocode of flow control in most programming languages ----- -if (condition): - code to run when condition is true -else: - code to run when condition is false -code to run in either case ----- - -In a stack-based language like Bitcoin Script, the logical condition comes before the +IF+, which makes it look "backward," like this: - -.Bitcoin Script flow control ----- -condition -IF - code to run when condition is true -ELSE - code to run when condition is false -ENDIF -code to run in either case ----- - -When reading Bitcoin Script, remember that the condition being evaluated comes _before_ the +IF+ opcode. - -==== Conditional Clauses with VERIFY Opcodes - -((("VERIFY opcodes")))((("IF clauses")))((("opcodes", "VERIFY")))Another form of conditional in Bitcoin Script is any opcode that ends in +VERIFY+. The +VERIFY+ suffix means that if the condition evaluated is not +TRUE+, execution of the script terminates immediately and the transaction is deemed invalid. - -((("guard clauses")))Unlike an +IF+ clause, which offers alternative execution paths, the +VERIFY+ suffix acts as a _guard clause_, continuing only if a precondition is met. - -For example, the following script requires Bob's signature and a pre-image (secret) that produces a specific hash. Both conditions must be satisfied to unlock it: - -.A redeem script with an +EQUALVERIFY+ guard clause. ----- -HASH160 EQUALVERIFY CHECKSIG ----- - -To redeem this, Bob must construct an unlocking script that presents a valid pre-image and a signature: - -.An unlocking script to satisfy the above redeem script ----- - ----- - -Without presenting the pre-image, Bob can't get to the part of the script that checks for his signature. - -[role="pagebreak-after"] -This script can be written with an +IF+ instead: - -.A redeem script with an +IF+ guard clause ----- -HASH160 EQUAL -IF - CHECKSIG -ENDIF ----- - -Bob's unlocking script is identical: - -.An unlocking script to satisfy the above redeem script ----- - ----- - -The script with +IF+ does the same thing as using an opcode with a +VERIFY+ suffix; they both operate as guard clauses. However, the +VERIFY+ construction is more efficient, using two fewer opcodes. - -So, when do we use +VERIFY+ and when do we use +IF+? If all we are trying to do is to attach a precondition (guard clause), then +VERIFY+ is better. If, however, we want to have more than one execution path (flow control), then we need an +IF...ELSE+ flow control clause. - -[TIP] -==== -((("EQUAL opcode")))((("opcodes", "EQUAL")))((("EQUALVERIFY opcode")))((("opcodes", "EQUALVERIFY")))An opcode such as +EQUAL+ will push the result (+TRUE+/+FALSE+) onto the stack, leaving it there for evaluation by subsequent opcodes. In contrast, the opcode +EQUALVERIFY+ suffix does not leave anything on the stack. Opcodes that end in +VERIFY+ do not leave the result on the stack. -==== - -==== Using Flow Control in Scripts - -A very common use for flow control in Bitcoin Script is to construct a redeem script that offers multiple execution paths, each a different way of redeeming the UTXO. - -((("use cases", "buying coffee")))Let's look at a simple example, where we have two signers, Alice and Bob, and either one is able to redeem. With multisig, this would be expressed as a 1-of-2 multisig script. For the sake of demonstration, we will do the same thing with an +IF+ clause: - ----- -IF - CHECKSIG -ELSE - CHECKSIG -ENDIF ----- - -Looking at this redeem script, you may be wondering: "Where is the condition? There is nothing preceding the +IF+ clause!" - -The condition is not part of the redeem script. Instead, the condition will be offered in the unlocking script, allowing Alice and Bob to "choose" which execution path they want. - -Alice redeems this with the unlocking script: ----- - 1 ----- - -The +1+ at the end serves as the condition (+TRUE+) that will make the +IF+ clause execute the first redemption path for which Alice has a signature. - -For Bob to redeem this, he would have to choose the second execution path by giving a +FALSE+ value to the +IF+ clause: - ----- - 0 ----- - -Bob's unlocking script puts a +0+ on the stack, causing the +IF+ clause to execute the second (+ELSE+) script, which requires Bob's signature. - -Since +IF+ clauses can be nested, we can create a "maze" of execution paths. The unlocking script can provide a "map" selecting which execution path is actually executed: - ----- -IF - script A -ELSE - IF - script B - ELSE - script C - ENDIF -ENDIF ----- - -In this scenario, there are three execution paths (+script A+, +script B+, and +script C+). The unlocking script provides a path in the form of a sequence of +TRUE+ or +FALSE+ values. To select path +script B+, for example, the unlocking script must end in +1 0+ (+TRUE+, +FALSE+). These values will be pushed onto the stack, so that the second value (+FALSE+) ends up at the top of the stack. The outer +IF+ clause pops the +FALSE+ value and executes the first +ELSE+ clause. Then the +TRUE+ value moves to the top of the stack and is evaluated by the inner (nested) +IF+, selecting the +B+ execution path. - -Using this construct, we can build redeem scripts with tens or hundreds of execution paths, each offering a different way to redeem the UTXO. To spend, we construct an unlocking script that navigates the execution path by putting the appropriate +TRUE+ and +FALSE+ values on the stack at each flow control point.((("", startref="Sflow07")))((("", startref="flow07")))((("", startref="condition07"))) - -=== Complex Script Example - -((("transactions", "advanced", "example")))((("scripting", "complex script example", id="Scomplex07")))In this section we combine many of the concepts from this chapter into a single example. - -((("use cases", "import/export", id="mohamseventwo")))Our example uses the story of Mohammed, the company owner in Dubai who is operating an import/export business. - -((("transactions", "advanced", "multisignature scripts")))((("scripting", "multisignature scripts", "import/export example")))((("multisignature scripts")))In this example, Mohammed wishes to construct a company capital account with flexible rules. The scheme he creates requires different levels of authorization depending on timelocks. The participants in the multisig scheme are Mohammed, his two partners Saeed and Zaira, and their company lawyer Abdul. The three partners make decisions based on a majority rule, so two of the three must agree. However, in the case of a problem with their keys, they want their lawyer to be able to recover the funds with one of the three partner signatures. Finally, if all partners are unavailable or incapacitated for a while, they want the lawyer to be able to manage the account directly. - -Here's the redeem script that Mohammed designs to achieve this (line number prefix as XX): - -.Variable Multi-Signature with Timelock ----- -01 IF -02 IF -03 2 -04 ELSE -05 <30 days> CHECKSEQUENCEVERIFY DROP -06 CHECKSIGVERIFY -07 1 -08 ENDIF -09 3 CHECKMULTISIG -10 ELSE -11 <90 days> CHECKSEQUENCEVERIFY DROP -12 CHECKSIG -13 ENDIF ----- - -Mohammed's script implements three execution paths using nested +IF...ELSE+ flow control clauses. - -In the first execution path, this script operates as a simple 2-of-3 multisig with the three partners. This execution path consists of lines 3 and 9. Line 3 sets the quorum of the multisig to +2+ (2-of-3). This execution path can be selected by putting +TRUE TRUE+ at the end of the unlocking script: - -.Unlocking script for the first execution path (2-of-3 multisig) ----- -0 TRUE TRUE ----- - - -[TIP] -==== -The +0+ at the beginning of this unlocking script is because of a bug in +CHECKMULTISIG+ that pops an extra value from the stack. The extra value is disregarded by the +CHECKMULTISIG+, but it must be present or the script fails. Pushing +0+ (customarily) is a workaround to the bug, as described in <>. -==== - -The second execution path can only be used after 30 days have elapsed from the creation of the UTXO. At that time, it requires the signature of Abdul the lawyer and one of the three partners (a 1-of-3 multisig). This is achieved by line 7, which sets the quorum for the multisig to +1+. To select this execution path, the unlocking script would end in +FALSE TRUE+: - -.Unlocking script for the second execution path (Lawyer + 1-of-3) ----- -0 FALSE TRUE ----- - -[TIP] -==== -Why +FALSE TRUE+? Isn't that backward? Because the two values are pushed on to the stack, with +FALSE+ pushed first, then +TRUE+ pushed second. +TRUE+ is therefore popped _first_ by the first +IF+ opcode. -==== - -Finally, the third execution path allows Abdul the lawyer to spend the funds alone, but only after 90 days. To select this execution path, the unlocking script has to end in +FALSE+: - -.Unlocking script for the third execution path (Lawyer only) ----- - FALSE ----- - -Try running the script on paper to see how it behaves on the stack. - -A few more things to consider when reading this example. See if you can find the answers: - -* Why can't the lawyer redeem the third execution path at any time by selecting it with +FALSE+ on the unlocking script? - -* How many execution paths can be used 5, 35, and 105 days, respectively, after the UTXO is mined? - -* Are the funds lost if the lawyer loses his key? Does your answer change if 91 days have elapsed? - -* How do the partners "reset" the clock every 29 or 89 days to prevent the lawyer from accessing the funds? - -* Why do some +CHECKSIG+ opcodes in this script have the +VERIFY+ suffix while others don't?((("", startref="Scomplex07")))((("", startref="mohamseventwo"))) - -[[segwit]] -=== Segregated Witness - -((("segwit (Segregated Witness)", id="Ssegwit07")))Segregated Witness (segwit) is an upgrade to the bitcoin consensus rules and network protocol, proposed and implemented as a BIP-9 soft-fork that was activated on bitcoin's mainnet on August 1st, 2017. - -In cryptography, the term "witness" is used to describe a solution to a cryptographic puzzle. In bitcoin terms, the witness satisfies a cryptographic condition placed on an unspent transaction output (UTXO). - -In the context of bitcoin, a digital signature is _one type of witness_, but a witness is more broadly any solution that can satisfy the conditions imposed on an UTXO and unlock that UTXO for spending. The term “witness” is a more general term for an “unlocking script” or “scriptSig.” - -Before segwit’s introduction, every input in a transaction was followed by the witness data that unlocked it. The witness data was embedded in the transaction as part of each input. The term _segregated witness_, or _segwit_ for short, simply means separating the signature or unlocking script of a specific output. Think "separate scriptSig," or “separate signature” in the simplest form. - -Segregated Witness therefore is an architectural change to bitcoin that aims to move the witness data from the +scriptSig+ (unlocking script) field of a transaction into a separate _witness_ data structure that accompanies a transaction. Clients may request transaction data with or without the accompanying witness data. - -In this section we will look at some of the benefits of Segregated Witness, describe the mechanism used to deploy and implement this architecture change, and demonstrate the use of Segregated Witness in transactions and addresses. - -Segregated Witness is defined by the following BIPs: - -https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki[BIP-141] :: The main definition of Segregated Witness. - -https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki[BIP-143] :: Transaction Signature Verification for Version 0 Witness Program - -https://github.com/bitcoin/bips/blob/master/bip-0144.mediawiki[BIP-144] :: Peer Services—New network messages and serialization formats - -https://github.com/bitcoin/bips/blob/master/bip-0145.mediawiki[BIP-145] :: getblocktemplate Updates for Segregated Witness (for mining) - -https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki[BIP-173]:: Base32 address format for native v0-16 witness outputs - - -==== Why Segregated Witness? - -Segregated Witness is an architectural change that has several effects on the scalability, security, economic incentives, and performance of bitcoin: - -Transaction Malleability :: By moving the witness outside the transaction data, the transaction hash used as an identifier no longer includes the witness data. Since the witness data is the only part of the transaction that can be modified by a third party (see <>), removing it also removes the opportunity for transaction malleability attacks. With Segregated Witness, transaction hashes become immutable by anyone other than the creator of the transaction, which greatly improves the implementation of many other protocols that rely on advanced bitcoin transaction construction, such as payment channels, chained transactions, and lightning networks. - -Script Versioning :: With the introduction of Segregated Witness scripts, every locking script is preceded by a _script version_ number, similar to how transactions and blocks have version numbers. The addition of a script version number allows the scripting language to be upgraded in a backward-compatible way (i.e., using soft fork upgrades) to introduce new script operands, syntax, or semantics. The ability to upgrade the scripting language in a nondisruptive way will greatly accelerate the rate of innovation in bitcoin. - -Network and Storage Scaling :: The witness data is often a big contributor to the total size of a transaction. More complex scripts such as those used for multisig or payment channels are very large. In some cases these scripts account for the majority (more than 75%) of the data in a transaction. By moving the witness data outside the transaction data, Segregated Witness improves bitcoin’s scalability. Nodes can prune the witness data after validating the signatures, or ignore it altogether when doing simplified payment verification. The witness data doesn’t need to be transmitted to all nodes and does not need to be stored on disk by all nodes. - -Signature Verification Optimization :: Segregated Witness upgrades the signature functions (+CHECKSIG+, +CHECKMULTISIG+, etc.) to reduce the algorithm's computational complexity. Before segwit, the algorithm used to produce a signature required a number of hash operations that was proportional to the size of the transaction. Data-hashing computations increased in O(n^2^) with respect to the number of signature operations, introducing a substantial computational burden on all nodes verifying the signature. With segwit, the algorithm is changed to reduce the complexity to O(n). - -Offline Signing Improvement :: Segregated Witness signatures incorporate the value (amount) referenced by each input in the hash that is signed. Previously, an offline signing device, such as a hardware wallet, would have to verify the amount of each input before signing a transaction. This was usually accomplished by streaming a large amount of data about the previous transactions referenced as inputs. Since the amount is now part of the commitment hash that is signed, an offline device does not need the previous transactions. If the amounts do not match (are misrepresented by a compromised online system), the signature will be invalid. - -==== How Segregated Witness Works - -At first glance, Segregated Witness appears to be a change to how transactions are constructed and therefore a transaction-level feature, but it is not. Rather, Segregated Witness is a change to how individual UTXO are spent and therefore is a per-output feature. - -A transaction can spend Segregated Witness outputs or traditional (inline-witness) outputs or both. Therefore, it does not make much sense to refer to a transaction as a “Segregated Witness transaction.” Rather we should refer to specific transaction outputs as “Segregated Witness outputs." - -When a transaction spends an UTXO, it must provide a witness. In a traditional UTXO, the locking script requires that witness data be provided _inline_ in the input part of the transaction that spends the UTXO. A Segregated Witness UTXO, however, specifies a locking script that can be satisfied with witness data outside of the input (segregated). - -==== Soft Fork (Backward Compatibility) - -Segregated Witness is a significant change to the way outputs and transactions are architected. Such a change would normally require a simultaneous change in every Bitcoin node and wallet to change the consensus rules—what is known as a hard fork. Instead, segregated witness is introduced with a much less disruptive change, which is backward compatible, known as a soft fork. This type of upgrade allows non-upgraded software to ignore the changes and continue to operate without any disruption. - -Segregated Witness outputs are constructed so that older systems that are not segwit-aware can still validate them. To an old wallet or node, a Segregated Witness output looks like an output that _anyone can spend_. Such outputs can be spent with an empty signature, therefore the fact that there is no signature inside the transaction (it is segregated) does not invalidate the transaction. Newer wallets and mining nodes, however, see the Segregated Witness output and expect to find a valid witness for it in the transaction’s witness data. - -==== Segregated Witness Output and Transaction Examples - -Let’s look at some of our example transactions and see how they would change with Segregated Witness. We’ll first look at how a Pay-to-Public-Key-Hash (P2PKH) payment is transformed with the Segregated Witness program. Then, we’ll look at the Segregated Witness equivalent for Pay-to-Script-Hash (P2SH) scripts. Finally, we’ll look at how both of the preceding Segregated Witness programs can be embedded inside a P2SH script. - -[[p2wpkh]] -===== Pay-to-Witness-Public-Key-Hash (P2WPKH) - -In <>, ((("use cases", "buying coffee", id="aliced")))Alice created a transaction to pay Bob for a cup of coffee. That transaction created a P2PKH output with a value of 0.015 BTC that was spendable by Bob. The output’s script looks like this: - -.Example P2PKH output script ----- -DUP HASH160 ab68025513c3dbd2f7b92a94e0581f5d50f654e7 EQUALVERIFY CHECKSIG ----- - -With Segregated Witness, Alice would create a Pay-to-Witness-Public-Key-Hash (P2WPKH) script, which looks like this: - -.Example P2WPKH output script ----- -0 ab68025513c3dbd2f7b92a94e0581f5d50f654e7 ----- - -As you can see, a Segregated Witness output’s locking script is much simpler than a traditional output. It consists of two values that are pushed on to the script evaluation stack. To an old (nonsegwit-aware) Bitcoin client, the two pushes would look like an output that anyone can spend and does not require a signature (or rather, can be spent with an empty signature). To a newer, segwit-aware client, the first number (0) is interpreted as a version number (the _witness version_) and the second part (20 bytes) is the equivalent of a locking script known as a _witness program_. The 20-byte witness program is simply the hash of the public key, as in a P2PKH script. - -Now, let’s look at the corresponding transaction that Bob uses to spend this output. For the original script (nonsegwit), Bob’s transaction would have to include a signature within the transaction input: - -.Decoded transaction showing a P2PKH output being spent with a signature ----- -[...] -“Vin” : [ -"txid": "0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2", -"vout": 0, - "scriptSig": “”, -] -[...] ----- - -However, to spend the Segregated Witness output, the transaction has no signature in the input part. Instead, Bob’s transaction has an empty +scriptSig+ in the transaction data (the first part of a transaction, which includes the input part) and includes his signature in the witness data (the second part of a transaction, which is separated from the transaction data): - -.Decoded transaction showing a P2WPKH output being spent with separate witness data ----- -[...] -“Vin” : [ -"txid": "0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2", -"vout": 0, - "scriptSig": “”, -] -[...] -“witness”: “” -[...] ----- - -===== Wallet construction of P2WPKH - -It is extremely important to note that P2WPKH should only be created by the payee (recipient) and not converted by the sender from a known public key, P2PKH script, or address. The receiver has no way of knowing if the sender's wallet has the ability to construct segwit transactions and spend P2WPKH outputs. - -Additionally, P2WPKH outputs must be constructed from the hash of a _compressed_ public key. Uncompressed public keys are nonstandard in segwit and may be explicitly disabled by a future soft fork. If the hash used in the P2WPKH came from an uncompressed public key, it may be unspendable and you may lose funds. P2WPKH outputs should be created by the payee's wallet by deriving a compressed public key from their private key. - -[WARNING] -==== -P2WPKH should be constructed by the payee (recipient) by converting a compressed public key to a P2WPKH hash. You should never transform a P2PKH script, Bitcoin address, or uncompressed public key to a P2WPKH witness script. -==== - -[[p2wsh]] -===== Pay-to-Witness-Script-Hash (P2WSH) - -The ((("use cases", "import/export", id="mohamappd")))second type of witness program corresponds to a Pay-to-Script-Hash (P2SH) script. We saw this type of script in <>. In that example, P2SH was used by Mohammed's company to express a multisignature script. Payments to Mohammed's company were encoded with a locking script like this: - -.Example P2SH output script ----- -HASH160 54c557e07dde5bb6cb791c7a540e0a4796f5e97e EQUAL ----- - -This P2SH script references the hash of a _redeem script_ that defines a 2-of-5 multisignature requirement to spend funds. To spend this output, Mohammed's company would present the redeem script (whose hash matches the script hash in the P2SH output) and the signatures necessary to satisfy that redeem script, all inside the transaction input: - -.Decoded transaction showing a P2SH output being spent ----- -[...] -“Vin” : [ -"txid": "abcdef12345...", -"vout": 0, - "scriptSig": “ <2 PubA PubB PubC PubD PubE 5 CHECKMULTISIG>”, -] ----- - -Now, let's look at how this entire example would be upgraded to segwit. If Mohammed's customers were using a segwit-compatible wallet, they would make a payment, creating a Pay-to-Witness-Script-Hash (P2WSH) output that would look like this: - -.Example P2WSH output script ----- -0 a9b7b38d972cabc7961dbfbcb841ad4508d133c47ba87457b4a0e8aae86dbb89 ----- - -Again, as with the example of P2WPKH, you can see that the Segregated Witness equivalent script is a lot simpler and omits the various script operands that you see in P2SH scripts. Instead, the Segregated Witness program consists of two values pushed to the stack: a witness version (0) and the 32-byte SHA256 hash of the redeem script. - -Mohammed's company can spend the P2WSH output by presenting the correct redeem script and sufficient signatures to satisfy it. Both the redeem script and the signatures would be segregated _outside_ the spending transaction data as part of the witness data. Within the transaction input, Mohammed's ((("", startref="mohamappd")))wallet would put an empty +scriptSig+: - -.Decoded transaction showing a P2WSH output being spent with separate witness data ----- -[...] -“Vin” : [ -"txid": "abcdef12345...", -"vout": 0, - "scriptSig": “”, -] -[...] -“witness”: “ <2 PubA PubB PubC PubD PubE 5 CHECKMULTISIG>” -[...] ----- - -[TIP] -==== -While P2SH uses the 20-byte +RIPEMD160(SHA256(script))+ hash, the P2WSH witness program uses a 32-byte +SHA256(script)+ hash. This difference in the selection of the hashing algorithm is deliberate and provides stronger security to P2WSH (128 bits of security in P2WSH versus 80 bits of security in P2SH). It is also used to differentiate between the two types of witness programs (P2WPKH and P2WSH) by using the length of the hash (see below). - -==== - -===== Differentiating between P2WPKH and P2WSH - -In the previous two sections, we demonstrated two types of witness programs: <> and <>. Both types of witness programs consist of a single byte version number followed by a longer hash. They look very similar, but are interpreted very differently: one is interpreted as a public key hash, which is satisfied by a signature and the other as a script hash, which is satisfied by a redeem script. The critical difference between them is the length of the hash: - -* The public key hash in P2WPKH is 20 bytes -* The script hash in P2WSH is 32 bytes - -This is the one difference that allows a wallet to differentiate between the two types of witness programs. By looking at the length of the hash, a wallet can determine what type of witness program it is, P2WPKH or P2WSH. - -==== Upgrading to Segregated Witness - -As we can see from the previous examples, upgrading to Segregated Witness is a two-step process. First, wallets must create special segwit type outputs. Then, these outputs can be spent by wallets that know how to construct Segregated Witness transactions. In the examples, Alice's wallet was segwit-aware and able to create special outputs with Segregated Witness scripts. Bob's wallet is also segwit-aware and able to spend those outputs. What may not be obvious from the example is that in practice, Alice's wallet needs to _know_ that Bob uses a segwit-aware wallet and can spend these outputs. Otherwise, if Bob's wallet is not upgraded and Alice tries to make segwit payments to Bob, Bob's wallet will not be able to detect these payments. - -[TIP] -==== -For P2WPKH and P2WSH payment types, both the sender and the recipient wallets need to be upgraded to be able to use segwit. Furthermore, the sender's wallet needs to know that the recipient's wallet is segwit-aware. -==== - -Segregated Witness will not be implemented simultaneously across the entire network. Rather, Segregated Witness is implemented as a backward-compatible upgrade, where _old and new clients can coexist_. Wallet developers will independently upgrade wallet software to add segwit capabilities. The P2WPKH and P2WSH payment types are used when both sender and recipient are segwit-aware. The traditional P2PKH and P2SH will continue to work for non-upgraded wallets. That leaves two important scenarios, which are addressed in the next section: - -* Ability of a sender's wallet that is not segwit-aware to make a payment to a recipient's wallet that can process segwit transactions - -* Ability of a sender's wallet that is segwit-aware to recognize and distinguish between recipients that are segwit-aware and ones that are not, by their _addresses_. - -===== Embedding Segregated Witness inside P2SH - -Let's assume, for example, that Alice's wallet is not upgraded to segwit, but Bob's wallet is upgraded and can handle segwit transactions. Alice and Bob can use "old" non-segwit transactions. But Bob would likely want to use segwit to reduce transaction fees, taking advantage of the discount that applies to witness data. - -In this case Bob's wallet can construct a P2SH address that contains a segwit script inside it. Alice's wallet sees this as a "normal" P2SH address and can make payments to it without any knowledge of segwit. Bob's wallet can then spend this payment with a segwit transaction, taking full advantage of segwit and reducing transaction fees. - -Both forms of witness scripts, P2WPKH and P2WSH, can be embedded in a P2SH address. The first is noted as P2SH(P2WPKH) and the second is noted as P2SH(P2WSH). - -===== Pay-to-Witness-Public-Key-Hash inside Pay-to-Script-Hash - -The first form of witness script we will examine is P2SH(P2WPKH). This is a Pay-to-Witness-Public-Key-Hash witness program, embedded inside a Pay-to-Script-Hash script, so that it can be used by a wallet that is not aware of segwit. - -Bob's wallet constructs a P2WPKH witness program with Bob's public key. This witness program is then hashed and the resulting hash is encoded as a P2SH script. The P2SH script is converted to a Bitcoin address, one that starts with a "3," as we saw in the <> section. - -Bob's wallet starts with the P2WPKH witness program we saw earlier: - -.Bob's P2WPKH witness program ----- -0 ab68025513c3dbd2f7b92a94e0581f5d50f654e7 ----- - -The P2WPKH witness program consists of the witness version and Bob's 20-byte public key hash. - -Bob's wallet then hashes the preceding witness program, first with SHA256, then with RIPEMD160, producing another 20-byte hash. - -Let's use +bx+ on the command-line to replicate that: - -.HASH160 of the P2WPKH witness program ----- -echo \ -'0 [ab68025513c3dbd2f7b92a94e0581f5d50f654e7]'\ - | bx script-encode | bx sha256 | bx ripemd160 -3e0547268b3b19288b3adef9719ec8659f4b2b0b ----- - - -Next, the redeem script hash is converted to a Bitcoin address. Let's use +bx+ on the command-line again: - -.P2SH address ----- -echo \ -'3e0547268b3b19288b3adef9719ec8659f4b2b0b' \ -| bx address-encode -v 5 -37Lx99uaGn5avKBxiW26HjedQE3LrDCZru ----- - -Now, Bob can display this address for customers to pay for their coffee. Alice's wallet can make a payment to +37Lx99uaGn5avKBxiW26HjedQE3LrDCZru+, just as it would to any other Bitcoin address. - -To pay Bob, Alice's wallet would lock the output with a P2SH script: ----- -HASH160 3e0547268b3b19288b3adef9719ec8659f4b2b0b EQUAL ----- - -Even though Alice's wallet has no support for segwit, the payment it creates can be spent by Bob with a segwit transaction.((("", startref="aliced"))) - -===== Pay-to-Witness-Script-Hash inside Pay-to-Script-Hash - -Similarly, a P2WSH witness program for a multisig script or other complicated script can be embedded inside a P2SH script and address, making it possible for any wallet to make payments that are segwit compatible. - -As we saw in <>, Mohammed's ((("use cases", "import/export")))company is using Segregated Witness payments with multisignature scripts. To make it possible for any client to pay his company, regardless of whether their wallets are upgraded for segwit, Mohammed's wallet can embed the P2WSH witness program inside a P2SH script. - -First, Mohammed's wallet hashes the redeem script with SHA256 (just once). Let's use +bx+ to do that on the command-line: - -.Mohammed's wallet creates a P2WSH witness program ----- -echo \ -2 \ [04C16B8698A9ABF84250A7C3EA7EEDEF9897D1C8C6ADF47F06CF73370D74DCCA01CDCA79DCC5C395D7EEC6984D83F1F50C900A24DD47F569FD4193AF5DE762C587] \ -[04A2192968D8655D6A935BEAF2CA23E3FB87A3495E7AF308EDF08DAC3C1FCBFC2C75B4B0F4D0B1B70CD2423657738C0C2B1D5CE65C97D78D0E34224858008E8B49] \ -[047E63248B75DB7379BE9CDA8CE5751D16485F431E46117B9D0C1837C9D5737812F393DA7D4420D7E1A9162F0279CFC10F1E8E8F3020DECDBC3C0DD389D9977965] \ -[0421D65CBD7149B255382ED7F78E946580657EE6FDA162A187543A9D85BAAA93A4AB3A8F044DADA618D087227440645ABE8A35DA8C5B73997AD343BE5C2AFD94A5] \ -[043752580AFA1ECED3C68D446BCAB69AC0BA7DF50D56231BE0AABF1FDEEC78A6A45E394BA29A1EDF518C022DD618DA774D207D137AAB59E0B000EB7ED238F4D800] \ -5 CHECKMULTISIG \ -| bx script-encode | bx sha256 -9592d601848d04b172905e0ddb0adde59f1590f1e553ffc81ddc4b0ed927dd73 ----- - -Next, the hashed redeem script is turned into a P2WSH witness program: - ----- -0 9592d601848d04b172905e0ddb0adde59f1590f1e553ffc81ddc4b0ed927dd73 ----- - -Then, the witness program itself is hashed with SHA256 and RIPEMD160, producing a new 20-byte hash, as used in traditional P2SH. Let's use +bx+ on the command-line to do that: - -.The HASH160 of the P2WSH witness program ----- - echo \ -'0 [9592d601848d04b172905e0ddb0adde59f1590f1e553ffc81ddc4b0ed927dd73]'\ - | bx script-encode | bx sha256 | bx ripemd160 -86762607e8fe87c0c37740cddee880988b9455b2 ----- - -Next, the wallet constructs a P2SH Bitcoin address from this hash. Again, we use +bx+ to calculate on the command-line: - -.P2SH Bitcoin address ----- -echo \ -'86762607e8fe87c0c37740cddee880988b9455b2'\ - | bx address-encode -v 5 -3Dwz1MXhM6EfFoJChHCxh1jWHb8GQqRenG ----- - -Now, Mohammed's clients can make payments to this address without any need to support segwit. To send a payment to Mohammed, a wallet would lock the output with the following P2SH script: - -.P2SH script used to lock payments to Mohammed's multisig ----- -HASH160 86762607e8fe87c0c37740cddee880988b9455b2 EQUAL ----- - -Mohammed's company can then construct segwit transactions to spend these payments, taking advantage of segwit features including lower transaction fees. - -===== Segregated Witness addresses - -Even after segwit activation, it will take some time until most wallets are upgraded. At first, segwit will be embedded in P2SH, as we saw in the previous section, to ease compatibility between segwit-aware and unaware wallets. - -However, once wallets are broadly supporting segwit, it makes sense to encode witness scripts directly in a native address format designed for segwit, rather than embed it in P2SH. - -The native segwit address format is defined in BIP-173: - -https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki[BIP-173]:: Base32 address format for native v0-16 witness outputs - -BIP-173 only encodes witness (P2WPKH and P2WSH) scripts. It is not compatible with non-segwit P2PKH or P2SH scripts. BIP-173 is a checksummed Base32 encoding, as compared to the Base58 encoding of a "traditional" Bitcoin address. BIP-173 addresses are also called _bech32_ addresses, pronounced "beh-ch thirty two", alluding to the use of a "BCH" error detection algorithm and 32-character encoding set. - -BIP-173 addresses use 32 lower-case-only alphanumeric character set, carefully selected to reduce errors from misreading or mistyping. By choosing a lower-case-only character set, bech32 is easier to read, speak, and 45% more efficient to encode in QR codes. - -The BCH error detection algorithm is a vast improvement over the previous checksum algorithm (from Base58Check), allowing not only detection but also _correction_ of errors. Address-input interfaces (such as text-fields in forms) can detect and highlight which character was most likely mistyped when they detect an error. - -From the BIP-173 specification, here are some examples of bech32 addresses: - -Mainnet P2WPKH:: bc1qw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t4 -Testnet P2WPKH:: tb1qw508d6qejxtdg4y5r3zarvary0c5xw7kxpjzsx -Mainnet P2WSH:: bc1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3qccfmv3 -Testnet P2WSH:: tb1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3q0sl5k7 - -As you can see in these examples, a segwit bech32 string is up to 90 characters long and consists of three parts: - -The human readable part:: This prefix "bc" or "tb" identifying mainnet or testnet - -The separator:: The digit "1", which is not part of the 32-character encoding set and can only appear in this position as a separator - -The data part:: A minimum of 6 alphanumeric characters, the checksum encoded witness script - -At this time, only a few wallets accept or produce native segwit bech32 addresses, but as segwit adoption increases, you will see these more and more often. - -<> shows bitcoin non-segwit (legacy) and segwit addresses. - -[[segwit_addresses]] -.Bitcoin non-segwit (legacy) and segwit addresses -|======= -|Type |Encoding |Prefix -| Legacy P2PKH Address | Base58 | 1 -| Legacy Testnet P2PKH Address | Base58 | m or n -| Legacy P2SH Address | Base58 | 3 -| Legacy Testnet P2SH Address | Base58 | 2 -| Nested (embedded) Segwit P2SH(P2WPKH) Address | Base58 | 3 -| Nested (embedded) Segwit P2SH(P2WSH) Address | Base58 | 3 -| Native Segwit P2WPKH Address | Bech32 | bc1 -| Native Segwit Testnet P2WPKH Address | Bech32 | tb1 -| Native Segwit P2WSH Address | Bech32 | bc1 -| Native Segwit Testnet P2WSH Address | Bech32 | tb1 -|======= - -[[segwit_txid]] -===== Transaction identifiers - -((("transaction IDs (txid)")))One of the greatest benefits of Segregated Witness is that it eliminates third-party transaction malleability. - -Before segwit, transactions could have their signatures subtly modified by third parties, changing their transaction ID (hash) without changing any fundamental properties (inputs, outputs, amounts). This created opportunities for denial-of-service attacks as well as attacks against poorly written wallet software that assumed unconfirmed transaction hashes were immutable. - -With the introduction of Segregated Witness, transactions have two identifiers, +txid+ and +wtxid+. The traditional transaction ID +txid+ is the double-SHA256 hash of the serialized transaction, without the witness data. A transaction +wtxid+ is the double-SHA256 hash of the new serialization format of the transaction with witness data. - -The traditional +txid+ is calculated in exactly the same way as with a nonsegwit transaction. However, since a pure segwit transaction (a transaction that only contains segwit inputs) has empty ++scriptSig++s in every input, there is no part of the transaction that can be modified by a third party. Therefore, in a pure segwit transaction, the +txid+ is immutable by a third party, even when the transaction is unconfirmed. - -The +wtxid+ is like an "extended" ID, in that the hash also incorporates the witness data. If a transaction is transmitted without witness data, then the +wtxid+ and +txid+ are identical. Note that since the +wtxid+ includes witness data (signatures) and since witness data may be malleable, the +wtxid+ should be considered malleable until the transaction is confirmed. Only the +txid+ of a pure segwit transaction can be considered immutable by third parties. - -[TIP] -==== -Segregated Witness transactions have two IDs: +txid+ and +wtxid+. The +txid+ is the hash of the transaction without the witness data and the +wtxid+ is the hash inclusive of witness data. Only pure segwit transactions (transactions that only contain segwit inputs) have a +txid+ that is not susceptible to third-party transaction malleability. -==== - -==== Segregated Witness' New Signing Algorithm - -Segregated Witness modifies the semantics of the four signature verification functions (+CHECKSIG+, +CHECKSIGVERIFY+, +CHECKMULTISIG+, and +CHECKMULTISIGVERIFY+), changing the way a transaction commitment hash is calculated. - -Signatures in bitcoin transactions are applied on a _commitment hash_, which is calculated from the transaction data, locking specific parts of the data indicating the signer's commitment to those values. For example, in a simple +SIGHASH_ALL+ type signature, the commitment hash includes all inputs and outputs. - -Unfortunately, the way the commitment hash was calculated introduced the possibility that a node verifying the signature can be forced to perform a significant number of hash computations. Specifically, the hash operations increase in O(n^2^) with respect to the number of signature operations in the transaction. An attacker could therefore create a transaction with a very large number of signature operations, causing the entire Bitcoin network to have to perform hundreds or thousands of hash operations to verify the transaction. - -Segwit represented an opportunity to address this problem by changing the way the commitment hash is calculated. For segwit version 0 witness programs, signature verification occurs using an improved commitment hash algorithm as specified in BIP-143. - -The new algorithm achieves two important goals. Firstly, the number of hash operations increases by a much more gradual O(n) to the number of signature operations, reducing the opportunity to create denial-of-service attacks with overly complex transactions. Secondly, the commitment hash now also includes the value (amounts) of each input as part of the commitment. This means that a signer can commit to a specific input value without needing to "fetch" and check the previous transaction referenced by the input. In the case of offline devices, such as hardware wallets, this greatly simplifies the communication between the host and the hardware wallet, removing the need to stream previous transactions for validation. A hardware wallet can accept the input value "as stated" by an untrusted host. Since the signature is invalid if that input value is not correct, the hardware wallet doesn't need to validate the value before signing the input. - -==== Economic Incentives for Segregated Witness - -Bitcoin mining nodes and full nodes incur costs for the resources used to support the Bitcoin network and the blockchain. As the volume of bitcoin transactions increases, so does the cost of resources (CPU, network bandwidth, disk space, memory). Miners are compensated for these costs through fees that are proportional to the size (in bytes) of each transaction. Nonmining full nodes are not compensated, so they incur these costs because they have a need to run an authoritative fully validating full-index node, perhaps because they use the node to operate a bitcoin business. - -Without transaction fees, the growth in bitcoin data would arguably increase dramatically. Fees are intended to align the needs of bitcoin users with the burden their transactions impose on the network, through a market-based price discovery mechanism. - -The calculation of fees based on transaction size treats all the data in the transaction as equal in cost. But from the perspective of full nodes and miners, some parts of a transaction carry much higher costs. Every transaction added to the Bitcoin network affects the consumption of four resources on nodes: - -Disk Space :: Every transaction is stored in the blockchain, adding to the total size of the blockchain. The blockchain is stored on disk, but the storage can be optimized by “pruning” (deleting) older transactions. - -CPU :: Every transaction must be validated, which requires CPU time. - -Bandwidth :: Every transaction is transmitted (through flood propagation) across the network at least once. Without any optimization in the block propagation protocol, transactions are transmitted again as part of a block, doubling the impact on network capacity. - -Memory :: Nodes that validate transactions keep the UTXO index or the entire UTXO set in memory to speed up validation. Because memory is at least one order of magnitude more expensive than disk, growth of the UTXO set contributes disproportionately to the cost of running a node. - -As you can see from the list, not every part of a transaction has an equal impact on the cost of running a node or on the ability of bitcoin to scale to support more transactions. The most expensive part of a transaction are the newly created outputs, as they are added to the in-memory UTXO set. By comparison, signatures (aka witness data) add the least burden to the network and the cost of running a node, because witness data are only validated once and then never used again. Furthermore, immediately after receiving a new transaction and validating witness data, nodes can discard that witness data. If fees are calculated on transaction size, without discriminating between these two types of data, then the market incentives of fees are not aligned with the actual costs imposed by a transaction. In fact, the current fee structure actually encourages the opposite behavior, because witness data is the largest part of a transaction. - -The incentives created by fees matter because they affect the behavior of wallets. All wallets must implement some strategy for assembling transactions that takes into consideration a number of factors, such as privacy (reducing address reuse), fragmentation (making lots of loose change), and fees. If the fees are overwhelmingly motivating wallets to use as few inputs as possible in transactions, this can lead to UTXO picking and change address strategies that inadvertently bloat the UTXO set. - -Transactions consume UTXO in their inputs and create new UTXO with their outputs. A transaction, therefore, that has more inputs than outputs will result in a decrease in the UTXO set, whereas a transaction that has more outputs than inputs will result in an increase in the UTXO set. Let’s consider the _difference_ between inputs and outputs and call that the “Net-new-UTXO.” That’s an important metric, as it tells us what impact a transaction will have on the most expensive network-wide resource, the in-memory UTXO set. A transaction with positive Net-new-UTXO adds to that burden. A transaction with a negative Net-new-UTXO reduces the burden. We would therefore want to encourage transactions that are either negative Net-new-UTXO or neutral with zero Net-new-UTXO. - -Let’s look at an example of what incentives are created by the transaction fee calculation, with and without Segregated Witness. We will look at two different transactions. Transaction A is a 3-input, 2-output transaction, which has a Net-new-UTXO metric of –1, meaning it consumes one more UTXO than it creates, reducing the UTXO set by one. Transaction B is a 2-input, 3-output transaction, which has a Net-new-UTXO metric of 1, meaning it adds one UTXO to the UTXO set, imposing additional cost on the entire Bitcoin network. Both transactions use multisignature (2-of-3) scripts to demonstrate how complex scripts increase the impact of segregated witness on fees. Let’s assume a transaction feerate of 30 satoshi per byte and a 75% fee discount on witness data: - -++++ -
-
Without Segregated Witness
-
-

Transaction A fee: 28,590 satoshi

-

Transaction B fee: 20,760 satoshi

-
- -
With Segregated Witness
-
-

Transaction A fee: 12,255 satoshi

-

Transaction B fee: 10,425 satoshi

-
-
-++++ - - -Both transactions are less expensive when segregated witness is implemented. Comparing the costs between the two transactions, we see that before Segregated Witness, the transaction with the positive Net-new-UTXO has significant cost-savings. With Segregated Witness, the cost difference shrinks significantly in absolute as well as relative terms. While it would require inputs to become cheaper than outputs to incentivize UTXO set consolidation, this discount reduces the incentive to create new UTXO in order to avoid using more inputs. - -Segregated Witness therefore has two main effects on the fees paid by bitcoin users. Firstly, segwit reduces the overall cost of transactions by discounting witness data and increasing the capacity of the Bitcoin blockchain. Secondly, segwit’s discount on witness data partially mitigates a misalignment of incentives that may have inadvertently created more bloat in the UTXO set.((("", startref="Tadv07")))((("", startref="Ssegwit07"))) diff --git a/ch07_authorization-authentication.adoc b/ch07_authorization-authentication.adoc new file mode 100644 index 00000000..69bdf972 --- /dev/null +++ b/ch07_authorization-authentication.adoc @@ -0,0 +1,2004 @@ +[[c_authorization_authentication]] +== Authorization and Authentication + +When you receive bitcoins, you have to decide who will have permission +to spend them, ((("authorization")))((("authentication")))called _authorization_. You also have to decide how full +nodes will distinguish the authorized spenders from everyone else, +called _authentication_. Your authorization instructions and the +spender proof of authentication will be checked by thousands of +independent full nodes, which all need to come to the same conclusion +that a spend was authorized and authenticated in order for the transaction +containing it to be valid. + +The original description of Bitcoin used a public key for authorization. +Alice paid Bob by putting his public key in the output of a transaction. +Authentication came from Bob in the form of a signature that committed to a +spending transaction, such as from Bob to Carol. + +The actual version of Bitcoin that was originally released provided a +more flexible mechanism for both authorization and authentication. +Improvements since then have only increased that flexibility. In this +chapter, we'll explore those features and see how they're most commonly +used. + +[[tx_script]] +=== Transaction Scripts and Script Language + +The((("transaction scripts", see="scripts")))((("Script programming language"))) original version of Bitcoin introduced a new +programming language called _Script_, a Forth-like stack-based +language. Both the script placed in an output and the legacy +input script used in a spending transaction are written in this scripting +language. + +Script is a very simple language. It requires minimal processing and +cannot easily do many of the fancy things modern programming languages +can do. + +When legacy transactions were the +most commonly used type of transaction, the majority of transactions processed +through the Bitcoin network had the form "Payment to Bob's Bitcoin +address" and used a script called a pay to public key hash (P2PKH) script. +However, Bitcoin transactions are not limited to the "Payment to Bob's +Bitcoin address" script. In fact, scripts can be written to express a +vast variety of complex conditions. In order to understand these more +complex scripts, we must first understand the basics of transaction +scripts and Script language. + +In this section, we will demonstrate the basic components of the Bitcoin +transaction scripting language and show how it can be used to express +conditions for spending and how those conditions can be satisfied. + +[TIP] +==== +Bitcoin transaction validation((("transactions", "validating")))((("validating", "transactions"))) is not based on +a static pattern but instead is achieved through the execution of a +scripting language. This language allows for a nearly infinite variety +of conditions to be expressed. +==== + +==== Turing Incompleteness + +The ((("scripts", "Turing incompleteness")))((("Turing Complete")))Bitcoin transaction script language +contains many operators, but is deliberately limited in one important +way--there are no loops or complex flow control capabilities other than +conditional flow control. This ensures that the language is not _Turing +Complete_, meaning that scripts have limited complexity and predictable +execution times. Script is not a general-purpose language. +These +limitations ensure that the language cannot be used to create an +infinite loop or other form of "logic bomb" that could be embedded in a +transaction in a way that causes a denial-of-service attack against the +Bitcoin network. Remember, every transaction is validated by every full +node on the Bitcoin network. A limited language prevents the transaction +validation mechanism from being used as a vulnerability. + +==== Stateless Verification + +The ((("scripts", "stateless verification")))((("stateless script verification")))((("verifying", "scripts")))Bitcoin transaction script language is +stateless, in that there is no state prior to execution of the script +or state saved after execution of the script. All the +information needed to execute a script is contained within the script +and the transaction executing the script. A +script will predictably execute the same way on any system. If your +system verified a script, you can be sure that every other system in the +Bitcoin network will also verify the script, meaning that a valid +transaction is valid for everyone and everyone knows this. This +predictability of outcomes is an essential benefit of the Bitcoin +system. + +[[tx_lock_unlock]] +==== Script Construction + +Bitcoin's ((("scripts", "input/output", "constructing", id="script-input-output-construct")))((("input scripts", "constructing", id="input-script-construct")))((("output scripts", "constructing", id="output-script-construct")))legacy transaction validation engine relies on two parts of scripts +to validate transactions: an output script and an input script. + +An output script +specifies the conditions that must be met to spend the output in the +future, such as who is authorized to spend the output and how they will +be authenticated. + +An input script is a script that satisfies the +conditions placed in an output script and allows the output +to be spent. Input scripts are part of every transaction input. Most +of the time in legacy transactions they contain a digital signature produced by the user's +wallet from his or her private key, but not all input scripts +must contain signatures. + +Every Bitcoin((("transactions", "validating")))((("validating", "transactions"))) validating node will validate transactions by executing +the output and input scripts. As we saw in +<>, each input contains an outpoint that refers to a +previous transaction output. The input also contains an input script. The +validation software will copy the input script, retrieve the UTXO +referenced by the input, and copy the output script from that UTXO. The +input and output scripts are then executed together. The input is +valid if the input script satisfies the output script's conditions +(see <>). All the inputs are validated independently as +part of the overall validation of the transaction. + +Note that the preceding steps involve making copies of all data. The +original data in the previous output and current input is never changed. +In particular, the previous output is invariable and unaffected by +failed attempts to spend it. Only a valid transaction that correctly +satisfies the conditions of the output script results in the output being +considered as "spent." + +<> is an example of the output and +input scripts for the most common type of legacy Bitcoin transaction (a +payment to a public key hash), showing the combined script resulting +from the concatenation of the scripts prior to +validation. + +[[input_and_output_scripts_legacy]] +.Combining input and output scripts to evaluate a transaction script. +image::images/mbc3_0701.png["input_and_output_scripts"] + + +===== The script execution stack + +Bitcoin's ((("scripts", "stack", id="script-stack")))((("stack", id="stack")))scripting language is called a stack-based language because it +uses a data structure called a _stack_. A stack is a very simple data +structure that can be visualized as a stack of cards. A stack has two +base operations: push and pop. Push adds an item on top of the stack. Pop +removes the top item from the stack. + +The scripting language executes the script by processing each item from +left to right. Numbers (data constants) are pushed onto the stack. +Operators push or pop one or more parameters from the stack, act on +them, and might push a result onto the stack. For example, +OP_ADD+ will +pop two items from the stack, add them, and push the resulting sum onto +the stack. + +Conditional operators evaluate a condition, producing a boolean result +of +TRUE+ or +FALSE+. For example, +OP_EQUAL+ pops two items from the stack +and pushes +TRUE+ (+TRUE+ is represented by the number 1) if they are equal +or +FALSE+ (represented by 0) if they are not equal. Bitcoin +transaction scripts usually contain a conditional operator so that they +can produce the +TRUE+ result that signifies a valid ((("scripts", "stack", startref="script-stack")))((("stack", startref="stack")))transaction. + +===== A simple script + +Now let's ((("scripts", "input/output", "examples of", id="script-input-output-example")))((("input scripts", "examples of", id="input-script-example")))((("output scripts", "examples of", id="output-script-example")))apply what we've learned about scripts and stacks to some simple examples. + +As we will see in <>, the script +2 3 OP_ADD 5 OP_EQUAL+ +demonstrates the arithmetic addition operator +OP_ADD+, adding two +numbers and putting the result on the stack, followed by the conditional +operator +OP_EQUAL+, which checks that the resulting sum is equal to ++5+. For brevity, the +OP_+ prefix may sometimes be omitted in examples +in this book. For more details on the available script operators and +functions, see https://oreil.ly/21vH9[Bitcoin Wiki's script +page]. + +Although most legacy output scripts refer to a public key hash (essentially, a +legacy Bitcoin address), thereby requiring proof of ownership to spend the +funds, the script does not have to be that complex. Any combination of +output and input scripts that results in a +TRUE+ value is valid. The +simple arithmetic we used as an example of the scripting language is +also a valid script. + +Use part of the arithmetic example script as the output script: + +---- +3 OP_ADD 5 OP_EQUAL +---- + +which can be satisfied by a transaction containing an input with the +input script: + +---- +2 +---- + +The validation software combines the scripts: + +---- +2 3 OP_ADD 5 OP_EQUAL +---- + +As we see in <>, when +this script is executed, the result is +OP_TRUE+, making the transaction valid. Although this is a valid transaction +output script, note that the resulting UTXO can be spent by anyone with +the arithmetic skills +to know that the number 2 satisfies the script. + + +[[simplemath_script]] +.Bitcoin's script validation doing simple math. +image::images/mbc3_0702.png["TxScriptSimpleMathExample"] + +[TIP] +==== +Transactions are valid if the +top result on the stack is +TRUE+, which is any +nonzero value. +Transactions are invalid if the top value on the stack is +FALSE+ (the +value zero or an empty stack), the script +execution is halted explicitly by an operator (such as +VERIFY+, ++OP_RETURN+), or the script was not semantically valid (such as +containing an +OP_IF+ statement that was not terminated by an +OP_ENDIF+ +opcode). For details, see https://oreil.ly/J2DXt[Bitcoin +Wiki's script page]. +==== + + +The following is a slightly more complex script, which calculates +2 + 7 – 3 + 1. Notice that when the script contains several operators in +a row, the stack allows the results of one operator to be acted upon by +the next operator: + +---- +2 7 OP_ADD 3 OP_SUB 1 OP_ADD 7 OP_EQUAL +---- + +Try validating the preceding script yourself using pencil and paper. +When the script execution ends, you should be left with a +TRUE+ value +on the ((("scripts", "input/output", "examples of", startref="script-input-output-example")))((("input scripts", "examples of", startref="input-script-example")))((("output scripts", "examples of", startref="output-script-example")))stack. + +[[script_exec]] +===== Separate execution of output and input scripts + +In the ((("scripts", "input/output", "separate execution")))((("input scripts", "separate execution from output scripts")))((("output scripts", "separate execution from input scripts")))original Bitcoin +client, output and input scripts were concatenated and executed +in sequence. For security reasons, this was changed in 2010 because of +a vulnerability known as the +1 OP_RETURN+ bug. In the current +implementation, the scripts are executed separately with the stack +transferred between the two executions. + +First, the input script is executed using the stack execution +engine. If the input script is executed without errors and has +no operations left over, the stack is copied and the +output script is executed. If the result of executing the output script +with the stack data copied from the input script is +TRUE+, +the input script has succeeded in resolving the conditions imposed +by the output script and, therefore, the input is a valid authorization +to spend the UTXO. If any result other than +TRUE+ remains after +execution of the combined script, the input is invalid because it has +failed to satisfy the spending conditions placed on the output. + +//SOMEDAY:implications of not being able to use script in input script + +[[p2pkh]] +==== Pay to Public Key Hash + + +A ((("P2PKH (pay to public key hash)")))((("addresses", "P2PKH (pay to public key hash)")))((("scripts", "P2PKH (pay to public key hash)")))pay to public key hash (P2PKH) script uses an output script that +contains a hash that commits to a public key. P2PKH is best known as +the basis for a legacy Bitcoin address. A P2PKH output can be spent by +presenting a public key that matches the hash commitment and a digital +signature created by the corresponding private key (see +<>). Let's look at an example of a P2PKH output script: + +---- +OP_DUP OP_HASH160 OP_EQUALVERIFY OP_CHECKSIG +---- + +[role="less_space pagebreak-before"] +The +Key Hash+ is the data that would be encoded into a legacy base58check +address. Most applications would show the _public key hash_ in a script +using hexadecimal encoding and not the familiar Bitcoin +address base58check format that begins with [.keep-together]#a "1."# + +The preceding output script can be satisfied with an input script +of the form: + +---- + +---- + +The two scripts together would form the following combined validation +script: + +---- + OP_DUP OP_HASH160 OP_EQUALVERIFY OP_CHECKSIG +---- + +The result will be +TRUE+ if the input script +has a valid signature from Bob's private key that corresponds to +the public key hash set as an encumbrance. + +Figures pass:[#P2PubKHash1] and pass:[#P2PubKHash2] show (in two parts) a step-by-step +execution of the combined script, which will prove this is a valid +transaction. + +[[P2PubKHash1]] +.Evaluating a script for a P2PKH transaction (part 1 of 2). +image::images/mbc3_0703.png["Tx_Script_P2PubKeyHash_1"] + +[[P2PubKHash2]] +.Evaluating a script for a P2PKH transaction (part 2 of 2). +image::images/mbc3_0704.png["Tx_Script_P2PubKeyHash_2"] + +[[multisig]] +=== Scripted Multisignatures + +Multisignature scripts((("scripted multisignatures", id="script-multisignature")))((("multisignature scripts", id="multi-script"))) set a condition where _k_ public keys +are recorded in the script and at least _t_ of those must provide +signatures to spend the funds, called _t_-of-_k_. +For example, a 2-of-3 multisignature is one +where three public keys are listed as potential signers and at least two +of those must be used to create signatures for a valid transaction to +spend the funds. + +[TIP] +==== +Some Bitcoin documentation, including earlier editions of this book, +uses the term "m-of-n" for a traditional multisignature. However, it's hard +to tell "m" and "n" apart when they're spoken, so we use the alternative +_t_-of-_k_. Both phrases refer to the same type of signature scheme. +==== + +The general form of an output script setting a _t_-of-_k_ multisignature +condition is: + +---- +t ... k OP_CHECKMULTISIG +---- + +where _k_ is the total number of listed public keys and _t_ is the threshold +of required signatures to spend the output. + +An output script setting a 2-of-3 multisignature condition looks like +this: + +---- +2 3 OP_CHECKMULTISIG +---- + +The preceding output script can be satisfied with an input script +containing [.keep-together]#signatures:# + +---- + +---- + +or any combination of two signatures from the private keys corresponding +to the three listed public keys. + +The two scripts together would form the combined validation script: + +---- + 2 3 OP_CHECKMULTISIG +---- + +When executed, this combined script will evaluate to +TRUE+ if +the input script has +two valid signatures from private keys that correspond to two of +the three public keys set as an encumbrance. + +At this time, Bitcoin Core's transaction relay policy limits multisignature output scripts to, at most, three +listed public keys, meaning you can do anything from a 1-of-1 to a +3-of-3 multisignature or any combination within that range. +You may want to check the +IsStandard()+ function to see what is currently +accepted by the network. Note that the limit of three keys applies only to +standard (also known as "bare") multisignature scripts, not to +scripts wrapped in another structure like P2SH, P2WSH, or P2TR. +P2SH multisignature scripts are limited by both policy and consensus to +15 keys, allowing for up to a 15-of-15 multisignature. We will learn about +P2SH in <>. All other scripts are consensus limited to 20 keys +per +OP_CHECKMULTISIG+ or +OP_CHECKMULTISIGVERIFY+ opcode, although one +script may include multiple of those opcodes. + +[role="less_space pagebreak-before"] +[[multisig_bug]] +==== An Oddity in CHECKMULTISIG Execution + +There ((("OP_CHECKMULTISIG execution", id="op-checkmultisig")))is an oddity in +++OP_CHECKMULTISIG++'s execution that requires a slight workaround. When ++OP_CHECKMULTISIG+ executes, it should consume _t_ + _k_ + 2 items on the stack as +parameters. However, due to the oddity, +OP_CHECKMULTISIG+ will pop an extra +value or one value more than expected. + +Let's look at this in greater detail using the previous validation +example: + +---- + 2 3 OP_CHECKMULTISIG +---- + +First, +OP_CHECKMULTISIG+ pops the top item, which is _k_ (in this example +"3"). Then it pops _k_ items, which are the public keys that can sign; +in this example, public keys A, B, and C. Then, it pops one item, which +is _t_, the quorum (how many signatures are needed). Here _t_ = 2. At this +point, +OP_CHECKMULTISIG+ should pop the final _t_ items, which are the +signatures, and see if they are valid. However, unfortunately, an oddity in +the implementation causes +OP_CHECKMULTISIG+ to pop one more item (_t_ + 1 +total) than it should. The extra item is called((("dummy stack element"))) the _dummy stack +element_, and it is disregarded when checking the +signatures so it has no direct effect on +OP_CHECKMULTISIG+ itself. +However, the dummy element must be present because, if it isn't present +when +OP_CHECKMULTISIG+ attempts to pop on an empty stack, it will cause a +stack error and script failure (marking the transaction as invalid). +Because the dummy element is disregarded, it can be anything. It became the custom early on to use +OP_0+, which later became a +relay policy rule and eventually a consensus rule (with the enforcement of BIP147). + +Because popping the dummy element is part of the consensus rules, it must now be +replicated forever. Therefore a script should look +like this: + +---- +OP_0 2 3 OP_CHECKMULTISIG +---- + +Thus the input script actually used in multisig is not: + +---- + +---- + +but instead it is: + +---- +OP_0 +---- + +Some people believe this oddity was a bug in the original code for +Bitcoin, but a plausible alternative explanation exists. Verifying +_t_-of-_k_ signatures can require many more than _t_ or _k_ signature checking +operations. Let's consider a simple example of 1-in-5, with the +following combined script: + +---- + 1 5 OP_CHECKMULTISIG +---- + +The signature is checked first against +key0+, then +key1+, and then +the other keys before it is finally compared to its corresponding ++key4+. That means five signature checking operations need to be +performed even though there's only one signature. One way to eliminate +this redundancy would have been to provide +OP_CHECKMULTISIG+ a map +indicating which provided signature corresponds to which public key, +allowing the +OP_CHECKMULTISIG+ operation to only perform exactly _t_ +signature-checking operations. It's possible that Bitcoin's original +developer added the extra element (which we now call the dummy stack +element) in the original version of Bitcoin so they could add the +feature for allowing a map to be passed in a later soft fork. However, +that feature was never implemented, and the BIP147 update to the +consensus rules in 2017 makes it impossible to add that feature in the +future. + +Only Bitcoin's original developer could tell us whether the dummy stack +element was the result of a bug or a plan for a future upgrade. In this +book, we simply call it an oddity. + +From now on, if you see a multisig script, you should expect +to see an extra +OP_0+ in the beginning, whose only purpose is as a +workaround to an oddity in the ((("scripted multisignatures", startref="script-multisignature")))((("multisignature scripts", startref="multi-script")))((("OP_CHECKMULTISIG execution", startref="op-checkmultisig")))consensus rules. + +[[p2sh]] +=== Pay to Script Hash + +Pay to script hash (P2SH) was((("scripts", "P2SH (pay to script hash)", id="script-p2sh")))((("addresses", "P2SH (pay to script hash)", id="address-p2sh-ch7")))((("P2SH (pay to script hash)", id="p2sh-ch7"))) +introduced in 2012 as a powerful new type of operation that greatly +simplifies the use of complex scripts. To explain the need +for P2SH, let's look at a practical example. + + +Mohammed is an electronics importer based in Dubai. Mohammed's +company uses Bitcoin's multisignature feature extensively for its +corporate accounts. Multisignature scripts are one of the most common +uses of Bitcoin's advanced scripting capabilities and are a very +powerful feature. Mohammed's company +uses a multisignature script for all customer payments. +Any payments made by customers are locked in such +a way that they require at least two signatures to release. Mohammed, +his three partners, and their attorney can each provide one signature. +A multisignature scheme like that offers corporate governance +controls and protects against theft, embezzlement, or loss. + +The resulting script is quite long and looks like this: + +---- +2 + 5 OP_CHECKMULTISIG +---- + +Although multisignature scripts are a powerful feature, they are +cumbersome to use. Given the preceding script, Mohammed would have to +communicate this script to every customer prior to payment. Each +customer would have to use special Bitcoin wallet software with the +ability to create custom transaction scripts. +Furthermore, the resulting transaction would be about five times larger +than a simple payment transaction, because this script contains very +long public keys. The burden of that extra data would be +borne by the customer in the form of extra transaction fees. Finally, a large transaction +script like this would be carried in the UTXO set in every full +node, until it was spent. All of these issues make using complex output scripts +difficult in practice. + +P2SH was developed to resolve these practical difficulties and to make +the use of complex scripts as easy as a payment to a single-key Bitcoin address. +With P2SH payments, the complex script is replaced with a +commitment, the digest of a cryptographic hash. When a transaction attempting +to spend the UTXO is presented later, it must contain the script that +matches the commitment in addition to the data that satisfies the script. In simple terms, +P2SH means "pay to a script matching this hash, a script that will be +presented later when this output is spent." + +In P2SH +transactions, the script that is replaced by a hash is ((("redeem scripts")))referred +to as the _redeem script_ because it is presented to the system at +redemption time rather than as an output script. <> shows +the script without P2SH and <> shows the same script encoded +with P2SH. + +++++ + + + + + + + + + + + + +
Complex script without P2SH

Output script

2 PubKey1 PubKey2 PubKey3 PubKey4 PubKey5 5 OP_CHECKMULTISIG

Input script

Sig1 Sig2

+ + + + + + + + + + + + + + + + +
Complex script as P2SH

Redeem script

2 PubKey1 PubKey2 PubKey3 PubKey4 PubKey5 5 OP_CHECKMULTISIG

Output script

OP_HASH160 <20-byte hash of redeem script> OP_EQUAL

Input script

Sig1 Sig2 <redeem script>

+++++ + +As you can see from the tables, with P2SH, the complex script that +details the conditions for spending the output (redeem script) is not +presented in the output script. Instead, only a hash of it is in the +output script, and the redeem script itself is presented later as part +of the input script when the output is spent. This shifts the burden +in fees and complexity from the spender to the receiver of the +transaction. + +Let's look at Mohammed's company, the complex multisignature script, and +the resulting P2SH scripts. + +First, the multisignature script that Mohammed's company uses for all +incoming payments from customers: + +---- +2 + 5 OP_CHECKMULTISIG +---- + +This entire script can instead be represented by a 20-byte cryptographic +hash by first applying the SHA256 hashing algorithm and then applying +the RIPEMD-160 algorithm on the result. For example, starting with the +hash of Mohammed's redeem script: + +---- +54c557e07dde5bb6cb791c7a540e0a4796f5e97e +---- + +A P2SH transaction locks the output to this hash instead of the longer +redeem script, using a special output script template: + +---- +OP_HASH160 54c557e07dde5bb6cb791c7a540e0a4796f5e97e OP_EQUAL +---- + +which, as you can see, is much shorter. Instead of "pay to this 5-key +multisignature script," the P2SH equivalent transaction is "pay to a +script with this hash." A customer making a payment to Mohammed's +company need only include this much shorter output script in his +payment. When Mohammed and his partners want to spend this UTXO, they +must present the original redeem script (the one whose hash locked the +UTXO) and the signatures necessary to unlock it, like this: + +---- + <2 PK1 PK2 PK3 PK4 PK5 5 OP_CHECKMULTISIG> +---- + +The two scripts are combined in two stages. First, the redeem script is +checked against the output script to make sure the hash matches: + +---- +<2 PK1 PK2 PK3 PK4 PK5 5 OP_CHECKMULTISIG> OP_HASH160