arno/bitcoinbook
1
0
Fork 0
mirror of https://github.com/bitcoinbook/bitcoinbook synced 2024-11-22 08:08:11 +00:00
Code Issues Releases Wiki Activity

added numbers to ch. files

Browse Source
This commit is contained in:
theresa jones 2023-09-11 18:22:28 -07:00
parent ea547997bd
commit df462b1a1c
15 changed files with 15780 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
atlas.json
View File

@ -7,20 +7,20 @@
"dedication.html",
"toc.html",
"preface.asciidoc",
"intro.adoc",
"overview.adoc",
"bitcoin-core.adoc",
"keys.adoc",
"wallets.adoc",
"transactions.adoc",
"authorization-authentication.adoc",
"signatures.adoc",
"fees.adoc",
"network.adoc",
"blockchain.adoc",
"mining.adoc",
"security.adoc",
"applications.adoc",
"ch01_intro.adoc",
"ch02_overview.adoc",
"ch03_bitcoin-core.adoc",
"ch04_keys.adoc",
"ch05_wallets.adoc",
"ch06_transactions.adoc",
"ch07_authorization-authentication.adoc",
"ch08_signatures.adoc",
"ch09_fees.adoc",
"ch10_network.adoc",
"ch11_blockchain.adoc",
"ch12_mining.adoc",
"ch13_security.adoc",
"ch14_applications.adoc",
"whitepaper.adoc",
"errata.adoc",
"bips.adoc",
@ -67,3 +67,4 @@
"lang": "en",
"accent_color": ""
}

502
ch01_intro.adoc Normal file
View File

@ -0,0 +1,502 @@
[role="pagenumrestart"]
[[ch01_intro_what_is_bitcoin]]
== Introduction
Bitcoin is a collection of concepts and technologies that form the basis of a digital money ecosystem. Units of currency called bitcoin are used to store and transmit value among participants in the Bitcoin network. Bitcoin users communicate with each other using the Bitcoin protocol primarily via the internet, although other transport networks can also be used. The Bitcoin protocol stack, available as open source software, can be run on a wide range of computing devices, including laptops and smartphones, making the technology easily accessible.
[TIP]
====
In this book, the unit of currency is called "bitcoin" with a small _b_,
and the system is called "Bitcoin", with a capital _B_.
====
Users can transfer bitcoin over the network to do just about anything
that can be done with conventional currencies, including buying and selling
goods, sending money to people or organizations, or extending credit. Bitcoin
can be purchased, sold, and exchanged for other currencies at
specialized currency exchanges. Bitcoin is arguably the perfect form
of money for the internet because it is fast, secure, and borderless.
Unlike traditional currencies, the bitcoin currency is entirely virtual. There are no
physical coins or even individual digital coins. The coins are implied in
transactions that transfer value from spender to receiver. Users of
Bitcoin control keys that allow them to prove ownership of bitcoin in the
Bitcoin network. With these keys, they can sign transactions to unlock
the value and spend it by transferring it to a new owner. Keys are often
stored in a digital wallet on each user’s computer or smartphone.
Possession of the key that can sign a transaction is the only
prerequisite to spending bitcoin, putting the control entirely in the
hands of each user.
Bitcoin is a distributed, peer-to-peer system. As such, there is no
central server or point of control. Units of bitcoin
are created through a process called "mining," which involves repeatedly
performing a computational task that references a list of recent Bitcoin
transactions. Any participant in the Bitcoin network may operate as a
miner, using their computing devices to help secure
transactions. Every 10 minutes, on average, one Bitcoin miner can add security to
past transactions and is rewarded with both brand new
bitcoins and the fees paid by recent transactions. Essentially, Bitcoin
mining decentralizes the currency-issuance
and clearing functions of a central bank and replaces the need for any
central bank.
//-- Math for following paragraph --
//total_btc = 0
//for i in range(0, 10_000_000):
// total_btc += (50 / (2**int(i/210000)) )
// if total_btc / 21e6 > 0.99:
// print(i)
// break
The Bitcoin protocol includes built-in algorithms that regulate the
mining function across the network. The difficulty of the computational
task that miners must perform is adjusted dynamically so that, on
average, someone succeeds every 10 minutes regardless of how many miners
(and how much processing) are competing at any moment. The protocol also
periodically decreases the number of new bitcoins that are created,
limiting the total number of bitcoins that will ever be created to a fixed total
just below 21 million coins. The result is that the number of bitcoins in
circulation closely follows an easily predictable curve where half of
the remaining coins are added to circulation every four years. At
approximately block 1,411,200, which is expected to be produced around
the year 2035, 99% of all bitcoins
that will ever exist will have been issued. Due to Bitcoin's
diminishing rate of issuance, over the long term, the Bitcoin currency
is deflationary. Furthermore, nobody can force you to accept
any bitcoins that were created beyond the
expected issuance rate.
Behind the scenes, Bitcoin is also the name of the protocol, a peer-to-peer network, and a distributed computing innovation. Bitcoin builds on decades of research in cryptography and distributed systems and includes at least four key innovations brought together in a unique and powerful combination. Bitcoin consists of:
* A decentralized peer-to-peer network (the Bitcoin protocol)
* A public transaction journal (the blockchain)
* A set of rules for independent transaction validation and currency issuance (consensus rules)
* A mechanism for reaching global decentralized consensus on the valid blockchain (Proof-of-Work algorithm)
As a developer, I see Bitcoin as akin to the internet of money, a network for propagating value and securing the ownership of digital assets via distributed computation. There's a lot more to Bitcoin than first meets the eye.
In this chapter we'll get started by explaining some of the main concepts and terms, getting the necessary software, and using Bitcoin for simple transactions. In the following chapters, we'll start unwrapping the layers of technology that make Bitcoin possible and examine the inner workings of the Bitcoin network and protocol.
[role="pagebreak-before less_space"]
.Digital Currencies Before Bitcoin
****
The emergence of viable digital money is closely linked to developments in cryptography. This is not surprising when one considers the fundamental challenges involved with using bits to represent value that can be exchanged for goods and services. Three basic questions for anyone accepting digital money are:
1. Can I trust that the money is authentic and not counterfeit?
2. Can I trust that the digital money can only be spent once (known as the “double-spend” problem)?
3. Can I be sure that no one else can claim this money belongs to them and not me?
Issuers of paper money are constantly battling the counterfeiting problem by using increasingly sophisticated papers and printing technology. Physical money addresses the double-spend issue easily because the same paper note cannot be in two places at once. Of course, conventional money is also often stored and transmitted digitally. In these cases, the counterfeiting and double-spend issues are handled by clearing all electronic transactions through central authorities that have a global view of the currency in circulation. For digital money, which cannot take advantage of esoteric inks or holographic strips, cryptography provides the basis for trusting the legitimacy of a user’s claim to value. Specifically, cryptographic digital signatures enable a user to sign a digital asset or transaction proving the ownership of that asset. With the appropriate architecture, digital signatures also can be used to address the double-spend issue.
When cryptography started becoming more broadly available and understood in the late 1980s, many researchers began trying to use cryptography to build digital currencies. These early digital currency projects issued digital money, usually backed by a national currency or precious metal such as gold.
Although these earlier digital currencies worked, they were centralized and, as a result, were easy to attack by governments and hackers. Early digital currencies used a central clearinghouse to settle all transactions at regular intervals, just like a traditional banking system. Unfortunately, in most cases these nascent digital currencies were targeted by worried governments and eventually litigated out of existence. Some failed in spectacular crashes when the parent company liquidated abruptly. To be robust against intervention by antagonists, whether legitimate governments or criminal elements, a _decentralized_ digital currency was needed to avoid a single point of attack. Bitcoin is such a system, decentralized by design, and free of any central authority or point of control that can be attacked or corrupted.
****
=== History of Bitcoin
Bitcoin was first described in 2008 with the publication of a
paper titled "Bitcoin: A Peer-to-Peer Electronic Cash
System,"footnote:["Bitcoin: A Peer-to-Peer Electronic Cash System,"
Satoshi Nakamoto (https://bitcoin.org/bitcoin.pdf).] written under the
alias of Satoshi Nakamoto (see <<satoshi_whitepaper>>). Nakamoto
combined several prior inventions such as digital signatures and Hashcash to create
a completely decentralized electronic cash system that does not rely on
a central authority for currency issuance or settlement and validation
of transactions. A key innovation was to use a distributed computation
system (called a "Proof-of-Work" algorithm) to conduct a global
lottery every 10 minutes on average, allowing the decentralized network to
arrive at _consensus_ about the state of transactions. This
elegantly solves the issue of double-spend where a single currency unit
can be spent twice. Previously, the double-spend problem was a weakness
of digital currency and was addressed by clearing all transactions
through a central clearinghouse.
The Bitcoin network started in 2009, based on a reference implementation
published by Nakamoto and since revised by many other programmers. The
number and power of machines running the Proof-of-Work algorithm
(mining) that provides security and resilience for Bitcoin have
increased exponentially, and their combined computational power now
exceeds the combined number of computing operations of the
world's top supercomputers.
Satoshi Nakamoto withdrew from the public in April 2011, leaving the responsibility of developing the code and network to a thriving group of volunteers. The identity of the person or people behind Bitcoin is still unknown. However, neither Satoshi Nakamoto nor anyone else exerts individual control over the Bitcoin system, which operates based on fully transparent mathematical principles, open source code, and consensus among participants. The invention itself is groundbreaking and has already spawned new science in the fields of distributed computing, economics, and econometrics.
.A Solution to a Distributed Computing Problem
****
Satoshi Nakamoto's invention is
also a practical and novel solution to a problem in distributed
computing, known as the "Byzantine Generals' Problem." Briefly, the
problem consists of trying to get multiple participants without a leader
to agree on a course of action by exchanging information over an
unreliable and potentially compromised network. Satoshi Nakamoto's solution, which uses the concept of
Proof-of-Work to achieve consensus _without a central trusted
authority_, represents a breakthrough in distributed computing.
****
=== Getting Started
Bitcoin is a protocol that can be accessed using an
application that speaks the protocol. A "Bitcoin wallet" is the
most common user interface to the Bitcoin system, just like a web
browser is the most common user interface for the HTTP protocol. There
are many implementations and brands of Bitcoin wallets, just like there
are many brands of web browsers (e.g., Chrome, Safari, and Firefox).
And just like we all have our favorite browsers,
Bitcoin wallets vary in quality, performance, security, privacy, and
reliability. There is also a reference implementation of the Bitcoin
protocol that includes a wallet, known as "Bitcoin Core," which is
derived from the original implementation written by Satoshi Nakamoto.
==== Choosing a Bitcoin Wallet
Bitcoin wallets are one of the most actively developed applications in the Bitcoin ecosystem. There is intense competition, and while a new wallet is probably being developed right now, several wallets from last year are no longer actively maintained. Many wallets focus on specific platforms or specific uses and some are more suitable for beginners while others are filled with features for advanced users. Choosing a wallet is highly subjective and depends on the use and user expertise. Therefore it would be pointless to recommend a specific brand or wallet. However, we can categorize Bitcoin wallets according to their platform and function and provide some clarity about all the different types of wallets that exist. It is worth trying out several different wallets until you find one that fits your needs.
===== Types of Bitcoin wallets
Bitcoin wallets can be categorized as follows, according to the platform:
Desktop wallet:: A desktop wallet was the first type of Bitcoin wallet created as a reference implementation and many users run desktop wallets for the features, autonomy, and control they offer. Running on general-use operating systems such as Windows and Mac OS has certain security disadvantages, however, as these platforms are often insecure and poorly configured.
Mobile wallet:: A mobile wallet is the most common type of Bitcoin
wallet. Running on smart-phone operating systems such as Apple iOS and
Android, these wallets are often a great choice for new users. Many are
designed for simplicity and ease-of-use, but there are also fully
featured mobile wallets for power users. To avoid downloading and
storing large amounts of data, most mobile wallets retrieve information
from remote servers, reducing your privacy by disclosing to third
parties information about your Bitcoin addresses and balances.
Web wallet:: Web wallets are accessed through a web browser and store
the user's wallet on a server owned by a third party. This is similar to
webmail in that it relies entirely on a third-party server. Some of
these services operate using client-side code running in the user's
browser, which keeps control of the Bitcoin keys in the hands of the
user, although the user's dependence on the server still compromises
their privacy. Most, however, take control of the Bitcoin keys from
users in exchange for ease-of-use. It is inadvisable
to store large amounts of bitcoin on third-party systems.
Hardware signing devices:: Hardware signing devices are devices that can
store keys and sign transactions using special-purpose hardware and
firmware. They usually
connect to a desktop, mobile, or web wallet via USB cable,
near-field-communication (NFC), or a camera with QR codes. By handling
all Bitcoin-related operations on the specialized hardware, these
wallets are less vulnerable to many types of attacks. Hardware signing
devices are sometimes called "hardware wallets", but they need to be
paired with a full-featured wallet to send and receive transactions, and
the security and privacy offered by that paired wallet plays a critical
role in how much security and privacy the user obtains when using the
hardware signing device.
===== Full-node vs. Lightweight
Another way to categorize Bitcoin wallets is by their degree of autonomy and how they interact with the Bitcoin network:
Full-node:: A full node is a program that validates the
entire history of Bitcoin transactions (every transaction by every user,
ever). Optionally, full nodes can also store previously validated
transactions and serve data to other Bitcoin programs, either on the
same computer or over the internet. A full node uses substantial
computer resources--about the same as watching an hour-long streaming
video for each day of Bitcoin transactions--but the full node offers
complete autonomy to its users.
Lightweight client::
A lightweight client, also known as a simplified-payment-verification (SPV) client,
connects to a full node or other remote server for receiving and sending
Bitcoin transaction information, but stores the user wallet locally,
partially validates the transactions it receives, and independently
creates outgoing transactions.
Third-party API client:: A third-party
API client is one that interacts with Bitcoin through a third-party
system of application programming interfaces (APIs), rather than by
connecting to the Bitcoin network directly. The wallet may be stored by
the user or by third-party servers, but the client trusts the remote
server to provide it with accurate information and protect its privacy.
[TIP]
====
Bitcoin is a Peer-to-Peer (P2P) network. Full nodes are the _peers:_
each peer individually validates every confirmed transaction and can
provide data to its user with complete authority. Lightweight wallets
and other software are _clients:_ each client depends on one or more peers
to provide it with valid data. Bitcoin clients can perform secondary
validation on some of the data they receive and make connections to
multiple peers to reduce their depedence on the integrity of a single
peer, but the security of a client ultimately relies on the integrity of
its peers.
====
===== Who controls the keys
A very important additional consideration is _who controls the keys_. As
we will see in subsequent chapters, access to bitcoins is
controlled by "private keys", which are like very long PIN numbers. If
you are the only one to have *control* over these private
keys, you are in control of your bitcoins. Conversely, if you do not have
control, then your bitcoins are managed by a third-party who
ultimately controls your funds on your behalf. Key management software falls into two
important categories based on control: _wallets_, where you
control the keys, and the funds and accounts with custodians where some
third-party controls the keys. To emphasize this point, I (Andreas)
coined the phrase:
_Your keys, your coins. Not your keys, not your coins_.
Combining these categorizations, many Bitcoin wallets fall into a few
groups, with the three most common being desktop full node
(you control the keys), mobile lightweight wallet (you control the keys), and web-based
accounts with third parties (you don't control the keys). The lines between different categories
are sometimes blurry, as software runs on multiple platforms and can
interact with the network in different ways.
==== Quick Start
Alice is not a
technical user and only recently heard about Bitcoin from her friend
Joe. While at a party, Joe is enthusiastically explaining
Bitcoin to everyone around him and is offering a demonstration. Intrigued,
Alice asks how she can get started with Bitcoin. Joe says that a mobile
wallet is best for new users and he recommends a few of his favorite
wallets. Alice downloads one of Joe's recommendations
and installs it on her phone.
When Alice runs her wallet application for the first time, she chooses
the option to create a new Bitcoin wallet. Because the wallet she has
chosen is a non-custodial wallet, Alice (and only Alice) will be in
control of her keys. Therefore, she bears responsibility for backing
them up, since losing the keys means she loses access to her bitcoins. To
facilitate this, her wallet produces a _recovery code_ that can be used
to restore her wallet.
[[recovery_code_intro]]
==== Recovery Codes
Most modern non-custodial Bitcoin wallets will provide a _recovery
code_ for their user
to back up. The recovery code usually consists of numbers, letters, or words
selected randomly by the software, and is used as the basis for the keys
that are generated by the wallet. See <<recovery_code_sample>> for
examples.
[[recovery_code_sample]]
.Sample Recovery Codes
[cols="1,1"]
|===
| Wallet | Recovery code
| BlueWallet
| (1) media (2) suspect (3) effort (4) dish (5) album (6) shaft (7) price (8) junk (9) pizza (10) situate (11) oyster (12) rib
| Electrum
| nephew dog crane clever quantum crazy purse traffic repeat fruit old clutch
| Muun
| LAFV TZUN V27E NU4D WPF4 BRJ4 ELLP BNFL
|===
[TIP]
====
A recovery code is sometimes called a "mnemonic" or "mnemonic phrase",
which implies you should memorize the phrase, but writing the phrase
down on paper takes less work and tends to be more reliable than most
people's memories. Another alternative name is "seed phrase" because
it provides the input ("seed") to the function which generates all of
a wallet's keys.
====
If something happens to Alice's wallet, she can download a new copy of
her wallet software and enter this recovery code to rebuild the wallet
database of all the onchain transactions she's ever sent or received.
However, recovering from the recovery code will not by itself restore any additional
data Alice entered into her wallet, such as the labels she associated
with particular addresses or transactions. Although losing access to
that metadata isn't as important as losing access to money, it can
still be important in its own way. Imagine you need to review an old
bank or credit card statement and the name of every entity you paid (or
who paid you) has been blanked out. To prevent losing metadata, many
wallets provide an additional backup feature beyond recovery codes.
For some wallets, that additional backup feature is even more important
today than it used to be. Many Bitcoin payments are now made using
_offchain_ technology, where not every payment is stored in the public block
chain. This reduces users costs and improves privacy, among other
benefits, but it means that a mechanism like recovery codes that depends on
onchain data can't guarantee recovery of all of a user's bitcoins. For
applications with offchain support, it's important to make frequent
backups of the wallet database.
Of note, when receiving funds to a new mobile wallet for the first time,
many wallets will often re-verify that you have securely backed-up your
recovery code. This can range from a simple prompt to requiring the
user to manually re-enter the code.
[WARNING]
====
Although many legitimate wallets will prompt you to re-enter
your recovery code, there are also many malware applications that mimic the
design of a wallet, insist you enter your recovery code, and then
relay any entered code to the malware developer so they can steal
your funds. This is the equivalent of phishing websites that try to
trick you into giving them your bank passphrase. For most wallet
applications, the only times they will ask for your recovery code are during
the initial set up (before you have received any bitcoins) and during
recovery (after you lost access to your original wallet). If the application
asks for your recovery code any other time, consult with an expert to
ensure you aren't being phished.
====
==== Bitcoin addresses
Alice is now ready to start using her new Bitcoin wallet. Her wallet application randomly generated a private key (described in more detail in <<private_keys>>) which will be used to derive Bitcoin addresses that direct to her wallet. At this point, her Bitcoin addresses are not known to the Bitcoin network or "registered" with any part of the Bitcoin system. Her Bitcoin addresses are simply numbers that correspond to her private key that she can use to control access to the funds. The addresses are generated independently by her wallet without reference or registration with any service.
[TIP]
====
There
are a variety of Bitcoin addresses and invoice formats. Addresses and
invoices can be shared with other Bitcoin users
who can use them to send bitcoins directly to your wallet. You can share
an address or invoice with other people without worrying about the
security of your bitcoins. Unlike a bank account number, nobody who
learns one of your Bitcoin addresses can withdraw money from your wallet--you
must initiate all spends. However, if you give two people the same
address, they will be able to see how many bitcoins the other person sent
you. If you post your address publicly, everyone will be able to see
how much bitcoin other people sent to that address. To protect your privacy, you
should generate a new invoice with a new address each time you request a
payment.
====
==== Receiving bitcoin
Alice uses the _Receive_ button, which displays a QR code, shown in <<wallet_receive>>.
[[wallet_receive]]
.Alice uses the Receive screen on her mobile Bitcoin wallet, and displays her address in a QR code format
image::images/receive.png["Wallet receive screen with QR code displayed. Image derived from Bitcoin Design Guide CC-BY"]
The QR code is the square with a pattern of black and white dots, serving as a form of barcode that contains the same information in a format that can be scanned by Joe's smartphone camera.
[WARNING]
====
Any funds sent to the addresses in this book will be lost. If you want
to test sending bitcoins, please consider donating it to a
bitcoin-accepting charity.
====
[[getting_first_bitcoin]]
==== Getting Your First Bitcoin
The first task for new users is to acquire some bitcoin.
Bitcoin transactions are irreversible. Most electronic payment networks such as credit cards, debit cards, PayPal, and bank account transfers are reversible. For someone selling bitcoin, this difference introduces a very high risk that the buyer will reverse the electronic payment after they have received bitcoin, in effect defrauding the seller. To mitigate this risk, companies accepting traditional electronic payments in return for bitcoin usually require buyers to undergo identity verification and credit-worthiness checks, which may take several days or weeks. As a new user, this means you cannot buy bitcoin instantly with a credit card. With a bit of patience and creative thinking, however, you won't need to.
[role="pagebreak-before"]
Here are some methods for acquiring bitcoin as a new user:
* Find a friend who has bitcoins and buy some from him or her directly. Many Bitcoin users start this way. This method is the least complicated. One way to meet people with bitcoins is to attend a local Bitcoin meetup listed at https://meetup.com[Meetup.com].
* Earn bitcoin by selling a product or service for bitcoin. If you are a programmer, sell your programming skills. If you're a hairdresser, cut hair for bitcoins.
* Use a Bitcoin ATM in your city. A Bitcoin ATM is a machine that accepts cash and sends bitcoins to your smartphone Bitcoin wallet.
* Use a Bitcoin currency exchange linked to your bank account. Many countries now have currency exchanges that offer a market for buyers and sellers to swap bitcoins with local currency. Exchange-rate listing services, such as https://bitcoinaverage.com[BitcoinAverage], often show a list of Bitcoin exchanges for each currency.
[TIP]
====
One of the advantages of
Bitcoin over other payment systems is that, when used correctly, it
affords users much more privacy. Acquiring, holding, and spending
bitcoin does not require you to divulge sensitive and personally
identifiable information to third parties. However, where bitcoin
touches traditional systems, such as currency exchanges, national and
international regulations often apply. In order to exchange bitcoin for
your national currency, you will often be required to provide proof of
identity and banking information. Users should be aware that once a
Bitcoin address is attached to an identity, other associated Bitcoin
transactions may also become easy to identify and track--including
transactions made earlier. This is one reason
many users choose to maintain dedicated exchange accounts independent from
their wallets.
====
Alice was introduced to Bitcoin by a friend so she has an easy way to acquire her first bitcoins. Next, we will look at how she buys bitcoins from her friend Joe and how Joe sends the bitcoins to her wallet.
[[bitcoin_price]]
==== Finding the Current Price of Bitcoin
Before Alice can buy bitcoin from Joe, they have to agree on the _exchange rate_ between bitcoin and US dollars. This brings up a common question for those new to Bitcoin: "Who sets the price
of bitcoins?" The short answer is that the price is set by markets.
Bitcoin, like most other currencies, has a _floating exchange rate_. That means that the value of bitcoin fluctuates according to supply and demand in the various markets where it is traded. For example, the "price" of bitcoin in US dollars is calculated in each market based on the most recent trade of bitcoins and US dollars. As such, the price tends to fluctuate minutely several times per second. A pricing service will aggregate the prices from several markets and calculate a volume-weighted average representing the broad market exchange rate of a currency pair (e.g., BTC/USD).
There are hundreds of applications and websites that can provide the current market rate. Here are some of the most popular:
https://bitcoinaverage.com/[Bitcoin Average]:: A site that provides a simple view of the volume-weighted-average for each currency.
https://coincap.io/[CoinCap]:: A service listing the market capitalization and exchange rates of hundreds of crypto-currencies, including bitcoins.
https://www.cmegroup.com/markets/cryptocurrencies/cme-cf-cryptocurrency-benchmarks.html?redirect=/trading/cryptocurrency-indices/cf-bitcoin-reference-rate.html[Chicago Mercantile Exchange Bitcoin Reference Rate]:: A reference rate that can be used for institutional and contractual reference, provided as part of investment data feeds by the CME.
In addition to these various sites and applications, some bitcoin
wallets will automatically convert amounts between bitcoin and other
currencies.
[[sending_receiving]]
==== Sending and Receiving Bitcoin
Alice has
decided to buy 0.001 bitcoins. After she and Joe check the exchange rate,
she gives Joe an appropriate amount of cash, opens her mobile wallet
application, and selects Receive. This
displays a QR code with Alice's first Bitcoin address.
Joe then selects Send on his smartphone wallet and opens the QR code
scanner. This allows Joe to scan the barcode with his smartphone camera
so that he doesn't have to type in Alice's Bitcoin address, which is
quite long.
Joe now has Alice's Bitcoin address set as the recipient. Joe enters the
amount as 0.001 bitcoins (BTC), see <<wallet-send>>. Some wallets may
show the amount in a different denomination: 0.001 BTC is 1 millibitcoin
(mBTC) or 100,000 satoshis (sats).
Some wallets may also suggest Joe enter a label for this transaction; if
so, Joe enters "Alice". Weeks or months from now, this will help Joe
remember why he sent these 0.001 bitcoins. Some wallets may also prompt
Joe about fees. Depending on the wallet and how the transaction is
being sent, the wallet may ask Joe to either enter a transaction fee rate or
prompt him with a suggested fee (or fee rate). The higher the transaction fee, the
faster the transaction will be confirmed (see <<confirmations>>).
[[wallet-send]]
[role="smallereighty"]
.Bitcoin wallet send screen
image::images/send.png["Wallet send screen. Image derived from Bitcoin Design Guide CC-BY"]
Joe then carefully checks to make sure he has entered the correct
amount, because he is about to transmit money and mistakes will soon become
irreversible. After double-checking the address and amount, he presses
Send to transmit the transaction. Joe's mobile Bitcoin wallet constructs
a transaction that assigns 0.001 BTC to the address provided by Alice,
sourcing the funds from Joe's wallet and signing the transaction with
Joe's private keys. This tells the Bitcoin network that Joe has
authorized a transfer of value to Alice's new address. As the
transaction is transmitted via the peer-to-peer protocol, it quickly
propagates across the Bitcoin network. After just a few seconds, most of
the well-connected nodes in the network receive the transaction and see
Alice's address for the first time.
Meanwhile, Alice's wallet is constantly "listening" for new
transactions on the Bitcoin network, looking for any that match the
addresses it contains. A few seconds after Joe's wallet transmits the
transaction, Alice's wallet will indicate that it is receiving
0.001 BTC.
[[confirmations]]
.Confirmations
****
At first, Alice's address will show the transaction from Joe as "Unconfirmed." This means that the transaction has been propagated to the network but has not yet been recorded in the Bitcoin transaction journal, known as the blockchain. To be confirmed, a transaction must be included in a block and added to the blockchain, which happens every 10 minutes, on average. In traditional financial terms this is known as _clearing_. For more details on propagation, validation, and clearing (confirmation) of bitcoin transactions, see <<mining>>.
****
Alice is now the proud owner of 0.001 BTC that she can spend. Over the next few days, Alice buys more bitcoin using an ATM and an exchange. In the next chapter we will look at her first purchase with Bitcoin, and examine the underlying transaction and propagation technologies in more detail.

636
ch02_overview.adoc Normal file
View File

@ -0,0 +1,636 @@
[[ch02_bitcoin_overview]]
== How Bitcoin Works
The Bitcoin system, unlike traditional banking and
payment systems, does not require trust in third parties. Instead of a central
trusted authority, in Bitcoin, each user can use software running on
their own computer to verify the correct operation of every
aspect of the Bitcoin system.
In this chapter, we will examine Bitcoin from a high level by tracking a
single transaction through the Bitcoin system and watch as it
is recorded on the blockchain, the distributed journal of all
transactions. Subsequent chapters will delve into the technology behind
transactions, the network, and mining.
=== Bitcoin Overview
In the overview diagram shown in <<bitcoin-overview>>, we see that the
Bitcoin system consists of users with wallets containing keys,
transactions that are propagated across the network, and miners who
produce (through competitive computation) the consensus blockchain,
which is the authoritative journal of all transactions.
Each example in this chapter is based
on an actual transaction made on the Bitcoin network, simulating the
interactions between several users by sending
funds from one wallet to another. While tracking a transaction through
the Bitcoin network to the blockchain, we will use a _blockchain
explorer_ site to visualize each step. A blockchain explorer is a web
application that operates as a bitcoin search engine, in that it allows
you to search for addresses, transactions, and blocks and see the
relationships and flows between them.
[[bitcoin-overview]]
.Bitcoin overview
image::images/mbc2_0201.png["Bitcoin Overview"]
Popular blockchain explorers include:
* https://blockstream.info/[Blockstream Explorer]
* https://mempool.space[Mempool.Space]
* https://live.blockcypher.com[BlockCypher Explorer]
Each of these has a search function that can take a Bitcoin address,
transaction hash, block number, or block hash and retrieve corresponding
information from the Bitcoin network. With each transaction or block
example, we will provide a URL so you can look it up yourself and study
it in detail.
[[block-explorer-privacy]]
.Block explorer privacy warning
[WARNING]
====
Searching information on a block explorer may disclose to its operator
that you're interested in that information, allowing them to associate
it with your IP address, browser details, past searches, or other
identifiable information. If you look up the transactions in this book,
the operator of the block explorer might guess that you're learning
about Bitcoin, which shouldn't be a problem. But if you look up your
own transactions, the operator may be able to guess how many bitcoins
you've received, spent, and currently own.
====
[[spending_bitcoin]]
==== Buying from an Online Store
Alice, introduced in the previous chapter, is a new user who has just
acquired her first bitcoins. In <<getting_first_bitcoin>>, Alice met with
her friend Joe to exchange some cash for bitcoins. Since then, Alice has
bought additional bitcoins. Now Alice will make
her first spending transaction, buying access to a premium podcast episode from Bob's online store.
Bob's web store recently started accepting bitcoin payments by adding a
Bitcoin option to its website. The prices at Bob's store are listed in
the local currency (US dollars), but at checkout, customers have the
option of paying in either dollars or bitcoin.
Alice finds the podcast episode she wants to buy and proceeds to the checkout page. At checkout,
Alice is offered the option to pay with bitcoin, in addition to the
usual options. The checkout cart displays the price in US dollars and
also in bitcoin (BTC), at Bitcoin's prevailing exchange rate.
Bob's
e-commerce system will automatically create a QR code containing an
_invoice_ (<<invoice-QR>>).
Unlike a QR code that simply contains a destination Bitcoin address, this
invoice is a QR-encoded URI that contains a destination address,
a payment amount, and a description.
This allows a Bitcoin wallet application to prefill the
information used to send the payment while showing a human-readable
description to the user. You can scan the QR code with a bitcoin wallet
application to see what Alice would see.
////
TODO: Replace QR code with test-BTC address
////
[[invoice-QR]]
.Invoice QR code
image::images/mbc2_0202.png["payment-request"]
[TIP]
====
Try to scan this with your wallet to see
the address and amount but DO NOT SEND MONEY.
====
[[invoice-URI]]
.The invoice QR code encodes the following URI, defined in BIP21:
----
bitcoin:bc1qk2g6u8p4qm2s2lh3gts5cpt2mrv5skcuu7u3e4?amount=0.01577764&
label=Bob%27s%20Store&
message=Purchase%20at%20Bob%27s%20Store
Components of the URI
A Bitcoin address: "bc1qk2g6u8p4qm2s2lh3gts5cpt2mrv5skcuu7u3e4"
The payment amount: "0.01577764"
A label for the recipient address: "Bob's Store"
A description for the payment: "Purchase at Bob's Store"
----
Alice uses her smartphone to scan the barcode on display. Her smartphone
shows a payment for the correct amount to +Bob's Store+ and she selects Send to
authorize the payment. Within a few seconds (about the same amount of
time as a credit card authorization), Bob sees the transaction on the
register.
[NOTE]
====
The
Bitcoin network can transact in fractional values, e.g., from
millibitcoin (1/1000th of a bitcoin) down to 1/100,000,000th of a
bitcoin, which is known as a satoshi. This book uses the same
pluralization rules used for dollars and other traditional currencies
when talking about amounts greater than one bitcoin and when using
decimal notation, such as "10 bitcoins" or "0.001 bitcoins." The same
rules also apply to other bitcoin bookkeeping units, such as
millibitcoins and satoshis.
====
You can examine Alice's transaction to Bob's Store on the blockchain
using a block explorer site (<<view_alice_transaction>>):
[[view_alice_transaction]]
.View Alice's transaction on https://blockstream.info/tx/674616f1fbc6cc748213648754724eebff0fc04506f2c81efb1349d1ebc8a2ef[Blockstream Explorer]
====
----
https://blockstream.info/tx/674616f1fbc6cc748213648754724eebff0fc04506f2c81efb1349d1ebc8a2ef
----
====
In the following sections, we will examine this transaction in more
detail. We'll see how Alice's wallet constructed it, how it was
propagated across the network, how it was verified, and finally, how Bob
can spend that amount in subsequent transactions.
=== Bitcoin Transactions
In simple terms, a transaction tells the
network that the owner of certain bitcoins has authorized the transfer
of that value to another owner. The new owner can now spend the bitcoin
by creating another transaction that authorizes the transfer to another
owner, and so on, in a chain of ownership.
==== Transaction Inputs and Outputs
Transactions are like lines in a double-entry
bookkeeping ledger. Each transaction contains one or more _inputs_,
which spend funds. On the other side of
the transaction, there are one or more _outputs_, which receive funds.
The inputs
and outputs do not necessarily add up to the same
amount. Instead, outputs add up to slightly less than inputs and the
difference represents an implied _transaction fee_, which is a small
payment collected by the miner who includes the transaction in the
blockchain. A Bitcoin transaction is shown as a bookkeeping ledger entry in
<<transaction-double-entry>>.
The transaction also contains proof of ownership for each amount of
bitcoins (inputs) whose value is being spent, in the form of a digital
signature from the owner, which can be independently validated by
anyone. In Bitcoin terms, spending
is signing a transaction that transfers value from a previous
transaction over to a new owner identified by a Bitcoin address.
[[transaction-double-entry]]
.Transaction as double-entry bookkeeping
image::images/mbc2_0203.png["Transaction Double-Entry"]
==== Transaction Chains
Alice's payment to Bob's Store uses a
previous transaction's output as its input. In the previous chapter,
Alice received bitcoins from her friend Joe in return for cash.
We've labeled that as _Transaction 1_ (Tx1) in <<transaction-chain>>.
Tx1 sent 0.001 bitcoins (100,000 satoshis) to an output locked by
Alice's key. Her new transaction to Bob's Store (Tx2) references the
previous output as an input. In the illustration, we show that
reference using an arrow and by labeling the input as "Tx1:0". In an
actual transaction, the reference is the 32-byte transaction identifier
(txid) for the transaction where Alice received the money from Joe. The
":0" indicates the position of the output where Alice received the
money; in this case, the first position (position 0).
As shown, actual Bitcoin transactions don't
explicitly include the value of their input. To determine the value of
an input, software needs to use the input's reference to find the
previous transaction output being spent.
Alice's Tx2 contains two new outputs, one paying 75,000 satoshis for the
podcast and another paying 20,000 satoshis back to Alice to receive
change.
////
@startditaa
Transaction 1 Tx2 Tx3
Inputs Outputs In Out In Out
+-------+---------+ +-------+--------+ +-------+--------+
| | | | | cDDD | | | |
<--+ Tx0꞉0 | 100,000 |<--+ Tx1꞉0 | 20,000 | +-+ Tx2꞉1 | 67,000 |
| | | | | | | | | |
+-------+---------+ +-------+--------+ | +-------+--------+
| | cDDD | | | | | | | |
| | 500,000 | | | 75,000 |<-+ | | |
| | | | | | | | |
+-------+---------+ +-------+--------+ +-------+--------+
Fee꞉ (unknown) Fee꞉ 5,000 Fee꞉ 8,000
@enddittaa
////
[[transaction-chain]]
.A chain of transactions, where the output of one transaction is the input of the next transaction
image::images/transaction-chain.png["Transaction chain"]
[TIP]
====
Serialized Bitcoin transactions--the data format that software uses for
sending transactions--encodes the value to transfer using an integer
of the smallest defined onchain unit of value. When Bitcoin was first
created, this unit didn't have a name and some developers simply called
it the _base unit._ Later many users began calling this unit a
_satoshi_ (sat) in honor of Bitcoin's creator. In <<transaction-chain>>
and some other illustrations in this book, we use satoshi values because
that's what the protocol itself uses.
====
==== Making Change
In addition to one or more outputs that pay the receiver of
bitcoins, many transactions will also include an output that pays the
spender of the bitcoins, called a _change_ output.
This is because transaction inputs,
like currency notes, cannot be partly spent. If you purchase a $5 US dollar
item in a store but use a $20 dollar bill to pay for the item, you
expect to receive $15 dollars in change. The same concept applies to
Bitcoin transaction inputs. If you purchased an item that costs 5
bitcoins but only had an input worth 20 bitcoins to use, you would send one
output of 5 bitcoins to the store owner and one output of 15 bitcoins back
to yourself as change (not counting your transaction fee).
At the level of the Bitcoin protocol, there is no difference between a
change output (and the address it pays, called a _change address_) and a
payment output.
Importantly, the change address does not have to be the
same address as that of the input and for privacy reasons is often a new
address from the owner's wallet. In ideal circumstances, the two
different uses of outputs both use never-before-been addresses and
otherwise look identical, preventing any third party from determining
which outputs are change and which are payments. However, for
illustration purposes, we've added shading to the change outputs in
<<transaction-chain>>.
Not every transaction has a change output. Those that don't are called
_changeless transactions_ and they can have only a single output.
Changeless transaction are only a practical option if the amount being
spent is roughly the same as the amount available in the transaction
inputs minus the anticipated transaction fee. In <<transaction-chain>>
we see Bob creating Tx3 as a changeless transaction that spends the
output he received in Tx2.
==== Coin selection
Different wallets use different strategies when choosing which
inputs to use to a payment, called _coin selection_.
They might aggregate many small
inputs, or use one that is equal to or larger than the desired payment.
Unless the wallet can aggregate inputs in such a way to exactly match
the desired payment plus transaction fees, the wallet will need to
generate some change. This is very similar to how people handle cash. If
you always use the largest bill in your pocket, you will end up with a
pocket full of loose change. If you only use the loose change, you'll
often have only big bills. People subconsciously find a balance between
these two extremes, and Bitcoin wallet developers strive to program this
balance.
==== Common Transaction Forms
A very common form of transaction is a simple payment. This type of
transaction has one input and two outputs and is shown in
<<transaction-common>>.
[[transaction-common]]
.Most common transaction
image::images/mbc2_0205.png["Common Transaction"]
Another common form of transaction is a _consolidation transaction_ that spends several inputs
into a single output (<<transaction-consolidating>>). This represents
the real-world equivalent of exchanging a pile of coins and currency
notes for a single larger note. Transactions like these are sometimes
generated by wallets and businesses to clean up lots of smaller amounts.
[[transaction-consolidating]]
.Consolidation transaction aggregating funds
image::images/mbc2_0206.png["Aggregating Transaction"]
Finally, another transaction form that is seen often on the
blockchain is _payment batching_ that pays to multiple outputs
representing multiple recipients (<<transaction-distributing>>).
This type of transaction is sometimes used by commercial entities to
distribute funds, such as when processing payroll payments to multiple
employees.
[[transaction-distributing]]
.Batch transaction distributing funds
image::images/mbc2_0207.png["Distributing Transaction"]
=== Constructing a Transaction
Alice's wallet application contains all
the logic for selecting inputs and generating outputs to build a
transaction to Alice's specification. Alice only needs to choose a
destination, amount, and transaction fee, and the rest happens in the wallet
application without her seeing the details. Importantly, if a wallet
already knows what inputs it controls, it can construct transactions
even if it is completely offline.
Like writing a check at home and later sending it to the bank in an
envelope, the transaction does not need to be constructed and signed
while connected to the Bitcoin network.
==== Getting the Right Inputs
Alice's wallet
application will first have to find inputs that can pay the amount she
wants to send to Bob. Most wallets keep track of all the available
outputs belonging to addresses in the wallet. Therefore, Alice's wallet
would contain a copy of the transaction output from Joe's transaction,
which was created in exchange for cash (see <<getting_first_bitcoin>>).
A bitcoin wallet application that runs on a full node actually
contains a copy of every confirmed transaction's unspent outputs, called
_Unspent Transaction Outputs_ (UTXOs).
However, because full nodes use more resources, many
user wallets run lightweight clients that track only the user's own
UTXOs.
In this case, this single
UTXO is sufficient to pay for the podcast. Had this not been the case,
Alice's wallet application might have to combine several
smaller UTXOs, like picking coins from a purse until it could
find enough to pay for the podcast. In both cases, there might be a need
to get some change back, which we will see in the next section, as the
wallet application creates the transaction outputs (payments).
==== Creating the Outputs
A transaction output is
created with a
script that says something like, "This output is paid to whoever can
present a signature from the key corresponding to Bob's public address."
Because only Bob has the wallet with the keys corresponding to that
address, only Bob's wallet can present such a signature to later spend this
output. Alice will therefore _encumber_ the output value with a demand
for a signature from Bob.
This transaction will also include a second output, because Alice's
funds contain more money than the cost of the
podcast. Alice's change
output is created in the very same
transaction as the payment to Bob. Essentially, Alice's wallet breaks
her funds into two outputs: one to Bob and one back to herself. She can
then spend the change output in a subsequent transaction.
Finally, for the transaction to be processed by the network in a timely
fashion, Alice's wallet application will add a small fee. The fee is not
explicitly stated in the transaction; it is implied by the difference in value between
inputs and outputs. This _transaction fee_ is collected by the
miner as a fee for including the transaction in a block
that gets recorded on the blockchain.
[[transaction-alice-url]]
[TIP]
====
View the https://blockstream.info/tx/466200308696215bbc949d5141a49a4138ecdfdfaa2a8029c1f9bcecd1f96177[transaction from Alice to Bob's Store].
====
==== Adding the Transaction to the Blockchain
The transaction created by Alice's wallet application
contains everything necessary to confirm ownership of the funds and
assign new owners. Now, the transaction must be transmitted to the
Bitcoin network where it will become part of the blockchain. In the next
section we will see how a transaction becomes part of a new block and
how the block is mined. Finally, we will see how the new block, once
added to the blockchain, is increasingly trusted by the network as more
blocks are added.
===== Transmitting the transaction
Because the transaction contains all
the information necessary for it to be processed, it does not matter how or where it
is transmitted to the Bitcoin network. The Bitcoin network is a
peer-to-peer network, with each Bitcoin peer participating by
connecting to several other Bitcoin peers. The purpose of the Bitcoin
network is to propagate transactions and blocks to all participants.
===== How it propagates
Peers in the Bitcoin peer-to-peer network are programs that have both
the software logic and the data necessary for them to fully verify the
correctness of a new transaction. The connections between peers are
often visualized as edges (lines) in a graph, with the peers themselves
being the nodes (dots). For that reason, Bitcoin peers are commonly
called "full verification nodes", or _full nodes_ for short.
Alice's wallet application can send the new
transaction to any Bitcoin node over any type of
connection: wired, WiFi, mobile, etc. It can also send the transaction
to another program (such as a block explorer) that will relay it to a
node. Her Bitcoin wallet does not have
to be connected to Bob's Bitcoin wallet directly and she does not have
to use the internet connection offered by Bob, though both those
options are possible, too. Any Bitcoin node that receives a
valid transaction it has not seen before will forward it to
all other nodes to which it is connected, a propagation technique known
as _gossiping_. Thus, the transaction rapidly propagates out across the
peer-to-peer network, reaching a large percentage of the nodes within a
few seconds.
===== Bob's view
If Bob's Bitcoin wallet application is directly connected to Alice's
wallet application, Bob's wallet application might be the first to
receive the transaction. However, even if Alice's wallet sends the
transaction through other nodes, it will reach Bob's wallet within a few
seconds. Bob's wallet will immediately identify Alice's transaction as
an incoming payment because it contains an output redeemable by Bob's
keys. Bob's wallet application can also independently verify that the
transaction is well formed. If Bob is using his own full node, his
wallet can further verify Alice's transaction only spends valid UTXOs.
=== Bitcoin Mining
Alice's transaction is now propagated on the Bitcoin
network. It does not become part of the _blockchain_ until it is
included in a block by a process called _mining_ and that block has been
validated by full nodes. See
<<mining>> for a detailed explanation.
Bitcoin's system of counterfeit protection is based on computation.
Transactions are bundled into _blocks_. Blocks have a very small header
that must be formed in a very specific way, requiring an enormous
amount of computation to get right--but only a small amount of
computation to verify as correct.
The mining process serves two purposes in Bitcoin:
* Miners can only
receive honest income from creating blocks that follow all of Bitcoin's
_consensus rules_. Therefore, miners are normally incentivized to
only include valid transactions in their blocks and the blocks they
build upon. This allows users to optionally make a trust-based
assumption that any transaction in a block is a valid transaction.
* Mining currently creates new bitcoins in each block, almost like a central bank
printing new money. The amount of bitcoin created per block is limited
and diminishes with time, following a fixed issuance schedule.
Mining achieves a fine balance between cost and reward. Mining uses
electricity to solve a computational problem. A successful miner will
collect a _reward_ in the form of new bitcoins and transaction fees.
However, the reward will only be collected if the miner has only
included valid transactions, with the Bitcoin Protocol's rules for
_consensus_ dermining what is valid. This delicate balance provides
security for Bitcoin without a central authority.
Mining is designed to be a decentralized lottery. Each miner can create
their own lottery ticket by creating a _candidate block_ that includes
the new transactions they want to mine plus some additional data fields.
The miner inputs their candidate into a specially-designed algorithm that
scrambles (or "hashes") the data, producing output that looks nothing
like the input data. This _hash_ function will always produce the same
output for the same input--but nobody can predict what the output will
look like for a new input, even if it is only slighly different from a
previous input. If the output of hash function matches a template
determined by the Bitcoin protocol, the miner wins the lottery and
Bitcoin users will accept the block with its transactions as a
valid block. If the output doesn't match the template, the miner makes
a small change to their candidate block and tries again. As of this
writing, the number of candidate blocks miners need to try before finding
a winning combination is about 168 billion trillions. That's also how
many times the hash function needs to be run.
However, once a winning combination has been found, anyone can verify
the block is valid by running the hash function just once. That makes a
valid block something that requires an incredible amount of work to
create but only a trivial amount of work to verify. The simple
verification process is able to probabalistically prove the work was
done, so the data necessary to generate that proof--in this case, the
block--is called Proof-of-Work (PoW).
Transactions are added to the new block, prioritized by the highest fee rate
transactions first and a few other criteria. Each miner starts the
process of mining a new candidate block of transactions as soon as they receive the
previous block from the network, knowing that some other miner won that
iteration of the lottery. They immediately create a new candidate block
with a commitment to the previous block, fill it with transactions, and start
calculating the Proof-of-Work for the candidate block. Each miner includes a
special transaction in their candidate blocks, one that pays their own Bitcoin address
the block reward plus the sum of
transaction fees from all the transactions included in the candidate block. If they
finds a solution that makes the candidate into a valid block, they receives this reward
after his successful block is added to the global blockchain and the
reward transaction he included becomes spendable. Miners who participates in a mining pool have set up their
software to create candidate blocks that assign the reward to a pool address.
From there, a share of the reward is distributed to members of the pool
miners in proportion to the amount of work they contributed.
Alice's
transaction was picked up by the network and included in the pool of
unverified transactions. Once validated by a full node, it was
included in a candidate block.
Approximately five minutes after the transaction was first transmitted
by Alice's wallet, a miner finds a solution for the
block and announces it to the network. After each other miner
validates the winning block, they start a new lottery to generate the next
block.
The winning block containing Alice's transaction became part of the
blockchain. The block containing Alice's transaction is counted as one
_confirmation_ of that transaction. After the block containing Alice's
transaction has propagated through the network, creating an alternative
block with a different version of Alice's transaction (such as a
transaction that doesn't pay Bob) would require performing the same
amount of work as it will take all Bitcoin miners to create an entirely
new block. When there are multiple alternative blocks to choose from,
Bitcoin full nodes choose the chain of valid blocks with the most total
Proof-of-Work, called the _best blockchain_. For the entire network to
accept an alternative block, an additional new block would need to be
mined on top of the alternative.
That means miners have a choice. They can work with Alice on an
alternative to the transaction where she pays Bob, perhaps with
Alice paying miners a share of the money she previously paid Bob. This
dishonest behavior will require they expend the effort required to
create two new blocks. Instead, miners who behave honestly can create a
single new block and and receive all of the fees from the transactions
they include in it, plus the block subsidy. Normally, the high cost of
dishonestly creating two blocks for a small additional payment is much
less profitable than honestly creating a new block, making it unlikely
that a confirmed transaction will be deliberately changed. For Bob, this
means that he can begin to believe that the payment from Alice can be
relied upon.
[TIP]
====
You can see the block that includes
https://blockstream.info/block/000000000000000000027d39da52dd790d98f85895b02e764611cb7acf552e90[Alice's transaction].
====
Approximately 19 minutes
after the block containing Alice's transaction is broadcast, a new block
is mined by another miner. Because this
new block is built on top of the block that contained Alice's
transaction (giving Alice's transaction two confirmations) Alice's
transaction can now only be changed if two alternative blocks are
mined--plus a new block built on top of them--for a total of three
blocks that would need to be mined for Alice to take back the money she
sent Bob. Each block mined on top of the one containing Alice's
transaction counts as an additional confirmation. As the blocks pile on
top of each other, it becomes harder to reverse the transaction, thereby
giving Bob more and more confidence that Alice's payment is secure.
In <<block-alice1>>, we can
the block which contains Alice's transaction. Below it are
hundreds of thousands of blocks, linked to each other in a chain of
blocks (blockchain) all the way back to block #0, known as the _genesis
block_. Over time, as the "height" of new blocks increases, so does the
computation difficulty for the chain as a whole.
By convention, any block with more than six confirmations
is considered very hard to change, because it would require an immense amount of
computation to recalculate six blocks (plus one new block). We will examine
the process of mining and the way it builds confidence in more detail in
<<mining>>.
[[block-alice1]]
.Alice's transaction included in a block
image::images/mbc2_0209.png["Alice's transaction included in a block"]
=== Spending the Transaction
Now that Alice's
transaction has been embedded in the blockchain as part of a block, it
is visible to all Bitcoin
applications. Each Bitcoin full node can independently verify the
transaction as valid and spendable. Full nodes validate every transfer
of the funds from the moment the bitcoins were first generated in
a block through each subsequent transaction until they reach
Bob's address. Lightweight clients can partially verify payments
by confirming that the
transaction is in the blockchain and has several blocks mined after it,
thus providing assurance that the miners expended significant effort
committing to it (see <<spv_nodes>>).
Bob can now spend the output from this and other transactions. For
example, Bob can pay a contractor or supplier by transferring value from
Alice's podcast payment to these new owners.
As Bob spends the payments received from Alice and other customers, he
extends the chain of transactions. Let's assume that Bob pays his web
designer Gopesh
for a new website page. Now the chain of transactions will
look like <<block-alice2>>.
[[block-alice2]]
.Alice's transaction as part of a transaction chain from Joe to Gopesh
image::images/mbc2_0210.png["Alice's transaction as part of a transaction chain"]
In this chapter, we saw how transactions build a chain that moves value
from owner to owner. We also tracked Alice's transaction, from the
moment it was created in her wallet, through the Bitcoin network and to
the miners who recorded it on the blockchain. In the rest of this book,
we will examine the specific technologies behind wallets, addresses,
signatures, transactions, the network, and finally mining.

1160
ch03_bitcoin-core.adoc Normal file
View File

File diff suppressed because it is too large Load Diff

1676
ch04_keys.adoc Normal file
View File

File diff suppressed because it is too large Load Diff

1367
ch05_wallets.adoc Normal file
View File

File diff suppressed because it is too large Load Diff

1179
ch06_transactions.adoc Normal file
View File

File diff suppressed because it is too large Load Diff

1978
ch07_authorization-authentication.adoc Normal file
View File

File diff suppressed because it is too large Load Diff

974
ch08_signatures.adoc Normal file
View File

@ -0,0 +1,974 @@
[[c_signatures]]
== Digital Signatures
Two signature algorithms are currently
used in Bitcoin, the _schnorr signature algorithm_ and the _Elliptic
Curve Digital Signature Algorithm_ (_ECDSA_).
These algorithms are used for digital signatures based on elliptic
curve private/public key pairs, as described in <<elliptic_curve>>.
They are used for spending segwit v0 P2WPKH outputs, segwit v1 P2TR
keypath spending, and by the script functions +OP_CHECKSIG+,
+OP_CHECKSIGVERIFY+, +OP_CHECKMULTISIG+, +OP_CHECKMULTISIGVERIFY+, and
+OP_CHECKSIGADD+.
Any time one of those is executed, a signature must be
provided.
A digital signature serves
three purposes in Bitcoin. First, the
signature proves that the controller of a private key, who is by
implication the owner of the funds, has _authorized_ the spending of
those funds. Secondly, the proof of authorization is _undeniable_
(nonrepudiation). Thirdly, that the authorized transaction cannot be
changed by unauthenticated third parties--that its _integrity_ is
intact.
[NOTE]
====
Each transaction input and any signatures it may contain is _completely_
independent of any other input or signature. Multiple parties can
collaborate to construct transactions and sign only one input each.
Several protocols uses this fact to create multi-party transactions for
privacy.
====
In this chapter we look at how digital signatures work and how they can
present proof of control of a private key without revealing that private
key.
=== How Digital Signatures Work
A digital signature
consists of two parts. The first part is an algorithm for creating a
signature for a message (the transaction) using a private key (the
signing key). The second part is an algorithm
that allows anyone to verify the signature, given also the message and the corresponding
public key.
==== Creating a digital signature
In Bitcoin's use of digital signature algorithms, the "message" being
signed is the transaction, or more accurately a hash of a specific
subset of the data in the transaction, called the _commitment hash_ (see
<<sighash_types>>). The
signing key is the user's private key. The result is the signature:
latexmath:[\(Sig = F_{sig}(F_{hash}(m), x)\)]
where:
* _x_ is the signing private key
* _m_ is the message to sign, the commitment hash (such as parts of a transaction)
* _F_~_hash_~ is the hashing function
* _F_~_sig_~ is the signing algorithm
* _Sig_ is the resulting signature
More details on the mathematics of schnorr and ECDSA signatures can be found in <<schnorr_signatures>>
and <<ecdsa_signatures>>.
In both schnorr and ECDSA signatures, the function _F_~_sig_~ produces a signature +Sig+ that is composed of
two values. There are differences between the two values in the
different algorithms, which we'll explore later.
After the two values
are calculated, they are serialized into a byte-stream. For ECDSA
signatures, the encoding uses an international standard encoding scheme
called the
_Distinguished Encoding Rules_, or _DER_. For schnorr signatures, a
simpler serialization format is used.
==== Verifying the Signature
The signature verification
algorithm takes the message (a hash of parts of the transaction and
related data), the signer's public key and the signature, and returns
TRUE if the signature is valid for this message and public key.
To verify the signature, one must have the signature, the serialized
transaction, some data about the output being spent, and the public key
that corresponds to the private key used to create the signature.
Essentially, verification of a signature means "Only the controller of
the private key that generated this public key could have produced this
signature on this transaction."
[[sighash_types]]
==== Signature Hash Types (SIGHASH)
Digital signatures apply to messages,
which in the case of Bitcoin, are the transactions themselves. The
signature prove a _commitment_ by the signer to specific transaction
data. In the simplest form, the signature applies to almost the entire
transaction, thereby committing to all the inputs, outputs, and other
transaction fields. However, a signature can commit to only a subset of
the data in a transaction, which is useful for a number of scenarios as
we will see in this section.
Bitcoin signatures have a way of indicating which
part of a transaction's data is included in the hash signed by the
private key using a +SIGHASH+ flag. The +SIGHASH+ flag is a single byte
that is appended to the signature. Every signature has either an
explicit or implicit +SIGHASH+ flag
and the flag can be different from input to input. A transaction with
three signed inputs may have three signatures with different +SIGHASH+
flags, each signature signing (committing) to different parts of the
transaction.
Remember, each input may contain one or more signatures. As
a result, an input may have signatures
with different +SIGHASH+ flags that commit to different parts of the
transaction. Note also that Bitcoin transactions
may contain inputs from different "owners," who may sign only one input
in a partially constructed transaction, collaborating with
others to gather all the necessary signatures to make a valid
transaction. Many of the +SIGHASH+ flag types only make sense if you
think of multiple participants collaborating outside the Bitcoin network
and updating a partially signed transaction.
[role="pagebreak-before"]
There are three +SIGHASH+ flags: +ALL+, +NONE+, and +SINGLE+, as shown
in <<sighash_types_and_their>>.
[[sighash_types_and_their]]
.SIGHASH types and their meanings
[options="header"]
|=======================
|+SIGHASH+ flag| Value | Description
| +ALL+ | 0x01 | Signature applies to all inputs and outputs
| +NONE+ | 0x02 | Signature applies to all inputs, none of the outputs
| +SINGLE+ | 0x03 | Signature applies to all inputs but only the one output with the same index number as the signed input
|=======================
In addition, there is a modifier flag +SIGHASH_ANYONECANPAY+, which can
be combined with each of the preceding flags. When +ANYONECANPAY+ is
set, only one input is signed, leaving the rest (and their sequence
numbers) open for modification. The +ANYONECANPAY+ has the value +0x80+
and is applied by bitwise OR, resulting in the combined flags as shown
in <<sighash_types_with_modifiers>>.
[[sighash_types_with_modifiers]]
.SIGHASH types with modifiers and their meanings
[options="header"]
|=======================
|SIGHASH flag| Value | Description
| ALL\|ANYONECANPAY | 0x81 | Signature applies to one input and all outputs
| NONE\|ANYONECANPAY | 0x82 | Signature applies to one input, none of the outputs
| SINGLE\|ANYONECANPAY | 0x83 | Signature applies to one input and the output with the same index number
|=======================
The way +SIGHASH+ flags are applied during signing and verification is
that a copy of the transaction is made and certain fields within are
either omitted or truncated (set to zero length and emptied). The resulting transaction is
serialized. The +SIGHASH+ flag is included in the serialized
transaction data and the result is hashed. The hash digest itself is the "message"
that is signed. Depending on which +SIGHASH+ flag is used, different
parts of the transaction are included.
By including the
+SIGHASH+ flag itself, the signature commits the
+SIGHASH+ type as well, so it can't be changed (e.g., by a miner).
In
<<serialization_of_signatures_der>>, we will see that the last part of the
DER-encoded signature was +01+, which is the +SIGHASH_ALL+ flag for ECDSA signatures. This
locks the transaction data, so Alice's signature is committing to the state
of all inputs and outputs. This is the most common signature form.
Let's look at some of the other +SIGHASH+ types and how they can be used
in practice:
+ALL|ANYONECANPAY+ :: This construction can be used to make a
"crowdfunding&#x201d;-style transaction. Someone attempting to raise
funds can construct a transaction with a single output. The single
output pays the "goal" amount to the fundraiser. Such a transaction is
obviously not valid, as it has no inputs. However, others can now amend
it by adding an input of their own, as a donation. They sign their own
input with +ALL|ANYONECANPAY+. Unless enough inputs are gathered to
reach the value of the output, the transaction is invalid. Each donation
is a "pledge," which cannot be collected by the fundraiser until the
entire goal amount is raised. Unfortunately, this protocol can be
circumvented by the fundraiser adding an input of their own (or from
someone who lends them funds), allowing them to collect the donations
even if they haven't reached the specified value.
+NONE+ :: This construction can be used to create a "bearer check" or
"blank check" of a specific amount. It commits to all inputs, but allows
the outputs to be changed. Anyone can write their own
Bitcoin address into the output script.
By itself, this allows any miner to change
the output destination and claim the funds for themselves, but if other
required signatures in the transaction use +SIGHASH_ALL+ or another type
that commits to the output, it allows those spenders to change the
destination without allowing any third-parties (like miners) to modify
the outputs.
+NONE|ANYONECANPAY+ :: This construction can be used to build a "dust
collector." Users who have tiny UTXOs in their wallets can't spend these
without the cost in fees exceeding the value of the UTXO, see
<<uneconomical_outputs>>. With this type
of signature, the uneconomical UTXOs can be donated for anyone to aggregate and
spend whenever they want.
There are some proposals to modify or
expand the +SIGHASH+ system. The most widely discussed proposal as of
this writing is BIP118, which proposes to add two
new sighash flags. A signature using +SIGHASH_ANYPREVOUT+ would not
commit to an input's outpoint field, allowing it to be used to spend any
previous output for a particular witness program. For example, if Alice
receives two outputs for the same amount to the same witness program
(e.g. requiring a single signature from her wallet), a
+SIGHASH_ANYPREVOUT+ signature for spending either one of those outputs
could be copied and used to spend the other output to the same
destination.
A signature using +SIGHASH_ANYPREVOUTANYSCRIPT+ would not
commit to the outpoint, the amount, the witness program, or the
specific leaf in the taproot merkle tree (script tree), allowing it to spend any previous output
which the signature could satisfy. For example, if Alice received two
outputs for different amounts and different witness programs (e.g. one
requiring a single signature and another requiring her signature plus some
other data), a +SIGHASH_ANYPREVOUTANYSCRIPT+ signature for spending
either one of those outputs could be copied and used to spend the other
output to the same destination (assuming the extra data for the second
output was known).
The main expected use for the two SIGHASH_ANYPREVOUT opcodes is improved
payment channels, such as those used in the Lightning Network, although
several other uses have been described.
[NOTE]
====
You will not often see +SIGHASH+ flags presented as an option in a user's
wallet application. Simple wallet applications
sign with +SIGHASH_ALL+ flags. More sophisticated applications, such as
Lightning Network nodes, may use alternative +SIGHASH+ flags, but they
use protocols that have been extensively reviewed to understand the
influence of the alternative flags.
====
[[schnorr_signatures]]
=== Schnorr signatures
In 1989, Claus Schnorr published a paper describing the signature
algorithm that's become eponymous with him. The algorithm isn't
specific to the Elliptic Curve Cryptography (ECC) that Bitcoin and many
other applications use, although it is perhaps most strongly associated
with ECC today. Schnorr signatures have a number of nice properties:
Provable security::
A mathematical proof of the security of schnorr signatures depends on
only the difficulty of solving the Discrete Logarithm Problem (DLP),
particularly for Elliptic Curves (EC) for Bitcoin, and the ability of
a hash function (like the SHA256 function used in Bitcoin) to produce
unpredictable values, called the Random Oracle Model (ROM). Other
signature algorithms have additional dependencies or require much
larger public keys or signatures for equivalent security to
ECC-Schnorr (when the threat is defined as classical computers; other
algorithms may provide more efficient security against quantum
computers).
Linearity::
Schnorr signatures have a property that mathematicians call
_linearity_, which applies to functions with two particular
properties. The first property is that summing together two or more
variables and then running a function on that sum will produce the
same value as running the function on each of the variables
independently and then summing together the results, e.g.
+f(x + y + z) == f(x) + f(y) + f(z)+; this property is called
_additivity_. The second property is that multiplying a variable and
then running a function on that product will produce the same value as
running the function on the variable and then multiplying it by the
same amount, e.g. +f(a * x) == a * f(x)+; this property is called
_homogeneity of degree 1_.
+
In cryptographic operations, some functions may be private (such
as functions involving private keys or secret nonces), so being able
to get the same result whether performing an operation inside or
outside of a function makes it easy for multiple parties to coordinate
and cooperate without sharing their secrets. We'll see some of the
specific benefits of linearity in schnorr signatures in
<<schnorr_multisignatures>> and <<schnorr_threshold_signatures>>.
Batch Verification::
When used in a certain way (which Bitcoin does), one consequence of
schnorr's linearity is that it's relatively straightforward to verify
more than one schnorr signature at the same time in less time than it
would take to verify each signature independently. The more
signatures that are verified in a batch, the greater the speed up.
For the typical number of signatures in a block, it's possible to
batch verify them in about half the amount of time it would take to
verify each signature independently.
Later in this chapter, we'll describe the schnorr signature algorithm
exactly as it's used in Bitcoin, but we're going to start with a
simplified version of it and work our way towards the actual protocol in
stages.
Alice starts by choosing a large random number (+x+), which we call her
_private key_. She also knows a public point on Bitcoin's Elliptic
Curve (EC) called the Generator (+G+) (see <<public_key_derivation>>). Alice uses EC
multiplication to multiply +G+ by her private key +x+, in which case +x+
is called a _scalar_ because it scales up +G+. The result is +xG+,
which we call Alice's _public key_. Alice gives her public key to Bob.
Even though Bob also knows +G+, the Discrete Logarithm Problem (DLP)
prevents Bob from being able to divide +xG+ by +G+ to derive Alice's
private key.
At some later time, Bob wants Alice to identify herself by proving
that she knows the scalar +x+ for the public key (+xG+) that Bob
received earlier. Alice can't give Bob +x+ directly because that would
allow him to identify as her to other people, so she needs to prove
her knowledge of +x+ without revealing +x+ to Bob, called a
_zero-knowledge proof_. For that, we begin the schnorr identity
process:
1. Alice chooses another large random number (+k+), which we call the
_private nonce_. Again she uses it as a scalar, multiplying it by +G+
to produce +kG+, which we call the _public nonce_. She gives the
public nonce to Bob.
2. Bob chooses a large random number of his own, +e+, which we call the
_challenge scalar_. We say "challenge" because it's used to challenge
Alice to prove that she knows the private key (+x+) for the public key
(+xG+) she previously gave Bob; we say "scalar" because it will later
be used to multiply an EC point.
3. Alice now has the numbers (scalars) +x+, +k+, and +e+. She combines
them together to produce a final scalar +s+ using the formula:
+s = k + ex+. She gives +s+ to Bob.
4. Bob now knows the scalars +s+ and +e+, but not +x+ or +k+. However,
Bob does know +xG+ and +kG+, and he can compute for himself +sG+ and
+exG+. That means he can check the equality of a scaled-up version of
the operation Alice performed: +sG == kG + exG+. If that is equal,
then Bob can be sure that Alice knew +x+ when she generated +s+.
.Schnorr identity protocol with integers instead of points
****
It might be easier to understand the interactive schnorr identity
protocol if we create an insecure oversimplification by substituting each of the values above
(including +G+) with simple integers instead of points on a elliptic curve.
For example, we'll use the prime numbers starting with 3:
Setup: Alice chooses +x=3+ as her private key. She multiplies it by the
generator +G=5+ to get her public key +xG=15+. She gives Bob +15+.
1. Alice chooses the private nonce +k=7+ and generates the public nonce
+kG=35+. She gives Bob +35+.
2. Bob chooses +e=11+ and gives it to Alice.
3. Alice generates +s = 40 = 7 + 11 * 3+. She gives Bob +40+.
4. Bob derives +sG = 200 = 40 * 5+ and +exG = 165 = 11 * 15+. He then
verifies that +200 == 35 + 165+. Note that this is the same operation
that Alice performed but all of the values have been scaled up by +5+
(the value of +G+).
Of course, this is an oversimplified example. When working with simple
integers, we can divide products by the generator +G+ to get the
underlying scalar, which isn't secure. This is why a critical property
of the Elliptic Curve Cryptography (ECC) used in Bitcoin is that
multiplication is easy but division by a point on the curve is impractical. Also, with numbers
this small, finding underlying values (or valid substitutes) through
brute force is easy; the numbers used in Bitcoin are much larger.
****
Let's discuss some of the features of the interactive schnorr
identity protocol that make it secure:
- The nonce (+k+). In step 1, Alice chooses a number that Bob doesn't
know and can't guess and gives him the scaled form of that number,
+kG+. At that point, Bob also already has her public key (+xG+),
which is the scaled form of +x+, her private key. That means when Bob is working on
the final equation (+sG = kG + exG+), there are two independent
variables that Bob doesn't know (+x+ and +k+). It's possible to use
simple algebra to solve an equation with one unknown variable but not
two independent unknown variables, so the presence of Alice's nonce
prevents Bob from being able to derive her private key. It's critical
to note that this protection depends on nonces being unguessable in
any way. If there's anything predictable about Alice's nonce, Bob may
be able to leverage that into figuring out Alice's private key. See
<<nonce_warning>> for more details.
- The challenge scalar (+e+). Bob waits to receive Alice's public nonce
and then proceeds in step 2 to give her a number (the challenge
scalar) that Alice didn't previously know and couldn't have guessed.
It's critical that Bob only give her the challenge scalar after she
commits to her public nonce. Consider what could happen if someone
who didn't know +x+ wanted to impersonate Alice, and Bob accidentally
gave them the challenge scalar +e+ before they told him the public
nonce +kG+. This allows the impersonator to change parameters on both sides of
the equation that Bob will use for verification, +sG == kG + exG+,
specifically they can change both +sG+ and +kG+. Think about a
simplified form of that expression: x = y + a. If you can change both
+x+ and +y+, you can cancel out +a+ using +x' = (x - a) + a+. Any
value you choose for +x+ will now satisfy the equation. For the
actual equation the impersonator simply chooses a random number for +s+, generates
+sG+, and then uses EC subtraction to select a +kG+ that equals +kG =
sG - exG+. They give Bob their calculated +kG+ and later their random
+sG+, and Bob thinks that's valid because +sG == (sG - exG) + exG+.
This explains why the order of operations in the protocol is
essential: Bob must only give Alice the challenge scalar after Alice
has committed to her public nonce.
The interactive identity protocol described here matches part of Claus
Schnorr's original description, but it lacks two essential features we
need for the decentralized Bitcoin network. The first of these is that
it relies on Bob waiting for Alice to commit to her public nonce and
then Bob giving her a random challenge scalar. In Bitcoin, the spender
of every transaction needs to be authenticated by thousands of Bitcoin
full nodes--including future nodes that haven't been started yet but
whose operators will one day want to ensure the bitcoins they receive
came from a chain of transfers where every transaction was valid. Any
Bitcoin node that is unable to communicate with Alice, today or in the
future, will be unable to authenticate her transaction and will be in
disagreement with every other node that did authenticate it. That's not
acceptable for a consensus system like Bitcoin. For Bitcoin to work, we
need a protocol that doesn't require interaction between Alice and each
node that wants to authenticate her.
A simple technique, known as the Fiat-Shamir transform after its
discoverers, can turn the schnorr interactive identity protocol
into a non-interactive digital signature scheme. Recall the importance
of steps 1 and 2--including that they be performed in order. Alice must
commit to an unpredictable nonce; Bob must give Alice an unpredictable
challenge scalar only after he has received her commitment. Recall also
the properties of secure cryptographic hash functions we've used
elsewhere in this book: it will always produce the same output when
given the same input but it will produce a value indistinguishable from
random data when given a different input.
This allows Alice to choose her private nonce, derive her public nonce,
and then hash the public nonce to get the challenge scalar. Because
Alice can't predict the output of the hash function (the challenge), and
because it's always the same for the same input (the nonce), this
ensures that Alice gets a random challenge even though she chooses the nonce
and hashes it herself. We no longer need interaction from Bob. She can
simply publish her public nonce +kG+ and the scalar +s+, and each of the
thousands of full nodes (past and future) can hash +kG+ to produce +e+,
use that to produce +exG+, and then verify +sG == kG + exG+. Written
explicitly, the verification equation becomes +sG == kG + hash(kG) * xG+.
We need one other thing to finish converting the interactive schnorr
identity protocol into a digital signature protocol useful for
Bitcoin. We don't just want Alice to prove that she knows her private
key; we also want to give her the ability to commit to a message. Specifically,
we want her to commit to the data related to the Bitcoin transaction she
wants to send. With the Fiat-Shamir transform in place, we already
have a commitment, so we can simply have it additionally commit to the
message. Instead of +hash(kG)+, we now also commit to the message
+m+ using +hash(kG || m)+, where +||+ stands for concatenation.
We've now defined a version of the schnorr signature protocol, but
there's one more thing we need to do to address a Bitcoin-specific
concern. In BIP32 key derivation, as described in
<<public_child_key_derivation>>, the algorithm for unhardened derivation
takes a public key and adds to it a non-secret value to produce a
derived public key. That means it's also possible to add that
non-secret value to a valid signature for one key to produce a signature
for a related key. That related signature is valid but it wasn't
authorized by the person possessing the private key, which is a major
security failure. To protect BIP32 unhardened derivation and
also support several protocols people wanted to build on top of schnorr
signatures, Bitcoin's version of schnorr signatures, called _BIP340
schnorr signatures for secp256k1_, also commits to the public key being
used in addition to the public nonce and the message. That makes the
full commitment +hash(kG || xG || m)+.
Now that we've described each part of the BIP340 schnorr signature
algorithm and explained what it does for us, we can define the protocol.
Multiplication of integers are performed _modulus p_, indicating that the
result of the operation divided by the number _p_ (as defined in the
secp256k1 standard) and the remainder is used. The number _p_ is very
large, but if it was 3 and the result of an operation was 5, the actual
number we would use is 2 (i.e., 5 divided by 3 is 2).
Setup: Alice chooses a large random number (+x+) as her private key
(either directly or by using a protocol like BIP32 to deterministically
generate a private key from a large random seed value). She uses the
parameters defined in secp256k1 (see <<elliptic_curve>>) to multiply the
generator +G+ by her scalar +x+, producing +xG+ (her public key). She
gives her public key to everyone who will later authenticate her Bitcoin
transactions (e.g. by having +xG+ included in a transaction output). When
she's ready to spend, she begins generating her signature:
1. Alice chooses a large random private nonce +k+ and derives the public
nonce +kG+.
2. She chooses her message +m+ (e.g. transaction data) and generates the
challenge scalar +e = hash(kG || xG || m)+.
3. She produces the scalar +s = k + ex+. The two values +kG+ and +s+
are her signature. She gives this signature to everyone who wants to
verify that signature; she also needs to ensure everyone receives her
message +m+. In Bitcoin, this is done by including her signature in
the witness structure of her spending transaction and then relaying that
transaction to full nodes.
4. The verifiers (e.g. full nodes) use +s+ to derive +sG+ and then
verify that +sG == kG + hash(kG || xG || m)*xG+. If the equation is
valid, Alice proved that she knows her private key +x+ (without
revealing it) and committed to the message +m+ (containing the
transaction data).
==== Serialization of schnorr signatures
A schnorr signature consists of two values, +kG+ and +s+. The value
+kG+ is a point on Bitcoin's elliptic curve (called secp256k1) and so
would normally be represented by two 32-byte coordinates, e.g. +(x,y)+.
However, only the x coordinate is needed, so only that value is
included. When you see +kG+ below, note that it's only that point's x
coordinate.
The value +s+ is a scalar (a number meant to multiply other numbers). For
Bitcoin's secp256k1 curve, it can never be more than 32 bytes long.
Although both +kG+ and +s+ can sometimes be values that can be
represented with fewer than 32 bytes, it's improbable that they'd be
much smaller than 32 bytes, and so they're serialized as two 32 byte
values (i.e., values smaller than 32 bytes have leading zeroes).
They're serialized in the order of +kG+ and then +s+, producing exactly
64 bytes.
The taproot soft fork, also called v1 segwit, introduced schnorr signatures
to Bitcoin and is the only way they are used in Bitcoin as of this writing. When
used with either taproot keypath or scriptpath spending, a 64-byte
schnorr signature is considered to use a default signature hash (sighash)
that is +SIGHASH_ALL+. If an alternative sighash is used, or if the
spender wants to waste space to explicitly specify +SIGHASH_ALL+, a
single additional byte is appended to the signature that specifies the
signature hash, making the signature 65 bytes.
As we'll see, either 64 or 65 bytes is considerably more efficient that
the serialization used for ECDSA signatures described in
<<serialization_of_signatures_der>>.
[[schnorr_multisignatures]]
==== Schnorr-based scriptless multisignatures
In the single-signature schnorr protocol described in <<schnorr_signatures>>, Alice
uses a signature (+kG+, +s+) to publicly prove her knowledge of her
private key, which in this case we'll call +y+. Imagine if Bob also has
a private key (+z+) and he's willing to work with Alice to prove that
together they know +x = y + z+ without either of them revealing their
private key to each other or anyone else. Let's go through the BIP340
schnorr signature protocol again.
[WARNING]
====
The simple protocol we are about to describe is not secure for the
reasons we will explain shortly. We use it only to demonstrate the
mechanics of schnorr multisignatures before describing related protocols
that are believed to be secure.
====
Alice and Bob need to derive the public key for +x+, which is +xG+.
Since it's possible to use Elliptic Curve (EC) operations to add two EC
points together, they start by Alice deriving +yG+ and Bob deriving
+zG+. Then then add them together to create +xG = yG + zG+. The point
+xG+ is their _aggregated public key_. To create a signature, they begin the
simple multisignature protocol:
1. They each individually choose a large random private nonce, +a+ for
Alice and +b+ for Bob. They also individually derive the corresponding
public nonce +aG+ and +bG+. Together, they produce an aggregated
public nonce +kG = aG + bG+.
2. They agree on the message to sign, +m+ (e.g. a transaction), and
each generate a copy of the challenge scalar: +e = hash(kG || xG || m)+.
3. Alice produces the scalar +q = a + ey+. Bob produces the scalar
+r = b + ez+. They add the scalars together to produce
+s = q + r+. Their signature is the two values +kG+ and +s+.
4. The verifiers check their public key and signature using the normal
equation: +sG == kG + hash(kG || xG || m)*xG+.
Alice and Bob have proven that they know the sum of their private keys without
either one of them revealing their private key to the other or anyone
else. The protocol can be extended to any number of participants, e.g.
a million people could prove they knew the sum of their million
different keys.
The protocol above has several security problems. Most notable is that one
party might learn the public keys of the other parties before committing
to their own public key. For example, Alice generates her public key
+yG+ honestly and shares it with Bob. Bob generates his public key
using +zG - yG+. When their two keys are combined (+yG + zG - yG+), the
positive and negative +yG+ terms cancel out so the public key only represents
the private key for +z+, i.e. Bob's private key. Now Bob can create a
valid signature without any assistance from Alice. This is called a
_key cancellation attack_.
There are various ways to solve the key cancellation attack. The
simplest scheme would be to require each participant commit to their
part of the public key before sharing anything about that key with all
of the other participants. For example, Alice and Bob each individually
hash their public keys and share their digests with each other. When
they both have the other's digest, they can share their keys. They
individually check that the other's key hashes to the previously
provided digest and then proceed with the protocol normally. This prevents
either one of them from choosing a public key that cancels out the keys
of the other participants. However, it's easy to fail to implement this
scheme correctly, such as using it in a naive way with unhardened
BIP32 public key derivation. Additionally, it adds an extra step for
communication between the participants, which may be undesirable in many
cases. More complex schemes have been proposed that address these
shortcomings.
In addition to the key cancellation attack, there are a number of
attacks possible against nonces. Recall that the purpose of the nonce
is to prevent anyone from being able to use their knowledge of other values
in the signature verification equation to solve for your private key,
determining its value. To effectively accomplish that, you must use a
different nonce every time you sign a different message or change other
signature parameters. The different nonces must not be related in any
way. For a multisignature, every participant must follow these rules or
it could compromise the security of other participants. In addition,
cancellation and other attacks need to be prevented. Different
protocols that accomplish these aims make different tradeoffs, so
there's no single multisignature protocol to recommend in all cases.
Instead, we'll note three from the MuSig family of protocols:
MuSig::
Also called _MuSig1_, this protocol requires three rounds of
communication during the signing process, making it similar to the
process we described above. MuSig1's greatest advantage is its
simplicity.
MuSig2::
This only requires two rounds of communication and can sometimes allow
one of the rounds to be combined with key exchange. This can
significantly speed up signing for certain protocols, such as how
scriptless multisignatures are planned to be used in the Lightning
Network. MuSig2 is specified in BIP327 (the only scriptless
multisignature protocol that has a BIP as of this writing).
MuSig-DN::
DN stands for Deterministic Nonce, which eliminates as a concern a
problem known as the _repeated session attack_. It can't be combined
with key exchange and it's significantly more complex to implement
than MuSig or Musig2.
For most applications, MuSig2 is the best multisignature protocol
available at the time of writing.
[[schnorr_threshold_signatures]]
==== Schnorr-based scriptless threshold signatures
Scriptless multisignature protocols only work for k-of-k signing.
Everyone with a partial public key that becomes part of the aggregated
public key must contribute a partial signature and partial nonce to the
final signature. Sometimes, though, the participants want to allow a
subset of them to sign, such as t-of-k where a threshold (t) number of participants can sign for
a key constructed by k participants. That type of signature is called a
_threshold signature_.
We saw script-based threshold signatures in
<<multisig>>. But just as
scriptless multisignatures save space and increase privacy compared to
scripted multisigantures, _scriptless threshold signatures_ save space and
increase privacy compared to _scripted threshold signatures_. To anyone
not involved in the signing, a _scriptless threshold signatures_ looks
like any other signature which could've been created by a single-sig
user or through a scriptless multisignature protocol.
Various methods are known for generating scriptless threshold
signatures, with the simplest being a slight modification of how we
created scriptless multisignatures previously. This protocol also
depends on verifiable secret sharing (which itself depends on secure
secret sharing).
Basic secret sharing can work through simple splitting. Alice has a
secret number that she splits into three equal-length parts and shares
with Bob, Carol, and Dan. Those three can combine the partial numbers
they received (called _shares_) in the correct order to reconstruct
Alice's secret. A more sophisticated scheme would involve Alice adding
on some additional information to each share, called a correction code,
that allows any two of them to recover the number. This scheme is not
secure because each share gives its holder partial knowledge of Alice's
secret, making it easier for the participant to guess Alice's secret
than a non-participant who didn't have a share.
A secure secret sharing scheme prevents participants from learning
anything about the secret unless they combine the minimum threshold
number of shares. For example, Alice can choose a threshold of
+2+ if she wants any two of Bob, Carol, and Dan to be able to
reconstruct her secret. The best known secure secret sharing algorithm
is _Shamir's Secret Sharing Scheme_, commonly abbreviated SSSS and named
after its discoverer, one of the same discoverers of the Fiat-Shamir
transform we saw in <<schnorr_signatures>>.
In some cryptographic protocols, such as the scriptless threshold signature
schemes we're working towards, it's critical for Bob, Carol, and Dan to
know that Alice followed her side of the protocol correctly. They need to
know that the shares she creates all derive from the same secret, that
she used the threshold value she claims, and that she gave each one of
them a different share. A protocol that can accomplish all of that,
and still be a secure secret sharing scheme, is a _verifiable secret
sharing scheme_.
To see how multisignatures and verifiable secret sharing works for
Alice, Bob, and Carol, imagine they each wish to receive funds that can
be spent by any two of them. They collaborate as described in
<<schnorr_multisignatures>> to produce a regular multisignature public
key to accept the funds (k-of-k). Then each participant derives two
secret shares from their private key--one for each of two the other
participants. The shares allow any two of them to reconstruct the
originating partial private key for the multisignature. Each participant
distributes one of their secret shares to the other two participants,
resulting in each participant storing their own partial private key and
one share for every other participant. Subsequently, each participant
verifies the authenticity and uniqueness of the shares they received
compared to the shares given to the other participants.
Later on, when (for example) Alice and Bob want to generate a scriptless
threshold signature without Carol's involvement, they exchange the two
shares they possess for Carol. This enables them to reconstruct Carol's
partial private key. Alice and Bob also have their private keys,
allowing them to create a scriptless multisignature with all three
necessary keys.
In other words, the scriptless threshold signature scheme described
above is the same as a scriptless multisignature scheme except that
a threshold number of participants have the ability to reconstruct the
partial private keys of any other participants who are unable or
unwilling to sign.
This does point to a few things to be aware about when considering a
scriptless threshold signature protocol:
1. No accountability: because Alice and Bob reconstruct Carol's partial
private key, there can be no fundamental difference between a scriptless
multisignature produced by a process that involved Carol and one that
didn't. Even if Alice, Bob, or Carol claim that they didn't sign,
there's no guaranteed way for them to prove that they didn't
help produce the signature. If it's important to know which members of
the group signed, you will need to use a script.
2. Manipulation attacks: imagine that Bob tells Alice that Carol is
unavailable, so they work together to reconstruct Carol's partial
private key. Then Bob tells Carol that Alice is unavailable, so they
work together to reconstruct Alice's partial private key. Now Bob has
his own partial private key plus the keys of Alice and Carol, allowing
him to spend the funds himself without their involvement. This attack can
be addressed if all of the participants agree to only communicate using a
scheme that allows any one of them to see all of the other's messages,
e.g. if Bob tells Alice that Carol is unavailable, Carol is able to see
that message before she begins working with Bob. Other solutions,
possibly more robust solutions, to this problem were being researched at
the time of writing.
No scriptless threshold signature protocol has been proposed as a BIP
yet, although significant research into the subject has been performed
by multiple Bitcoin contributors and we expect peer-reviewed solutions
will become available after the publication of this book.
[[ecdsa_signatures]]
=== ECDSA signatures
Unfortunately for the future development of Bitcoin and many other
applications, Claus Schnorr patented the algorithm he discovered and
prevented its use in open standards and open source software for almost
two decades. Cryptographers in the early 1990s who were blocked from
using the schnorr signature scheme developed an alternative construction
called the _Digital Signature Algorithm_ (DSA), with a version adapted
to Elliptic Curves called ECDSA.
The ECDSA scheme and standardized parameters for suggested curves it could be used
with were widely implemented in cryptographic libraries by the time
development on Bitcoin began in 2007. This was almost certainly the
reason why ECDSA was the only digital signature protocol that Bitcoin
supported from its first release version until the activation of the
taproot soft fork in 2021. ECDSA remains supported today for all
non-taproot transactions. Some of the differences compared to schnorr
signatures include:
More complex::
As we'll see, ECDSA requires more operations to create or verify a
signature than the schnorr signature protocol. It's not significantly
more complex from an implementation standpoint, but that extra
complexity makes ECDSA less flexible, less performant, and harder to
prove secure.
Less provable security::
The interactive schnorr signature identification protocol depends only
on the strength of the Elliptic Curve Discrete Logarithm Problem
(ECDLP). The non-interactive authentication protocol used in Bitcoin
also relies on the Random Oracle Model (ROM). However, ECDSA's extra
complexity has prevented a complete proof of its security being
published (to the best of our knowledge). We are not experts in
proving cryptographic algorithms, but it seems unlikely after 30 years
that ECDSA will be proven to only require the same two assumptions as
schnorr.
Non-linear::
ECDSA signatures cannot be easily combined to create scriptless
multisignatures or used in related advanced applications such as
multi-party signature adaptors. There are workarounds for this
problem, but they involve additional extra complexity which
significantly slows down operations and which, in some cases, has
resulted in software accidentally leaking private keys.
Looking at the math of ECDSA,
signatures are created by a mathematical function _F_~_sig_~
that produces a signature composed of two values. In ECDSA, those two
values are _R_ and _s_.
The signature
algorithm first generates a private nonce (_k_) and derives from it a public
nonce (_K_). The _R_ value of the digital signature is then the x
coordinate of the nonce _K_.
From there, the algorithm calculates the _s_ value of the signature,
such that. Like we did with schnorr signatures, operations involving
integers are modulus p.
_s_ = __k__^-1^ (__Hash__(__m__) + __x__ * __R__)
where:
* _k_ is the private nonce
* _R_ is the x coordinate of the public nonce
* _x_ is the Alice's private key
* _m_ is the message (transaction data)
Verification is the inverse of the signature generation function, using
the _R_, _s_ values and the public key to calculate a value _K_, which
is a point on the elliptic curve (the public nonce used in
signature creation):
_K_ = __s__^-1^ * __Hash__(__m__) * _G_ + __s__^-1^ * _R_ * _X_
where:
- _R_ and _s_ are the signature values
- _X_ is Alice's public key
- _m_ is the message (the transaction data that was signed)
- _G_ is the elliptic curve generator point
If the x coordinate of the calculated point _K_ is equal to _R_, then
the verifier can conclude that the signature is valid.
[TIP]
====
ECDSA is necessarily a fairly complicated piece of math; a full
explanation is beyond the scope of this book. A number of great guides
online take you through it step by step: search for "ECDSA explained".
====
[[serialization_of_signatures_der]]
==== Serialization of ECDSA signatures (DER)
Let's look at
the following DER-encoded signature:
----
3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e381301
----
That signature is a serialized byte-stream of the +R+ and +S+ values
produced by to prove control of the private key authorized
to spend an output. The serialization format consists of nine elements
as follows:
* +0x30+&#x2014;indicating the start of a DER sequence
* +0x45+&#x2014;the length of the sequence (69 bytes)
* +0x02+&#x2014;an integer value follows
* +0x21+&#x2014;the length of the integer (33 bytes)
* +R+&#x2014;++00884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb++
* +0x02+&#x2014;another integer follows
* +0x20+&#x2014;the length of the integer (32 bytes)
* +S+&#x2014;++4b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813++
* A suffix (+0x01+) indicating the type of hash used (+SIGHASH_ALL+)
[[nonce_warning]]
=== The Importance of Randomness in Signatures
As we saw in <<schnorr_signatures>> and <<ecdsa_signatures>>,
the signature generation algorithm uses a random number _k_, as the basis
for a private/public nonce pair. The value of _k_ is not
important, _as long as it is random_. If signatures from the same
private key use the private nonce _k_ with different messages
(transactions), then the
signing _private key_ can be calculated by anyone. Reuse of the same
value for _k_ in a signature algorithm leads to exposure of the private
key!
[WARNING]
====
If the same value _k_
is used in the signing algorithm on two different transactions, the
private key can be calculated and exposed to the world!
====
This is not just a theoretical possibility. We have seen this issue lead
to exposure of private keys in a few different implementations of
transaction-signing algorithms in Bitcoin. People have had funds stolen
because of inadvertent reuse of a _k_ value. The most common reason for
reuse of a _k_ value is an improperly initialized random-number
generator.
To avoid this
vulnerability, the industry best practice is to not generate _k_ with a
random-number generator seeded only with entropy, but instead to use a
process seeded in part with the transaction data itself plus the
private key being used to sign.
This ensures that each transaction produces a different _k_. The
industry-standard algorithm for deterministic initialization of _k_ for
ECDSA is defined in https://tools.ietf.org/html/rfc6979[RFC6979], published by
the Internet Engineering Task Force. For schnorr signatures, BIP340
recommends a default signing algorithm.
BIP340 and RFC6979 can generate _k_ entirely deterministically, meaning the same
transaction data will always produce the same _k_. Many wallets do this
because it makes it easy to write tests to verify their safety-critical
signing code is producing _k_ values correctly. RFC6979 also allows
including additional data in the calculation. If that data is entropy,
then a different _k_ will be produced even if the exact same transaction
data is signed. This can increase protection against sidechannel and
fault-injection attacks.
If you are implementing an algorithm to sign transactions in Bitcoin,
you _must_ use BIP340, RFC6979, or a similar algorithm to
ensure you generate a different _k_ for each transaction.
=== Segregated Witness's New Signing Algorithm
Signatures in bitcoin transactions are applied on a _commitment hash_,
which is calculated from the transaction data, locking specific parts of
the data indicating the signer's commitment to those values. For
example, in a simple +SIGHASH_ALL+ type signature, the commitment hash
includes all inputs and outputs.
Unfortunately, the way the legacy commitment hashes were calculated introduced the
possibility that a node verifying a signature can be forced to perform
a significant number of hash computations. Specifically, the hash
operations increase roughly quadratically with respect to the number of
inputs in the transaction. An attacker could therefore create a
transaction with a very large number of signature operations, causing
the entire Bitcoin network to have to perform hundreds or thousands of
hash operations to verify the transaction.
Segwit represented an opportunity to address this problem by changing
the way the commitment hash is calculated. For segwit version 0 witness
programs, signature verification occurs using an improved commitment
hash algorithm as specified in BIP143.
The new algorithm allows the number of
hash operations to increase by a much more gradual O(n) to the number of
signature operations, reducing the opportunity to create
denial-of-service attacks with overly complex transactions.
In this chapter, we learned about schnorr and ECDSA signatures for
Bitcoin. This explains how full nodes authenticate transactions to
ensure that only someone controlling the key to which bitcoins were
received can spend those bitcoins. We also examined several advanced
applications of signatures, such as scriptless multisignatures and
scriptless threshold signatures that can be used to improve the
efficiency and privacy of Bitcoin. In the past few chapters, we've
learned how to create transactions, how to secure them with
authorization and authentication, and how to sign them. We will next
learn how to encourage miners to confirm them by adding fees to the
transactions we create.
//FIXME: mention segwit v0 and v1 coverage of values to aid hardware
//wallets

669
ch09_fees.adoc Normal file
View File

@ -0,0 +1,669 @@
[[tx_fees]]
== Transaction Fees
The digital signature we saw Alice create in <<c_signatures>> only
proves that she knows her private key and that she committed to a
transaction that pays Bob. She can create another signature that
instead commits to a transaction paying Carol--a transaction that spends
the same output (bitcoins) that she used to pay Bob. Those two
transactions are now _conflicting transactions_ because only one
transaction spending a particular output can be included in the valid
blockchain with the most proof of work--the blockchain that full nodes
use to determine which keys control which bitcoins.
To protect himself against conflicting transactions, it would be wise
for Bob to wait until the transaction from Alice is included in the
blockchain to a sufficient depth before he considers the money he
received as his to spend (see <<confirmations>>).
For Alice's transaction to be included in the
blockchain, it must be included in a _block_ of transactions. There are
a limited number of blocks produced in a given amount of time and each
block only has a limited amount of space. Only the miner who creates
that block gets to choose which transactions to include. Miners may
select transactions by any criteria they want, including refusing to
include any transactions at all.
[NOTE]
====
When we say "transactions" in this chapter, we refer to every
transaction in a block except for the first transaction. The first
transaction in a block is a _coinbase transaction_, described in
<<coinbase_transactions>>, which allows the miner of the block to
collect their reward for producing the block. Unlike other
transactions, a coinbase transaction doesn't spend the output of a
previous transaction and is also an exception to several other rules
that apply to other transactions. Coinbase transactions don't pay
transaction fees, don't need to be fee bumped, aren't subject to
transaction pinning, and are largely uninteresting to the following
discussion about fees--so we're going to ignore them in this chapter.
====
The criteria that almost all miners use to select which transactions to
include in their blocks is to maximize their revenue. Bitcoin was
specifically designed to accommodate this by providing a mechanism that
allows a transaction to give money to the miner who includes that
transaction in a block. We call that mechanism _transaction fees_,
although it's not a fee in the usual sense of that word. It's not an
amount set by the protocol or by any particular miner--it's much more
like a bid in an auction. The good being purchased is the portion of
limited space in a block that a transaction will consume. Miners choose
the set of transactions whose bids will allow them to earn the greatest
revenue.
In this chapter, we'll explore various aspects of those
bids--transaction fees--and how they influence the creation and
management of Bitcoin transactions.
=== Who pays the transaction fee?
Most payment systems involve some sort of fee for transacting, but
often this fee is hidden from typical buyers. For example, a merchant
may advertise the same item for the same price whether you pay with cash
or a credit card even though their payment processor may charge them
a higher fee for credit transactions than their bank charges them for
cash deposits.
In Bitcoin, every spend of bitcoins must be authenticated (typically
with a signature), so it's not possible for a transaction to pay a fee
without the permission of the spender. It is possible for the receiver
of a transaction to pay a fee in a different transaction--and we'll see
that in use later--but if we want a single transaction to pay its own
fee, that fee needs to be something agreed upon by the spender. It
can't be hidden.
Bitcoin transactions are designed so that it doesn't take any extra
space in a transaction for a spender to commit to the fee it pays. That
means that, even though it's possible to pay the fee in a different
transaction, it's most efficient (and thus cheapest) to pay the fee in a
single transaction.
In Bitcoin,
the fee is a bid and the amount paid contributes to determining how long
it will take the transaction to confirm. Both spenders and receivers of
a payment typically have an interest in having it confirming quickly, so
normally allowing only spenders to choose fees can sometimes be a
problem; we'll look at a solution to that problem in <<cpfp>>. However,
in many common payment flows, the parties with the highest desire to see a
transaction confirm quickly--that is, the parties who'd be the most
willing to pay higher fees--are the the spenders.
For those reasons, both technical and practical, it is customary in
Bitcoin for spenders to pay transaction fees. There are exceptions,
such as for merchants which accept unconfirmed transactions and in
protocols that don't immediately broadcast transactions after they are
signed (preventing the spender from being able to choose an appropriate
fee for the current market). We'll explore those exceptions later.
=== Fees and fee rates
Each transaction only pays a single fee--it doesn't matter how large the
transaction is. However, the larger transactions become, the fewer of
them a miner will be able to fit in a block. For that reason, miners
evaluate transactions the same way you might comparison shop between
several equivalent items at the market: they divide the price by the
quantity.
Whereas you might divide the cost of several different bags of rice by
each bag's weight to find the lowest price per weight (best deal), miners
divide the fee of a transaction by its size (also called its weight) to
find the highest fee per weight (most revenue). In Bitcoin, we use the
term _fee rate_ for a transaction's size divided by weight. Due to
changes in Bitcoin over the years, fee rate can be expressed in
different units:
- BTC/Bytes (a legacy unit rarely used anymore)
- BTC/Kilobytes (a legacy unit rarely used anymore)
- BTC/Vbytes (rarely used)
- BTC/Kilo-vbyte (used mainly in Bitcoin Core)
- Satoshi/Vbyte (most commonly used today)
- Satoshi/Weight (also commonly used today)
We recommend either the sat/vbyte or sat/weight units for displaying
fee rates.
[WARNING]
====
Be careful accepting input for fee rates. If a user copies and pastes a
fee rate printed in one denominator into a field using a different
enumerator, they could overpay fees by 1,000 times. If they instead
switch the enumerator, they could theoretically overpay by 100,000,000
times. Wallets should make it hard for the user to pay an excessive
fee rate and may want to prompt the user to confirm any fee rate that was
not generated by the wallet itself using a trusted data source.
An excessive fee, also called an _absurd fee_, is any fee rate that's
significantly higher than the amount that fee rate estimators currently
expect is necessary to get a transaction confirmed in the next block.
Note that wallets should not entirely prevent users from choosing an
excessive fee rate--they should only make using such a fee rate hard to do
by accident. There are legitimate reasons for users to overpay fees on
rare occasions.
====
=== Estimating appropriate fee rates
We've established that you can pay a lower fee rate if you're willing to
wait longer for your transaction to be confirmed, with the exception
that paying too low of a fee rate could result in your transaction never
confirming. Because fee rates are bids in an open auction for block
space, it's not possible to perfectly predict what fee rate you need to
pay to get your transaction confirmed by a certain time. However, we
can generate a rough estimate based on what fee rates other transactions
have paid in the recent past.
A full node can record three pieces of information about each
transactions it sees: the time (block height) when it first received
that transaction, the block height when that transaction was confirmed,
and the fee rate paid by that transaction. By grouping together
transactions that arrived at similar heights, were confirmed at similar
heights, and which paid similar fees, we can calculate how many blocks it
took to confirm transactions paying a certain fee rate. We can then
assume that a transaction paying a similar fee rate now will take a
similar number of blocks to confirm. Bitcoin Core includes a fee rate
estimator that uses these principles, which can be called using the
`estimatesmartfee` RPC with a parameter specifying how many blocks
you're willing to wait before the transaction is highly likely to
confirm (for example, 144 blocks is about 1 day).
----
$ bitcoin-cli -named estimatesmartfee conf_target=144
{
"feerate": 0.00006570,
"blocks": 144
}
----
Many web-based services also provide fee estimation as an API. For a
current list, see https://www.lopp.net/bitcoin-information/fee-estimates.html
As mentioned, fee rate estimation can never be perfect. One common
problem is that the fundamental demand might change, adjusting the
equilibrium and either increasing prices (fees) to new heights or
decreasing them towards the minimum.
If fee rates go down, then a transaction
that previously paid a normal fee rate might now be paying a high fee
rate and it will be confirmed earlier than expected. There's no way to
lower the fee rate on a transaction you've already sent, so you're stuck
paying a higher fee rate. But, when fee rates go up, there's a need for
methods to be able to increase the fee rates on those transactions,
which is called _fee bumping_. There are two commonly used types of fee
bumping in Bitcoin, Replace-By-Fee (RBF) and Child Pays For Parent
(CPFP).
[[rbf]]
=== Replace-By-Fee (RBF) Fee Bumping
To increase the fee of a transaction using RBF fee bumping, you create
a conflicting version of the transaction which pays a higher fee. Two
or more transactions are considered to be _conflicting transactions_ if
only one of them can be included in a valid blockchain, forcing a miner
to chose only one of them. Conflicts occur when two or more transactions
each try to spend one of the same UTXOs, i.e. they each include an input
that has the same outpoint (reference to the output of a previous
transaction).
To prevent someone from consuming large amounts of bandwidth by creating
an unlimited number of conflicting transactions and sending them through
the network of relaying full nodes, Bitcoin Core and other full nodes
that support transaction replacement require each replacement
transaction to pay a higher fee rate than the transaction being
replaced. Bitcoin Core also currently requires the replacement
transaction to pay a higher total fee than the original transaction, but
this requirement has undesired side effects and developers have been
looking for ways to remove it at the time of writing.
Bitcoin Core currently supports two variations of RBF:
Opt-in RBF::
An unconfirmed transaction can signal to miners and full nodes that
the creator of the transaction wants to allow it to be replaced by a
higher fee rate version. This signal and the rules for using it
are specified in BIP125. As of this writing, this has been enabled by
default in Bitcoin Core for several years.
Full RBF::
Any unconfirmed transaction can be replaced by a higher fee rate
version. As of this writing, this can be optionally enabled in
Bitcoin Core (but it is disabled by default).
.Why are there two variants of RBF?
****
The reason for the two different versions of RBF is that full RBF has
been controversial. Early versions of Bitcoin allowed transaction
replacement but this behavior was disabled for several releases. During
that time, a miner or full node using the software now called Bitcoin
Core would not replace the first version of an unconfirmed transaction
they received with any different version. Some merchants came to expect
this behavior: they assumed that any valid unconfirmed transaction which
paid an appropriate fee rate would eventually become a confirmed
transaction, so they provided their goods or services shortly after
receiving such an unconfirmed transaction.
However, there's no way for the Bitcoin protocol to guarantee that any
unconfirmed transaction will eventually be confirmed. As mentioned
earlier in this chapter, every miner gets to choose for themselves which
transactions they will try to confirm--including which versions of those
transactions. Bitcoin Core is open source software, so anyone with a
copy of its source code can add (or remove) transaction replacement.
Even if Bitcoin Core wasn't open source, Bitcoin is an open protocol
that can be reimplemented from scratch by a sufficiently competent
programmer, allowing the reimplementor to include or not include
transaction replacement.
Transaction replacement breaks the assumption of some merchants that
every reasonable unconfirmed transaction will eventually be confirmed.
An alternative version of a transaction can pay the same outputs as the
original, but it isn't required to pay any of those outputs. If the
first version of an unconfirmed transaction pays a merchant, the second
version might not pay them. If the merchant provided goods or services
based on the first version, but the second version gets confirmed, then
the merchant will not receive payment for its costs.
Some merchants, and people supporting them, requested that transaction
replacement not be re-enabled in Bitcoin Core. Other people pointed out
that transaction replacement provides benefits, including the ability to
fee bump transactions that initially paid too low of a fee rate.
Eventually, developers working on Bitcoin Core implemented a compromise:
instead of allowing every unconfirmed transaction to be replaced (full
RBF), they only programmed Bitcoin Core to allow replacement of
transactions that signaled they wanted to allow replacement (opt-in RBF).
Merchants can check the transactions they receive for the opt-in
signal and treat those transactions differently than those without the
signal.
This doesn't change the fundamental concern: anyone can still alter
their copy of Bitcoin Core, or create a reimplementation, to allow full
RBF--and some developers even did this, but seemingly few people used
their software.
After several years, developers working on Bitcoin Core changed the
compromise slightly. In addition to keeping opt-in RBF by default, they
added an option that allows users to enable full RBF. If enough mining
hash rate and relaying full nodes enable this option, it will be
possible for any unconfirmed transaction to eventually be replaced by a
version paying a higher fee rate. As of this writing, it's not clear
whether or not that has happened yet.
****
As a user, if you plan to use RBF fee bumping, you will first need to
choose a wallet that supports it, such as one of the wallets listed as
having "Sending support" on
https://bitcoinops.org/en/compatibility/#replace-by-fee-rbf
As a developer, if you plan to implement RBF fee bumping, you will first
need to decided whether to perform opt-in RBF or full RBF. At the time
of writing, opt-in RBF is the only method that's sure to work. Even if
full RBF becomes reliable, there will likely be several years where
replacements of opt-in transactions get confirmed slightly faster than
full-RBF replacements. If you choose opt-in RBF, your wallet will need
to implement the signaling specified in BIP125, which is a simple
modification to any one of the sequence fields in a transaction (see
<<sequence>>). If you choose full RBF, you don't need to include any
signaling in your transactions. Everything else related to RBF is the
same for both approaches.
When you need to fee bump a transaction, you will simply create a new
transaction that spends at least one of the same UTXOs as the original
transaction you want to replace. You will likely want to keep the
same outputs in the transaction which pay the receiver (or receivers).
You may pay the increased fee by reducing the the value of your change
output or by adding additional inputs to the transaction. Developers
should provide users with a fee-bumping interface that does all of this
work for them and simply asks them (or suggests to them) how much the
fee rate should be increased.
[WARNING]
====
Be very careful when creating more than one replacement of the same
transaction. You must ensure than all versions of the transactions
conflict with each other. If they aren't all conflicts, it may be
possible for multiple separate transactions to confirm, leading you to
overpay the receivers. For example:
- Transaction version 0 includes input _A_.
- Transaction version 1 includes inputs _A_ and _B_ (e.g., you had to add
input _B_ to pay the extra fees)
- Transaction version 2 includes inputs B and C (e.g., you had to add input
_C_ to pay the extra fees but _C_ was large enough that you no longer
need input _A_).
In the above scenario, any miner who saved version 0 of the transaction
will be able to confirm both it and version 2 of the transaction. If
both versions pay the same receivers, they'll be paid twice (and the
miner will receive transaction fees from two separate transactions).
A simple method to avoid this problem is to ensure the replacement
transaction always includes all of the same inputs as the previous
version of the transaction.
====
The advantage of RBF fee bumping over other types of fee bumping is that
it can be very efficient at using block space. Often, a replacement
transaction is the same size as the transaction it replaces. Even when
it's larger, it's often the same size as the transaction the user would
have created if they had paid the increased fee rate in the first place.
The fundamental disadvantage of RBF fee bumping is that it can normally
only be performed by the creator of the transaction--the person or
people who were required to provide signatures or other authentication
data for the transaction. An exception to this is transactions which
were designed to allow additional inputs to be added by using sighash
flags, see <<sighash_types>>, but that presents its own challenges. In
general, if you're the receiver of an unconfirmed transaction and you
want to make it confirm faster (or at all), you can't use an RBF fee
bump; you need some other method.
There are additional problems with RBF that we'll explore in the
subsequent <<transaction_pinning>> section.
[[cpfp]]
=== Child Pays For Parent (CPFP) Fee Bumping
Anyone who receives the output of an unconfirmed transaction can
incentivize miners to confirm that transaction by spending that output.
The transaction you want to get confirmed is called the _parent
transaction_. A transaction which spends an output of the parent
transaction is called a _child transaction_.
As we learned in <<outpoints>>, every input in a confirmed transaction
must reference the unspent output of a transaction which appears earlier
in the blockchain (whether earlier in the same block or in a previous
block). That means a miner who wants to confirm a child transaction
must also ensure that its parent transaction is confirmed. If the
parent transaction hasn't been confirmed yet but the child transaction
pays a high enough fee, the miner can consider whether it would be
profitable to confirm both of them in the same block.
To evaluate the profitability of mining both a parent and child
transaction, the miner looks at them as a _package of transactions_ with
an aggregate size and aggregate fees, from which the fees can be divided
by the size to calculate a _package fee rate_. The miner can then sort
all of the individual transactions and transaction packages they know
about by fee rate and include the highest-revenue ones in the block
they're attempting to mine, up to the maximum size (weight) allowed to
be included in a block. To find even more packages that might be
profitable to mine, the miner can evaluate packages across multiple
generations (e.g. an unconfirmed parent transaction being combined with
both its child and grandchild). This is called _ancestor fee rate
mining_.
Bitcoin Core has implemented ancestor fee rate mining for many years,
and it's believed that almost all miners use it at the time of writing.
That means it's practical for wallets to use this feature to fee bump an
incoming transaction by using a child transaction to pay for its parent
(CPFP).
CPFP has several advantages over RBF. Anyone who receives an output
from a transaction can use CPFP--that includes both the receivers of
payments and the spender (if the spender included a change output). It
also doesn't require replacing the original transaction, which makes it
less disruptive to some merchants than RBF.
The primary disadvantage of CPFP compared to RBF is that CPFP typically
uses more block space. In RBF, a fee bump transaction is often the same
size as the transaction it replaces. In CPFP, a fee bump adds a whole
separate transaction. Using extra block space requires paying extra
fees beyond the the cost of the fee bump.
There are several challenges with CPFP, some of which we'll explore in
<<transaction_pinning>>. One other problem which we
specifically need to mention is the minimum relay fee rate problem,
which is addressed by package relay.
==== Package Relay
Early versions of Bitcoin Core didn't place any limits on the number of
unconfirmed transactions they stored for later relay and mining in their
mempools (see <<mempool>>). Of course, computers have physical limits, whether
it's the memory (RAM) or disk space--it's not possible for a full node
to store an unlimited number of unconfirmed transactions. Later
versions of Bitcoin Core limited the size of the mempool to hold about
one day's worth of transactions, storing only the transactions or packages
with the highest fee rate.
That works extremely well for most things, but it creates a dependency
problem. In order to calculate the fee rate for a transaction package,
we need both the parent and descendant transactions--but if the parent
transaction doesn't pay a high enough fee rate, it won't be kept in a
node's mempool. If a node receives a child transaction without having
access to its parent, it can't do anything with that transaction.
The solution to this problem is the ability to relay transactions as a
package, called _package relay_, allowing the receiving node to evaluate
the fee rate of the entire package before operating on any individual
transaction. As of this writing, developers working on Bitcoin Core
have made significant progress on implementing package relay and a
limited early version of it may be available by the time this book is
published.
Package relay is especially important for protocols based on
time-sensitive presigned transactions, such as Lightning Network. In
non-cooperative cases, some presigned transactions can't be fee bumped
using RBF, forcing them to depend on CPFP. In those protocols, some
transactions may also be created long before they need to be broadcast,
making it effectively impossible to estimate an appropriate fee rate.
If a presigned transaction pays a fee rate below the amount necessary to
get into a node's mempool, there's no way to fee bump it with a child.
If that prevents the transaction from confirming in time, an honest user
might lose money. Package relay is the solution for this critical
problem.
[[transaction_pinning]]
=== Transaction Pinning
Although both RBF and CPFP fee bumping work in the basic cases we
described, there are rules related to both
methods that are designed to prevent denial of service attacks on miners
and relaying full nodes. An unfortunate side effect of those rules
is that they can sometimes prevent someone from being able to use fee
bumping. Making it impossible or difficult to fee bump a transaction is
called _transaction pinning_.
One of the major denial of service concerns revolve's around the effect of
transaction relationships. Whenever the output of a transaction is
spent, that transaction's identifier (txid) is referenced by the child
transaction. However, when a transaction is replaced, the replacement
has a different txid. If that replacement transaction gets confirmed,
none of its descendants can be included in the same blockchain. It's
possible to recreate and re-sign the descendant transactions, but that's
not guaranteed to happen. This has related but divergent implications
for RBF and CPFP:
- In the context of RBF, when Bitcoin Core accepts a replacement
transaction, it keeps things simple by forgetting about the original
transaction and all descendant transactions which depended on that
original. To ensure that it's more profitable for miners to accept
replacements, Bitcoin Core only accepts a replacement transaction if it
pays more fees than all the transactions that will be forgotten.
+
The downside of this approach is that Alice can create a small
transaction that pays Bob. Bob can then use his output to create a
large child transaction. If Alice then wants to replace her original
transaction, she needs to pay a fee that's larger than what both she and
Bob originally paid. For example, if Alice's original transaction was
about 100 vbytes and Bob's transaction was about 100,000 vbytes, and
they both used the same fee rate, Alice now needs to pay more than 1,000
times as much as she originally paid in order to RBF fee bump her
transaction.
- In the context of CPFP, any time the node considers including a
package in a block, it must remove the transactions in that package
from any other package it wants to consider for the same block. For
example, if a child transaction pays for 25 ancestors, and each of
those ancestors has 25 other children, then including the package in
the block requires updating approximately 625 packages (25^2^).
Similarly, if a transaction with 25 descendants is removed from a
node's mempool (such as for being included in a block), and each of
those descendants has 25 other ancestors, another 625 packages need to
be updated. Each time we double our parameter (e.g. from 25 to 50),
we quadruple the amount of work our node needs to perform.
+
Additionally, a transaction and all of its descendants is not
useful to keep in a mempool long term if an alternative version of
that transaction is mined--none of those transactions can now be
confirmed unless there's a rare blockchain reorganization. Bitcoin
Core will remove from its mempool every transaction that can no longer
be confirmed on the current blockchain. At it's worst, that can
waste an enormous amount of your node's bandwidth and possibly be used
to prevent transactions from propagating correctly.
+
To prevent these problems, and other related
problems, Bitcoin Core limits a parent transaction to having a maximum
of 25 ancestors or descendants in its mempool and limits the
total size of all those transactions to 100,000 vbytes. The downside
of this approach is that users are prevented from creating CPFP fee
bumps if a transaction already has too many descendants (or if it and
its descendants are too large).
Transaction pinning can happen by accident, but it also represents a
serious vulnerability for multiparty time-sensitive protocols such as
Lightning Network. If your counterparty can prevent one of your
transactions from confirming by a deadline, they may be able to steal
money from you.
Protocol developers have been working on mitigating problems with
transaction pinning for several years. One partial solution is
described in <<cpfp_carve_out>>. Several other solutions have been
proposed, and at least one solution is being actively developed as of
this writing (ephemeral anchors, see
https://bitcoinops.org/en/topics/ephemeral-anchors/).
[[cpfp_carve_out]]
=== CPFP Carve Out and Anchor Outputs
In 2018, developers working on Lightning Network (LN) had a problem.
Their protocol uses transactions which require signatures from two
different parties. Neither party wants to trust the other, so they sign
transactions at a point in the protocol when trust isn't needed,
allowing either of them to broadcast one of those transactions at a
later time when the other party may not want to (or be able to) fulfill
its obligations. The problem with this approach is that the
transactions might need to be broadcast at an unknown time, far in the future, beyond any
reasonable ability to estimate an appropriate fee rate for the
transactions.
In theory, the developers could have designed their transactions to
allow fee bumping with either RBF (using special sighash flags) or CPFP,
but both of those protocols are vulnerable to transaction pinning.
Given that the involved transactions are time sensitive, allowing a
counterparty to use transaction pinning to delay confirmation of a
transaction can easily lead to a repeatable exploit that malicious
parties could use to steal money from honest parties.
LN developer Matt Corallo proposed a solution: give the rules for CPFP
fee bumping a special exception, called _CPFP carve out_. The normal
rules for CPFP forbid the inclusion of an additional descendant if it
would cause a parent transaction to have 26 or more descendants or if it
would cause a parent and all of its descendants to exceed 100,000 vbytes
in size. Under the rules of CPFP carve out, a single additional
transaction up to 1,000 vbytes in size can be added to a package even if
it would exceed the other limits as long as it is a direct child of an
unconfirmed transaction with no unconfirmed ancestors.
For example, Bob and Mallory both co-sign a transaction with two
outputs, one to each of them. Mallory broadcasts that transaction and
uses her output to attach either 25 child transactions or any smaller
number of child transactions equaling 100,000 vbytes in size. Without
carve-out, Bob would be unable to attach another child transaction to
his output for CPFP fee bumping. With carve-out, he can spend one of
the two outputs in the transaction, the one that belongs to him, as long
as his child transaction is less than 1,000 vbytes in size (which should
be more than enough space).
It's not allowed to use CPFP carve-out more than once, so it only works
for two-party protocols. There have been proposals to extend it to
protocols involving more participants, but there hasn't been much demand
for that and developers are focused on building more generic solutions
to transaction pinning attacks.
As of this writing, most popular LN implementations use a transaction
template called _anchor outputs_, which is designed to be used with CPFP
carve out.
=== Adding Fees to Transactions
The data structure of transactions does not have a field for fees.
Instead, fees are implied as the difference between the sum of inputs
and the sum of outputs. Any excess amount that remains after all outputs
have been deducted from all inputs is the fee that is collected by the
miners:
[[tx_fee_equation]]
.Transaction fees are implied, as the excess of inputs minus outputs:
----
Fees = Sum(Inputs) – Sum(Outputs)
----
This is a somewhat confusing element of transactions and an important
point to understand, because if you are constructing your own
transactions you must ensure you do not inadvertently include a very
large fee by underspending the inputs. That means that you must account
for all inputs, if necessary by creating change, or you will end up
giving the miners a very big tip!
For example, if you spend a 20-bitcoin UTXO to make a 1-bitcoin
payment, you must include a 19-bitcoin change output back to your
wallet. Otherwise, the 19-bitcoin "leftover" will be counted as a
transaction fee and will be collected by the miner who mines your
transaction in a block. Although you will receive priority processing
and make a miner very happy, this is probably not what you intended.
[WARNING]
====
If you forget to add a
change output in a manually constructed transaction, you will be paying
the change as a transaction fee. "Keep the change!" might not be what
you intended.
====
[[fee_sniping]]
=== Timelock Defense Against Fee Sniping
Fee-sniping is a theoretical
attack scenario, where miners attempting to rewrite past blocks "snipe"
higher-fee transactions from future blocks to maximize their
profitability.
For example, let's say the highest block in existence is block
#100,000. If instead of attempting to mine block #100,001 to extend the
chain, some miners attempt to remine #100,000. These miners can choose
to include any valid transaction (that hasn't been mined yet) in their
candidate block #100,000. They don't have to remine the block with the
same transactions. In fact, they have the incentive to select the most
profitable (highest fee per kB) transactions to include in their block.
They can include any transactions that were in the "old" block
#100,000, as well as any transactions from the current mempool.
Essentially they have the option to pull transactions from the "present"
into the rewritten "past" when they re-create block #100,000.
Today, this attack is not very lucrative, because the block subsidy is much
higher than total fees per block. But at some point in the future,
transaction fees will be the majority of the reward (or even the
entirety of the reward). At that time, this scenario becomes inevitable.
Several wallets discourage fee sniping by creating transactions with a
lock time that limits those transactions to being included in the next
block or any later block. In our
scenario, our wallet would set lock time to 100,001 on any
transaction it created. Under normal circumstances, this lock time has
no effect&#x2014;the transactions could only be included in block
#100,001 anyway; it's the next block.
But under a reorganization attack, the miners would not be able to pull
high-fee transactions from the mempool, because all those transactions
would be timelocked to block #100,001. They can only remine #100,000
with whatever transactions were valid at that time, essentially gaining
no new fees.
This does not entirely prevent fee sniping, but it does make it less
profitable in some cases and so can help preserve the stability of the
Bitcoin network as the block subsidy declines. We recommend all wallets
implement anti fee sniping when it doesn't interfere with the wallet's
other uses of the lock time field.
As Bitcoin continues to mature, and as the subsidy continues to decline,
fees become more and more important to Bitcoin users, both in their
day-to-day use for getting transactions confirmed quickly and in
providing an incentive for miners to continue securing Bitcoin
transactions with new proof-of-work.

1214
ch10_network.adoc Normal file
View File

File diff suppressed because it is too large Load Diff

782
ch11_blockchain.adoc Normal file
View File

@ -0,0 +1,782 @@
[[blockchain]]
== The Blockchain
The blockchain is the history of every confirmed Bitcoin transaction.
It's what allows every full node to independently determine what keys and
scripts control which bitcoins. In this chapter, we'll look at the
structure of the blockchain and see how it uses cryptographic
commitments and other clever tricks to make every part of it easy for
full nodes (and sometimes light clients) to validate.
The blockchain data structure is
an ordered, back-linked list of blocks of transactions. The blockchain
can be stored as a flat file, or in a simple database.
Blocks are linked "back," each referring to the previous block in the
chain. The blockchain is often visualized
as a vertical stack, with blocks layered on top of each other and the
first block serving as the foundation of the stack. The visualization of
blocks stacked on top of each other results in the use of terms such as
"height" to refer to the distance from the first block, and "top" or
"tip" to refer to the most recently added block.
Each block
within the blockchain is identified by a hash, generated using the
SHA256 cryptographic hash algorithm on the header of the block. Each
block also commits to the previous block, known as the _parent_ block,
through the "previous block hash" field in the block header.
The sequence of hashes linking each block to its parent creates a chain
going back all the way to the first block ever created, known as the
_genesis block_.
Although a block has just one parent, it can have multiple
children. Each of the children commits to the same parent block.
Multiple children arise during a blockchain "fork," a temporary
situation that can occur when different blocks are discovered almost
simultaneously by different miners (see <<forks>>). Eventually, only one
child block becomes part of the blockchain accepted by all full nodes and the "fork" is resolved.
The "previous block hash" field is inside the block header and thereby
affects the _current_ block's hash.
Any change to a parent block
requires a child block's hash to change, which requires a change in the
pointer of the grandchild, which in turn changes the grandchild, and so
on. This sequence ensures that, once a block has many generations
following it, it cannot be changed without forcing a recalculation of
all subsequent blocks. Because such a recalculation would require
enormous computation (and therefore energy consumption), the existence
of a long chain of blocks makes the blockchain's deep history impractical to change,
which is a key feature of Bitcoin's security.
One way to think about the blockchain is like layers in a geological
formation, or glacier core sample. The surface layers might change with
the seasons, or even be blown away before they have time to settle. But
once you go a few inches deep, geological layers become more and more
stable. By the time you look a few hundred feet down, you are looking at
a snapshot of the past that has remained undisturbed for millions of
years. In the blockchain, the most recent few blocks might be revised if
there is a chain reorganization due to a fork. The top six blocks are
like a few inches of topsoil. But once you go more deeply into the
blockchain, beyond six blocks, blocks are less and less likely to
change. After 100 blocks back there is so much stability that
the coinbase transaction--the transaction containing the reward in
bitcoin for creating a new block--can be spent.
While the
protocol always allows a chain to be undone by a longer chain and while
the possibility of any block being reversed always exists, the
probability of such an event decreases as time passes until it becomes
infinitesimal.
=== Structure of a Block
A block is a container data structure that aggregates
transactions for inclusion in the blockchain. The
block is made of a header, containing metadata, followed by a long list
of transactions that make up the bulk of its size. The block header is
80 bytes, whereas the total size of all transactions in a block can be
up to about 4,000,000 bytes. A complete block,
with all transactions, can therefore be almost 50,000 times larger than the block
header. <<block_structure1>> describes how Bitcoin Core stores the structure of a block.
[[block_structure1]]
[role="pagebreak-before"]
.The structure of a block
[options="header"]
|=======
|Size| Field | Description
| 4 bytes | Block Size | The size of the block, in bytes, following this field
| 80 bytes | Block Header | Several fields form the block header
| 1-3 bytes (compactSize) | Transaction Counter | How many transactions follow
| Variable | Transactions | The transactions recorded in this block
|=======
[[block_header]]
=== Block Header
The block header consists of
block metadata as shown in <<block_header_structure_ch09>>.
[[block_header_structure_ch09]]
.The structure of the block header
[options="header"]
|=======
|Size| Field | Description
| 4 bytes | Version | Originally a version field; its use has evolved over time
| 32 bytes | Previous Block Hash | A hash of the previous (parent) block in the chain
| 32 bytes | Merkle Root | The root hash of the merkle tree of this block's transactions
| 4 bytes | Timestamp | The approximate creation time of this block (Unix epoch time)
| 4 bytes | Target | A compact encoding of the Proof-of-Work target for this block
| 4 bytes | Nonce | Arbitrary data used for the Proof-of-Work algorithm
|=======
The nonce, target, and timestamp are used in the mining
process and will be discussed in more detail in <<mining>>.
[[block_hash]]
=== Block Identifiers: Block Header Hash and Block Height
The primary identifier of a block
is its cryptographic hash, a commitment made by hashing the
block header twice through the SHA256 algorithm. The resulting 32-byte
hash is called the _block hash_ but is more accurately the _block header
hash_, pass:[<span role="keep-together">because only the block header is
used to compute it. For example,</span>]
+000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+ is
the block hash of the first block on Bitcoin's blockchain. The block hash
identifies a block uniquely and unambiguously and can be independently
derived by any node by simply hashing the block header.
Note that the block hash is not actually included inside the block's
data structure.
Instead, the block's hash is computed by each node as the
block is received from the network. The block hash might be stored in a
separate database table as part of the block's metadata, to facilitate
indexing and faster retrieval of blocks from disk.
A second way to identify a block is by its position in the blockchain,
called the pass:[<span role="keep-together"><em>block height</em>. The
genesis block is at block height 0 (zero) and is the</span>]
pass:[<span role="keep-together">same block that was previously
referenced by the following block hash</span>]
+000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f+. A
block can thus be identified in two ways: by referencing the block hash
or by referencing the block height. Each subsequent block added "on top"
of that first block is one position "higher" in the blockchain, like
boxes stacked one on top of the other. The block height 800,000 was
reached during the writing of this book in mid-2023, meaning there were
800,000 blocks stacked on top of the first block created in January
2009.
Unlike the block hash, the block height is not a unique identifier.
Although a single block will always have a specific and invariant block
height, the reverse is not true—the block height does not always
identify a single block. Two or more blocks might have the same block
height, competing for the same position in the blockchain. This scenario
is discussed in detail in the section <<forks>>. In early blocks, the block height was
also not a part of the block's data structure; it was not stored within
the block. Each node dynamically identified a block's position (height)
in the blockchain when it was received from the Bitcoin network. A
later protocol change (BIP34) began including the block height in the
coinbase transaction, although its purpose was to ensure each block had
a different coinbase transaction. Nodes still need to dynamically
identify a block's height in order to validate the coinbase field. The
block height might also be stored as metadata in an indexed database
table for faster retrieval.
[TIP]
====
A block's _block hash_ always identifies a single block uniquely. A
block also always has a specific _block height_. However, it is not
always the case that a specific block height identifies a single
block. Rather, two or more blocks might compete for a single position in
the blockchain.
====
=== The Genesis Block
The first block in the blockchain is called the _genesis block_
and was created in 2009. It is the common ancestor of all the blocks in
the blockchain, meaning that if you start at any block and follow the
chain backward in time, you will eventually arrive at the genesis block.
Every node always starts with a blockchain of at least one block because
the genesis block is statically encoded within Bitcoin Core,
such that it cannot be altered. Every node always "knows" the
genesis block's hash and structure, the fixed time it was created, and
even the single transaction within. Thus, every node has the starting
point for the blockchain, a secure "root" from which to build a trusted
blockchain.
See the statically encoded genesis block inside the Bitcoin Core client,
in https://github.com/bitcoin/bitcoin/blob/3955c3940eff83518c186facfec6f50545b5aab5/src/chainparams.cpp#L123[_chainparams.cpp_].
The following identifier hash belongs to the genesis block:
----
000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
----
You can search for that block hash in almost any block explorer website, such
as _blockstream.info_, and you will find a page describing the contents
of this block, with a URL containing that hash:
https://blockstream.info/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
Using the Bitcoin Core reference client on the command line:
----
$ bitcoin-cli getblock 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
----
[source,json]
----
{
"hash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
"confirmations": 790496,
"height": 0,
"version": 1,
"versionHex": "00000001",
"merkleroot": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
"time": 1231006505,
"mediantime": 1231006505,
"nonce": 2083236893,
"bits": "1d00ffff",
"difficulty": 1,
"chainwork": "0000000000000000000000000000000000000000000000000000000100010001",
"nTx": 1,
"nextblockhash": "00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048",
"strippedsize": 285,
"size": 285,
"weight": 1140,
"tx": [
"4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b"
]
}
----
The genesis block contains a message within it. The coinbase
transaction input contains the text "The Times 03/Jan/2009 Chancellor on
brink of second bailout for banks." This message was intended to offer
proof of the earliest date this block could have been created, by referencing the
headline of the British newspaper _The Times_. It also serves as a
tongue-in-cheek reminder of the importance of an independent monetary
system, with Bitcoin's launch occurring at the same time as an
unprecedented worldwide monetary crisis. The message was embedded in the
first block by Satoshi Nakamoto, Bitcoin's creator.
=== Linking Blocks in the Blockchain
Bitcoin full nodes validate every
block in the blockchain after the genesis block. Their local view of
the blockchain is constantly updated as new blocks are found and used to
extend the chain. As a node receives incoming blocks from the network,
it will validate these blocks and then link them to its view of the existing
blockchain. To establish a link, a node will examine the incoming block
header and look for the "previous block hash."
Let's assume, for example, that a node has 277,314 blocks in the local
copy of the blockchain. The last block the node knows about is block
277,314, with a block header hash of:
----
00000000000000027e7ba6fe7bad39faf3b5a83daed765f05f7d1b71a1632249
----
The Bitcoin node then receives a new block from the network, which it
parses as follows:
[source,json]
----
{
"size" : 43560,
"version" : 2,
"previousblockhash" :
"00000000000000027e7ba6fe7bad39faf3b5a83daed765f05f7d1b71a1632249",
"merkleroot" :
"5e049f4030e0ab2debb92378f53c0a6e09548aea083f3ab25e1d94ea1155e29d",
"time" : 1388185038,
"difficulty" : 1180923195.25802612,
"nonce" : 4215469401,
"tx" : [
"257e7497fb8bc68421eb2c7b699dbab234831600e7352f0d9e6522c7cf3f6c77",
"[... many more transactions omitted ...]",
"05cfd38f6ae6aa83674cc99e4d75a1458c165b7ab84725eda41d018a09176634"
]
}
----
Looking at this new block, the node finds the +previousblockhash+ field,
which contains the hash of its parent block. It is a hash known to the
node, that of the last block on the chain at height 277,314. Therefore,
this new block is a child of the last block on the chain and extends the
existing blockchain. The node adds this new block to the end of the
chain, making the blockchain longer with a new height of 277,315.
<<chain_of_blocks>> shows the chain of three blocks, linked by
references in the +previousblockhash+ field.
[[chain_of_blocks]]
[role="smallerfourtyfive"]
.Blocks linked in a chain by each referencing the previous block header hash
image::images/mbc2_0901.png[]
[[merkle_trees]]
=== Merkle Trees
Each block in the Bitcoin blockchain contains
a summary of all the transactions in the block using a _merkle tree_.
A _merkle tree_, also known
as a _binary hash tree_, is a data structure used for efficiently
summarizing and verifying the integrity of large sets of data. Merkle
trees are binary trees containing cryptographic hashes. The term "tree"
is used in computer science to describe a branching data structure, but
these trees are usually displayed upside down with the "root" at the top
and the "leaves" at the bottom of a diagram, as you will see in the
examples that follow.
Merkle trees are used in bitcoin to summarize all the transactions in a
block, producing an overall commitment to the entire set of
transactions and permitting a very efficient process to verify whether a
transaction is included in a block. A merkle tree is constructed by
recursively hashing pairs of elements until there is only one hash, called
the _root_, or _merkle root_. The cryptographic hash algorithm used in
Bitcoin's merkle trees is SHA256 applied twice, also known as
double-SHA256.
When N data elements are hashed and summarized in a merkle tree, you can
check to see if any one data element is included in the tree with at
about +log~2~(N)+ calculations, making this a very efficient data
structure.
The merkle tree is constructed bottom-up. In the following example, we
start with four transactions, A, B, C, and D, which form the _leaves_ of
the merkle tree, as shown in <<simple_merkle>>. The transactions are not
stored in the merkle tree; rather, their data is hashed and the
resulting hash is stored in each leaf node as H~A~, H~B~, H~C~, and
H~D~:
++++
<pre data-type="codelisting">
H<sub>A</sub> = SHA256(SHA256(Transaction A))
</pre>
++++
Consecutive pairs of leaf nodes are then summarized in a parent node, by
concatenating the two hashes and hashing them together. For example, to
construct the parent node H~AB~, the two 32-byte hashes of the children
are concatenated to create a 64-byte string. That string is then
double-hashed to produce the parent node's hash:
++++
<pre data-type="codelisting">
H<sub>AB</sub> = SHA256(SHA256(H<sub>A</sub> || H<sub>B</sub>))
</pre>
++++
The process continues until there is only one node at the top, the node
known as the merkle root. That 32-byte hash is stored in the block
header and summarizes all the data in all four transactions.
<<simple_merkle>> shows how the root is calculated by pair-wise hashes
of the nodes.
[[simple_merkle]]
.Calculating the nodes in a merkle tree
image::images/mbc2_0902.png["merkle_tree"]
Because the merkle tree is a binary tree, it needs
an even number of leaf nodes. If there is an odd number of transactions
to summarize, the last transaction hash will be duplicated to create an
even number of leaf nodes, also known as a _balanced tree_. This is
shown in <<merkle_tree_odd>>, where transaction C is duplicated.
Similarly, if there are an odd number of hashes to process at any level,
the last hash is duplicated.
[[merkle_tree_odd]]
.Duplicating one data element achieves an even number of data elements
image::images/mbc2_0903.png["merkle_tree_odd"]
.A design flaw in Bitcoin's merkle tree
****
An extended comment in Bitcoin Core's source code describes a
significant problems in the design of Bitcoin's duplication of odd
elements in its merkle tree:
[quote,Bitcoin Core src/consensus/merkle.cpp]
____
WARNING! If you're reading this because you're learning about crypto
and/or designing a new system that will use merkle trees, keep in mind
that the following merkle tree algorithm has a serious flaw related to
duplicate txids, resulting in a vulnerability (CVE-2012-2459).
The reason is that if the number of hashes in the list at a given level
is odd, the last one is duplicated before computing the next level (which
is unusual in Merkle trees). This results in certain sequences of
transactions leading to the same merkle root. For example, these two
trees:
[[cve_tree]]
.Two Bitcoin-style merkle tree with the same root but a different number of leaves
image::images/cve-2012-2459.dot.png["Two Bitcoin-style merkle tree with the same root but a different number of leaves"]
For transaction lists [1,2,3,4,5,6] and [1,2,3,4,5,6,5,6] (where 5 and
6 are repeated) result in the same root hash A (because the hash of both
of (F) and (F,F) is C).
The vulnerability results from being able to send a block with such a
transaction list, with the same merkle root, and the same block hash as
the original without duplication, resulting in failed validation. If the
receiving node proceeds to mark that block as permanently invalid
however, it will fail to accept further unmodified (and thus potentially
valid) versions of the same block. We defend against this by detecting
the case where we would hash two identical hashes at the end of the list
together, and treating that identically to the block having an invalid
merkle root. Assuming no double-SHA256 collisions, this will detect all
known ways of changing the transactions without affecting the merkle
root.
____
****
The same method for constructing a tree from four transactions can be
generalized to construct trees of any size. In Bitcoin it is common to
have several thousand transactions in a single
block, which are summarized in exactly the same way, producing just 32
bytes of data as the single merkle root. In <<merkle_tree_large>>, you
will see a tree built from 16 transactions. Note that although the root
looks bigger than the leaf nodes in the diagram, it is the exact same
size, just 32 bytes. Whether there is one transaction or ten
thousand transactions in the block, the merkle root always summarizes
them into 32 bytes.
To prove that a specific transaction is
included in a block, a node only needs to produce approximately +log~2~(N)+ 32-byte
hashes, constituting an _authentication path_ or _merkle path_
connecting the specific transaction to the root of the tree. This is
especially important as the number of transactions increases, because
the base-2 logarithm of the number of transactions increases much more
slowly. This allows Bitcoin nodes to efficiently produce paths of 10 or
12 hashes (320–384 bytes), which can provide proof of a single
transaction out of more than a thousand transactions in a multi-megabyte
block.
[[merkle_tree_large]]
.A merkle tree summarizing many data elements
image::images/mbc2_0904.png["merkle_tree_large"]
In <<merkle_tree_path>>, a node can prove that a transaction K is
included in the block by producing a merkle path that is only four
32-byte hashes long (128 bytes total). The path consists of the four
hashes (shown with a shaded background) H~L~,
H~IJ~, H~MNOP~, and H~ABCDEFGH~. With those four hashes provided as an
authentication path, any node can prove that H~K~ (with a black
background at the bottom of the diagram) is included in the merkle root
by computing four additional pair-wise hashes H~KL~, H~IJKL~,
H~IJKLMNOP~, and the merkle tree root (outlined in a dashed line in the
diagram).
[[merkle_tree_path]]
.A merkle path used to prove inclusion of a data element
image::images/mbc2_0905.png["merkle_tree_path"]
The efficiency of merkle trees becomes obvious as the scale increases.
The largest possible block can hold almost 16,000 transactions in 4,000,000
bytes, but proving any particular one of those sixteen thousand transactions
is a part of that block only requires a copy of the transaction, a copy
of the 80-byte block header, and 448 bytes for the merkle proof. That
makes the largest possible proof almost 10,000 times smaller than the
largest possible Bitcoin block.
=== Merkle Trees and Lightweight Clients
Merkle trees are used extensively by lightweight clients. Lightweight clients don't
have all transactions and do not download full blocks, just block
headers. In order to verify that a transaction is included in a block,
without having to download all the transactions in the block, they use
a merkle path.
Consider, for example, a lightweight client that is interested in incoming
payments to an address contained in its wallet. The lightweight client will
establish a bloom filter (see <<bloom_filters>>) on its connections to
peers to limit the transactions received to only those containing
addresses of interest. When a peer sees a transaction that matches the
bloom filter, it will send that block using a +merkleblock+ message. The
+merkleblock+ message contains the block header as well as a merkle path
that links the transaction of interest to the merkle root in the block.
The lightweight client can use this merkle path to connect the transaction to the
block header and verify that the transaction is included in the block. The lightweight
client also uses the block header to link the block to the rest of the
blockchain. The combination of these two links, between the transaction
and block, and between the block and blockchain, proves that the
transaction is recorded in the blockchain. All in all, the lightweight client will
have received less than a kilobyte of data for the block header and
merkle path, an amount of data that is more than a thousand times less
than a full block (about 2 megabytes currently).
=== Bitcoin's Test Blockchains
You might be
surprised to learn that there is more than one blockchain used with Bitcoin. The
"main" Bitcoin blockchain, the one created by Satoshi Nakamoto on
January 3rd, 2009, the one with the genesis block we studied in this
chapter, is called _mainnet_. There are other Bitcoin blockchains that
are used for testing purposes: at this time _testnet_, _signet_, and
_regtest_. Let's look at each in turn.
==== Testnet: Bitcoin's Testing Playground
Testnet is the name of the test blockchain, network, and currency that
is used for testing purposes. The testnet is a fully featured live P2P
network, with wallets, test bitcoins (testnet coins), mining, and all
the other features of mainnet. The most important difference is that
testnet coins are meant to be worthless.
Any software development that is intended for production use on
Bitcoin's mainnet can first be tested on testnet with test coins.
This protects both the developers from monetary losses due to bugs and
the network from unintended behavior due to bugs.
The current testnet is called _testnet3_, the third iteration of
testnet, restarted in February 2011 to reset the difficulty from the
previous testnet. Testnet3 is a large blockchain, in excess of 30 GB in
2023. It will take a while to sync fully and use up resources
on your computer. Not as much as mainnet, but not exactly "lightweight"
either.
[TIP]
====
Testnet and the other test blockchains described in this book don't use
the same address prefixes as mainnet addresses to prevent someone from
accidentally sending real bitcoins to a test address. Mainnet addresses
begin with +1+, +3+, or +bc1+. Addresses for the test networks
mentioned in this book begin with +m+, +n+, or +tb1+. Other test
networks, or new protocols being developed on test networks, may use
other address prefixes or alterations.
====
===== Using testnet
Bitcoin Core, like many other Bitcoin programs, has full support
for operation on testnet as an alternative mainnet. All of Bitcoin Core's
functions work on testnet, including the wallet, mining testnet coins,
and syncing a full testnet node.
To start Bitcoin Core on testnet instead of mainnet you use the
+testnet+ switch:
----
$ bitcoind -testnet
----
In the logs you should see that bitcoind is building a new blockchain in
the +testnet3+ subdirectory of the default bitcoind directory:
----
bitcoind: Using data directory /home/username/.bitcoin/testnet3
----
To connect to bitcoind, you use the +bitcoin-cli+ command-line tool, but
you must also switch it to testnet mode:
----
$ bitcoin-cli -testnet getblockchaininfo
{
"chain": "test",
"blocks": 1088,
"headers": 139999,
"bestblockhash": "0000000063d29909d475a1c4ba26da64b368e56cce5d925097bf3a2084370128",
"difficulty": 1,
"mediantime": 1337966158,
"verificationprogress": 0.001644065914099759,
"chainwork": "0000000000000000000000000000000000000000000000000000044104410441",
"pruned": false,
"softforks": [
[...]
----
You can also run on testnet3 with other full-node implementations, such
as +btcd+ (written in Go) and +bcoin+ (written in JavaScript), to
experiment and learn in other programming languages and frameworks.
Testnet3 supports all the features of mainnet, including
Segregated Witness v0 and v1 (see <<segwit>> and <<taproot>>). Therefore, testnet3 can also be
used to test Segregated Witness features.
===== Problems With Testnet
Testnet doesn't just use the same data structures as Bitcoin, it also
uses almost exactly the same Proof-of-Work (PoW) security mechanism as
Bitcoin. The notable differences for testnet are that it's minimum
difficulty is half that of Bitcoin and that it's allowed to include a
block at the minimum difficulty if that block's timestamp is more than
20 minutes after the previous block.
Unfortunately, Bitcoin's PoW security mechanism was designed to depend
on economic incentives--incentives which don't exist in a test
blockchain that is forbidden from having value. On mainnet, miners are
incentivized to include user transactions in their blocks because those
transactions pay fees. On testnet, transactions still contain something
called fees, but those fees don't have any economic value. That means
the only incentive for a testnet miner to include transactions is
because they want to help users and developers to test their software.
Alas, people who like to disrupt systems often feel a stronger
incentive, at least in the short term. Because PoW mining is designed
to be permissionless, anyone can mine, whether their intention is good
or not. That means disruptive miners can create many blocks in a row on
testnet without including any user transactions. When those attacks
happen, testnet becomes unusable for users and developers.
==== Signet: The Proof of Authority Testnet
There's no known way for a system dependent on permissionless PoW to
provide a highly usable blockchain without introducing economic
incentives, so Bitcoin protocol developers began considering
alternatives. The primary goal was to preserve as much of the structure of
Bitcoin as possible so that software could run on a testnet with minimal
changes--but to also provide an environment that would remain useful.
A secondary goal was to produce a reusable design that would allow
developers of new software to easily create their own test networks.
The solution implemented in Bitcoin Core and other software is called
_signet_, as defined by BIP325. A signet is a test network where each
block must contain proof (such as a signature) that the creation of that
block was sanctioned by a trusted authority.
Whereas mining in Bitcoin is permissionless--anyone can do it--mining on
signet is fully permissioned. Only those with permission can do it.
This would be a completely unacceptable change to Bitcoin's mainnet--no
one would use that software--but it's reasonable on a testnet where coins have
no value and the only purpose is testing software and systems.
BIP325 signets are designed to make it very easy to create your own. If
you disagree with how someone else is running their signet, you can
start your own signet and connect your software to it.
===== The Default Signet and Custom Signets
Bitcoin Core supports a default signet, which we believe to be the most
widely used signet at the time of writing. It is currently operated by
two contributors to that project. If you start Bitcoin Core with the
+-signet+ parameter and no other signet-related parameters, this is the
signet you will be using.
As of this writing, the default signet has about 150,000 blocks and is
about a gigabyte in size. It supports all of the same features as
Bitcoin's mainnet and is also used for testing proposed upgrades through
the Bitcoin Inquisition project, which is a software fork of Bitcoin
Core that's only designed to run on signet.
If you want to use a different signet, called a _custom signet_, you
will need to know the script used to determine when a block is
authorized, called the _challenge_ script. This is a standard Bitcoin
script, so it can use features such as multisig to allow multiple people
to authorize blocks. You may also need to connect to a seed node that
will provide you with the addresses of peers on the custom signet. For
example:
----
bitcoind -signet -signetchallenge=0123...cdef -signetseednode=example.com:1234
----
As of this writing, we generally recommend that the public testing of
mining software occur on testnet3 and that all other public testing of
Bitcoin software occur on the default signet.
To interact with your chosen signet, you can use the +-signet+ parameter
with +bitcoin-cli+, similar to how you used testnet. For example:
----
$ bitcoin-cli -signet getblockchaininfo
{
"chain": "signet",
"blocks": 143619,
"headers": 143619,
"bestblockhash": "000000c46cb3505ddd29653720686b6a3565ad1c5768e2908439382447572a93",
"difficulty": 0.003020638517858618,
"time": 1684530244,
"mediantime": 1684526116,
"verificationprogress": 0.999997961940662,
"initialblockdownload": false,
"chainwork": "0000000000000000000000000000000000000000000000000000019ab37d2194",
"size_on_disk": 769525915,
"pruned": false,
"warnings": ""
}
----
==== Regtest&#x2014;The Local Blockchain
Regtest, which stands for
"Regression Testing," is a Bitcoin Core feature that allows you to
create a local blockchain for testing purposes. Unlike signet and testnet3, which
are a public and shared test blockchain, the regtest blockchains are
intended to be run as closed systems for local testing. You launch a
regtest blockchain from scratch. You may
add other nodes to the network, or run it with a single node only to
test the Bitcoin Core software.
To start Bitcoin Core in regtest mode, you use the +regtest+ flag:
----
$ bitcoind -regtest
----
Just like with testnet, Bitcoin Core will initialize a new blockchain
under the _regtest_ subdirectory of your bitcoind default directory:
----
bitcoind: Using data directory /home/username/.bitcoin/regtest
----
To use the command-line tool, you need to specify the +regtest+ flag
too. Let's try the +getblockchaininfo+ command to inspect the regtest
blockchain:
----
$ bitcoin-cli -regtest getblockchaininfo
{
"chain": "regtest",
"blocks": 0,
"headers": 0,
"bestblockhash": "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
"difficulty": 4.656542373906925e-10,
"mediantime": 1296688602,
"verificationprogress": 1,
"chainwork": "0000000000000000000000000000000000000000000000000000000000000002",
"pruned": false,
[...]
----
As you can see, there are no blocks yet. Let's create a default wallet,
get an address, and then mine some (500 blocks) to earn the reward:
----
$ bitcoin-cli -regtest createwallet ""
$ bitcoin-cli -regtest getnewaddress
bcrt1qwvfhw8pf79kw6tvpmtxyxwcfnd2t4e8v6qfv4a
$ bitcoin-cli -regtest generatetoaddress 500 bcrt1qwvfhw8pf79kw6tvpmtxyxwcfnd2t4e8v6qfv4a
[
"3153518205e4630d2800a4cb65b9d2691ac68eea99afa7fd36289cb266b9c2c0",
"621330dd5bdabcc03582b0e49993702a8d4c41df60f729cc81d94b6e3a5b1556",
"32d3d83538ba128be3ba7f9dbb8d1ef03e1b536f65e8701893f70dcc1fe2dbf2",
...,
"32d55180d010ffebabf1c3231e1666e9eeed02c905195f2568c987c2751623c7"
]
----
It will only take a few seconds to mine all these blocks, which
certainly makes it easy for testing. If you check your wallet balance,
you will see that you earned reward for the first 400 blocks (coinbase
rewards must be 100 blocks deep before you can spend them):
----
$ bitcoin-cli -regtest getbalance
12462.50000000
----
=== Using Test Blockchains for Development
Bitcoin's various
blockchains (+regtest+, +signet+, +testnet3+, +mainnet+) offer a range
of testing environments for bitcoin development. Use the test
blockchains whether you are developing for Bitcoin Core, or another
full-node consensus client; an application such as a wallet, exchange,
ecommerce site; or even developing novel smart contracts and complex
scripts.
You can use the test blockchains to establish a development pipeline.
Test your code locally on a +regtest+ as you develop it. Once you are
ready to try it on a public network, switch to +signet+ or +testnet+ to expose your
code to a more dynamic environment with more diversity of code and
applications. Finally, once you are confident your code works as
expected, switch to +mainnet+ to deploy it in production. As you make
changes, improvements, bug fixes, etc., start the pipeline again,
deploying each change first on +regtest+, then on +signet+ or +testnet+, and finally
into production.
Now that we know what data the blockchain contains and how cryptographic
commitments securely tie the various parts together, we will look at the
specical commitment that both provides computational security and
ensures no block can be changed without invalidating all other blocks
built on top of it: Bitcoin's mining function.

2087
ch12_mining.adoc Normal file
View File

File diff suppressed because it is too large Load Diff

291
ch13_security.adoc Normal file
View File

@ -0,0 +1,291 @@
[[ch11]]
== Bitcoin Security
Securing your bitcoins is challenging because bitcoins are
are not like a balance in a bank account. Your bitcoins are very
much like digital cash or gold. You've probably heard the expression,
"Possession is nine-tenths of the law." Well, in Bitcoin, possession is
ten-tenths of the law. Possession of the keys to spend certain bitcoins is
equivalent to possession of cash or a chunk of precious metal. You can
lose it, misplace it, have it stolen, or accidentally give the wrong
amount to someone. In every one of these cases, users have no recourse
within the protocol, just as if they dropped cash on a public sidewalk.
However, the Bitcoin system has capabilities that cash, gold, and bank accounts do
not. A Bitcoin wallet, containing your keys, can be backed up like any
file. It can be stored in multiple copies, even printed on paper for
hard-copy backup. You can't "back up" cash, gold, or bank accounts.
Bitcoin is different enough from anything that has come before that we
need to think about securing your bitcoins in a novel way too.
=== Security Principles
The core principle in Bitcoin is
decentralization and it has important implications for security. A
centralized model, such as a traditional bank or payment network,
depends on access control and vetting to keep bad actors out of the
system. By comparison, a decentralized system like Bitcoin pushes the
responsibility and control to the users. Because the security of the network
is based on independent verification, the network can be open
and no encryption is required for Bitcoin traffic (although encryption
can still be useful).
On a traditional payment network, such as a credit card system, the
payment is open-ended because it contains the user's private identifier
(the credit card number). After the initial charge, anyone with access
to the identifier can "pull" funds and charge the owner again and again.
Thus, the payment network has to be secured end-to-end with encryption
and must ensure that no eavesdroppers or intermediaries can compromise
the payment traffic, in transit or when it is stored (at rest). If a bad
actor gains access to the system, he can compromise current transactions
_and_ payment tokens that can be used to create new transactions. Worse,
when customer data is compromised, the customers are exposed to identity
theft and must take action to prevent fraudulent use of the compromised
accounts.
Bitcoin is dramatically different. A Bitcoin transaction authorizes only
a specific value to a specific recipient and cannot be forged.
It does not reveal any private information, such as the
identities of the parties, and cannot be used to authorize additional
payments. Therefore, a Bitcoin payment network does not need to be
encrypted or protected from eavesdropping. In fact, you can broadcast
Bitcoin transactions over an open public channel, such as unsecured WiFi
or Bluetooth, with no loss of security.
Bitcoin's decentralized security model puts a lot of power in the hands
of the users. With that power comes responsibility for maintaining the
secrecy of their keys. For most users that is not easy to do, especially
on general-purpose computing devices such as internet-connected
smartphones or laptops. Although Bitcoin's decentralized model prevents
the type of mass compromise seen with credit cards, many users are not
able to adequately secure their keys and get hacked, one by one.
==== Developing Bitcoin Systems Securely
A critical principle
for Bitcoin developers is decentralization. Most developers will be
familiar with centralized security models and might be tempted to apply
these models to their Bitcoin applications, with disastrous results.
Bitcoin's security relies on decentralized control over keys and on
independent transaction validation by users. If you want to leverage
Bitcoin's security, you need to ensure that you remain within the
Bitcoin security model. In simple terms: don't take control of keys away
from users and don't outsource validation.
For example, many early Bitcoin exchanges concentrated all user funds in
a single "hot" wallet with keys stored on a single server. Such a design
removes control from users and centralizes control over keys in a single
system. Many such systems have been hacked, with disastrous consequences
for their customers.
Unless you are prepared to invest heavily in operational security,
multiple layers of access control, and audits (as the traditional banks
do) you should think very carefully before taking funds outside of
Bitcoin's decentralized security context. Even if you have the funds and
discipline to implement a robust security model, such a design merely
replicates the fragile model of traditional financial networks, plagued
by identity theft, corruption, and embezzlement. To take advantage of
Bitcoin's unique decentralized security model, you have to avoid the
temptation of centralized architectures that might feel familiar but
ultimately subvert Bitcoin's security.
==== The Root of Trust
Traditional security architecture is based
upon a concept called the _root of trust_, which is a trusted core used
as the foundation for the security of the overall system or application.
Security architecture is developed around the root of trust as a series
of concentric circles, like layers in an onion, extending trust outward
from the center. Each layer builds upon the more-trusted inner layer
using access controls, digital signatures, encryption, and other
security primitives. As software systems become more complex, they are
more likely to contain bugs, which make them vulnerable to security
compromise. As a result, the more complex a software system becomes, the
harder it is to secure. The root of trust concept ensures that most of
the trust is placed within the least complex part of the system, and
therefore the least vulnerable parts of the system, while more complex
software is layered around it. This security architecture is repeated at
different scales, first establishing a root of trust within the hardware
of a single system, then extending that root of trust through the
operating system to higher-level system services, and finally across
many servers layered in concentric circles of diminishing trust.
Bitcoin security
architecture is different. In Bitcoin, the consensus system creates a
trusted blockchain that is completely decentralized. A correctly
validated blockchain uses the genesis block as the root of trust,
building a chain of trust up to the current block. Bitcoin systems can
and should use the blockchain as their root of trust. When designing a
complex Bitcoin application that consists of services on many different
systems, you should carefully examine the security architecture in order
to ascertain where trust is being placed. Ultimately, the only thing
that should be explicitly trusted is a fully validated blockchain. If
your application explicitly or implicitly vests trust in anything but
the blockchain, that should be a source of concern because it introduces
vulnerability. A good method to evaluate the security architecture of
your application is to consider each individual component and evaluate a
hypothetical scenario where that component is completely compromised and
under the control of a malicious actor. Take each component of your
application, in turn, and assess the impacts on the overall security if
that component is compromised. If your application is no longer secure
when components are compromised, that shows you have misplaced trust in
those components. A Bitcoin application without vulnerabilities should
be vulnerable only to a compromise of the Bitcoin consensus mechanism,
meaning that its root of trust is based on the strongest part of the
Bitcoin security architecture.
The numerous examples of hacked Bitcoin exchanges serve to underscore
this point because their security architecture and design fails even
under the most casual scrutiny. These centralized implementations had
invested trust explicitly in numerous components outside the Bitcoin
blockchain, such as hot wallets, centralized databases,
vulnerable encryption keys, and similar schemes.
=== User Security Best Practices
Humans have
used physical security controls for thousands of years. By comparison,
our experience with digital security is less than 50 years old. Modern
general-purpose operating systems are not very secure and not
particularly suited to storing digital money. Our computers are
constantly exposed to external threats via always-on internet
connections. They run thousands of software components from hundreds of
authors, often with unconstrained access to the user's files. A single
piece of rogue software, among the many thousands installed on your
computer, can compromise your keyboard and files, stealing any bitcoins
stored in wallet applications. The level of computer maintenance
required to keep a computer virus-free and trojan-free is beyond the
skill level of all but a tiny minority of computer users.
Despite decades of research and advancements in information security,
digital assets are still woefully vulnerable to a determined adversary.
Even the most highly protected and restricted systems, in financial
services companies, intelligence agencies, and defense contractors, are
frequently breached. Bitcoin creates digital assets that have intrinsic
value and can be stolen and diverted to new owners instantly and
irrevocably. This creates a massive incentive for hackers. Until now,
hackers had to convert identity information or account tokens—such as
credit cards and bank accounts—into value after compromising them.
Despite the difficulty of fencing and laundering financial information,
we have seen ever-escalating thefts. Bitcoin escalates this problem
because it doesn't need to be fenced or laundered; bitcoins are valuable
by themselves.
Bitcoin also creates the incentives to improve computer
security. Whereas previously the risk of computer compromise was vague
and indirect, Bitcoin makes these risks clear and obvious. Holding
bitcoins on a computer serves to focus the user's mind on the need for
improved computer security. As a direct result of the proliferation and
increased adoption of Bitcoin and other digital currencies, we have seen
an escalation in both hacking techniques and security solutions. In
simple terms, hackers now have a very juicy target and users have a
clear incentive to defend themselves.
Over the past three years, as a direct result of Bitcoin adoption, we
have seen tremendous innovation in the realm of information security in
the form of hardware encryption, key storage and hardware signing devices,
multisignature technology, and digital escrow. In the following sections
we will examine various best practices for practical user security.
==== Physical Bitcoin Storage
Because most users are far more
comfortable with physical security than information security, a very
effective method for protecting bitcoins is to convert them into physical
form. Bitcoin keys, and the seeds used to create them, are nothing more than long numbers. This means that
they can be stored in a physical form, such as printed on paper or
etched on a metal plate. Securing the keys then becomes as simple as
physically securing a printed copy of the key seed. A seed
that is printed on paper is called a "paper backup," and
many wallets can create them.
Keeping bitcoins
offline is called _cold storage_ and it is one of the most effective
security techniques. A cold storage system is one where the keys are
generated on an offline system (one never connected to the internet) and
stored offline either on paper or on digital media, such as a USB memory
stick.
==== Hardware Signing Devices
In the long term, Bitcoin security may increasingly take the
form of tamper-proof hardware signing devices. Unlike a smartphone or desktop
computer, a Bitcoin hardware signing device only needs to hold keys and
use them to generate signatures. Without general-purpose software to
compromise and
with limited interfaces, hardware signing devices can deliver strong
security to nonexpert users. Hardware
signing devices may become the predominant method of storing bitcoins.
==== Ensuring Your Access
Although
most users are rightly concerned about theft of thir bitcoins, there is an even
bigger risk. Data files get lost all the time. If they contain Bitcoin keys,
the loss is much more painful. In the effort to secure their Bitcoin
wallets, users must be very careful not to go too far and end up losing
their bitcoins. In July 2011, a well-known Bitcoin awareness and education
project lost almost 7,000 bitcoin. In their effort to prevent theft, the
owners had implemented a complex series of encrypted backups. In the end
they accidentally lost the encryption keys, making the backups worthless
and losing a fortune. Like hiding money by burying it in the desert, if
you secure your bitcoins too well you might not be able to find them again.
[WARNING]
====
To spend bitcoins, you may need to backup more than just your private
keys or the BIP32 seed used to derive them. This is especially the case
when multisignatures or complex scripts are being used. Most output
scripts commit to the actual conditions that must be fulfilled to spend
the bitcoins in that output, and it's not possible to fulfill that
commitment unless your wallet software can reveal those conditions to
the network. Wallet recovery codes must include this information. For
more details, see <<ch05_wallets>>.
====
==== Diversifying Risk
Would you carry your entire net worth in cash in your wallet? Most
people would consider that reckless, yet Bitcoin users often keep all
their bitcoins using a single wallet application. Instead, users should spread the risk
among multiple and diverse Bitcoin applications. Prudent users will keep only
a small fraction, perhaps less than 5%, of their bitcoins in an online or
mobile wallet as "pocket change." The rest should be split between a few
different storage mechanisms, such as a desktop wallet and offline (cold
storage).
==== Multisig and Governance
Whenever a company or individual stores large amounts of
bitcoins, they should consider using a multisignature Bitcoin address.
Multisignature addresses secure funds by requiring more than one
signature to make a payment. The signing keys should be stored in a
number of different locations and under the control of different people.
In a corporate environment, for example, the keys should be generated
independently and held by several company executives, to ensure no
single person can compromise the funds. Multisignature addresses can
also offer redundancy, where a single person holds several keys that are
stored in different locations.
==== Survivability
One important
security consideration that is often overlooked is availability,
especially in the context of incapacity or death of the key holder.
Bitcoin users are told to use complex passwords and keep their keys
secure and private, not sharing them with anyone. Unfortunately, that
practice makes it almost impossible for the user's family to recover any
funds if the user is not available to unlock them. In most cases, in
fact, the families of Bitcoin users might be completely unaware of the
existence of the bitcoin funds.
If you have a lot of bitcoins, you should consider sharing access details
with a trusted relative or lawyer. A more complex survivability scheme
can be set up with multi-signature access and estate planning through a
lawyer specialized as a "digital asset executor."
Bitcoin is a completely new, unprecedented, and complex technology. Over
time we will develop better security tools and practices that are easier
to use by nonexperts. For now, Bitcoin users can use many of the tips
discussed here to enjoy a secure and trouble-free Bitcoin experience.

1250
ch14_applications.adoc Normal file
View File

File diff suppressed because it is too large Load Diff