From 311eac3b3413f9e67b6b85436fd96294c4a25106 Mon Sep 17 00:00:00 2001 From: Clare Laylock Date: Tue, 3 Oct 2023 09:25:59 -0400 Subject: [PATCH] part 3 CE --- ch08_signatures.adoc | 217 +++++++++++++++++++++---------------------- ch09_fees.adoc | 82 ++++++++-------- ch10_network.adoc | 120 ++++++++++++------------ ch11_blockchain.adoc | 46 ++++----- 4 files changed, 230 insertions(+), 235 deletions(-) diff --git a/ch08_signatures.adoc b/ch08_signatures.adoc index a098eb58..2c55a635 100644 --- a/ch08_signatures.adoc +++ b/ch08_signatures.adoc @@ -27,7 +27,7 @@ intact. Each transaction input and any signatures it may contain is _completely_ independent of any other input or signature. Multiple parties can collaborate to construct transactions and sign only one input each. -Several protocols uses this fact to create multi-party transactions for +Several protocols use this fact to create multiparty transactions for privacy. ==== @@ -44,7 +44,7 @@ signing key). The second part is an algorithm that allows anyone to verify the signature, given also the message and the corresponding public key. -==== Creating a digital signature +==== Creating a Digital Signature In Bitcoin's use of digital signature algorithms, the "message" being signed is the transaction, or more accurately a hash of a specific @@ -62,7 +62,7 @@ where: * _F_~_sig_~ is the signing algorithm * _Sig_ is the resulting signature -More details on the mathematics of schnorr and ECDSA signatures can be found in <> +You can find more details on the mathematics of schnorr and ECDSA signatures in <> and <>. In both schnorr and ECDSA signatures, the function _F_~_sig_~ produces a signature +Sig+ that is composed of @@ -70,7 +70,7 @@ two values. There are differences between the two values in the different algorithms, which we'll explore later. After the two values -are calculated, they are serialized into a byte-stream. For ECDSA +are calculated, they are serialized into a byte stream. For ECDSA signatures, the encoding uses an international standard encoding scheme called the _Distinguished Encoding Rules_, or _DER_. For schnorr signatures, a @@ -78,10 +78,7 @@ simpler serialization format is used. ==== Verifying the Signature - The signature verification -algorithm takes the message (a hash of parts of the transaction and -related data), the signer's public key and the signature, and returns -TRUE if the signature is valid for this message and public key. +The signature verification algorithm takes the message (a hash of parts of the transaction and related data), the signer's public key and the signature, and returns ++TRUE++ if the signature is valid for this message and public key. To verify the signature, one must have the signature, the serialized transaction, some data about the output being spent, and the public key @@ -127,16 +124,16 @@ There are three +SIGHASH+ flags: +ALL+, +NONE+, and +SINGLE+, as shown in <>. [[sighash_types_and_their]] -.SIGHASH types and their meanings +.++SIGHASH++ types and their meanings [options="header"] |======================= -|+SIGHASH+ flag| Value | Description -| +ALL+ | 0x01 | Signature applies to all inputs and outputs -| +NONE+ | 0x02 | Signature applies to all inputs, none of the outputs -| +SINGLE+ | 0x03 | Signature applies to all inputs but only the one output with the same index number as the signed input +|++SIGHASH++ flag| Value | Description +| ++ALL++ | ++0x01++ | Signature applies to all inputs and outputs +| ++NONE++ | ++0x02++ | Signature applies to all inputs, none of the outputs +| ++SINGLE++ | ++0x03++ | Signature applies to all inputs but only the one output with the same index number as the signed input |======================= -In addition, there is a modifier flag +SIGHASH_ANYONECANPAY+, which can +In addition, there is a modifier flag, +SIGHASH_ANYONECANPAY+, which can be combined with each of the preceding flags. When +ANYONECANPAY+ is set, only one input is signed, leaving the rest (and their sequence numbers) open for modification. The +ANYONECANPAY+ has the value +0x80+ @@ -144,13 +141,13 @@ and is applied by bitwise OR, resulting in the combined flags as shown in <>. [[sighash_types_with_modifiers]] -.SIGHASH types with modifiers and their meanings +.++SIGHASH++ types with modifiers and their meanings [options="header"] |======================= -|SIGHASH flag| Value | Description -| ALL\|ANYONECANPAY | 0x81 | Signature applies to one input and all outputs -| NONE\|ANYONECANPAY | 0x82 | Signature applies to one input, none of the outputs -| SINGLE\|ANYONECANPAY | 0x83 | Signature applies to one input and the output with the same index number +|++SIGHASH++ flag| Value | Description +| ++ALL\++|++ANYONECANPAY++ | ++0x81++ | Signature applies to one input and all outputs +| ++NONE\++|++ANYONECANPAY++ | ++0x82++ | Signature applies to one input, none of the outputs +| ++SINGLE\++|++ANYONECANPAY++ | ++0x83++ | Signature applies to one input and the output with the same index number |======================= The way +SIGHASH+ flags are applied during signing and verification is @@ -195,12 +192,12 @@ By itself, this allows any miner to change the output destination and claim the funds for themselves, but if other required signatures in the transaction use +SIGHASH_ALL+ or another type that commits to the output, it allows those spenders to change the -destination without allowing any third-parties (like miners) to modify +destination without allowing any third parties (like miners) to modify the outputs. +NONE|ANYONECANPAY+ :: This construction can be used to build a "dust collector." Users who have tiny UTXOs in their wallets can't spend these -without the cost in fees exceeding the value of the UTXO, see +without the cost in fees exceeding the value of the UTXO; see <>. With this type of signature, the uneconomical UTXOs can be donated for anyone to aggregate and spend whenever they want. @@ -212,23 +209,22 @@ new sighash flags. A signature using +SIGHASH_ANYPREVOUT+ would not commit to an input's outpoint field, allowing it to be used to spend any previous output for a particular witness program. For example, if Alice receives two outputs for the same amount to the same witness program -(e.g. requiring a single signature from her wallet), a +(e.g., requiring a single signature from her wallet), a +SIGHASH_ANYPREVOUT+ signature for spending either one of those outputs could be copied and used to spend the other output to the same destination. A signature using +SIGHASH_ANYPREVOUTANYSCRIPT+ would not commit to the outpoint, the amount, the witness program, or the -specific leaf in the taproot merkle tree (script tree), allowing it to spend any previous output -which the signature could satisfy. For example, if Alice received two -outputs for different amounts and different witness programs (e.g. one +specific leaf in the taproot merkle tree (script tree), allowing it to spend any previous output that the signature could satisfy. For example, if Alice received two +outputs for different amounts and different witness programs (e.g., one requiring a single signature and another requiring her signature plus some other data), a +SIGHASH_ANYPREVOUTANYSCRIPT+ signature for spending either one of those outputs could be copied and used to spend the other output to the same destination (assuming the extra data for the second output was known). -The main expected use for the two SIGHASH_ANYPREVOUT opcodes is improved +The main expected use for the two ++SIGHASH_ANYPREVOUT++ opcodes is improved payment channels, such as those used in the Lightning Network, although several other uses have been described. @@ -236,27 +232,27 @@ several other uses have been described. ==== You will not often see +SIGHASH+ flags presented as an option in a user's wallet application. Simple wallet applications -sign with +SIGHASH_ALL+ flags. More sophisticated applications, such as +sign with [.keep-together]#+SIGHASH_ALL+# flags. More sophisticated applications, such as Lightning Network nodes, may use alternative +SIGHASH+ flags, but they use protocols that have been extensively reviewed to understand the influence of the alternative flags. ==== [[schnorr_signatures]] -=== Schnorr signatures +=== Schnorr Signatures In 1989, Claus Schnorr published a paper describing the signature algorithm that's become eponymous with him. The algorithm isn't -specific to the Elliptic Curve Cryptography (ECC) that Bitcoin and many +specific to the elliptic curve cryptography (ECC) that Bitcoin and many other applications use, although it is perhaps most strongly associated with ECC today. Schnorr signatures have a number of nice properties: Provable security:: A mathematical proof of the security of schnorr signatures depends on only the difficulty of solving the Discrete Logarithm Problem (DLP), - particularly for Elliptic Curves (EC) for Bitcoin, and the ability of + particularly for elliptic curves (EC) for Bitcoin, and the ability of a hash function (like the SHA256 function used in Bitcoin) to produce - unpredictable values, called the Random Oracle Model (ROM). Other + unpredictable values, called the random oracle model (ROM). Other signature algorithms have additional dependencies or require much larger public keys or signatures for equivalent security to ECC-Schnorr (when the threat is defined as classical computers; other @@ -269,12 +265,12 @@ Linearity:: properties. The first property is that summing together two or more variables and then running a function on that sum will produce the same value as running the function on each of the variables - independently and then summing together the results, e.g. + independently and then summing together the results, e.g., +f(x + y + z) == f(x) + f(y) + f(z)+; this property is called _additivity_. The second property is that multiplying a variable and then running a function on that product will produce the same value as running the function on the variable and then multiplying it by the - same amount, e.g. +f(a * x) == a * f(x)+; this property is called + same amount, e.g., +f(a * x) == a * f(x)+; this property is called _homogeneity of degree 1_. + In cryptographic operations, some functions may be private (such @@ -285,7 +281,7 @@ In cryptographic operations, some functions may be private (such specific benefits of linearity in schnorr signatures in <> and <>. -Batch Verification:: +Batch verification:: When used in a certain way (which Bitcoin does), one consequence of schnorr's linearity is that it's relatively straightforward to verify more than one schnorr signature at the same time in less time than it @@ -297,17 +293,16 @@ Batch Verification:: Later in this chapter, we'll describe the schnorr signature algorithm exactly as it's used in Bitcoin, but we're going to start with a -simplified version of it and work our way towards the actual protocol in +simplified version of it and work our way toward the actual protocol in stages. Alice starts by choosing a large random number (+x+), which we call her -_private key_. She also knows a public point on Bitcoin's Elliptic -Curve (EC) called the Generator (+G+) (see <>). Alice uses EC +_private key_. She also knows a public point on Bitcoin's elliptic +curve called the Generator (+G+) (see <>). Alice uses EC multiplication to multiply +G+ by her private key +x+, in which case +x+ is called a _scalar_ because it scales up +G+. The result is +xG+, which we call Alice's _public key_. Alice gives her public key to Bob. -Even though Bob also knows +G+, the Discrete Logarithm Problem (DLP) -prevents Bob from being able to divide +xG+ by +G+ to derive Alice's +Even though Bob also knows +G+, the Discrete Logarithm Problem prevents Bob from being able to divide +xG+ by +G+ to derive Alice's private key. At some later time, Bob wants Alice to identify herself by proving @@ -330,7 +325,7 @@ process: be used to multiply an EC point. 3. Alice now has the numbers (scalars) +x+, +k+, and +e+. She combines - them together to produce a final scalar +s+ using the formula: + them together to produce a final scalar +s+ using the formula +s = k + ex+. She gives +s+ to Bob. 4. Bob now knows the scalars +s+ and +e+, but not +x+ or +k+. However, @@ -339,11 +334,10 @@ process: the operation Alice performed: +sG == kG + exG+. If that is equal, then Bob can be sure that Alice knew +x+ when she generated +s+. -.Schnorr identity protocol with integers instead of points +.Schnorr Identity Protocol with Integers Instead of Points **** It might be easier to understand the interactive schnorr identity -protocol if we create an insecure oversimplification by substituting each of the values above -(including +G+) with simple integers instead of points on a elliptic curve. +protocol if we create an insecure oversimplification by substituting each of the preceding values (including +G+) with simple integers instead of points on an elliptic curve. For example, we'll use the prime numbers starting with 3: Setup: Alice chooses +x=3+ as her private key. She multiplies it by the @@ -364,7 +358,7 @@ generator +G=5+ to get her public key +xG=15+. She gives Bob +15+. Of course, this is an oversimplified example. When working with simple integers, we can divide products by the generator +G+ to get the underlying scalar, which isn't secure. This is why a critical property -of the Elliptic Curve Cryptography (ECC) used in Bitcoin is that +of the elliptic curve cryptography used in Bitcoin is that multiplication is easy but division by a point on the curve is impractical. Also, with numbers this small, finding underlying values (or valid substitutes) through brute force is easy; the numbers used in Bitcoin are much larger. @@ -373,7 +367,8 @@ brute force is easy; the numbers used in Bitcoin are much larger. Let's discuss some of the features of the interactive schnorr identity protocol that make it secure: -- The nonce (+k+). In step 1, Alice chooses a number that Bob doesn't +The nonce (+k+):: +In step 1, Alice chooses a number that Bob doesn't know and can't guess and gives him the scaled form of that number, +kG+. At that point, Bob also already has her public key (+xG+), which is the scaled form of +x+, her private key. That means when Bob is working on @@ -387,7 +382,8 @@ identity protocol that make it secure: be able to leverage that into figuring out Alice's private key. See <> for more details. -- The challenge scalar (+e+). Bob waits to receive Alice's public nonce +The challenge scalar (+e+):: +Bob waits to receive Alice's public nonce and then proceeds in step 2 to give her a number (the challenge scalar) that Alice didn't previously know and couldn't have guessed. It's critical that Bob only give her the challenge scalar after she @@ -395,8 +391,8 @@ identity protocol that make it secure: who didn't know +x+ wanted to impersonate Alice, and Bob accidentally gave them the challenge scalar +e+ before they told him the public nonce +kG+. This allows the impersonator to change parameters on both sides of - the equation that Bob will use for verification, +sG == kG + exG+, - specifically they can change both +sG+ and +kG+. Think about a + the equation that Bob will use for verification, +sG == kG + exG+; + specifically, they can change both +sG+ and +kG+. Think about a simplified form of that expression: x = y + a. If you can change both +x+ and +y+, you can cancel out +a+ using +x' = (x - a) + a+. Any value you choose for +x+ will now satisfy the equation. For the @@ -426,7 +422,7 @@ node that wants to authenticate her. A simple technique, known as the Fiat-Shamir transform after its discoverers, can turn the schnorr interactive identity protocol -into a non-interactive digital signature scheme. Recall the importance +into a noninteractive digital signature scheme. Recall the importance of steps 1 and 2--including that they be performed in order. Alice must commit to an unpredictable nonce; Bob must give Alice an unpredictable challenge scalar only after he has received her commitment. Recall also @@ -460,9 +456,9 @@ We've now defined a version of the schnorr signature protocol, but there's one more thing we need to do to address a Bitcoin-specific concern. In BIP32 key derivation, as described in <>, the algorithm for unhardened derivation -takes a public key and adds to it a non-secret value to produce a +takes a public key and adds to it a nonsecret value to produce a derived public key. That means it's also possible to add that -non-secret value to a valid signature for one key to produce a signature +nonsecret value to a valid signature for one key to produce a signature for a related key. That related signature is valid but it wasn't authorized by the person possessing the private key, which is a major security failure. To protect BIP32 unhardened derivation and @@ -486,13 +482,13 @@ generate a private key from a large random seed value). She uses the parameters defined in secp256k1 (see <>) to multiply the generator +G+ by her scalar +x+, producing +xG+ (her public key). She gives her public key to everyone who will later authenticate her Bitcoin -transactions (e.g. by having +xG+ included in a transaction output). When +transactions (e.g., by having +xG+ included in a transaction output). When she's ready to spend, she begins generating her signature: 1. Alice chooses a large random private nonce +k+ and derives the public nonce +kG+. -2. She chooses her message +m+ (e.g. transaction data) and generates the +2. She chooses her message +m+ (e.g., transaction data) and generates the challenge scalar +e = hash(kG || xG || m)+. 3. She produces the scalar +s = k + ex+. The two values +kG+ and +s+ @@ -502,19 +498,19 @@ she's ready to spend, she begins generating her signature: the witness structure of her spending transaction and then relaying that transaction to full nodes. -4. The verifiers (e.g. full nodes) use +s+ to derive +sG+ and then +4. The verifiers (e.g., full nodes) use +s+ to derive +sG+ and then verify that +sG == kG + hash(kG || xG || m)*xG+. If the equation is valid, Alice proved that she knows her private key +x+ (without revealing it) and committed to the message +m+ (containing the transaction data). -==== Serialization of schnorr signatures +==== Serialization of Schnorr Signatures A schnorr signature consists of two values, +kG+ and +s+. The value +kG+ is a point on Bitcoin's elliptic curve (called secp256k1) and so -would normally be represented by two 32-byte coordinates, e.g. +(x,y)+. -However, only the x coordinate is needed, so only that value is -included. When you see +kG+ below, note that it's only that point's x +would normally be represented by two 32-byte coordinates, e.g., +(x,y)+. +However, only the _x_ coordinate is needed, so only that value is +included. When you see +kG+ in schnorr signatures for Bitcoin, note that it's only that point's _x_ coordinate. The value +s+ is a scalar (a number meant to multiply other numbers). For @@ -522,8 +518,8 @@ Bitcoin's secp256k1 curve, it can never be more than 32 bytes long. Although both +kG+ and +s+ can sometimes be values that can be represented with fewer than 32 bytes, it's improbable that they'd be -much smaller than 32 bytes, and so they're serialized as two 32 byte -values (i.e., values smaller than 32 bytes have leading zeroes). +much smaller than 32 bytes, and so they're serialized as two 32-byte +values (i.e., values smaller than 32 bytes have leading zeros). They're serialized in the order of +kG+ and then +s+, producing exactly 64 bytes. @@ -541,7 +537,7 @@ the serialization used for ECDSA signatures described in <>. [[schnorr_multisignatures]] -==== Schnorr-based scriptless multisignatures +==== Schnorr-based Scriptless Multisignatures In the single-signature schnorr protocol described in <>, Alice uses a signature (+kG+, +s+) to publicly prove her knowledge of her @@ -560,9 +556,9 @@ that are believed to be secure. ==== Alice and Bob need to derive the public key for +x+, which is +xG+. -Since it's possible to use Elliptic Curve (EC) operations to add two EC +Since it's possible to use elliptic curve operations to add two EC points together, they start by Alice deriving +yG+ and Bob deriving -+zG+. Then then add them together to create +xG = yG + zG+. The point ++zG+. They then add them together to create +xG = yG + zG+. The point +xG+ is their _aggregated public key_. To create a signature, they begin the simple multisignature protocol: @@ -571,8 +567,8 @@ simple multisignature protocol: public nonce +aG+ and +bG+. Together, they produce an aggregated public nonce +kG = aG + bG+. -2. They agree on the message to sign, +m+ (e.g. a transaction), and - each generate a copy of the challenge scalar: +e = hash(kG || xG || m)+. +2. They agree on the message to sign, +m+ (e.g., a transaction), and + each generates a copy of the challenge scalar: +e = hash(kG || xG || m)+. 3. Alice produces the scalar +q = a + ey+. Bob produces the scalar +r = b + ez+. They add the scalars together to produce @@ -583,17 +579,17 @@ simple multisignature protocol: Alice and Bob have proven that they know the sum of their private keys without either one of them revealing their private key to the other or anyone -else. The protocol can be extended to any number of participants, e.g. +else. The protocol can be extended to any number of participants; e.g., a million people could prove they knew the sum of their million different keys. -The protocol above has several security problems. Most notable is that one +The preceding protocol has several security problems. Most notable is that one party might learn the public keys of the other parties before committing to their own public key. For example, Alice generates her public key +yG+ honestly and shares it with Bob. Bob generates his public key using +zG - yG+. When their two keys are combined (+yG + zG - yG+), the positive and negative +yG+ terms cancel out so the public key only represents -the private key for +z+, i.e. Bob's private key. Now Bob can create a +the private key for +z+, i.e., Bob's private key. Now Bob can create a valid signature without any assistance from Alice. This is called a _key cancellation attack_. @@ -623,14 +619,14 @@ signature parameters. The different nonces must not be related in any way. For a multisignature, every participant must follow these rules or it could compromise the security of other participants. In addition, cancellation and other attacks need to be prevented. Different -protocols that accomplish these aims make different tradeoffs, so +protocols that accomplish these aims make different trade-offs, so there's no single multisignature protocol to recommend in all cases. Instead, we'll note three from the MuSig family of protocols: MuSig:: Also called _MuSig1_, this protocol requires three rounds of communication during the signing process, making it similar to the - process we described above. MuSig1's greatest advantage is its + process we just described. MuSig1's greatest advantage is its simplicity. MuSig2:: @@ -645,13 +641,13 @@ MuSig-DN:: DN stands for Deterministic Nonce, which eliminates as a concern a problem known as the _repeated session attack_. It can't be combined with key exchange and it's significantly more complex to implement - than MuSig or Musig2. + than MuSig or MuSig2. For most applications, MuSig2 is the best multisignature protocol available at the time of writing. [[schnorr_threshold_signatures]] -==== Schnorr-based scriptless threshold signatures +==== Schnorr-based Scriptless Threshold Signatures Scriptless multisignature protocols only work for k-of-k signing. Everyone with a partial public key that becomes part of the aggregated @@ -664,10 +660,10 @@ _threshold signature_. We saw script-based threshold signatures in <>. But just as scriptless multisignatures save space and increase privacy compared to -scripted multisigantures, _scriptless threshold signatures_ save space and +scripted multisignatures, _scriptless threshold signatures_ save space and increase privacy compared to _scripted threshold signatures_. To anyone -not involved in the signing, a _scriptless threshold signatures_ looks -like any other signature which could've been created by a single-sig +not involved in the signing, a _scriptless threshold signature_ looks +like any other signature that could've been created by a single-sig user or through a scriptless multisignature protocol. Various methods are known for generating scriptless threshold @@ -685,7 +681,7 @@ on some additional information to each share, called a correction code, that allows any two of them to recover the number. This scheme is not secure because each share gives its holder partial knowledge of Alice's secret, making it easier for the participant to guess Alice's secret -than a non-participant who didn't have a share. +than a nonparticipant who didn't have a share. A secure secret sharing scheme prevents participants from learning anything about the secret unless they combine the minimum threshold @@ -697,7 +693,7 @@ after its discoverer, one of the same discoverers of the Fiat-Shamir transform we saw in <>. In some cryptographic protocols, such as the scriptless threshold signature -schemes we're working towards, it's critical for Bob, Carol, and Dan to +schemes we're working toward, it's critical for Bob, Carol, and Dan to know that Alice followed her side of the protocol correctly. They need to know that the shares she creates all derive from the same secret, that she used the threshold value she claims, and that she gave each one of @@ -705,7 +701,7 @@ them a different share. A protocol that can accomplish all of that, and still be a secure secret sharing scheme, is a _verifiable secret sharing scheme_. -To see how multisignatures and verifiable secret sharing works for +To see how multisignatures and verifiable secret sharing work for Alice, Bob, and Carol, imagine they each wish to receive funds that can be spent by any two of them. They collaborate as described in <> to produce a regular multisignature public @@ -726,8 +722,8 @@ partial private key. Alice and Bob also have their private keys, allowing them to create a scriptless multisignature with all three necessary keys. -In other words, the scriptless threshold signature scheme described -above is the same as a scriptless multisignature scheme except that +In other words, the scriptless threshold signature scheme just described +is the same as a scriptless multisignature scheme except that a threshold number of participants have the ability to reconstruct the partial private keys of any other participants who are unable or unwilling to sign. @@ -735,7 +731,8 @@ unwilling to sign. This does point to a few things to be aware about when considering a scriptless threshold signature protocol: -1. No accountability: because Alice and Bob reconstruct Carol's partial +No accountability:: +Because Alice and Bob reconstruct Carol's partial private key, there can be no fundamental difference between a scriptless multisignature produced by a process that involved Carol and one that didn't. Even if Alice, Bob, or Carol claim that they didn't sign, @@ -743,15 +740,16 @@ there's no guaranteed way for them to prove that they didn't help produce the signature. If it's important to know which members of the group signed, you will need to use a script. -2. Manipulation attacks: imagine that Bob tells Alice that Carol is +Manipulation attacks:: +Imagine that Bob tells Alice that Carol is unavailable, so they work together to reconstruct Carol's partial private key. Then Bob tells Carol that Alice is unavailable, so they work together to reconstruct Alice's partial private key. Now Bob has his own partial private key plus the keys of Alice and Carol, allowing him to spend the funds himself without their involvement. This attack can be addressed if all of the participants agree to only communicate using a -scheme that allows any one of them to see all of the other's messages, -e.g. if Bob tells Alice that Carol is unavailable, Carol is able to see +scheme that allows any one of them to see all of the other's messages; +e.g., if Bob tells Alice that Carol is unavailable, Carol is able to see that message before she begins working with Bob. Other solutions, possibly more robust solutions, to this problem were being researched at the time of writing. @@ -762,7 +760,7 @@ by multiple Bitcoin contributors and we expect peer-reviewed solutions will become available after the publication of this book. [[ecdsa_signatures]] -=== ECDSA signatures +=== ECDSA Signatures Unfortunately for the future development of Bitcoin and many other applications, Claus Schnorr patented the algorithm he discovered and @@ -770,7 +768,7 @@ prevented its use in open standards and open source software for almost two decades. Cryptographers in the early 1990s who were blocked from using the schnorr signature scheme developed an alternative construction called the _Digital Signature Algorithm_ (DSA), with a version adapted -to Elliptic Curves called ECDSA. +to elliptic curves called ECDSA. The ECDSA scheme and standardized parameters for suggested curves it could be used with were widely implemented in cryptographic libraries by the time @@ -790,20 +788,20 @@ More complex:: Less provable security:: The interactive schnorr signature identification protocol depends only - on the strength of the Elliptic Curve Discrete Logarithm Problem + on the strength of the elliptic curve Discrete Logarithm Problem (ECDLP). The non-interactive authentication protocol used in Bitcoin - also relies on the Random Oracle Model (ROM). However, ECDSA's extra + also relies on the random oracle model (ROM). However, ECDSA's extra complexity has prevented a complete proof of its security being published (to the best of our knowledge). We are not experts in proving cryptographic algorithms, but it seems unlikely after 30 years that ECDSA will be proven to only require the same two assumptions as schnorr. -Non-linear:: +Nonlinear:: ECDSA signatures cannot be easily combined to create scriptless multisignatures or used in related advanced applications such as - multi-party signature adaptors. There are workarounds for this - problem, but they involve additional extra complexity which + multiparty signature adaptors. There are workarounds for this + problem, but they involve additional extra complexity that significantly slows down operations and which, in some cases, has resulted in software accidentally leaking private keys. @@ -814,19 +812,18 @@ values are _R_ and _s_. The signature algorithm first generates a private nonce (_k_) and derives from it a public -nonce (_K_). The _R_ value of the digital signature is then the x +nonce (_K_). The _R_ value of the digital signature is then the _x_ coordinate of the nonce _K_. -From there, the algorithm calculates the _s_ value of the signature, -such that. Like we did with schnorr signatures, operations involving -integers are modulus p. +From there, the algorithm calculates the _s_ value of the signature. Like we did with schnorr signatures, operations involving +integers are modulus p: _s_ = __k__^-1^ (__Hash__(__m__) + __x__ * __R__) where: * _k_ is the private nonce -* _R_ is the x coordinate of the public nonce +* _R_ is the _x_ coordinate of the public nonce * _x_ is the Alice's private key * _m_ is the message (transaction data) @@ -844,18 +841,18 @@ where: - _m_ is the message (the transaction data that was signed) - _G_ is the elliptic curve generator point -If the x coordinate of the calculated point _K_ is equal to _R_, then +If the _x_ coordinate of the calculated point _K_ is equal to _R_, then the verifier can conclude that the signature is valid. [TIP] ==== ECDSA is necessarily a fairly complicated piece of math; a full explanation is beyond the scope of this book. A number of great guides -online take you through it step by step: search for "ECDSA explained". +online take you through it step by step: search for "ECDSA explained." ==== [[serialization_of_signatures_der]] -==== Serialization of ECDSA signatures (DER) +==== Serialization of ECDSA Signatures (DER) Let's look at the following DER-encoded signature: @@ -864,19 +861,19 @@ the following DER-encoded signature: 3045022100884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb02204b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e381301 ---- -That signature is a serialized byte-stream of the +R+ and +S+ values -produced by to prove control of the private key authorized +That signature is a serialized byte stream of the +R+ and +S+ values +produced by the signer to prove control of the private key authorized to spend an output. The serialization format consists of nine elements as follows: -* +0x30+—indicating the start of a DER sequence -* +0x45+—the length of the sequence (69 bytes) - * +0x02+—an integer value follows - * +0x21+—the length of the integer (33 bytes) - * +R+—++00884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb++ - * +0x02+—another integer follows - * +0x20+—the length of the integer (32 bytes) - * +S+—++4b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813++ +* +0x30+, indicating the start of a DER sequence +* +0x45+, the length of the sequence (69 bytes) + * +0x02+, an integer value follows + * +0x21+, the length of the integer (33 bytes) + * +R+, ++00884d142d86652a3f47ba4746ec719bbfbd040a570b1deccbb6498c75c4ae24cb++ + * +0x02+, another integer follows + * +0x20+, the length of the integer (32 bytes) + * +S+, ++4b9f039ff08df09cbe9f6addac960298cad530a863ea8f53982c09db8f6e3813++ * A suffix (+0x01+) indicating the type of hash used (+SIGHASH_ALL+) [[nonce_warning]] @@ -932,7 +929,7 @@ ensure you generate a different _k_ for each transaction. === Segregated Witness's New Signing Algorithm -Signatures in bitcoin transactions are applied on a _commitment hash_, +Signatures in Bitcoin transactions are applied on a _commitment hash_, which is calculated from the transaction data, locking specific parts of the data indicating the signer's commitment to those values. For example, in a simple +SIGHASH_ALL+ type signature, the commitment hash diff --git a/ch09_fees.adoc b/ch09_fees.adoc index e3580589..6ce81dbe 100644 --- a/ch09_fees.adoc +++ b/ch09_fees.adoc @@ -38,7 +38,7 @@ transaction pinning, and are largely uninteresting to the following discussion about fees--so we're going to ignore them in this chapter. ==== -The criteria that almost all miners use to select which transactions to +The criterion that almost all miners use to select which transactions to include in their blocks is to maximize their revenue. Bitcoin was specifically designed to accommodate this by providing a mechanism that allows a transaction to give money to the miner who includes that @@ -54,7 +54,7 @@ In this chapter, we'll explore various aspects of those bids--transaction fees--and how they influence the creation and management of Bitcoin transactions. -=== Who pays the transaction fee? +=== Who Pays the Transaction Fee? Most payment systems involve some sort of fee for transacting, but often this fee is hidden from typical buyers. For example, a merchant @@ -85,16 +85,16 @@ normally allowing only spenders to choose fees can sometimes be a problem; we'll look at a solution to that problem in <>. However, in many common payment flows, the parties with the highest desire to see a transaction confirm quickly--that is, the parties who'd be the most -willing to pay higher fees--are the the spenders. +willing to pay higher fees--are the spenders. For those reasons, both technical and practical, it is customary in Bitcoin for spenders to pay transaction fees. There are exceptions, -such as for merchants which accept unconfirmed transactions and in +such as for merchants that accept unconfirmed transactions and in protocols that don't immediately broadcast transactions after they are signed (preventing the spender from being able to choose an appropriate fee for the current market). We'll explore those exceptions later. -=== Fees and fee rates +=== Fees and Fee Rates Each transaction only pays a single fee--it doesn't matter how large the transaction is. However, the larger transactions become, the fewer of @@ -140,7 +140,7 @@ by accident. There are legitimate reasons for users to overpay fees on rare occasions. ==== -=== Estimating appropriate fee rates +=== Estimating Appropriate Fee Rates We've established that you can pay a lower fee rate if you're willing to wait longer for your transaction to be confirmed, with the exception @@ -163,7 +163,7 @@ similar number of blocks to confirm. Bitcoin Core includes a fee rate estimator that uses these principles, which can be called using the `estimatesmartfee` RPC with a parameter specifying how many blocks you're willing to wait before the transaction is highly likely to -confirm (for example, 144 blocks is about 1 day). +confirm (for example, 144 blocks is about 1 day): ---- $ bitcoin-cli -named estimatesmartfee conf_target=144 @@ -174,12 +174,12 @@ $ bitcoin-cli -named estimatesmartfee conf_target=144 ---- Many web-based services also provide fee estimation as an API. For a -current list, see https://www.lopp.net/bitcoin-information/fee-estimates.html +current list, see https://www.lopp.net/bitcoin-information/fee-estimates.html. As mentioned, fee rate estimation can never be perfect. One common problem is that the fundamental demand might change, adjusting the equilibrium and either increasing prices (fees) to new heights or -decreasing them towards the minimum. +decreasing them toward the minimum. If fee rates go down, then a transaction that previously paid a normal fee rate might now be paying a high fee rate and it will be confirmed earlier than expected. There's no way to @@ -187,18 +187,18 @@ lower the fee rate on a transaction you've already sent, so you're stuck paying a higher fee rate. But, when fee rates go up, there's a need for methods to be able to increase the fee rates on those transactions, which is called _fee bumping_. There are two commonly used types of fee -bumping in Bitcoin, Replace-By-Fee (RBF) and Child Pays For Parent +bumping in Bitcoin, replace by fee (RBF) and child pays for parent (CPFP). [[rbf]] -=== Replace-By-Fee (RBF) Fee Bumping +=== Replace By Fee (RBF) Fee Bumping To increase the fee of a transaction using RBF fee bumping, you create -a conflicting version of the transaction which pays a higher fee. Two +a conflicting version of the transaction that pays a higher fee. Two or more transactions are considered to be _conflicting transactions_ if only one of them can be included in a valid blockchain, forcing a miner -to chose only one of them. Conflicts occur when two or more transactions -each try to spend one of the same UTXOs, i.e. they each include an input +to choose only one of them. Conflicts occur when two or more transactions +each try to spend one of the same UTXOs, i.e., they each include an input that has the same outpoint (reference to the output of a previous transaction). @@ -226,7 +226,7 @@ Full RBF:: version. As of this writing, this can be optionally enabled in Bitcoin Core (but it is disabled by default). -.Why are there two variants of RBF? +.Why Are There Two Variants of RBF? **** The reason for the two different versions of RBF is that full RBF has been controversial. Early versions of Bitcoin allowed transaction @@ -260,7 +260,7 @@ based on the first version, but the second version gets confirmed, then the merchant will not receive payment for its costs. Some merchants, and people supporting them, requested that transaction -replacement not be re-enabled in Bitcoin Core. Other people pointed out +replacement not be reenabled in Bitcoin Core. Other people pointed out that transaction replacement provides benefits, including the ability to fee bump transactions that initially paid too low of a fee rate. @@ -289,7 +289,7 @@ whether or not that has happened yet. As a user, if you plan to use RBF fee bumping, you will first need to choose a wallet that supports it, such as one of the wallets listed as having "Sending support" on -https://bitcoinops.org/en/compatibility/#replace-by-fee-rbf +https://bitcoinops.org/en/compatibility/#replace-by-fee-rbf. As a developer, if you plan to implement RBF fee bumping, you will first need to decided whether to perform opt-in RBF or full RBF. At the time @@ -306,8 +306,8 @@ same for both approaches. When you need to fee bump a transaction, you will simply create a new transaction that spends at least one of the same UTXOs as the original transaction you want to replace. You will likely want to keep the -same outputs in the transaction which pay the receiver (or receivers). -You may pay the increased fee by reducing the the value of your change +same outputs in the transaction that pay the receiver (or receivers). +You may pay the increased fee by reducing the value of your change output or by adding additional inputs to the transaction. Developers should provide users with a fee-bumping interface that does all of this work for them and simply asks them (or suggests to them) how much the @@ -324,11 +324,11 @@ overpay the receivers. For example: - Transaction version 0 includes input _A_. - Transaction version 1 includes inputs _A_ and _B_ (e.g., you had to add input _B_ to pay the extra fees) -- Transaction version 2 includes inputs B and C (e.g., you had to add input +- Transaction version 2 includes inputs _B_ and _C_ (e.g., you had to add input _C_ to pay the extra fees but _C_ was large enough that you no longer need input _A_). -In the above scenario, any miner who saved version 0 of the transaction +In this scenario, any miner who saved version 0 of the transaction will be able to confirm both it and version 2 of the transaction. If both versions pay the same receivers, they'll be paid twice (and the miner will receive transaction fees from two separate transactions). @@ -347,27 +347,26 @@ have created if they had paid the increased fee rate in the first place. The fundamental disadvantage of RBF fee bumping is that it can normally only be performed by the creator of the transaction--the person or people who were required to provide signatures or other authentication -data for the transaction. An exception to this is transactions which +data for the transaction. An exception to this is transactions that were designed to allow additional inputs to be added by using sighash -flags, see <>, but that presents its own challenges. In +flags (see <>), but that presents its own challenges. In general, if you're the receiver of an unconfirmed transaction and you want to make it confirm faster (or at all), you can't use an RBF fee bump; you need some other method. -There are additional problems with RBF that we'll explore in the -subsequent <> section. +There are additional problems with RBF that we'll explore in <>. [[cpfp]] -=== Child Pays For Parent (CPFP) Fee Bumping +=== Child Pays for Parent (CPFP) Fee Bumping Anyone who receives the output of an unconfirmed transaction can incentivize miners to confirm that transaction by spending that output. The transaction you want to get confirmed is called the _parent -transaction_. A transaction which spends an output of the parent +transaction_. A transaction that spends an output of the parent transaction is called a _child transaction_. As we learned in <>, every input in a confirmed transaction -must reference the unspent output of a transaction which appears earlier +must reference the unspent output of a transaction that appears earlier in the blockchain (whether earlier in the same block or in a previous block). That means a miner who wants to confirm a child transaction must also ensure that its parent transaction is confirmed. If the @@ -384,7 +383,7 @@ about by fee rate and include the highest-revenue ones in the block they're attempting to mine, up to the maximum size (weight) allowed to be included in a block. To find even more packages that might be profitable to mine, the miner can evaluate packages across multiple -generations (e.g. an unconfirmed parent transaction being combined with +generations (e.g., an unconfirmed parent transaction being combined with both its child and grandchild). This is called _ancestor fee rate mining_. @@ -404,14 +403,14 @@ The primary disadvantage of CPFP compared to RBF is that CPFP typically uses more block space. In RBF, a fee bump transaction is often the same size as the transaction it replaces. In CPFP, a fee bump adds a whole separate transaction. Using extra block space requires paying extra -fees beyond the the cost of the fee bump. +fees beyond the cost of the fee bump. There are several challenges with CPFP, some of which we'll explore in -<>. One other problem which we +<>. One other problem that we specifically need to mention is the minimum relay fee rate problem, which is addressed by package relay. -==== Package Relay +=== Package Relay Early versions of Bitcoin Core didn't place any limits on the number of unconfirmed transactions they stored for later relay and mining in their @@ -454,25 +453,25 @@ problem. Although both RBF and CPFP fee bumping work in the basic cases we described, there are rules related to both -methods that are designed to prevent denial of service attacks on miners +methods that are designed to prevent denial-of-service attacks on miners and relaying full nodes. An unfortunate side effect of those rules is that they can sometimes prevent someone from being able to use fee bumping. Making it impossible or difficult to fee bump a transaction is called _transaction pinning_. -One of the major denial of service concerns revolve's around the effect of +One of the major denial of service concerns revolves around the effect of transaction relationships. Whenever the output of a transaction is spent, that transaction's identifier (txid) is referenced by the child transaction. However, when a transaction is replaced, the replacement has a different txid. If that replacement transaction gets confirmed, none of its descendants can be included in the same blockchain. It's -possible to recreate and re-sign the descendant transactions, but that's +possible to re-create and re-sign the descendant transactions, but that's not guaranteed to happen. This has related but divergent implications for RBF and CPFP: - In the context of RBF, when Bitcoin Core accepts a replacement transaction, it keeps things simple by forgetting about the original - transaction and all descendant transactions which depended on that + transaction and all descendant transactions that depended on that original. To ensure that it's more profitable for miners to accept replacements, Bitcoin Core only accepts a replacement transaction if it pays more fees than all the transactions that will be forgotten. @@ -496,7 +495,7 @@ transaction. Similarly, if a transaction with 25 descendants is removed from a node's mempool (such as for being included in a block), and each of those descendants has 25 other ancestors, another 625 packages need to - be updated. Each time we double our parameter (e.g. from 25 to 50), + be updated. Each time we double our parameter (e.g., from 25 to 50), we quadruple the amount of work our node needs to perform. + Additionally, a transaction and all of its descendants is not @@ -526,14 +525,13 @@ Protocol developers have been working on mitigating problems with transaction pinning for several years. One partial solution is described in <>. Several other solutions have been proposed, and at least one solution is being actively developed as of -this writing (ephemeral anchors, see -https://bitcoinops.org/en/topics/ephemeral-anchors/). +this writing—https://bitcoinops.org/en/topics/ephemeral-anchors/[ephemeral anchors]. [[cpfp_carve_out]] === CPFP Carve Out and Anchor Outputs In 2018, developers working on Lightning Network (LN) had a problem. -Their protocol uses transactions which require signatures from two +Their protocol uses transactions that require signatures from two different parties. Neither party wants to trust the other, so they sign transactions at a point in the protocol when trust isn't needed, allowing either of them to broadcast one of those transactions at a @@ -620,7 +618,7 @@ you intended. [[fee_sniping]] === Timelock Defense Against Fee Sniping -Fee-sniping is a theoretical +Fee sniping is a theoretical attack scenario, where miners attempting to rewrite past blocks "snipe" higher-fee transactions from future blocks to maximize their profitability. @@ -666,4 +664,4 @@ As Bitcoin continues to mature, and as the subsidy continues to decline, fees become more and more important to Bitcoin users, both in their day-to-day use for getting transactions confirmed quickly and in providing an incentive for miners to continue securing Bitcoin -transactions with new proof-of-work. +transactions with new proof of work. diff --git a/ch10_network.adoc b/ch10_network.adoc index 7289a77b..edee8689 100644 --- a/ch10_network.adoc +++ b/ch10_network.adoc @@ -3,7 +3,7 @@ Bitcoin is structured as a peer-to-peer network architecture on top of the internet. The term peer-to-peer, or P2P, means that the -full nodes which participate in the network are peers to each other, that +full nodes that participate in the network are peers to each other, that they can all perform the same functions, and that there are no "special" nodes. The network nodes interconnect in a mesh network with a "flat" topology. There is no @@ -52,7 +52,7 @@ only a subset of the blockchain and partly verify transactions using a method called _simplified payment verification_, or SPV. These clients are known as lightweight clients. Miners compete to create new blocks by -running specialized hardware to solve the Proof-of-Work algorithm. Some +running specialized hardware to solve the proof-of-work algorithm. Some miners operate full nodes, validating every block on the blockchain, while others are clients participating in pool mining and depending on a pool server to provide them with work. @@ -119,7 +119,7 @@ some of those transactions are confirmed in a new block, the node doesn't need to receive a second copy of those transactions. Instead of receiving redundant unconfirmed transactions, compact blocks -allows a peer to instead send a short 6-byte identifier for each transaction. +allow a peer to instead send a short 6-byte identifier for each transaction. When your node receives a compact block with one or more identifiers, it checks its mempool for those transactions and uses them if they are found. For any transaction that isn't found in your local node's @@ -133,7 +133,7 @@ always sends a block's coinbase transaction. If the remote peer guesses correctly about what transactions your node has in its mempool, and which it does not, it will send a block nearly as efficiently as is theoretically possible (for a typical block, it'll -be between 97% to 99% efficient). +be between 97% and 99% efficient). [TIP] ==== @@ -175,7 +175,7 @@ quickly announcing blocks). // released into the public domain by Nicolas Dorier [[bip152_illustration]] -.BIP152 modes compared (from BIP152). They grey bar indicates the time it takes the node to validate the block +.BIP152 modes compared (from BIP152). The gray bar indicates the time it takes the node to validate the block. image::images/mbc3_1002.png["BIP152"] The names of the two methods (which are taken from BIP152) can be a bit @@ -186,15 +186,15 @@ relay before compact blocks were implemented. === Private Block Relay Networks -Although compact blocks go a long way towards minimizing the time it +Although compact blocks go a long way toward minimizing the time it takes for blocks to propagate across the network, it's possible to minimize latency further. Unlike -compact blocks, though, the other solutions involve tradeoffs that +compact blocks, though, the other solutions involve trade-offs that make them unavailable or unsuitable for the public P2P relay network. For that reason, there has been experimentation with private relay networks for blocks. -One simple technique is to pre-select a route between endpoints. For +One simple technique is to preselect a route between endpoints. For example, a relay network with servers running in datacenters near major trans-oceanic fiber optic lines might be able to forward new blocks faster than waiting for the block to arrive at the node run by some home @@ -214,18 +214,18 @@ because we mostly use protocols that automatically re-request the missing data. However, requesting missing data triples the time to receive it. For example: -1. Alice sends some data to Bob +1. Alice sends some data to Bob. 2. Bob doesn't receive the data (or it is damaged). Bob re-requests - the data from Alice -3. Alice sends the data again + the data from Alice. +3. Alice sends the data again. A third technique is to assume all nodes receiving the data have almost all of the same transactions in their mempool, so they can all accept the same compact block. That not only saves us time computing -a compact block at each hop but it means that all each hop can simply +a compact block at each hop, but it means that all each hop can simply relay the FEC packets to the next hop even before validating them. -The tradeoff for each of the above methods is that they work well with +The trade-off for each of the preceding methods is that they work well with centralization but not in a decentralized network where individual nodes can't trust other nodes. Servers in datacenters cost money and can often be accessed by operators of the datacenter, making them less @@ -270,7 +270,7 @@ information, including: +nTime+:: The current time +addrYou+:: The IP address of the remote node as seen from this node +addrMe+:: The IP address of the local node, as discovered by the local node -+subver+:: A sub-version showing the type of software running on this node (e.g., pass:[/Satoshi:0.9.2.1/]) ++subver+:: A subversion showing the type of software running on this node (e.g., pass:[/Satoshi:0.9.2.1/]) +BestHeight+:: The block height of this node's blockchain +fRelay+:: A field added by BIP37 for requesting not to receive unconfirmed transactions @@ -435,8 +435,8 @@ decisions. However, the most common implementation is Bitcoin Core. More than 95% of full nodes on the Bitcoin network run various versions of Bitcoin Core. It is identified as "Satoshi" in the -sub-version string sent in the +version+ message and shown by the -command +getpeerinfo+ as we saw earlier; for example, +/Satoshi:24.0.1/+. +subversion string sent in the +version+ message and shown by the +command +getpeerinfo+ as we saw earlier; for example, [.keep-together]#+/Satoshi:24.0.1/+#. === Exchanging "Inventory" @@ -458,7 +458,7 @@ a +getheaders+ message that contains the hash of the top block on their local blockchain. One of the peers will be able to identify the received hash as belonging to a block that is not at the top, but rather belongs to an older block, thus deducing that its own -local blockchain is longer than its peer's. +local blockchain is longer than the remote node's blockchain. The peer that has the longer blockchain has more blocks than the other node and can identify which headers the other node needs in order to @@ -474,7 +474,7 @@ connections to peers that are significantly slower than the average in order to find newer (and possibly faster) peers. Let's assume, for example, that a node only has the genesis block. It -will then receive an +headers+ message from its peers containing the headers +will then receive a +headers+ message from its peers containing the headers of the next 2,000 blocks in the chain. It will start requesting blocks from all of its connected peers, keeping a queue of up to 1,024 blocks. Blocks need to be validated in order, so if the oldest block in the @@ -542,7 +542,7 @@ on top of it is proof, by proxy, that the transaction actually exists. A lightweight client cannot normally be persuaded that a transaction exists in a block when the transaction does not in fact exist. The lightweight client establishes the existence of a transaction in a block by requesting a merkle path -proof and by validating the Proof-of-Work in the chain of blocks. +proof and by validating the proof of work in the chain of blocks. However, a transaction's existence can be "hidden" from a lightweight client. A lightweight client can definitely verify that a transaction exists but cannot verify that a transaction, such as a double-spend of the same UTXO, @@ -565,7 +565,7 @@ node. ==== A full node verifies a transaction by checking the entire chain of thousands of blocks below it in order to guarantee that the UTXO exists -and is not spent, whereas an lightweight client only proves that a transaction +and is not spent, whereas a lightweight client only proves that a transaction exists and checks that the block containing that transaction is buried by a handful of blocks above it. ==== @@ -607,19 +607,19 @@ transactions matching a specific pattern, without revealing exactly which addresses, keys, or transactions they are searching for. In our previous analogy, a tourist without a map is asking for -directions to a specific address, "23 Church St." If she asks strangers -for directions to this street, she inadvertently reveals her +directions to a specific address, "23 Church St." If they ask a stranger +for directions to this street, they inadvertently reveal their destination. A bloom filter is like asking, "Are there any streets in this neighborhood whose name ends in R-C-H?" A question like that reveals slightly less about the desired destination than asking for "23 Church St." Using this technique, a tourist could specify the desired -address in more detail such as "ending in U-R-C-H" or less detail as +address in more detail such as "ending in U-R-C-H" or less detail such as "ending in H." By varying the precision of the search, the tourist reveals more or less information, at the expense of getting more or less -specific results. If she asks a less specific pattern, she gets a lot +specific results. If they ask for a less specific pattern, they get a lot more possible addresses and better privacy, but many of the results are -irrelevant. If she asks for a very specific pattern, she gets fewer -results but loses privacy. +irrelevant. If they ask for a very specific pattern, they get fewer +results but lose privacy. Bloom filters serve this function by allowing a lightweight client to specify a search pattern for transactions that can be tuned toward precision or @@ -782,14 +782,14 @@ that they didn't offer very much privacy. A full node receiving a bloom filter from a peer could apply that filter to the entire blockchain to find all of the client's transactions (plus false positives). It could then look for patterns and relationships between the transactions. -Randomly-selected false positive transactions would be unlikely to have +Randomly selected false positive transactions would be unlikely to have a parent-child relationship from output to input, but transactions from the user's wallet would be very likely to have that relationship. If all of the related transactions have certain characteristics, such as at least one P2PKH output, then transactions without that characteristic can be assumed not to belong to the wallet. -It was also discovered that specially-constructed filters could force +It was also discovered that specially constructed filters could force the full nodes that processed them to perform a large amount of work, which could lead to denial-of-service attacks. @@ -814,7 +814,7 @@ the entire block. [NOTE] ==== Despite the similarities in names, BIP152 _compact blocks_ and -BIP157/158 _compact block filters_ are unrelated +BIP157/158 _compact block filters_ are unrelated. ==== This allows nodes to create a single filter for every block, which they @@ -863,8 +863,8 @@ numerical order: Then, Alice sends the first number. For the remaining numbers, she sends the difference between that number and the preceding number. For -example, for the second number, she sends 97 (476 - 379); for the third -number, she sends 177 (653 - 476); and so on: +example, for the second number, she sends 97 (476 – 379); for the third +number, she sends 177 (653 – 476); and so on: ---- 379 @@ -885,8 +885,8 @@ more numbers we select, the smaller the average (mean) size of the differences. That means the amount of data we need to transfer doesn't increase as fast as the length of our list increases (up to a point). -Even more usefully, the length of the randomly-selected numbers in a -list of differences is naturally biased towards smaller lengths. +Even more usefully, the length of the randomly selected numbers in a +list of differences is naturally biased toward smaller lengths. Consider selecting two random numbers from 1 to 6; this is the same as rolling two dice. There are 36 distinct combinations of two dice: @@ -939,17 +939,17 @@ Golomb coding provides that facility. Rice coding is a subset of Golomb coding that's more convenient to use in some situations, including the application of Bitcoin block filters. -==== What data to include in a block filter +==== What Data to Include in a Block Filter Our primary goal is to allow wallets to learn whether a block contains a transaction affecting that wallet. For a wallet to be effective, it needs to learn two types of information: -1. When it has received money. Specifically, when a transaction +. When it has received money. Specifically, when a transaction output contains a script that the wallet controls (such as by controlling the authorized private key). -2. When it has spent money. Specifically, when a transaction input +. When it has spent money. Specifically, when a transaction input references a previous transaction output that the wallet controlled. A secondary goal during the design of compact block filters was to allow @@ -963,9 +963,9 @@ accurate filter. For both the primary and secondary goals to be met, a filter would need to reference two types of information: -1. The script for every output in every transaction in a block. +* The script for every output in every transaction in a block. -2. The outpoint for every input in every transaction in a block. +* The outpoint for every input in every transaction in a block. An early design for compact block filters included both of those pieces of information, but it was realized that there was a more efficient way @@ -973,15 +973,15 @@ to accomplish the primary goal if we sacrificed the secondary goal. In the new design, a block filter would still references two types of information, but they'd be more closely related: -1. As before, the script for every output in every transaction in a +* As before, the script for every output in every transaction in a block. -2. In a change, it would also reference the script of the output +* In a change, it would also reference the script of the output referenced by the outpoint for every input in every transaction in a block. In other words, the output script being spent. This had several advantages. First, it meant that wallets didn't need -to track outpoints; they could instead just scan for the the +to track outpoints; they could instead just scan for the output scripts to which they expected to receive money. Second, any time a later transaction in a block spends the output of an earlier transaction in the same block, they'll both reference the same @@ -999,12 +999,12 @@ to verify a block filter was built correctly. However, there is an alternative that can help a client detect if a peer is lying to it: obtaining the same filter from multiple peers. -==== Downloading block filters from multiple peers +==== Downloading Block Filters from Multiple Peers -A peer can provide a wallet with an inaccurate filter. There's two ways +A peer can provide a wallet with an inaccurate filter. There are two ways to create an inaccurate filter. The peer can create a filter that references transactions that don't actually appear in the associated -block (a false positive). Alternatively, the peer can crate a filter +block (a false positive). Alternatively, the peer can create a filter that doesn't reference transactions that do actually appear in the associated block (a false negative). @@ -1021,7 +1021,7 @@ block, the client can download all of them. It can then also download the associated block. If the block contains any transaction related to the wallet that is not part of one of the filters, then the wallet can be sure that whichever peer created that filter was -inaccurate--Golomb-Rice Coded Sets (GCSes) will always include a +inaccurate--Golomb-Rice Coded Sets will always include a potential match. Alternatively, if the block doesn't contain a transaction that the @@ -1035,7 +1035,7 @@ designed to use, the wallet can stop using that peer. In most cases, the only consequence of the inaccurate filter is that the wallet uses more bandwidth than expected. -==== Reducing bandwidth with lossy encoding +==== Reducing Bandwidth with Lossy Encoding The data about the transactions in a block that we want to communicate is an output script. Output scripts vary in length and follow patterns, @@ -1051,7 +1051,7 @@ faster and more configurable non-cryptographic hash functions, such as the SipHash function we'll use for compact block filters. The details of the algorithm used are described in BIP158, but the gist -is that each output script is reduced to a 64 bit commitment using +is that each output script is reduced to a 64-bit commitment using SipHash and some arithmetic operations. You can think of this as taking a set of large numbers and truncating them to shorter numbers, a process that loses data (so it's called _lossy encoding_). By losing @@ -1059,9 +1059,9 @@ some information, we don't need to store as much information later, which saves space. In this case, we go from a typical output script that's 160 bits or longer down to just 64 bits. -==== Using compact block filters +==== Using Compact Block Filters -The 64 bit value for every commitment to an output script in a block are +The 64-bit values for every commitment to an output script in a block are sorted, duplicate entries are removed, and the GCS is constructed by finding the differences (deltas) between each entry. That compact block filter is then distributed by peers to their clients (such as wallets). @@ -1074,23 +1074,23 @@ the filter. Recall our example of the lossiness of compact block filters being similar to truncating a number. Imagine a client is looking for a block -that contains the number 123456 and that an an accurate (but lossy) +that contains the number 123456 and that an accurate (but lossy) compact block filter contains the number 1234. When a client sees that 1234, it will download the associated block. There's a 100% guarantee that an accurate filter containing 1234 will allow a client to learn about a block containing 123456, called a _true positive_. However, there's also chance that the block might contain -123400, 123401, or almost a hundred other entries that are not when the +123400, 123401, or almost a hundred other entries that are not what the client is looking for (in this example), called a _false positive_. A 100% true positive match rate is great. It means that a wallet can depend on compact block filters to find every transaction affecting that -wallet. A non-zero false positive rate means that the wallet will end +wallet. A nonzero false positive rate means that the wallet will end up downloading some blocks that don't contain transactions interesting to the wallet. The main consequence of this is that the client will use extra bandwidth, which is not a huge problem. The actual -false-positive rate for BIP158 compact block filters is very low, so +false positive rate for BIP158 compact block filters is very low, so it's not a major problem. A false positive rate can also help improve a client's privacy, as it does with bloom filters, although anyone wanting the best possible privacy should still use their own full node. @@ -1129,7 +1129,7 @@ at the time of writing. As a way to increase the privacy and security of the Bitcoin P2P network, there is a solution that provides encryption of the -communications: _Tor Transport_. +communications: _Tor transport_. ==== Tor Transport @@ -1155,7 +1155,7 @@ debugging for the Tor service like this: $ bitcoind --daemon --debug=tor ---- -You should see "tor: ADD_ONION successful" in the logs, indicating that +You should see ++tor: ADD_ONION successful++ in the logs, indicating that Bitcoin Core has added a hidden service to the Tor network. You can find more instructions on running Bitcoin Core as a Tor hidden @@ -1163,7 +1163,7 @@ service in the Bitcoin Core documentation (_docs/tor.md_) and various online tutorials. [[mempool]] -=== Mempools and orphan pools +=== Mempools and Orphan Pools Almost every node on the Bitcoin network maintains a temporary list of unconfirmed transactions called @@ -1190,10 +1190,10 @@ of the newly added transaction, which is no longer an orphan, the process is repeated recursively looking for any further descendants, until no more descendants are found. Through this process, the arrival of a parent transaction triggers a cascade reconstruction of an entire -chain of interdependent transactions by re-uniting the orphans with +chain of interdependent transactions by reuniting the orphans with their parents all the way down the chain. -Some implementations of the Bitcoin also maintain an UTXO +Some implementations of Bitcoin also maintain a UTXO database, which is the set of all unspent outputs on the blockchain. This represents a different set of data from the mempool. Unlike the mempool and orphan pools, the UTXO database @@ -1204,11 +1204,11 @@ table on persistent storage. Whereas the mempool and orphan pools represent a single node's local perspective and might vary significantly from node to node depending -upon when the node was started or restarted, the UTXO database represents +on when the node was started or restarted, the UTXO database represents the emergent consensus of the network and therefore will not usually vary between nodes. -Now that we have a understanding of many of the data types and +Now that we have an understanding of many of the data types and structures used by nodes and clients to send data across the Bitcoin network, it's time to look at the software that's responsible for keeping the network secure and operational. diff --git a/ch11_blockchain.adoc b/ch11_blockchain.adoc index 7ccc71af..d6339c08 100644 --- a/ch11_blockchain.adoc +++ b/ch11_blockchain.adoc @@ -84,7 +84,7 @@ header. <> describes how Bitcoin Core stores the structure of |Size| Field | Description | 4 bytes | Block Size | The size of the block, in bytes, following this field | 80 bytes | Block Header | Several fields form the block header -| 1-3 bytes (compactSize) | Transaction Counter | How many transactions follow +| 1–3 bytes (compactSize) | Transaction Counter | How many transactions follow | Variable | Transactions | The transactions recorded in this block |======= @@ -103,8 +103,8 @@ block metadata as shown in <>. | 32 bytes | Previous Block Hash | A hash of the previous (parent) block in the chain | 32 bytes | Merkle Root | The root hash of the merkle tree of this block's transactions | 4 bytes | Timestamp | The approximate creation time of this block (Unix epoch time) -| 4 bytes | Target | A compact encoding of the Proof-of-Work target for this block -| 4 bytes | Nonce | Arbitrary data used for the Proof-of-Work algorithm +| 4 bytes | Target | A compact encoding of the proof-of-work target for this block +| 4 bytes | Nonce | Arbitrary data used for the proof-of-work algorithm |======= The nonce, target, and timestamp are used in the mining @@ -185,7 +185,7 @@ even the single transaction within. Thus, every node has the starting point for the blockchain, a secure "root" from which to build a trusted blockchain. -See the statically encoded genesis block inside the Bitcoin Core client, +See the statically encoded genesis block inside the Bitcoin Core client in https://github.com/bitcoin/bitcoin/blob/3955c3940eff83518c186facfec6f50545b5aab5/src/chainparams.cpp#L123[_chainparams.cpp_]. The following identifier hash belongs to the genesis block: @@ -200,7 +200,7 @@ of this block, with a URL containing that hash: https://blockstream.info/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f -Using the Bitcoin Core reference client on the command line: +Alternatively, you can get the block using Bitcoin Core on the command line: ---- $ bitcoin-cli getblock 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f @@ -311,7 +311,7 @@ these trees are usually displayed upside down with the "root" at the top and the "leaves" at the bottom of a diagram, as you will see in the examples that follow. -Merkle trees are used in bitcoin to summarize all the transactions in a +Merkle trees are used in Bitcoin to summarize all the transactions in a block, producing an overall commitment to the entire set of transactions and permitting a very efficient process to verify whether a transaction is included in a block. A merkle tree is constructed by @@ -361,7 +361,7 @@ of the nodes. image::images/mbc3_1102.png["merkle_tree"] Because the merkle tree is a binary tree, it needs -an even number of leaf nodes. If there is an odd number of transactions +an even number of leaf nodes. If there are an odd number of transactions to summarize, the last transaction hash will be duplicated to create an even number of leaf nodes, also known as a _balanced tree_. This is shown in <>, where transaction C is duplicated. @@ -372,10 +372,10 @@ the last hash is duplicated. .Duplicating one data element achieves an even number of data elements image::images/mbc3_1103.png["merkle_tree_odd"] -.A design flaw in Bitcoin's merkle tree +.A Design Flaw in Bitcoin's Merkle Tree **** -An extended comment in Bitcoin Core's source code describes a -significant problems in the design of Bitcoin's duplication of odd +An extended comment in Bitcoin Core's source code, reproduced here with slight revisions, describes a +significant problem in the design of Bitcoin's duplication of odd elements in its merkle tree: [quote,Bitcoin Core src/consensus/merkle.cpp] @@ -388,14 +388,14 @@ duplicate txids, resulting in a vulnerability (CVE-2012-2459). The reason is that if the number of hashes in the list at a given level is odd, the last one is duplicated before computing the next level (which is unusual in Merkle trees). This results in certain sequences of -transactions leading to the same merkle root. For example, these two -trees: +transactions leading to the same merkle root. For example, the two +trees in <>: [[cve_tree]] .Two Bitcoin-style merkle tree with the same root but a different number of leaves -image::images/mbc3_1104.png["Two Bitcoin-style merkle tree with the same root but a different number of leaves"] +image::images/mbc3_1104.png["Two Bitcoin-style merkle trees with the same root but a different number of leaves"] -For transaction lists [1,2,3,4,5,6] and [1,2,3,4,5,6,5,6] (where 5 and +The transaction lists [1,2,3,4,5,6] and [1,2,3,4,5,6,5,6] (where 5 and 6 are repeated) result in the same root hash A (because the hash of both of (F) and (F,F) is C). @@ -433,7 +433,7 @@ especially important as the number of transactions increases, because the base-2 logarithm of the number of transactions increases much more slowly. This allows Bitcoin nodes to efficiently produce paths of 10 or 12 hashes (320–384 bytes), which can provide proof of a single -transaction out of more than a thousand transactions in a multi-megabyte +transaction out of more than a thousand transactions in a multimegabyte block. [[merkle_tree_large]] @@ -457,7 +457,7 @@ image::images/mbc3_1106.png["merkle_tree_path"] The efficiency of merkle trees becomes obvious as the scale increases. The largest possible block can hold almost 16,000 transactions in 4,000,000 -bytes, but proving any particular one of those sixteen thousand transactions +bytes, but proving any particular one of those 16,000 transactions is a part of that block only requires a copy of the transaction, a copy of the 80-byte block header, and 448 bytes for the merkle proof. That makes the largest possible proof almost 10,000 times smaller than the @@ -576,14 +576,14 @@ as +btcd+ (written in Go) and +bcoin+ (written in JavaScript), to experiment and learn in other programming languages and frameworks. Testnet3 supports all the features of mainnet, including -Segregated Witness v0 and v1 (see <> and <>). Therefore, testnet3 can also be -used to test Segregated Witness features. +segregated witness v0 and v1 (see <> and <>). Therefore, testnet3 can also be +used to test segregated witness features. -===== Problems With Testnet +===== Problems with testnet Testnet doesn't just use the same data structures as Bitcoin, it also -uses almost exactly the same Proof-of-Work (PoW) security mechanism as -Bitcoin. The notable differences for testnet are that it's minimum +uses almost exactly the same proof-of-work (PoW) security mechanism as +Bitcoin. The notable differences for testnet are that its minimum difficulty is half that of Bitcoin and that it's allowed to include a block at the minimum difficulty if that block's timestamp is more than 20 minutes after the previous block. @@ -630,7 +630,7 @@ BIP325 signets are designed to make it very easy to create your own. If you disagree with how someone else is running their signet, you can start your own signet and connect your software to it. -===== The Default Signet and Custom Signets +===== The default signet and custom signets Bitcoin Core supports a default signet, which we believe to be the most widely used signet at the time of writing. It is currently operated by @@ -757,7 +757,7 @@ $ bitcoin-cli -regtest getbalance === Using Test Blockchains for Development Bitcoin's various -blockchains (+regtest+, +signet+, +testnet3+, +mainnet+) offer a range +blockchains (regtest, signet, testnet3, mainnet) offer a range of testing environments for bitcoin development. Use the test blockchains whether you are developing for Bitcoin Core, or another full-node consensus client; an application such as a wallet, exchange,