Each transaction input and any signatures it may contain is _completely_
independent of any other input or signature. Multiple parties can
collaborate to construct transactions and sign only one input each.
Several protocols uses this fact to create multi-party transactions for
Several protocols use this fact to create multiparty transactions for
privacy.
====
@ -44,7 +44,7 @@ signing key). The second part is an algorithm
that allows anyone to verify the signature, given also the message and the corresponding
public key.
==== Creating a digital signature
==== Creating a Digital Signature
In Bitcoin's use of digital signature algorithms, the "message" being
signed is the transaction, or more accurately a hash of a specific
@ -62,7 +62,7 @@ where:
* _F_~_sig_~ is the signing algorithm
* _Sig_ is the resulting signature
More details on the mathematics of schnorr and ECDSA signatures can be found in <<schnorr_signatures>>
You can find more details on the mathematics of schnorr and ECDSA signatures in <<schnorr_signatures>>
and <<ecdsa_signatures>>.
In both schnorr and ECDSA signatures, the function _F_~_sig_~ produces a signature +Sig+ that is composed of
@ -70,7 +70,7 @@ two values. There are differences between the two values in the
different algorithms, which we'll explore later.
After the two values
are calculated, they are serialized into a byte-stream. For ECDSA
are calculated, they are serialized into a bytestream. For ECDSA
signatures, the encoding uses an international standard encoding scheme
called the
_Distinguished Encoding Rules_, or _DER_. For schnorr signatures, a
@ -78,10 +78,7 @@ simpler serialization format is used.
==== Verifying the Signature
The signature verification
algorithm takes the message (a hash of parts of the transaction and
related data), the signer's public key and the signature, and returns
TRUE if the signature is valid for this message and public key.
The signature verification algorithm takes the message (a hash of parts of the transaction and related data), the signer's public key and the signature, and returns ++TRUE++ if the signature is valid for this message and public key.
To verify the signature, one must have the signature, the serialized
transaction, some data about the output being spent, and the public key
@ -127,16 +124,16 @@ There are three +SIGHASH+ flags: +ALL+, +NONE+, and +SINGLE+, as shown
in <<sighash_types_and_their>>.
[[sighash_types_and_their]]
.SIGHASH types and their meanings
.++SIGHASH++ types and their meanings
[options="header"]
|=======================
|+SIGHASH+ flag| Value | Description
| +ALL+ | 0x01 | Signature applies to all inputs and outputs
| +NONE+ | 0x02 | Signature applies to all inputs, none of the outputs
| +SINGLE+ | 0x03 | Signature applies to all inputs but only the one output with the same index number as the signed input
|++SIGHASH++ flag| Value | Description
| ++ALL++ | ++0x01++ | Signature applies to all inputs and outputs
| ++NONE++ | ++0x02++ | Signature applies to all inputs, none of the outputs
| ++SINGLE++ | ++0x03++ | Signature applies to all inputs but only the one output with the same index number as the signed input
|=======================
In addition, there is a modifier flag +SIGHASH_ANYONECANPAY+, which can
In addition, there is a modifier flag, +SIGHASH_ANYONECANPAY+, which can
be combined with each of the preceding flags. When +ANYONECANPAY+ is
set, only one input is signed, leaving the rest (and their sequence
numbers) open for modification. The +ANYONECANPAY+ has the value +0x80+
@ -144,13 +141,13 @@ and is applied by bitwise OR, resulting in the combined flags as shown
in <<sighash_types_with_modifiers>>.
[[sighash_types_with_modifiers]]
.SIGHASH types with modifiers and their meanings
.++SIGHASH++ types with modifiers and their meanings
[options="header"]
|=======================
|SIGHASH flag| Value | Description
| ALL\|ANYONECANPAY | 0x81 | Signature applies to one input and all outputs
| NONE\|ANYONECANPAY | 0x82 | Signature applies to one input, none of the outputs
| SINGLE\|ANYONECANPAY | 0x83 | Signature applies to one input and the output with the same index number
|++SIGHASH++ flag| Value | Description
| ++ALL\++|++ANYONECANPAY++ | ++0x81++ | Signature applies to one input and all outputs
| ++NONE\++|++ANYONECANPAY++ | ++0x82++ | Signature applies to one input, none of the outputs
| ++SINGLE\++|++ANYONECANPAY++ | ++0x83++ | Signature applies to one input and the output with the same index number
|=======================
The way +SIGHASH+ flags are applied during signing and verification is
@ -195,12 +192,12 @@ By itself, this allows any miner to change
the output destination and claim the funds for themselves, but if other
required signatures in the transaction use +SIGHASH_ALL+ or another type
that commits to the output, it allows those spenders to change the
destination without allowing any third-parties (like miners) to modify
destination without allowing any thirdparties (like miners) to modify
the outputs.
+NONE|ANYONECANPAY+ :: This construction can be used to build a "dust
collector." Users who have tiny UTXOs in their wallets can't spend these
without the cost in fees exceeding the value of the UTXO, see
without the cost in fees exceeding the value of the UTXO; see
<<uneconomical_outputs>>. With this type
of signature, the uneconomical UTXOs can be donated for anyone to aggregate and
spend whenever they want.
@ -212,23 +209,22 @@ new sighash flags. A signature using +SIGHASH_ANYPREVOUT+ would not
commit to an input's outpoint field, allowing it to be used to spend any
previous output for a particular witness program. For example, if Alice
receives two outputs for the same amount to the same witness program
(e.g. requiring a single signature from her wallet), a
(e.g., requiring a single signature from her wallet), a
+SIGHASH_ANYPREVOUT+ signature for spending either one of those outputs
could be copied and used to spend the other output to the same
destination.
A signature using +SIGHASH_ANYPREVOUTANYSCRIPT+ would not
commit to the outpoint, the amount, the witness program, or the
specific leaf in the taproot merkle tree (script tree), allowing it to spend any previous output
which the signature could satisfy. For example, if Alice received two
outputs for different amounts and different witness programs (e.g. one
specific leaf in the taproot merkle tree (script tree), allowing it to spend any previous output that the signature could satisfy. For example, if Alice received two
outputs for different amounts and different witness programs (e.g., one
requiring a single signature and another requiring her signature plus some
other data), a +SIGHASH_ANYPREVOUTANYSCRIPT+ signature for spending
either one of those outputs could be copied and used to spend the other
output to the same destination (assuming the extra data for the second
output was known).
The main expected use for the two SIGHASH_ANYPREVOUT opcodes is improved
The main expected use for the two ++SIGHASH_ANYPREVOUT++ opcodes is improved
payment channels, such as those used in the Lightning Network, although
several other uses have been described.
@ -236,27 +232,27 @@ several other uses have been described.
====
You will not often see +SIGHASH+ flags presented as an option in a user's
wallet application. Simple wallet applications
sign with +SIGHASH_ALL+ flags. More sophisticated applications, such as
sign with [.keep-together]#+SIGHASH_ALL+# flags. More sophisticated applications, such as
Lightning Network nodes, may use alternative +SIGHASH+ flags, but they
use protocols that have been extensively reviewed to understand the
influence of the alternative flags.
====
[[schnorr_signatures]]
=== Schnorr signatures
=== Schnorr Signatures
In 1989, Claus Schnorr published a paper describing the signature
algorithm that's become eponymous with him. The algorithm isn't
specific to the Elliptic Curve Cryptography (ECC) that Bitcoin and many
specific to the elliptic curve cryptography (ECC) that Bitcoin and many
other applications use, although it is perhaps most strongly associated
with ECC today. Schnorr signatures have a number of nice properties:
Provable security::
A mathematical proof of the security of schnorr signatures depends on
only the difficulty of solving the Discrete Logarithm Problem (DLP),
particularly for Elliptic Curves (EC) for Bitcoin, and the ability of
particularly for elliptic curves (EC) for Bitcoin, and the ability of
a hash function (like the SHA256 function used in Bitcoin) to produce
unpredictable values, called the Random Oracle Model (ROM). Other
unpredictable values, called the random oracle model (ROM). Other
signature algorithms have additional dependencies or require much
larger public keys or signatures for equivalent security to
ECC-Schnorr (when the threat is defined as classical computers; other
@ -269,12 +265,12 @@ Linearity::
properties. The first property is that summing together two or more
variables and then running a function on that sum will produce the
same value as running the function on each of the variables
independently and then summing together the results, e.g.
independently and then summing together the results, e.g.,
+f(x + y + z) == f(x) + f(y) + f(z)+; this property is called
_additivity_. The second property is that multiplying a variable and
then running a function on that product will produce the same value as
running the function on the variable and then multiplying it by the
same amount, e.g. +f(a * x) == a * f(x)+; this property is called
same amount, e.g., +f(a * x) == a * f(x)+; this property is called
_homogeneity of degree 1_.
+
In cryptographic operations, some functions may be private (such
@ -285,7 +281,7 @@ In cryptographic operations, some functions may be private (such
specific benefits of linearity in schnorr signatures in
<<schnorr_multisignatures>> and <<schnorr_threshold_signatures>>.
Batch Verification::
Batch verification::
When used in a certain way (which Bitcoin does), one consequence of
schnorr's linearity is that it's relatively straightforward to verify
more than one schnorr signature at the same time in less time than it
@ -297,17 +293,16 @@ Batch Verification::
Later in this chapter, we'll describe the schnorr signature algorithm
exactly as it's used in Bitcoin, but we're going to start with a
simplified version of it and work our way towards the actual protocol in
simplified version of it and work our way toward the actual protocol in
stages.
Alice starts by choosing a large random number (+x+), which we call her
_private key_. She also knows a public point on Bitcoin's Elliptic
Curve (EC) called the Generator (+G+) (see <<public_key_derivation>>). Alice uses EC
_private key_. She also knows a public point on Bitcoin's elliptic
curve called the Generator (+G+) (see <<public_key_derivation>>). Alice uses EC
multiplication to multiply +G+ by her private key +x+, in which case +x+
is called a _scalar_ because it scales up +G+. The result is +xG+,
which we call Alice's _public key_. Alice gives her public key to Bob.
Even though Bob also knows +G+, the Discrete Logarithm Problem (DLP)
prevents Bob from being able to divide +xG+ by +G+ to derive Alice's
Even though Bob also knows +G+, the Discrete Logarithm Problem prevents Bob from being able to divide +xG+ by +G+ to derive Alice's
private key.
At some later time, Bob wants Alice to identify herself by proving
@ -330,7 +325,7 @@ process:
be used to multiply an EC point.
3. Alice now has the numbers (scalars) +x+, +k+, and +e+. She combines
them together to produce a final scalar +s+ using the formula:
them together to produce a final scalar +s+ using the formula
+s = k + ex+. She gives +s+ to Bob.
4. Bob now knows the scalars +s+ and +e+, but not +x+ or +k+. However,
@ -339,11 +334,10 @@ process:
the operation Alice performed: +sG == kG + exG+. If that is equal,
then Bob can be sure that Alice knew +x+ when she generated +s+.
.Schnorr identity protocol with integers instead of points
.Schnorr Identity Protocol with Integers Instead of Points
****
It might be easier to understand the interactive schnorr identity
protocol if we create an insecure oversimplification by substituting each of the values above
(including +G+) with simple integers instead of points on a elliptic curve.
protocol if we create an insecure oversimplification by substituting each of the preceding values (including +G+) with simple integers instead of points on an elliptic curve.
For example, we'll use the prime numbers starting with 3:
Setup: Alice chooses +x=3+ as her private key. She multiplies it by the
@ -364,7 +358,7 @@ generator +G=5+ to get her public key +xG=15+. She gives Bob +15+.
Of course, this is an oversimplified example. When working with simple
integers, we can divide products by the generator +G+ to get the
underlying scalar, which isn't secure. This is why a critical property
of the Elliptic Curve Cryptography (ECC) used in Bitcoin is that
of the elliptic curve cryptography used in Bitcoin is that
multiplication is easy but division by a point on the curve is impractical. Also, with numbers
this small, finding underlying values (or valid substitutes) through
brute force is easy; the numbers used in Bitcoin are much larger.
@ -373,7 +367,8 @@ brute force is easy; the numbers used in Bitcoin are much larger.
Let's discuss some of the features of the interactive schnorr
identity protocol that make it secure:
- The nonce (+k+). In step 1, Alice chooses a number that Bob doesn't
The nonce (+k+)::
In step 1, Alice chooses a number that Bob doesn't
know and can't guess and gives him the scaled form of that number,
+kG+. At that point, Bob also already has her public key (+xG+),
which is the scaled form of +x+, her private key. That means when Bob is working on
@ -387,7 +382,8 @@ identity protocol that make it secure:
be able to leverage that into figuring out Alice's private key. See
<<nonce_warning>> for more details.
- The challenge scalar (+e+). Bob waits to receive Alice's public nonce
The challenge scalar (+e+)::
Bob waits to receive Alice's public nonce
and then proceeds in step 2 to give her a number (the challenge
scalar) that Alice didn't previously know and couldn't have guessed.
It's critical that Bob only give her the challenge scalar after she
@ -395,8 +391,8 @@ identity protocol that make it secure:
who didn't know +x+ wanted to impersonate Alice, and Bob accidentally
gave them the challenge scalar +e+ before they told him the public
nonce +kG+. This allows the impersonator to change parameters on both sides of
the equation that Bob will use for verification, +sG == kG + exG+,
specifically they can change both +sG+ and +kG+. Think about a
the equation that Bob will use for verification, +sG == kG + exG+;
specifically, they can change both +sG+ and +kG+. Think about a
simplified form of that expression: x = y + a. If you can change both
+x+ and +y+, you can cancel out +a+ using +x' = (x - a) + a+. Any
value you choose for +x+ will now satisfy the equation. For the
@ -426,7 +422,7 @@ node that wants to authenticate her.
A simple technique, known as the Fiat-Shamir transform after its
discoverers, can turn the schnorr interactive identity protocol
into a non-interactive digital signature scheme. Recall the importance
into a noninteractive digital signature scheme. Recall the importance
of steps 1 and 2--including that they be performed in order. Alice must
commit to an unpredictable nonce; Bob must give Alice an unpredictable
challenge scalar only after he has received her commitment. Recall also
@ -460,9 +456,9 @@ We've now defined a version of the schnorr signature protocol, but
there's one more thing we need to do to address a Bitcoin-specific
concern. In BIP32 key derivation, as described in
<<public_child_key_derivation>>, the algorithm for unhardened derivation
takes a public key and adds to it a non-secret value to produce a
takes a public key and adds to it a nonsecret value to produce a
derived public key. That means it's also possible to add that
non-secret value to a valid signature for one key to produce a signature
nonsecret value to a valid signature for one key to produce a signature
for a related key. That related signature is valid but it wasn't
authorized by the person possessing the private key, which is a major
security failure. To protect BIP32 unhardened derivation and
@ -486,13 +482,13 @@ generate a private key from a large random seed value). She uses the
parameters defined in secp256k1 (see <<elliptic_curve>>) to multiply the
generator +G+ by her scalar +x+, producing +xG+ (her public key). She
gives her public key to everyone who will later authenticate her Bitcoin
transactions (e.g. by having +xG+ included in a transaction output). When
transactions (e.g., by having +xG+ included in a transaction output). When
she's ready to spend, she begins generating her signature:
1. Alice chooses a large random private nonce +k+ and derives the public
nonce +kG+.
2. She chooses her message +m+ (e.g. transaction data) and generates the
2. She chooses her message +m+ (e.g., transaction data) and generates the
challenge scalar +e = hash(kG || xG || m)+.
3. She produces the scalar +s = k + ex+. The two values +kG+ and +s+
@ -502,19 +498,19 @@ she's ready to spend, she begins generating her signature:
the witness structure of her spending transaction and then relaying that
transaction to full nodes.
4. The verifiers (e.g. full nodes) use +s+ to derive +sG+ and then
4. The verifiers (e.g., full nodes) use +s+ to derive +sG+ and then
verify that +sG == kG + hash(kG || xG || m)*xG+. If the equation is
valid, Alice proved that she knows her private key +x+ (without
revealing it) and committed to the message +m+ (containing the
transaction data).
==== Serialization of schnorr signatures
==== Serialization of Schnorr Signatures
A schnorr signature consists of two values, +kG+ and +s+. The value
+kG+ is a point on Bitcoin's elliptic curve (called secp256k1) and so
would normally be represented by two 32-byte coordinates, e.g. +(x,y)+.
However, only the x coordinate is needed, so only that value is
included. When you see +kG+ below, note that it's only that point's x
would normally be represented by two 32-byte coordinates, e.g., +(x,y)+.
However, only the _x_ coordinate is needed, so only that value is
included. When you see +kG+ in schnorr signatures for Bitcoin, note that it's only that point's _x_
coordinate.
The value +s+ is a scalar (a number meant to multiply other numbers). For
@ -522,8 +518,8 @@ Bitcoin's secp256k1 curve, it can never be more than 32 bytes long.
Although both +kG+ and +s+ can sometimes be values that can be
represented with fewer than 32 bytes, it's improbable that they'd be
much smaller than 32 bytes, and so they're serialized as two 32byte
values (i.e., values smaller than 32 bytes have leading zeroes).
much smaller than 32 bytes, and so they're serialized as two 32-byte
values (i.e., values smaller than 32 bytes have leading zeros).
They're serialized in the order of +kG+ and then +s+, producing exactly
64 bytes.
@ -541,7 +537,7 @@ the serialization used for ECDSA signatures described in
<<serialization_of_signatures_der>>.
[[schnorr_multisignatures]]
==== Schnorr-based scriptless multisignatures
==== Schnorr-based Scriptless Multisignatures
In the single-signature schnorr protocol described in <<schnorr_signatures>>, Alice
uses a signature (+kG+, +s+) to publicly prove her knowledge of her
@ -560,9 +556,9 @@ that are believed to be secure.
====
Alice and Bob need to derive the public key for +x+, which is +xG+.
Since it's possible to use Elliptic Curve (EC) operations to add two EC
Since it's possible to use elliptic curve operations to add two EC
points together, they start by Alice deriving +yG+ and Bob deriving
+zG+. Then then add them together to create +xG = yG + zG+. The point
+zG+. They then add them together to create +xG = yG + zG+. The point
+xG+ is their _aggregated public key_. To create a signature, they begin the
Bitcoin is structured as a peer-to-peer network architecture on
top of the internet. The term peer-to-peer, or P2P, means that the
full nodes which participate in the network are peers to each other, that
full nodes that participate in the network are peers to each other, that
they can all perform the same functions, and that there are no "special" nodes.
The network nodes
interconnect in a mesh network with a "flat" topology. There is no
@ -52,7 +52,7 @@ only a subset of the blockchain and partly verify transactions using a method
called _simplified payment verification_, or SPV. These clients are known as lightweight clients.
Miners compete to create new blocks by
running specialized hardware to solve the Proof-of-Work algorithm. Some
running specialized hardware to solve the proof-of-work algorithm. Some
miners operate full nodes, validating every block on the
blockchain, while others are clients participating in pool
mining and depending on a pool server to provide them with work.
@ -119,7 +119,7 @@ some of those transactions are confirmed in a new block, the node
doesn't need to receive a second copy of those transactions.
Instead of receiving redundant unconfirmed transactions, compact blocks
allows a peer to instead send a short 6-byte identifier for each transaction.
allow a peer to instead send a short 6-byte identifier for each transaction.
When your node receives a compact block with one or more identifiers, it
checks its mempool for those transactions and uses them if they are
found. For any transaction that isn't found in your local node's
@ -133,7 +133,7 @@ always sends a block's coinbase transaction.
If the remote peer guesses correctly about what transactions your node
has in its mempool, and which it does not, it will send a block nearly
as efficiently as is theoretically possible (for a typical block, it'll
be between 97% to 99% efficient).
be between 97% and 99% efficient).
[TIP]
====
@ -175,7 +175,7 @@ quickly announcing blocks).
// released into the public domain by Nicolas Dorier
[[bip152_illustration]]
.BIP152 modes compared (from BIP152). They grey bar indicates the time it takes the node to validate the block
.BIP152 modes compared (from BIP152). The gray bar indicates the time it takes the node to validate the block.
image::images/mbc3_1002.png["BIP152"]
The names of the two methods (which are taken from BIP152) can be a bit
@ -186,15 +186,15 @@ relay before compact blocks were implemented.
=== Private Block Relay Networks
Although compact blocks go a long way towards minimizing the time it
Although compact blocks go a long way toward minimizing the time it
takes for blocks to propagate across the network,
it's possible to minimize latency further. Unlike
compact blocks, though, the other solutions involve tradeoffs that
compact blocks, though, the other solutions involve trade-offs that
make them unavailable or unsuitable for the public P2P relay network.
For that reason, there has been experimentation with private relay
networks for blocks.
One simple technique is to pre-select a route between endpoints. For
One simple technique is to preselect a route between endpoints. For
example, a relay network with servers running in datacenters near major
trans-oceanic fiber optic lines might be able to forward new blocks
faster than waiting for the block to arrive at the node run by some home
@ -214,18 +214,18 @@ because we mostly use protocols that automatically re-request the
missing data. However, requesting missing data triples the time to
receive it. For example:
1. Alice sends some data to Bob
1. Alice sends some data to Bob.
2. Bob doesn't receive the data (or it is damaged). Bob re-requests
the data from Alice
3. Alice sends the data again
the data from Alice.
3. Alice sends the data again.
A third technique is to assume all nodes receiving the data have
almost all of the same transactions in their mempool, so they can all
accept the same compact block. That not only saves us time computing
a compact block at each hop but it means that all each hop can simply
a compact block at each hop, but it means that all each hop can simply
relay the FEC packets to the next hop even before validating them.
The tradeoff for each of the above methods is that they work well with
The trade-off for each of the preceding methods is that they work well with
centralization but not in a decentralized network where individual nodes
can't trust other nodes. Servers in datacenters cost money and can
often be accessed by operators of the datacenter, making them less
@ -270,7 +270,7 @@ information, including:
+nTime+:: The current time
+addrYou+:: The IP address of the remote node as seen from this node
+addrMe+:: The IP address of the local node, as discovered by the local node
+subver+:: A sub-version showing the type of software running on this node (e.g., pass:[<span class="keep-together"><code>/Satoshi:0.9.2.1/</code></span>])
+subver+:: A subversion showing the type of software running on this node (e.g., pass:[<span class="keep-together"><code>/Satoshi:0.9.2.1/</code></span>])
+BestHeight+:: The block height of this node's blockchain
+fRelay+:: A field added by BIP37 for requesting not to receive unconfirmed transactions
@ -435,8 +435,8 @@ decisions. However, the most common
implementation is Bitcoin Core.
More than 95% of full nodes on the Bitcoin network run
various versions of Bitcoin Core. It is identified as "Satoshi" in the
sub-version string sent in the +version+ message and shown by the
command +getpeerinfo+ as we saw earlier; for example, +/Satoshi:24.0.1/+.
subversion string sent in the +version+ message and shown by the
command +getpeerinfo+ as we saw earlier; for example, [.keep-together]#+/Satoshi:24.0.1/+#.
=== Exchanging "Inventory"
@ -458,7 +458,7 @@ a +getheaders+ message that contains the hash of the top
block on their local blockchain. One of the peers will be able to
identify the received hash as belonging to a block that is not at the
top, but rather belongs to an older block, thus deducing that its own
local blockchain is longer than its peer's.
local blockchain is longer than the remote node's blockchain.
The peer that has the longer blockchain has more blocks than the other
node and can identify which headers the other node needs in order to
@ -474,7 +474,7 @@ connections to peers that are significantly slower than the average in
order to find newer (and possibly faster) peers.
Let's assume, for example, that a node only has the genesis block. It
will then receive an +headers+ message from its peers containing the headers
will then receive a +headers+ message from its peers containing the headers
of the next 2,000 blocks in the chain. It will start requesting blocks
from all of its connected peers, keeping a queue of up to 1,024 blocks.
Blocks need to be validated in order, so if the oldest block in the
@ -542,7 +542,7 @@ on top of it is proof, by proxy, that the transaction actually exists.
A lightweight client cannot normally be persuaded that a transaction exists in a block
when the transaction does not in fact exist. The lightweight client establishes
the existence of a transaction in a block by requesting a merkle path
proof and by validating the Proof-of-Work in the chain of blocks.
proof and by validating the proof of work in the chain of blocks.
However, a transaction's existence can be "hidden" from a lightweight client. A
lightweight client can definitely verify that a transaction exists but cannot
verify that a transaction, such as a double-spend of the same UTXO,
@ -565,7 +565,7 @@ node.
====
A full node verifies a transaction by checking the entire chain of
thousands of blocks below it in order to guarantee that the UTXO exists
and is not spent, whereas an lightweight client only proves that a transaction
and is not spent, whereas a lightweight client only proves that a transaction
exists and checks that the block containing that transaction is
buried by a handful of blocks above it.
====
@ -607,19 +607,19 @@ transactions matching a specific pattern, without revealing exactly
which addresses, keys, or transactions they are searching for.
In our previous analogy, a tourist without a map is asking for
directions to a specific address, "23 Church St." If she asks strangers
for directions to this street, she inadvertently reveals her
directions to a specific address, "23 Church St." If they ask a stranger
for directions to this street, they inadvertently reveal their
destination. A bloom filter is like asking, "Are there any streets in
this neighborhood whose name ends in R-C-H?" A question like that
reveals slightly less about the desired destination than asking for "23
Church St." Using this technique, a tourist could specify the desired
address in more detail such as "ending in U-R-C-H" or less detail as
address in more detail such as "ending in U-R-C-H" or less detail such as
"ending in H." By varying the precision of the search, the tourist
reveals more or less information, at the expense of getting more or less
specific results. If she asks a less specific pattern, she gets a lot
specific results. If they ask for a less specific pattern, they get a lot
more possible addresses and better privacy, but many of the results are
irrelevant. If she asks for a very specific pattern, she gets fewer
results but loses privacy.
irrelevant. If they ask for a very specific pattern, they get fewer
results but lose privacy.
Bloom filters serve this function by allowing a lightweight client to specify a
search pattern for transactions that can be tuned toward precision or
@ -782,14 +782,14 @@ that they didn't offer very much privacy. A full node receiving a bloom
filter from a peer could apply that filter to the entire blockchain to
find all of the client's transactions (plus false positives). It could
then look for patterns and relationships between the transactions.
Randomly-selected false positive transactions would be unlikely to have
Randomlyselected false positive transactions would be unlikely to have
a parent-child relationship from output to input, but transactions from
the user's wallet would be very likely to have that relationship. If
all of the related transactions have certain characteristics, such as
at least one P2PKH output, then transactions without that characteristic
can be assumed not to belong to the wallet.
It was also discovered that specially-constructed filters could force
It was also discovered that speciallyconstructed filters could force
the full nodes that processed them to perform a large amount of work,
which could lead to denial-of-service attacks.
@ -814,7 +814,7 @@ the entire block.
[NOTE]
====
Despite the similarities in names, BIP152 _compact blocks_ and
BIP157/158 _compact block filters_ are unrelated
BIP157/158 _compact block filters_ are unrelated.
====
This allows nodes to create a single filter for every block, which they
@ -863,8 +863,8 @@ numerical order:
Then, Alice sends the first number. For the remaining numbers, she
sends the difference between that number and the preceding number. For
example, for the second number, she sends 97 (476 - 379); for the third
number, she sends 177 (653 - 476); and so on:
example, for the second number, she sends 97 (476 – 379); for the third
number, she sends 177 (653 – 476); and so on:
----
379
@ -885,8 +885,8 @@ more numbers we select, the smaller the average (mean) size of the
differences. That means the amount of data we need to transfer doesn't
increase as fast as the length of our list increases (up to a point).
Even more usefully, the length of the randomly-selected numbers in a
list of differences is naturally biased towards smaller lengths.
Even more usefully, the length of the randomlyselected numbers in a
list of differences is naturally biased toward smaller lengths.
Consider selecting two random numbers from 1 to 6; this is the same
as rolling two dice. There are 36 distinct combinations of two dice:
@ -939,17 +939,17 @@ Golomb coding provides that facility. Rice coding is a subset of Golomb
coding that's more convenient to use in some situations, including the
application of Bitcoin block filters.
==== What data to include in a block filter
==== What Data to Include in a Block Filter
Our primary goal is to allow wallets to learn whether a block contains a
transaction affecting that wallet. For a wallet to be effective, it
needs to learn two types of information:
1. When it has received money. Specifically, when a transaction
. When it has received money. Specifically, when a transaction
output contains a script that the wallet controls (such as by
controlling the authorized private key).
2. When it has spent money. Specifically, when a transaction input
. When it has spent money. Specifically, when a transaction input
references a previous transaction output that the wallet controlled.
A secondary goal during the design of compact block filters was to allow
@ -963,9 +963,9 @@ accurate filter.
For both the primary and secondary goals to be met, a filter would need
to reference two types of information:
1. The script for every output in every transaction in a block.
* The script for every output in every transaction in a block.
2. The outpoint for every input in every transaction in a block.
* The outpoint for every input in every transaction in a block.
An early design for compact block filters included both of those pieces
of information, but it was realized that there was a more efficient way
@ -973,15 +973,15 @@ to accomplish the primary goal if we sacrificed the secondary goal. In
the new design, a block filter would still references two types of
information, but they'd be more closely related:
1. As before, the script for every output in every transaction in a
* As before, the script for every output in every transaction in a
block.
2. In a change, it would also reference the script of the output
* In a change, it would also reference the script of the output
referenced by the outpoint for every input in every transaction in a
block. In other words, the output script being spent.
This had several advantages. First, it meant that wallets didn't need
to track outpoints; they could instead just scan for the the
to track outpoints; they could instead just scan for the
output scripts to which they expected to receive money. Second, any time a
later transaction in a block spends the output of an earlier
transaction in the same block, they'll both reference the same
@ -999,12 +999,12 @@ to verify a block filter was built correctly. However, there is an
alternative that can help a client detect if a peer is lying to it:
obtaining the same filter from multiple peers.
==== Downloading block filters from multiple peers
==== Downloading Block Filters from Multiple Peers
A peer can provide a wallet with an inaccurate filter. There's two ways
A peer can provide a wallet with an inaccurate filter. There are two ways
to create an inaccurate filter. The peer can create a filter that
references transactions that don't actually appear in the associated
block (a false positive). Alternatively, the peer can crate a filter
block (a false positive). Alternatively, the peer can create a filter
that doesn't reference transactions that do actually appear in the
associated block (a false negative).
@ -1021,7 +1021,7 @@ block, the client can download all of them. It can then also download
the associated block. If the block contains any transaction related to
the wallet that is not part of one of the filters, then the wallet can
be sure that whichever peer created that filter was
inaccurate--Golomb-Rice Coded Sets (GCSes) will always include a
inaccurate--Golomb-Rice Coded Sets will always include a
potential match.
Alternatively, if the block doesn't contain a transaction that the
@ -1035,7 +1035,7 @@ designed to use, the wallet can stop using that peer. In most cases,
the only consequence of the inaccurate filter is that the wallet uses
more bandwidth than expected.
==== Reducing bandwidth with lossy encoding
==== Reducing Bandwidth with Lossy Encoding
The data about the transactions in a block that we want to communicate
is an output script. Output scripts vary in length and follow patterns,
@ -1051,7 +1051,7 @@ faster and more configurable non-cryptographic hash functions, such as
the SipHash function we'll use for compact block filters.
The details of the algorithm used are described in BIP158, but the gist
is that each output script is reduced to a 64bit commitment using
is that each output script is reduced to a 64-bit commitment using
SipHash and some arithmetic operations. You can think of this as
taking a set of large numbers and truncating them to shorter numbers, a
process that loses data (so it's called _lossy encoding_). By losing
@ -1059,9 +1059,9 @@ some information, we don't need to store as much information later,
which saves space. In this case, we go from a typical output script
that's 160 bits or longer down to just 64 bits.
==== Using compact block filters
==== Using Compact Block Filters
The 64 bit value for every commitment to an output script in a block are
The 64-bit values for every commitment to an output script in a block are
sorted, duplicate entries are removed, and the GCS is constructed by
finding the differences (deltas) between each entry. That compact block
filter is then distributed by peers to their clients (such as wallets).
@ -1074,23 +1074,23 @@ the filter.
Recall our example of the lossiness of compact block filters being
similar to truncating a number. Imagine a client is looking for a block
that contains the number 123456 and that an an accurate (but lossy)
that contains the number 123456 and that an accurate (but lossy)
compact block filter contains the number 1234. When a client sees that
1234, it will download the associated block.
There's a 100% guarantee that an accurate filter containing 1234 will
allow a client to learn about a block containing 123456, called a _true
positive_. However, there's also chance that the block might contain
123400, 123401, or almost a hundred other entries that are not when the
123400, 123401, or almost a hundred other entries that are not what the
client is looking for (in this example), called a _false positive_.
A 100% true positive match rate is great. It means that a wallet can
depend on compact block filters to find every transaction affecting that
wallet. A non-zero false positive rate means that the wallet will end
wallet. A nonzero false positive rate means that the wallet will end
up downloading some blocks that don't contain transactions interesting
to the wallet. The main consequence of this is that the client will use
extra bandwidth, which is not a huge problem. The actual
false-positive rate for BIP158 compact block filters is very low, so
falsepositive rate for BIP158 compact block filters is very low, so
it's not a major problem. A false positive rate can also help improve a
client's privacy, as it does with bloom filters, although anyone wanting
the best possible privacy should still use their own full node.
@ -1129,7 +1129,7 @@ at the time of writing.
As a way to increase the privacy and security of the Bitcoin P2P
network, there is a solution that provides encryption of the
communications: _Tor Transport_.
communications: _Tor transport_.
==== Tor Transport
@ -1155,7 +1155,7 @@ debugging for the Tor service like this:
$ bitcoind --daemon --debug=tor
----
You should see "tor: ADD_ONION successful" in the logs, indicating that
You should see ++tor: ADD_ONION successful++ in the logs, indicating that
Bitcoin Core has added a hidden service to the Tor network.
You can find more instructions on running Bitcoin Core as a Tor hidden
@ -1163,7 +1163,7 @@ service in the Bitcoin Core documentation (_docs/tor.md_) and various
online tutorials.
[[mempool]]
=== Mempools and orphan pools
=== Mempools and Orphan Pools
Almost every node on the Bitcoin
network maintains a temporary list of unconfirmed transactions called
@ -1190,10 +1190,10 @@ of the newly added transaction, which is no longer an orphan, the
process is repeated recursively looking for any further descendants,
until no more descendants are found. Through this process, the arrival
of a parent transaction triggers a cascade reconstruction of an entire
chain of interdependent transactions by re-uniting the orphans with
chain of interdependent transactions by reuniting the orphans with
their parents all the way down the chain.
Some implementations of the Bitcoin also maintain an UTXO
Some implementations of Bitcoin also maintain a UTXO
database, which is the set of all unspent outputs on the
blockchain. This represents a different set of data from the mempool. Unlike the
mempool and orphan pools, the UTXO database
@ -1204,11 +1204,11 @@ table on persistent storage.
Whereas the mempool and orphan pools represent a single node's local
perspective and might vary significantly from node to node depending
upon when the node was started or restarted, the UTXO database represents
on when the node was started or restarted, the UTXO database represents
the emergent consensus of the network and therefore will not usually
vary between nodes.
Now that we have a understanding of many of the data types and
Now that we have an understanding of many of the data types and
structures used by nodes and clients to send data across the Bitcoin
network, it's time to look at the software that's responsible for
Clare Laylock
8 months ago