CH09: add info about fees

- New introduction to fees
- More detail about how the fee market works
- Adds RBF and CPFP fee bumping
- Adds transaction pinning
- Adds package relay
- Adds CPFP carve out
- Small edits to 'Adding fees'
- Tiny edits to fee sniping

ch02.asciidoc

@@ -599,6 +599,7 @@ mine for bitcoin 24 hours a day. The company pays its electricity costs
 by selling the bitcoin it is able to generate from mining, creating some
 income from the profits.
 
+[[confirmation_score]]
 === Mining Transactions in Blocks
 
 ((("blocks", "mining transactions in")))New transactions are constantly

chapters/fees.adoc

@@ -1,134 +1,669 @@
-//FIXME
-// - Economic incentives for segwit
-// - RBF (opt-in, full)
-// - CPFP
-// - CPFP carve out
-// - Package relay
 [[tx_fees]]
-==== Transaction Fees
+== Transaction Fees
 
-((("transactions", "outputs and inputs", "transaction fees")))((("fees",
+The digital signature we saw Alice create in <<c_signatures>> only
-"transaction fees")))((("mining and consensus", "rewards and
+proves that she knows her private key and that she committed to a
-fees")))Most transactions include transaction fees, which compensate the
+transaction that pays Bob.  She can create another signature that
-bitcoin miners for securing the network. Fees also serve as a security
+instead commits to a transaction paying Carol--a transaction that spends
-mechanism themselves, by making it economically infeasible for attackers
+the same output (bitcoins) that she used to pay Bob.  Those two
-to flood the network with transactions. Mining and the fees and rewards
+transactions are now _conflicting transactions_ because only one
-collected by miners are discussed in more detail in <<mining>>.
+transaction spending a particular output can be included in the valid
+blockchain with the most proof of work--the blockchain that full nodes
+use to determine which keys control which bitcoins.
 
-This section examines how transaction fees are included in a typical
+To protect himself against conflicting transactions, it would be wise
-transaction. Most wallets calculate and include transaction fees
+for Bob to wait until the transaction from Alice is included in the
-automatically. However, if you are constructing transactions
+blockchain before he considers the money he received as his to spend.
-programmatically, or using a command-line interface, you must manually
+He may want to wait even longer than that, as we discussed in
-account for and include these fees.
+<<confirmation_score>>.  For Alice's transaction to be included in the
+blockchain, it must be included in a _block_ of transactions.  There are
+a limited number of blocks produced in a given amount of time and each
+block only has a limited amount of space.  Only the miner who creates
+that block gets to choose which transactions to include.  Miners may
+select transactions by any criteria they want, including refusing to
+include any transactions at all.
 
-Transaction fees serve as an incentive to include (mine) a transaction
+[NOTE]
-into the next block and also as a disincentive against abuse of the
-system by imposing a small cost on every transaction. Transaction fees
-are collected by the miner who mines the block that records the
-transaction on the blockchain.
-
-Transaction fees are calculated based on the size of the transaction in
-kilobytes, not the value of the transaction in bitcoin. Overall,
-transaction fees are set based on market forces within the Bitcoin
-network. Miners prioritize transactions based on many different
-criteria, including fees, and might even process transactions for free
-under certain circumstances. Transaction fees affect the processing
-priority, meaning that a transaction with sufficient fees is likely to
-be included in the next block mined, whereas a transaction with
-insufficient or no fees might be delayed, processed on a best-effort
-basis after a few blocks, or not processed at all. Transaction fees are
-not mandatory, and transactions without fees might be processed
-eventually; however, including transaction fees encourages priority
-processing.
-
-Over time, the way transaction fees are calculated and the effect they
-have on transaction prioritization has evolved. At first, transaction
-fees were fixed and constant across the network. Gradually, the fee
-structure relaxed and may be influenced by market forces, based on
-network capacity and transaction volume. Since at least the beginning of
-2016, capacity limits in bitcoin have created competition between
-transactions, resulting in higher fees and effectively making free
-transactions a thing of the past. Zero fee or very low fee transactions
-rarely get mined and sometimes will not even be propagated across the
-network.
-
-((("fees", "fee relay policies")))((("minrelaytxfee option")))In Bitcoin
-Core, fee relay policies are set by the +minrelaytxfee+ option. The
-current default +minrelaytxfee+ is 0.00001 bitcoin or a hundredth of a
-millibitcoin per kilobyte. Therefore, by default, transactions with a
-fee less than 0.00001 bitcoin are treated as free and are only relayed
-if there is space in the mempool; otherwise, they are dropped. Bitcoin
-nodes can override the default fee relay policy by adjusting the value
-of +minrelaytxfee+.
-
-((("dynamic fees")))((("fees", "dynamic fees")))Any bitcoin service that
-creates transactions, including wallets, exchanges, retail applications,
-etc., _must_ implement dynamic fees. Dynamic fees can be implemented
-through a third-party fee estimation service or with a built-in fee
-estimation algorithm. If you're unsure, begin with a third-party service
-and as you gain experience design and implement your own algorithm if
-you wish to remove the third-party dependency.
-
-Fee estimation algorithms calculate the appropriate fee, based on
-capacity and the fees offered by "competing" transactions. These
-algorithms range from simplistic (average or median fee in the last
-block) to sophisticated (statistical analysis). They estimate the
-necessary fee (in satoshis per byte) that will give a transaction a high
-probability of being selected and included within a certain number of
-blocks. Most services offer users the option of choosing high, medium,
-or low priority fees. High priority means users pay higher fees but the
-transaction is likely to be included in the next block. Medium and low
-priority means users pay lower transaction fees but the transactions may
-take much longer to confirm.
-
-((("bitcoinfees (third-party service)")))Many wallet applications use
-third-party services for fee calculations. One popular service is
-http://bitcoinfees.21.co/[_http://bitcoinfees.21.co_], which provides an
-API and a visual chart showing the fee in satoshi/byte for different
-priorities.
-
-[TIP]
 ====
-((("static fees")))((("fees", "static fees")))Static fees are no longer
+When we say "transactions" in this chapter, we refer to every
-viable on the Bitcoin network. Wallets that set static fees will produce
+transaction in a block except for the first transaction.  The first
-a poor user experience as transactions will often get "stuck" and remain
+transaction in a block is a _coinbase transaction_, described in
-unconfirmed. Users who don't understand bitcoin transactions and fees
+<<coinbase_transactions>>, which allows the miner of the block to
-are dismayed by "stuck" transactions because they think they've lost
+collect their reward for producing the block.  Unlike other
-their money.
+transactions, a coinbase transaction doesn't spend the output of a
+previous transaction and is also an exception to several other rules
+that apply to other transactions.  Coinbase transactions don't pay
+transaction fees, don't need to be fee bumped, aren't subject to
+transaction pinning, and are largely uninteresting to the following
+discussion about fees--so we're going to ignore them in this chapter.
 ====
 
-The chart in <<bitcoinfees21co>> shows the real-time estimate of fees in
+The criteria that almost all miners use to select which transactions to
-10 satoshi/byte increments and the expected confirmation time (in
+include in their blocks is to maximize their revenue.  Bitcoin was
-minutes and number of blocks) for transactions with fees in each range.
+specifically designed to accommodate this by providing a mechanism that
-For each fee range (e.g., 61&#x2013;70 satoshi/byte), two horizontal
+allows a transaction to give money to the miner who includes that
-bars show the number of unconfirmed transactions (1405) and total number
+transaction in a block.  We call that mechanism _transaction fees_,
-of transactions in the past 24 hours (102,975), with fees in that range.
+although it's not a fee in the usual sense of that word.  It's not an
-Based on the graph, the recommended high-priority fee at this time was
+amount set by the protocol or by any particular miner---it's much more
-80 satoshi/byte, a fee likely to result in the transaction being mined
+like a bid in an auction.  The good being purchased is the portion of
-in the very next block (zero block delay). For perspective, the median
+limited space in a block that a transaction will consume.  Miners choose
-transaction size is 226 bytes, so the recommended fee for a transaction
+the set of transactions whose bids will allow them to earn the greatest
-size would be 18,080 satoshis (0.00018080 BTC).
+revenue.
 
-The fee estimation data can be retrieved via a simple HTTP REST API, at
+In this chapter, we'll explore various aspects of those
-https://bitcoinfees.21.co/api/v1/fees/recommended[https://bitcoinfees.21.co/api/v1/fees/recommended].
+bids--transaction fees--and how they influence the creation and
-For example, on the command line using the +curl+ command:
+management of Bitcoin transactions.
+
+=== Bids on space and time
+
+There's a chart taught in most economics courses that shows simplified
+supply and demand curves, shown in <<img_supply_demand>>.  In these
+simple scenarios, increasing aggregate demand or decreasing aggregate
+supply results in higher prices; alternatively, decreasing demand or
+increasing supply results in lower prices.
+
+[[img_supply_demand]]
+.Traditional supply and demand curves
+image::../images/supply-demand.png[Traditional supply and demand curves]
+
+It's possible for the supply of block space in Bitcoin to significantly
+change, but normally it's very consistent when averaged over the course
+of several hours or longer.  That means, for the purposes of considering
+transaction fees, price is almost entirely a function of aggregate demand,
+as shown in <<img_block_supply_demand>>.
+
+[[img_block_supply_demand]]
+.Bitcoin block space supply and demand
+image::../images/supply-demand-constant.png[Bitcoin block space supply and demand]
+
+When demand is less than supply, the transaction fees should be nearly
+zero.  Most full nodes which help relay transactions from users to
+miners require that every transaction they relay pay a minimum
+transaction fee.  This prevents their relay bandwidth being wasted.
+That minimum fee is the effective floor on transaction fees.  Note, the
+minimum transaction relay fee must be a fairly small amount or it would
+create an incentive for users to begin sending transactions to miners
+directly, which would be bad for mining decentralization.
+
+When demand exceeds supply, then the only thing that prevents fee costs
+from rising to infinity is that higher prices curtail demand.  Each time
+the price increases, the number of people who want to pay that price to
+include a transaction in a block decreases.  Conversely, when prices
+decrease, the the number of people willing to pay that price to confirm
+a transaction increases.  Engineers call this a _negative feedback
+loop_, as shown in <<img_negative_feedback>>.
+
+[[img_negative_feedback]]
+.Negative feedback loop
+image::../images/fee-negative-feedback.png[Negative feedback loop]
+
+In particular, people who want to send a transaction during a period of
+high fees will try to find an alternative.  There are many alternatives
+for most reasons people need to send a transaction, such as moving to an
+offchain payment system like Lightning Network, using an alternative
+payment network, or taking care to construct the smallest transaction
+possible.  But it's also possible to take advantage of the negative
+feedback loop to pay less fees.  All you need to do is create a
+transaction now which pays a fee that's equal to or higher than what
+miners will accept in the future--even if that price is below what they
+accept now.  As the market of other people adapts to the higher price by
+lowering demand, and as miners confirm all transactions paying a higher
+price, the price will drop until miners are willing to accept the
+lower fee you offered earlier.
+
+We can imagine four separate transactions that are equivalent except
+that each pays a different fee.  The transaction paying the highest fee
+gets confirmed first.  As the high prices result in lower demand, which
+subsequently leads to lower prices, the second highest fee transaction
+gets confirmed, as does the third.  Eventually the lowest fee
+transaction gets confirmed.  However, the lower the price becomes, the
+more demand there is to confirm transactions at that price, so a
+transaction paying too low of a fee might never be confirmed.  We can
+see this scenario illustrated in <<img_wait_times>>.
+
+[[img_wait_times]]
+.Wait times for different fee rates
+image::../images/fee-patience.png[Wait times for different feerates]
+
+In other words, you don't need to pay peak prices, or switch to another
+payment method, if you're willing to wait longer for your transaction to
+confirm.
+
+Miners keep a pool of transactions that they will attempt to mine,
+either presently or in the near future.  Full nodes which relay
+unconfirmed transactions will also keep their own pool.  In some
+Bitcoin implementations these pools are kept in memory and are called
+memory pools or _mempools_.  The mempools across most nodes tend to be
+very similar, so it's common to talk about _the mempool_ as if it was a
+single pool synchronized across multiple peers.  Although that's
+convenient most of the time, anyone evaluating potential problems with
+transaction relay (both accidental problems and deliberate attacks)
+needs to remember that every relay node keeps its own mempool.
+
+=== Who pays the transaction fee?
+
+Most payment systems involve some sort of fee for transacting, but
+often this fee is hidden from typical buyers.  For example, a merchant
+may advertise the same item for the same price whether you pay with cash
+or a credit card even though their payment processor may charge them
+a higher fee for credit transactions than their bank charges them for
+cash deposits.
+
+In Bitcoin, every spend of bitcoins must be authenticated (typically
+with a signature), so it's not possible for a transaction to pay a fee
+without the permission of the spender.  It is possible for the receiver
+of a transaction to pay a fee in a different transaction--and we'll see
+that in use later--but if we want a single transaction to pay its own
+fee, that fee needs to be something agreed upon by the spender.  It
+can't be hidden.
+
+Bitcoin transactions are designed so that it doesn't take any extra
+space in a transaction for a spender to commit to the fee it pays.  That
+means that, even though it's possible to pay the fee in a different
+transaction, it's most efficient (and thus cheapest) to pay the fee in a
+single transaction.
+
+Those technicalities leads us to design Bitcoin so that spenders are
+normally responsible for fees.
+
+That's not always what's most convenient for the situation.  In Bitcoin,
+the fee is a bid and the amount paid contributes to determining how long
+it will take the transaction to confirm.  Both spenders and receivers of
+a payment typically have an interest in having it confirming quickly, so
+normally allowing only spenders to choose fees can sometimes be a
+problem; we'll look at a solution to that problem in <<cpfp>>.  However,
+in many common payment flows, the parties with the highest desire to see a
+transaction confirm quickly--that is, the parties who'd be the most
+willing to pay higher fees--are the the spenders.
+
+For those reasons, both technical and practical, it is customary in
+Bitcoin for spenders to pay transaction fees.  There are exceptions,
+such as for merchants which accept unconfirmed transactions and in
+protocols that don't immediately broadcast transactions after they are
+signed (preventing the spender from being able to choose an appropriate
+fee for the current market).
+
+=== Fees and fee rates
+
+Each transaction only pays a single fee--it doesn't matter how large the
+transaction is.  However, the larger transactions become, the fewer of
+them a miner will be able to fit in a block.  For that reason, miners
+evaluate transactions the same way you might comparison shop between
+several equivalent items at the market: they divide quantity by price.
+Whereas you might divide the weight of several different bags of rice by
+their cost to find the lowest price per weight (best deal), miners
+divide the size of a transaction (also called its weight) by the fee to
+find the highest fee per weight (most revenue).  In Bitcoin, we use the
+term _fee rate_ for a transaction's size divided by weight.  Due to
+changes in Bitcoin over the years, fee rate can be expressed in
+different units:
+
+- Bytes/BTC (a legacy unit rarely used anymore)
+- Kilobytes/BTC (a legacy unit rarely used anymore)
+- vBytes/BTC (rarely used)
+- Kilo-vByte/BTC (used mainly in Bitcoin Core)
+- vByte/satoshi (most commonly used today)
+- weight/satoshi (also commonly used today)
+
+We recommend either the vByte/sat or weight/sat units for displaying
+feerates.
+
+[WARNING]
+====
+Be careful accepting input for feerates.  If a user copy and pastes a
+feerate printed in one enumerator into a field using a different
+enumerator, they could overpay fees by 1,000 times.  If they instead
+switch the denominator, they could theoretically overpay by 100,000,000
+times.  Wallets should make it hard for the user to pay an excessive
+feerate and may want to prompt the user to confirm any feerate that was
+not generated by the wallet itself using a trusted data source.
+
+An excessive fee, also called an _absurd fee_, is any feerate that's
+significantly higher than the amount that feerate estimators currently
+expect is necessary to get a transaction confirmed in the next block.
+Note that wallets should not entirely prevent users from choosing an
+excessive feerate--they should only make using such a feerate hard to do
+by accident.  There are legitimate reasons for users to overpay fees on
+rare occasions.
+====
+
+=== Estimating appropriate feerates
+
+We've established that you can pay a lower fee rate if you're willing to
+wait longer for your transaction to be confirmed, with the exception
+that paying too low of a fee rate could result in your transaction never
+confirming.  Because fee rates are bids in an open auction for block
+space, it's not possible to perfectly predict what fee rate you need to
+pay to get your transaction confirmed by a certain time.  However, we
+can generate a rough estimate based on what fee rates other transactions
+have paid in the recent past.
+
+A full node can record three pieces of information about each
+transactions it sees: the time (block height) when it first received
+that transaction, the block height when that transaction was confirmed,
+and the fee rate paid by that transaction.  By grouping together
+transactions that arrived at similar heights, were confirmed at similar
+heights, and which paid similar fees, we can calculate how many blocks it
+took to confirm transactions paying a certain fee rate.  We can then
+assume that a transaction paying a similar fee rate now will take a
+similar number of blocks to confirm.  Bitcoin Core includes a fee rate
+estimator that uses these principles, which can be called using the
+`estimatesmartfee` RPC with a parameter specifying how many blocks
+you're willing to wait before the transaction is highly likely to
+confirm (for example, 144 blocks is about 1 day).
 
-.Using the fee estimation API
 ----
-$ curl https://bitcoinfees.21.co/api/v1/fees/recommended
+$ bitcoin-cli -named estimatesmartfee conf_target=144
-
+{
-{"fastestFee":80,"halfHourFee":80,"hourFee":60}
+  "feerate": 0.00006570,
+  "blocks": 144
+}
 ----
 
-The API returns a JSON object with the current fee estimate for fastest
+Many web-based services also provide fee estimation as an API.  For a
-confirmation (+fastestFee+), confirmation within three blocks
+current list, see https://www.lopp.net/bitcoin-information/fee-estimates.html
-(+halfHourFee+) and six blocks (+hourFee+), in satoshi per byte.
 
-[[bitcoinfees21co]]
+As mentioned, fee rate estimation can never be perfect.  One common
-.Fee estimation service bitcoinfees.21.co
+problem is that the fundamental demand might change, adjusting the
-image::images/mbc2_0602.png[Fee Estimation Service bitcoinfees.21.co]
+equilibrium and either increasing prices (fees) to new heights or
+decreasing them towards the minimum, as show in
+<<img_equilibrium_change>>.  If fee rates go down, then a transaction
+that previously paid a normal fee rate might now be paying a high fee
+rate and it will be confirmed earlier than expected.  There's no way to
+lower the fee rate on a transaction you've already sent, so you're stuck
+paying a higher fee rate.  But, when fee rates go up, there's a need for
+methods to be able to increase the fee rates on those transactions,
+which is called _fee bumping_.  There are two commonly used types of fee
+bumping in Bitcoin, Replace-By-Fee (RBF) and Child Pays For Parent
+(CPFP).
 
-==== Adding Fees to Transactions
+[[img_equilibrium_change]]
+.Equilibrium change
+image::../images/equilibrium-change.png[Equilibrium change]
+
+[[rbf]]
+=== Replace-By-Fee (RBF) Fee Bumping
+
+To increase the fee of a transaction using RBF fee bumping, you create
+a conflicting version of the transaction which pays a higher fee.  Two
+or more transactions are considered to be _conflicting transactions_ if
+only one of them can be included in a valid block chain, forcing a miner
+to chose only one of them.  Conflicts occur when two or more transaction
+both try to spend one of the same UTXOs, i.e. they each include an input
+that has the same outpoint (reference to the output of a previous
+transaction).
+
+To prevent someone from consuming large amounts of bandwidth by creating
+an unlimited number of conflicting transactions and sending them through
+the network of relaying full nodes, Bitcoin Core and other full nodes
+that support transaction replacement require each replacement
+transaction to pay a higher fee rate than the transaction being
+replaced.  Bitcoin Core also currently requires the replacement
+transaction to pay a higher total fee than the original transaction, but
+this requirement has undesired side effects and developers have been
+looking for ways to remove it at the time of writing.
+
+Bitcoin Core currently supports two variations of RBF:
+
+Opt-in RBF::
+  An unconfirmed transaction can signal to miners and full nodes that
+  the creator of the transaction wants to allow it to be replaced by a
+  higher fee rate version.  This signal and the rules for using it
+  are specified in BIP125.  As of this writing, this has been enabled by
+  default in Bitcoin Core for several years.
+
+Full RBF::
+  Any unconfirmed transaction can be replaced by a higher fee rate
+  version.  As of this writing, this can be optionally enabled in
+  Bitcoin Core (but it is disabled by default).
+
+.Why are there two variants of RBF?
+****
+The reason for the two different versions of RBF is that full RBF has
+been controversial.  Early versions of Bitcoin allowed transaction
+replacement but this behavior was disabled for several releases.  During
+that time, a miner or full node using the software now called Bitcoin
+Core would not replace the first version of an unconfirmed transaction
+they received with any different version.  Some merchants came to expect
+this behavior: they assumed that any valid unconfirmed transaction which
+paid an appropriate fee rate would eventually become a confirmed
+transaction, so they provided their goods or services shortly after
+receiving such an unconfirmed transaction.
+
+However, there's no way for the Bitcoin protocol to guarantee that any
+unconfirmed transaction will eventually be confirmed.  As mentioned
+earlier in this chapter, every miner gets to choose for themselves which
+transactions they will try to confirm--including which versions of those
+transactions.  Bitcoin Core is open source software, so anyone with a
+copy of its source code can add (or remove) transaction replacement.
+Even if Bitcoin Core wasn't open source, Bitcoin is an open protocol
+that can be reimplemented from scratch by a sufficiently competent
+programmer, allowing the reimplementor to include or not include
+transaction replacement.
+
+Transaction replacement breaks the assumption of some merchants that
+every reasonable unconfirmed transaction will eventually be confirmed.
+An alternative version of a transaction can pay the same outputs as the
+original, but it isn't required to pay any of those outputs.  If the
+first version of an unconfirmed transaction pays a merchant, the second
+version might not pay them.  If the merchant provided goods or services
+based on the first version, but the second version gets confirmed, then
+the merchant will not receive payment for its costs.
+
+Some merchants, and people supporting them, requested that transaction
+replacement not be re-enabled in Bitcoin Core.  Other people pointed out
+that transaction replacement provides benefits, including the ability to
+fee bump transactions that initially paid too low of a fee rate.
+
+Eventually, developers working on Bitcoin Core implemented a compromise:
+instead of allowing every unconfirmed transaction to be replaced (full
+RBF), they only programmed Bitcoin Core to allow replacement of
+transactions that signaled they wanted to allow replacement (opt-in RBF).
+Merchants can check the transactions they receive for the opt-in
+signal and treat those transactions differently than those without the
+signal.
+
+This doesn't change the fundamental concern: anyone can still alter
+their copy of Bitcoin Core, or create a reimplementation, to allow full
+RBF--and some developers even did this, but seemingly few people used
+their software.
+
+After several years, developers working on Bitcoin Core changed the
+compromise slightly.  In addition to keeping opt-in RBF by default, they
+added an option that allows users to enable full RBF.  If enough mining
+hash rate and relaying full nodes enable this option, it will be
+possible for any unconfirmed transaction to eventually be replaced by a
+version paying a higher fee rate.  As of this writing, it's not clear
+whether or not that has happened yet.
+****
+
+As a user, if you plan to use RBF fee bumping, you will first need to
+choose a wallet that supports it, such as one of the wallets listed as
+having "Sending support" on
+https://bitcoinops.org/en/compatibility/#replace-by-fee-rbf
+
+As a developer, if you plan to implement RBF fee bumping, you will first
+need to implement the signaling specified in BIP125, which is a simple
+modification to any one of the nSequence fields in a transaction (see
+<<nsequence>>).
+
+When you need to fee bump a transaction, you will simply create a new
+transaction that spends at least one of the same UTXOs as the original
+transaction you want to replace.  You will likely want to keep the
+same outputs in the transaction which pay the receiver (or receivers).
+You may pay the increased fee by reducing the the value of your change
+output or by adding additional inputs to the transaction.  Developers
+should provide users with a fee-bumping interface that does all of this
+work for them and simply asks them (or suggests to them) how much the
+fee rate should be increased.
+
+[WARNING]
+====
+Be very careful when creating more than one replacement of the same
+transaction.  You must ensure than all versions of the transactions
+conflict with each other.  If they aren't all conflicts, it may be
+possible for multiple separate transactions to confirm, leading you to
+overpay the receivers.  For example:
+
+- Transaction version 0 includes input _A_.
+- Transaction version 1 includes inputs _A_ and _B_ (e.g., you had to add
+  input _B_ to pay the extra fees)
+- Transaction version 2 includes inputs B and C (e.g., you had to add input
+  _C_ to pay the extra fees but _C_ was large enough that you no longer
+  need input _A_).
+
+In the above scenario, any miner who saved version 0 of the transaction
+will be able to confirm both it and version 2 of the transaction.  If
+both versions pay the same receivers, they'll be paid twice (and the
+miner will receive transaction fees from two separate transactions).
+
+A simple method to avoid this problem is to ensure the replacement
+transaction always includes all of the same inputs as the previous
+version of the transaction.
+====
+
+The advantage of RBF fee bumping over other types of fee bumping is that
+it can be very efficient at using block space.  Often, a replacement
+transaction is the same size as the transaction it replaces.  Even when
+it's larger, it's often the same size as the transaction the user would
+have created if they had paid the increased fee rate in the first place.
+
+The fundamental disadvantage of RBF fee bumping is that it can normally
+only be performed by the creator of the transaction--the person or
+people who were required to provide signatures or other authentication
+data for the transaction.  An exception to this is transactions which
+were designed to allow additional inputs to be added by using sighash
+flags, see <<sighash_types>>, but that presents its own challenges.  In
+general, if you're the receiver of an unconfirmed transaction and you
+want to make it confirm faster (or at all), you can't use an RBF fee
+bump; you need some other method.
+
+There are additional problems with RBF that we'll explore in the
+subsequent <<transaction_pinning>> section.
+
+[[cpfp]]
+=== Child Pays For Parent (CPFP) Fee Bumping
+
+Anyone who receives the output of an unconfirmed transaction can
+incentivize miners to confirm that transaction by spending that output.
+The transaction you want to get confirmed is called the _parent
+transaction_.  A transaction which spends an output of the parent
+transaction is called a _child transaction_.
+
+As we learned in <<outpoints>>, every input in a confirmed transaction
+must reference the unspent output of a transaction which appears earlier
+in the block chain (whether earlier in the same block or in a previous
+block).  That means a miner who wants to confirm a child transaction
+must also ensure that its parent transaction is confirmed.  If the
+parent transaction hasn't been confirmed yet but the child transaction
+pays a high enough fee, the miner can consider whether it would be
+profitable to confirm both of them in the same block.
+
+To evaluate the profitability of mining both a parent and child
+transaction, the miner looks at them as a _package of transactions_ with
+an aggregate size and aggregate fees, from which the fees can be divided
+by the size to calculate a _package fee rate_.  The miner can then sort
+all of the individual transactions and transaction packages they know
+about by fee rate and include the highest-revenue ones in the block
+they're attempting to mine, up to the maximum size (weight) allowed to
+be included in a block.  To find even more packages that might be
+profitable to mine, the miner can evaluate packages across multiple
+generations (e.g. an unconfirmed parent transaction being combined with
+both its child and grandchild).  This is called _ancestor fee rate
+mining_.
+
+Bitcoin Core has implemented ancestor fee rate mining for many years,
+and it's believed that almost all miners use it at the time of writing.
+That means it's practical for wallets to use this feature to fee bump an
+incoming transaction by using a child transaction to pay for its parent
+(CPFP).
+
+CPFP has several advantages over RBF.  Anyone who receives an output
+from a transaction can use CPFP--that includes both the receivers of
+payments and the spender (if the spender included a change output).  It
+also doesn't require replacing the original transaction, which makes it
+less disruptive to some merchants than RBF.
+
+The primary disadvantage of CPFP compared to RBF is that CPFP typically
+uses more block space.  In RBF, a fee bump transaction is often the same
+size as the transaction it replaces.  In CPFP, a fee bump is a whole
+separate transaction.  Using extra block space requires paying extra
+fees beyond the the cost of the fee bump.
+
+There are several challenges with CPFP, some of which we'll explore in
+the <<transaction_pinning>> section.  One other problem which we
+specifically need to mention is the minimum relay fee rate problem,
+which is addressed by package relay.
+
+==== Package Relay
+
+Early versions of Bitcoin Core didn't place any limits on the size of
+each node's mempool.  Of course, computers have physical limits, whether
+it's the memory (RAM) or disk space--it's not possible for a full node
+to store an unlimited number of unconfirmed transactions.  Later
+versions of Bitcoin Core limited the size of the mempool to hold about
+one day worth of transactions, storing only the transactions or packages
+with the highest fee rate.
+
+That works extremely well for most things, but it creates a dependency
+problem.  In order to calculate the fee rate for a transaction package,
+we need both the parent and descendant transactions--but if the parent
+transaction doesn't pay a high enough fee rate, it won't be kept in a
+node's mempool.  If a node receives a child transaction without having
+access to its parent, it can't do anything with that transaction.
+
+The solution to this problem is the ability to relay transactions as a
+package, called _package relay_, allowing the receiving node to evaluate
+the fee rate of the entire package before operating on any individual
+transaction.  As of this writing, developers working on Bitcoin Core
+have made significant progress on implementing package relay and a
+limited early version of it may be available by the time this book is
+published.
+
+Package relay is especially important for protocols based on
+time-sensitive presigned transactions, such as Lightning Network.  In
+non-cooperative cases, some presigned transactions can't be fee bumped
+using RBF, forcing them to depend on CPFP.  In those protocols, some
+transactions may also be created long before they need to be broadcast,
+making it effectively impossible to estimate an appropriate fee rate.
+If a presigned transaction pays a fee rate below the amount necessary to
+get into a node's mempool, there's no way to fee bump it with a child.
+If that prevents the transaction from confirming in time, an honest user
+might lose money.  Package relay is the solution for this critical
+problem.
+
+[[transaction_pinning]]
+=== Transaction Pinning
+
+Although both RBF and CPFP fee bumping work in the basic cases we
+described in the sections about them, there are rules related to both
+methods that are designed to prevent denial of service attacks on miners
+and relaying full nodes.  An unfortunate side effect of those rules
+is that they can sometimes prevent someone from being able to use fee
+bumping.  Making it impossible or difficult to fee bump a transaction is
+called _transaction pinning_.
+
+One of the major denial of service concerns revolve around the effect of
+transaction relationships.  Whenever the output of a transaction is
+spent, that transaction's identifier (txid) is referenced by the child
+transaction.  However, when a transaction is replaced, the replacement
+has a different txid.  If that replacement transaction gets confirmed,
+none of its descendants can be included in the same block chain.  It's
+possible to recreate and re-sign the descendant transactions, but that's
+not guaranteed to happen.  This has related but divergent implications
+for RBF and CPFP:
+
+- In the context of RBF, when Bitcoin Core accepts a replacement
+  transaction, it keeps things simple by forgetting about the original
+  transaction and all descendant transactions which depended on that
+  original.  To ensure that it's more profitable for miners to accept
+  replacements, Bitcoin Core only accepts a replacement transaction if it
+  pays more fees than all the transactions that will be forgotten.
++
+The downside of this approach is that Alice can create a small
+transaction that pays Bob.  Bob can then use his output to create a
+large child transaction.  If Alice then wants to replace her original
+transaction, she needs to pay a fee that's larger than what both she and
+Bob originally paid.  For example, if Alice's original transaction was
+about 100 vbytes and Bob's transaction was about 100,000 vbytes, and
+they both used the same feerate, Alice now needs to pay more than 1,000
+times as much as she originally paid in order to RBF fee bump her
+transaction.
+
+- In the context of CPFP, any time the node considers including a
+  package in a block, it must remove the transactions in that package
+  from any other package it wants to consider for the same block.  For
+  example, if a child transaction pays for 25 ancestors, and each of
+  those ancestors has 25 other children, then including the package in
+  the block requires updating approximately 625 packages (25^2^).
+  Similarly, if a transaction with 25 descendants is removed from a
+  node's mempool (such as for being included in a block), and each of
+  those descendants has 25 other ancestors, another 625 packages need to
+  be updated.  Each time we double our parameter (e.g. from 25 to 50),
+  we quadruple the amount of work our node needs to perform.
++
+Additionally, a transaction and all of its descendants is not
+  useful to keep in a mempool long term if an alternative version of
+  that transaction is mined--none of those transactions can now be
+  confirmed unless there's a rare blockchain reorganization.  Bitcoin
+  Core will remove from its mempool every transaction that can no longer
+  be confirmed on the current block chain.  At it's worst, that can
+  waste an enormous amount of your node's bandwidth and possibly be used
+  to prevent transactions from propagating correctly.
++
+To prevent these problems, and other related
+  problems, Bitcoin Core limits a parent transaction to having a maximum
+  of 25 ancestors or descendants in its mempool pool and limits the
+  total size of all those transactions to 100,000 vbytes.  The downside
+  of this approach is that users are prevented from creating CPFP fee
+  bumps if a transaction already has too many descendants (or if it and
+  its descendants are too large).
+
+Transaction pinning can happen by accident, but it also represents a
+serious vulnerability for multiparty time-sensitive protocols such as
+Lightning Network.  If your counterparty can prevent one of your
+transactions from confirming by a deadline, they may be able to steal
+money from you.
+
+Protocol developers have been working on mitigating problems with
+transaction pinning for several years.  One partial solution is
+described in <<cpfp_carve_out>>.  Several other solutions have been
+proposed, and at least one solution is being actively developed as of
+this writing (ephemeral anchors, see
+https://bitcoinops.org/en/topics/ephemeral-anchors/).
+
+[[cpfp_carve_out]]
+=== CPFP Carve Out and Anchor Outputs
+
+In 2018, developers working on Lightning Network (LN) had a problem.
+Their protocol uses transactions which require signatures from two
+different parties.  Neither party wants to trust the other, so they sign
+transactions at a point in the protocol when trust isn't needed,
+allowing either of them to broadcast one of those transactions at a
+later time when the other party may not want to (or be able to) fulfill
+its obligations.  The problem with this approach is that the
+transactions might need to be broadcast far in the future, beyond any
+reasonable ability to estimate an appropriate fee rate for the
+transactions.
+
+In theory, the developers could have designed their transactions to
+allow fee bumping with either RBF (using special sighash flags) or CPFP,
+but both of those protocols are vulnerable to transaction pinning.
+Given that the involved transactions are time sensitive, allowing a
+counterparty to use transaction pinning to delay confirmation of a
+transaction can easily lead to an repeatable exploit that malicious
+parties could use to steal money from honest parties.
+
+LN developer Matt Corallo proposed a solution: give the rules for CPFP
+fee bumping a special exception, called _CPFP carve out_.  The normal
+rules for CPFP forbid the inclusion of an additional descendant if it
+would cause a parent transaction to have 26 or more descendants or if it
+would cause a parent and all of its descendants to exceed 100,000 vbytes
+in size.  Under the rules of CPFP carve out, a single additional
+transaction up to 1,000 vbytes in size can be added to a package even if
+it would exceed the other limits as long as it is a direct child of an
+unconfirmed transaction with no unconfirmed ancestors.
+
+For example, Bob and Mallory both co-sign a transaction with two
+outputs, one to each of them.  Mallory broadcasts that transaction and
+uses her output to attach either 25 child transactions or any smaller
+number of child transactions equaling 100,000 vbytes in size.  Without
+carve-out, Bob would be unable to attach another child transaction to
+his output for CPFP fee bumping.  With carve-out, he can do that as long
+as his child transaction is less than 1,000 vbytes in size (which should
+be more than enough space).
+
+It's not allowed to use CPFP carve-out more than once, so it only works
+for two-party protocols.  There have been proposals to extend it to
+protocols involving more participants, but there hasn't been much demand
+for that and developers are focused on building more generic solutions
+to transaction pinning attacks.
+
+As of this writing, most popular LN implementations use a transaction
+template called _anchor outputs_, which is designed to be used with CPFP
+carve out.
+
+=== Adding Fees to Transactions
 
 The data structure of transactions does not have a field for fees.
 Instead, fees are implied as the difference between the sum of inputs
@@ -164,55 +699,8 @@ the change as a transaction fee. "Keep the change!" might not be what
 you intended.
 ====
 
-((("use cases", "buying coffee")))Let's see how this works in practice,
-by looking at Alice's coffee purchase again. Alice wants to spend 0.015
-bitcoin to pay for coffee. To ensure this transaction is processed
-promptly, she will want to include a transaction fee, say 0.001. That
-will mean that the total cost of the transaction will be 0.016. Her
-wallet must therefore source a set of UTXO that adds up to 0.016 bitcoin
-or more and, if necessary, create change. Let's say her wallet has a
-0.2-bitcoin UTXO available. It will therefore need to consume this UTXO,
-create one output to Bob's Cafe for 0.015, and a second output with
-0.184 bitcoin in change back to her own wallet, leaving 0.001 bitcoin
-unallocated, as an implicit fee for the transaction.
-
-((("use cases", "charitable donations")))((("charitable donations")))Now
-let's look at a different scenario. Eugenia, our children's charity
-director in the Philippines, has completed a fundraiser to purchase
-schoolbooks for the children. She received several thousand small
-donations from people all around the world, totaling 50 bitcoin, so her
-wallet is full of very small payments (UTXO). Now she wants to purchase
-hundreds of schoolbooks from a local publisher, paying in bitcoin.
-
-As Eugenia's wallet application tries to construct a single larger
-payment transaction, it must source from the available UTXO set, which
-is composed of many smaller amounts. That means that the resulting
-transaction will source from more than a hundred small-value UTXO as
-inputs and only one output, paying the book publisher. A transaction
-with that many inputs will be larger than one kilobyte, perhaps several
-kilobytes in size. As a result, it will require a much higher fee than
-the median-sized transaction.
-
-Eugenia's wallet application will calculate the appropriate fee by
-measuring the size of the transaction and multiplying that by the
-per-kilobyte fee. Many wallets will overpay fees for larger transactions
-to ensure the transaction is processed promptly. The higher fee is not
-because Eugenia is spending more money, but because her transaction is
-more complex and larger in size--the fee is independent of the
-transaction's bitcoin value.((("", startref="Tout06")))
-
-[[transaction-pinning]]
-=== Transaction Pinning
-
-FIXME
-
-[[rbf]]
-=== Replace by Fee (RBF)
-
-FIXME
-
 [[fee_sniping]]
-==== Timelock Defense Against Fee Sniping
+=== Timelock Defense Against Fee Sniping
 
 ((("scripting", "timelocks", "defense against
 fee-sniping")))((("timelocks", "defense against
@@ -234,14 +722,15 @@ They can include any transactions that were in the "old" block
 Essentially they have the option to pull transactions from the "present"
 into the rewritten "past" when they re-create block  #100,000.
 
-Today, this attack is not very lucrative, because block reward is much
+Today, this attack is not very lucrative, because the block subsidy is much
 higher than total fees per block. But at some point in the future,
 transaction fees will be the majority of the reward (or even the
 entirety of the reward). At that time, this scenario becomes inevitable.
 
-To prevent "fee sniping," when Bitcoin Core creates transactions, it
+Several wallets discourage fee sniping by creating transactions with an
-uses +nLocktime+ to limit them to the "next block," by default. In our
++nLocktime+ that limits those transactions to being included in the next
-scenario, Bitcoin Core would set +nLocktime+ to 100,001 on any
+block or any later block. In our
+scenario, our wallet would set +nLocktime+ to 100,001 on any
 transaction it created. Under normal circumstances, this +nLocktime+ has
 no effect&#x2014;the transactions could only be included in block
 #100,001 anyway; it's the next block.
@@ -252,7 +741,8 @@ would be timelocked to block #100,001. They can only remine  #100,000
 with whatever transactions were valid at that time, essentially gaining
 no new fees.
 
-To achieve this, Bitcoin Core sets the +nLocktime+ on all new
+This does not entirely prevent fee sniping, but it does make it less
-transactions to <current block # + 1> and sets the +nSequence+ on all
+profitable in some cases and so can help preserve the stability of the
-the inputs to 0xFFFFFFFE to enable +nLocktime+.((("",
+Bitcoin network as the block subsidy declines.  We recommend all wallets
-startref="Stimelock07")))
+implement anti fee sniping when it doesn't interfere with the wallet's
+other uses of the nLockTime field.

chapters/transactions.adoc

@@ -141,7 +141,7 @@ being widely considered.  That proposal does not seek to change the
 consensus rules but only the policy that Bitcoin full nodes use to relay
 transactions.  Under the proposal, version 3 transactions would be
 subject to additional constraints in order to prevent certain Denial of
-Service (DoS) attacks that we'll discuss further in <<transaction-pinning>>
+Service (DoS) attacks that we'll discuss further in <<transaction_pinning>>
 
 === Extended Marker and Flag
 
@@ -372,6 +372,7 @@ byte (0x6b) indicating the scriptSig is 107 bytes.  We'll cover parsing
 and using scripts in detail in the next chapter,
 <<c_authorization_authentication>>.
 
+[[nsequence]]
 ==== nSequence
 
 The final four bytes of an input are its sequence number, called