diff --git a/ch06.asciidoc b/ch06.asciidoc index c8b3d08d..085bf708 100644 --- a/ch06.asciidoc +++ b/ch06.asciidoc @@ -1,198 +1,2 @@ [[ch06]] -[[tx_fees]] -==== Transaction Fees - -((("transactions", "outputs and inputs", "transaction fees")))((("fees", -"transaction fees")))((("mining and consensus", "rewards and -fees")))Most transactions include transaction fees, which compensate the -bitcoin miners for securing the network. Fees also serve as a security -mechanism themselves, by making it economically infeasible for attackers -to flood the network with transactions. Mining and the fees and rewards -collected by miners are discussed in more detail in <>. - -This section examines how transaction fees are included in a typical -transaction. Most wallets calculate and include transaction fees -automatically. However, if you are constructing transactions -programmatically, or using a command-line interface, you must manually -account for and include these fees. - -Transaction fees serve as an incentive to include (mine) a transaction -into the next block and also as a disincentive against abuse of the -system by imposing a small cost on every transaction. Transaction fees -are collected by the miner who mines the block that records the -transaction on the blockchain. - -Transaction fees are calculated based on the size of the transaction in -kilobytes, not the value of the transaction in bitcoin. Overall, -transaction fees are set based on market forces within the Bitcoin -network. Miners prioritize transactions based on many different -criteria, including fees, and might even process transactions for free -under certain circumstances. Transaction fees affect the processing -priority, meaning that a transaction with sufficient fees is likely to -be included in the next block mined, whereas a transaction with -insufficient or no fees might be delayed, processed on a best-effort -basis after a few blocks, or not processed at all. Transaction fees are -not mandatory, and transactions without fees might be processed -eventually; however, including transaction fees encourages priority -processing. - -Over time, the way transaction fees are calculated and the effect they -have on transaction prioritization has evolved. At first, transaction -fees were fixed and constant across the network. Gradually, the fee -structure relaxed and may be influenced by market forces, based on -network capacity and transaction volume. Since at least the beginning of -2016, capacity limits in bitcoin have created competition between -transactions, resulting in higher fees and effectively making free -transactions a thing of the past. Zero fee or very low fee transactions -rarely get mined and sometimes will not even be propagated across the -network. - -((("fees", "fee relay policies")))((("minrelaytxfee option")))In Bitcoin -Core, fee relay policies are set by the +minrelaytxfee+ option. The -current default +minrelaytxfee+ is 0.00001 bitcoin or a hundredth of a -millibitcoin per kilobyte. Therefore, by default, transactions with a -fee less than 0.00001 bitcoin are treated as free and are only relayed -if there is space in the mempool; otherwise, they are dropped. Bitcoin -nodes can override the default fee relay policy by adjusting the value -of +minrelaytxfee+. - -((("dynamic fees")))((("fees", "dynamic fees")))Any bitcoin service that -creates transactions, including wallets, exchanges, retail applications, -etc., _must_ implement dynamic fees. Dynamic fees can be implemented -through a third-party fee estimation service or with a built-in fee -estimation algorithm. If you're unsure, begin with a third-party service -and as you gain experience design and implement your own algorithm if -you wish to remove the third-party dependency. - -Fee estimation algorithms calculate the appropriate fee, based on -capacity and the fees offered by "competing" transactions. These -algorithms range from simplistic (average or median fee in the last -block) to sophisticated (statistical analysis). They estimate the -necessary fee (in satoshis per byte) that will give a transaction a high -probability of being selected and included within a certain number of -blocks. Most services offer users the option of choosing high, medium, -or low priority fees. High priority means users pay higher fees but the -transaction is likely to be included in the next block. Medium and low -priority means users pay lower transaction fees but the transactions may -take much longer to confirm. - -((("bitcoinfees (third-party service)")))Many wallet applications use -third-party services for fee calculations. One popular service is -http://bitcoinfees.21.co/[_http://bitcoinfees.21.co_], which provides an -API and a visual chart showing the fee in satoshi/byte for different -priorities. - -[TIP] -==== -((("static fees")))((("fees", "static fees")))Static fees are no longer -viable on the Bitcoin network. Wallets that set static fees will produce -a poor user experience as transactions will often get "stuck" and remain -unconfirmed. Users who don't understand bitcoin transactions and fees -are dismayed by "stuck" transactions because they think they've lost -their money. -==== - -The chart in <> shows the real-time estimate of fees in -10 satoshi/byte increments and the expected confirmation time (in -minutes and number of blocks) for transactions with fees in each range. -For each fee range (e.g., 61–70 satoshi/byte), two horizontal -bars show the number of unconfirmed transactions (1405) and total number -of transactions in the past 24 hours (102,975), with fees in that range. -Based on the graph, the recommended high-priority fee at this time was -80 satoshi/byte, a fee likely to result in the transaction being mined -in the very next block (zero block delay). For perspective, the median -transaction size is 226 bytes, so the recommended fee for a transaction -size would be 18,080 satoshis (0.00018080 BTC). - -The fee estimation data can be retrieved via a simple HTTP REST API, at -https://bitcoinfees.21.co/api/v1/fees/recommended[https://bitcoinfees.21.co/api/v1/fees/recommended]. -For example, on the command line using the +curl+ command: - -.Using the fee estimation API ----- -$ curl https://bitcoinfees.21.co/api/v1/fees/recommended - -{"fastestFee":80,"halfHourFee":80,"hourFee":60} ----- - -The API returns a JSON object with the current fee estimate for fastest -confirmation (+fastestFee+), confirmation within three blocks -(+halfHourFee+) and six blocks (+hourFee+), in satoshi per byte. - -[[bitcoinfees21co]] -.Fee estimation service bitcoinfees.21.co -image::images/mbc2_0602.png[Fee Estimation Service bitcoinfees.21.co] - -==== Adding Fees to Transactions - -The data structure of transactions does not have a field for fees. -Instead, fees are implied as the difference between the sum of inputs -and the sum of outputs. Any excess amount that remains after all outputs -have been deducted from all inputs is the fee that is collected by the -miners: - -[[tx_fee_equation]] -.Transaction fees are implied, as the excess of inputs minus outputs: ----- -Fees = Sum(Inputs) – Sum(Outputs) ----- - -This is a somewhat confusing element of transactions and an important -point to understand, because if you are constructing your own -transactions you must ensure you do not inadvertently include a very -large fee by underspending the inputs. That means that you must account -for all inputs, if necessary by creating change, or you will end up -giving the miners a very big tip! - -For example, if you consume a 20-bitcoin UTXO to make a 1-bitcoin -payment, you must include a 19-bitcoin change output back to your -wallet. Otherwise, the 19-bitcoin "leftover" will be counted as a -transaction fee and will be collected by the miner who mines your -transaction in a block. Although you will receive priority processing -and make a miner very happy, this is probably not what you intended. - -[WARNING] -==== -((("warnings and cautions", "change outputs")))If you forget to add a -change output in a manually constructed transaction, you will be paying -the change as a transaction fee. "Keep the change!" might not be what -you intended. -==== - -((("use cases", "buying coffee")))Let's see how this works in practice, -by looking at Alice's coffee purchase again. Alice wants to spend 0.015 -bitcoin to pay for coffee. To ensure this transaction is processed -promptly, she will want to include a transaction fee, say 0.001. That -will mean that the total cost of the transaction will be 0.016. Her -wallet must therefore source a set of UTXO that adds up to 0.016 bitcoin -or more and, if necessary, create change. Let's say her wallet has a -0.2-bitcoin UTXO available. It will therefore need to consume this UTXO, -create one output to Bob's Cafe for 0.015, and a second output with -0.184 bitcoin in change back to her own wallet, leaving 0.001 bitcoin -unallocated, as an implicit fee for the transaction. - -((("use cases", "charitable donations")))((("charitable donations")))Now -let's look at a different scenario. Eugenia, our children's charity -director in the Philippines, has completed a fundraiser to purchase -schoolbooks for the children. She received several thousand small -donations from people all around the world, totaling 50 bitcoin, so her -wallet is full of very small payments (UTXO). Now she wants to purchase -hundreds of schoolbooks from a local publisher, paying in bitcoin. - -As Eugenia's wallet application tries to construct a single larger -payment transaction, it must source from the available UTXO set, which -is composed of many smaller amounts. That means that the resulting -transaction will source from more than a hundred small-value UTXO as -inputs and only one output, paying the book publisher. A transaction -with that many inputs will be larger than one kilobyte, perhaps several -kilobytes in size. As a result, it will require a much higher fee than -the median-sized transaction. - -Eugenia's wallet application will calculate the appropriate fee by -measuring the size of the transaction and multiplying that by the -per-kilobyte fee. Many wallets will overpay fees for larger transactions -to ensure the transaction is processed promptly. The higher fee is not -because Eugenia is spending more money, but because her transaction is -more complex and larger in size--the fee is independent of the -transaction's bitcoin value.((("", startref="Tout06"))) diff --git a/ch07.asciidoc b/ch07.asciidoc index 8c40f9ac..5396fa40 100644 --- a/ch07.asciidoc +++ b/ch07.asciidoc @@ -46,49 +46,3 @@ for it when estimating the desired value to encode in +nLocktime+, Median-Time-Past is specified in https://github.com/bitcoin/bips/blob/master/bip-0113.mediawiki[BIP-113]. - -[[fee_sniping]] -==== Timelock Defense Against Fee Sniping - -((("scripting", "timelocks", "defense against -fee-sniping")))((("timelocks", "defense against -fee-sniping")))((("fees", "fee sniping")))((("security", "defense -against fee-sniping")))((("sniping")))Fee-sniping is a theoretical -attack scenario, where miners attempting to rewrite past blocks "snipe" -higher-fee transactions from future blocks to maximize their -profitability. - -For example, let's say the highest block in existence is block -#100,000. If instead of attempting to mine block #100,001 to extend the -chain, some miners attempt to remine #100,000. These miners can choose -to include any valid transaction (that hasn't been mined yet) in their -candidate block #100,000. They don't have to remine the block with the -same transactions. In fact, they have the incentive to select the most -profitable (highest fee per kB) transactions to include in their block. -They can include any transactions that were in the "old" block -#100,000, as well as any transactions from the current mempool. -Essentially they have the option to pull transactions from the "present" -into the rewritten "past" when they re-create block #100,000. - -Today, this attack is not very lucrative, because block reward is much -higher than total fees per block. But at some point in the future, -transaction fees will be the majority of the reward (or even the -entirety of the reward). At that time, this scenario becomes inevitable. - -To prevent "fee sniping," when Bitcoin Core creates transactions, it -uses +nLocktime+ to limit them to the "next block," by default. In our -scenario, Bitcoin Core would set +nLocktime+ to 100,001 on any -transaction it created. Under normal circumstances, this +nLocktime+ has -no effect—the transactions could only be included in block -#100,001 anyway; it's the next block. - -But under a blockchain fork attack, the miners would not be able to pull -high-fee transactions from the mempool, because all those transactions -would be timelocked to block #100,001. They can only remine #100,000 -with whatever transactions were valid at that time, essentially gaining -no new fees. - -To achieve this, Bitcoin Core sets the +nLocktime+ on all new -transactions to and sets the +nSequence+ on all -the inputs to 0xFFFFFFFE to enable +nLocktime+.((("", -startref="Stimelock07"))) diff --git a/chapters/fees.adoc b/chapters/fees.adoc index 02f58337..b90a7797 100644 --- a/chapters/fees.adoc +++ b/chapters/fees.adoc @@ -1,3 +1,200 @@ +[[tx_fees]] +==== Transaction Fees + +((("transactions", "outputs and inputs", "transaction fees")))((("fees", +"transaction fees")))((("mining and consensus", "rewards and +fees")))Most transactions include transaction fees, which compensate the +bitcoin miners for securing the network. Fees also serve as a security +mechanism themselves, by making it economically infeasible for attackers +to flood the network with transactions. Mining and the fees and rewards +collected by miners are discussed in more detail in <>. + +This section examines how transaction fees are included in a typical +transaction. Most wallets calculate and include transaction fees +automatically. However, if you are constructing transactions +programmatically, or using a command-line interface, you must manually +account for and include these fees. + +Transaction fees serve as an incentive to include (mine) a transaction +into the next block and also as a disincentive against abuse of the +system by imposing a small cost on every transaction. Transaction fees +are collected by the miner who mines the block that records the +transaction on the blockchain. + +Transaction fees are calculated based on the size of the transaction in +kilobytes, not the value of the transaction in bitcoin. Overall, +transaction fees are set based on market forces within the Bitcoin +network. Miners prioritize transactions based on many different +criteria, including fees, and might even process transactions for free +under certain circumstances. Transaction fees affect the processing +priority, meaning that a transaction with sufficient fees is likely to +be included in the next block mined, whereas a transaction with +insufficient or no fees might be delayed, processed on a best-effort +basis after a few blocks, or not processed at all. Transaction fees are +not mandatory, and transactions without fees might be processed +eventually; however, including transaction fees encourages priority +processing. + +Over time, the way transaction fees are calculated and the effect they +have on transaction prioritization has evolved. At first, transaction +fees were fixed and constant across the network. Gradually, the fee +structure relaxed and may be influenced by market forces, based on +network capacity and transaction volume. Since at least the beginning of +2016, capacity limits in bitcoin have created competition between +transactions, resulting in higher fees and effectively making free +transactions a thing of the past. Zero fee or very low fee transactions +rarely get mined and sometimes will not even be propagated across the +network. + +((("fees", "fee relay policies")))((("minrelaytxfee option")))In Bitcoin +Core, fee relay policies are set by the +minrelaytxfee+ option. The +current default +minrelaytxfee+ is 0.00001 bitcoin or a hundredth of a +millibitcoin per kilobyte. Therefore, by default, transactions with a +fee less than 0.00001 bitcoin are treated as free and are only relayed +if there is space in the mempool; otherwise, they are dropped. Bitcoin +nodes can override the default fee relay policy by adjusting the value +of +minrelaytxfee+. + +((("dynamic fees")))((("fees", "dynamic fees")))Any bitcoin service that +creates transactions, including wallets, exchanges, retail applications, +etc., _must_ implement dynamic fees. Dynamic fees can be implemented +through a third-party fee estimation service or with a built-in fee +estimation algorithm. If you're unsure, begin with a third-party service +and as you gain experience design and implement your own algorithm if +you wish to remove the third-party dependency. + +Fee estimation algorithms calculate the appropriate fee, based on +capacity and the fees offered by "competing" transactions. These +algorithms range from simplistic (average or median fee in the last +block) to sophisticated (statistical analysis). They estimate the +necessary fee (in satoshis per byte) that will give a transaction a high +probability of being selected and included within a certain number of +blocks. Most services offer users the option of choosing high, medium, +or low priority fees. High priority means users pay higher fees but the +transaction is likely to be included in the next block. Medium and low +priority means users pay lower transaction fees but the transactions may +take much longer to confirm. + +((("bitcoinfees (third-party service)")))Many wallet applications use +third-party services for fee calculations. One popular service is +http://bitcoinfees.21.co/[_http://bitcoinfees.21.co_], which provides an +API and a visual chart showing the fee in satoshi/byte for different +priorities. + +[TIP] +==== +((("static fees")))((("fees", "static fees")))Static fees are no longer +viable on the Bitcoin network. Wallets that set static fees will produce +a poor user experience as transactions will often get "stuck" and remain +unconfirmed. Users who don't understand bitcoin transactions and fees +are dismayed by "stuck" transactions because they think they've lost +their money. +==== + +The chart in <> shows the real-time estimate of fees in +10 satoshi/byte increments and the expected confirmation time (in +minutes and number of blocks) for transactions with fees in each range. +For each fee range (e.g., 61–70 satoshi/byte), two horizontal +bars show the number of unconfirmed transactions (1405) and total number +of transactions in the past 24 hours (102,975), with fees in that range. +Based on the graph, the recommended high-priority fee at this time was +80 satoshi/byte, a fee likely to result in the transaction being mined +in the very next block (zero block delay). For perspective, the median +transaction size is 226 bytes, so the recommended fee for a transaction +size would be 18,080 satoshis (0.00018080 BTC). + +The fee estimation data can be retrieved via a simple HTTP REST API, at +https://bitcoinfees.21.co/api/v1/fees/recommended[https://bitcoinfees.21.co/api/v1/fees/recommended]. +For example, on the command line using the +curl+ command: + +.Using the fee estimation API +---- +$ curl https://bitcoinfees.21.co/api/v1/fees/recommended + +{"fastestFee":80,"halfHourFee":80,"hourFee":60} +---- + +The API returns a JSON object with the current fee estimate for fastest +confirmation (+fastestFee+), confirmation within three blocks +(+halfHourFee+) and six blocks (+hourFee+), in satoshi per byte. + +[[bitcoinfees21co]] +.Fee estimation service bitcoinfees.21.co +image::images/mbc2_0602.png[Fee Estimation Service bitcoinfees.21.co] + +==== Adding Fees to Transactions + +The data structure of transactions does not have a field for fees. +Instead, fees are implied as the difference between the sum of inputs +and the sum of outputs. Any excess amount that remains after all outputs +have been deducted from all inputs is the fee that is collected by the +miners: + +[[tx_fee_equation]] +.Transaction fees are implied, as the excess of inputs minus outputs: +---- +Fees = Sum(Inputs) – Sum(Outputs) +---- + +This is a somewhat confusing element of transactions and an important +point to understand, because if you are constructing your own +transactions you must ensure you do not inadvertently include a very +large fee by underspending the inputs. That means that you must account +for all inputs, if necessary by creating change, or you will end up +giving the miners a very big tip! + +For example, if you consume a 20-bitcoin UTXO to make a 1-bitcoin +payment, you must include a 19-bitcoin change output back to your +wallet. Otherwise, the 19-bitcoin "leftover" will be counted as a +transaction fee and will be collected by the miner who mines your +transaction in a block. Although you will receive priority processing +and make a miner very happy, this is probably not what you intended. + +[WARNING] +==== +((("warnings and cautions", "change outputs")))If you forget to add a +change output in a manually constructed transaction, you will be paying +the change as a transaction fee. "Keep the change!" might not be what +you intended. +==== + +((("use cases", "buying coffee")))Let's see how this works in practice, +by looking at Alice's coffee purchase again. Alice wants to spend 0.015 +bitcoin to pay for coffee. To ensure this transaction is processed +promptly, she will want to include a transaction fee, say 0.001. That +will mean that the total cost of the transaction will be 0.016. Her +wallet must therefore source a set of UTXO that adds up to 0.016 bitcoin +or more and, if necessary, create change. Let's say her wallet has a +0.2-bitcoin UTXO available. It will therefore need to consume this UTXO, +create one output to Bob's Cafe for 0.015, and a second output with +0.184 bitcoin in change back to her own wallet, leaving 0.001 bitcoin +unallocated, as an implicit fee for the transaction. + +((("use cases", "charitable donations")))((("charitable donations")))Now +let's look at a different scenario. Eugenia, our children's charity +director in the Philippines, has completed a fundraiser to purchase +schoolbooks for the children. She received several thousand small +donations from people all around the world, totaling 50 bitcoin, so her +wallet is full of very small payments (UTXO). Now she wants to purchase +hundreds of schoolbooks from a local publisher, paying in bitcoin. + +As Eugenia's wallet application tries to construct a single larger +payment transaction, it must source from the available UTXO set, which +is composed of many smaller amounts. That means that the resulting +transaction will source from more than a hundred small-value UTXO as +inputs and only one output, paying the book publisher. A transaction +with that many inputs will be larger than one kilobyte, perhaps several +kilobytes in size. As a result, it will require a much higher fee than +the median-sized transaction. + +Eugenia's wallet application will calculate the appropriate fee by +measuring the size of the transaction and multiplying that by the +per-kilobyte fee. Many wallets will overpay fees for larger transactions +to ensure the transaction is processed promptly. The higher fee is not +because Eugenia is spending more money, but because her transaction is +more complex and larger in size--the fee is independent of the +transaction's bitcoin value.((("", startref="Tout06"))) + [[transaction-pinning]] === Transaction Pinning @@ -7,3 +204,49 @@ FIXME === Replace by Fee (RBF) FIXME + +[[fee_sniping]] +==== Timelock Defense Against Fee Sniping + +((("scripting", "timelocks", "defense against +fee-sniping")))((("timelocks", "defense against +fee-sniping")))((("fees", "fee sniping")))((("security", "defense +against fee-sniping")))((("sniping")))Fee-sniping is a theoretical +attack scenario, where miners attempting to rewrite past blocks "snipe" +higher-fee transactions from future blocks to maximize their +profitability. + +For example, let's say the highest block in existence is block +#100,000. If instead of attempting to mine block #100,001 to extend the +chain, some miners attempt to remine #100,000. These miners can choose +to include any valid transaction (that hasn't been mined yet) in their +candidate block #100,000. They don't have to remine the block with the +same transactions. In fact, they have the incentive to select the most +profitable (highest fee per kB) transactions to include in their block. +They can include any transactions that were in the "old" block +#100,000, as well as any transactions from the current mempool. +Essentially they have the option to pull transactions from the "present" +into the rewritten "past" when they re-create block #100,000. + +Today, this attack is not very lucrative, because block reward is much +higher than total fees per block. But at some point in the future, +transaction fees will be the majority of the reward (or even the +entirety of the reward). At that time, this scenario becomes inevitable. + +To prevent "fee sniping," when Bitcoin Core creates transactions, it +uses +nLocktime+ to limit them to the "next block," by default. In our +scenario, Bitcoin Core would set +nLocktime+ to 100,001 on any +transaction it created. Under normal circumstances, this +nLocktime+ has +no effect—the transactions could only be included in block +#100,001 anyway; it's the next block. + +But under a blockchain fork attack, the miners would not be able to pull +high-fee transactions from the mempool, because all those transactions +would be timelocked to block #100,001. They can only remine #100,000 +with whatever transactions were valid at that time, essentially gaining +no new fees. + +To achieve this, Bitcoin Core sets the +nLocktime+ on all new +transactions to and sets the +nSequence+ on all +the inputs to 0xFFFFFFFE to enable +nLocktime+.((("", +startref="Stimelock07")))