CH10: add compact block relay & update relay networks

- Describe BIP152 compact block relay
- Describe why we still need relay networks (what they do that BIP152
  can't)
- Drop FALCON relay (never used, AFAIK)
- Minor updates to relay section

ch08.asciidoc

@@ -102,51 +102,172 @@ protocols. These other protocol nodes are mostly pool mining nodes (see
 <<mining>>) and lightweight wallet clients, which do not carry a full
 copy of the blockchain.
 
-=== Bitcoin Relay Networks
 
-((("Bitcoin network", "Bitcoin Relay Networks")))((("relay
+=== Compact Block Relay
-networks")))While the Bitcoin P2P network serves the general needs of a
-broad variety of node types, it exhibits too high network latency for
-the specialized needs of Bitcoin mining nodes.
 
-((("propagation", "relay networks and")))Bitcoin miners are engaged in a
+When a miner finds a new block, they announce it to the Bitcoin network
-time-sensitive competition to solve the Proof-of-Work problem and extend
+(which includes other miners).  The miner who found that block can start
-the blockchain (see <<mining>>). While participating in this
+building on top of it immediately; all other miners who haven't learned
-competition, Bitcoin miners must minimize the time between the
+about the block yet will continue building on top of the previous block
-propagation of a winning block and the beginning of the next round of
+until they do learn about it (and, ideally, they also validate it).
-competition. In mining, network latency is directly related to profit
-margins.
 
-A _Bitcoin Relay Network_ is a network that attempts to minimize the
+If, before they learn about the new block, one of those other
-latency in the transmission of blocks between miners. The original
+miners creates a block, their block will be in competition with the
+first miner's new block.  Only one of the blocks will ever be included
+in the blockchain used by all full nodes, and miners only get paid for
+blocks that are widely accepted.
+
+Whichever block has a second block built on top of it first wins (unless
+there's another near-tie), which is called a _block-finding race_.
+Block-finding races give the advantage to the largest miners, so they
+act in opposition to Bitcoin's essential decentralization.  To prevent
+block-finding races and allow miners of any size to participate equally
+in the lottery that is Bitcoin mining, it's extremely useful to minimize
+the time between when one miner announces a new block and when other
+miners receive that block.
+
+.A blockchain fork requiring a mining race
+image::images/race1.dot.png["Mining race"]
+
+In 2015, a new version of Bitcoin Core added a feature called
+_compact block relay_ (specified in BIP152) that allows transferring new
+blocks both faster and with less bandwidth.
+
+As background, full nodes that relay unconfirmed transactions also store
+many of those transactions in their mempools (see <<mempool>>).  When
+some of those transactions are confirmed in a new block, the node
+doesn't need to receive a second copy of those transactions.
+
+Instead of receiving redundant unconfirmed transactions, compact blocks
+allows a peer that believes your node already has a transaction in that
+block to instead send a short 6-byte identifier for that transaction.
+When your node receives a compact block with one or more identifiers, it
+checks its mempool for those transactions and uses them if they are
+found.  For any transaction that isn't found in your local node's
+mempool, your node can send a request to the peer for a copy.
+
+Conversely, if the remote peer believes your node's mempool doesn't have
+some of transactions that appear in the block, it can include a copy of
+those transaction in the compact block.
+
+If the remote peer guesses correctly about what transactions your node
+has in its mempool, and which it does not, it will send a block nearly
+as efficiently as is theoretically possible (for a typical block, it'll
+be between 97% to 99% efficient).
+
+[TIP]
+====
+Compact block relay does not decrease the size of blocks.  It just
+prevents the redundant transfer of information that a node already has.
+When a node doesn't previously have information about a block, for
+example when a node is first started, it must receive complete copies of
+each block.
+====
+
+There are two modes that Bitcoin Core currently implements for sending
+compact blocks:
+
+Low-bandwidth mode::
+  When your node requests that a peer use low-bandwidth mode (the default),
+  that peer will tell your node the 32-byte identifier (header hash) of a
+  new block but will not send your node any details about it.  If your
+  node acquires that block first from another source, this avoids
+  wasting any more of your bandwidth acquiring a redundant copy of that
+  block.  If your node does need the block, it will request a compact
+  block.
+
+High-bandwidth mode::
+  When your node requests that a peer use high-bandwidth mode, that peer
+  will send your node a compact block for a new block even before it has
+  fully verified that the block is valid.  The only validation the peer
+  will perform is ensuring that the block's header contains the correct
+  amount of proof of work.  Since proof of work is expensive to generate
+  (about $150,000 USD at the time of writing), it's unlikely that a
+  miner would fake it just to waste the bandwidth of relay nodes.
+  Skipping validation before relay allows new blocks to travel across
+  the network with minimal delays at each hop.
++
+The downside of high-bandwidth node is that your node is likely to
+receive redundant information from each high-bandwidth peer it chooses.
+As of this writing, Bitcoin Core currently only asks three peers to use
+high-bandwidth mode (and it tries to choose peers that have a history of
+quickly announcing blocks).
+
+// released into the public domain by Nicolas Dorier
+.BIP152 modes compared (from BIP152)
+image::images/bip152.png["BIP152"]
+
+The names of the two methods (which are taken from BIP152) can be a bit
+confusing.  Low-bandwidth mode saves bandwidth by not sending blocks in
+most cases.  High-bandwidth mode uses more bandwidth than low-bandwidth
+mode but, in most cases, much less bandwidth than was used for block
+relay before compact blocks were implemented.
+
+=== Private Block Relay Networks
+
+Although compact blocks go a long way towards minimizing the latency
+of block propagation, it's possible to minimize latency further.  Unlike
+compact blocks, though, the other solutions involve tradeoffs that
+make them unavailable or unsuitable for the public P2P relay network.
+For that reason, there has been experimentation with private relay
+networks for blocks.
+
+One simple technique is to pre-select a route between endpoints.  For
+example, a relay network with servers running in datacenters near major
+trans-oceanic fiber optic lines might be able to forward new blocks
+faster than waiting for the block to arrive at the node run by some home
+user many kilometers away from the fiber optic line.
+
+Another, more complex technique, is Forward Error Correction (FEC).
+This allows a compact block message to be split into several parts, with
+each part having extra data appended.  If any of the parts isn't
+received, that part can be reconstructed from the parts that are
+received.  Depending on the settings, up to several parts may be
+reconstructed if they are lost.
+
+FEC avoids the problem of a compact block (or some parts of it) not
+arriving due to problems with the underlying network connection.
+Those problems frequently occur but we mostly don't notice them
+because we mostly use protocols that automatically re-request the
+missing data.  However, requesting missing data triples the time to
+receive it.  For example:
+
+1. Alice sends some data to Bob
+2. Bob doesn't receive the data (or it is damaged).   Bob re-requests
+   the data from Alice
+3. Alice sends the data again
+
+A third technique is to assume all nodes receiving the data have
+almost all of the same transactions in their mempool, so they can all
+accept the same compact block.  That not only saves us time computing
+a compact block at each hop but it means that all each hop can simply
+relay the FEC packets to the next hop even before validating them.
+
+The tradeoff for each of the above methods is that they work well with
+centralization but not in a decentralized network where individual nodes
+can't trust other nodes.  Servers in datacenters cost money and can
+often be accessed by operators of the datacenter, making them less
+trustworthy than a secure home computer.  Relaying data before
+validating makes it easy to waste bandwidth, so it can only reasonably
+be used on a private network where there's some level of trust and
+accountability between parties.
+
+The original
 http://www.bitcoinrelaynetwork.org[Bitcoin Relay Network] was created by
-core developer Matt Corallo in 2015 to enable fast synchronization of
+developer Matt Corallo in 2015 to enable fast synchronization of
 blocks between miners with very low latency. The network consisted of
-several specialized nodes hosted on the Amazon Web Services
+several Virtual Private Servers (VPSes) hosted on
 infrastructure around the world and served to connect the majority of
 miners and mining pools.
 
 ((("Fast Internet Bitcoin Relay Engine (FIBRE)")))((("Compact Block
 optimization")))The original Bitcoin Relay Network was replaced in 2016
 with the introduction of the _Fast Internet Bitcoin Relay Engine_ or
-http://bitcoinfibre.org[_FIBRE_], also created by core developer Matt
+http://bitcoinfibre.org[_FIBRE_], also created by developer Matt
 Corallo. FIBRE is a UDP-based relay network that relays blocks within a
-network of nodes. FIBRE implements _compact block_ optimization to
+network of nodes. FIBRE implements FEC and the _compact block_ optimization to
 further reduce the amount of data transmitted and the network latency.
 
-((("Falcon Relay Network")))Another relay network (still in the proposal
-phase) is http://www.falcon-net.org/about[_Falcon_], based on research
-at Cornell University. Falcon uses "cut-through-routing" instead of
-"store-and-forward" to reduce latency by propagating parts of blocks as
-they are received rather than waiting until a complete block is
-received.
-
-Relay networks are not replacements for Bitcoin's P2P network. Instead
-they are overlay networks that provide additional connectivity between
-nodes with specialized needs. Like freeways are not replacements for
-rural roads, but rather shortcuts between two points with heavy traffic,
-you still need small roads to connect to the freeways.
-
 === Network Discovery
 
 ((("Bitcoin network", "extended network discovery",

images/race1.dot

@@ -0,0 +1,37 @@
+digraph blockchain {
+    rankdir=LR; // Flow from left to right
+    node [shape=box, width=0.3, height=0.3];
+    
+    A [label=""];
+    B [label=""];
+    C [label=""];
+    C1 [label=""];
+    D [label="?", style=dashed];
+    D1 [label="?", style=dashed];
+
+    A -> B;
+    B -> C;
+    B -> C1;
+    C -> D [style=dashed];
+    C1 -> D1 [style=dashed];
+
+    {
+        rank=same;
+        A;
+    }
+    {
+        rank=same;
+        B;
+    }
+    {
+        rank=same;
+        C;
+        C1;
+    }
+    {
+        rank=same;
+        D;
+        D1;
+    }
+}
+