Bitcoin is an _open source_ project and the source code is available under an open (MIT) license, free to download and use for any purpose. Open source means more than simply free to use. It also means that bitcoin is developed by an open community of volunteers. At first, that community consisted of only Satoshi Nakamoto. By 2016, bitcoin's source code has more than 340 contributors with about a dozen developers working on the code almost full time and several dozen more on a part-time basis. Anyone can contribute to the code - including you!
When bitcoin was created by Satoshi Nakamoto, the software was actually completed before the white paper <<satoshi_whitepaper>>. Satoshi wanted to make sure it worked before writing about it. That first implementation, then simply known as "Bitcoin" or "Satoshi client", has been heavily modified and improved. It has evolved into what is known as _Bitcoin Core_, to differentiate it from other compatible implementations. Bitcoin Core is the _reference implementation_ of the bitcoin system, meaning that it is the authoritative reference on how each part of the technology should be implemented. Bitcoin Core implements all aspects of bitcoin, including wallets, a transaction and block validation engine, and a full network node in the peer-to-peer bitcoin network.
[WARNING]
====
Even though Bitcoin Core includes a reference implementation of a wallet, this is not intended to be used as a production wallet for users or for applications. Application developers are advised to build wallets using modern standards such as BIP39 and BIP32 (see <<mnemonic_code_words>> and <<hd_wallets>>).
If you're a developer, you will want to setup a development environment with all the tools, libraries and support software for writing bitcoin applications. In this highly technical chapter, we'll walk through that process step-by-step. If the material becomes too dense (and you're not actually setting up a development environment) feel free to skip to the next chapter, which is less technical.
((("Bitcoin Core client","compiling from source code", id="ix_ch03-asciidoc3", range="startofrange")))((("bitcoind client", see="Bitcoin Core client")))Bitcoin Core's source code can be downloaded as a ZIP archive or by cloning the authoritative source repository from GitHub. ((("Bitcoin Core client","source code, downloading")))((("GitHub, downloading Bitcoin Core from"))) On the https://github.com/bitcoin/bitcoin[GitHub bitcoin page], select Download ZIP from the sidebar. Alternatively, use the git command line to create a local copy of the source code on your system.
In many of the examples in this chapter we will be using the operating system's command-line interface (also known as a "shell"), accessed via a "terminal" application. The shell will display a prompt; you type a command; and the shell responds with some text and a new prompt for your next command. The prompt may look different on your system, but in the examples below it is denoted by a +$+ symbol. In the examples, when you see text after a +$+ symbol, don't type the +$+ symbol but type the command immediately following it, then press enter to execute the command. In the examples, the lines below each command are the operating system's responses to that command. When you see the next +$+ prefix, you'll know it's a new command and you should repeat the process.
Git is the most widely used distributed version control system, an essential part of any software developer's toolkit. You may need to install the +git+ command, or a graphical user interface for git, on your operating system if you do not have it already.
When the git cloning operation has completed, you will have a complete local copy of the source code repository in the directory _bitcoin_. Change to this directory by typing +cd bitcoin+ at the prompt:
By default, the local copy will be synchronized with the most recent code, which might be an unstable or beta version of bitcoin. Before compiling the code, select a specific version by checking out a((("release tags"))) release _tag_. This will synchronize the local copy with a specific snapshot of the code repository identified by a keyword tag. Tags are used by the developers to mark specific releases of the code by version number. First, to find the available tags, we use the +git tag+ command:
The list of tags shows all the released versions of bitcoin. By convention,((("release candidates"))) _release candidates_, which are intended for testing, have the suffix "rc". Stable releases that can be run on production systems have no suffix. From the preceding list, select the highest version release, which at this writing was v0.11.2. To synchronize the local code with this version, use the +git checkout+ command:
((("Bitcoin Core client","documentation")))The source code includes documentation, which can be found in a number of files. Review the main documentation located in _README.md_ in the bitcoin directory by typing +more README.md+ at the prompt and using the space bar to progress to the next page. In this chapter, we will build the command-line bitcoin client, also known as +bitcoind+ on Linux. Review the instructions for compiling the bitcoind command-line client on your platform by typing +more doc/build-unix.md+. Alternative instructions for Mac OS X and Windows can be found in the _doc_ directory, as _build-osx.md_ or _build-msw.md_, respectively.
Carefully review the build prerequisites, which are in the first part of the build documentation. These are libraries that must be present on your system before you can begin to compile bitcoin. If these prerequisites are missing, the build process will fail with an error. If this happens because you missed a prerequisite, you can install it and then resume the build process from where you left off. Assuming the prerequisites are installed, you start the build process by generating a set of build scripts using the _autogen.sh_ script.
The Bitcoin Core build process was changed to use the autogen/configure/make system starting with version 0.9. Older versions use a simple Makefile and work slightly differently from the following example. Follow the instructions for the version you want to compile. The autogen/configure/make introduced in 0.9 is likely to be the build system used for all future versions of the code and is the system demonstrated in the following examples.
((("autogen.sh script")))The _autogen.sh_ script creates a set of automatic configuration scripts that will interrogate your system to discover the correct settings and ensure you have all the necessary libraries to compile the code. The most important of these is the((("configure script"))) +configure+ script that offers a number of different options to customize the build process. Type +./configure --help+ to see the various options:
The +configure+ script allows you to enable or disable certain features of bitcoind through the use of the +--enable-FEATURE+ and +--disable-FEATURE+ flags, where +FEATURE+ is replaced by the feature name, as listed in the help output. In this chapter, we will build the bitcoind client with all the default features. We won't be using the configuration flags, but you should review them to understand what optional features are part of the client. If you are in an academic setting, computer lab restrictions may require you to install applications in your home directory (e.g. using --prefix=$HOME).
[TIP]
====
Here are some useful options that override the default behavior of the configure script:
--prefix=$HOME:: This overrides the default installation location (which is +/usr/local/+) for the resulting executable. Use $HOME to put everything in your home directory, or a different path.
If all goes well, the +configure+ command will end by creating the customized build scripts that will allow us to compile bitcoind. If there are any missing libraries or errors, the +configure+ command will terminate with an error instead of creating the build scripts. If an error occurs, it is most likely because of a missing or incompatible library. Review the build documentation again and make sure you install the missing prerequisites. Then run +configure+ again and see if that fixes the error.
==== Building the Bitcoin Core Executables
Next, you will compile the source code, a process that can take up to an hour to complete, depending on the speed of your CPU and available memory. During the compilation process you should see output every few seconds or every few minutes, or an error if something goes wrong. If an error occurs, or the compilation process is interrupted, it can be resumed any time by typing +make+ again. Type +make+ to start compiling the executable application:
If all goes well, Bitcoin Core is now compiled. The final step is to install the various executables on your system using the +sudo make install+ command. You may be prompted for your user password, because this step requires administrative privileges:
The default installation of bitcoind puts it in _/usr/local/bin_. You can confirm that Bitcoin Core is correctly installed by asking the system for the path of the executables, as follows:
Bitcoin's peer-to-peer network is composed of network "nodes", run mostly by volunteers and some of the businesses that build bitcoin applications. Those running bitcoin nodes have a direct and authoritative view of the bitcoin blockchain, with a local copy of all the transactions, independently validated by their own system. By running a node, you don't have to rely on any third party to validate a transaction. Moreover, by running a bitcoin node you contribute to the bitcoin network by making it more robust.
Running a node, however, requires a permanently connected system with enough resources to process all bitcoin transactions. Depending on whether you choose to index all transactions and keep a full copy of the blockchain, you may also need a lot of disk space and RAM. In early 2016, a full-index node needs 2GB of RAM and 80GB of disk space. Bitcoin nodes also transmit and receive bitcoin transactions and blocks, consuming Internet bandwidth. If your Internet connection is limited, has a low data cap, or is metered (charged by the gigabit), you should probably not run a bitcoin node on it, or run it in a way that constrains its bandwidth (see <<constrained_resources>>).
[TIP]
====
((("Bitcoin Core","runtime requirement for")))((("runtime requirements for Bitcoin Core")))Bitcoin Core keeps a full copy of the blockchain by default, with every transaction that has ever occurred on the bitcoin network since its inception in 2009. This dataset is several gigabytes in size and is downloaded incrementally over several hours or days, depending on the speed of your CPU and Internet connection. Bitcoin Core will not be able to process transactions or update account balances until the full blockchain dataset is downloaded. Make sure you have enough disk space, bandwidth, and time to complete the initial synchronization. You can configure Bitcoin Core to reduce the size of the blockchain by discarding old blocks (see <<constrained_resources>>) but it will still download the entire dataset before discarding data.
====
Despite these resource requirements, thousands of volunteers run bitcoin nodes. Some are running on systems as simple as a Raspberry Pi (a $35 USD computer the size of a pack of cards). Many volunteers also run bitcoin nodes on rented servers, usually some variant of Linux. A _Virtual Private Server_ (VPS) or _Cloud Computing_ server instance can be used to run a bitcoin node. Such servers can be rented for $12 to $18 USD per month from a variety of providers.
Why would you want to run a node? Here are some of the most common reasons for running a node:
* If you are developing bitcoin software and need to rely on a bitcoin node for programmable (API) access to the network and blockchain.
* If you are building applications that must validate transactions according to bitcoin's consensus rules. Typically, bitcoin software companies run several nodes.
* If you want to support bitcoin. Running a node makes the network more robust and able to serve more wallets, more users and more transactions.
* If you do not want to rely on any third party for processing your own transactions or validating transactions.
If you're reading this book and interested in developing bitcoin software, you should be running your own node.
==== Running Bitcoin Core for the First Time
When you first run bitcoind, it will remind you to create a configuration file with a strong password for the JSON-RPC interface. This password controls access to the Application Programming Interface (API) offered by Bitcoin Core.
Run bitcoind by typing +bitcoind+ into the terminal:
As you can see, the first time you run +bitcoind+ it tells you that you need to build a configuration file, with at least an rpcuser and rpcpassword entry. Additionally, it is recommended you set up the alerting mechanism. In the next section we will examine the various configuration options and set up a configuration file.
Edit the configuration file in your preferred editor and set the parameters, replacing the password with a strong password as recommended by bitcoind. Do _not_ use the password shown in the book. Create a file inside the +.bitcoin+ directory (under your user's home directory) so that it is named +.bitcoin/bitcoin.conf+ and provide a username and password:
In addition to the +rpcuser+ and +rpcpassword+ options, Bitcoin Core offers more than one hundred configuration options that modify the behavior of the network node, the storage of the blockchain and many other aspects of its operation. To see a listing of these options, run +bitcoind --help+:
conf:: An alternative location for the configuration file. This only makes sense as a command-line parameter to bitcoind, as it can't be inside the configuration file it refers to.
datadir:: Select the directory and filesystem to put all the blockchain data. By default this is the +.bitcoin+ subdirectory of your home directory. Make sure this filesystem has several gigabytes free space.
prune:: Reduce the disk space requirements to this many megabytes, by deleting old blocks. Use this on a resource-constrained node that can't fit the full blockchain.
txindex:: Maintain an index of all transactions. This means a complete copy of the blockchain and allows you to programmatically retrieve any transaction by ID.
maxconnections:: Set the maximum number of nodes from which to accept connections. Reducing this from the default will reduce your bandwidth consumption. Use if you have a data cap or pay by the gigabyte.
maxmempool:: Limit the transaction memorypool to this many megabytes. Use it to reduce memory use of the node.
maxreceivebuffer/maxsendbuffer:: Limit per-connection memory buffer to this many * 1000 bytes. Use on memory-constrained nodes.
minrelaytxfee:: Set the minimum fee transaction you will relay. Below this value, the transaction is treated as zero fee. Use this on memory-constrained nodes to reduce the size of the in-memory transaction pool.
((("transaction database index")))By default, Bitcoin Core builds a database containing _only_ the transactions related to the user's wallet. If you want to be able to access _any_ transaction with commands like +getrawtransaction+ (see ), you need to configure Bitcoin Core to build a complete transaction index, which can be achieved with the((("txindex option (Bitcoin Core)"))) +txindex+ option. Set +txindex=1+ in the Bitcoin Core configuration file. If you don't set this option at first and later set it to full indexing, you need to restart bitcoind with the +-reindex+ option and wait for it to rebuild the index.
Once you've edited the configuration file and set the options that best represent your needs, we can test +bitcoind+ with this configuration. Run Bitcoin Core with the option +printtoconsole+ to run in the foreground with output to the console:(((range="endofrange", startref="ix_ch03-asciidoc3")))
This shows a node running Bitcoin Core version 0.11.2, with a blockchain height of 396328 blocks and 15 active network connections.
Once you are happy with the configuration options you have selected, you should add bitcoin to the startup scripts in your operating system, so that it runs continuously and restarts when the operating system restarts. You will find a number of example startup scripts for various operating systems in bitcoin's source directory under +contrib/init+ and a +README.md+ file showing which system uses which script.
((("Bitcoin Core client","JSON-RPC API for", id="ix_ch03-asciidoc5", range="startofrange")))((("Bitcoin Core client","using from command line", id="ix_ch03-asciidoc6", range="startofrange")))((("bitcoin-cli command line helper", id="ix_ch03-asciidoc7", range="startofrange")))The Bitcoin Core client implements a JSON-RPC interface that can also be accessed using the command-line helper +bitcoin-cli+. The command line allows us to experiment interactively with the capabilities that are also available programmatically via the API. To start, invoke the +help+ command to see a list of the available bitcoin RPC commands:
Each of these commands may take a number of parameters. To get additional help, a detailed description and information on the parameters, add the command name after help. For example, to see help on the +getblockhash+ RPC command:
----
$ bitcoin-cli help getblockhash
getblockhash index
Returns hash of block in best-block-chain at index provided.
At the end of the help information you will see two examples of the RPC command, using the +bitcoin-cli+ helper or the HTTP client +curl+. These examples demonstrate how you might call the command. Copy the first example and see the result:
The result is a block hash, which is described in more detail in the following chapters. But for now, this command should return the same result on your system, demonstrating that your Bitcoin Core node is running, is accepting commands and has information about block 1000 to return to you.
In the next sections we will demonstrate some very useful RPC commands and their expected output.
((("Bitcoin Core client","client status, getting")))((("bitcoin-cli command line helper","getinfo command")))((("client status, getting")))((("getinfo command (bitcoin-cli)")))Bitcoin's +getinfo+ RPC command displays basic information about the status of the bitcoin network node, the wallet, and the blockchain database. Use +bitcoin-cli+ to run it:
The data is returned in((("JavaScript Object Notation (JSON)"))) JavaScript Object Notation (JSON), a format that can easily be "consumed" by all programming languages but is also quite human-readable. Among this data we see the version numbers for the bitcoin software client (110200 and bitcoin protocol (70002). We see the current block height, showing us how many blocks are known to this client (396367). We also see various statistics about the bitcoin network and the settings related to this client.
It will take some time, perhaps more than a day, for the bitcoind client to "catch up" to the current blockchain height as it downloads blocks from other bitcoin clients. You can check its progress using +getinfo+ to see the number of known blocks.
In <<cup_of_coffee>>, Alice bought a cup of coffee from Bob's Cafe. Her transaction was recorded on the blockchain with transaction ID +0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2+. Let's use the API to retrieve and examine that transaction, by passing the transaction ID as a parameter:
Transaction IDs are not authoritative until a transaction has been confirmed. Absence of a transaction hash in the blockchain does not mean the transaction was not processed. This is known as((("transaction malleability"))) "transaction malleability," because transaction hashes can be modified prior to confirmation in a block. After confirmation, the txid is immutable and authoritative.
====
The command +getrawtransaction+ returns a serialized transaction in hexadecimal notation. To decode that, we use the +decoderawtransaction+ command, passing the hex data as a parameter. You can copy the hex returned by +getrawtransaction+ and paste it as a parameter to +decoderawtransaction+:
The transaction decode shows all the components of this transaction, including the transaction inputs and outputs. In this case we see that the transaction that credited our new address with 50 millibits used one input and generated two outputs. The input to this transaction was the output from a previously confirmed transaction (shown as the vin txid starting with +7957a35fe+). The two outputs correspond to the 50 millibit credit and an output with change back to the sender.
We can further explore the blockchain by examining the previous transaction referenced by its txid in this transaction using the same commands (e.g., +getrawtransaction+). Jumping from transaction to transaction we can follow a chain of transactions back as the coins are transmitted from owner address to owner address.
Exploring blocks is similar to exploring transactions. However, blocks can be referenced either by the block _height_ or by the block _hash_. First, let's find a block by its height. In <<cup_of_coffee>>, we saw that Alice's transaction was included in block 277316.
Now that we know which block Alice's transaction was included in, we can query that block. We use the +getblock+ command with the block hash as the parameter:
The block contains 419 transactions the 64th transaction listed (+0627052b...+) is Alice's coffee payment. The +height+ entry tells us this is the 277316th block in the blockchain.
The +bitcoin-cli+ helper is very useful for exploring the Bitcoin Core API and testing functions. But the whole point of an Application Programming Interface is to access functions programmatically. In this section we will demonstrate accessing Bitcoin Core from another program.
Bitcoin Core's API is a JSON-RPC interface. JSON stands for JavaScript Object Notation and it is a very convenient way to present data that both humans and programs can easily read. RPC stands for Remote Procedure Call, which means that we are calling procedures (functions) that are remote (on the Bitcoin Core node) via a network protocol. In this case, the network protocol is HTTP, or HTTPS (for encrypted connections).
When we used the +bitcoin-cli+ command to get help on a command, it showed us an example of using +curl+, the versatile command-line HTTP client to construct one of these JSON-RPC calls:
This command shows that +curl+ submits an HTTP request to the local host (127.0.0.1), connecting to the default bitcoin port (8332), and submitting a +jsonrpc+ request for the +getinfo+ method, using a +text/plain+ encoding.
If you're implementing a JSON-RPC call in your own program, you can use a generic HTTP library to construct the call, similar to what is shown in the +curl+ example above.
However, there are libraries in most every programming language that "wrap" the Bitcoin Core API in a way that makes this a lot simpler. We will use the +python-bitcoinlib+ library to simplify API access. Remember, this requires you to have a running Bitcoin Core instance which will be used to make JSON-RPC calls.
The Python script below makes a simple +getinfo+ call and prints the +block+ parameter from the data returned by Bitcoin Core:
[[rpc_example]]
.Running +getinfo+ via Bitcoin Core's JSON-RPC API
====
[source,python]
----
include::code/rpc_example.py[]
----
====
Running it, gives us the following result:
----
$ python rpc_example.py
394075
----
It tells us that our local Bitcoin Core node has 394075 blocks in its blockchain. Not a spectacular result, but it demonstrates the basic use of the library as a simplified interface to Bitcoin Core's JSON-RPC API.
Next, let's use the +getrawtransaction+ and +decodetransaction+ calls to retrieve the details of Alice's coffee payment. In the following example, we retrieve Alice's transaction and list the transaction's outputs. For each output, we show the recipient address and value. As a reminder, Alice's transaction had one output paying Bob's Cafe and one output for change back to Alice.
[[rpc_transaction]]
.Retrieving a transaction and iterating its outputs
Both of the examples above are rather simple. You don't really need a program to run them, you could just as easily use the +bitcoin-cli+ helper. The next example, however, requires several hundred RPC calls and more clearly demonstrates the use of a programmatic interface.
In <<rpc_block>>, we first retrieve block 277316, then retrieve each of the 419 transactions within by reference to each transaction ID. Next, iterate through each of the transaction's outputs and add up the value.
[[rpc_block]]
.Retrieving a block and adding all the transaction outputs
====
[source,python]
----
include::code/rpc_block.py[]
----
====
Running this code, we get:
----
$ python rpc_block.py
('Total value in block: ', Decimal('10322.07722534'))
Our example code calculates the total value transacted in this block is 10,322.07722534 BTC (inclusive of the 25 BTC reward and 0.0909 BTC in fees). Compare that to the amount reported by a block explorer site, by searching for the block hash or height. Some block explorers report the total value excluding the reward and excluding the fees. See if you can spot the difference.
((("clients, alternative", id="ix_ch03-asciidoc47", range="startofrange")))((("libraries, alternative", id="ix_ch03-asciidoc48", range="startofrange")))((("toolkits, alternative", id="ix_ch03-asciidoc49", range="startofrange"))) There are many alternative clients, libraries, toolkits and even full-node implementations in the bitcoin ecosystem. These are implemented in a variety of programming languages, offering programmers native interfaces in their preferred language.
Below we list some of the best libraries, clients and toolkits, organized by programming languages:
https://github.com/petertodd/python-bitcoinlib[python-bitcoinlib]:: ((("python-bitcoinlib library"))) A Python bitcoin library, consensus library and node by Peter Todd
https://github.com/richardkiss/pycoin[pycoin]:: ((("pycoin library")))A Python bitcoin library by Richard Kiss
https://github.com/vbuterin/pybitcointools[pybitcointools]:: ((("pybitcointools library")))A Python bitcoin library by Vitalik Buterin
==== Ruby
https://github.com/sinisterchipmunk/bitcoin-client[bitcoin-client]:: A Ruby library wrapper for the JSON-RPC API
