13 KiB

Raw Permalink Blame History

Blockbook Build Guide

Setting up your development environment

Supported environment to develop Blockbook is Linux. Although it is possible build and run Blockbook on macOS or Windows our build process is not prepared for it. But you can still build Blockbook manually.

The only dependency required to build Blockbook is Docker. You can see how to install Docker here. Manual build require additional dependencies that are described in appropriate section.

Build in Docker environment

All build operations run in Docker container in order to keep build environment isolated. Makefile in root of repository define few targets used for building, testing and packaging of Blockbook. With Docker image definitions and Debian package templates in build/docker and build/templates respectively, they are only inputs that make build process.

Docker build images are created at first execution of Makefile and that information is persisted. (Actually there are created two files in repository – .bin-image and .deb-image – that are used as tags.) Sometimes it is necessary to rebuild Docker images, it is possible by executing make build-images.

Building binary

Just run make and that is it. Output binary is stored in build directory. Note that although Blockbook is Go application it is dynamically linked with RocksDB dependencies and ZeroMQ. Therefore operating system where Blockbook will be executed still need that dependencies installed. See Manual build instructions below or install Blockbook via Debian packages.

Building debug binary

Standard binary contains no debug symbols. Execute make build-debug to get binary for debugging.

Testing

How to execute tests is described in separate document here.

Building Debian packages

Blockbook and particular coin back-end are usually deployed together. They are defined in same place as well. So typical way to build Debian packages is build Blockbook and back-end deb packages by single command. But it is not mandatory, of course.

Early releases of Blockbook weren't so friendly for extending. One had to define back-end package, Blockbook package, back-end configuration and Blockbook configuration as well. There were many options that were duplicated across configuration files and therefore error prone.

Now, all configuration options and also build options for both Blockbook and back-end are defined in single JSON file and all stuff required during build is generated dynamically.

Makefile targets follow simple pattern, there are a few prefixes that define what to build.

deb-blockbook-<coin> – Build Blockbook package for given coin.
deb-backend-<coin> – Build back-end package for given coin.
deb-<coin> – Build both Blockbook and back-end packages for given coin.
all-<coin> – Similar to deb-<coin> but clean repository and rebuild Docker image before package build. It is useful for production deployment.
all – Build both Blockbook and back-end packages for all coins.

Which coins are possible to build is defined in configs/coins. Each coin has to have JSON config file there.

For example we want to build some packages for Bitcoin and Bitcoin Testnet.

# make all-bitcoin deb-blockbook-bitcoin_testnet
...
# ls build/*.deb
build/backend-bitcoin_0.21.0-satoshilabs-1_amd64.deb  build/blockbook-bitcoin_0.3.5_amd64.deb  build/blockbook-bitcoin-testnet_0.3.5_amd64.deb

We have built one back-end package, for Bitcoin, and two Blockbook packages, for Bitcoin and Bitcoin Testnet. The all-bitcoin initially cleaned the build directory and rebuilt the Docker build image.

Extra variables

There are few variables that can be passed to make in order to modify build process:

BASE_IMAGE: Specifies the base image of the Docker build image. By default, it chooses the same Linux distro as the host machine but you can override it this way make BASE_IMAGE=debian:10 all-bitcoin to make a build for Debian 10.

Please be aware that we are running our Blockbooks on Debian 9 and Debian 10 and do not offer support with running it on other distros.

NO_CACHE: Common behaviour of Docker image build is that build steps are cached and next time they are executed much faster. Although this is a good idea, when something went wrong you will need to override this behaviour somehow. Execute this command: make NO_CACHE=true all-bitcoin.

TCMALLOC: RocksDB, the storage engine used by Blockbook, allows to use alternative memory allocators. Use the TCMALLOC variable to specify Google's TCMalloc allocator make TCMALLOC=true all-bitcoin. To run Blockbook built with TCMalloc, the library must be installed on the target server, for example by sudo apt-get install google-perftools.

Naming conventions and versioning

All configuration keys described below are in coin definition file in configs/coins.

install and data directories

Both Blockbook and back-end have separated install and data directories. They use common preffix and are defined in configs/environ.json and all templates use them.

back-end install directory is /opt/coins/nodes/<coin>.
back-end data directory is /opt/coins/data/<coin>/backend.
Blockbook install directory is /opt/coins/blockbook/<coin>.
Blockbook data directory is /opt/coins/data/<coin>/blockbook.

coin used above is defined in coin.alias in coin definition file.

package names

Package names are defined in backend.package_name and blockbook.package_name in coin definition file. We use simple pattern <prefix>-<coin> to name packages where prefix is either blockbook or backend and coin is made similarly to coin.alias. We use convention that coin name uses lowercase characters and dash '-' as a word delimiter. Testnet versions of coins must have -testnet suffix. That differs from coin.alias because underscore has a special meaning in Debian packaging. For example there are packages backend-bitcoin and blockbook-bitcoin-testnet.

user names

User names are defined in backend.system_user and blockbook.system_user in coin definition file. We follow common Linux conventions, user names use lowercase characters and dash '-' as a word delimiter.

Back-end user name use coin name only, including testnet services. For example there is bitcoin user for both backend-bitcoin and backend-bitcoin-testnet packages.

Blockbook user name has blockbook- prefix and coin name (made same as back-end version). For example there is blockbook-bitcoin user for both blockbook-bitcoin and blockbook-bitcoin-testnet packages.

back-end versioning

Since we have to distinguish version of coin distribution and version of our configuration we follow standard Debian package versioning rules (for details see Debian policy). There is upstream version and revision both defined in coin definition file in backend.version and backend.package_revision, respectively.

blockbook versioning

Blockbook versioning is much simpler. There is only one version defined in configs/environ.json.

Back-end building

Because we don't keep back-end archives inside out repository we download them during build process. Build steps are these: download, verify and extract archive, prepare distribution and make package.

All configuration keys described below are in coin definition file in configs/coins.

download archive

URL from where is archive downloaded is defined in backend.binary_url.

verify archive

There are three different approaches how is archive verification done. Some projects use PGP sign of archive, some have signed sha256 sums and some don't care about verification at all. So there is option backend.verification_type that could be gpg, gpg-sha256 or sha256 and chooses particular method.

gpg type require file with digital sign and maintainer's public key imported in Docker build image (see below). Sign file is downloaded from URL defined in backend.verification_source. Than is passed to gpg in order to verify archvie.

gpg-sha256 type require signed checksum file and maintainer's public key imported in Docker build image (see below). Checksum file is downloaded from URL defined in backend.verification_source. Then is verified by gpg and passed to sha256sum in order to verify archive.

sha256 type is used for coins that don't support verification at all. In backend.verification_source is defined hexadecimal string that is compared with output of sha256sum. Although this solution is not secure, it avoid download errors and other surprises at least.

gpg and gpg-sha256 types require maintainer's public key imported in Docker build image. It is not expected that maintainer's key will change requently while sing or checksum files are changed every release, so it is ideal to store maintainer's key within image definition. Public keys are stored in build/docker/deb/gpg-keys directory. Docker image must be rebuilt by calling make build-images.

extract archive

Extraction command is defined in backend.extract_command. Content of archive must be extracted to ./backend directory. See bitcoin.json and vertcoin.json for different approaches.

prepare distribution

There are two steps in this stage – exclude unnecessary files and generate configuration.

Some files are not required for server deployment, some binaries have unnecessary dependencies, so it is good idea to extract these files from output package. Files to extract are listed in backend.exclude_files. Note that paths are relative to backend directory where archive is extracted.

Configuration is described in config.md.

Manual build

Instructions below are focused on Debian 9 (Stretch). If you want to use another Linux distribution or operating system like macOS or Windows, please read instructions specific for each project.

Setup go environment:

wget https://golang.org/dl/go1.14.2.linux-amd64.tar.gz && tar xf go1.14.2.linux-amd64.tar.gz
sudo mv go /opt/go
sudo ln -s /opt/go/bin/go /usr/bin/go
# see `go help gopath` for details
mkdir $HOME/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin

Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md and compile the static_lib and tools

sudo apt-get update && sudo apt-get install -y \
    build-essential git wget pkg-config libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
git checkout v6.8.1
CFLAGS=-fPIC CXXFLAGS=-fPIC make release

Setup variables for gorocksdb: https://github.com/tecbot/gorocksdb

export CGO_CFLAGS="-I/path/to/rocksdb/include"
export CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -ldl -lbz2 -lsnappy -llz4"

Install ZeroMQ: https://github.com/zeromq/libzmq

git clone https://github.com/zeromq/libzmq
cd libzmq
./autogen.sh
./configure
make
sudo make install

Get blockbook sources, install dependencies, build:

cd $GOPATH/src
git clone https://github.com/trezor/blockbook.git
cd blockbook
go build

Example command

Blockbook require full node daemon as its back-end. You are responsible for proper installation. Port numbers and daemon configuration are defined in configs/coins and build/templates/backend/config directories. You should use specific installation process for particular coin you want run (e.g. https://bitcoin.org/en/full-node#other-linux-distributions for Bitcoin).

When you have running back-end daemon you can start Blockbook. It is highly recommended use ports described in ports.md for both Blockbook and back-end daemon. You can use contrib/scripts/build-blockchaincfg.sh that will generate Blockbook's blockchain configuration from our coin definition files.

Also, check that your operating system open files limit is set to high enough value - recommended is at least 20000.

Example for Bitcoin:

./contrib/scripts/build-blockchaincfg.sh <coin>
./blockbook -sync -blockchaincfg=build/blockchaincfg.json -internal=:9030 -public=:9130 -certfile=server/testcert -logtostderr

This command starts Blockbook with parallel synchronization and providing HTTP and Socket.IO interface, with database in local directory data and established ZeroMQ and RPC connections to back-end daemon specified in configuration file passed to -blockchaincfg option.

Blockbook logs to stderr (option -logtostderr) or to directory specified by parameter -log_dir . Verbosity of logs can be tuned by command line parameters -v and -vmodule, for details see https://godoc.org/github.com/golang/glog.

You can check that Blockbook is running by simple HTTP request: curl https://localhost:9130. Returned data is JSON with some run-time information. If the port is closed, Blockbook is syncing data.

13 KiB Raw Permalink Blame History Unescape Escape