blockbook/docs/build.md

# Blockbook Build Guide

## Setting up your development environment

Supported environment to develop Blockbook is Linux. Although it is possible build and run Blockbook on macOS
or Windows our build process is not prepared for it. But you can still build Blockbook [manually](#manual-build).

The only dependency required to build Blockbook is Docker. You can see how to install Docker [here](https://docs.docker.com/install/linux/docker-ce/debian/).
Manual build require additional dependencies that are described in appropriate section.

## Build in Docker environment

All build operations run in Docker container in order to keep build environment isolated. Makefile in root of repository
define few targets used for building, testing and packaging of Blockbook. With Docker image definitions and Debian
package templates in *build/docker* and *build/templates* respectively, they are only inputs that make build process.

Docker build images are created at first execution of Makefile and that information is persisted. (Actually there are
created two files in repository – .bin-image and .deb-image – that are used as tags.) Sometimes it is necessary to
rebuild Docker images, it is possible by executing `make build-images`.

### Building binary

Just run `make` and that is it. Output binary is stored in *build* directory. Note that although Blockbook is Go application
it is dynamically linked with RocksDB dependencies and ZeroMQ. Therefore operating system where Blockbook will be
executed still need that dependencies installed. See [Manual build](#manual-build) instructions below or install
Blockbook via Debian packages.

### Building debug binary

Standard binary contains no debug symbols. Execute `make build-debug` to get binary for debugging.

### Testing

How to execute tests is described in separate document [here](/docs/testing.md).

### Building Debian packages

Blockbook and particular coin back-end are usually deployed together. They are defined in same place as well.
So typical way to build Debian packages is build Blockbook and back-end deb packages by single command. But it is not
mandatory, of course.

> Early releases of Blockbook weren't so friendly for extending. One had to define back-end package, Blockbook package,
> back-end configuration and Blockbook configuration as well. There were many options that were duplicated across
> configuration files and therefore error prone.
>
> Actually all configuration options and also build options for both Blockbook and back-end are defined in single JSON
> file and all stuff required during build is generated dynamically.

Makefile targets follow simple pattern, there are few prefixes that define what to build.

* *deb-blockbook-<coin>* – Build Blockbook package for given coin.

* *deb-backend-<coin>* – Build back-end package for given coin.

* *deb-<coin>* – Build both Blockbook and back-end packages for given coin.

* *all-<coin>* – Similar to deb-<coin> but clean repository and rebuild Docker image before package build. It is useful
  for production deployment.

* *all* – Build both Blockbook and back-end packages for all coins.

Which coins are possible to build is defined in *configs/coins*. Particular coin has to have JSON config file there.

For example we want to build some packages for Bitcoin and Bitcoin Testnet.

```bash
# make all-bitcoin deb-backend-bitcoin_testnet
...
# ls build/*.deb
build/backend-bitcoin_0.16.1-satoshilabs-1_amd64.deb  build/backend-bitcoin-testnet_0.16.1-satoshilabs-1_amd64.deb  build/blockbook-bitcoin_0.0.6_amd64.deb
```

We have built two back-end packages – for Bitcoin and Testnet – and Blockbook package for Bitcoin. Before build have
been performed there was cleaned build directory and rebuilt Docker image.

### Extra variables

There are few variables that can be passed to make in order to modify build process.

In general, build of Blockbook binary require some dependencies. They are downloaded automatically during build process
but if you need to build binary repeatedly it consumes a lot of time. Here comes variable *UPDATE_VENDOR* that if is
unset says that build process uses *vendor* (i.e. dependencies) from your local repository. For example:
`make deb-bitcoin UPDATE_VENDOR=0`. But before the command is executed there must be *vendor* directory populated,
you can do it by calling `go mod vendor`. See [Manual build](#manual-build) instructions below.

All build targets allow pass additional parameters to underlying command inside container. It is possible via ARGS
variable. For example if you want run only subset of unit-tests, you will perform it by calling:
`make test ARGS='-run TestBitcoinRPC' UPDATE_VENDOR=0`

Common behaviour of Docker image build is that build steps are cached and next time they are executed much faster.
Although this is a good idea, when something went wrong you will need to override this behaviour somehow. Execute this
command: `make build-images NO_CACHE=true`.

### On naming conventions and versioning

All configuration keys described below are in coin definition file in *configs/coins*.

**install and data directories**

Both Blockbook and back-end have separated install and data directories. They use common preffix and are defined in
*configs/environ.json* and all templates use them.

* back-end install directory is */opt/coins/nodes/<coin>*.
* back-end data directory is */opt/coins/data/<coin>/backend*.
* Blockbook install directory is */opt/coins/blockbook/<coin>*.
* Blockbook data directory is */opt/coins/data/<coin>/blockbook*.

*coin* used above is defined in *coin.alias* in coin definition file.

**package names**

Package names are defined in *backend.package_name* and *blockbook.package_name* in coin definition file. We use
simple pattern *<prefix>-<coin>* to name packages where *prefix* is either *blockbook* or *backend* and
*coin* is made similarly to *coin.alias*. We use convention that coin name uses lowercase characters and dash '-' as
a word delimiter. Testnet versions of coins must have *-testnet* suffix. That differs from *coin.alias* because
underscore has a special meaning in Debian packaging. For example there are packages *backend-bitcoin* and
*blockbook-bitcoin-testnet*.

**user names**

User names are defined in *backend.system_user* and *blockbook.system_user* in coin definition file. We follow common
Linux conventions, user names use lowercase characters and dash '-' as a word delimiter.

Back-end user name use coin name only, including testnet services. For example there is *bitcoin* user for both
*backend-bitcoin* and *backend-bitcoin-testnet* packages.

Blockbook user name has *blockbook-* prefix and coin name (made same as back-end version). For example there is
*blockbook-bitcoin*  user for both *blockbook-bitcoin* and *blockbook-bitcoin-testnet* packages.

**back-end versioning**

Since we have to distinguish version of coin distribution and version of our configuration we follow standard Debian
package versioning rules (for details see
[Debian policy](https://www.debian.org/doc/debian-policy/ch-controlfields.html#version)). There is upstream version
and revision both defined in coin definition file in *backend.version* and *backend.package_revision*, respectively.

**blockbook versioning**

Blockbook versioning is much simpler. There is only one version defined in *configs/environ.json*.

### On back-end building

Because we don't keep back-end archives inside out repository we download them during build process. Build steps
are these: download, verify and extract archive, prepare distribution and make package.

All configuration keys described below are in coin definition file in *configs/coins*.

**download archive**

URL from where is archive downloaded is defined in *backend.binary_url*.

**verify archive**

There are three different approaches how is archive verification done. Some projects use PGP sign of archive, some
have signed sha256 sums and some don't care about verification at all. So there is option *backend.verification_type* that
could be *gpg*, *gpg-sha256* or *sha256* and chooses particular method.

*gpg* type require file with digital sign and maintainer's public key imported in Docker build image (see below). Sign
file is downloaded from URL defined in *backend.verification_source*. Than is passed to gpg in order to verify archvie.

*gpg-sha256* type require signed checksum file and maintainer's public key imported in Docker build image (see below).
Checksum file is downloaded from URL defined in *backend.verification_source*. Then is verified by gpg and passed to
sha256sum in order to verify archive.

*sha256* type is used for coins that don't support verification at all. In *backend.verification_source* is defined
hexadecimal string that is compared with output of sha256sum. Although this solution is not secure, it avoid download
errors and other surprises at least.

*gpg* and *gpg-sha256* types require maintainer's public key imported in Docker build image. It is not expected that
maintainer's key will change requently while sing or checksum files are changed every release, so it is ideal to
store maintainer's key within image definition. Public keys are stored in *build/docker/deb/gpg-keys* directory. Docker
image must be rebuilt by calling `make build-images`.

**extract archive**

Extraction command is defined in *backend.extract_command*. Content of archive must be extracted to `./backend` directory.
See bitcoin.json and vertcoin.json for different approaches.

**prepare distribution**

There are two steps in this stage – exclude unnecessary files and generate configuration.

Some files are not required for server deployment, some binaries have unnecessary dependencies, so it is good idea to
extract these files from output package. Files to extract are listed in *backend.exclude_files*. Note that paths are
relative to *backend* directory where archive is extracted.

Configuration is described in [config.md](/docs/config.md).

## Manual build

Instructions below are focused on Debian 9 (Stretch). If you want to use another Linux distribution or operating system
like macOS or Windows, please read instructions specific for each project.

Setup go environment:

```
wget https://dl.google.com/go/go1.14.2.linux-amd64.tar.tz && tar xf go1.14.2.linux-amd64.tar.gz
sudo mv go /opt/go
sudo ln -s /opt/go/bin/go /usr/bin/go
# see `go help gopath` for details
mkdir $HOME/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
```

Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md
and compile the static_lib and tools

```
sudo apt-get update && sudo apt-get install -y \
    build-essential git wget pkg-config libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
CFLAGS=-fPIC CXXFLAGS=-fPIC make release
```

Setup variables for gorocksdb: https://github.com/tecbot/gorocksdb

```
export CGO_CFLAGS="-I/path/to/rocksdb/include"
export CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -ldl -lbz2 -lsnappy -llz4"
```

Install ZeroMQ: https://github.com/zeromq/libzmq

Get blockbook sources, install dependencies, build:

```
cd $GOPATH/src
git clone https://github.com/trezor/blockbook.git
cd blockbook
go build
```

### Example command

Blockbook require full node daemon as its back-end. You are responsible for proper installation. Port numbers and
daemon configuration are defined in *configs/coins* and *build/templates/backend/config* directories. You should use
specific installation process for particular coin you want run (e.g. https://bitcoin.org/en/full-node#other-linux-distributions for Bitcoin).

When you have running back-end daemon you can start Blockbook. It is highly recommended use ports described in [ports.md](/docs/ports.md)
for both Blockbook and back-end daemon. You can use *contrib/scripts/build-blockchaincfg.sh* that will generate
Blockbook's blockchain configuration from our coin definition files.

Example for Bitcoin:
```
contrib/scripts/build-blockchaincfg.sh
./blockbook -sync -blockchaincfg=build/blockchaincfg.json -internal=:9030 -public=:9130 -certfile=server/testcert -logtostderr
```

This command starts Blockbook with parallel synchronization and providing HTTP and Socket.IO interface, with database
in local directory *data* and established ZeroMQ and RPC connections to back-end daemon specified in configuration
file passed to *-blockchaincfg* option.

Blockbook logs to stderr (option *-logtostderr*) or to directory specified by parameter *-log_dir* . Verbosity of logs can be tuned
by command line parameters *-v* and *-vmodule*, for details see https://godoc.org/github.com/golang/glog.

You can check that Blockbook is running by simple HTTP request: `curl https://localhost:9130`. Returned data is JSON with some
run-time information. If port is closed, Blockbook is syncing data.