From ef12666f706150e58cf455b3ca2c432bab160fe1 Mon Sep 17 00:00:00 2001 From: Jakub Matys Date: Wed, 1 Aug 2018 15:49:06 +0200 Subject: [PATCH] Upgrade of documentation: build, config, contributing, readme --- CONTRIBUTING.md | 85 +++++++ README.md | 153 ++---------- bchain/coins/btc/btc.md | 50 ---- bchain/coins/btc/btctestnet.md | 49 ---- bchain/coins/eth/eth.md | 25 -- bchain/coins/eth/ethropsten.md | 25 -- bchain/coins/zec/zec.md | 57 ----- configs/coins/ethereum.json | 2 + configs/coins/ethereum_classic.json | 14 ++ configs/coins/ethereum_testnet_ropsten.json | 1 + contrib/scripts/generate-port-registry.go | 263 ++++++++++++++++++++ docs/build.md | 263 ++++++++++++++++++++ docs/config.md | 97 ++++++++ docs/ports.md | 24 ++ docs/rocksdb.md | 44 ++++ 15 files changed, 809 insertions(+), 343 deletions(-) create mode 100644 CONTRIBUTING.md delete mode 100644 bchain/coins/btc/btc.md delete mode 100644 bchain/coins/btc/btctestnet.md delete mode 100644 bchain/coins/eth/eth.md delete mode 100644 bchain/coins/eth/ethropsten.md delete mode 100644 bchain/coins/zec/zec.md create mode 100644 configs/coins/ethereum_classic.json create mode 100755 contrib/scripts/generate-port-registry.go create mode 100644 docs/build.md create mode 100644 docs/config.md create mode 100644 docs/ports.md create mode 100644 docs/rocksdb.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..36c67fd8 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,85 @@ +# Blockbook Contributor Guide + +Blockbook is back-end service for Trezor wallet. Although it is open source, design and development of core packages +is done by Trezor developers in order to keep Blockbook compatible with Trezor. If you feel you could use Blockbook +for another purposes, we recommend you to make a fork. + +However you can still help us find bugs or add support for new coins. + +## Development environment + +Instructions to set up your development environment and build Blockbook are described in separated +[document](/docs/build.md). + +## How can I contribute? + +### Reporting bugs + +### Adding coin support + +Trezor harware wallet supports over 500 coins, see https://trezor.io/coins/. You are free to add support for any of +them to Blockbook. Actually implemented coins are listed [here](/docs/ports.md). + +You should follow few steps bellow to get smooth merge of your PR. + +> Altough we are happy for support of new coins we have not enough capacity to run them all on our infrastructure. +> Actually we can run Blockbook instances only for coins supported by Trezor wallet. If you want to have Blockbook +> instance for your coin, you will have to deploy your own server. + +#### Add coin definition + +Coin definitions are stored in JSON files in *configs/coins* directory. They are single source of Blockbook +configuration, Blockbook and back-end package definition and build metadata. Since Blockbook supports only single +coin index per running instance, every coin (including testnet) must have single definition file. + +All options of coin definition are described in [config.md](/docs/config.md). + +Because most of coins are fork of Bitcoin and they have similar way to install and configure their daemon, we use +templates to generate package definition and configuration files during build process. It is similar to build Blockbook +package too. Templates are filled with data from coin definition. Although build process generate packages +automatically, there is sometimes necessary see intermediate step. You can generate all files by calling +`go run build/templates/generate.go coin` where *coin* is name of definition file without .json extension. Files are +generated to *build/pkg-defs* directory. + +Good examples of coin configuration are +[*configs/coins/bitcoin.json*](configs/coins/bitcoin.json) and +[*configs/coins/ethereum.json*](configs/coins/ethereum.json) for Bitcoin-like coins and different coins, respectively. + +Usually you have to update only few options that differ from Bitcoin definition. At first there are base information +about coin in section *coin* – name, alias etc. Then update port information in *port* section. We keep port series as +listed in [port registry](/docs/ports.md). Select next port numbers in series. Port numbers must be unique across all +port definitions. + +In section *backend* update information how to build and configure backend service. When back-end package is built, +build process downloads installation archive, verify and extract it. How it is done is described in +[build guide](/docs/build.md#on-back-end-building). Naming conventions and versioning are described +also in [build guide](/docs/build.md#on-naming-conventions-and-versioning). You have to update *package_name*, +*package_revision*, *system_user*, *version*, *binary_url*, *verification_type*, *verification_source*, *extract_command* and +*exclude_files*. Also update information whether service runs mainnet or testnet network in *mainnet* option. + +In section *blockbook* update information how to build and configure Blockbook service. Usually they are only +*package_name*, *system_user* and *explorer_url* options. Naming conventions are are described +[here](/docs/build.md#on-naming-conventions-and-versioning). + +Update *package_maintainer* and *package_maintainer_email* options in section *meta*. + +Execute script *contrib/scripts/generate-port-registry.go* that will update *docs/ports.md*. + +Now you can try generate package definitions as described above in order to check outputs. + +##### Go template evaluation note + +We use *text/template* package to generate package definitions and configuration files. Some options in coin definition +are also templates and are executed inside base template. Use `{{.path}}` syntax to refer values in coin definition, +where *.path* can be for example *.Blockbook.BlockChain.Parse*. Go uses CammelCase notation so references inside templates +as well. Note that dot at the beginning is mandatory. Go template syntax is fully documented +[here](https://godoc.org/text/template). + +TODO: +* script that checks unique port numbers + +#### Add coin implementation + +#### Add tests + +#### Deploy public server diff --git a/README.md b/README.md index 1fe7325b..fe9fdfa7 100644 --- a/README.md +++ b/README.md @@ -1,161 +1,40 @@ -# blockbook +# Blockbook -## **blockbook is currently in the state of heavy development, do not expect this documentation to be up to date** +> **Blockbook is currently in the state of heavy development, do not expect this documentation to be up to date** -## Build and install using docker +## Build and installation instructions -Run in the project root - -``` -make all -``` - -to create blockbook debian packages. - -## Install manually - -Setup go environment (Debian 9): - -``` -sudo apt-get update && apt-get install -y \ - build-essential git wget pkg-config lxc-dev libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev -cd /opt -wget https://storage.googleapis.com/golang/go1.9.2.linux-amd64.tar.gz && tar xf go1.9.2.linux-amd64.tar.gz -sudo ln -s /opt/go/bin/go /usr/bin/go -go help gopath -``` - -Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md -and compile the static_lib and tools - -``` -git clone https://github.com/facebook/rocksdb.git -cd rocksdb -make release -``` - -Setup variables for gorocksdb: https://github.com/tecbot/gorocksdb - -``` -export CGO_CFLAGS="-I/path/to/rocksdb/include" -export CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4" -``` - -Install ZeroMQ: https://github.com/zeromq/libzmq - -Install go-dep tool: -``` -RUN go get github.com/golang/dep/cmd/dep -``` - -Get blockbook sources, install dependencies, build: - -``` -cd $GOPATH/src -git clone https://github.com/trezor/blockbook.git -cd blockbook -dep ensure -go build -``` - -## Usage - -``` -./blockbook --help -``` - -## Example command -To run blockbook with fast synchronization, connection to ZeroMQ and providing https and socket.io interface, with database in local directory *data* and connected to local bitcoind with configuration specified by parameter *-blockchaincfg*: -``` -./blockbook -sync -blockchaincfg=configs/bitcoin_testnet.json -internal=127.0.0.1:8333 -public=127.0.0.1:8334 -certfile=server/testcert -logtostderr -``` -Blockbook logs to stderr *-logtostderr* or to directory specified by parameter *-log_dir* . Verbosity of logs can be tuned by command line parameters *-v* and *-vmodule*, details at https://godoc.org/github.com/golang/glog +Develper build guide is [here](/docs/build.md). +Sysadmin installation guide is [here](https://wiki.trezor.io/Blockbook). # Implemented coins -- [Bitcoin](bchain/coins/btc/btc.md) -- [Bitcoin Testnet](bchain/coins/btc/btctestnet.md) +The most significant coins implemented by Blockbook are: + +- Bitcoin +- Bitcoin Testnet - Bcash - Bcash Testnet - Bgold -- [ZCash](bchain/coins/zec/zec.md) +- ZCash - ZCash Testnet - Dash - Dash Testnet - Litecoin - Litecoin Testnet -- [Ethereum](bchain/coins/eth/eth.md) -- [Ethereum Testnet Ropsten](bchain/coins/eth/ethropsten.md) +- Ethereum +- Ethereum Testnet Ropsten + +They are also supported by Trezor wallet. List of all coins is [here](/docs/ports.md). # Data storage in RocksDB -Blockbook stores data the key-value store RocksDB. Data are stored in binary form to save space. -The data are separated to different column families: - -- **default** - - at the moment not used, will store statistical data etc. - -- **height** - maps *block height* to *block hash* - - *Block heigh* stored as array of 4 bytes (big endian uint32) - *Block hash* stored as array of 32 bytes - - Example - the first four blocks (all data hex encoded) -``` -0x00000000 : 0x000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943 -0x00000001 : 0x00000000b873e79784647a6c82962c70d228557d24a747ea4d1b8bbe878e1206 -0x00000002 : 0x000000006c02c8ea6e4ff69651f7fcde348fb9d557a06e6957b65552002a7820 -0x00000003 : 0x000000008b896e272758da5297bcd98fdc6d97c9b765ecec401e286dc1fdbe10 -``` - -- **outputs** - maps *output script+block height* to *array of outpoints* - - *Output script (ScriptPubKey)+block height* stored as variable length array of bytes for output script + 4 bytes (big endian uint32) block height - *array of outpoints* stored as array of 32 bytes for transaction id + variable length outpoint index for each outpoint - - Example - (all data hex encoded) -``` -0x001400efeb484a24a1c1240eafacef8566e734da429c000e2df6 : 0x1697966cbd76c75eb9fc736dfa3ba0bc045999bab1e8b10082bc0ba546b0178302 -0xa9143e3d6abe282d92a28cb791697ba001d733cefdc7870012c4b1 : 0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d301 -``` - -- **inputs** - maps *transaction outpoint* to *input transaction* that spends it - - *Transaction outpoint* stored as array of 32 bytes for transaction id + variable length outpoint index - *Input transaction* stored as array of 32 bytes for transaction id + variable length input index - - Example - (all data hex encoded) -``` -0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d300 : 0x0a7aa90ea0269c79f844c516805e4cac594adb8830e56fca894b66aab19136a428 -0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d301 : 0x4303a9fcfe6026b4d33ba488df6443c9a99bca7b7fcb7c6f6cd65cea24a749b700 -``` +Blockbook stores data the key-value store RocksDB. Database format is described [here](/docs/rocksdb.md). ## Registry of ports -| coin | blockbook internal port | blockbook public port | backend rpc port | zmq port | -|--------------------------|-------------------------|-----------------------|--------------------|----------| -| Bitcoin | 9030 | 9130 | 8030 | 38330 | -| Bcash | 9031 | 9131 | 8031 | 38331 | -| Zcash | 9032 | 9132 | 8032 | 38332 | -| Dash | 9033 | 9133 | 8033 | 38333 | -| Litecoin | 9034 | 9134 | 8034 | 38334 | -| Bgold | 9035 | 9135 | 8035 | 38335 | -| Ethereum | 9036 | 9136 | 8036 ws, 8136 http | 38336* | -| Ethereum Classic | 9037 | 9137 | 8037 | 38337* | -| Dogecoin | 9038 | 9138 | 8038 | 38338 | -| Namecoin | 9039 | 9139 | 8039 | 38339 | -| Vertcoin | 9040 | 9140 | 8040 | 38340 | -| Bitcoin Testnet | 19030 | 1913 | 18030 | 48330 | -| Bcash Testnet | 19031 | 1913 | 18031 | 48331 | -| Zcash Testnet | 19032 | 1913 | 18032 | 48332 | -| Dash Testnet | 19033 | 1913 | 18033 | 48333 | -| Litecoin Testnet | 19034 | 1913 | 18034 | 48334 | -| Ethereum Testnet Ropsten | 19036 | 19136 | 18036 | 48336* | -| Vertcoin Testnet | 19040 | 19140 | 18040 | 48340 | - -\* geth listens on this port, however not as zmq service +Reserved ports are described [here](/docs/ports.md) ## Todo diff --git a/bchain/coins/btc/btc.md b/bchain/coins/btc/btc.md deleted file mode 100644 index 2be2fdd7..00000000 --- a/bchain/coins/btc/btc.md +++ /dev/null @@ -1,50 +0,0 @@ -## BTC Setup -Get Bitcoin Core -``` -wget https://bitcoin.org/bin/bitcoin-core-0.16.0/bitcoin-0.15.1-x86_64-linux-gnu.tar.gz -tar -xf bitcoin-0.16.0-x86_64-linux-gnu.tar.gz -``` - -Data are stored in */data/btc*, in folders */data/btc/bitcoin* for Bitcoin Core data, */data/btc/blockbook* for Blockbook data. - -Create configuration file */data/btc/bitcoin/bitcoin.conf* with content -``` -daemon=1 -server=1 -rpcuser=rpc -rpcpassword=rpc -rpcport=8030 -txindex=1 -``` -Create script that starts the bitcoind daemon *run-btc-bitcoind.sh* with increased rpcworkqueue and configured zeromq -``` -#!/bin/bash - -bitcoin-0.15.1/bin/bitcoind -datadir=/data/btc/bitcoin -rpcworkqueue=32 -zmqpubhashtx=tcp://127.0.0.1:38330 -zmqpubhashblock=tcp://127.0.0.1:38330 -zmqpubrawblock=tcp://127.0.0.1:38330 -zmqpubrawtx=tcp://127.0.0.1:38330 -``` -Run the *run-btc-bitcoind.sh* to get initial import of data. - -Create blockchain configuration file */data/testnet/blockbook/btc.json* -``` -{ - "rpcURL": "http://127.0.0.1:8030", - "rpcUser": "rpc", - "rpcPass": "rpc", - "rpcTimeout": 25, - "parse": true, - "zeroMQBinding": "tcp://127.0.0.1:38330" -} -``` - -Create script that runs blockbook *run-btc-blockbook.sh* -``` -#!/bin/bash - -cd go/src/blockbook -./blockbook -coin=btc -blockchaincfg=/data/btc/blockbook/btc.json -datadir=/data/btc/blockbook/db -sync -internal=:9030 -public=:9130 -certfile=server/testcert -explorer=https://bitcore1.trezor.io/ $1 -``` -To run blockbook with logging to file (run with nohup or daemonize or using screen) -``` -./run-btc-blockbook.sh 2>/data/btc/blockbook/blockbook.log -``` - diff --git a/bchain/coins/btc/btctestnet.md b/bchain/coins/btc/btctestnet.md deleted file mode 100644 index bbb59ac0..00000000 --- a/bchain/coins/btc/btctestnet.md +++ /dev/null @@ -1,49 +0,0 @@ -## BTC Testnet Setup -Get Bitcoin Core -``` -wget https://bitcoin.org/bin/bitcoin-core-0.16.0/bitcoin-0.15.1-x86_64-linux-gnu.tar.gz -tar -xf bitcoin-0.16.0-x86_64-linux-gnu.tar.gz -``` -Data are stored in */data/testnet*, in folders */data/testnet/bitcoin* for Bitcoin Core data, */data/testnet/blockbook* for Blockbook data. - -Create configuration file */data/testnet/bitcoin/bitcoin.conf* with content -``` -testnet=1 -daemon=1 -server=1 -rpcuser=rpc -rpcpassword=rpc -rpcport=18030 -txindex=1 -``` -Create script that starts the bitcoind daemon *run-testnet-bitcoind.sh* with increased rpcworkqueue and configured zeromq -``` -#!/bin/bash - -bitcoin-0.15.1/bin/bitcoind -datadir=/data/testnet/bitcoin -rpcworkqueue=32 -zmqpubhashtx=tcp://127.0.0.1:48330 -zmqpubhashblock=tcp://127.0.0.1:48330 -zmqpubrawblock=tcp://127.0.0.1:48330 -zmqpubrawtx=tcp://127.0.0.1:48330 -``` -Run the *run-testnet-bitcoind.sh* to get initial import of data. - -Create blockchain configuration file */data/testnet/blockbook/btc-testnet.json* -``` -{ - "rpcURL": "http://127.0.0.1:18030", - "rpcUser": "rpc", - "rpcPass": "rpc", - "rpcTimeout": 25, - "parse": true, - "zeroMQBinding": "tcp://127.0.0.1:48330" -} -``` - -Create script that runs blockbook *run-testnet-blockbook.sh* -``` -#!/bin/bash - -cd go/src/blockbook -./blockbook -coin=btc-testnet -blockchaincfg=/data/testnet/blockbook/btc-testnet.json -datadir=/data/testnet/blockbook/db -sync -internal=:19030 -public=:19130 -certfile=server/testcert -explorer=https://testnet-bitcore1.trezor.io $1 -``` -To run blockbook with logging to file (run with nohup or daemonize or using screen) -``` -./run-testnet-blockbook.sh 2>/data/testnet/blockbook/blockbook.log -``` diff --git a/bchain/coins/eth/eth.md b/bchain/coins/eth/eth.md deleted file mode 100644 index 1db56280..00000000 --- a/bchain/coins/eth/eth.md +++ /dev/null @@ -1,25 +0,0 @@ -## Ethereum Testnet Setup -Get Ethereum -``` -git clone https://github.com/ethereum/go-ethereum -cd go-ethereum/ -make geth -``` -Data are stored in */data/eth*, in folders */data/eth/eth* for Ethereum data, */data/eth/blockbook* for Blockbook data. - -Run geth with rpc and websocket interfaces, bound to all ip addresses - insecure! (run with nohup or daemonize or using screen) -``` -go-ethereum/build/bin/geth --syncmode "full" --cache 1024 --datadir /data/eth/eth --port "35555" --rpc --rpcport 8545 -rpcaddr 0.0.0.0 --rpccorsdomain "*" --ws --wsaddr 0.0.0.0 --wsport 8546 --wsorigins "*" 2>/data/eth/eth/eth.log -``` - -Create script that runs blockbook *run-eth-blockbook.sh* -``` -#!/bin/bash - -cd go/src/blockbook -./blockbook -coin=eth -blockchaincfg=/data/eth/blockbook/eth.json -datadir=/data/eth/blockbook/db -sync -internal=:8555 -public=:8556 -certfile=server/testcert $1 -``` -To run blockbook with logging to file (run with nohup or daemonize or using screen) -``` -./run-eth-blockbook.sh 2>/data/eth/blockbook/blockbook.log -``` diff --git a/bchain/coins/eth/ethropsten.md b/bchain/coins/eth/ethropsten.md deleted file mode 100644 index 0e4bb161..00000000 --- a/bchain/coins/eth/ethropsten.md +++ /dev/null @@ -1,25 +0,0 @@ -## Ethereum Testnet Setup -Get Ethereum -``` -git clone https://github.com/ethereum/go-ethereum -cd go-ethereum/ -make geth -``` -Data are stored in */data/eth-testnet*, in folders */data/eth-testnet/eth* for Ethereum data, */data/eth-testnet/eth/blockbook* for Blockbook data. - -Run geth with rpc and websocket interfaces, bound to all ip addresses - insecure! (run with nohup or daemonize or using screen) -``` -go-ethereum/build/bin/geth --testnet --datadir /data/eth-testnet/eth --rpc --rpcport 18545 -rpcaddr 0.0.0.0 --rpccorsdomain "*" --ws --wsaddr 0.0.0.0 --wsport 18546 --wsorigins "*" 2>/data/eth-testnet/eth/eth.log -``` - -Create script that runs blockbook *run-eth-testnet-blockbook.sh* -``` -#!/bin/bash - -cd go/src/blockbook -./blockbook -coin=eth-testnet -blockchaincfg=/data/eth-testnet/blockbook/eth-testnet.json -datadir=/data/eth-testnet/blockbook/db -sync -internal=:18555 -public=:18556 -certfile=server/testcert $1 -``` -To run blockbook with logging to file (run with nohup or daemonize or using screen) -``` -./run-eth-testnet-blockbook.sh 2>/data/eth-testnet/blockbook/blockbook.log -``` diff --git a/bchain/coins/zec/zec.md b/bchain/coins/zec/zec.md deleted file mode 100644 index e7cf690d..00000000 --- a/bchain/coins/zec/zec.md +++ /dev/null @@ -1,57 +0,0 @@ -## Zcash Setup -Get Zcash client -``` -wget https://z.cash/downloads/zcash-1.0.15-linux64.tar.gz -tar xzf zcash-1.0.15-linux64.tar.gz -``` - -Run command to download the parameters used to create and verify shielded transactions: -``` -zcash-1.0.15/bin/zcash-fetch-params -``` - -Data are stored in */data/zec* , in folders */data/zec/zcash* for Zcash client data, */data/zec/blockbook* for Blockbook data. - -Create configuration file */data/zec/zcash/zcash.conf* with content -``` -daemon=1 -server=1 -rpcuser=rpc -rpcpassword=rpc -rpcport=8032 -txindex=1 -mainnet=1 -addnode=mainnet.z.cash -``` - -Create script *run-zec-zcashd.sh* that starts the zcashd daemon with increased rpcworkqueue and configured zeromq -``` -#!/bin/bash - -zcash-1.0.15/bin/zcashd -datadir=/data/zec/zcash -rpcworkqueue=32 -zmqpubhashblock=tcp://127.0.0.1:38332 -zmqpubrawblock=tcp://127.0.0.1:38332 -zmqpubhashtx=tcp://127.0.0.1:38332 -zmqpubrawtx=tcp://127.0.0.1:38332 -``` - -Run the *run-zec-zcashd.sh* to get initial import of data. - -Create blockchain configuration file */data/zec/blockbook/zec.json* -``` -{ - "rpcURL": "http://127.0.0.1:8032", - "rpcUser": "rpc", - "rpcPass": "rpc", - "rpcTimeout": 25, - "parse": true, - "zeroMQBinding": "tcp://127.0.0.1:38332" -} -``` - -Create *run-zec-blockbook.sh* script that starts blockbook -``` -#!/bin/bash -./blockbook -coin=zec -blockchaincfg=/data/zec/blockbook/zec.json -datadir=/data/zec/blockbook/db -sync -internal=:9032 -public=:9132 -certfile=server/testcert -explorer=https://zec-bitcore1.trezor.io $1 -``` - -To run blockbook with logging to file (run with nohup or daemonize using screen) -``` -./run-zec-blockbook.sh 2> /data/zec/blockbook/blockbook.log -``` diff --git a/configs/coins/ethereum.json b/configs/coins/ethereum.json index c51ccec9..fcc63d3d 100644 --- a/configs/coins/ethereum.json +++ b/configs/coins/ethereum.json @@ -8,6 +8,8 @@ "ports": { "backend_rpc": 8036, "backend_message_queue": 0, + "backend_p2p": 38336, + "backend_http": 8136, "blockbook_internal": 9036, "blockbook_public": 9136 }, diff --git a/configs/coins/ethereum_classic.json b/configs/coins/ethereum_classic.json new file mode 100644 index 00000000..cde3d834 --- /dev/null +++ b/configs/coins/ethereum_classic.json @@ -0,0 +1,14 @@ +{ + "coin": { + "name": "Ethereum Classic", + "shortcut": "ETC", + "label": "Ethereum Classic", + "alias": "ethereum_classic" + }, + "ports": { + "backend_rpc": 8037, + "backend_message_queue": 0, + "blockbook_internal": 9037, + "blockbook_public": 9137 + } +} diff --git a/configs/coins/ethereum_testnet_ropsten.json b/configs/coins/ethereum_testnet_ropsten.json index cb241065..a236e16d 100644 --- a/configs/coins/ethereum_testnet_ropsten.json +++ b/configs/coins/ethereum_testnet_ropsten.json @@ -8,6 +8,7 @@ "ports": { "backend_rpc": 18036, "backend_message_queue": 0, + "backend_p2p": 48336, "blockbook_internal": 19036, "blockbook_public": 19136 }, diff --git a/contrib/scripts/generate-port-registry.go b/contrib/scripts/generate-port-registry.go new file mode 100755 index 00000000..20375eea --- /dev/null +++ b/contrib/scripts/generate-port-registry.go @@ -0,0 +1,263 @@ +//usr/bin/go run $0 $@ ; exit +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "io/ioutil" + "math" + "os" + "path/filepath" + "sort" + "strings" +) + +const ( + inputDir = "configs/coins" + outputFile = "docs/ports.md" +) + +type PortInfo struct { + CoinName string + BlockbookInternalPort uint16 + BlockbookPublicPort uint16 + BackendRPCPort uint16 + BackendServicePorts map[string]uint16 +} + +type PortInfoSlice []*PortInfo + +type Config struct { + Coin struct { + Name string `json:"name"` + } + Ports map[string]uint16 `json:"ports"` +} + +func main() { + output := "stdout" + if len(os.Args) > 1 { + if len(os.Args) == 2 && os.Args[1] == "-w" { + output = outputFile + } else { + fmt.Fprintf(os.Stderr, "Usage: %s [-w]\n", filepath.Base(os.Args[0])) + fmt.Fprintf(os.Stderr, " -w write output to %s instead of stdout\n", outputFile) + os.Exit(1) + } + } + + slice, err := loadPortInfo(inputDir) + if err != nil { + panic(err) + } + + sortPortInfo(slice) + + err = writeMarkdown(output, slice) + if err != nil { + panic(err) + } +} + +func loadPortInfo(dir string) (PortInfoSlice, error) { + files, err := ioutil.ReadDir(dir) + if err != nil { + return nil, err + } + + items := make(PortInfoSlice, 0, len(files)) + + for _, fi := range files { + if fi.IsDir() || fi.Name()[0] == '.' { + continue + } + + path := filepath.Join(dir, fi.Name()) + f, err := os.Open(path) + if err != nil { + return nil, fmt.Errorf("%s: %s", path, err) + } + defer f.Close() + + v := Config{} + d := json.NewDecoder(f) + err = d.Decode(&v) + if err != nil { + return nil, fmt.Errorf("%s: json: %s", path, err) + } + + item := &PortInfo{CoinName: v.Coin.Name, BackendServicePorts: map[string]uint16{}} + for k, v := range v.Ports { + if v == 0 { + continue + } + + switch k { + case "blockbook_internal": + item.BlockbookInternalPort = v + case "blockbook_public": + item.BlockbookPublicPort = v + case "backend_rpc": + item.BackendRPCPort = v + default: + if len(k) > 8 && k[:8] == "backend_" { + item.BackendServicePorts[k[8:]] = v + } + } + } + + items = append(items, item) + } + + return items, nil +} + +func sortPortInfo(slice PortInfoSlice) { + // normalizes values in order to sort zero values at the bottom of the slice + normalize := func(a, b uint16) (uint16, uint16) { + if a == 0 { + a = math.MaxUint16 + } + if b == 0 { + b = math.MaxUint16 + } + return a, b + } + + // sort values by BlockbookPublicPort, then by BackendRPCPort and finally by + // CoinName; zero values are sorted at the bottom of the slice + sort.Slice(slice, func(i, j int) bool { + a, b := normalize(slice[i].BlockbookPublicPort, slice[j].BlockbookPublicPort) + + if a < b { + return true + } + if a > b { + return false + } + + a, b = normalize(slice[i].BackendRPCPort, slice[j].BackendRPCPort) + + if a < b { + return true + } + if a > b { + return false + } + + return strings.Compare(slice[i].CoinName, slice[j].CoinName) == -1 + }) +} + +func writeMarkdown(output string, slice PortInfoSlice) error { + var ( + buf bytes.Buffer + err error + ) + + fmt.Fprintf(&buf, "# Registry of ports\n\n") + + header := []string{"coin", "blockbook internal port", "blockbook public port", "backend rpc port", "backend service ports (zmq)"} + writeTable(&buf, header, slice) + + fmt.Fprintf(&buf, "\n> NOTE: This document is generated from coin definitions in `configs/coins`.\n") + + out := os.Stdout + if output != "stdout" { + out, err = os.OpenFile(output, os.O_CREATE|os.O_WRONLY, 0644) + if err != nil { + return err + } + defer out.Close() + } + + n, err := out.Write(buf.Bytes()) + if err != nil { + return err + } + if n < len(buf.Bytes()) { + return io.ErrShortWrite + } + + return nil +} + +func writeTable(w io.Writer, header []string, slice PortInfoSlice) { + rows := make([][]string, len(slice)) + for i, item := range slice { + row := make([]string, len(header)) + row[0] = item.CoinName + if item.BlockbookInternalPort > 0 { + row[1] = fmt.Sprintf("%d", item.BlockbookInternalPort) + } + if item.BlockbookPublicPort > 0 { + row[2] = fmt.Sprintf("%d", item.BlockbookPublicPort) + } + if item.BackendRPCPort > 0 { + row[3] = fmt.Sprintf("%d", item.BackendRPCPort) + } + + svcPorts := make([]string, 0, len(item.BackendServicePorts)) + for k, v := range item.BackendServicePorts { + var s string + if k == "message_queue" { + s = fmt.Sprintf("%d", v) + } else { + s = fmt.Sprintf("%d %s", v, k) + } + svcPorts = append(svcPorts, s) + } + + row[4] = strings.Join(svcPorts, ", ") + + rows[i] = row + } + + padding := make([]int, len(header)) + for column := range header { + padding[column] = len(header[column]) + + for _, row := range rows { + padding[column] = maxInt(padding[column], len(row[column])) + } + } + + content := make([][]string, 0, len(rows)+2) + + content = append(content, paddedRow(header, padding)) + content = append(content, delim("-", padding)) + + for _, row := range rows { + content = append(content, paddedRow(row, padding)) + } + + for _, row := range content { + fmt.Fprintf(w, "|%s|\n", strings.Join(row, "|")) + } +} + +func maxInt(a, b int) int { + if a > b { + return a + } + return b +} + +func paddedRow(row []string, padding []int) []string { + out := make([]string, len(row)) + for i := 0; i < len(row); i++ { + format := fmt.Sprintf(" %%-%ds ", padding[i]) + out[i] = fmt.Sprintf(format, row[i]) + } + return out +} + +func delim(str string, padding []int) []string { + out := make([]string, len(padding)) + for i := 0; i < len(padding); i++ { + out[i] = strings.Repeat(str, padding[i]+2) + } + return out +} diff --git a/docs/build.md b/docs/build.md new file mode 100644 index 00000000..cd7b050b --- /dev/null +++ b/docs/build.md @@ -0,0 +1,263 @@ +# Blockbook Build Guide + +## Setting up your development environment + +Supported environment to develop Blockbook is Linux. Although it is possible build and run Blockbook on macOS +or Windows our build process is not prepared for it. But you can still build Blockbook [manually](#manual-build). + +The only dependency required to build Blockbook is Docker. You can see how to install Docker [here](https://docs.docker.com/install/linux/docker-ce/debian/). +Manual build require additional dependencies that are described in appropriate section. + +## Build in Docker environment + +All build operations run in Docker container in order to keep build environment isolated. Makefile in root of repository +define few targets used for building, testing and packaging of Blockbook. With Docker image definitions and Debian +package templates in *build/docker* and *build/templates* respectively, they are only inputs that make build process. + +Docker build images are created at first execution of Makefile and that information is persisted. (Actually there are +created two files in repository – .bin-image and .deb-image – that are used as tags.) Sometimes it is necessary to +rebuild Docker images, it is possible by executing `make build-images`. + +### Building binary + +Just run `make` and that is it. Output binary is stored in *build* directory. Note that although Blockbook is Go application +it is dynamically linked with RocksDB dependencies and ZeroMQ. Therefore operating system where Blockbook will be +executed still need that dependencies installed. See [Manual build](#manual-build) instructions below or install +Blockbook via Debian packages. + +### Building debug binary + +Standard binary contains no debug symbols. Execute `make build-debug` to get binary for debugging. + +### Testing + +How to execute tests is described in separate document [here](/docs/testing.md). + +### Building Debian packages + +Blockbook and particular coin back-end are usually deployed together. They are defined in same place as well. +So typical way to build Debian packages is build Blockbook and back-end deb packages by single command. But it is not +mandatory, of course. + +> Early releases of Blockbook weren't so friendly for extending. One had to define back-end package, Blockbook package, +> back-end configuration and Blockbook configuration as well. There were many options that were duplicated across +> configuration files and therefore error prone. +> +> Actually all configuration options and also build options for both Blockbook and backend are defined in single JSON +> file and all stuff required during build is generated dynamically. + +Makefile targets follow simple pattern, there are few prefixes that define what to build. + +* *deb-blockbook-<coin>* – Build Blockbook package for given coin. + +* *deb-backend-<coin>* – Build back-end package for given coin. + +* *deb-<coin>* – Build both Blockbook and back-end packages for given coin. + +* *all-<coin>* – Similar to deb-<coin> but clean repository and rebuild Docker image before package build. It is useful + for production deployment. + +* *all* – Build both Blockbook and back-end packages for all coins. + +Which coins are possible to build is defined in *configs/coins*. Particular coin has to have JSON config file there. + +For example we want to build some packages for Bitcoin and Bitcoin Testnet. + +```bash +# make all-bitcoin deb-backend-bitcoin_testnet +... +# ls build/*.deb +build/backend-bitcoin_0.16.1-satoshilabs-1_amd64.deb build/backend-bitcoin-testnet_0.16.1-satoshilabs-1_amd64.deb build/blockbook-bitcoin_0.0.6_amd64.deb +``` + +We have built two backend packages – for Bitcoin and Testnet – and Blockbook package for Bitcoin. Before build have been +performed there was cleaned build directory and rebuilt Docker image. + +### Extra variables + +There are few variables that can be passed to make in order to modify build process. + +In general, build of Blockbook binary require some dependencies. They are downloaded automatically during build process +but if you need to build binary repeatedly it consumes a lot of time. Here comes variable *UPDATE_VENDOR* that if is +unset says that build process uses *vendor* (i.e. dependencies) from your local repository. For example: +`make deb-bitcoin UPDATE_VENDOR=0`. But before the command is executed there must be *vendor* directory populated, +you can do it by calling `dep ensure --vendor-only`. See [Manual build](#manual-build) instructions below. + +All build targets allow pass additional parameters to underlying command inside container. It is possible via ARGS +variable. For example if you want run only subset of unit-tests, you will perform it by calling: +`make test ARGS='-run TestBitcoinRPC' UPDATE_VENDOR=0` + +Common behaviour of Docker image build is that build steps are cached and next time they are executed much faster. +Although this is a good idea, when something went wrong you will need to override this behaviour somehow. Execute this +command: `make build-images NO_CACHE=true`. + +### On naming conventions and versioning + +**install and data directories** + +Both Blockbook and back-end have separated install and data directories. They use common preffix and are defined in +*configs/environ.json* and all templates use them. + +Back-end install directory is */opt/coins/nodes/<coin>*. +Back-end data directory is */opt/coins/data/<coin>/backend*. +Blockbook install directory is */opt/coins/blockbook/<coin>*. +Blockbook data directory is */opt/coins/data/<coin>/blockbook*. + +*coin* used above is defined in *coin.alias* in coin definition file. + +**package names** + +Package names are defined in *backend.package_name* and *blockbook.package_name* in coin definition file. We use +simple pattern *<prefix>-<coin>* to name packages where *prefix* is either *blockbook* or *backend* and +*coin* is made similarly to *coin.alias*. We use convention that coin name uses lowercase characters and dash '-' as +a word delimiter. Testnet versions of coins must have *-testnet* suffix. That differs from *coin.alias* because +underscore has a special meaning in Debian packaging. For example there are packages *backend-bitcoin* and +*blockbook-bitcoin-testnet*. + +**user names** + +User names are defined in *backend.system_user* and *blockbook.system_user* in coin definition file. We follow common +Linux conventions, user names use lowercase characters and dash '-' as a word delimiter. + +Back-end user name use coin name only, including testnet services. For example there is *bitcoin* user for both +*backend-bitcoin* and *backend-bitcoin-testnet* packages. + +Blockbook user name has *blockbook-* prefix and coin name (made same as back-end version). For example there is +*blockbook-bitcoin* user for both *blockbook-bitcoin* and *blockbook-bitcoin-testnet* packages. + +**back-end versioning** + +Since we have to distinguish version of coin distribution and version of our configuration we follow standard Debian +package versioning rules (for details see +[Debian policy](https://www.debian.org/doc/debian-policy/ch-controlfields.html#version)). There is upstream version +and revision both defined in coin definition file in *backend.version* and *backend.package_revision*, respectively. + +**blockbook versioning** + +Blockbook versioning is much simpler. There is only one version defined in *configs/environ.json*. + +### On back-end building + +Because we don't keep back-end archives inside out repository we download them during build process. Build steps +are these: download, verify and extract archive, prepare distribution and make package. + +All configuration keys described below are in coin definition file in *configs/coins*. + +**download archive** + +URL from where is archive downloaded is defined in *backend.binary_url*. + +**verify archive** + +There are three different approaches how is archive verification done. Some projects use PGP sign of archive, some +have signed sha256 sums and some don't care about verification at all. So there is option *backend.verification_type* that +could be *gpg*, *gpg-sha256* or *sha256* and chooses particular method. + +*gpg* type require file with digital sign and maintainer's public key imported in Docker build image (see below). Sign +file is downloaded from URL defined in *backend.verification_source*. Than is passed to gpg in order to verify archvie. + +*gpg-sha256* type require signed checksum file and maintainer's public key imported in Docker build image (see below). +Checksum file is downloaded from URL defined in *backend.verification_source*. Then is verified by gpg and passed to +sha256sum in order to verify archive. + +*sha256* type is used for coins that don't support verification at all. In *backend.verification_source* is defined +hexadecimal string that is compared with output of sha256sum. Although this solution is not secure, it avoid download +errors and other surprises at least. + +*gpg* and *gpg-sha256* types require maintainer's public key imported in Docker build image. It is not expected that +maintainer's key will change requently while sing or checksum files are changed every release, so it is ideal to +store maintainer's key within image definition. Public keys are stored in *build/docker/deb/gpg-keys* directory. Docker +image must be rebuilt by calling `make build-images`. + +**extract archive** + +Extraction command is defined in *backend.extract_command*. Content of archive must be extracted to `./backend` directory. +See bitcoin.json and vertcoin.json for different approaches. + +**prepare distribution** + +There are two steps in this stage – exclude unnecessary files and generate configuration. + +Some files are not required for server deployment, some binaries have unnecessary dependencies, so it is good idea to +extract these files from output package. Files to extract are listed in *backend.exclude_files*. Note that paths are +relative to *backend* directory where archive is extracted. + +Configuration is described in [config.md](/docs/config.md). + +## Manual build + +Instructions below are focused on Debian 9 (Stretch). If you want to use another Linux distribution or operating system +like macOS or Windows, please read instructions specific for each project. + +Setup go environment: + +``` +wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz && tar xf go1.10.3.linux-amd64.tar.gz +sudo mv go /opt/go +sudo ln -s /opt/go/bin/go /usr/bin/go +# see `go help gopath` for details +mkdir $HOME/go +export GOPATH=$HOME/go +export PATH=$PATH:$GOPATH/bin +``` + +Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md +and compile the static_lib and tools + +``` +sudo apt-get update && sudo apt-get install -y \ + build-essential git wget pkg-config libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev +git clone https://github.com/facebook/rocksdb.git +cd rocksdb +CFLAGS=-fPIC CXXFLAGS=-fPIC make release +``` + +Setup variables for gorocksdb: https://github.com/tecbot/gorocksdb + +``` +export CGO_CFLAGS="-I/path/to/rocksdb/include" +export CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4" +``` + +Install ZeroMQ: https://github.com/zeromq/libzmq + +Install go-dep tool: +``` +go get github.com/golang/dep/cmd/dep +``` + +Get blockbook sources, install dependencies, build: + +``` +cd $GOPATH/src +git clone https://github.com/trezor/blockbook.git +cd blockbook +dep ensure -vendor-only +go build +``` + +### Example command + +Blockbook require full node daemon as its back-end. You are responsible for proper installation. Port numbers and +daemon configuration are defined in *configs/coins* and *build/templates/backend/config* directories. You should use +specific installation process for particular coin you want run (e.g. https://bitcoin.org/en/full-node#other-linux-distributions for Bitcoin). + +When you have running back-end daemon you can start Blockbook. It is highly recomended use ports described in [ports.md](/docs/ports.md) +for both Blockbook and back-end daemon. You can use *contrib/scripts/build-blockchaincfg.sh* that will generate +Blockbook's blockchain configuration from our coin definition files. + +Example for Bitcoin: +``` +contrib/scripts/build-blockchaincfg.sh +./blockbook -sync -blockchaincfg=build/blockchaincfg.json -internal=:9030 -public=:9130 -certfile=server/testcert -logtostderr +``` + +This command starts Blockbook with parallel synchronization and providing HTTP and Socket.IO interface, with database +in local directory *data* and established ZeroMQ and RPC connections to back-end daemon specified in configuration +file passed to *-blockchaincfg* option. + +Blockbook logs to stderr (option *-logtostderr*) or to directory specified by parameter *-log_dir* . Verbosity of logs can be tuned +by command line parameters *-v* and *-vmodule*, for details see https://godoc.org/github.com/golang/glog. + +You can check that Blockbook is running by simple HTTP request: `curl https://localhost:9130`. Returned data is JSON with some +run-time information. If port is closed, Blockbook is syncing data. diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 00000000..2d0a9e67 --- /dev/null +++ b/docs/config.md @@ -0,0 +1,97 @@ +# Configuration + +Coin definitions are stored in JSON files in *configs/coins* directory. They are single source of Blockbook +configuration, Blockbook and back-end package definition and build metadata. Since Blockbook supports only single +coin index per running instance, every coin (including testnet) must have single definition file. + +Because most of coins are fork of Bitcoin and they have similar way to install and configure their daemon, we use +templates to generate package definition and configuration files during build process. It is similar to build Blockbook +package too. Templates are filled with data from coin definition. Although build process generate packages +automatically, there is sometimes necessary see intermediate step. You can generate all files by calling +`go run build/templates/generate.go coin` where *coin* is name of definition file without .json extension. Files are +generated to *build/pkg-defs* directory. + +Good examples of coin configuration are +[*configs/coins/bitcoin.json*](configs/coins/bitcoin.json) and +[*configs/coins/ethereum.json*](configs/coins/ethereum.json) for Bitcoin-like coins and different coins, respectively. + +## Description of coin definition + +* `coin` – Base information about coin. + * `name` – Name of coin used internally (e.g. "Bcash Testnet"). + * `shortcut` – Ticker symbol (code) of coin (e.g. "TBCH"). + * `label` – Name of coin used publicly (e.g. "Bitcoin Cash Testnet"). + * `alias` – Name of coin used in file system paths and config files. We use convention that name uses lowercase + characters and underscore '_' as a word delimiter. Testnet versions of coins must have *_testnet* + suffix. For example "bcash_testnet". + +* `ports` – List of ports used by both back-end and Blockbook. Ports defined here are used in configuration templates + and also as source for generated documentation. + * `backend_rpc` – Port of back-end RPC that is connected by Blockbook service. + * `backend_message_queue` – Port of back-end MQ (if used) that is connected by Blockbook service. + * `backend_*` – Additional back-end ports can be documented here. Actually the only purpose is to get them to + port table (prefix is removed and rest of string is used as note). + * `blockbook_internal` – Blockbook's internal port that is used for metric collecting, debugging etc. + * `blockbook_public` – Blockbook's public port that is used to comunicate with Trezor wallet (via Socket.IO). + +* `ipc` – Defines how Blockbook connects its back-end service. + * `rpc_url_template` – Template that defines URL of back-end RPC service. See note on templates below. + * `rpc_user` – User name of back-end RPC service, used by both Blockbook and back-end configuration templates. + * `rpc_pass` – Password of back-end RPC service, used by both Blockbook and back-end configuration templates. + * `rpc_timeout` – RPC timeout used by Blockbook. + * `message_queue_binding_template` – Template that defines URL of back-end's message queue (ZMQ), used by both + Blockbook and back-end configuration template. See note on templates below. + +* `backend` – Definition of back-end package, configuration and service. + * `package_name` – Name of package. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning). + * `package_revision` – Revision of package. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning). + * `system_user` – User used to run back-end service. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning). + * `version` – Upstream version. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning). + * `binary_url` – URL of back-end archive. + * `verification_type` – Type of back-end archive verification. Possible values are *gpg*, *gpg-sha256*, *sha256*. + * `verification_source` – Source of sign/checksum of back-end archive. + * `extract_command` – Command to extract back-end archive. It is required to extract content of archive to + *backend* directory. + * `exclude_files` – List of files from back-end archive to exclude. Some files are not required for server + deployment, some binaries have unnecessary dependencies, so it is good idea to extract these files from output + package. Note that paths are relative to *backend* directory where archive is extracted. + * `exec_command_template` – Template of command to execute back-end node daemon. Every back-end node daemon has its + service that is managed by systemd. Template is evaluated to *ExecStart* option in *Service* section of + service unit. See note on templates below. + * `logrotate_files_template` – Template that define log files rotated by logrotate daemon. See note on templates + below. + * `postinst_script_template` – Additional steps in postinst script. See [ZCash definition](configs/coins/zcash.json) + for more information. + * `service_type` – Type of service. Services that daemonize must have *forking* type and write their PID to + *PIDFile*. Services that don't support daemonization must have *simple* type. See examples above. + * `service_additional_params_template` – Additional parameters in service unit. See + [ZCash definition](configs/coins/zcash.json) for more information. + * `protect_memory` – Enables *MemoryDenyWriteExecute* option in service unit if *true*. + * `mainnet` – Set *false* for testnet back-end. + * `config_file` – Name of template of back-end configuration file. Templates are defined in *build/backend/config*. + For Bitcoin-like coins it is not necessary to add extra template, most options can be added via + *additional_params*. For coins that don't require configuration option should be empty (e.g. Ethereum). + * `additional_params` – Object of extra parameters that are added to back-end configuration file as key=value pairs. + Exception is *addnode* key that contains list of nodes that is expanded as addnode=item lines. + +* `blockbook` – Definition of Blockbook package, configuration and service. + * `package_name` – Name of package. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning). + * `system_user` – User used to run Blockbook service. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning). + * `internal_binding_template` – Template for *-internal* parameter. See note on templates below. + * `public_binding_template` – Template for *-public* parameter. See note on templates below. + * `explorer_url` – URL of blockchain explorer. + * `additional_params` – Additional params of exec command (see [Dogecoin definition](configs/coins/dogecoin.json)). + * `block_chain` – Configuration of BlockChain type that ensures communication with back-end service. All options + must be tweaked for each individual coin separely. + * `parse` – Use binary parser for block decoding if *true* else call verbose back-end RPC method that returns + JSON. Note that verbose method is slow and not every coin support it. However there are coin implementations + that don't support binary parsing (e.g. ZCash). + * `mempool_workers` – Number of workers for UTXO mempool. + * `mempool_sub_workers` – Number of subworkers for UTXO mempool. + * `block_addresses_to_keep` – Number of blocks that are to be kept in blockaddresses column. + * `additional_params` – Object of coin-specific params. + +* `meta` – Common package metadata. + * `package_maintainer` – Full name of package maintainer. + * `package_maintainer_email` – E-mail of package maintainer. + diff --git a/docs/ports.md b/docs/ports.md new file mode 100644 index 00000000..3ab71f08 --- /dev/null +++ b/docs/ports.md @@ -0,0 +1,24 @@ +# Registry of ports + +| coin | blockbook internal port | blockbook public port | backend rpc port | backend service ports (zmq) | +|--------------------------|-------------------------|-----------------------|------------------|-----------------------------| +| Bitcoin | 9030 | 9130 | 8030 | 38330 | +| Bcash | 9031 | 9131 | 8031 | 38331 | +| Zcash | 9032 | 9132 | 8032 | 38332 | +| Dash | 9033 | 9133 | 8033 | 38333 | +| Litecoin | 9034 | 9134 | 8034 | 38334 | +| Bgold | 9035 | 9135 | 8035 | 38335 | +| Ethereum | 9036 | 9136 | 8036 | 38336 p2p, 8136 http | +| Ethereum Classic | 9037 | 9137 | 8037 | | +| Dogecoin | 9038 | 9138 | 8038 | 38338 | +| Namecoin | 9039 | 9139 | 8039 | 38339 | +| Vertcoin | 9040 | 9140 | 8040 | 38340 | +| Testnet | 19030 | 19130 | 18030 | 48330 | +| Bcash Testnet | 19031 | 19131 | 18031 | 48331 | +| Zcash Testnet | 19032 | 19132 | 18032 | 48332 | +| Dash Testnet | 19033 | 19133 | 18033 | 48333 | +| Litecoin Testnet | 19034 | 19134 | 18034 | 48334 | +| Ethereum Testnet Ropsten | 19036 | 19136 | 18036 | 48336 p2p | +| Vertcoin Testnet | 19040 | 19140 | 18040 | 48340 | + +> NOTE: This document is generated from coin definitions in `configs/coins`. diff --git a/docs/rocksdb.md b/docs/rocksdb.md new file mode 100644 index 00000000..6d98694e --- /dev/null +++ b/docs/rocksdb.md @@ -0,0 +1,44 @@ +# Data storage in RocksDB + +Blockbook stores data the key-value store RocksDB. Data are stored in binary form to save space. +The data are separated to different column families: + +- **default** + + at the moment not used, will store statistical data etc. + +- **height** - maps *block height* to *block hash* + + *Block heigh* stored as array of 4 bytes (big endian uint32) + *Block hash* stored as array of 32 bytes + + Example - the first four blocks (all data hex encoded) +``` +0x00000000 : 0x000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943 +0x00000001 : 0x00000000b873e79784647a6c82962c70d228557d24a747ea4d1b8bbe878e1206 +0x00000002 : 0x000000006c02c8ea6e4ff69651f7fcde348fb9d557a06e6957b65552002a7820 +0x00000003 : 0x000000008b896e272758da5297bcd98fdc6d97c9b765ecec401e286dc1fdbe10 +``` + +- **outputs** - maps *output script+block height* to *array of outpoints* + + *Output script (ScriptPubKey)+block height* stored as variable length array of bytes for output script + 4 bytes (big endian uint32) block height + *array of outpoints* stored as array of 32 bytes for transaction id + variable length outpoint index for each outpoint + + Example - (all data hex encoded) +``` +0x001400efeb484a24a1c1240eafacef8566e734da429c000e2df6 : 0x1697966cbd76c75eb9fc736dfa3ba0bc045999bab1e8b10082bc0ba546b0178302 +0xa9143e3d6abe282d92a28cb791697ba001d733cefdc7870012c4b1 : 0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d301 +``` + +- **inputs** - maps *transaction outpoint* to *input transaction* that spends it + + *Transaction outpoint* stored as array of 32 bytes for transaction id + variable length outpoint index + *Input transaction* stored as array of 32 bytes for transaction id + variable length input index + + Example - (all data hex encoded) +``` +0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d300 : 0x0a7aa90ea0269c79f844c516805e4cac594adb8830e56fca894b66aab19136a428 +0x7246e79f97b5f82e7f51e291d533964028ec90be0634af8a8ef7d5a903c7f6d301 : 0x4303a9fcfe6026b4d33ba488df6443c9a99bca7b7fcb7c6f6cd65cea24a749b700 +``` +