Update documentation to match version 0.2.0
Martin Boehm
parent
499d65460f
commit
e27821f231
|
|
@ -1,15 +1,15 @@
|
|||
# Blockbook Contributor Guide
|
||||
|
||||
Blockbook is back-end service for Trezor wallet. Although it is open source, design and development of core packages
|
||||
Blockbook is back-end service for Trezor wallet. Although it is open source, the design and development of the core packages
|
||||
is done by Trezor developers in order to keep Blockbook compatible with Trezor.
|
||||
|
||||
Bug fixes and support for new coins are welcome. Please take note that non-fixing pull requests that change base
|
||||
packages or another coin code will not be accepted. If you will have to change some of existing code, please file
|
||||
packages or another coin code will not be accepted. If you have a need to change some of the existing code, please file
|
||||
an issue and discuss your request with Blockbook maintainers.
|
||||
|
||||
## Development environment
|
||||
|
||||
Instructions to set up your development environment and build Blockbook are described in separated
|
||||
Instructions to set up your development environment and build Blockbook are described in a separate
|
||||
[document](/docs/build.md).
|
||||
|
||||
## How can I contribute?
|
||||
|
|
@ -25,8 +25,7 @@ updates. Do not leave random "+1" or "I have this too" comments, as they only cl
|
|||
resolving it. However, if you have ways to reproduce the issue or have additional information that may help resolving
|
||||
the issue, please leave a comment.
|
||||
|
||||
Include information about Blockbook instance which is exposed at internal HTTP port. Ports are listed in
|
||||
[port registry](/docs/ports.md). For example execute `curl -k https://localhost:9030` for Bitcoin.
|
||||
Include information about the Blockbook instance, which is shown at the Blockbook status page or returned by API call. For example execute `curl -k https://<server name>:<public port>/api` to get JSON containing details about Blockbook and Backend installation. Ports are listed in the [port registry](/docs/ports.md).
|
||||
|
||||
Also include the steps required to reproduce the problem if possible and applicable. This information will help us
|
||||
review and fix your issue faster. When sending lengthy log-files, consider posting them as a gist
|
||||
|
|
@ -34,35 +33,35 @@ review and fix your issue faster. When sending lengthy log-files, consider posti
|
|||
|
||||
### Adding coin support
|
||||
|
||||
> **Important notice**: Although we are happy for support of new coins, we do not have enough capacity to run them all
|
||||
> on our infrastructure. We run Blockbook instances only for selected number of coins. If you want to have Blockbook
|
||||
> instance for your coin, you will have to deploy it to your own server.
|
||||
|
||||
Trezor harware wallet supports over 500 coins, see https://trezor.io/coins/. You are free to add support for any of
|
||||
them to Blockbook. Actually implemented coins are listed [here](/docs/ports.md).
|
||||
them to Blockbook. Currently implemented coins are listed [here](/docs/ports.md).
|
||||
|
||||
You should follow few steps bellow to get smooth merge of your PR.
|
||||
|
||||
> Although we are happy for support of new coins we have not enough capacity to run them all on our infrastructure.
|
||||
> Actually we can run Blockbook instances only for coins supported by Trezor wallet. If you want to have Blockbook
|
||||
> instance for your coin, you will have to deploy your own server.
|
||||
You should follow the steps below to get smooth merge of your PR.
|
||||
|
||||
#### Add coin definition
|
||||
|
||||
Coin definitions are stored in JSON files in *configs/coins* directory. They are single source of Blockbook
|
||||
Coin definitions are stored in JSON files in *configs/coins* directory. They are the single source of Blockbook
|
||||
configuration, Blockbook and back-end package definition and build metadata. Since Blockbook supports only single
|
||||
coin index per running instance, every coin (including testnet) must have single definition file.
|
||||
|
||||
All options of coin definition are described in [config.md](/docs/config.md).
|
||||
|
||||
Because most of coins are fork of Bitcoin and they have similar way to install and configure their daemon, we use
|
||||
templates to generate package definition and configuration files during build process. Similarly, there templates for Blockbook
|
||||
templates to generate package definition and configuration files during build process. Similarly, there are templates for Blockbook
|
||||
package. Templates are filled with data from coin definition. Although normally all package definitions are generated automatically
|
||||
during the build process, sometimes there is a reason to see them. You can create them by calling
|
||||
during the build process, sometimes there is a reason to check what was generated. You can create them by calling
|
||||
`go run build/templates/generate.go coin`, where *coin* is name of definition file without .json extension. Files are
|
||||
generated to *build/pkg-defs* directory.
|
||||
|
||||
Good examples of coin configuration are
|
||||
[*configs/coins/bitcoin.json*](configs/coins/bitcoin.json) and
|
||||
[*configs/coins/ethereum.json*](configs/coins/ethereum.json) for Bitcoin-like coins and different coins, respectively.
|
||||
[*configs/coins/ethereum.json*](configs/coins/ethereum.json) for Bitcoin type coins and Ethereum type coins, respectively.
|
||||
|
||||
Usually you have to update only few options that differ from Bitcoin definition. At first there is base information
|
||||
Usually you have to update only a few options that differ from the Bitcoin definition. At first there is base information
|
||||
about coin in section *coin* – name, alias etc. Then update port information in *port* section. We keep port series as
|
||||
listed in [the port registry](/docs/ports.md). Select next port numbers in the series. Port numbers must be unique across all
|
||||
port definitions.
|
||||
|
|
@ -109,15 +108,15 @@ different concept than Bitcoin.
|
|||
|
||||
##### BlockChain interface
|
||||
|
||||
Type that implements *bchain.BlockChain* interface ensures communication with block chain network. Because
|
||||
it calls node RPCs it usually has suffix RPC.
|
||||
Type that implements *bchain.BlockChain* interface ensures communication with the block chain network. Because
|
||||
it calls node RPCs, it usually has suffix RPC.
|
||||
|
||||
Initialization of object is separated into two stages. At first there is called factory method (details described
|
||||
in next section) and then *bchain.BlockChain.Initialize()* method. Separated initialization method allows you call
|
||||
in the next section) and then *bchain.BlockChain.Initialize()* method. Separated initialization method allows you call
|
||||
inherited methods during initialization. However it is common practice override fields of embedded structure in factory
|
||||
method.
|
||||
|
||||
During initialization, there is usually loaded chain information, registered message queue callback and created mempool
|
||||
Initialization routine usually loads chain information, registers message queue callback and creates mempool
|
||||
and parser objects.
|
||||
|
||||
BitcoinRPC uses *btc.RPCMarshaller* ([btc/codec.go](/bchain/coins/btc/codec.go)) in order to distinguish API version of
|
||||
|
|
@ -136,7 +135,7 @@ must correspond to *coin.name* in coin definition file (see above).
|
|||
Configuration passed to factory method is coin specific. For types that embed *btc.BitcoinRPC,* configuration must
|
||||
contain at least fields referred in *btc.Configuration* ([btc/bitcoinrpc.go](/bchain/coins/btc/bitcoinrpc.go)).
|
||||
|
||||
For types that embed base struct it is common practise call factory method of embedded type in order to
|
||||
For types that embed base struct it is common practice to call factory method of the embedded type in order to
|
||||
create & initialize it. It is much more robust than simple struct composition.
|
||||
|
||||
For example see [zec/zcashrpc.go](/bchain/coins/zec/zcashrpc.go).
|
||||
|
|
@ -146,7 +145,7 @@ For example see [zec/zcashrpc.go](/bchain/coins/zec/zcashrpc.go).
|
|||
Type that implements *bchain.BlockChainParser* interface ensures parsing and conversions of block chain data. It is
|
||||
initialized by *bchain.BlockChain* during initialization.
|
||||
|
||||
There are several groups of methods defined in *bchian.BlockChainParser*:
|
||||
There are several groups of methods defined in *bchain.BlockChainParser*:
|
||||
|
||||
* *GetAddrDescFromVout* and *GetAddrDescFromAddress* – Convert transaction addresses to *Address Descriptor* that is used as database ID.
|
||||
Most of coins use output script as *Address Descriptor*.
|
||||
|
|
@ -168,9 +167,9 @@ different approach for address representation than Bitcoin.
|
|||
|
||||
#### Add tests
|
||||
|
||||
Add unit tests and integration tests. PR without passing tests won't be accepted.
|
||||
Tests are described [here](/docs/testing.md).
|
||||
Add unit tests and integration tests. **Pull requests without passing tests will not be accepted**.
|
||||
How to implement tests is described [here](/docs/testing.md).
|
||||
|
||||
#### Deploy public server
|
||||
|
||||
Deploy Blockbook server on public IP address. Blockbook maintainers will check implementation before merging.
|
||||
Deploy Blockbook server on public IP address. Blockbook maintainers will check your implementation before merging.
|
||||
|
|
|
|||
23
README.md
23
README.md
|
|
@ -6,19 +6,19 @@
|
|||
|
||||
**Blockbook** is back-end service for Trezor wallet. Main features of **Blockbook** are:
|
||||
|
||||
- create missing indexes in the blockchain - index of addresses and address balances
|
||||
- allow fast searches in the index of addresses
|
||||
- implement parts Insight socket.io interface as required by Trezor wallet
|
||||
- support of multiple coins
|
||||
- simple blockchain explorer for implemented coins
|
||||
- index of addresses and address balances of the connected block chain
|
||||
- fast searches in the indexes
|
||||
- simple blockchain explorer
|
||||
- websocket, API and legacy Bitcore Insight compatible socket.io interfaces
|
||||
- support of multiple coins (Bitcoin and Ethereum type), with easy extensibility for other coins
|
||||
- scripts for easy creation of debian packages for backend and blockbook
|
||||
|
||||
## Build and installation instructions
|
||||
|
||||
Officially supported platform is **Debian Linux** and **AMD64** architecture.
|
||||
|
||||
Memory and disk requirements for initial synchronization of **Bitcoin mainnet** are around 32 GB RAM and over 150 GB of disk size. After initial synchronization, fully synchronized instance takes around 10 GB RAM.
|
||||
Other coins should have lower requirements depending on size of their block chain. Note that fast SSD disks are highly
|
||||
Memory and disk requirements for initial synchronization of **Bitcoin mainnet** are around 32 GB RAM and over 160 GB of disk space. After initial synchronization, fully synchronized instance uses about 10 GB RAM.
|
||||
Other coins should have lower requirements, depending on the size of their block chain. Note that fast SSD disks are highly
|
||||
recommended.
|
||||
|
||||
User installation guide is [here](https://wiki.trezor.io/User_manual:Running_a_local_instance_of_Trezor_Wallet_backend_(Blockbook)).
|
||||
|
|
@ -29,14 +29,11 @@ Contribution guide is [here](CONTRIBUTING.md).
|
|||
|
||||
# Implemented coins
|
||||
|
||||
The most significant coins implemented by Blockbook are:
|
||||
- Bitcoin, Bcash, Bgold, ZCash, Dash, Litecoin
|
||||
|
||||
Incomplete, experimental support is for:
|
||||
- Ethereum, Ethereum Classic
|
||||
Blockbook currently supports over 20 coins, among them:
|
||||
- Bitcoin, Litecoin, Bitcoin Cash, Bgold, ZCash, Dash, Ethereum, Ethereum Classic
|
||||
|
||||
Testnets for some coins are also supported, for example:
|
||||
- Bitcoin Testnet, Bcash Testnet, ZCash Testnet, Ethereum Testnet Ropsten
|
||||
- Bitcoin Testnet, Bitcoin Cash Testnet, ZCash Testnet, Ethereum Testnet Ropsten
|
||||
|
||||
List of all implemented coins is in [the registry of ports](/docs/ports.md).
|
||||
|
||||
|
|
|
|||
|
|
@ -75,8 +75,8 @@ Good examples of coin configuration are
|
|||
Exception is *addnode* key that contains list of nodes that is expanded as addnode=item lines.
|
||||
|
||||
* `blockbook` – Definition of Blockbook package, configuration and service.
|
||||
* `package_name` – Name of package. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning).
|
||||
* `system_user` – User used to run Blockbook service. See convention note in [build guide](/docs/build.md#on-naming-conventions-and-versioning).
|
||||
* `package_name` – Name of package. See convention note in the [build guide](/docs/build.md#on-naming-conventions-and-versioning).
|
||||
* `system_user` – User used to run Blockbook service. See convention note in the [build guide](/docs/build.md#on-naming-conventions-and-versioning).
|
||||
* `internal_binding_template` – Template for *-internal* parameter. See note on templates below.
|
||||
* `public_binding_template` – Template for *-public* parameter. See note on templates below.
|
||||
* `explorer_url` – URL of blockchain explorer. Leave empty for internal explorer.
|
||||
|
|
@ -108,8 +108,8 @@ as well. Note that dot at the beginning is mandatory. Go template syntax is full
|
|||
Since Blockbook is an open-source project and we don't prevent anybody from running independent instances, it is possible
|
||||
to alter built-in text that is specific for Trezor. Text fields that could be updated are:
|
||||
|
||||
* about – A note about instance shown on the Application status page and returned by an API.
|
||||
* tos – A link to Terms of service shown as the footer on the Explorer pages.
|
||||
* [about](/build/text/about) – A note about instance shown on the Application status page and returned by an API.
|
||||
* [tos_link](/build/text/tos_link) – A link to Terms of service shown as the footer on the Explorer pages.
|
||||
|
||||
Text data are stored as plain text files in *build/text* directory and are embedded to binary during build. A change of
|
||||
theese files is mean for a private purpose and PRs that would update them won't be accepted.
|
||||
|
|
|
|||
|
|
@ -1,74 +1,115 @@
|
|||
# Data storage in RocksDB
|
||||
|
||||
**Blockbook** stores data the key-value store RocksDB. Each index is stored in its own column family.
|
||||
**Blockbook** stores data the key-value store [RocksDB](https://github.com/facebook/rocksdb/wiki). As there are multiple indexes, Blockbook uses RocksDB **column families** feature to store indexes separately.
|
||||
|
||||
>Database content is described in golang pseudo types in the form *(name type)*.
|
||||
>The database structure is described in golang pseudo types in the form *(name type)*.
|
||||
>
|
||||
>Operators used in the description:
|
||||
>- -> mapping from key to value.
|
||||
>- \+ concatenation,
|
||||
>- [] array
|
||||
>- *->* mapping from key to value.
|
||||
>- *\+* concatenation,
|
||||
>- *[]* array
|
||||
>
|
||||
>Types used in the description:
|
||||
>- []byte - array of bytes
|
||||
>- uint32 - unsigned integer, stored as array of 4 bytes in big endian*
|
||||
>- vint, vuint - variable length signed/unsigned int
|
||||
>- addrDesc - address descriptor, abstraction of an address. In all bitcoin like coins it is output script. Stored as variable length array of bytes.
|
||||
>- bigInt - unsigned big integer, stored as length of array (1 byte) followed by array of bytes of big int, i.e. *(int_len byte)+(int_value []byte)*. Zero is stored as one byte containing 0.
|
||||
>- *[]byte* - array of bytes
|
||||
>- *uint32* - unsigned integer, stored as array of 4 bytes in big endian*
|
||||
>- *vint*, *vuint* - variable length signed/unsigned int
|
||||
>- *addrDesc* - address descriptor, abstraction of an address.
|
||||
For Bitcoin type coins it is the transaction output script, stored as variable length array of bytes.
|
||||
For Ethereum type coins it is fixed size array of 20 bytes.
|
||||
>- *bigInt* - unsigned big integer, stored as length of the array (1 byte) followed by array of bytes of big int, i.e. *(int_len byte)+(int_value []byte)*. Zero is stored as one byte containing 0.
|
||||
|
||||
**Column families:**
|
||||
**Database structure:**
|
||||
|
||||
The database structure described here is of Blockbook version **0.2.0** (data format version 4).
|
||||
|
||||
The database structure for **Bitcoin type** and **Ethereum type** coins is slightly different. Column families used for both types:
|
||||
- default, height, addresses, transactions, blockTxs
|
||||
|
||||
Column families used only by **Bitcoin type** coins:
|
||||
- addressBalance, txAddresses
|
||||
|
||||
Column families used only by **Ethereum type** coins:
|
||||
- addressContracts
|
||||
|
||||
**Column families description:**
|
||||
|
||||
- **default**
|
||||
|
||||
stores internal state in json format, under the key *internalState*.
|
||||
Stores internal state in json format, under the key *internalState*.
|
||||
|
||||
Most important internal state values are:
|
||||
- coin - which coin is indexed in DB
|
||||
- data format version - currently 3
|
||||
- data format version - currently 4
|
||||
- dbState - closed, open, inconsistent
|
||||
|
||||
Blockbook is on startup checking these values and does not allow to run against wrong coin, data format version and in inconsistent state.
|
||||
Blockbook is on startup checking these values and does not allow to run against wrong coin, data format version and in inconsistent state. The database must be recreated if the internal state does not match.
|
||||
|
||||
- **height**
|
||||
|
||||
maps *block height* to *block hash* and additional data about block
|
||||
Maps *block height* to *block hash* and additional data about block.
|
||||
```
|
||||
(height uint32) -> (hash [32]byte)+(time uint32)+(nr_txs vuint)+(size vuint)
|
||||
```
|
||||
|
||||
- **addresses**
|
||||
|
||||
maps *addrDesc+block height* to *array of outpoints* (array of transactions with input/output index). Input/output is recognized by the sign of the number, output is positive, input is negative, with operation bitwise complement ^ performed on the number.
|
||||
Maps *addrDesc+block height* to *array of transactions with array of input/output indexes*.
|
||||
|
||||
The *block height* in the key is stored as bitwise complement ^ of the height to sort the keys in the order from newest to oldest.
|
||||
|
||||
As there can be multiple inputs/outputs for the same address in one transaction, each txid is followed by variable length array of input/output indexes.
|
||||
The index values in the array are multiplied by two, the last element of the array has the lowest bit set to 1.
|
||||
Input or output is distinguished by the sign of the index, output is positive, input is negative (by operation bitwise complement ^ performed on the number).
|
||||
```
|
||||
(addrDesc []byte)+(height uint32) -> []((txid [32]byte)+(index vint))
|
||||
(addrDesc []byte)+(^height uint32) -> []((txid [32]byte)+[](index vint))
|
||||
```
|
||||
|
||||
- **addressBalance**
|
||||
- **addressBalance** (used only by Bitcoin type coins)
|
||||
|
||||
maps *addrDesc* to *number of transactions*, *sent amount* and *total balance* of given address
|
||||
Maps *addrDesc* to *number of transactions*, *sent amount* and *total balance* of given address.
|
||||
```
|
||||
(addrDesc []byte) -> (nr_txs vuint)+(sent_amount bigInt)+(balance bigInt)
|
||||
```
|
||||
|
||||
- **txAddresses**
|
||||
- **txAddresses** (used only by Bitcoin type coins)
|
||||
|
||||
maps *txid* to *block height* and array of *input addrDesc* with *amounts* and array of *output addrDesc* with *amounts*, with flag if output is spent. In case of spent output, *addrDesc_len* is negative (negative sign is achieved by bitwise complement ^).
|
||||
Maps *txid* to *block height* and array of *input addrDesc* with *amounts* and array of *output addrDesc* with *amounts*, with flag if output is spent. In case of spent output, *addrDesc_len* is negative (negative sign is achieved by bitwise complement ^).
|
||||
```
|
||||
(txid []byte) -> (height vuint)+
|
||||
(nr_inputs vuint)+[]((addrDesc_len vuint)+(addrDesc []byte)+(amount bigInt))+
|
||||
(nr_outputs vuint)+[]((addrDesc_len vint)+(addrDesc []byte)+(amount bigInt))
|
||||
```
|
||||
|
||||
- **addressContracts** (used only by Ethereum type coins)
|
||||
|
||||
Maps *addrDesc* to *total number of transactions*, *number of non contract transactions* and array of *contracts* with *number of transfers* of given address.
|
||||
```
|
||||
(addrDesc []byte) -> (total_txs vuint)+(non-contract_txs vuint)+[]((contractAddrDesc []byte)+(nr_transfers vuint))
|
||||
```
|
||||
|
||||
- **blockTxs**
|
||||
|
||||
maps *block height* to an array of *txids* and *input points* in the block - only last 300 (by default) blocks are kept, the column is used in case of rollback.
|
||||
Maps *block height* to data necessary for blockchain rollback. Only last 300 (by default) blocks are kept.
|
||||
The content of value data differs for Bitcoin and Ethereum types.
|
||||
|
||||
- Bitcoin type
|
||||
|
||||
The value is an array of *txids* and *input points* in the block.
|
||||
```
|
||||
(height uint32) -> []((txid [32]byte)+(nr_inputs vuint)+[]((txid [32]byte)+(index vint)))
|
||||
```
|
||||
|
||||
- Ethereum type
|
||||
|
||||
The value is an array of transaction data. For each transaction is stored *txid*,
|
||||
*from* and *to* address descriptors and array of *contract address descriptors* with *transfer address descriptors*.
|
||||
```
|
||||
(height uint32) -> []((txid [32]byte)+(from addrDesc)+(to addrDesc)+(nr_contracts vuint)+[]((contract addrDesc)+(addr addrDesc)))
|
||||
```
|
||||
|
||||
- **transactions**
|
||||
|
||||
transaction cache, *txdata* is generated by coin specific parser function PackTx
|
||||
Transaction cache, *txdata* is generated by coin specific parser function PackTx.
|
||||
```
|
||||
(txid []byte) -> (txdata []byte)
|
||||
```
|
||||
|
|
|
|||
Loading…
Reference in New Issue