2017-08-28 09:50:57 -06:00
package main
import (
2018-01-18 09:33:20 -07:00
"context"
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
"encoding/json"
2017-08-28 09:50:57 -06:00
"flag"
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
"io/ioutil"
2018-03-09 09:05:27 -07:00
"log"
2018-06-12 13:23:34 -06:00
"math/rand"
2018-09-06 04:02:31 -06:00
"net/http"
_ "net/http/pprof"
2018-01-18 09:33:20 -07:00
"os"
"os/signal"
2019-06-11 08:27:47 -06:00
"runtime/debug"
2018-05-31 04:22:53 -06:00
"strings"
2018-05-30 06:37:30 -06:00
"sync/atomic"
2018-01-18 09:33:20 -07:00
"syscall"
2017-08-28 09:50:57 -06:00
"time"
2017-09-12 18:50:34 -06:00
2018-01-30 10:22:25 -07:00
"github.com/golang/glog"
2018-09-06 04:02:31 -06:00
"github.com/juju/errors"
2020-02-26 09:17:43 -07:00
"github.com/trezor/blockbook/api"
"github.com/trezor/blockbook/bchain"
"github.com/trezor/blockbook/bchain/coins"
"github.com/trezor/blockbook/common"
"github.com/trezor/blockbook/db"
"github.com/trezor/blockbook/fiat"
"github.com/trezor/blockbook/server"
2017-08-28 09:50:57 -06:00
)
2018-02-06 01:12:50 -07:00
// debounce too close requests for resync
const debounceResyncIndexMs = 1009
// debounce too close requests for resync mempool (ZeroMQ sends message for each tx, when new block there are many transactions)
const debounceResyncMempoolMs = 1009
2018-05-23 00:54:02 -06:00
// store internal state about once every minute
const storeInternalStatePeriodMs = 59699
2019-04-23 06:18:32 -06:00
// exit codes from the main function
const exitCodeOK = 0
const exitCodeFatal = 255
2017-08-28 09:50:57 -06:00
var (
2018-03-15 05:34:29 -06:00
blockchain = flag . String ( "blockchaincfg" , "" , "path to blockchain RPC service configuration json file" )
2017-08-28 09:50:57 -06:00
2018-09-18 03:49:39 -06:00
dbPath = flag . String ( "datadir" , "./data" , "path to database directory" )
dbCache = flag . Int ( "dbcache" , 1 << 29 , "size of the rocksdb cache" )
dbMaxOpenFiles = flag . Int ( "dbmaxopenfiles" , 1 << 14 , "max open files by rocksdb" )
2017-08-28 09:50:57 -06:00
2018-03-01 10:37:01 -07:00
blockFrom = flag . Int ( "blockheight" , - 1 , "height of the starting block" )
2018-01-29 09:27:42 -07:00
blockUntil = flag . Int ( "blockuntil" , - 1 , "height of the final block" )
rollbackHeight = flag . Int ( "rollback" , - 1 , "rollback to the given height and quit" )
2017-10-05 06:35:07 -06:00
2018-01-31 07:03:06 -07:00
synchronize = flag . Bool ( "sync" , false , "synchronizes until tip, if together with zeromq, keeps index synchronized" )
repair = flag . Bool ( "repair" , false , "repair the database" )
2020-01-26 12:12:12 -07:00
fixUtxo = flag . Bool ( "fixutxo" , false , "check and fix utxo db and exit" )
2018-03-09 09:05:27 -07:00
prof = flag . String ( "prof" , "" , "http server binding [address]:port of the interface to profiling data /debug/pprof/ (default no profiling)" )
2017-10-06 04:57:51 -06:00
2018-09-18 03:49:54 -06:00
syncChunk = flag . Int ( "chunk" , 100 , "block chunk size for processing in bulk mode" )
syncWorkers = flag . Int ( "workers" , 8 , "number of workers to process blocks in bulk mode" )
2018-02-26 08:44:25 -07:00
dryRun = flag . Bool ( "dryrun" , false , "do not index blocks, only download" )
2018-01-18 08:44:31 -07:00
2018-08-24 08:17:43 -06:00
debugMode = flag . Bool ( "debug" , false , "debug mode, return more verbose errors, reload templates on each request" )
2018-06-21 07:42:21 -06:00
internalBinding = flag . String ( "internal" , "" , "internal http server binding [address]:port, (default no internal server)" )
2018-01-19 07:58:46 -07:00
2018-09-18 03:49:54 -06:00
publicBinding = flag . String ( "public" , "" , "public http server binding [address]:port[/path] (default no public server)" )
2018-02-06 04:06:30 -07:00
2018-09-18 03:49:54 -06:00
certFiles = flag . String ( "certfile" , "" , "to enable SSL specify path to certificate files without extension, expecting <certfile>.crt and <certfile>.key (default no SSL)" )
2018-02-07 12:42:25 -07:00
2018-03-08 10:36:01 -07:00
explorerURL = flag . String ( "explorer" , "" , "address of blockchain explorer" )
2018-05-14 07:49:08 -06:00
noTxCache = flag . Bool ( "notxcache" , false , "disable tx cache" )
2018-06-01 08:01:58 -06:00
2019-04-28 03:54:12 -06:00
computeColumnStats = flag . Bool ( "computedbstats" , false , "compute column stats and exit" )
2019-05-06 00:38:12 -06:00
computeFeeStatsFlag = flag . Bool ( "computefeestats" , false , "compute fee stats for blocks in blockheight-blockuntil range and exit" )
2019-04-28 03:54:12 -06:00
dbStatsPeriodHours = flag . Int ( "dbstatsperiod" , 24 , "period of db stats collection in hours, 0 disables stats collection" )
2018-06-16 05:52:15 -06:00
// resync index at least each resyncIndexPeriodMs (could be more often if invoked by message from ZeroMQ)
resyncIndexPeriodMs = flag . Int ( "resyncindexperiod" , 935093 , "resync index period in milliseconds" )
// resync mempool at least each resyncMempoolPeriodMs (could be more often if invoked by message from ZeroMQ)
resyncMempoolPeriodMs = flag . Int ( "resyncmempoolperiod" , 60017 , "resync mempool period in milliseconds" )
2017-08-28 09:50:57 -06:00
)
2018-01-31 07:03:06 -07:00
var (
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
chanSyncIndex = make ( chan struct { } )
chanSyncMempool = make ( chan struct { } )
chanStoreInternalState = make ( chan struct { } )
chanSyncIndexDone = make ( chan struct { } )
chanSyncMempoolDone = make ( chan struct { } )
chanStoreInternalStateDone = make ( chan struct { } )
chain bchain . BlockChain
mempool bchain . Mempool
index * db . RocksDB
txCache * db . TxCache
metrics * common . Metrics
syncWorker * db . SyncWorker
internalState * common . InternalState
callbacksOnNewBlock [ ] bchain . OnNewBlockFunc
callbacksOnNewTxAddr [ ] bchain . OnNewTxAddrFunc
2020-05-24 09:58:29 -06:00
callbacksOnNewTx [ ] bchain . OnNewTxFunc
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
callbacksOnNewFiatRatesTicker [ ] fiat . OnNewFiatRatesTicker
chanOsSignal chan os . Signal
inShutdown int32
2018-01-31 07:03:06 -07:00
)
2018-04-12 07:23:16 -06:00
func init ( ) {
2018-05-17 04:31:45 -06:00
glog . MaxSize = 1024 * 1024 * 8
2018-04-12 07:23:16 -06:00
glog . CopyStandardLogTo ( "INFO" )
}
2017-08-28 09:50:57 -06:00
func main ( ) {
2019-06-11 08:27:47 -06:00
defer func ( ) {
if e := recover ( ) ; e != nil {
glog . Error ( "main recovered from panic: " , e )
debug . PrintStack ( )
os . Exit ( - 1 )
}
} ( )
2019-04-23 06:18:32 -06:00
os . Exit ( mainWithExitCode ( ) )
}
// allow deferred functions to run even in case of fatal error
func mainWithExitCode ( ) int {
2017-08-28 09:50:57 -06:00
flag . Parse ( )
2018-01-31 07:03:06 -07:00
defer glog . Flush ( )
2018-06-12 13:23:34 -06:00
rand . Seed ( time . Now ( ) . UTC ( ) . UnixNano ( ) )
2018-02-28 16:59:25 -07:00
chanOsSignal = make ( chan os . Signal , 1 )
signal . Notify ( chanOsSignal , syscall . SIGHUP , syscall . SIGINT , syscall . SIGQUIT , syscall . SIGTERM )
2018-08-24 08:17:43 -06:00
glog . Infof ( "Blockbook: %+v, debug mode %v" , common . GetVersionInfo ( ) , * debugMode )
2018-05-22 09:22:22 -06:00
2018-03-09 09:05:27 -07:00
if * prof != "" {
go func ( ) {
log . Println ( http . ListenAndServe ( * prof , nil ) )
} ( )
2018-01-19 07:58:46 -07:00
}
2017-09-12 08:53:40 -06:00
if * repair {
2018-01-18 09:33:20 -07:00
if err := db . RepairRocksDB ( * dbPath ) ; err != nil {
2019-04-12 07:44:11 -06:00
glog . Errorf ( "RepairRocksDB %s: %v" , * dbPath , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2017-09-12 08:53:40 -06:00
}
2019-04-23 06:18:32 -06:00
return exitCodeOK
2017-09-12 08:53:40 -06:00
}
2018-06-05 08:14:46 -06:00
if * blockchain == "" {
2019-04-12 07:44:11 -06:00
glog . Error ( "Missing blockchaincfg configuration parameter" )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-06-05 08:14:46 -06:00
}
2018-10-09 16:23:31 -06:00
coin , coinShortcut , coinLabel , err := coins . GetCoinNameFromConfig ( * blockchain )
2018-03-13 04:34:49 -06:00
if err != nil {
2019-04-12 07:44:11 -06:00
glog . Error ( "config: " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-03-13 04:34:49 -06:00
}
2018-09-20 04:29:13 -06:00
// gspt.SetProcTitle("blockbook-" + normalizeName(coin))
2018-09-06 04:02:31 -06:00
2018-09-19 08:51:16 -06:00
metrics , err = common . GetMetrics ( coin )
2018-06-05 08:14:46 -06:00
if err != nil {
2019-04-12 07:44:11 -06:00
glog . Error ( "metrics: " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-03-15 05:34:29 -06:00
}
2019-04-23 06:18:32 -06:00
if chain , mempool , err = getBlockChainWithRetry ( coin , * blockchain , pushSynchronizationHandler , metrics , 120 ) ; err != nil {
2019-04-12 07:44:11 -06:00
glog . Error ( "rpc: " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2017-10-07 03:05:35 -06:00
}
2018-09-18 03:49:39 -06:00
index , err = db . NewRocksDB ( * dbPath , * dbCache , * dbMaxOpenFiles , chain . GetChainParser ( ) , metrics )
2017-08-28 09:50:57 -06:00
if err != nil {
2019-04-12 07:44:11 -06:00
glog . Error ( "rocksDB: " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2017-08-28 09:50:57 -06:00
}
2018-01-31 07:03:06 -07:00
defer index . Close ( )
2017-08-28 09:50:57 -06:00
2020-02-16 15:07:50 -07:00
internalState , err = newInternalState ( coin , coinShortcut , coinLabel , index )
if err != nil {
glog . Error ( "internalState: " , err )
return exitCodeFatal
}
// fix possible inconsistencies in the UTXO index
if * fixUtxo || ! internalState . UtxoChecked {
2020-01-26 12:12:12 -07:00
err = index . FixUtxos ( chanOsSignal )
if err != nil {
glog . Error ( "fixUtxos: " , err )
return exitCodeFatal
}
2020-02-16 15:07:50 -07:00
internalState . UtxoChecked = true
}
index . SetInternalState ( internalState )
if * fixUtxo {
err = index . StoreInternalState ( internalState )
if err != nil {
glog . Error ( "StoreInternalState: " , err )
return exitCodeFatal
}
2020-01-26 12:12:12 -07:00
return exitCodeOK
}
2018-05-29 03:37:35 -06:00
if internalState . DbState != common . DbStateClosed {
2018-08-18 16:23:26 -06:00
if internalState . DbState == common . DbStateInconsistent {
glog . Error ( "internalState: database is in inconsistent state and cannot be used" )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-08-18 16:23:26 -06:00
}
glog . Warning ( "internalState: database was left in open state, possibly previous ungraceful shutdown" )
2018-05-22 04:56:51 -06:00
}
2019-04-28 03:54:12 -06:00
if * computeFeeStatsFlag {
internalState . DbState = common . DbStateOpen
err = computeFeeStats ( chanOsSignal , * blockFrom , * blockUntil , index , chain , txCache , internalState , metrics )
if err != nil && err != db . ErrOperationInterrupted {
glog . Error ( "computeFeeStats: " , err )
return exitCodeFatal
}
return exitCodeOK
}
2018-06-01 08:01:58 -06:00
if * computeColumnStats {
internalState . DbState = common . DbStateOpen
2018-06-08 05:19:57 -06:00
err = index . ComputeInternalStateColumnStats ( chanOsSignal )
2018-06-01 08:01:58 -06:00
if err != nil {
glog . Error ( "internalState: " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-06-01 08:01:58 -06:00
}
2018-06-04 05:24:40 -06:00
glog . Info ( "DB size on disk: " , index . DatabaseSizeOnDisk ( ) , ", DB size as computed: " , internalState . DBSizeTotal ( ) )
2019-04-23 06:18:32 -06:00
return exitCodeOK
2018-06-01 08:01:58 -06:00
}
2018-05-29 03:37:35 -06:00
syncWorker , err = db . NewSyncWorker ( index , chain , * syncWorkers , * syncChunk , * blockFrom , * dryRun , chanOsSignal , metrics , internalState )
2018-03-14 05:34:13 -06:00
if err != nil {
2019-04-12 07:44:11 -06:00
glog . Errorf ( "NewSyncWorker %v" , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-03-14 05:34:13 -06:00
}
2018-05-22 04:56:51 -06:00
// set the DbState to open at this moment, after all important workers are initialized
2018-05-29 03:37:35 -06:00
internalState . DbState = common . DbStateOpen
err = index . StoreInternalState ( internalState )
2018-05-22 04:56:51 -06:00
if err != nil {
2019-04-12 07:44:11 -06:00
glog . Error ( "internalState: " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-05-22 04:56:51 -06:00
}
2018-01-29 09:27:42 -07:00
if * rollbackHeight >= 0 {
2019-04-23 06:18:32 -06:00
err = performRollback ( )
if err != nil {
return exitCodeFatal
}
return exitCodeOK
2018-01-29 09:27:42 -07:00
}
2018-10-08 06:55:21 -06:00
if txCache , err = db . NewTxCache ( index , chain , metrics , internalState , ! * noTxCache ) ; err != nil {
2018-03-06 04:36:24 -07:00
glog . Error ( "txCache " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-03-06 04:36:24 -07:00
}
2018-09-19 08:51:16 -06:00
// report BlockbookAppInfo metric, only log possible error
if err = blockbookAppInfoMetric ( index , chain , txCache , internalState , metrics ) ; err != nil {
glog . Error ( "blockbookAppInfoMetric " , err )
}
2018-06-21 07:37:46 -06:00
var internalServer * server . InternalServer
if * internalBinding != "" {
2019-03-27 05:46:45 -06:00
internalServer , err = startInternalServer ( )
2018-01-18 08:44:31 -07:00
if err != nil {
2019-03-27 05:46:45 -06:00
glog . Error ( "internal server: " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-01-18 08:44:31 -07:00
}
}
2018-06-21 07:37:46 -06:00
var publicServer * server . PublicServer
if * publicBinding != "" {
2019-03-27 05:46:45 -06:00
publicServer , err = startPublicServer ( )
2018-02-06 04:06:30 -07:00
if err != nil {
2019-03-27 05:46:45 -06:00
glog . Error ( "public server: " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-02-06 04:06:30 -07:00
}
2018-02-22 05:01:35 -07:00
}
if * synchronize {
2018-09-20 04:15:46 -06:00
internalState . SyncMode = true
internalState . InitialSync = true
2018-10-01 05:43:38 -06:00
if err := syncWorker . ResyncIndex ( nil , true ) ; err != nil {
2019-04-28 03:54:12 -06:00
if err != db . ErrOperationInterrupted {
2019-04-23 06:18:32 -06:00
glog . Error ( "resyncIndex " , err )
return exitCodeFatal
}
return exitCodeOK
2018-09-20 04:15:46 -06:00
}
2019-03-25 09:43:57 -06:00
// initialize mempool after the initial sync is complete
2019-03-29 10:01:20 -06:00
var addrDescForOutpoint bchain . AddrDescForOutpointFunc
if chain . GetChainParser ( ) . GetChainType ( ) == bchain . ChainBitcoinType {
addrDescForOutpoint = index . AddrDescForOutpoint
}
2020-05-24 09:58:29 -06:00
err = chain . InitializeMempool ( addrDescForOutpoint , onNewTxAddr , onNewTx )
2019-03-25 09:43:57 -06:00
if err != nil {
glog . Error ( "initializeMempool " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2019-03-25 09:43:57 -06:00
}
2018-09-20 04:15:46 -06:00
var mempoolCount int
2019-04-01 09:00:34 -06:00
if mempoolCount , err = mempool . Resync ( ) ; err != nil {
2018-09-20 04:15:46 -06:00
glog . Error ( "resyncMempool " , err )
2019-04-23 06:18:32 -06:00
return exitCodeFatal
2018-09-20 04:15:46 -06:00
}
internalState . FinishedMempoolSync ( mempoolCount )
2018-02-22 05:01:35 -07:00
go syncIndexLoop ( )
go syncMempoolLoop ( )
2018-09-20 04:15:46 -06:00
internalState . InitialSync = false
}
go storeInternalStateLoop ( )
2019-03-27 05:46:45 -06:00
if publicServer != nil {
2018-09-20 04:15:46 -06:00
// start full public interface
2019-04-01 09:00:34 -06:00
callbacksOnNewBlock = append ( callbacksOnNewBlock , publicServer . OnNewBlock )
callbacksOnNewTxAddr = append ( callbacksOnNewTxAddr , publicServer . OnNewTxAddr )
2020-05-24 09:58:29 -06:00
callbacksOnNewTx = append ( callbacksOnNewTx , publicServer . OnNewTx )
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
callbacksOnNewFiatRatesTicker = append ( callbacksOnNewFiatRatesTicker , publicServer . OnNewFiatRatesTicker )
2018-09-20 04:15:46 -06:00
publicServer . ConnectFullPublicInterface ( )
2018-02-06 04:06:30 -07:00
}
2018-03-01 10:37:01 -07:00
if * blockFrom >= 0 {
2017-08-28 09:50:57 -06:00
if * blockUntil < 0 {
2018-03-01 10:37:01 -07:00
* blockUntil = * blockFrom
2017-08-28 09:50:57 -06:00
}
2018-03-01 10:37:01 -07:00
height := uint32 ( * blockFrom )
2017-08-28 09:50:57 -06:00
until := uint32 ( * blockUntil )
2019-01-03 09:19:56 -07:00
if ! * synchronize {
2018-04-28 16:17:30 -06:00
if err = syncWorker . ConnectBlocksParallel ( height , until ) ; err != nil {
2019-04-28 03:54:12 -06:00
if err != db . ErrOperationInterrupted {
2019-04-23 06:18:32 -06:00
glog . Error ( "connectBlocksParallel " , err )
return exitCodeFatal
}
return exitCodeOK
2017-08-28 09:50:57 -06:00
}
}
}
2018-01-18 09:33:20 -07:00
2018-06-21 07:37:46 -06:00
if internalServer != nil || publicServer != nil || chain != nil {
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
// start fiat rates downloader only if not shutting down immediately
initFiatRatesDownloader ( index , * blockchain )
2018-06-21 07:37:46 -06:00
waitForSignalAndShutdown ( internalServer , publicServer , chain , 10 * time . Second )
2018-01-18 09:33:20 -07:00
}
2018-01-31 07:03:06 -07:00
2018-02-02 08:17:33 -07:00
if * synchronize {
close ( chanSyncIndex )
close ( chanSyncMempool )
2018-05-23 00:54:02 -06:00
close ( chanStoreInternalState )
2018-02-02 08:17:33 -07:00
<- chanSyncIndexDone
<- chanSyncMempoolDone
2018-06-08 05:19:57 -06:00
<- chanStoreInternalStateDone
2018-02-02 08:17:33 -07:00
}
2019-04-23 06:18:32 -06:00
return exitCodeOK
2018-01-31 07:03:06 -07:00
}
2019-03-27 05:46:45 -06:00
func getBlockChainWithRetry ( coin string , configfile string , pushHandler func ( bchain . NotificationType ) , metrics * common . Metrics , seconds int ) ( bchain . BlockChain , bchain . Mempool , error ) {
var chain bchain . BlockChain
var mempool bchain . Mempool
var err error
timer := time . NewTimer ( time . Second )
for i := 0 ; ; i ++ {
if chain , mempool , err = coins . NewBlockChain ( coin , configfile , pushHandler , metrics ) ; err != nil {
if i < seconds {
glog . Error ( "rpc: " , err , " Retrying..." )
select {
case <- chanOsSignal :
return nil , nil , errors . New ( "Interrupted" )
case <- timer . C :
timer . Reset ( time . Second )
continue
}
} else {
return nil , nil , err
}
}
return chain , mempool , nil
}
}
func startInternalServer ( ) ( * server . InternalServer , error ) {
internalServer , err := server . NewInternalServer ( * internalBinding , * certFiles , index , chain , mempool , txCache , internalState )
if err != nil {
return nil , err
}
go func ( ) {
err = internalServer . Run ( )
if err != nil {
if err . Error ( ) == "http: Server closed" {
glog . Info ( "internal server: closed" )
} else {
glog . Error ( err )
return
}
}
} ( )
return internalServer , nil
}
func startPublicServer ( ) ( * server . PublicServer , error ) {
// start public server in limited functionality, extend it after sync is finished by calling ConnectFullPublicInterface
publicServer , err := server . NewPublicServer ( * publicBinding , * certFiles , index , chain , mempool , txCache , * explorerURL , metrics , internalState , * debugMode )
if err != nil {
return nil , err
}
go func ( ) {
err = publicServer . Run ( )
if err != nil {
if err . Error ( ) == "http: Server closed" {
glog . Info ( "public server: closed" )
} else {
glog . Error ( err )
return
}
}
} ( )
return publicServer , err
}
2019-04-23 06:18:32 -06:00
func performRollback ( ) error {
2019-03-25 09:43:57 -06:00
bestHeight , bestHash , err := index . GetBestBlock ( )
if err != nil {
glog . Error ( "rollbackHeight: " , err )
2019-04-23 06:18:32 -06:00
return err
2019-03-25 09:43:57 -06:00
}
if uint32 ( * rollbackHeight ) > bestHeight {
glog . Infof ( "nothing to rollback, rollbackHeight %d, bestHeight: %d" , * rollbackHeight , bestHeight )
} else {
hashes := [ ] string { bestHash }
for height := bestHeight - 1 ; height >= uint32 ( * rollbackHeight ) ; height -- {
hash , err := index . GetBlockHash ( height )
if err != nil {
glog . Error ( "rollbackHeight: " , err )
2019-04-23 06:18:32 -06:00
return err
2019-03-25 09:43:57 -06:00
}
hashes = append ( hashes , hash )
}
err = syncWorker . DisconnectBlocks ( uint32 ( * rollbackHeight ) , bestHeight , hashes )
if err != nil {
glog . Error ( "rollbackHeight: " , err )
2019-04-23 06:18:32 -06:00
return err
2019-03-25 09:43:57 -06:00
}
}
2019-04-23 06:18:32 -06:00
return nil
2019-03-25 09:43:57 -06:00
}
2018-09-19 08:51:16 -06:00
func blockbookAppInfoMetric ( db * db . RocksDB , chain bchain . BlockChain , txCache * db . TxCache , is * common . InternalState , metrics * common . Metrics ) error {
2019-03-25 09:43:57 -06:00
api , err := api . NewWorker ( db , chain , mempool , txCache , is )
2018-09-19 08:51:16 -06:00
if err != nil {
return err
}
si , err := api . GetSystemInfo ( false )
if err != nil {
return err
}
2018-09-26 03:26:38 -06:00
metrics . BlockbookAppInfo . Reset ( )
2018-09-19 08:51:16 -06:00
metrics . BlockbookAppInfo . With ( common . Labels {
2018-09-26 04:16:56 -06:00
"blockbook_version" : si . Blockbook . Version ,
2018-09-19 08:51:16 -06:00
"blockbook_commit" : si . Blockbook . GitCommit ,
2018-10-02 04:09:04 -06:00
"blockbook_buildtime" : si . Blockbook . BuildTime ,
2018-09-19 08:51:16 -06:00
"backend_version" : si . Backend . Version ,
"backend_subversion" : si . Backend . Subversion ,
"backend_protocol_version" : si . Backend . ProtocolVersion } ) . Set ( float64 ( 0 ) )
return nil
}
2018-10-09 16:23:31 -06:00
func newInternalState ( coin , coinShortcut , coinLabel string , d * db . RocksDB ) ( * common . InternalState , error ) {
2018-05-30 06:44:40 -06:00
is , err := d . LoadInternalState ( coin )
if err != nil {
return nil , err
}
2018-06-27 16:36:56 -06:00
is . CoinShortcut = coinShortcut
2018-10-09 16:23:31 -06:00
if coinLabel == "" {
coinLabel = coin
}
is . CoinLabel = coinLabel
2018-05-30 06:44:40 -06:00
name , err := os . Hostname ( )
if err != nil {
glog . Error ( "get hostname " , err )
} else {
2018-05-31 04:22:53 -06:00
if i := strings . IndexByte ( name , '.' ) ; i > 0 {
name = name [ : i ]
}
2018-05-30 06:44:40 -06:00
is . Host = name
}
return is , nil
2018-05-29 03:37:35 -06:00
}
2018-02-01 03:24:53 -07:00
func tickAndDebounce ( tickTime time . Duration , debounceTime time . Duration , input chan struct { } , f func ( ) ) {
timer := time . NewTimer ( tickTime )
2018-04-03 14:09:46 -06:00
var firstDebounce time . Time
2018-02-01 03:24:53 -07:00
Loop :
for {
select {
case _ , ok := <- input :
2018-02-01 04:26:12 -07:00
if ! timer . Stop ( ) {
<- timer . C
}
// exit loop on closed input channel
2018-02-01 03:24:53 -07:00
if ! ok {
break Loop
}
2018-04-03 14:09:46 -06:00
if firstDebounce . IsZero ( ) {
firstDebounce = time . Now ( )
}
// debounce for up to debounceTime period
// afterwards execute immediately
if firstDebounce . Add ( debounceTime ) . After ( time . Now ( ) ) {
timer . Reset ( debounceTime )
} else {
timer . Reset ( 0 )
}
2018-02-01 04:26:12 -07:00
case <- timer . C :
2018-05-30 06:37:30 -06:00
// do the action, if not in shutdown, then start the loop again
if atomic . LoadInt32 ( & inShutdown ) == 0 {
f ( )
}
2018-02-01 04:26:12 -07:00
timer . Reset ( tickTime )
2018-04-03 14:09:46 -06:00
firstDebounce = time . Time { }
2018-02-01 03:24:53 -07:00
}
}
}
2018-01-31 09:51:48 -07:00
func syncIndexLoop ( ) {
defer close ( chanSyncIndexDone )
glog . Info ( "syncIndexLoop starting" )
2018-02-01 03:24:53 -07:00
// resync index about every 15 minutes if there are no chanSyncIndex requests, with debounce 1 second
2018-06-16 05:52:15 -06:00
tickAndDebounce ( time . Duration ( * resyncIndexPeriodMs ) * time . Millisecond , debounceResyncIndexMs * time . Millisecond , chanSyncIndex , func ( ) {
2018-09-25 03:20:58 -06:00
if err := syncWorker . ResyncIndex ( onNewBlockHash , false ) ; err != nil {
2019-05-22 07:47:28 -06:00
glog . Error ( "syncIndexLoop " , errors . ErrorStack ( err ) , ", will retry..." )
// retry once in case of random network error, after a slight delay
time . Sleep ( time . Millisecond * 2500 )
if err := syncWorker . ResyncIndex ( onNewBlockHash , false ) ; err != nil {
glog . Error ( "syncIndexLoop " , errors . ErrorStack ( err ) )
}
2018-01-31 09:51:48 -07:00
}
2018-02-01 03:24:53 -07:00
} )
2018-01-31 09:51:48 -07:00
glog . Info ( "syncIndexLoop stopped" )
}
2018-09-11 05:37:12 -06:00
func onNewBlockHash ( hash string , height uint32 ) {
2020-10-29 01:55:04 -06:00
defer func ( ) {
if r := recover ( ) ; r != nil {
glog . Error ( "onNewBlockHash recovered from panic: " , r )
}
} ( )
2018-09-11 05:37:12 -06:00
for _ , c := range callbacksOnNewBlock {
c ( hash , height )
2018-02-22 05:01:35 -07:00
}
}
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
func onNewFiatRatesTicker ( ticker * db . CurrencyRatesTicker ) {
2020-10-29 01:55:04 -06:00
defer func ( ) {
if r := recover ( ) ; r != nil {
glog . Error ( "onNewFiatRatesTicker recovered from panic: " , r )
}
} ( )
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
for _ , c := range callbacksOnNewFiatRatesTicker {
c ( ticker )
}
}
2018-01-31 09:51:48 -07:00
func syncMempoolLoop ( ) {
defer close ( chanSyncMempoolDone )
glog . Info ( "syncMempoolLoop starting" )
2018-02-01 03:24:53 -07:00
// resync mempool about every minute if there are no chanSyncMempool requests, with debounce 1 second
2018-06-16 05:52:15 -06:00
tickAndDebounce ( time . Duration ( * resyncMempoolPeriodMs ) * time . Millisecond , debounceResyncMempoolMs * time . Millisecond , chanSyncMempool , func ( ) {
2018-05-29 03:37:35 -06:00
internalState . StartedMempoolSync ( )
2019-04-01 09:00:34 -06:00
if count , err := mempool . Resync ( ) ; err != nil {
2018-03-01 03:22:37 -07:00
glog . Error ( "syncMempoolLoop " , errors . ErrorStack ( err ) )
2018-05-22 09:34:37 -06:00
} else {
2018-06-01 05:22:56 -06:00
internalState . FinishedMempoolSync ( count )
2018-01-31 09:51:48 -07:00
}
2018-02-01 03:24:53 -07:00
} )
2018-01-31 09:51:48 -07:00
glog . Info ( "syncMempoolLoop stopped" )
2018-01-18 09:33:20 -07:00
}
2018-05-23 00:54:02 -06:00
func storeInternalStateLoop ( ) {
2018-06-08 05:19:57 -06:00
stopCompute := make ( chan os . Signal )
defer func ( ) {
close ( stopCompute )
close ( chanStoreInternalStateDone )
} ( )
2020-01-26 12:12:12 -07:00
signal . Notify ( stopCompute , syscall . SIGHUP , syscall . SIGINT , syscall . SIGQUIT , syscall . SIGTERM )
2018-06-08 05:19:57 -06:00
var computeRunning bool
2018-06-12 13:23:34 -06:00
lastCompute := time . Now ( )
2018-09-19 08:51:16 -06:00
lastAppInfo := time . Now ( )
logAppInfoPeriod := 15 * time . Minute
2018-12-27 13:35:40 -07:00
// randomize the duration between ComputeInternalStateColumnStats to avoid peaks after reboot of machine with multiple blockbooks
computePeriod := time . Duration ( * dbStatsPeriodHours ) * time . Hour + time . Duration ( rand . Float64 ( ) * float64 ( ( 4 * time . Hour ) . Nanoseconds ( ) ) )
if ( * dbStatsPeriodHours ) > 0 {
glog . Info ( "storeInternalStateLoop starting with db stats recompute period " , computePeriod )
} else {
glog . Info ( "storeInternalStateLoop starting with db stats compute disabled" )
}
2018-05-23 00:54:02 -06:00
tickAndDebounce ( storeInternalStatePeriodMs * time . Millisecond , ( storeInternalStatePeriodMs - 1 ) * time . Millisecond , chanStoreInternalState , func ( ) {
2018-12-27 13:35:40 -07:00
if ( * dbStatsPeriodHours ) > 0 && ! computeRunning && lastCompute . Add ( computePeriod ) . Before ( time . Now ( ) ) {
2018-06-08 05:19:57 -06:00
computeRunning = true
go func ( ) {
err := index . ComputeInternalStateColumnStats ( stopCompute )
if err != nil {
glog . Error ( "computeInternalStateColumnStats error: " , err )
}
lastCompute = time . Now ( )
computeRunning = false
} ( )
}
2018-05-29 03:37:35 -06:00
if err := index . StoreInternalState ( internalState ) ; err != nil {
2018-05-23 00:54:02 -06:00
glog . Error ( "storeInternalStateLoop " , errors . ErrorStack ( err ) )
}
2018-09-19 08:51:16 -06:00
if lastAppInfo . Add ( logAppInfoPeriod ) . Before ( time . Now ( ) ) {
2018-08-26 13:29:10 -06:00
glog . Info ( index . GetMemoryStats ( ) )
2018-09-19 08:51:16 -06:00
if err := blockbookAppInfoMetric ( index , chain , txCache , internalState , metrics ) ; err != nil {
glog . Error ( "blockbookAppInfoMetric " , err )
}
lastAppInfo = time . Now ( )
2018-08-26 13:29:10 -06:00
}
2018-05-23 00:54:02 -06:00
} )
glog . Info ( "storeInternalStateLoop stopped" )
}
2018-12-19 02:06:25 -07:00
func onNewTxAddr ( tx * bchain . Tx , desc bchain . AddressDescriptor ) {
2020-10-29 01:55:04 -06:00
defer func ( ) {
if r := recover ( ) ; r != nil {
glog . Error ( "onNewTxAddr recovered from panic: " , r )
}
} ( )
2018-02-22 05:32:06 -07:00
for _ , c := range callbacksOnNewTxAddr {
2018-12-19 02:06:25 -07:00
c ( tx , desc )
2018-02-22 05:32:06 -07:00
}
}
2020-05-24 09:58:29 -06:00
func onNewTx ( tx * bchain . MempoolTx ) {
2020-10-29 01:55:04 -06:00
defer func ( ) {
if r := recover ( ) ; r != nil {
glog . Error ( "onNewTx recovered from panic: " , r )
}
} ( )
2020-05-24 09:58:29 -06:00
for _ , c := range callbacksOnNewTx {
c ( tx )
}
}
2018-03-27 15:39:06 -06:00
func pushSynchronizationHandler ( nt bchain . NotificationType ) {
2018-12-11 04:14:05 -07:00
glog . V ( 1 ) . Info ( "MQ: notification " , nt )
2018-05-30 06:37:30 -06:00
if atomic . LoadInt32 ( & inShutdown ) != 0 {
return
}
2018-03-27 15:39:06 -06:00
if nt == bchain . NotificationNewBlock {
2018-01-31 09:51:48 -07:00
chanSyncIndex <- struct { } { }
2018-03-27 15:39:06 -06:00
} else if nt == bchain . NotificationNewTx {
2018-01-31 09:51:48 -07:00
chanSyncMempool <- struct { } { }
2018-01-31 07:03:06 -07:00
} else {
2018-03-27 15:39:06 -06:00
glog . Error ( "MQ: unknown notification sent" )
2018-01-31 07:03:06 -07:00
}
2018-01-22 08:46:54 -07:00
}
2018-01-18 09:33:20 -07:00
2018-06-21 07:37:46 -06:00
func waitForSignalAndShutdown ( internal * server . InternalServer , public * server . PublicServer , chain bchain . BlockChain , timeout time . Duration ) {
2018-02-28 16:59:25 -07:00
sig := <- chanOsSignal
2018-05-30 06:37:30 -06:00
atomic . StoreInt32 ( & inShutdown , 1 )
2018-06-21 07:37:46 -06:00
glog . Infof ( "shutdown: %v" , sig )
2018-01-18 09:33:20 -07:00
ctx , cancel := context . WithTimeout ( context . Background ( ) , timeout )
defer cancel ( )
2018-06-21 07:37:46 -06:00
if internal != nil {
if err := internal . Shutdown ( ctx ) ; err != nil {
glog . Error ( "internal server: shutdown error: " , err )
2018-01-18 12:32:10 -07:00
}
2018-01-18 09:33:20 -07:00
}
2018-02-06 04:06:30 -07:00
2018-06-21 07:37:46 -06:00
if public != nil {
if err := public . Shutdown ( ctx ) ; err != nil {
glog . Error ( "public server: shutdown error: " , err )
2018-02-06 04:06:30 -07:00
}
}
2018-03-15 05:34:29 -06:00
if chain != nil {
2018-05-30 06:37:30 -06:00
if err := chain . Shutdown ( ctx ) ; err != nil {
2018-06-21 07:37:46 -06:00
glog . Error ( "rpc: shutdown error: " , err )
2018-03-15 05:34:29 -06:00
}
}
2017-08-28 09:50:57 -06:00
}
2018-12-04 03:54:15 -07:00
func printResult ( txid string , vout int32 , isOutput bool ) error {
2018-02-03 11:14:27 -07:00
glog . Info ( txid , vout , isOutput )
2017-08-28 09:50:57 -06:00
return nil
}
2018-09-06 04:02:31 -06:00
func normalizeName ( s string ) string {
s = strings . ToLower ( s )
s = strings . Replace ( s , " " , "-" , - 1 )
return s
}
2019-04-28 03:54:12 -06:00
// computeFeeStats computes fee distribution in defined blocks
func computeFeeStats ( stopCompute chan os . Signal , blockFrom , blockTo int , db * db . RocksDB , chain bchain . BlockChain , txCache * db . TxCache , is * common . InternalState , metrics * common . Metrics ) error {
start := time . Now ( )
glog . Info ( "computeFeeStats start" )
api , err := api . NewWorker ( db , chain , mempool , txCache , is )
if err != nil {
return err
}
err = api . ComputeFeeStats ( blockFrom , blockTo , stopCompute )
glog . Info ( "computeFeeStats finished in " , time . Since ( start ) )
return err
}
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
func initFiatRatesDownloader ( db * db . RocksDB , configfile string ) {
data , err := ioutil . ReadFile ( configfile )
if err != nil {
glog . Errorf ( "Error reading file %v, %v" , configfile , err )
return
}
var config struct {
FiatRates string ` json:"fiat_rates" `
FiatRatesParams string ` json:"fiat_rates_params" `
}
err = json . Unmarshal ( data , & config )
if err != nil {
glog . Errorf ( "Error parsing config file %v, %v" , configfile , err )
return
}
if config . FiatRates == "" || config . FiatRatesParams == "" {
glog . Infof ( "FiatRates config (%v) is empty, so the functionality is disabled." , configfile )
} else {
fiatRates , err := fiat . NewFiatRatesDownloader ( db , config . FiatRates , config . FiatRatesParams , nil , onNewFiatRatesTicker )
if err != nil {
glog . Errorf ( "NewFiatRatesDownloader Init error: %v" , err )
return
}
glog . Infof ( "Starting %v FiatRates downloader..." , config . FiatRates )
go fiatRates . Run ( )
}
}