Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
|
|
|
package fiat
|
|
|
|
|
|
|
|
import (
|
|
|
|
"encoding/json"
|
|
|
|
"errors"
|
|
|
|
"io/ioutil"
|
|
|
|
"net/http"
|
|
|
|
"strconv"
|
|
|
|
"time"
|
|
|
|
|
|
|
|
"github.com/golang/glog"
|
2020-02-26 09:17:43 -07:00
|
|
|
"github.com/trezor/blockbook/db"
|
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
|
|
|
)
|
|
|
|
|
|
|
|
// Coingecko is a structure that implements RatesDownloaderInterface
|
|
|
|
type Coingecko struct {
|
|
|
|
url string
|
|
|
|
coin string
|
|
|
|
httpTimeoutSeconds time.Duration
|
|
|
|
timeFormat string
|
|
|
|
}
|
|
|
|
|
|
|
|
// NewCoinGeckoDownloader creates a coingecko structure that implements the RatesDownloaderInterface
|
|
|
|
func NewCoinGeckoDownloader(url string, coin string, timeFormat string) RatesDownloaderInterface {
|
|
|
|
return &Coingecko{
|
|
|
|
url: url,
|
|
|
|
coin: coin,
|
|
|
|
httpTimeoutSeconds: 15 * time.Second,
|
|
|
|
timeFormat: timeFormat,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// makeRequest retrieves the response from Coingecko API at the specified date.
|
|
|
|
// If timestamp is nil, it fetches the latest market data available.
|
|
|
|
func (cg *Coingecko) makeRequest(timestamp *time.Time) ([]byte, error) {
|
|
|
|
requestURL := cg.url + "/coins/" + cg.coin
|
|
|
|
if timestamp != nil {
|
|
|
|
requestURL += "/history"
|
|
|
|
}
|
|
|
|
|
|
|
|
req, err := http.NewRequest("GET", requestURL, nil)
|
|
|
|
if err != nil {
|
|
|
|
glog.Errorf("Error creating a new request for %v: %v", requestURL, err)
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
req.Close = true
|
|
|
|
req.Header.Set("Content-Type", "application/json")
|
|
|
|
|
|
|
|
// Add query parameters
|
|
|
|
q := req.URL.Query()
|
|
|
|
|
|
|
|
// Add a unix timestamp to query parameters to get uncached responses
|
|
|
|
currentTimestamp := strconv.FormatInt(time.Now().UTC().UnixNano(), 10)
|
|
|
|
q.Add("current_timestamp", currentTimestamp)
|
|
|
|
|
|
|
|
if timestamp == nil {
|
|
|
|
q.Add("market_data", "true")
|
|
|
|
q.Add("localization", "false")
|
|
|
|
q.Add("tickers", "false")
|
|
|
|
q.Add("community_data", "false")
|
|
|
|
q.Add("developer_data", "false")
|
|
|
|
} else {
|
|
|
|
timestampFormatted := timestamp.Format(cg.timeFormat)
|
|
|
|
q.Add("date", timestampFormatted)
|
|
|
|
}
|
|
|
|
req.URL.RawQuery = q.Encode()
|
|
|
|
|
|
|
|
client := &http.Client{
|
|
|
|
Timeout: cg.httpTimeoutSeconds,
|
|
|
|
}
|
|
|
|
resp, err := client.Do(req)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
defer resp.Body.Close()
|
|
|
|
if resp.StatusCode != http.StatusOK {
|
|
|
|
return nil, errors.New("Invalid response status: " + string(resp.Status))
|
|
|
|
}
|
|
|
|
bodyBytes, err := ioutil.ReadAll(resp.Body)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return bodyBytes, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// GetData gets fiat rates from API at the specified date and returns a CurrencyRatesTicker
|
|
|
|
// If timestamp is nil, it will download the current fiat rates.
|
|
|
|
func (cg *Coingecko) getTicker(timestamp *time.Time) (*db.CurrencyRatesTicker, error) {
|
|
|
|
dataTimestamp := timestamp
|
|
|
|
if timestamp == nil {
|
|
|
|
timeNow := time.Now()
|
|
|
|
dataTimestamp = &timeNow
|
|
|
|
}
|
|
|
|
dataTimestampUTC := dataTimestamp.UTC()
|
|
|
|
ticker := &db.CurrencyRatesTicker{Timestamp: &dataTimestampUTC}
|
|
|
|
bodyBytes, err := cg.makeRequest(timestamp)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
type FiatRatesResponse struct {
|
|
|
|
MarketData struct {
|
2019-12-19 09:30:19 -07:00
|
|
|
Prices map[string]float64 `json:"current_price"`
|
Add fiat rates functionality (#316)
* Add initial commit for fiat rates functionality
* templates.go: use bash from current user's environment
* bitcoinrpc.go: add FiatRates and FiatRatesParams to config
* blockbook.go: add initFiatRatesDownloader kickoff
* bitcoin.json: add coingecko API URL
* rockdb.go: add FindTicker and StoreTicker functions
* rocksdb_test.go: add a simple test for storing and getting FiatRate tickers
* rocksdb: add FindLastTicker and convertDate, make FindTicker return strings
* rocksdb: add ConvertDate function and CoinGeckoTicker struct, update tests
* blockbook.go, fiat: finalize the CoinGecko downloader
* coingecko.go: do not stop syncing when encountered an error
* rocksdb_test: fix the exported function name
* worker.go: make getBlockInfoFromBlockID a public function
* public.go: apiTickers kickoff
* rocksdb_test: fix the unittest comment
* coingecko.go: update comments
* blockbook.go, fiat: reword CoinGecko -> FiatRates, fix binary search upper bound, remove assignment of goroutine call result
* rename coingecko -> fiat_rates
* fiat_rates: export only the necessary methods
* blockbook.go: update log message
* bitcoinrpc.go: remove fiatRates settings
* use CurrencyRatesTicker structure everywhere, fix time format string, update tests, use UTC time
* add /api/v2/tickers tests, store rates as strings (json.Number)
* fiat_rates: add more tests, metrics and tickers-list endpoint, make the "currency" parameter mandatory
* public, worker: move FiatRates API logic to worker.go
* fiat_rates: add a future date test, fix comments, add more checks, store time as a pointer
* rocksdb_test: remove unneeded code
* fiat_rates: add a "ping" call to check server availability
* fiat_rates: do not return empty ticker, return nil instead if not found
add a test for non-existent ticker
* rocksdb_test: remove Sleep from tests
* worker.go: do not propagate all API errors to the client
* move InitTestFiatRates from rocksdb.go to public_test.go
* public.go: fix FiatRatesFindLastTicker result check
* fiat_rates: mock API server responses
* remove commented-out code
* fiat_rates: add comment explaining what periodSeconds attribute is used for
* websocket.go: implement fiatRates websocket endpoints & add tests
* fiatRates: add getFiatRatesTickersList websocket endpoint & test
* fiatRates: make websocket getFiatRatesByDate accept an array of dates, add more tests
* fiatRates: remove getFiatRatesForBlockID from websocket endpoints
* fiatRates: remove "if test", use custom startTime instead
Update tests and mock data
* fiatRates: finalize websocket functionality
add "date" parameter to TickerList
return data timestamps where needed
fix sync bugs (nil timestamp, duplicate save)
* fiatRates: add FiatRates configs for different coins
* worker.go: make GetBlockInfoFromBlockID private again
* fiatRates: wait & retry on errors, remove Ping function
* websocket.go: remove incorrect comment
* fiatRates: move coingecko-related code to a separate file, use interface
* fiatRates: if the new rates are the same as previous, try five more times, and only then store them
* coingecko: fix getting actual rates, add a timestamp parameter to get uncached responses
* vertcoin_testnet.json: remove fiat rates parameters
* fiat_rates: add timestamp to log message about skipping the repeating rates
2019-12-17 02:40:02 -07:00
|
|
|
} `json:"market_data"`
|
|
|
|
}
|
|
|
|
|
|
|
|
var data FiatRatesResponse
|
|
|
|
err = json.Unmarshal(bodyBytes, &data)
|
|
|
|
if err != nil {
|
|
|
|
glog.Errorf("Error parsing FiatRates response: %v", err)
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
ticker.Rates = data.MarketData.Prices
|
|
|
|
return ticker, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// MarketDataExists checks if there's data available for the specific timestamp.
|
|
|
|
func (cg *Coingecko) marketDataExists(timestamp *time.Time) (bool, error) {
|
|
|
|
resp, err := cg.makeRequest(timestamp)
|
|
|
|
if err != nil {
|
|
|
|
glog.Error("Error getting market data: ", err)
|
|
|
|
return false, err
|
|
|
|
}
|
|
|
|
type FiatRatesResponse struct {
|
|
|
|
MarketData struct {
|
|
|
|
Prices map[string]interface{} `json:"current_price"`
|
|
|
|
} `json:"market_data"`
|
|
|
|
}
|
|
|
|
var data FiatRatesResponse
|
|
|
|
err = json.Unmarshal(resp, &data)
|
|
|
|
if err != nil {
|
|
|
|
glog.Errorf("Error parsing Coingecko response: %v", err)
|
|
|
|
return false, err
|
|
|
|
}
|
|
|
|
return len(data.MarketData.Prices) != 0, nil
|
|
|
|
}
|