galmon/README.md

# galmon
galileo/GPS/GLONASS/BeiDou open source monitoring. GPL3 licensed. 
(C) AHU Holding BV - bert@hubertnet.nl - https://berthub.eu/

Live website: https://galmon.eu/

Multi-vendor, with support for U-blox 8 and 9 chipsets and many Septentrio
devices.  Navilock NL-8012U receiver works really well, as does the U-blox
evaluation kit for the 8MT.  In addition, many stations have reported
success with this very cheap [AliExpress sourced
device](https://www.aliexpress.com/item/32816656706.html).  

For ublox, there is good support for the F9P, several of us use the
[ArdusimpleRTK2B](https://www.ardusimple.com/simplertk2b/) board. It adds
the Galileo E5b band.

Septentrio devices support even more bands.

An annotated presentation about our project aimed at GNSS professionals can
be found [here](https://berthub.eu/galileo/The%20galmon.eu%20project.pdf). 

> NOTE: One of our programs is called 'ubxtool'. Sadly, we did not do our
> research, and there is another '[ubxtool](https://gpsd.io/ubxtool.html)' already, part of
> [gpsd](https://gpsd.io). You might have ended up on our page by mistake.
> Sorry!

To deliver data to the project, please read
[The Galmon GNSS Monitoring Project](https://berthub.eu/articles/posts/galmon-project/)
and consult the rules outlined in [the operator
guidelines](https://github.com/ahupowerdns/galmon/blob/master/Operator.md).

Highlights
----------

 * Support for Septentrio and U-blox.
 * Processes raw frames/strings/words from GPS, GLONASS, BeiDou and Galileo
 * All-band support (E1, E5a, E5b, B1I, B2I, Glonass L1, Glonass L2, GPS L1C/A)
   so far.
 * Calculate ephemeris positions
 * Comparison of ephemerides to independent SP3 data to determine SISE
   * Globally, locally, worst user location
 * Record discontinuities between subsequent ephemerides (in time and space)
 * Compare doppler shift as reported by receiver with that expected from ephemeris
 * Track atomic clock & report jumps
 * Coverage maps (number of satellites >5, >10, >20 elevation)
 * HDOP/VDOP/PDOP maps
 * Compare orbit to TLE, match up to best matching satellite
 * Tear out every bit that tells us how well an SV is doing
 * Full almanac processing to see what _should_ be transmitting
 * Distributed receivers, combined into a single source of all messages
 * Ready to detect/report spoofing/jamming

Data is made available as JSON, as a user-friendly website and as a
time-series database. This time-series database is easily mated to the
industry standard Matplotlib/Pandas/Jupyter combination (details 
[here](https://github.com/ahupowerdns/galmon/blob/master/influxdb.md).

There is also tooling to extract raw frames/strings/words from specific
timeframes.

Goals:

1) Support multiple wildly distributed receivers
2) Combine these into a forensic archive of all Galileo/GPS NAV messages
3) Make this archive available, offline and as a stream
4) Consume this stream and turn it into an attractive live website
   (https://galmon.eu/). As part of this, perform higher-level calculations
   to determine ephemeris discontinuities, live gst/gps/galileo time
   offsets, atomic clock jumps etc.
5) Populate an InfluxDB timeseries database with raw measurements and higher
   order calculations

Works on Linux (including Raspbian Buster on Pi Zero W), OSX and OpenBSD.

Build locally (Linux, Debian, Ubuntu)
-------------------------------------

To get started, make sure you have a C++17 compiler (like g++ 8 or higher),
git, protobuf-compiler.  Then run 'make ubxtool navdump' to build the
receiver-only tools.

To build everything, including the webserver, try:

```
apt-get install protobuf-compiler libh2o-dev libcurl4-openssl-dev libssl-dev libprotobuf-dev \
libh2o-evloop-dev libwslay-dev libncurses5-dev libeigen3-dev libzstd-dev g++
git clone https://github.com/ahupowerdns/galmon.git --recursive
cd galmon
make
```

If this doesn't succeed with an error about h2o, make sure you have this
library installed. If you get an error about 'wslay', do the following, and run make again:

```
echo WSLAY=-lwslay > Makefile.local
```

Building on OSX
---------------
With thanks to a contributor from Prague.  First make sure you've installed
brew, which you can get [here](https://brew.sh/). Then do:

```
brew install protobuf lzlib zstd h2o eigen
```

And then:
```
git clone https://github.com/ahupowerdns/galmon.git --recursive
cd galmon
make
```

Running in Docker
-----------------

We publish official Docker images for galmon on
[docker hub](https://hub.docker.com/r/berthubert/galmon) for multiple architectures.

To run a container with a shell in there (this will also expose a port so you
can view the UI too and assumes a ublox GPS device too -
you may need to tweak as necessary):

```
docker run -it --rm --device=/dev/ttyACM0 -p 10000:10000 berthubert/galmon
```

Running a daemonized docker container reporting data to a remote server
might look like:

```
docker run -d --restart=always --device=/dev/ttyACM0 --name=galmon berthubert/galmon ubxtool --wait --port /dev/ttyACM0 --gps --galileo --glonass --destination [server] --station [station-id] --owner [owner]
```

To make your docker container update automatically you could use a tool such as
[watchtower](https://containrrr.github.io/watchtower/).

Running
-------
On u-blox: 
Once compiled, run for example `./ubxtool --wait --port /dev/ttyACM0
--station 1 --stdout --galileo | ./navparse --bind [::1]:10000`

For Septentrio, try: `nc 192.168.1.1 29000 | ./septool --station x --stdout |
./navparse --bind [::1]:10000`, assuming your Septentrio can be reached on
192.168.1.1.1 and you have defined an SBF stream on port 29000. For more
details, please see below.

Next up, browse to http://[::1]:10000 (or try http://localhost:10000/ and
you should be in business. ubxtool changes (non-permanently) the
configuration of your u-blox receiver so it emits the required frames for
GPS and Galileo. If you have a u-blox timing receiver it will also enable
the doppler frames.

By default the ublox receiver module will be configured to use the USB port,
if you want to use a different interface port on the ublox module then add
the `--ubxport <id>` option using one of the following numeric IDs:

   0 : DDC (aka. I2C)  
   1 : UART[1]  
   2 : UART2  
   3 : USB (default)  
   4 : SPI  

To see what is going on, try:

```
./ubxtool --wait --port /dev/ttyACM0 --station 1 --stdout --galileo | ./navdump
```

To distribute data to a remote `navrecv`, use:

```
./ubxtool --wait --port /dev/ttyACM0 --galileo --station 255 --destination 127.0.0.1
```

This will send protobuf to 127.0.0.1:29603. You can add as many destinations
as you want, they will buffer and automatically reconnect. To also send data
to stdout, add `--stdout`.

Tooling:

 * ubxtool: can configure a u-blox 8 chipset, parses its output & will
   convert it into a protbuf stream of GNSS NAV frames + metadata
   Adds 64-bit timestamps plus origin information to each message
 * septool: ingests the Septentrio binary format (SBF) and converts it to our
   protobuf format. Supports same protocol as ubxtool.
 * rtcmtool: ingest ntripclient output, decodes RTCM messages and converts
   them to our protobuf format
 * xtool: if you have another chipset, build something that extracts NAV
   frames & metadata. Not done yet.
 * navrecv: receives GNSS NAV frames and stores them on disk, split out per
   sender. 
 * navnexus: tails the files stored by navrecv, makes them available over
   TCP
 * navparse: consumes these ordered nav updates for a nice website
   and puts "ready to graph" data in influxdb - this is the first
   step that breaks "store everything in native format". Also does
   computations on ephemerides. 
 * grafana dashboard: makes pretty graphs

Linux Systemd
-------------
First make sure 'ubxtool' has been compiled (run: make ubxtool). Then, as
root:
```
mkdir /usr/local/ubxtool
cp ubxtool ubxtool.sh /usr/local/ubxtool/
cp ubxtool.service /etc/systemd/system/
```

Then please reach out as indicated in [Operator.md](Operator.md) to obtain your
station ID and the receiver hostname and run:

```
echo RECEIVER-NAME > /usr/local/ubxtool/destination
echo STATION-NUMBER > /usr/local/ubxtool/station
```

Then start up the service (as root):
```
systemctl enable ubxtool
systemctl start ubxtool
```

To check if it is all working, do 'service ubxtool status'.

> NOTE!  If you don't use one of the AliExpress or Navilock devices, it may
> be that your U-blox is not connected to the USB-port of the U-blox chip
> but to the UART1 or UART2 port.  If so, you'll need to edit the script so
> it finds your USB-to-serial adapter.  At the very least you'll have to
> update the DEVICE line.  You'll likely also have to add --ubxport 1 at the
> end, and likely also the baudrate (-b) and/or --rtscts=0.

To change the default constellations, create a file called
/usr/local/ubxtool/constellations and set your favorites. To set all four
constellations (which only F9-receivers support), do as root:

```
echo --gps --glonass --beidou --galileo > /usr/local/ubxtool/constellations
```

And then 'service ubxtool restart'.

Distributed setup
-----------------
Run `navrecv -b :: --storage ./storage` to receive frames on port 29603 of
::, aka all your IPv6 addresses (and IPv4 too on Linux).  This allows anyone
to send you frames, so be aware.

Next up, run `navnexus --storage ./storage -b ::`, which will serve your
recorded data from port 29601.  It will merge messages coming in from all
sources and serve them in time order.

Finally, you can do `nc 127.0.0.1 29601 | ./navdump`, which will give you all messages over the past 24 hours, and stream you more.
This also works for `navparse` for the pretty website and influx storage, `nc 127.0.0.1 29601 | ./navparse --influxdb=galileo`,
if you have an influxdb running on localhost with a galileo database in there.
The default URL is http://127.0.0.1:29599/ 

Septentrio specifics
--------------------
Unlike `ubxtool`, our `septool` does not (re)configure your Septentrio
device. Instead, the tool expects Septentrio Binary Format (SBF) on input,
and that this stream includes at least the following messages:

 * MeasEpoch
 * PVTCartesian

We currently parse and understand:
 * GALRawFNAV
 * GALRawINAV

Support will be added soon for:
 * GPSRawCA
 * GPSRawL2C
 * GPSRawL5
 * GLORawCA
 * BDSRaw
 * BDSRawB1C
 * BDSRawB2a

A typical invocation of `septool` looks like this:

```
nc 192.168.1.1 29000 | ./septool --station x --destination galmon-eu-server.example.com
```

Or to test, try:

```
nc 192.168.1.1 29000 | ./septool --station x --stdout | ./navdump
```

This is assuming that you can reach your Septentrio on 192.168.1.1 and that
you have defined a TCP server stream on port 29000. 

Septool will also accept input from a serial port or basically anything that
can provide SBF. Please let us know if our tooling can make your life
easier.

Internals
---------
The transport format consists of repeats of:

1) Four byte magic value
2) Two-byte frame length
3) A protobuf frame

The magic value is there to help us resync from partially written data.

The whole goal is that we can continue to rebuild the database by 
rerunning 'navstore' and 'navinflux'.

Documents
---------

 * [BeiDou](http://m.beidou.gov.cn/xt/gfxz/201902/P020190227593621142475.pdf)
 * [Galileo](https://www.gsc-europa.eu/sites/default/files/sites/all/files/Galileo-OS-SIS-ICD.pdf)
 * [GLONASS](https://www.unavco.org/help/glossary/docs/ICD_GLONASS_4.0_(1998)_en.pdf),
    old 1998 version, but unlike newer versions, this one is not full of
    mistakes. [New version](http://gauss.gge.unb.ca/GLONASS.ICD.pdf) is more complete but is worryingly messy.
 * [GLONASS CDMA](http://russianspacesystems.ru/wp-content/uploads/2016/08/ICD-GLONASS-CDMA-General.-Edition-1.0-2016.pdf)
   not actually relevant for the CDMA aspects, but has appendices on more
   precise orbit determinations.
 * [GPS](https://www.gps.gov/technical/icwg/IS-GPS-200K.pdf)
 * [U-blox 8 interface specification](https://www.u-blox.com/sites/default/files/products/documents/u-blox8-M8_ReceiverDescrProtSpec_%28UBX-13003221%29_Public.pdf)
 * [U-blox 9 interface specification](https://www.u-blox.com/sites/default/files/u-blox_ZED-F9P_InterfaceDescription_%28UBX-18010854%29.pdf)
 * [U-blox 9 integration manual](https://www.u-blox.com/sites/default/files/ZED-F9P_IntegrationManual_%28UBX-18010802%29.pdf)

Data sources
------------
The software can interpret SP3 files, good sources:

 * ESA/ESOC: http://navigation-office.esa.int/products/gnss-products/ - pick the
   relevant GPS week number, and then a series (.sp3 extension):
   * ESU = ultra rapid, 2-8h delay, only GPS and GLONASS
   * ESR = rapid, 2-26h delay, only GPS and GLONASS
   * ESM = finals, 6-13d delay, GPS, GLONASS, Galileo, BeiDou, QZSS
   * File format is esXWWWWD.sp3 - where X is U, R or M, WWWW is the
   (non-wrapping) GPS week number and D is day of week, Sunday is 0.
   * Further description: http://navigation-office.esa.int/GNSS_based_products.html
 * GFZ Potsdam: ftp://ftp.gfz-potsdam.de/GNSS/products/mgnss
   * The GBM series covers GPS, GLONASS, Galileo, BeiDou, QZSS and appears
     to have less of a delay than the ESA ESM series.
   * GBU = ultra rapid, still a few days delay, but much more recent.

To get SP3 GBM from GFZ Potsdam for GPS week number 2111:

```
WN=2111
lftp -c "mget ftp://ftp.gfz-potsdam.de/GNSS/products/mgnss/${WN}/gbm*sp3.Z"
gunzip gbm*sp3.Z
```

To feed data, use: 

```
./sp3feed --sp3src=gbm --influxdb=galileo gbm*sp3
```

This will populate the sp3 tables in the database. A subsequent run of
`reporter`, while setting the `--sp3src` parameter, will provide SP3
deviation statistics, and fill out the `sp3delta` table in there, which
stores deviations from the SP3 provided position, per sp3src.

Further interesting (ephemeris) data is on http://mgex.igs.org/IGS_MGEX_Products.php

RTCM
----
RTCM is the Radio Technical Commission for Maritime Services, and
confusingly, also the name of a protocol. 

This protocol is proprietary, but search for a file called `RTCM3.2.pdf` or
`104-2013-SC104-STD - Vers. 3.2.docx` and you might find a copy. 

This project can parse RTCM 10403.1 messages, and currently processes State
Space Representation (SSR) messages, specifically types 1057/1240
(GPS/Galileo Orbit corrections to broadcast ephemeris) and 1058/1241
(GPS/Galileo Clock corrections to broadcast ephemeris).

RTCM messages need to be converted to protobuf format, and the `rtcmtool` is
provided for this purpose.

RTCM is frequently transmitted over the internet using 'ntrip', a typical
commandline to process RTCM in our project is:
```
$ ntripclient ntrip:CLKA0_DEU1/user:password@navcast.spaceopal.com:2101 | ./rtcmtool --station x --destination y
```

User and password can be obtained from https://spaceopal.com/navcast/ - the
Galileo operating company. 

The IGS also offers excellent streams, but without Galileo. Information is
[here](http://www.igs.org/rts/products). A typical commandline is:

```
$ ntripclient ntrip:IGS01/user:password@products.igs-ip.net:2101 | ./rtcmtool --station x --destination y
```

User and password can be requested through http://www.igs.org/rts/access

An interesting list is here: http://products.igs-ip.net/

There are many other sources of RTCM but currently not many offer the SSR
messages we can use.

Tooling
-------

 * ubxtool: Configure and use a Ublox chip to gather messages, and send as
   protobuf to standard output or a remote server (with buffering).
 * navdump: convert protobuf format data into a textual display of messages
 * navparse: consume protobuf and turn into a webserver with data, plus
   optionally fill an influxdb time-series database for graphing and analysis
   purposes.
 * navrecv: receive protobuf messages over the network and store them on
   disk
 * navnexus: serve protobuf messages from disk over the network
 * navcat: serve protobuf messages from disk directly to stdout
 * reporter: make "the galmon.eu weekly galileo report"
 * rinreport: rinex analysis tooling (not generically useful yet)
 * galmonmon: monitor a navparse instance for changes, tweet them out
 * navdisplay: some eye-candy that converts protobuf into a live display
   (not very good)
 * rtcmtool: accepts RTCM messages on standard input (for example coming
   from ntripclient) and transmits them as protobuf messages, either to
   stdout or to a navrecv server. This is the equivalent of 'ubxtool'
   except for submitting RTCM messages. 

Sample command lines
--------------------
Look at old data:

```
$ ./navcat storage "2020-01-01 00:00" "2020-01-02 00:00" | ./navdump  
```

Global coverage (via volunteers)
--------------------------------

In alphabetical order:

 * Austria (Vienna area)
 * Brazil
 * Holland (Nootdorp, Hilversum, etc)
 * India (New Delhi area)
 * Israel (Jerusalem)
 * Italy (Rome)
 * New Zealand (Auckland area)
 * Rusia (Moscow area)
 * Singapore
 * South Africa (Cape Town area)
 * Spain
 * Tonga 
 * USA
   * Alaska (Anchorage)
   * California (Santa Cruz, Los Angeles area, etc)
   * Massachusetts (Boston area)
 * Uruguay
 
Additional sites are welcome (and encouraged) as the more data receiving sites that exist, then more accurate data and absolute coverage of each constellation can be had.

The galmon project is very grateful to all its volunteering receiving stations.

ubxtool
-------
 * Will also spool raw serial data to disk (in a filename that includes the
   start date)
 * Can also read from disk
 * Careful to add the right timestamps