Merge pull request #25 from kerel-fs/idea_dump
WIP: Add SatNOGS Artifact Analysis Toolspull/27/head
commit
badca9ac4f
|
@ -25,3 +25,6 @@
|
||||||
/rfinfo
|
/rfinfo
|
||||||
/rfplot
|
/rfplot
|
||||||
/rfpng
|
/rfpng
|
||||||
|
/rfinfo
|
||||||
|
|
||||||
|
.env
|
||||||
|
|
|
@ -0,0 +1,73 @@
|
||||||
|
# Guide for RF doppler analysis from SatNOGS Observation Waterfall Images
|
||||||
|
|
||||||
|
Forum post: https://community.libre.space/t/new-software-satnogs-waterfall-tabulation-helper/4380
|
||||||
|
|
||||||
|
This tool evolved from the need to tabulate doppler data from the lunar
|
||||||
|
Change-4 probe,
|
||||||
|
see <https://gitlab.com/kerel-fs/jupyter-notebooks/-/tree/master/change4/data#data-tabulation-method-new>.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
Install dependencies
|
||||||
|
```
|
||||||
|
pip install -r contrib/requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
# First Setup
|
||||||
|
Choose a folder where TLEs are stored and
|
||||||
|
a folder where RF doppler observatons (`.dat`-files) are stored.
|
||||||
|
|
||||||
|
For now the paths are configured via environment variables,
|
||||||
|
so make sure to set them correctly before each usage.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
# Filename convention: {TLE_DIR}/{observation_id}.txt
|
||||||
|
SATNOGS_TLE_DIR="./data/tles"
|
||||||
|
|
||||||
|
# absulute frequency measurement storage
|
||||||
|
# Filename convention: {DOPPLER_OBS_DIR}/{observation_id}.txt
|
||||||
|
SATNOGS_DOPPLER_OBS_DIR="./data/obs"
|
||||||
|
|
||||||
|
# SATTOOLS/STRF/STVID sites.txt file
|
||||||
|
SATNOGS_SITES_TXT="./data/sites.txt"
|
||||||
|
|
||||||
|
mkdir -p $SATNOGS_TLE_DIR
|
||||||
|
mkdir -p $SATNOGS_DOPPLER_OBS_DIR
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
0. Make sure the (3) env variables are set.
|
||||||
|
1. Choose SatNOGS Observation ID from network and run the tabulation helper
|
||||||
|
|
||||||
|
```
|
||||||
|
./contrib/satnogs_waterfall_tabulation_helper.py 1102230
|
||||||
|
```
|
||||||
|
|
||||||
|
An interactive plot will show up.
|
||||||
|
Clicking inside the plot will add a signal marker.
|
||||||
|
If you are finished with adding signal markers,
|
||||||
|
save the signal markers using the keyboard shortcut `f`.
|
||||||
|
|
||||||
|
Custom keyboard shortcuts:
|
||||||
|
|
||||||
|
- u - undo last signal marker
|
||||||
|
- f - save the signal markers in an strf-compatible file
|
||||||
|
|
||||||
|
Useful Matplotlib navigation keyboard shortcuts (documentation):
|
||||||
|
|
||||||
|
p - toggle 'Pan/Zoom' modus
|
||||||
|
o - toggle 'Zoom-to-rect' modus
|
||||||
|
h - Home/Reset (view)
|
||||||
|
c - Back (view)
|
||||||
|
|
||||||
|
2. Run rffit for orbit fitting, e.g.
|
||||||
|
```
|
||||||
|
./rffit -d $SATNOGS_DOPPLER_OBS_DIR/1102230.dat -i 44356 -c $SATNOGS_TLE_DIR/1102230.txt -s 7669
|
||||||
|
```
|
||||||
|
|
||||||
|
## Known issues
|
||||||
|
- A site id of the form `7{station_id}` is automatically assigned and written to
|
||||||
|
the `sites.txt` (e.g. station 669 should get `7669` assigned).
|
||||||
|
Only SatNOGS stations <999 are supported, as the strf sites.txt parse only allows
|
||||||
|
4-digit site ids. In case of problems, choose a free site id and manually correct the
|
||||||
|
doppler obs (`.dat`-files, last colum).
|
|
@ -0,0 +1,65 @@
|
||||||
|
# STRF contrib Scripts
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
- Create Pyhton virtual environment and install dependencies via pip:
|
||||||
|
```
|
||||||
|
mkvirtualenv strf
|
||||||
|
pip insatll -r contrib/requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
- Create initial configuration from `env-dist`
|
||||||
|
```
|
||||||
|
cp env-dist .env
|
||||||
|
```
|
||||||
|
- Configure the data paths (they must already exist!) and add your SatNOGS DB API Token in `.env`
|
||||||
|
|
||||||
|
## Find good observations with SatNOGS artifacts
|
||||||
|
```
|
||||||
|
$ ./contrib/find_good_satnogs_artifacts.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Download SatNOGS Artifacts
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ./contrib/download_satnogs_artifact.py 4950356
|
||||||
|
Artifact Metadata for Observation #4950356 found.
|
||||||
|
Artifact saved in /home/pi/data/artifacts/4950356.h5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Download SatNOGS Observation TLEs
|
||||||
|
|
||||||
|
To add the TLE used in a specific SatNOGS Observation to your catalog, the following command can be used:
|
||||||
|
```
|
||||||
|
$ ./contrib/download_satnogs_tle.py 4950356
|
||||||
|
TLE saved in /home/pi/data/tles/satnogs/4950356.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## Plot SatNOGS Artifacts
|
||||||
|
|
||||||
|
This can be done using [spectranalysis](https://github.com/kerel-fs/spectranalysis).
|
||||||
|
|
||||||
|
## Ultra-Short Analysis Guide
|
||||||
|
|
||||||
|
```
|
||||||
|
source .env
|
||||||
|
|
||||||
|
OBSERVATION_ID=4950356
|
||||||
|
./contrib/download_satnogs_tle.py $OBSERVATION_ID
|
||||||
|
./contrib/download_satnogs_artifact.py $OBSERVATION_ID
|
||||||
|
./contrib/analyze_artifact.py --observation_id $OBSERVATION_ID --site_id 977
|
||||||
|
rffit -d "$SATNOGS_DOPPLER_OBS_DIR/$OBSERVATION_ID.dat" -c "$SATNOGS_TLE_DIR/$OBSERVATION_ID.txt" -i 27844 -s 977
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ./contrib/download_satnogs_tle.py 4950356
|
||||||
|
TLE saved in /mnt/old_home/kerel/c4/satnogs/data/tles/satnogs/4950356.txt
|
||||||
|
$ ./contrib/download_satnogs_artifact.py 4950356
|
||||||
|
Artifact Metadata for Observation #4950356 found.
|
||||||
|
Artifact saved in /mnt/old_home/kerel/c4/satnogs/data/artifacts/4950356.h5
|
||||||
|
$ ./contrib/analyze_artifact.py --observation_id 4950356 --site_id 977
|
||||||
|
Load /mnt/old_home/kerel/c4/satnogs/data/artifacts/4950356.h5
|
||||||
|
Extract measurements...
|
||||||
|
Data written in /mnt/old_home/kerel/c4/satnogs/data/doppler_obs/4950356.dat
|
||||||
|
$ rffit -d ${SATNOGS_DOPPLER_OBS_DIR}/4950356.dat -c ${SATNOGS_TLE_DIR}/4950356.txt -i 27844 -s 977
|
||||||
|
```
|
|
@ -0,0 +1,170 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
from astropy.time import Time
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import ephem
|
||||||
|
import h5py
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
|
||||||
|
def load_artifact(filename):
|
||||||
|
hdf5_file = h5py.File(filename, 'r')
|
||||||
|
if hdf5_file.attrs['artifact_version'] != 2:
|
||||||
|
print("unsupported artifact version {}".format(hdf5_file.attrs['artifact_version']))
|
||||||
|
# return
|
||||||
|
wf = hdf5_file.get('waterfall')
|
||||||
|
data = (np.array(wf['data']) * np.array(wf['scale']) + np.array(wf['offset']))
|
||||||
|
metadata = json.loads(hdf5_file.attrs['metadata'])
|
||||||
|
|
||||||
|
return hdf5_file, wf, data, metadata
|
||||||
|
|
||||||
|
def extract_peaks(data):
|
||||||
|
"""
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
points: (time_index, freq_index)
|
||||||
|
measurements: measurement tuples of relative time in seconds and relative frequency in hertz
|
||||||
|
"""
|
||||||
|
|
||||||
|
snr = (np.max(data, axis=1) - np.mean(data, axis=1)) / np.std(data, axis=1)
|
||||||
|
|
||||||
|
# Integrate each channel along the whole observation,
|
||||||
|
# This filter helps under the assumption of minimal Doppler deviaion
|
||||||
|
snr_integrated = np.zeros(data[0].shape)
|
||||||
|
for row in data:
|
||||||
|
snr_integrated += (row - np.mean(row)) / np.std(row)
|
||||||
|
snr_integrated = pd.Series(snr_integrated/len(snr_integrated))
|
||||||
|
|
||||||
|
SNR_CUTOFF = 4
|
||||||
|
CHANNEL_WINDOW_SIZE = 4
|
||||||
|
channel_mask = (snr_integrated.diff().abs() / snr_integrated.diff().std() > SNR_CUTOFF).rolling(CHANNEL_WINDOW_SIZE, min_periods=1).max()
|
||||||
|
rows=[]
|
||||||
|
for i, row in enumerate(channel_mask):
|
||||||
|
if not row:
|
||||||
|
continue
|
||||||
|
rows.append(i)
|
||||||
|
|
||||||
|
points_all = []
|
||||||
|
points_filtered = []
|
||||||
|
for i,x in enumerate(np.argmax(data, axis=1)):
|
||||||
|
points_all.append([i, x])
|
||||||
|
if x in rows:
|
||||||
|
points_filtered.append([i, x])
|
||||||
|
|
||||||
|
points_all = np.array(points_all)
|
||||||
|
points_filtered = np.array(points_filtered)
|
||||||
|
|
||||||
|
if len(points_filtered) == 0:
|
||||||
|
print("WARNING: No measurement passed filter, loosening requirements now.")
|
||||||
|
points_filtered = points_all
|
||||||
|
measurements = None
|
||||||
|
|
||||||
|
measurements = np.vstack((wf['relative_time'][points_filtered[:,0]],
|
||||||
|
[wf['frequency'][x] for x in points_filtered[:,1]]))
|
||||||
|
return snr, measurements, points_filtered
|
||||||
|
|
||||||
|
def plot_measurements_all(wf, data):
|
||||||
|
plt.plot(wf['relative_time'][:],
|
||||||
|
[wf['frequency'][x] for x in np.argmax(data[:], axis=1)],
|
||||||
|
'.',
|
||||||
|
label='all')
|
||||||
|
|
||||||
|
def plot_measurements_selected(measurements):
|
||||||
|
plt.plot(measurements[0,:],
|
||||||
|
measurements[1,:],
|
||||||
|
'.',
|
||||||
|
label='filtered')
|
||||||
|
|
||||||
|
def plot_legend(observation_id):
|
||||||
|
plt.legend()
|
||||||
|
plt.title("Observation #{} - Maximum Values".format(observation_id))
|
||||||
|
plt.grid()
|
||||||
|
plt.xlabel('Elapsed Time / s')
|
||||||
|
plt.ylabel('rel. Frequency / kHz')
|
||||||
|
plt.show()
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def dedoppler(measurements, metadata):
|
||||||
|
tle = metadata['tle'].split('\n')
|
||||||
|
start_time = datetime.strptime(wf.attrs['start_time'].decode('ascii'),
|
||||||
|
'%Y-%m-%dT%H:%M:%S.%fZ')
|
||||||
|
f_center = float(metadata['frequency'])
|
||||||
|
|
||||||
|
# Initialize SGP4 propagator / pyephem
|
||||||
|
satellite = ephem.readtle('sat', tle[1], tle[2])
|
||||||
|
observer = ephem.Observer()
|
||||||
|
observer.lat = str(metadata['location']['latitude'])
|
||||||
|
observer.lon = str(metadata['location']['longitude'])
|
||||||
|
observer.elevation = metadata['location']['altitude']
|
||||||
|
|
||||||
|
def remove_doppler_correction(t, freq):
|
||||||
|
"""
|
||||||
|
Arguments
|
||||||
|
---------
|
||||||
|
t - float: Time in seconds
|
||||||
|
freq - float: Relative Frequency in herz
|
||||||
|
"""
|
||||||
|
observer.date = t
|
||||||
|
satellite.compute(observer)
|
||||||
|
v = satellite.range_velocity
|
||||||
|
df = f_center * v / ephem.c
|
||||||
|
return f_center + freq - df*2
|
||||||
|
|
||||||
|
output = []
|
||||||
|
for dt,df in measurements.T:
|
||||||
|
t = start_time + timedelta(seconds=dt)
|
||||||
|
f = f_center + df
|
||||||
|
freq_recv = remove_doppler_correction(t, f)
|
||||||
|
output.append((t, freq_recv))
|
||||||
|
return output
|
||||||
|
|
||||||
|
def save_rffit_data(filename, measurements, site_id):
|
||||||
|
with open(filename, 'w') as file_out:
|
||||||
|
for time, freq in measurements:
|
||||||
|
line = '{:.6f}\t{:.2f}\t1.0\t{}\n'.format(Time(time).mjd, freq, site_id)
|
||||||
|
file_out.write(line)
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
parser = argparse.ArgumentParser(description='Analyze SatNOGS Artifact.')
|
||||||
|
parser.add_argument('--observation_id', type=str,
|
||||||
|
help='SatNOGS Observation ID')
|
||||||
|
parser.add_argument('--filename', type=str,
|
||||||
|
help='SatNOGS Artifacts File')
|
||||||
|
parser.add_argument('--output_file', type=str,
|
||||||
|
help='STRF-compatible output file')
|
||||||
|
parser.add_argument('--site_id', type=int, required=True,
|
||||||
|
help='STRF Site ID')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if args.observation_id:
|
||||||
|
# Use canonic file paths, ignoring `--filename` and `--output_path`
|
||||||
|
filename = '{}/{}.h5'.format(os.getenv('SATNOGS_ARTIFACTS_DIR'), args.observation_id)
|
||||||
|
output_file = '{}/{}.dat'.format(os.getenv('SATNOGS_DOPPLER_OBS_DIR'), args.observation_id)
|
||||||
|
else:
|
||||||
|
if not any([args.filename, args.output_file]):
|
||||||
|
print('ERROR: Missing arguments')
|
||||||
|
filename = args.filename
|
||||||
|
output_file = args.output_file
|
||||||
|
|
||||||
|
print('Load {}'.format(filename))
|
||||||
|
hdf5_file, wf, data, metadata = load_artifact(filename)
|
||||||
|
|
||||||
|
print('Extract measurements...')
|
||||||
|
snr, measurements, points = extract_peaks(data)
|
||||||
|
plot_measurements_all(wf, data)
|
||||||
|
plot_measurements_selected(measurements)
|
||||||
|
plot_legend(args.observation_id)
|
||||||
|
|
||||||
|
m2 = dedoppler(measurements, metadata)
|
||||||
|
save_rffit_data(output_file, m2, site_id=args.site_id)
|
||||||
|
print('Data written in {}'.format(output_file))
|
|
@ -0,0 +1,85 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import logging
|
||||||
|
import requests
|
||||||
|
import settings
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import os
|
||||||
|
|
||||||
|
from urllib.parse import urljoin
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
def fetch_artifact_metadata(network_obs_id):
|
||||||
|
url = urljoin(settings.SATNOGS_DB_API_URL, 'artifacts/',)
|
||||||
|
params = {'network_obs_id': network_obs_id}
|
||||||
|
headers = {'Authorization': 'Token {0}'.format(settings.SATNOGS_DB_API_TOKEN)}
|
||||||
|
|
||||||
|
response = requests.get(url,
|
||||||
|
params=params,
|
||||||
|
headers=headers,
|
||||||
|
timeout=10)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
return response.json()
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_artifact(url, artifact_filename):
|
||||||
|
headers = {'Authorization': 'Token {0}'.format(settings.SATNOGS_DB_API_TOKEN)}
|
||||||
|
|
||||||
|
response = requests.get(url,
|
||||||
|
headers=headers,
|
||||||
|
stream=True,
|
||||||
|
timeout=10)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
with open(artifact_filename, 'wb') as fname:
|
||||||
|
for chunk in response.iter_content(chunk_size=1024):
|
||||||
|
fname.write(chunk)
|
||||||
|
|
||||||
|
|
||||||
|
def download_artifact(observation_id, filename):
|
||||||
|
try:
|
||||||
|
artifact_metadata = fetch_artifact_metadata(network_obs_id=observation_id)
|
||||||
|
except requests.HTTPError:
|
||||||
|
print('An error occurred trying to GET artifact metadata from db')
|
||||||
|
return
|
||||||
|
|
||||||
|
if not len(artifact_metadata):
|
||||||
|
print('No artifact found in db for network_obs_id {}'.format(network_obs_id))
|
||||||
|
return
|
||||||
|
|
||||||
|
print("Artifact Metadata for Observation #{} found.".format(observation_id))
|
||||||
|
|
||||||
|
try:
|
||||||
|
artifact_file_url = artifact_metadata[0]['artifact_file']
|
||||||
|
artifact_file = tempfile.NamedTemporaryFile(delete=False)
|
||||||
|
|
||||||
|
fetch_artifact(artifact_file_url, artifact_file.name)
|
||||||
|
|
||||||
|
artifact_file.close()
|
||||||
|
shutil.copy(artifact_file.name, filename)
|
||||||
|
os.remove(artifact_file.name)
|
||||||
|
print("Artifact saved in {}".format(filename))
|
||||||
|
except requests.HTTPError:
|
||||||
|
print('Download failed for {}'.format(artifact_file_url))
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
parser = argparse.ArgumentParser(description='Download SatNOGS Artifacts from SatNOGS DB.')
|
||||||
|
parser.add_argument('observation_ids', metavar='ID', type=int, nargs='+',
|
||||||
|
help='SatNOGS Observation ID')
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
logging.basicConfig(level=logging.INFO)
|
||||||
|
|
||||||
|
for observation_id in args.observation_ids:
|
||||||
|
download_artifact(observation_id,
|
||||||
|
'{}/{}.h5'.format(os.getenv('SATNOGS_ARTIFACTS_DIR'), observation_id))
|
|
@ -0,0 +1,26 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
|
||||||
|
from satnogs_api_client import fetch_observation_data
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
parser = argparse.ArgumentParser(description='Download TLE from a specific Observation in SatNOGS Network.')
|
||||||
|
parser.add_argument('observation_id', type=int,
|
||||||
|
help='SatNOGS Observation ID')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
obs = fetch_observation_data([args.observation_id])[0]
|
||||||
|
|
||||||
|
filename = '{}/{}.txt'.format(os.getenv('SATNOGS_TLE_DIR'), args.observation_id)
|
||||||
|
|
||||||
|
with open(filename, 'w') as f:
|
||||||
|
f.write(obs['tle0'])
|
||||||
|
f.write('\n')
|
||||||
|
f.write(obs['tle1'])
|
||||||
|
f.write('\n')
|
||||||
|
f.write(obs['tle2'])
|
||||||
|
f.write('\n')
|
||||||
|
print("TLE saved in {}".format(filename))
|
|
@ -0,0 +1,54 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import logging
|
||||||
|
import requests
|
||||||
|
import settings
|
||||||
|
|
||||||
|
from urllib.parse import urljoin
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
from satnogs_api_client import fetch_observation_data, fetch_tle_of_observation
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_latest_artifacts_metadata():
|
||||||
|
url = urljoin(settings.SATNOGS_DB_API_URL, 'artifacts/',)
|
||||||
|
params = {}
|
||||||
|
headers = {'Authorization': 'Token {0}'.format(settings.SATNOGS_DB_API_TOKEN)}
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = requests.get(url,
|
||||||
|
params=params,
|
||||||
|
headers=headers,
|
||||||
|
timeout=10)
|
||||||
|
response.raise_for_status()
|
||||||
|
except (requests.ConnectionError, requests.Timeout, requests.TooManyRedirects):
|
||||||
|
logger.exception('An error occurred trying to GET artifact metadata from db')
|
||||||
|
|
||||||
|
artifacts_metadata = response.json()
|
||||||
|
|
||||||
|
if not len(artifacts_metadata):
|
||||||
|
logger.info('No artifacts found in db')
|
||||||
|
sys.exit(-1)
|
||||||
|
|
||||||
|
return artifacts_metadata
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
logging.basicConfig(level=logging.INFO)
|
||||||
|
|
||||||
|
# Fetch list of artifacts
|
||||||
|
artifacts_metadata = fetch_latest_artifacts_metadata()
|
||||||
|
|
||||||
|
observation_ids = [artifact['network_obs_id'] for artifact in artifacts_metadata]
|
||||||
|
|
||||||
|
# Load corresponding obs from network
|
||||||
|
observations = fetch_observation_data(sorted(observation_ids, reverse=True))
|
||||||
|
|
||||||
|
# Filter by good status
|
||||||
|
for observation in observations:
|
||||||
|
if not observation['vetted_status'] == 'good':
|
||||||
|
pass
|
||||||
|
|
||||||
|
print("{}/observations/{}/".format(settings.SATNOGS_NETWORK_API_URL[:-5], observation['id']))
|
|
@ -0,0 +1,10 @@
|
||||||
|
astropy
|
||||||
|
matplotlib
|
||||||
|
numpy
|
||||||
|
Pillow
|
||||||
|
python-decouple
|
||||||
|
requests
|
||||||
|
git+https://gitlab.com/librespacefoundation/satnogs/python-satnogs-api.git@e20a7d3c
|
||||||
|
h5py~=3.6.0
|
||||||
|
pandas~=1.3.4
|
||||||
|
ephem~=4.3.1
|
|
@ -0,0 +1,213 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import requests
|
||||||
|
import ephem
|
||||||
|
import csv
|
||||||
|
|
||||||
|
|
||||||
|
from astropy.time import Time
|
||||||
|
from astropy import constants as const
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from io import BytesIO
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
import numpy as np
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
import settings
|
||||||
|
from satnogs_api_client import fetch_observation_data, fetch_tle_of_observation
|
||||||
|
|
||||||
|
|
||||||
|
def tabulation_helper_dialog(lower_left, upper_right,
|
||||||
|
bandwidth, duration,
|
||||||
|
f_center, filename_out,
|
||||||
|
waterfall_matrix,
|
||||||
|
epoch_start, site_id, correction_method=None):
|
||||||
|
# Assumptions: x-axis - freqency, y-axis - time; up-time advance, right-freq advance
|
||||||
|
|
||||||
|
# Derive conversion parameters
|
||||||
|
x_range = upper_right[0] - lower_left[0]
|
||||||
|
y_range = lower_left[1] - upper_right[1]
|
||||||
|
x_offset = int(lower_left[0] + 0.5 * x_range)
|
||||||
|
y_offset = lower_left[1]
|
||||||
|
|
||||||
|
t_step = duration / y_range
|
||||||
|
f_step = bandwidth / x_range
|
||||||
|
|
||||||
|
# Highlight center and calibration lines
|
||||||
|
waterfall_matrix[:,x_offset] = (255,0,0,255)
|
||||||
|
waterfall_matrix[y_offset,:] = (255,0,0,255)
|
||||||
|
waterfall_matrix[:,upper_right[0]] = (255,0,0,255)
|
||||||
|
waterfall_matrix[upper_right[1],:] = (255,0,0,255)
|
||||||
|
|
||||||
|
|
||||||
|
markers = {'x': [], 'y': [], 't': [], 'f': []}
|
||||||
|
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
ax.imshow(waterfall_matrix)
|
||||||
|
marker_line, = ax.plot([0], [0], marker='.', c='k', zorder=100)
|
||||||
|
def on_mouse_press(event):
|
||||||
|
tb = plt.get_current_fig_manager().toolbar
|
||||||
|
if not (event.button==1 and event.inaxes and tb.mode == ''):
|
||||||
|
return
|
||||||
|
|
||||||
|
markers['x'].append(event.xdata)
|
||||||
|
markers['y'].append(event.ydata)
|
||||||
|
markers['t'].append(epoch_start - (event.ydata - y_offset) * t_step)
|
||||||
|
markers['f'].append(f_step * (event.xdata - x_offset))
|
||||||
|
print('x:{:.2f} y:{:.2f} --> {:7.0f} --> {} {:7.0f}'.format(markers['x'][-1],
|
||||||
|
markers['y'][-1],
|
||||||
|
markers['f'][-1] - f_center,
|
||||||
|
markers['t'][-1],
|
||||||
|
markers['f'][-1]))
|
||||||
|
update_plot_markers()
|
||||||
|
|
||||||
|
def update_plot_markers():
|
||||||
|
marker_line.set_xdata(markers['x'])
|
||||||
|
marker_line.set_ydata(markers['y'])
|
||||||
|
# ax.scatter(x=[int(event.xdata)], y=[int(event.ydata)], marker='.', c='k', zorder=100)
|
||||||
|
ax.figure.canvas.draw()
|
||||||
|
|
||||||
|
def undo_track_point_selection():
|
||||||
|
markers['x'] = markers['x'][:-1]
|
||||||
|
markers['y'] = markers['y'][:-1]
|
||||||
|
markers['t'] = markers['t'][:-1]
|
||||||
|
markers['f'] = markers['f'][:-1]
|
||||||
|
update_plot_markers()
|
||||||
|
|
||||||
|
def write_strf_spectrum(filename, site_id, markers):
|
||||||
|
with open(filename, 'w') as f:
|
||||||
|
for t,freq in list(sorted(zip(markers['t'],markers['f']), key=lambda x: x[0])):
|
||||||
|
|
||||||
|
if correction_method:
|
||||||
|
freq_recv = correction_method(t, freq)
|
||||||
|
else:
|
||||||
|
freq_recv = freq
|
||||||
|
line = '{:.6f}\t{:.2f}\t1.0\t{}\n'.format(Time(t).mjd, freq_recv, site_id)
|
||||||
|
print(line, end='')
|
||||||
|
f.write(line)
|
||||||
|
|
||||||
|
def save_selected_track_points():
|
||||||
|
write_strf_spectrum(filename_out, site_id, markers)
|
||||||
|
print('Stored {} selected track points in {}.'.format(len(markers['t']), filename_out))
|
||||||
|
|
||||||
|
def on_key(event):
|
||||||
|
if event.key == 'u':
|
||||||
|
undo_track_point_selection()
|
||||||
|
elif event.key == 'f':
|
||||||
|
save_selected_track_points()
|
||||||
|
|
||||||
|
fig.canvas.mpl_connect('button_press_event', on_mouse_press)
|
||||||
|
fig.canvas.mpl_connect('key_press_event', on_key)
|
||||||
|
plt.tight_layout()
|
||||||
|
plt.show()
|
||||||
|
|
||||||
|
|
||||||
|
def read_sitestxt(path):
|
||||||
|
sites = {}
|
||||||
|
with open(path) as fp:
|
||||||
|
d = csv.DictReader([row for row in fp if not row.startswith('#')],
|
||||||
|
delimiter=' ',
|
||||||
|
fieldnames=["no", "id", "lat", "lon", "alt"],
|
||||||
|
restkey="observer",
|
||||||
|
skipinitialspace=True)
|
||||||
|
for line in d:
|
||||||
|
print(line)
|
||||||
|
sites[line["no"]] = {'lat': line["lat"],
|
||||||
|
'lon': line["lon"],
|
||||||
|
'sn': line["id"],
|
||||||
|
'alt': line["alt"],
|
||||||
|
'name': ' '.join(line["observer"])}
|
||||||
|
return sites
|
||||||
|
|
||||||
|
|
||||||
|
def add_station_to_sitestxt(station):
|
||||||
|
# Read the sites.txt
|
||||||
|
sites = read_sitestxt(settings.SITES_TXT)
|
||||||
|
if '7{}'.format(station['id']) in sites:
|
||||||
|
# Station is already available
|
||||||
|
return
|
||||||
|
else:
|
||||||
|
with open(settings.SITES_TXT, 'a') as f:
|
||||||
|
f.write('7{} {} {} {} {} {}\n'.format(station['id'],
|
||||||
|
'SN',
|
||||||
|
station['lat'],
|
||||||
|
station['lng'],
|
||||||
|
station['alt'],
|
||||||
|
station['name']))
|
||||||
|
|
||||||
|
|
||||||
|
def tabulation_helper(observation_id):
|
||||||
|
# Fetch waterfall image and observation data
|
||||||
|
observation = fetch_observation_data([observation_id])[0]
|
||||||
|
tle = fetch_tle_of_observation(observation_id)
|
||||||
|
result = requests.get(observation['waterfall'])
|
||||||
|
image = Image.open(BytesIO(result.content))
|
||||||
|
waterfall_matrix = np.array(image)
|
||||||
|
|
||||||
|
epoch_start = datetime.strptime(observation['start'], '%Y-%m-%dT%H:%M:%SZ')
|
||||||
|
epoch_end = datetime.strptime(observation['end'], '%Y-%m-%dT%H:%M:%SZ')
|
||||||
|
site_id = int("7" + str(observation['ground_station']))
|
||||||
|
|
||||||
|
station = {'lat': observation['station_lat'],
|
||||||
|
'lng': observation['station_lng'],
|
||||||
|
'alt': observation['station_alt'],
|
||||||
|
'name': "SatNOGS No.{}".format(observation['ground_station']),
|
||||||
|
'id': observation['ground_station']}
|
||||||
|
|
||||||
|
# Store TLE to file
|
||||||
|
with open('{}/{}.txt'.format(settings.TLE_DIR, observation_id), 'w') as f:
|
||||||
|
f.write('0 OBJECT\n')
|
||||||
|
for line in tle:
|
||||||
|
f.write(f'{line}\n')
|
||||||
|
|
||||||
|
# Add SatNOGS station to sites.txt
|
||||||
|
add_station_to_sitestxt(station)
|
||||||
|
|
||||||
|
# Set image parameters
|
||||||
|
lower_left = (66, 1553)
|
||||||
|
upper_right = (686, 13)
|
||||||
|
|
||||||
|
# Set data parameters
|
||||||
|
bandwidth = 48e3 # Hz
|
||||||
|
duration = epoch_end - epoch_start
|
||||||
|
f = observation['transmitter_downlink_low']
|
||||||
|
drift = observation['transmitter_downlink_drift']
|
||||||
|
if drift:
|
||||||
|
f_center = f * (1 + drift / 1e9)
|
||||||
|
else:
|
||||||
|
f_center = f
|
||||||
|
filename_out = '{}/{}.dat'.format(settings.OBS_DIR, observation_id)
|
||||||
|
|
||||||
|
# Initialize SGP4 propagator / pyephem
|
||||||
|
satellite = ephem.readtle('sat', tle[0], tle[1])
|
||||||
|
observer = ephem.Observer()
|
||||||
|
observer.lat = str(station['lat'])
|
||||||
|
observer.lon = str(station['lng'])
|
||||||
|
observer.elevation = station['alt']
|
||||||
|
|
||||||
|
|
||||||
|
def remove_doppler_correction(t, freq):
|
||||||
|
# Remove the doppler correction
|
||||||
|
observer.date = t
|
||||||
|
satellite.compute(observer)
|
||||||
|
v = satellite.range_velocity
|
||||||
|
df = f_center * v / const.c.value
|
||||||
|
return f_center + freq - df
|
||||||
|
|
||||||
|
tabulation_helper_dialog(lower_left, upper_right,
|
||||||
|
bandwidth, duration,
|
||||||
|
f_center, filename_out,
|
||||||
|
waterfall_matrix,
|
||||||
|
epoch_start, site_id, correction_method=remove_doppler_correction)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
parser = argparse.ArgumentParser(description='Interactive helper to tabulate signals from SatNOGS waterfalls.')
|
||||||
|
parser.add_argument('observation_ids', metavar='ID', type=int, nargs='+',
|
||||||
|
help='SatNOGS Observation ID')
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
for observation_id in args.observation_ids:
|
||||||
|
tabulation_helper(observation_id)
|
|
@ -0,0 +1,19 @@
|
||||||
|
from decouple import config
|
||||||
|
|
||||||
|
# Path to TLE
|
||||||
|
# Filename convention: {TLE_DIR}/{observation_id}.txt
|
||||||
|
TLE_DIR = config('SATNOGS_TLE_DIR')
|
||||||
|
|
||||||
|
# absulute frequency measurement storage
|
||||||
|
# Filename convention: {DOPPLER_OBS_DIR}/{observation_id}.dat
|
||||||
|
DOPPLER_OBS_DIR = config('SATNOGS_DOPPLER_OBS_DIR')
|
||||||
|
# legacy name:
|
||||||
|
OBS_DIR = DOPPLER_OBS_DIR
|
||||||
|
|
||||||
|
# SATTOOLS/STRF/STVID sites.txt file
|
||||||
|
SITES_TXT = config('SATNOGS_SITES_TXT')
|
||||||
|
|
||||||
|
|
||||||
|
SATNOGS_NETWORK_API_URL = config('SATNOGS_NETWORK_API_URL')
|
||||||
|
SATNOGS_DB_API_URL = config('SATNOGS_DB_API_URL')
|
||||||
|
SATNOGS_DB_API_TOKEN = config('SATNOGS_DB_API_TOKEN')
|
|
@ -0,0 +1,7 @@
|
||||||
|
#!/usr/bin/bash
|
||||||
|
|
||||||
|
DB_API_TOKEN="4f20a493a3f5fd85074d61db944365bf94987613"
|
||||||
|
URL="https://db-satnogs.freetls.fastly.net/media/artifacts/b4975058-04eb-4ab7-9c40-9bcce76d94db.h5"
|
||||||
|
echo "Authorization: Token $DB_API_TOKEN"
|
||||||
|
curl -H "Authorization: Token $DB_API_TOKEN" "$URL"
|
||||||
|
|
|
@ -0,0 +1,17 @@
|
||||||
|
# "New" Python STRF environment variables
|
||||||
|
|
||||||
|
# SATNOGS_DIR is used by config.sh for STRF configuration
|
||||||
|
SATNOGS_DIR=/home/pi/satnogs_data
|
||||||
|
|
||||||
|
# SatNOGS Waterfall Tabulation Helper
|
||||||
|
export SATNOGS_TLE_DIR=/home/pi/satnogs_data/tles/satnogs
|
||||||
|
export SATNOGS_ARTIFACTS_DIR=/home/pi/satnogs_data/artifacts
|
||||||
|
export SATNOGS_DOPPLER_OBS_DIR=/home/pi/satnogs_data/doppler_obs
|
||||||
|
SATNOGS_SITES_TXT=$SATNOGS_DIR/sites.txt
|
||||||
|
|
||||||
|
|
||||||
|
# SatNOGS Artifacts Helpers
|
||||||
|
SATNOGS_NETWORK_API_URL=https://network.satnogs.org/api/
|
||||||
|
SATNOGS_DB_API_URL=https://db.satnogs.org/api/
|
||||||
|
|
||||||
|
SATNOGS_DB_API_TOKEN=your-db-api-token
|
Loading…
Reference in New Issue