update docs

master 0.23
ml server 2020-01-10 18:05:57 -07:00
parent 68a37ea874
commit 6a444136f7
6 changed files with 38 additions and 16 deletions

3
.gitignore vendored
View File

@ -1,5 +1,6 @@
*.swp
data/
dump*.json
download/
dump*.json
preprocess/
tmp/

View File

@ -28,15 +28,21 @@ Learning/testing, results are inaccurate.
The following scripts are in the repo:
* `wut` --- Feed it an observation ID and it returns if it is a "good", "bad", or "failed" observation.
* `wut-audio-archive` --- Downloads audio files from archive.org.
* `wut-compare` --- Compare an observations' current presumably human vetting with a `wut` vetting.
* `wut-compare-all` --- Compare all the observations in `download/` with `wut` vettings.
* `wut-compare-tx` --- Compare all the observations in `download/` with `wut` vettings using selected transmitter UUID.
* `wut-compare-txmode` --- Compare all the observations in `download/` with `wut` vettings using selected encoding.
* `wut-compare-txmode-csv` --- Compare all the observations in `download/` with `wut` vettings using selected encoding, CSV output.
* `wut-dl-sort` --- Populate `data/` dir with waterfalls from `download/`.
* `wut-dl-sort-tx` --- Populate `data/` dir with waterfalls from `download/` using selected transmitter UUID.
* `wut-dl-sort-txmode` --- Populate `data/` dir with waterfalls from `download/` using selected encoding.
* `wut-files` --- Tells you about what files you have in `downloads/` and `data/`.
* `wut-ml` --- Main machine learning Python script using Tensorflow and Keras.
* `wut-ml-load` --- Machine learning Python script using Tensorflow and Keras, load `data/wut.h5`.
* `wut-ml-save` --- Machine learning Python script using Tensorflow and Keras, save `data/wut.h5`.
* `wut-obs` --- Download the JSON for an observation ID.
* `wut-ogg2wav` --- Convert `.ogg` files in `downloads/` to `.wav` files.
* `wut-review-staging` --- Review all images in `data/staging`.
* `wut-water` --- Download waterfall for an observation ID to `download/[ID]`.
* `wut-water-range` --- Download waterfalls for a range of observation IDs to `download/[ID]`.
@ -132,15 +138,21 @@ to sync.
## Data Caching Downloads
The scripts are designed to not download a waterfall or make a JSON request
for an observation it has already requested. The first time an observation
is requested, it is downloaded from the SatNOGS network to the `download`
directory. That `download` directory is the download cache.
is requested, it is downloaded from the SatNOGS network to the `download/`
directory. That `download/` directory is the download cache.
The `data` directory is just temporary files,mostly linked from the
`downloads` directory. Files in the `data` directory are deleted by many
The `data/` directory is just temporary files, mostly linked from the
`downloads/` directory. Files in the `data/` directory are deleted by many
scripts, so don't put anything you want to keep in there.
## Preprocessed Files
Files in the `preprocess/` directory have been preprocessed to be used
further in the pipeline. This contains `.wav` files that have been
decoded from `.ogg` files.
## SatNOGS Observation Data Mirror
The downloaded waterfalls are available below via `http` and `rsync`.
Use this instead of downloading from SatNOGS to save their bandwidth.

View File

@ -1,8 +1,10 @@
#!/bin/bash
# wut-compare-txmode
# wut-compare-txmode-csv
#
# Check the results of a prediction against vetted results
# using a selected encoding.
# Outputs CSV (comma separated values).
#
# Uses all files in download/ directory.
# Available encodings:
# AFSK AFSK1k2 AHRPT APT BPSK BPSK1k2 BPSK9k6 BPSK12k5 BPSK400 CERTO CW DUV
@ -11,9 +13,9 @@
# MSK1k2 MSK2k4 MSK4k8 PSK PSK31 SSTV USB WSJT
#
# Usage:
# wut-compare-txmode [Encoding]
# wut-compare-txmode-csv [Encoding]
# Example:
# wut-compare-txmode DUV
# wut-compare-txmode-csv DUV
MAIN_DIR=`pwd`
OBSENC="$1"

View File

@ -1,17 +1,21 @@
#!/usr/bin/python3
# wut-ml
#
# XXX This does not work yet XXX
#
# wut-ml-load
#
# Vet a SatNOGS image using machine learning (guessing).
# It will vet the image located at test/unvetted/waterfall.png.
# Load data/wut.h5 file.
#
# Note, there is an issue to fix where it will vet everything
# under the data/test directory, so fix that. For now, just delete
# everything else. :)
#
# Usage:
# wut-ml
# wut-ml-load
# Example:
# wut-ml
# wut-ml-load
import os
import numpy as np
@ -26,7 +30,7 @@ from tensorflow.python.keras.models import load_model
from tensorflow.python.keras.preprocessing.image import load_img
from tensorflow.python.keras.preprocessing.image import img_to_array
model = load_model('wut.h5')
model = load_model('date/wut.h5')
img_width=256
img_height=256
model = Sequential()

View File

@ -1,5 +1,8 @@
#!/usr/bin/python3
# wut-ml
#
# XXX This does not work yet XXX
#
# wut-ml-save
#
# Vet a SatNOGS image using machine learning (guessing).
# It will vet the image located at test/unvetted/waterfall.png.
@ -9,9 +12,9 @@
# everything else. :)
#
# Usage:
# wut-ml
# wut-ml-save
# Example:
# wut-ml
# wut-ml-save
import os
import numpy as np

View File

@ -16,7 +16,7 @@ cd download/ || exit
# Compile a list of ogg files (NOTE: THIS WILL BREAK AS ARCHIVE GROWS XXX)
echo "Total audio files: `ls -1 */satnogs_*.ogg | wc -l`"
for i in 129*/satnogs_*.ogg
for i in 13*/satnogs_*.ogg
do OBSID=`dirname $i`
# Go into directories with audiofiles
echo $OBSID