diff --git a/.gitignore b/.gitignore index 2f3fbde..1088ca9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ *.swp data/ -dump*.json download/ +dump*.json +preprocess/ tmp/ diff --git a/README.md b/README.md index 42bb699..1221f9f 100644 --- a/README.md +++ b/README.md @@ -28,15 +28,21 @@ Learning/testing, results are inaccurate. The following scripts are in the repo: * `wut` --- Feed it an observation ID and it returns if it is a "good", "bad", or "failed" observation. +* `wut-audio-archive` --- Downloads audio files from archive.org. * `wut-compare` --- Compare an observations' current presumably human vetting with a `wut` vetting. * `wut-compare-all` --- Compare all the observations in `download/` with `wut` vettings. * `wut-compare-tx` --- Compare all the observations in `download/` with `wut` vettings using selected transmitter UUID. * `wut-compare-txmode` --- Compare all the observations in `download/` with `wut` vettings using selected encoding. +* `wut-compare-txmode-csv` --- Compare all the observations in `download/` with `wut` vettings using selected encoding, CSV output. * `wut-dl-sort` --- Populate `data/` dir with waterfalls from `download/`. * `wut-dl-sort-tx` --- Populate `data/` dir with waterfalls from `download/` using selected transmitter UUID. * `wut-dl-sort-txmode` --- Populate `data/` dir with waterfalls from `download/` using selected encoding. +* `wut-files` --- Tells you about what files you have in `downloads/` and `data/`. * `wut-ml` --- Main machine learning Python script using Tensorflow and Keras. +* `wut-ml-load` --- Machine learning Python script using Tensorflow and Keras, load `data/wut.h5`. +* `wut-ml-save` --- Machine learning Python script using Tensorflow and Keras, save `data/wut.h5`. * `wut-obs` --- Download the JSON for an observation ID. +* `wut-ogg2wav` --- Convert `.ogg` files in `downloads/` to `.wav` files. * `wut-review-staging` --- Review all images in `data/staging`. * `wut-water` --- Download waterfall for an observation ID to `download/[ID]`. * `wut-water-range` --- Download waterfalls for a range of observation IDs to `download/[ID]`. @@ -132,15 +138,21 @@ to sync. ## Data Caching Downloads The scripts are designed to not download a waterfall or make a JSON request for an observation it has already requested. The first time an observation -is requested, it is downloaded from the SatNOGS network to the `download` -directory. That `download` directory is the download cache. +is requested, it is downloaded from the SatNOGS network to the `download/` +directory. That `download/` directory is the download cache. -The `data` directory is just temporary files,mostly linked from the -`downloads` directory. Files in the `data` directory are deleted by many +The `data/` directory is just temporary files, mostly linked from the +`downloads/` directory. Files in the `data/` directory are deleted by many scripts, so don't put anything you want to keep in there. +## Preprocessed Files +Files in the `preprocess/` directory have been preprocessed to be used +further in the pipeline. This contains `.wav` files that have been +decoded from `.ogg` files. + + ## SatNOGS Observation Data Mirror The downloaded waterfalls are available below via `http` and `rsync`. Use this instead of downloading from SatNOGS to save their bandwidth. diff --git a/wut-compare-txmode-csv b/wut-compare-txmode-csv index 27860e5..75d1f60 100755 --- a/wut-compare-txmode-csv +++ b/wut-compare-txmode-csv @@ -1,8 +1,10 @@ #!/bin/bash -# wut-compare-txmode +# wut-compare-txmode-csv # # Check the results of a prediction against vetted results # using a selected encoding. +# Outputs CSV (comma separated values). +# # Uses all files in download/ directory. # Available encodings: # AFSK AFSK1k2 AHRPT APT BPSK BPSK1k2 BPSK9k6 BPSK12k5 BPSK400 CERTO CW DUV @@ -11,9 +13,9 @@ # MSK1k2 MSK2k4 MSK4k8 PSK PSK31 SSTV USB WSJT # # Usage: -# wut-compare-txmode [Encoding] +# wut-compare-txmode-csv [Encoding] # Example: -# wut-compare-txmode DUV +# wut-compare-txmode-csv DUV MAIN_DIR=`pwd` OBSENC="$1" diff --git a/wut-ml-load b/wut-ml-load index 4f1080b..d186de2 100755 --- a/wut-ml-load +++ b/wut-ml-load @@ -1,17 +1,21 @@ #!/usr/bin/python3 -# wut-ml +# +# XXX This does not work yet XXX +# +# wut-ml-load # # Vet a SatNOGS image using machine learning (guessing). # It will vet the image located at test/unvetted/waterfall.png. +# Load data/wut.h5 file. # # Note, there is an issue to fix where it will vet everything # under the data/test directory, so fix that. For now, just delete # everything else. :) # # Usage: -# wut-ml +# wut-ml-load # Example: -# wut-ml +# wut-ml-load import os import numpy as np @@ -26,7 +30,7 @@ from tensorflow.python.keras.models import load_model from tensorflow.python.keras.preprocessing.image import load_img from tensorflow.python.keras.preprocessing.image import img_to_array -model = load_model('wut.h5') +model = load_model('date/wut.h5') img_width=256 img_height=256 model = Sequential() diff --git a/wut-ml-save b/wut-ml-save index e580bc4..88e8eaa 100755 --- a/wut-ml-save +++ b/wut-ml-save @@ -1,5 +1,8 @@ #!/usr/bin/python3 -# wut-ml +# +# XXX This does not work yet XXX +# +# wut-ml-save # # Vet a SatNOGS image using machine learning (guessing). # It will vet the image located at test/unvetted/waterfall.png. @@ -9,9 +12,9 @@ # everything else. :) # # Usage: -# wut-ml +# wut-ml-save # Example: -# wut-ml +# wut-ml-save import os import numpy as np diff --git a/wut-ogg2wav b/wut-ogg2wav index dc54cc5..d259c1b 100755 --- a/wut-ogg2wav +++ b/wut-ogg2wav @@ -16,7 +16,7 @@ cd download/ || exit # Compile a list of ogg files (NOTE: THIS WILL BREAK AS ARCHIVE GROWS XXX) echo "Total audio files: `ls -1 */satnogs_*.ogg | wc -l`" -for i in 129*/satnogs_*.ogg +for i in 13*/satnogs_*.ogg do OBSID=`dirname $i` # Go into directories with audiofiles echo $OBSID