update docs

.gitignore

@@ -1,5 +1,6 @@
 *.swp
 data/
-dump*.json
 download/
+dump*.json
+preprocess/
 tmp/

README.md

@@ -28,15 +28,21 @@ Learning/testing, results are inaccurate.
 The following scripts are in the repo:
 
 * `wut` --- Feed it an observation ID and it returns if it is a "good", "bad", or "failed" observation.
+* `wut-audio-archive` --- Downloads audio files from archive.org.
 * `wut-compare` --- Compare an observations' current presumably human vetting with a `wut` vetting.
 * `wut-compare-all` --- Compare all the observations in `download/` with `wut` vettings.
 * `wut-compare-tx` --- Compare all the observations in `download/` with `wut` vettings using selected transmitter UUID.
 * `wut-compare-txmode` --- Compare all the observations in `download/` with `wut` vettings using selected encoding.
+* `wut-compare-txmode-csv` --- Compare all the observations in `download/` with `wut` vettings using selected encoding, CSV output.
 * `wut-dl-sort` --- Populate `data/` dir with waterfalls from `download/`.
 * `wut-dl-sort-tx` --- Populate `data/` dir with waterfalls from `download/` using selected transmitter UUID.
 * `wut-dl-sort-txmode` --- Populate `data/` dir with waterfalls from `download/` using selected encoding.
+* `wut-files` --- Tells you about what files you have in `downloads/` and `data/`.
 * `wut-ml` --- Main machine learning Python script using Tensorflow and Keras.
+* `wut-ml-load` --- Machine learning Python script using Tensorflow and Keras, load `data/wut.h5`.
+* `wut-ml-save` --- Machine learning Python script using Tensorflow and Keras, save `data/wut.h5`.
 * `wut-obs` --- Download the JSON for an observation ID.
+* `wut-ogg2wav` --- Convert `.ogg` files in `downloads/` to `.wav` files.
 * `wut-review-staging` --- Review all images in `data/staging`.
 * `wut-water` --- Download waterfall for an observation ID to `download/[ID]`.
 * `wut-water-range` --- Download waterfalls for a range of observation IDs to `download/[ID]`.
@@ -132,15 +138,21 @@ to sync.
 ## Data Caching Downloads
 The scripts are designed to not download a waterfall or make a JSON request
 for an observation it has already requested. The first time an observation
-is requested, it is downloaded from the SatNOGS network to the `download`
+is requested, it is downloaded from the SatNOGS network to the `download/`
-directory. That `download` directory is the download cache. 
+directory. That `download/` directory is the download cache. 
 
 
-The `data` directory is just temporary files,mostly linked from the
+The `data/` directory is just temporary files, mostly linked from the
-`downloads` directory. Files in the `data` directory are deleted by many
+`downloads/` directory. Files in the `data/` directory are deleted by many
 scripts, so don't put anything you want to keep in there.
 
 
+## Preprocessed Files
+Files in the `preprocess/` directory have been preprocessed to be used
+further in the pipeline. This contains `.wav` files that have been
+decoded from `.ogg` files.
+
+
 ## SatNOGS Observation Data Mirror
 The downloaded waterfalls are available below via `http` and `rsync`.
 Use this instead of downloading from SatNOGS to save their bandwidth.

wut-compare-txmode-csv

@@ -1,8 +1,10 @@
 #!/bin/bash
-# wut-compare-txmode
+# wut-compare-txmode-csv
 #
 # Check the results of a prediction against vetted results
 # using a selected encoding.
+# Outputs CSV (comma separated values).
+#
 # Uses all files in download/ directory.
 # Available encodings:
 # AFSK AFSK1k2 AHRPT APT BPSK BPSK1k2 BPSK9k6 BPSK12k5 BPSK400 CERTO CW DUV
@@ -11,9 +13,9 @@
 # MSK1k2 MSK2k4 MSK4k8 PSK PSK31 SSTV USB WSJT
 #
 # Usage:
-# wut-compare-txmode [Encoding]
+# wut-compare-txmode-csv [Encoding]
 # Example:
-# wut-compare-txmode DUV 
+# wut-compare-txmode-csv DUV 
 
 MAIN_DIR=`pwd`
 OBSENC="$1"

wut-ml-load

@@ -1,17 +1,21 @@
 #!/usr/bin/python3
-# wut-ml
+#
+# XXX This does not work yet XXX
+#
+# wut-ml-load
 #
 # Vet a SatNOGS image using machine learning (guessing).
 # It will vet the image located at test/unvetted/waterfall.png.
+# Load data/wut.h5 file.
 #
 # Note, there is an issue to fix where it will vet everything
 # under the data/test directory, so fix that. For now, just delete
 # everything else. :)
 #
 # Usage:
-# wut-ml
+# wut-ml-load
 # Example:
-# wut-ml
+# wut-ml-load
 
 import os
 import numpy as np
@@ -26,7 +30,7 @@ from tensorflow.python.keras.models import load_model
 from tensorflow.python.keras.preprocessing.image import load_img
 from tensorflow.python.keras.preprocessing.image import img_to_array
 
-model = load_model('wut.h5')
+model = load_model('date/wut.h5')
 img_width=256
 img_height=256
 model = Sequential()

wut-ml-save

@@ -1,5 +1,8 @@
 #!/usr/bin/python3
-# wut-ml
+#
+# XXX This does not work yet XXX
+#
+# wut-ml-save
 #
 # Vet a SatNOGS image using machine learning (guessing).
 # It will vet the image located at test/unvetted/waterfall.png.
@@ -9,9 +12,9 @@
 # everything else. :)
 #
 # Usage:
-# wut-ml
+# wut-ml-save
 # Example:
-# wut-ml
+# wut-ml-save
 
 import os
 import numpy as np

wut-ogg2wav

@@ -16,7 +16,7 @@ cd download/ || exit
 
 # Compile a list of ogg files (NOTE: THIS WILL BREAK AS ARCHIVE GROWS XXX)
 echo "Total audio files: `ls -1 */satnogs_*.ogg | wc -l`"
-for i in 129*/satnogs_*.ogg
+for i in 13*/satnogs_*.ogg
 	do OBSID=`dirname $i`
 	# Go into directories with audiofiles
 	echo $OBSID