wut? --- What U Think? SatNOGS Observation AI.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
Jeff Moe 35dc603832 Various download Internet Archive cruft 3 weeks ago
docs voila dark theme 1 month ago
notebooks Add systemd voila scripts 1 month ago
pics spacecruft svg 2 years ago
scripts uh, voila on wut, jupyter on ml0 duh 2 years ago
.gitignore ignore tmp model render 1 month ago
LICENSE-CC license 3 years ago
LICENSE-GPL license 3 years ago
README.md wut-aria-* scripts for downloading Internet Archive torrents 4 weeks ago
requirements-node.txt CPU nodes for tensorflow 1 month ago
requirements-voila.txt Update notebooks to current Keras, pip deps 1 month ago
requirements.txt wut-ia Internet Archive download stub 4 weeks ago
wut update paths to /srv/satnogs 2 years ago
wut-aria-active Various download Internet Archive cruft 3 weeks ago
wut-aria-add Various download Internet Archive cruft 3 weeks ago
wut-aria-daemon download torrents from Internet Archive 3 weeks ago
wut-aria-info Various download Internet Archive cruft 3 weeks ago
wut-aria-methods Various download Internet Archive cruft 3 weeks ago
wut-aria-shutdown Various download Internet Archive cruft 3 weeks ago
wut-aria-stat Various download Internet Archive cruft 3 weeks ago
wut-aria-stopped Various download Internet Archive cruft 3 weeks ago
wut-aria-waiting Various download Internet Archive cruft 3 weeks ago
wut-audio-archive update paths to /srv/satnogs 2 years ago
wut-audio-sha1 wut-ogg2wav, misc 3 years ago
wut-compare wut-compare scripts 3 years ago
wut-compare-all Flip scoring, rename variables 3 years ago
wut-compare-tx wut-ml-load wut-ml-save 3 years ago
wut-compare-txmode new output 3 years ago
wut-compare-txmode-csv update docs 3 years ago
wut-dl-sort update paths to /srv/satnogs 2 years ago
wut-dl-sort-tx update paths to /srv/satnogs 2 years ago
wut-dl-sort-txmode prep txmode waterfalls into respective subdirs 2 years ago
wut-dl-sort-txmode-all dec range 2 years ago
wut-files update paths to /srv/satnogs 2 years ago
wut-files-data update paths to /srv/satnogs 2 years ago
wut-files-data-all wut-files-data-all... 2 years ago
wut-ia-sha1 Various download Internet Archive cruft 3 weeks ago
wut-ia-torrents Various download Internet Archive cruft 3 weeks ago
wut-img-ck.py check all PNGs 1 month ago
wut-ml update paths to /srv/satnogs 2 years ago
wut-ml-auto update paths to /srv/satnogs 2 years ago
wut-ml-load update paths to /srv/satnogs 2 years ago
wut-ml-save update paths to /srv/satnogs 2 years ago
wut-obs update paths to /srv/satnogs 2 years ago
wut-ogg2wav update docs 3 years ago
wut-review-staging update paths to /srv/satnogs 2 years ago
wut-rm-random update paths to /srv/satnogs 2 years ago
wut-tf wtf scripts to check tensorflow setup 2 years ago
wut-tf.py wtf scripts to check tensorflow setup 2 years ago
wut-water update paths to /srv/satnogs 2 years ago
wut-water-range Observation ID by year 1 month ago
wut-worker renames 2 years ago
wut-worker-mas cruft 2 years ago
wut-worker-mas.py update paths to /srv/satnogs 2 years ago
wut-worker.py renames 2 years ago

README.md

wut?

wut --- What U Think? SatNOGS Observation AI.

Website:

satnogs-wut

Image

The goal of satnogs-wut is to have a script that will take an observation ID and return an answer whether the observation is "good", "bad", or "failed".

Good Observation

Image

Bad Observation

Image

Failed Observation

Image

wut Web

Main site:

Source code:

Beta (test) site:

Alpha (development) site:

Observations

See also:

Machine Learning

The system at present is built upon the following:

  • Debian Bookworm (testing/12).
  • Tensorflow.
  • Jupyter Lab.
  • Voila.

Learning/testing, results are good. The main AI/ML development is being done in Jupyter.

Jupyter

There Jupyter Lab Notebook files in the notebooks/ subdirectory. These are producing usable results. Voila is used to convert Jupyter notebooks into websites.

wut scripts

The following scripts are in the repo.

  • wut --- Feed it an observation ID and it returns if it is a "good", "bad", or "failed" observation.
  • wut-aria-add --- Add a torrent from the Internet Archive to the aria daemon for downloading.
  • wut-aria-daemon --- Run an aria daemon for torrent downloads from the Internet Archive.
  • wut-audio-archive --- Downloads audio files from archive.org.
  • wut-audio-sha1 --- Verifies sha1 checksums of files downloaded from archive.org.
  • wut-compare --- Compare an observations' current presumably human vetting with a wut vetting.
  • wut-compare-all --- Compare all the observations in download/ with wut vettings.
  • wut-compare-tx --- Compare all the observations in download/ with wut vettings using selected transmitter UUID.
  • wut-compare-txmode --- Compare all the observations in download/ with wut vettings using selected encoding.
  • wut-compare-txmode-csv --- Compare all the observations in download/ with wut vettings using selected encoding, CSV output.
  • wut-dl-sort --- Populate data/ dir with waterfalls from download/.
  • wut-dl-sort-tx --- Populate data/ dir with waterfalls from download/ using selected transmitter UUID.
  • wut-dl-sort-txmode --- Populate data/ dir with waterfalls from download/ using selected encoding.
  • wut-dl-sort-txmode-all --- Populate data/ dir with waterfalls from download/ using all encodings.
  • wut-files --- Tells you about what files you have in downloads/ and data/.
  • wut-files-data --- Tells you about what files you have in data/.
  • wut-ia --- Download SatNOGS data from the Internet Archive at archive.org.
  • wut-ia-torrents --- Download SatNOGS torrents from the Internet Archive at archive.org.
  • wut-img-ck.py --- Validate image files are not corrupt with PIL.
  • wut-ml --- Main machine learning Python script using Tensorflow and Keras.
  • wut-ml-auto --- Machine learning Python script using Tensorflow and Keras, auto.
  • wut-ml-load --- Machine learning Python script using Tensorflow and Keras, load data/wut.h5.
  • wut-ml-save --- Machine learning Python script using Tensorflow and Keras, save data/wut.h5.
  • wut-obs --- Download the JSON for an observation ID.
  • wut-ogg2wav --- Convert .ogg files in downloads/ to .wav files.
  • wut-rm-random --- Randomly deletes stuff. Very bad.
  • wut-review-staging --- Review all images in data/staging.
  • wut-tf --- Shell script to set variables when launching wut-tf.py.
  • wut-tf.py --- Distributed learning script to be run on multiple nodes.
  • wut-water --- Download waterfall for an observation ID to download/[ID].
  • wut-water-range --- Download waterfalls for a range of observation IDs to download/[ID].
  • wut-worker --- Shell script to set variables when launching wut-worker.py.
  • wut-worker.py --- Distributed training script to run on multiple nodes.
  • wut-worker-mas --- Shell script to set variables when launching wut-worker-mas.py.
  • wut-worker-mas.py --- Distributed training script to run on multiple nodes, alt version.

Installation

Installation notes...

There's more docs on a few different setups in the docs/ subdir.

Setup

The scripts use files that are ignored in the git repo. So you need to create those directories:

mkdir -p download
mkdir -p data/train/good
mkdir -p data/train/bad
mkdir -p data/train/failed
mkdir -p data/val/good
mkdir -p data/val/bad
mkdir -p data/val/failed
mkdir -p data/staging
mkdir -p data/test/unvetted

Debian Packages

Install dependencies from Debian.

sudo apt update
sudo apt install curl jq python3-pip graphviz

Install Python Packages

For the machine learning scripts, like wut-ml, Tensorflow needs to be installed.

You need to add ~/.local/bin to your $PATH:

echo 'PATH=~/.local/bin:$PATH' >> ~/.bashrc

Then log out and back in, or reload ala:

. ~/.bashrc

Update pip to latest pretty version, in local directory. Vary Python package install, suited to taste.

pip install --user --upgrade pip

Make sure you have right pip:

debian@workstation:~$ which pip
/home/debian/.local/bin/pip

Install Python packages:

pip install --user --upgrade -r requirements.txt

Tensorflow KVM Notes

Note, for KVM, pass cpu=host if host has "avx" in /proc/cpuinfo.

Recent versions of Tensorflow can handle many more CPU build options to optimize for speed, such as AVX. By default, Proxmox and likely other virtual machine systems pass kvm/qemu "type=kvm" for CPU type. To use all possible CPU options available on the bare metal server, use "type=host". For more info about this in Proxmox, see CPU Type If you don't have this enabled, CPU instructions will fail or Tensorflow will run slower than it could.

Jupyter

Jupyter is a cute little web interface that makes Python programming easy. It works well for machine learning because you can step through just parts of the code, changing variables and immediately seeing output in the web browser.

Usage

The main purpose of the script is to evaluate an observation, but to do that, it needs to build a corpus of observations to learn from. So many of the scripts in this repo are just for downloading and managing observations.

The following steps need to be performed:

  1. Download waterfalls and JSON descriptions with wut-water-range. These get put in the downloads/[ID]/ directories.

  2. Organize downloaded waterfalls into categories (e.g. "good", "bad", "failed"). Use wut-dl-sort script. The script will sort them into their respective directories under:

    • data/train/good/
    • data/train/bad/
    • data/train/failed/
    • data/val/good/
    • data/val/bad/
    • data/val/failed/
  3. Use machine learning script wut-ml to build a model based on the files in the data/train and data/val directories.

  4. Rate an observation using the wut script.

Data Caching Downloads

The scripts are designed to not download a waterfall or make a JSON request for an observation it has already requested. The first time an observation is requested, it is downloaded from the SatNOGS network to the download/ directory. That download/ directory is the download cache.

The data/ directory is just temporary files, mostly linked from the downloads/ directory. Files in the data/ directory are deleted by many scripts, so don't put anything you want to keep in there.

Preprocessed Files

Files in the preprocess/ directory have been preprocessed to be used further in the pipeline. This contains .wav files that have been decoded from .ogg files.

Internet Archive Downloads

The Internet Archive has a mirror of data from the SatNOGS network. It is better to download from there to save on Libre Space Foundation resources.

To download, perhaps do something like the following. Get an account at archive.org, then run this to set up your account locally:

ia configure

To download all the SatNOGS collections .torrent files from the Internet Archive, run:

wut-ia-torrents

Caveats

This is the first artificial intelligence script I've done, I know little about radio and less about satellites, and I'm not a programmer.

Source License / Copying

Main repository is available here:

License: CC By SA 4.0 International and/or GPLv3+ at your discretion. Other code licensed under their own respective licenses.

Copyright (C) 2019, 2020, 2022 Jeff Moe