Start update after ~2 years...
Jeff Moe
parent
b52c07e8ed
commit
b7c1712853
182
README.md
182
README.md
|
|
@ -24,20 +24,9 @@ observation ID and return an answer whether the observation is
|
|||
|
||||
## wut Web
|
||||
|
||||
Main site:
|
||||
* https://wut.spacecruft.org/
|
||||
|
||||
Source code:
|
||||
* https://spacecruft.org/spacecruft/satnogs-wut
|
||||
|
||||
Beta (test) site:
|
||||
|
||||
* https://wut-beta.spacecruft.org/
|
||||
|
||||
Alpha (development) site:
|
||||
|
||||
* https://wut-alpha.spacecruft.org/
|
||||
|
||||
## Observations
|
||||
See also:
|
||||
|
||||
|
|
@ -50,8 +39,8 @@ See also:
|
|||
# Machine Learning
|
||||
The system at present is built upon the following:
|
||||
|
||||
* Debian Buster.
|
||||
* Tensorflow 2 with Keras.
|
||||
* Debian Bookworm (testing/12).
|
||||
* Tensorflow.
|
||||
* Jupyter Lab.
|
||||
* Voila.
|
||||
|
||||
|
|
@ -66,9 +55,9 @@ Jupyter notebooks into websites.
|
|||
* `wut.ipynb` --- Machine learning Python script using Tensorflow and Keras in a Jupyter Notebook.
|
||||
* `wut-predict.ipynb` --- Make prediction (rating) of observation from pre-existing model.
|
||||
* `wut-train.ipynb` --- Train models to be using by prediction engine.
|
||||
* `wut-web.ipynb` --- Website: https://wut.spacecruft.org/
|
||||
* `wut-web-beta.ipynb` --- Website: https://wut-beta.spacecruft.org/
|
||||
* `wut-web-alpha.ipynb` --- Website: https://wut-alpha.spacecruft.org/
|
||||
* `wut-web.ipynb`
|
||||
* `wut-web-beta.ipynb`
|
||||
* `wut-web-alpha.ipynb`
|
||||
|
||||
# wut scripts
|
||||
The following scripts are in the repo.
|
||||
|
|
@ -128,60 +117,49 @@ mkdir -p data/test/unvetted
|
|||
```
|
||||
|
||||
## Debian Packages
|
||||
You'll need `curl` and `jq`, both in Debian's repos.
|
||||
Install dependencies from Debian.
|
||||
|
||||
```
|
||||
apt update
|
||||
apt install curl jq
|
||||
apt install curl jq python3-pip
|
||||
```
|
||||
|
||||
## Install Tensorflow
|
||||
## Install Python Packages
|
||||
For the machine learning scripts, like `wut-ml`, Tensorflow
|
||||
needs to be installed.
|
||||
As of version 2 of Tensorflow, Keras no longer needs to be
|
||||
installed separately.
|
||||
|
||||
|
||||
The verions of Tensorflow installed with `pip3` on Debian
|
||||
Buster crashes. It is perhaps best to do a custom install,
|
||||
best preferred build options, of the most preferred version.
|
||||
At this point, the `remotes/origin/r2.1` branch is preferred.
|
||||
|
||||
|
||||
To install Tensorflow:
|
||||
|
||||
* https://www.tensorflow.org/install/source
|
||||
|
||||
1. Install dependencies in Debian.
|
||||
|
||||
1. Install Bazel to build Tensorflow.
|
||||
|
||||
1. Build Tensorflow pip package.
|
||||
|
||||
1. Install Tensorflow from custom pip package.
|
||||
|
||||
You need to add `~/.local/bin` to your `$PATH`:
|
||||
|
||||
```
|
||||
# Install deps
|
||||
apt update
|
||||
apt install python3-pip
|
||||
# Install bazel .deb from releases here:
|
||||
firefox https://github.com/bazelbuild/bazel/releases
|
||||
# Install Tensorflow
|
||||
git clone tensorflow...
|
||||
cd tensorflow
|
||||
git checkout v2.1.0
|
||||
bazel clean
|
||||
# Get flags to pass:
|
||||
grep flags -m1 /proc/cpuinfo | cut -d ":" -f 2 | tr '[:upper:]' '[:lower:]' | { read FLAGS; OPT="-march=native"; for flag in $FLAGS; do case "$flag" in "sse4_1" | "sse4_2" | "ssse3" | "fma" | "cx16" | "popcnt" | "avx" | "avx2") OPT+=" -m$flag";; esac; done; MODOPT=${OPT//_/\.}; echo "$MODOPT"; }
|
||||
./configure
|
||||
# Run Bazel to build pip package. Takes nearly 2 hours to build.
|
||||
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
|
||||
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
|
||||
pip3 install --user /tmp/tensorflow_pkg/tensorflow-2.1.0-cp37-cp37m-linux_x86_64.whl
|
||||
echo 'PATH=~/.local/bin:$PATH' >> ~/.bashrc
|
||||
```
|
||||
|
||||
Then log out and back in, or reload ala:
|
||||
```
|
||||
. ~/.bashrc
|
||||
```
|
||||
|
||||
Update pip to latest pretty version, in local directory.
|
||||
Vary Python package install, suited to taste.
|
||||
|
||||
```
|
||||
pip install --user --upgrade pip
|
||||
```
|
||||
|
||||
Make sure you have right pip:
|
||||
```
|
||||
debian@workstation:~$ which pip
|
||||
/home/debian/.local/bin/pip
|
||||
```
|
||||
|
||||
Install Python packages:
|
||||
```
|
||||
pip install --user --upgrade -r requirements.txt
|
||||
```
|
||||
|
||||
### Tensorflow KVM Notes
|
||||
Note, for KVM, pass cpu=host if host has "avx" in `/proc/cpuinfo`.
|
||||
|
||||
Recent versions of Tensorflow can handle many more CPU build options
|
||||
to optimize for speed, such as
|
||||
[AVX](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions).
|
||||
|
|
@ -193,81 +171,13 @@ For more info about this in Proxmox, see
|
|||
If you don't have this enabled, CPU instructions will fail or
|
||||
Tensorflow will run slower than it could.
|
||||
|
||||
### Tensor Configuration
|
||||
```
|
||||
$ ./configure
|
||||
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
|
||||
You have bazel 0.29.1 installed.
|
||||
Please specify the location of python. [Default is /usr/bin/python3]:
|
||||
|
||||
|
||||
Found possible Python library paths:
|
||||
/usr/lib/python3/dist-packages
|
||||
/usr/local/lib/python3.7/dist-packages
|
||||
Please input the desired Python library path to use. Default is [/usr/lib/python3/dist-packages]
|
||||
|
||||
Do you wish to build TensorFlow with XLA JIT support? [Y/n]:
|
||||
XLA JIT support will be enabled for TensorFlow.
|
||||
|
||||
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]:
|
||||
No OpenCL SYCL support will be enabled for TensorFlow.
|
||||
|
||||
Do you wish to build TensorFlow with ROCm support? [y/N]:
|
||||
No ROCm support will be enabled for TensorFlow.
|
||||
|
||||
Do you wish to build TensorFlow with CUDA support? [y/N]:
|
||||
No CUDA support will be enabled for TensorFlow.
|
||||
|
||||
Do you wish to download a fresh release of clang? (Experimental) [y/N]:
|
||||
Clang will not be downloaded.
|
||||
|
||||
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native -Wno-sign-compare]: -march=native -mssse3 -mcx16 -msse4.1 -msse4.2 -mpopcnt -mavx
|
||||
|
||||
|
||||
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
|
||||
Not configuring the WORKSPACE for Android builds.
|
||||
|
||||
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
|
||||
--config=mkl # Build with MKL support.
|
||||
--config=monolithic # Config for mostly static monolithic build.
|
||||
--config=ngraph # Build with Intel nGraph support.
|
||||
--config=numa # Build with NUMA support.
|
||||
--config=dynamic_kernels # (Experimental) Build kernels into separate shared objects.
|
||||
--config=v2 # Build TensorFlow 2.x instead of 1.x.
|
||||
Preconfigured Bazel build configs to DISABLE default on features:
|
||||
--config=noaws # Disable AWS S3 filesystem support.
|
||||
--config=nogcp # Disable GCP support.
|
||||
--config=nohdfs # Disable HDFS support.
|
||||
--config=nonccl # Disable NVIDIA NCCL support.
|
||||
Configuration finished
|
||||
```
|
||||
|
||||
## KVM
|
||||
Note, for KVM, pass cpu=host if host has "avx" in `/proc/cpuinfo`.
|
||||
|
||||
## Install Jupyter
|
||||
## Jupyter
|
||||
Jupyter is a cute little web interface that makes Python programming
|
||||
easy. It works well for machine learning because you can step through
|
||||
just parts of the code, changing variables and immediately seeing
|
||||
output in the web browser.
|
||||
|
||||
Probably installed like this:
|
||||
|
||||
```
|
||||
pip3 install --user jupyterlab
|
||||
# Also other good packages, maybe like:
|
||||
pip3 install --user jupyter-tensorboard
|
||||
pip3 list | grep jupyter
|
||||
# returns:
|
||||
jupyter 1.0.0
|
||||
jupyter-client 5.3.4
|
||||
jupyter-console 6.0.0
|
||||
jupyter-core 4.6.1
|
||||
jupyter-tensorboard 0.1.10
|
||||
jupyterlab 1.2.4
|
||||
jupyterlab-server 1.0.6
|
||||
```
|
||||
|
||||
|
||||
# Usage
|
||||
The main purpose of the script is to evaluate an observation,
|
||||
|
|
@ -296,12 +206,6 @@ The following steps need to be performed:
|
|||
|
||||
1. Rate an observation using the `wut` script.
|
||||
|
||||
# ml.spacecruft.org
|
||||
This server is processing the data and has directories available
|
||||
to sync.
|
||||
|
||||
* https://ml.spacecruft.org/
|
||||
|
||||
## Data Caching Downloads
|
||||
The scripts are designed to not download a waterfall or make a JSON request
|
||||
for an observation it has already requested. The first time an observation
|
||||
|
|
@ -320,18 +224,6 @@ further in the pipeline. This contains `.wav` files that have been
|
|||
decoded from `.ogg` files.
|
||||
|
||||
|
||||
## SatNOGS Observation Data Mirror
|
||||
The downloaded waterfalls are available below via `http` and `rsync`.
|
||||
Use this instead of downloading from SatNOGS to save their bandwidth.
|
||||
|
||||
```
|
||||
# Something like:
|
||||
wget --mirror https://ml.spacecruft.org/download
|
||||
# Or with rsync:
|
||||
mkdir download
|
||||
rsync -ultav rsync://ml.spacecruft.org/download/ download/
|
||||
```
|
||||
|
||||
# Caveats
|
||||
This is the first artificial intelligence script I've done,
|
||||
I know little about radio and less about satellites,
|
||||
|
|
@ -346,4 +238,4 @@ Main repository is available here:
|
|||
|
||||
License: CC By SA 4.0 International and/or GPLv3+ at your discretion. Other code licensed under their own respective licenses.
|
||||
|
||||
Copyright (C) 2019, 2020, Jeff Moe
|
||||
Copyright (C) 2019, 2020, 2022 Jeff Moe
|
||||
|
|
|
|||
Loading…
Reference in New Issue