sndid/README.md

# sndid
`sndid` identifies sounds.

At present only birds are identified.


# Install
Install thusly.

Using Debian Stable (12/Bookworm).

Install thine dependencies:
```
sudo apt update
sudo apt install git ffmpeg python3-pip python3-venv python-is-python3 \
  netcat-traditional sox alsa-utils
```

Clone repo:
```
git clone https://spacecruft.org/deepcrayon/sndid
cd sndid/
```

If you want some sound samples, pull the submodule:

```
git submodule init
git submodule update
```


Set up Python to taste, such as:
```
python -m venv venv
source venv/bin/activate
pip install --upgrade pip setuptools wheel
```

Install Python dependencies.
This will download ~750 megabytes of files to `~/.cache/pip`.
The install will use an additional ~2+ gigabytes in `env/`.

```
pip install -e .
```


# Usage
Note, BirdNet tensorflow works fine with just the CPU, no GPU required to use the model.

## Command line
As such:

```
sndid
```

Help:
```
$ sndid -h
usage: sndid.py [-h] [-i INPUT] [-t LATITUDE] [-n LONGITUDE] [-y YEAR] [-m MONTH] [-d DAY] [-c CONFIDENCE] [-l | --list | --no-list]

Run sndid

options:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Input filename to process (default samples/sample.wav)
  -t LATITUDE, --latitude LATITUDE
                        Latitude (default 40.57)
  -n LONGITUDE, --longitude LONGITUDE
                        Longitude (default -105.23)
  -y YEAR, --year YEAR  Year (default today)
  -m MONTH, --month MONTH
                        Month (default today)
  -d DAY, --day DAY     Day (default today)
  -c CONFIDENCE, --confidence CONFIDENCE
                        Minimum Confidence (default 0.50)
  -l, --list, --no-list
                        Output as human readable list not terse (default True)
```

Sample output:

```
$ sndid
Hairy Woodpecker, Dryobates villosus, 15.0, 18.0, 0.8371534943580627
Hairy Woodpecker, Dryobates villosus, 18.0, 21.0, 0.8111729025840759
Hairy Woodpecker, Dryobates villosus, 30.0, 33.0, 0.50068598985672
Hairy Woodpecker, Dryobates villosus, 33.0, 36.0, 0.7170186042785645
Hairy Woodpecker, Dryobates villosus, 39.0, 42.0, 0.6576249003410339
Hairy Woodpecker, Dryobates villosus, 48.0, 51.0, 0.8048814535140991
Hairy Woodpecker, Dryobates villosus, 51.0, 54.0, 0.9604988694190979
Hairy Woodpecker, Dryobates villosus, 54.0, 57.0, 0.8156641125679016
Hairy Woodpecker, Dryobates villosus, 57.0, 60.0, 0.8230038285255432
```


## Server
The sndid-server waits for connections to feed it wav file segments,
then processes the sound.

It prints to the terminal and also logs to `sndid.log`.

Run thusly:

```
sndid-server
```

Help:
```
$ sndid-server -h
usage: sndid-server.py [-h] [-i IP] [-p PORT] [-t LATITUDE] [-n LONGITUDE] [-y YEAR] [-m MONTH] [-d DAY] [-c CONFIDENCE]

Run sndid-server

options:
  -h, --help            show this help message and exit
  -i IP, --ip IP        Server IP address (default 127.0.0.1)
  -p PORT, --port PORT  Server network port (default 9988)
  -t LATITUDE, --latitude LATITUDE
                        Latitude (default 40.57)
  -n LONGITUDE, --longitude LONGITUDE
                        Longitude (default -105.23)
  -y YEAR, --year YEAR  Year (default 2023)
  -m MONTH, --month MONTH
                        Month (default 9)
  -d DAY, --day DAY     Day (default 19)
  -c CONFIDENCE, --confidence CONFIDENCE
                        Minimum Confidence (default 0.25)
```

Sample output on startup:

```
$ sndid-server
2023-10-03 10:45:47.343270-06:00 sndid-server started 127.0.0.1:9988
```

After client connects and sends mono wav:

```
$ sndid-server 
2023-10-03 10:45:47.343270-06:00 sndid-server started 127.0.0.1:9988
2023-10-03 10:46:34.060983-06:00 Hairy Woodpecker, Dryobates villosus, 0.3896884322166443
2023-10-03 10:46:34.061248-06:00 Hairy Woodpecker, Dryobates villosus, 0.47060683369636536
2023-10-03 10:46:34.061312-06:00 Hairy Woodpecker, Dryobates villosus, 0.5013241171836853
2023-10-03 10:46:34.061377-06:00 Hairy Woodpecker, Dryobates villosus, 0.6557420492172241
2023-10-03 10:46:34.061429-06:00 Hairy Woodpecker, Dryobates villosus, 0.7146830558776855
2023-10-03 10:46:34.061498-06:00 Hairy Woodpecker, Dryobates villosus, 0.806126594543457
2023-10-03 10:46:34.061553-06:00 Hairy Woodpecker, Dryobates villosus, 0.8105885982513428
2023-10-03 10:46:34.061628-06:00 Hairy Woodpecker, Dryobates villosus, 0.8147749900817871
2023-10-03 10:46:34.061678-06:00 Hairy Woodpecker, Dryobates villosus, 0.8241879343986511
2023-10-03 10:46:34.061725-06:00 Hairy Woodpecker, Dryobates villosus, 0.837184488773346
2023-10-03 10:46:34.061771-06:00 Hairy Woodpecker, Dryobates villosus, 0.9604253768920898
```

Sample server output from a realtime stream:

```
$ sndid-server
2023-10-03 10:38:52.975515-06:00 sndid-server started 127.0.0.1:9988
2023-10-03 10:41:27.162509-06:00 Northern Flicker, Colaptes auratus, 0.2552548944950104
2023-10-03 10:41:27.162775-06:00 White-crowned Sparrow, Zonotrichia leucophrys, 0.3168713450431824
2023-10-03 10:41:27.172693-06:00 White-crowned Sparrow, Zonotrichia leucophrys, 0.3708573877811432
2023-10-03 10:41:27.172858-06:00 White-crowned Sparrow, Zonotrichia leucophrys, 0.4243549406528473
2023-10-03 10:41:38.446815-06:00 White-crowned Sparrow, Zonotrichia leucophrys, 0.2577541172504425
2023-10-03 10:41:49.688914-06:00 White-crowned Sparrow, Zonotrichia leucophrys, 0.3252100646495819
2023-10-03 10:42:45.590852-06:00 Rock Wren, Salpinctes obsoletus, 0.34000715613365173
2023-10-03 10:43:52.545070-06:00 Spotted Towhee, Pipilo maculatus, 0.31051698327064514
2023-10-03 10:44:15.007297-06:00 Clark's Nutcracker, Nucifraga columbiana, 0.6180582642555237
2023-10-03 10:44:15.007515-06:00 Northern Flicker, Colaptes auratus, 0.35758695006370544
2023-10-03 10:44:26.109497-06:00 Clark's Nutcracker, Nucifraga columbiana, 0.27328306436538696
2023-10-03 10:44:26.109734-06:00 Clark's Nutcracker, Nucifraga columbiana, 0.32902488112449646
2023-10-03 10:44:26.109799-06:00 Clark's Nutcracker, Nucifraga columbiana, 0.6783570647239685
```

## Client
Requires mono wav file.

To convert stereo to mono with sox:

```
sox -c 2 stereo.wav -c 1 mono.wav
```


Run client to submit file to server thusly:

```
sndid-client
```

Help:
```
$ sndid-client -h
usage: sndid-client [-h] [-i IP] [-p PORT] [-f FILE]

Run sndid-client

options:
  -h, --help            show this help message and exit
  -i IP, --ip IP        Server IP address (default 127.0.0.1)
  -p PORT, --port PORT  Server network port (default 9988)
  -f FILE, --file FILE  Input filename to process (default samples/mono.wav)
```

Sample output:

```
$ sndid-client
Sending samples/mono.wav to 127.0.0.1:9988
```

## ALSA Client
Use this script to stream from the microphone to the server,
using ALSA.

```
sndid-alsa
```


Help:

```
$ sndid-alsa -h
usage: sndid-alsa [-h] [-i IP] [-p PORT] [-r RATE] [-t TIME]

Run sndid-alsa

options:
  -h, --help            show this help message and exit
  -i IP, --ip IP        Server IP address (default 127.0.0.1)
  -p PORT, --port PORT  Server network port (default 9988)
  -r RATE, --rate RATE  Rate in Hertz (default 48000)
  -t TIME, --time TIME  Length of segments in seconds (default 10)
```

Sample output:

```
$ sndid-alsa 
Streaming ALSA in to 127.0.0.1:9988
Recording WAVE 'stdin' : Float 32 bit Little Endian, Rate 48000 Hz, Mono
```

Exit with `CTRL-C`.

## Stream
`sndid-stream` streams *from* a URL to the `sndid-server`.
Input URL can be anything ffmpeg can read (everything).


Help:
```
$ sndid-stream -h
usage: sndid-stream [-h] [-i IP] [-p PORT] [-t TIME] -u URL

Run sndid-stream

options:
  -h, --help            show this help message and exit
  -i IP, --ip IP        Server IP address (default 127.0.0.1)
  -p PORT, --port PORT  Server network port (default 9988)
  -t TIME, --time TIME  Length of segments in seconds (default 60)
  -u URL, --url URL     Input url
```

Exit with `CTRL-Z` and `kill %1`  :) por ahora.


## List
`sndid-list` lists sounds available to be identified at a particular
time and location.

XXX This script is foobar at the moment, as it uses a different
model than the other scripts.

Use:

```
sndid-list
```

Help:
```
$ sndid-list -h
usage: sndid-list.py [-h] [-t LATITUDE] [-n LONGITUDE] [-y YEAR] [-m MONTH] [-d DAY] [-g | --geo | --no-geo] [-c | --cal | --no-cal]

Run sndid-list

options:
  -h, --help            show this help message and exit
  -t LATITUDE, --latitude LATITUDE
                        Latitude (default 40.57)
  -n LONGITUDE, --longitude LONGITUDE
                        Longitude (default -105.23)
  -y YEAR, --year YEAR  Year (default today)
  -m MONTH, --month MONTH
                        Month (default today)
  -d DAY, --day DAY     Day (default today)
  -g, --geo, --no-geo   Limit list by geocoordinates (default False)
  -c, --cal, --no-cal   Limit list by calendar date (default False, use with --geo)
```

Sample output:

```
$ sndid-list -c -g
American Robin, Turdus migratorius
Black-billed Magpie, Pica hudsonia
Black-capped Chickadee, Poecile atricapillus
Blue Jay, Cyanocitta cristata
Canada Goose, Branta canadensis
Common Raven, Corvus corax
Eurasian Collared-Dove, Streptopelia decaocto
European Starling, Sturnus vulgaris
House Finch, Haemorhous mexicanus
Mallard, Anas platyrhynchos
Mountain Bluebird, Sialia currucoides
Mountain Chickadee, Poecile gambeli
Mourning Dove, Zenaida macroura
Northern Flicker, Colaptes auratus
Red-tailed Hawk, Buteo jamaicensis
Turkey Vulture, Cathartes aura
Western Meadowlark, Sturnella neglecta
White-crowned Sparrow, Zonotrichia leucophrys
Wilson's Warbler, Cardellina pusilla
Yellow-rumped Warbler, Setophaga coronata
```

The current count of all species in model:

```
$ sndid-list |& wc -l
6522
```


## JACK
`sndid-jack` needs work.

It still causes xruns and chops up the sample and is less
than ideal, but it runs and IDs.

I feed Master out from Ardour into this JACK input.

I haven't added command line arguments and all that to it yet.

```
sndid-jack
```


# Development
To "develop", install the requirements:

Run black on the Python files for nice formatting:

```
black src/sndid/*.py
```


# Upstream
## Birds
### BirdNet
* BirdNet is the main upstream project:
https://birdnet.cornell.edu/

* The source code is available here:
https://github.com/kahst/BirdNET-Analyzer

Unfortunately the project has a horrible license, CC-NC-BY-SA 4.0,
which isn't even a software license. The authors of the license
themselves say it shouldn't be used for software.
It's not just the data files,
such as the models that are NC (which would be ok), but the source
code itself is NC.
I haven't double checked the Python imports, but it would be interesting
to see if they are even compatible with the NC license,
since the NC license is incompatible with so many Free Software
and Open Source Software licenses.
There is a bug open to fix the license, and change it to Apache 2.0,
which would be sane.
I hope this happens:

https://github.com/kahst/BirdNET-Analyzer/issues/154


### birdnetlib
birdnetlib is based on BirdNet, but with a different codebase and author.
birdnetlib uses BirdNet's non-libre NC model files.
birdnetlib has a dependency on the non-free BirdNet-Analyzer Python code
just for testing.

The source code to birdnetlib itself is
under the Apache 2.0 license.
birdnetlib is Free Software / Open Source Software, with non-libre dependency
for testing.


https://github.com/joeweiss/birdnetlib

In sum, AFAICT, building upon birdnetlib, then creating a libre model,
would be a fully libre system without any non-libre dependencies.


# Status
Alpha, initial development.

* The system ran from October, 2023 to May, 2024
and generated 671,080 identifications.

* Analyzing files works.

* Using `sndid-server` then `sndid-stream` works best now for "realtime",
but is kludgy.

* `sndid-jack` can work sometimes, but causes xruns and still needs
work for non-blocking processing of the sound.


# Disclaimer
I'm not a programmer and I know less about birds.

## AI
Starting on October 19th, 2023, I switched from `vi`, which I've used since the 1990s to `vscodium`, an "open source" build of `vscode`. I set up a self-hosted `tabbyml` server and am using it for completion. It has been more "in the way" for me at the moment, since I am hardened on `vi`, but I am getting accustomed to it and will likely experiment with using more "AI" generated code.

I also have set up an `ollama` server, but haven't used code generated from it for `sndid` yet.

Many of the models don't quite work yet. The larger ones can produce perhaps usable code, but strain system resources and need a lot of GPU VRAM. `ollama`, which is in GO, and `tabbyml`, which is in Rust can use a lot of the same models. Both use `llama.cpp` and `tabblyml` also uses `CTranslate2`.

 I have been using the so-called "open source" models, but many of them aren't under licenses listed by the Open Source Initiative (OSI) nor the Free Software Foundation (FSF) and aren't Wikipedia nor Debian "compatible". Ideally CC by SA 4.0 International licensed models become more common.

Models tried with `tabbyml`: santacoder, starcoder, vicuna, wizardcoder.

Models tried with `ollama`: codellama, everythinglm, falcon, llama2, mistral, nexusraven, orca-mini, phind-codellama, sqlcoder, starcoder, vicuna, wizardcoder, and wizard-math.


# Copyright
Unofficial project, not related to BirdNet or Joe Weiss.

Upstream sources under their respective copyrights.

# License
Apache 2.0.

*Copyright © 2023, Jeff Moe.*