Update/Write some documentation describing the overall project

CONTRIBUTING.md

@@ -1,2 +1,4 @@
 # Contributing
 To get started, <a href="https://www.clahub.com/agreements/FarmBot/farmbot_os">sign the Contributor License Agreement</a>.
+
+After that, read through the [documentation](docs/index.md) 

README.md

@@ -49,8 +49,8 @@ _Refer to the [software documentation Configurator page](https://software.farm.b
 
 ## Problems?
 
-See the [FAQ](docs/FAQ.md)
-If your problem isn't solved there please file an issue on [Github](https://github.com/FarmBot-Labs/farmbot_os/issues/new)
+See the [FAQ](docs/target_development/target_faq.md)
+If your problem isn't solved there please file an issue on [Github](https://github.com/FarmBot/farmbot_os/issues/new)
 
 ## Security Concerns?
 
@@ -58,5 +58,5 @@ We take security seriously and value the input of independent researchers. Pleas
 
 ## Want to Help?
 
-[Low Hanging Fruit](https://github.com/FarmBot-Labs/farmbot_os/search?utf8=%E2%9C%93&q=TODO)
+[Low Hanging Fruit](https://github.com/FarmBot/farmbot_os/search?utf8=%E2%9C%93&q=TODO)
 [Development](CONTRIBUTING.md)

docs/celery_script/celery_script.md

@@ -1,4 +1,5 @@
 # CeleryScript
+
 CeleryScript is an AST definition of commands, rpcs, and functions that
 can all be executed by Farmbot. The basic syntax is as follows:
 
@@ -16,6 +17,7 @@ optional, `body` should be supplied when working with any and all modules
 in this project.
 
 ## kind
+
 `kind` is the identifier for a command. Examples include:
 * `move_absolute`
 * `sync`
@@ -26,6 +28,7 @@ Each `kind` will have it's own set of rules for execution. These rules will
 define what is required inside of both `args` and `body`.
 
 ## args
+
 `args` is arguments to be passed to `kind`. Each `kind` defines it's own
 set of optional and required `args`. Args can any of the following types:
 * `number`
@@ -42,6 +45,7 @@ executing the parent AST. Examples of `args` include:
 * `milliseconds`
 
 ## body
+
 `body` is the only way a `list` or `array` type is aloud in CeleryScript.
 It may only contain _more CeleryScript nodes_. This is useful for
 enumeration, scripting looping etc. Here's a syntacticly correct example:

docs/glossary.md

@@ -0,0 +1,48 @@
+# FarmBot OS Source Glossary
+
+This file contains a basic glossary of commonly used terms
+
+## FarmBot Specific Terms
+
+* Asset - REST resource stored in Farmbot's database stored on the SD card
+* Arduino Firmware - The code that runs on the Arduino.
+  * [Source](https://github.com/farmbot/farmbot-arduino-firmware)
+* CelleryScript - FarmBot OS's scripting language
+* FarmbBot API/Web App - The REST server FarmBot communicates with
+
+## General Terms
+
+* Elixir - Programming language FarmBot is developed in
+  * [More info](https://elixir-lang.org/)
+  * [Docs](https://hexdocs.pm/elixir/Kernel.html)
+* Erlang - Programming language and VM that Elixir compiles down too
+  * [More info](https://elixir-lang.org/)
+  * [Even more info](#OTP-Terms)
+  * [Docs](https://www.erlang.org/docs)
+* UART - **U**niversal **A**synchronous **R**eceiver/**T**ransmitter. hardware based transport mechanism
+* SSH - **S**ecure **S**hell. 
+* MQTT/AMQP - network protocols for pub/sub data transport
+* HTTP - network protocol for accessing REST resource
+
+## Nerves Specific Terms
+
+* Nerves - Framework that allows cross compilation of Elixir code
+  * [More info](https://nerves-project.org/)
+  * [Docs](https://hexdocs.pm/nerves/getting-started.html)
+* NervesHub - Cloud based firmware management
+  * [More info](https://www.nerves-hub.org/)
+  * [Docs](https://github.com/nerves-hub/documentation)
+* Firmware - Usually refers to the code that gets deployed onto the Raspberry Pi
+
+## OTP Terms
+
+* Beam - Virtual machine that runs compiled Erlang bytecode
+* OTP - Open Telecom Platform. Erlang's runtime libraries
+  * [More info](https://erlang.org/doc/design_principles/des_princ.html)
+* Supervisor - OTP `Process` responsible for supervising `Workers`
+* Worker - OTP `Process` responsible for doing `work`. Usually `Supervised`
+* Process - OTP concept responsible for sending/receiving messages. **everything** is a process in erlang
+* Application - OTP concept responsible for containing many `Supervisor`s and `Worker`s
+* Distribution - OTP concept of networking multiple Beam instances together
+* ETS - **E**rlang **T**erm **S**torage. OTP application for storing data in memory
+* DETS - **D**isk **E**rlang **T**erm **S**torage. OTP application for storing data on disk

docs/index.md

@@ -0,0 +1,38 @@
+# FarmBot OS Documentation
+
+This document will act as an index to available documentation.
+
+## Glossary
+
+* [FarmBot Source Code common terms](/docs/glossary.md)
+
+## Hardware specifics
+
+Most FarmBot development/testing is done on a standard desktop PC. 
+
+* [Developing on your local PC](/docs/host_development/host_development.md)
+* [Deploying on Raspberry Pi](/docs/target_development/building_target_firmware.md)
+  * [Publishing Firmware (OTAs)](/docs/target_development/releasing_target_firmware.md)
+  * [Why doesn't my device boot after building firmware](docs/target_development/target_faq.md)
+  * [Inspecting a running devicve](/docs/target_development/consoles/target_console.md)
+
+## CeleryScript
+
+CeleryScript is FarmBot's native scripting language. See the below 
+documentation for information about it as it relates to FarmBot OS.
+
+* [CeleryScript intro](/docs/celery_script/celery_script.md)
+* [A list of all supported commands](/docs/celery_script/all_nodes.md)
+* [Lua (embedded scripting inside CeleryScript)](/docs/celery_script/assert_expressions.md)
+
+## Project structure
+
+The FarmBot OS application is broken into several sub applications. 
+
+* [Project structure overview](/docs/project/structure.md)
+  * [farmbot_celery_script](/docs/project/farmbot_celery_script.md)
+  * [farmbot_core](/docs/project/farmbot_core.md)
+  * [farmbot_ext](/docs/project/farmbot_ext.md)
+  * [farmbot_firmware](/docs/project/farmbot_firmware.md)
+  * [farmbot_os](/docs/project/farmbot_os.md)
+  * [farmbot_telemetry](/docs/project/farmbot_telemetry.md)

docs/project/farmbot_celery_script.md

@@ -0,0 +1,12 @@
+# FarmBot CeleryScript OTP App
+
+`farmbot_celery_script` is responsible for implementing the 
+runtime that execute's [CeleryScript](/docs/celery_script/celery_script.md). 
+It contains a handful of helpers, and several subsystems for working with CeleryScript. 
+The most important being:
+* AST - definition of the AST as it relates to FarmBot OS
+* Compiler - Compiles CeleryScript to Elixir AST. 
+  * See the [Elixir Macro Docs](https://hexdocs.pm/elixir/Macro.html)
+* StepRunner - Process responsible for actually executing CeleryScript
+* Scheduler - Process responsible for scheduling calls to the `StepRunner`
+* SysCalls - module responsible for dispatching calls to the configured implementation

docs/project/farmbot_core.md

@@ -0,0 +1,58 @@
+# FarmBot Core OTP App
+
+`farmbot_core` is responsible for the core funcionality of the FarmBot application.
+This contains things such as resource (asset) management, plugin (farmware) management,
+central state, and schedule management. FarmBot specific network requests are not made
+from the `farmbot_core` app. Below describes the important subsystems
+
+## Asset storage subsystem
+
+Sqlite database responsible for storing data needed for FarmBot to operate.
+Most device specific REST resources are mirrored here.
+* Device
+* FarmEvent
+* Regimen
+* Sequence
+* Peripheral
+
+## Asset Worker subsystem
+
+All assets that need to have a process assosiated with it will be found
+in this subsystem. Examples of this include:
+* FarmEvent scheduling
+* Regimen scheduling
+* PinBinding monitoring
+* FbosConfig/FirmwareConfig
+
+## Farmware subsystem
+
+Farmbot's external plugin system. See the Farmware documentation for more details.
+
+## BotState subsystem
+
+Central in-memory state process/tracker. This process keeps a cache of
+all the moving parts of FarmBot. Some examples of what is stored
+in this cache:
+
+* Firmware reporting
+  * current axis position
+  * encoder data
+  * arduino pin data
+  * currently configured firmware paramaters
+* Current configuration
+  * mirror of `fbos_config` asset
+* System info
+  * version info
+  * (nerves) firmware info
+  * memory usage
+  * disk usage
+* Network info
+  * WiFi signal quality
+  * private ip address
+
+## Logging subsystem
+
+This is where the `Messages` panel gets it's data from. Calls to this subsystem
+push data into an sqlite database. This data is collected on a timer and dispatched
+over AMQP/MQTT when/if that subsystem is available. See [farmbot_ext](/docs/project/farmbot_ext.md)
+for information on how that works.

docs/project/farmbot_ext.md

@@ -0,0 +1,38 @@
+# FarmBot Ext OTP App
+
+the `farmbot_ext` OTP app contains extended FarmbotCore functionality. This includes
+mostly network functionality that isn't possible to do in `farmbot_core`. 
+
+## Bootstrap subsystem
+
+Subsystem responsible for bootstrapping a connection to the FarmBot network services.
+This includes authenticating with the FarmBot API, connecting to AMQP/MQTT and syncing 
+the bare minimum resources to get up and running.
+
+## HTTP/Sync subsystem
+
+This is the subsystem that syncronizes FarmBot with the remote API. It uses HTTP to
+download an index of all the data FarmBot cares about, and compares timestamps to 
+determine who has the most up to date data. The basic flow is whoever has the most 
+recent `updated_at` field will become the "most truthy". If FarmBot has a more recent
+`updated_at` field, FarmBot will do an HTTP PUT of it's data. If the remote resource
+does not exist, FarmBot will do an HTTP POST of it's data. If the remote data has a more
+recent `updated_at` field, FarmBot will do an HTTP GET and replace it's own data.
+
+## AMQP/MQTT substem
+
+FarmBot maintains a connection to the API for real time communication. This real time communication
+connection is multiplexed over multiple `channel`s. Below is a description of the channels
+
+* bot_state - pushes a JSON encoded version of the `bot_state` process (from `farmbot_core`)
+* celery_script - receives/sends JSON encoded celery_script. Used for controling FarmBot externally
+* log - sends log messages from `farmbot_core`'s logger
+* ping/pong - echos everything received. used for detecting active conncetion
+* auto_sync - the API dispatches every REST resource change on this channel. Used to speed up HTTP requests
+* telemetry - similar to the log channel, but sends consumable events, rather than human readable messages
+
+## Image uploader subsystem
+
+This subsystem watches a local directory, and as matching files appear in that directory, 
+it uploads them using the FarmBot image upload protocol. Basically an HTTP request to fetch
+credentials that are used to preform another HTTP request to upload the photo. 

docs/project/farmbot_firmware.md

@@ -0,0 +1,23 @@
+# FarmBot Firmware OTP App
+
+The `farmbot_firmware` OTP application is responsible for maintaining a connection to
+the arduino-firmware.
+
+## GCODE encoder/decoder subsystem
+
+The official Farmbot-Arduino-Firmware communicates over UART using a ASCII based
+protocol based on CNC GCODE. This subsystem is responsible for translating
+the ASCII data into an intermediate representation that can be `transport`ed
+proto agnostically. 
+
+## Transport subsystem
+
+This subsystem is responsible for abstracting the details of transporting 
+FarmBot GCODEs to/from the firmware implementation. A `transport` will take 
+in the intermediate (farmbot specific) representation of a GCODE, and dispatch/handle
+it in it's own specific manor. This keeps the usage of the overall application uniform
+with or without a firmware plugged in. 
+
+## UART subsystem
+
+Responsible for the official communication mechanism with the official arduino hardware.

docs/project/farmbot_os.md

@@ -0,0 +1,50 @@
+# FarmBot OTP App
+
+the `farmbot` OTP app is the container Nerves based application. It contains mostly
+glue code between all the subsystems in the other applications along with it's own 
+platform specific subsystems.
+
+## CeleryScript System Calls
+
+The "official" implementation of all the CeleryScript syscalls. These calls are mostly
+glue to other existing implementations from the other otp apps.
+
+## Lua Subsystem
+
+The implementation of the embedded scripting language inside CeleryScript.
+Also contains glue code for glueing together the real implementation to the
+Lua vm.
+
+## Configurator Subsystem
+
+HTTP server responsible for configuring the running FarmBot OS instance. Will
+server a web page that allows a user to supply a username, password, network credentials
+etc.
+
+# Platform specific subsystems
+
+the `farmbot_os` OTP app contains target/hardware specific systems. This code is
+located in the `platform` directory.
+
+## Network subsystem
+
+Responsible for getting FarmBot connected to the (inter)net. If no network configuration is
+available, FarmBot will create a captive portal access point to allow external devices to 
+configure it.
+
+## GPIO subsystem
+
+Responsible for implementing LED and Button support at the hardware level.
+
+## RTC subsystem
+
+Responsible for syncronizing network time to an attached hardware clock.
+
+## Info Worker subsystem
+
+Responsible for simple workers that handle things like
+
+* CPU temperature
+* CPU usage
+* memory usage
+* disk usage

docs/project/structure.md

@@ -0,0 +1,33 @@
+## FarmBot Source Project structure
+
+The FarmBot OS application is broken into several sub OTP applications. 
+
+* [farmbot_celery_script](/docs/project/farmbot_celery_script.md)
+* [farmbot_core](/docs/project/farmbot_core.md)
+* [farmbot_ext](/docs/project/farmbot_ext.md)
+* [farmbot_firmware](/docs/project/farmbot_firmware.md)
+* [farmbot_os](/docs/project/farmbot_os.md)
+* [farmbot_telemetry](/docs/project/farmbot_telemetry.md)
+
+## Commonality
+
+All of these folders share a common structure.
+
+<OTP APP ROOT>
+├── lib/
+│   ├── application.ex
+│   └── some_file.ex
+|
+├── test/
+|   └── test_helper.exs
+|
+├── config/
+|   └── config.exs
+|
+├─── mix.exs
+└─── mix.lock
+
+* The `lib` folder contains Elixir source code
+* the `test` folder contains Elixir scripts responsible for testing the `lib` code
+* the `config` folder contains Elixir scripts responsible for configuring the **current** OTP app
+* `mix.exs` and `mix.lock` files are responsible describing the OTP app, and managing external dependencies

docs/target_development/consoles/erlang_distribution.md

@@ -0,0 +1,26 @@
+# Erlang Distribution Console
+
+Accessing an Erlang Distribution console
+
+## Setup
+
+No additional setup is required for setting up erlang distribution on the target.
+On your `host` machine, you need to have Erlang Distribution running. This can
+usually be accomplished by starting `epmd`:
+
+```bash
+epmd --daemon
+```
+
+## Connecting
+
+From your `host` terminal, connecting to a running device can be done by using the
+`remsh` feature of elixir's built in console.
+
+```bash
+iex --name console --cookie democookie --remsh farmbot@farmbot-<SERIAL_NUMBER>.local
+```
+
+## Disconnecting
+
+Issuing a `ctrl+c` to the `host` terminal should disconnect you from the session. 

docs/target_development/consoles/ssh.md

@@ -1,42 +1,25 @@
-# SSH
+# SSH console
+
+Accessing an SSH console.
 
 ## Setup
+
 FarmBot can be configured to start an SSH server to aid in debugging and development.
 During configuration of Network, select `Advanced Settings` and paste your [ssh
 public key](https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key) into the
 optional input section labeled: `id_rsa.pub`. FarmBot requires a public key and
-will not allow a username + password combination. FarmBot also only allows one
-key to be configured for security reasons.
+will not allow a username + password combination. If you followed the documentation
+described in [building target firmware](/docs/target_development/building_target_firmware.md)
+then your SSH key will be automatically added to the device.
 
 ## Connecting
+
 From the same machine that owns the `id_rsa.pub` key and assosiated private key
 you can simply `ssh <ip address>`. If your machine supports `mdns`, you can also
 do `ssh farmbot-<node_name>` where `node_name` can be found in the `Device` panel
 on the FarmBot web app.
 
-## Usage
-The console a user will be presented with is _not_ a Linux console. There are
-pretty much no Linux Utilities built-in. This includes but is not limited to:
-* `bash`
-* `apt-get`
-* `make`
-* `screen`
-* `vi`
-* `cp`
-* `mkdir`
-* `ln`
-* `echo`
-* etc
-
-What is available is a console to the FarmBot OS runtime. You will need to be
-familiar with the FarmBotOS Source code for this to be helpful.
-
-If all you are looking for is Logs, you will probably want to do:
-```elixir
-RingLogger.attach()
-```
-
-After that command you will see logs come across the screen in real time.
+## Disconnecting
 
 To exit the SSH session, type `~.`.
 This is an ssh escape sequence (See the ssh man page for other escape sequences).

docs/target_development/consoles/target_console.md

@@ -0,0 +1,42 @@
+# FarmBot target console
+
+The FarmBot OS target console is a repl for interacting with a running
+farmbot instance. Depending on your development configuration, there are several
+ways to access a console.
+
+If you are using `host` mode, your console will be presented on stdin of your
+terminal.
+
+If you are on `target` mode (IE: deployed to the raspberry pi), there will
+be a console available in 3 locations:
+* UART (except on RPI0 since the on-board UART is used for the arduino-firmware)
+  * [Connect to a UART console](/docs/target_development/consoles/uart.md)
+* SSH
+  * [Connect to an SSH console](/docs/target_development/consoles/ssh.md)
+* Erlang Distribution
+  * [Connect to an Erlang Distribution console](/docs/target_development/consoles/erlang_distribution.md)
+
+## Console Usage
+
+The console a user will be presented with is _not_ a Linux console. There are
+pretty much no Linux Utilities built-in. This includes but is not limited to:
+* `bash`
+* `apt-get`
+* `make`
+* `screen`
+* `vi`
+* `cp`
+* `mkdir`
+* `ln`
+* `echo`
+* etc
+
+What is available is a console to the FarmBot OS runtime. You will need to be
+familiar with the FarmBotOS Source code for this to be helpful.
+
+If all you are looking for is Logs, you will probably want to do:
+```elixir
+RingLogger.tail()
+```
+
+After that command you will see logs come across the screen in real time.

docs/target_development/consoles/uart.md

@@ -0,0 +1,31 @@
+# UART Console
+
+Accessing an target UART console
+
+## Setup
+
+No additional setup is required for setting up the UART console on the target.
+
+On your `host` machine, you need to have a console cable, as well as a console client.
+The console cable must be 3.3 volts, **not** 5 volts. a 5 volt cable will harm your
+Raspberry Pi. Here are some known working cables:
+* https://www.amazon.com/Converter-Terminated-Galileo-BeagleBone-Minnowboard/dp/B06ZYPLFNB
+* https://www.amazon.com/DSD-TECH-Adapter-FT232RL-Compatible/dp/B07BBPX8B8
+* https://www.amazon.com/JANSANE-PL2303TA-Serial-Console-Raspberry/dp/B07D9R5JFK
+
+The most common client is probably `screen` on *Nix based systems or `putty` on windows. 
+See your distribution's package manager for installation and usage instructions.
+
+## Connecting
+
+Connecting to a console is dependent on your particular client. Here is an example 
+`screen` command:
+
+```bash
+screen /dev/ttyUSB0 115200
+```
+
+## Disconnecting
+
+Disconnecting is also dependent on your particular client. To exit screen, 
+issue a `ctrl+\+y` sequence to escape the console.