Commit Graph

217 Commits (staging)

Author SHA1 Message Date
Connor Rigby 209ded96cb
Add Lua helpers for updating and getting resources 2019-09-13 16:03:32 -07:00
Connor Rigby a242cce830
Add new log type for assertion logs 2019-09-13 16:03:32 -07:00
Connor Rigby b173b70675
Fix PATCH HTTP requests for singular resources 2019-09-13 16:03:31 -07:00
Connor Rigby 61d63afb17
Fix typo AMPQ -> AMQP 2019-09-13 16:03:31 -07:00
Connor Rigby 86843c4553
Make FirstPartyFarmware syncable 2019-09-13 16:03:31 -07:00
connor rigby 21eef22c55
Fix DirtyWorker doing patch requests 2019-09-13 16:03:31 -07:00
Gabriel Burnworth 9a3fadcd80
fix image upload bug 2019-09-13 16:03:30 -07:00
connor rigby 82042f98df
Clean up error message for uploading images 2019-09-13 16:03:30 -07:00
connor rigby 5610f2e9e3
Fix typo in PingPong channel reconnect message 2019-09-13 16:03:30 -07:00
Connor Rigby a6f365b4c0
Implement public_key worker 2019-09-13 16:03:30 -07:00
Connor Rigby c35b81decb
Add new table public_keys
This will be to load new ssh keys onto a device to allow
remote console for debugging over a network
2019-09-13 16:03:30 -07:00
connor rigby 302ddd3bf0
Fix race condition in AMQP connecting to PingPong channel 2019-09-13 16:03:30 -07:00
connor rigby 3a5666429a
Allow Direct-Upload of images 2019-09-13 16:03:29 -07:00
connor rigby 7b19b73e6f
Revert "Allow direct-upload"
This reverts commit 462681d5da.
2019-09-13 16:03:29 -07:00
Connor Rigby 01c3723877
Fix LEDs and Pinbindings working intermitently
The Internal pullups need to be disabled for the
FarmBot gpio hat to work properly
2019-09-13 16:03:29 -07:00
connor rigby e199cb8a90
Add LED handler for green (sync status) 2019-09-13 16:03:29 -07:00
connor rigby 7fa252e9d7
Add LED handler for blue (connection status) 2019-09-13 16:03:29 -07:00
connor rigby bb804fd510
Allow direct-upload
Tesla HTTP client is formatting the formdata wrong for some reason
so just shell out to hackney
2019-09-13 16:03:29 -07:00
connor rigby 830f6894d8
Add a background timer to do an HTTP for the consistency checker 2019-09-13 16:03:29 -07:00
connor rigby 373a0b2faf
Fix ImageUploader automatic upload
* Serialize DateTime as an iso string for bot_state filesystem
* Set user-agent for uploading images
* Check for more types of images other than jpeg
2019-09-13 16:03:27 -07:00
connor rigby 8b2ac2935e
Create images directory when starting image_uploader 2019-09-13 16:03:27 -07:00
gabrielburnworth ab1b8bbda5
add farmware execution rpc debug logs
Fix AMQP and network disconnects causing limbo state
2019-09-13 16:03:27 -07:00
Connor Rigby de78091f6d
Add device side log filter for bad words
This prevents the frontend from crashing on bad words
2019-09-13 16:03:26 -07:00
Connor Rigby 1900bdba57
Allow firmware to be open/flashed infinity times
* add `infinity` as an option for `fbos_config_worker` and
`firmware_open_task`
* reenable uevent montior flashing/opening firmware
2019-09-13 16:03:26 -07:00
Connor Rigby 5d54795d11
Fix updating a regimen remotely not updating locally
If a user updated a regimen (started or not by a FarmEvent) it would
take two changes to actually dispatch the change since Regimens
update their own `updated_at` without `put`ing it to the API
2019-09-13 16:03:26 -07:00
Connor Rigby 3d9be8d902
Silence repeating firmware open log 2019-09-13 16:03:26 -07:00
Connor Rigby 91ccd93ef0
Re-enable API client UserAgent header 2019-09-13 16:03:26 -07:00
connor rigby 767ef5279d
Silence PingPong channel connection log 2019-09-13 16:03:26 -07:00
connor rigby 712256a1a3
Implement task for opening the firmware
Adds a new flag for `firmware_needs_open` and a task that polls that
value with a bit of state to open the firmware tty
2019-09-13 16:03:25 -07:00
Connor Rigby ce51edeb99
Revert Alert/Enigma system
* Revert 9cc461b512
* Revert 7309a19fe7
* Revert 7fd82d3a58
* Revert 890ddfbd86
* Revert 839ce3e470
* Revert 68cc8e6471
* Revert 3991a2c516
* Revert db8da31018
2019-09-13 16:03:25 -07:00
Connor Rigby a160d6a41b
Fix factory_resetting the arduino-firmware
* Adds `api_migrated` field to FirmwareConfig
* Changes `factory_reset()` syscall to be `factory_reset(package)`
2019-09-13 16:03:25 -07:00
connor rigby d2239d76c3
Try add log channels more aggressively 2019-09-13 16:03:24 -07:00
Connor Rigby 47d024e561
Stabilize AMQP connect and disconnects
* AMQP wasn't disconnected properly, which was causing bots to appear to
be offline because the connection was broken, but not closed.
2019-09-13 16:03:24 -07:00
Connor Rigby 60b075cf88
Normalize asset commans
Many clauses were not getting hit depending on if
`update/3` was called with a Module or a "String".
This normalizes everything early.
2019-09-13 16:03:23 -07:00
Connor Rigby e7ea334ab2
Change Preloading and Caching to use the same system as auto_sync
* preloader and reconciler no longer use a transaction
   * This will prevent rolling back a failed sync, but allows farmbot to
   continue operating if a sync does fail
* usage of the preloader updated to reflect this
2019-09-13 16:03:23 -07:00
Connor Rigby e6b29ea559
Refactor CeleryScript execution
* FarmEvents now schedule all events up front
  * Executions are stored in the database
* Regimens now schedule all items up front
  * Executions are stored in the database
* fix Deleting a FarmEvent with a running RegimenInstance
* `execute`int and `schedule`ing a CeleryScript are not separate things
* Everywhere `scheduling` and `executing` CeleryScript updated to use
the new API
2019-09-13 16:03:23 -07:00
Connor Rigby 890aac46db
Add asset commands for syncing SensorValues
* New Asset.Command for SensorValue
* new/update/delete for SensorValue
2019-09-13 16:03:22 -07:00
Rick Carlino 1730bf8c29
Configurable EXUNIT_TIMEOUT 2019-09-13 16:03:22 -07:00
Connor Rigby d5215b6608
Handle unknown auto_sync messages
Adds a blacklist of known asset types that FarmBot can
handle and sync/cache. This is important for when new assets are added or
assets are dispatched for the frontend to consume.
2019-09-13 16:03:22 -07:00
Connor Rigby eefdbfc883
Further refactor Preloader to allow for error recovery
Previous system would crash the calling processes with a `MatchError`
instead of returning an error. The end result after this commit is
still the same - the auto_sync channel will attempt a resync
every 5 seconds. The log messages/user experience should just
be better now.
2019-09-13 16:03:22 -07:00
Connor Rigby 11220c3697
Refactor Preloader test system
* Gets rid of the Preloader behaviour
  * This prevents the Preloader mock from overriding the _actual_
    implementation.
* Update auto_sync_channel_test to mock the API, not the Preloader
* remove old preloader_test as it didn't do anything
2019-09-13 16:03:22 -07:00
Connor Rigby 9a16c79ee8
Implement error handling for api 429 errors
* Changes auto_sync_channel to no longer use `handle_continue`
  * If preloading fails, retry instead of crashing.
* Make `get_changeset` return an error rather than raising an exception
  * Update every use of that function to handle posibility of error
2019-09-13 16:03:22 -07:00
Connor Rigby 7cda16f144
Fix argument order in auto_sync channel 2019-09-13 16:03:22 -07:00
Connor Rigby 2105fb8a70
Fix NervesHubChannel compile warning 2019-09-13 16:03:22 -07:00
Connor Rigby b6120041ef
Rename bot_state_channel to reflect the previous commit 2019-09-13 16:03:22 -07:00
Connor Rigby 3ff7a78f68
Revert multiple state channels over amqp 2019-09-13 16:03:22 -07:00
Rick Carlino b619cb3576
Add scafold tests for celery_script_channel
* Test case: `celery_script_channel` `start_link`
2019-09-13 16:03:22 -07:00
Connor Rigby 26578a7ae2
Update `mox` to be used correctly.
I previously misunderstood how the black magic of mox actually works.
This updates `farmbot_ext` to not require setting excess data in every
config.exs entry. Also removes calls to `Application.get_env/2`
2019-09-13 16:03:22 -07:00
Connor Rigby 2d66f09485
Enable `warnings_as_errors` for each app 2019-09-13 16:03:21 -07:00
Connor Rigby 1483a3881f
Cleanup farmbot_ext auto_sync_channel
Also add some documentation for stubable
modules Query and Command
2019-09-13 16:03:21 -07:00
Connor Rigby 686c9be7c9
Fix process naming colision 2019-09-13 16:03:21 -07:00
Rick Carlino ab9e434da8
Cleanup of auto_sync tests
* Move EagerLoader and friends back into AutoSyncChannel to avoid cross cutting
2019-09-13 16:03:21 -07:00
Rick Carlino c3819359b0
Test auto_sync ignore case
* Test case: auto_sync is disabled.

* Test case: auto_sync of Device assets

* Test case: auto_sync of FbosConfig

* Test case: auto_sync of FbosConfig assets

* Test case: auto_sync of FarmwareInstallation assets

Test case: `cache_sync/3`
2019-09-13 16:03:21 -07:00
Connor Rigby 1ec7e429bb
Set sync status to synced after any auto sync handler 2019-09-13 16:03:21 -07:00
Connor Rigby 5ef96a8db3
Update sync_status after preload in auto_sync channel 2019-09-13 16:03:21 -07:00
Connor Rigby 57f4927973
Make bot_state_channel send data on legacy, and new channel 2019-09-13 16:03:21 -07:00
Connor Rigby c664b10627
Remove noisy ping/pong calls 2019-09-13 16:03:21 -07:00
Connor Rigby 6ffdc43c6e
Remove changeset from logger call in asset dirty worker 2019-09-13 16:03:21 -07:00
Connor Rigby 6cde33012c
Add initial support for a new PingPong channel. 2019-09-13 16:03:21 -07:00
Connor Rigby 7e5a7d557f
Change the v8 channel name 2019-09-13 16:03:21 -07:00
Rick Carlino 8c7d6f6608
AutoSyncChannel Testing: Device, `auto_sync = false` (#714)
* Test auto_sync ignore case

* Test case: auto_sync is disabled.

* Test for auto_sync of Device assets
2019-09-13 16:03:21 -07:00
Rick Carlino 4248c93962
Add mox and excoveralls
* and tests for AutoSyncChannel
* add tests for auto sync preloader
2019-09-13 16:03:20 -07:00
connor rigby f6b1839ffe
Quick example on how to use mox 2019-09-13 16:03:20 -07:00
Rick Carlino 0c75d61d2d
Add mox, excoveralls and a test for AutoSyncChannel 2019-09-13 16:03:20 -07:00
connor rigby b968827d4b
Add compiler warning for preloader impl 2019-09-13 16:03:20 -07:00
connor rigby dd05ffc40e
Quick example on how to use mox 2019-09-13 16:03:20 -07:00
connor rigby ac2a5a67e9
Add coveralls aggregator 2019-09-13 16:03:20 -07:00
Connor Rigby f893611dd0
Consolidate NervesHub logic and implementation 2019-09-13 16:03:20 -07:00
Connor Rigby 4fff304871
Change force state pushes to use the old method of publish the entire
state.
2019-09-13 16:03:19 -07:00
Connor Rigby 199e1edb97
Rename Enigma to Alert 2019-09-13 16:03:19 -07:00
Connor Rigby 1b31c32b3d
Implement change_ownership sys_call
This required some changes to NervesHub because of there
being a brief moment after changing ownership where there
are no certs.
2019-09-13 16:03:19 -07:00
Connor Rigby 00003c2a96
Implement dump_info sys_call 2019-09-13 16:03:19 -07:00
Connor Rigby 2c73a9b2d8
Update job_progress to not use atoms 2019-09-13 16:03:19 -07:00
Connor Rigby c711df2dfc
Remove botstate channel 2019-09-13 16:03:19 -07:00
Connor Rigby 33c0046fff
Update auto_sync_channel to cast updates to respective workers 2019-09-13 16:03:18 -07:00
Connor Rigby 35996f571f
Add dummy test to hopefully help CI 2019-09-13 16:03:18 -07:00
Connor Rigby 4c611d3faa
Format farmbot_ext/mix.exs 2019-09-13 16:03:18 -07:00
Connor Rigby a228805caa
Update AMQP to 1.1.1 2019-09-13 16:03:18 -07:00
Rick Carlino 399ddf50c8
PersistentRegimen => RegimenInstance 2019-09-13 16:03:17 -07:00
Rick Carlino 295cb45d49
Transport => Channel 2019-09-13 16:03:17 -07:00
Rick Carlino 62173829e9
AutoSyncTransport => AutoSyncChannel 2019-09-13 16:03:17 -07:00
Rick Carlino bca13f600b
LogTransport => LogChannel 2019-09-13 16:03:17 -07:00
Rick Carlino 673d8a834c
Add docs to modules that were missing them 2019-09-13 16:03:17 -07:00
Connor Rigby d4245cf2df
Fix/update dirty worker 2019-09-13 16:03:17 -07:00
Connor Rigby d2c5a1a72b
Add enigma_worker config to farmbot_ext and frambot_os 2019-09-13 16:03:16 -07:00
Connor Rigby 1e7ae7a744
Finish implementing Farmware.
Farmware runtime seems to be working really well. The only
problem is currently the way Farmware is stored on the bot state
can not be properly serialized to the new method. This will require
a breaking change
2019-09-13 16:03:16 -07:00
connor rigby 218b33785d
Fix more namespaces 2019-09-13 16:03:16 -07:00
Connor Rigby 08cec79db6
Rename FarmbotExt Namespace 2019-09-13 16:03:16 -07:00
Connor Rigby 21bb1a6f25
Misc cleanup and bug fixes.
* Fixes botstate changes being dispatched as nested lists
* Fixes jwt failing to decode `nil`. Doesn't fix token actually being`nil` though.
* Adds a syscall for read_pin
* Add/fix calibrate syscall
* tweak tty detector. This is still broken i think
2019-09-13 16:03:16 -07:00
Connor Rigby 4114e26804
Implement new CeleryScript Runtime environment.
This is obviously a rather large change warranting an essay describing
it.

A Brief overview
Basically the old implementation had quite a few down sides preventing
it from really working as intended, especially with the addition of the
variables feature. Here is the shortlist of things that needed
addressing:

* No scoping between sequences.
What this essentially means is that a sequence that executes another
sequence is unable to add data to the calle. This is important for using
Variables.

* Error recovery
certain nodes have a high likelyhood of failing such as anything that
interfaces the firmware. Much focus was spent ensuring that errors would
be recoverable when desired.

* Complexity of control flow asts versus action asts.
Nodes such as `if` will always work in the same way regardless of the
state of the rest of the system meaning there is no reason for it to
have a special implementation per environment. on the other hand
`move_absolute` is bound to a specific part of the system. Seperating
these concerns allows for better testing of each piece independently.

A More In Depth overview
The core of this change resolves around 1 really big change resulting in
many more small changes. This change is the CeleryScript `compiler`. The
TLDR of this system is that now CeleryScript ASTs are deterministicly
compiled to Elixir's AST and executed. Doing this has some big benifits
as described below.

1) CeleryScript "runtime" environment is now much simpiler in favor of a somewhat
complex "compile time" environment. Basically instead of EVERY single
CeleryScript AST having a custom runtime implementation, only a subset
of ASTs that require external services such as the Firmware, Database,
HTTP, etc require having a runtime implementation. This subset of ASTs
are called `SysCalls`. Also the runtime implementations are compiled to
a single function call that can be implemented instead of needing to
have a contextual environment and making decisions at runtime to
evaluate variables and the like.

2) Static analysis is now possible. This means an incorrectly crafted
sequence can be validated at compile time rather than getting half way
through a sequence before finding the error.

3) Having the "external services" separated leads to better plugability.
There is now a behaviour to be implemented for the subset of syscalls
that are system specific.
2019-09-13 16:03:15 -07:00
connor rigby 09be64fe4e
Fix AMQP issue with hanging forever 2019-09-13 16:03:15 -07:00
connor rigby d0edb1e19e
Update to Elixir 1.8!! 2019-09-13 16:03:15 -07:00
connor rigby 87c3ecf83b
Pull server location from config instead of jwt 2019-09-13 16:03:15 -07:00
Connor Rigby bb7306dabc
Work towards refactoring Farmware
* Add bot state serializer to filesystem
* Add helper to turn ecto types into TSC types
* Add module to extract ecto changes as a list of instructions
* Add new transport for bot_state_ng
* Update CSVM Runtime to poll until `await` can complete
2019-09-13 16:03:15 -07:00
Connor Rigby 8f3abc335a
Update NervesHub logs and abstractions 2019-09-13 16:03:14 -07:00
Connor Rigby cf5077e2a9
Start implementing firmware startup
Misc other cleanup
2019-09-13 16:03:14 -07:00
Connor Rigby 7cde2b2390
Change to ElixirCircuits. 2019-09-13 16:03:14 -07:00
Connor Rigby 56f6749595
Nerves hub pt2 (#654)
* Rework Networking

* Rework Configurator

* Fix NervesHub not connecting
2019-09-13 16:03:14 -07:00
Connor Rigby bfa1b7f966
Add some logs to connection_worker to help debug race 2019-09-13 16:03:14 -07:00
Connor Rigby 6e2a018598
Implement Regimens again, add tests and test support
* This also includes changes to all Farmbot API Assets adding
  a `monitor: :boolean` field. This allows tests to be run
  causing minimal side effects on the rest of the systems.
2019-09-13 16:03:14 -07:00
Connor Rigby a4c1eac2d2
Update image uploader to add `file_type` 2019-09-13 16:03:14 -07:00
Connor Rigby f2b8abd692
Botstate refactor (#645)
* Refactor PinBindings

* Refactor logging to not require Farmbot.Registry

* Fix AMQP bot_state_transport and log_transport

* Update to use floats everywhere

* Refactor Farmbot.Firmware

* Write Firmware Tests

* dipping into FarmbotOS finally

* Cleanup peripheral_worker error message

* Implement remaining CeleryScript RPCs

* Refactor job progress

* Image Upload status notifications updates

* Fix compiler warnings and things

* Update Firmware submodule

* Fix FarmbotEXT tests
2019-09-13 16:03:14 -07:00
Connor Rigby 3b8f8d591f
Config + dead code cleanup 2019-09-13 16:03:14 -07:00
Connor Rigby b3f1369a77
Re-implement image uploading 2019-09-13 16:03:14 -07:00
Connor Rigby 83be454f94
Progress on Refactoring farmware 2019-09-13 16:03:13 -07:00
Connor Rigby 869d1ad1a4
Progress on refactoring data syncing (#643) 2019-09-13 16:03:13 -07:00
connor rigby 8c34f2eacd
Refactor farmbot application into separate folders/apps. 2019-09-13 16:03:13 -07:00
Connor Rigby 0d569301f1
Replace httpoison 2019-09-13 16:03:13 -07:00
Connor Rigby 690ae222e8
Fix compile error 2019-09-13 16:03:13 -07:00
Connor Rigby 512f9d8910
Fix AMQP opening a new connection without closing the older ones 2019-09-13 16:03:13 -07:00
connor rigby 10a4379568
Work toward farmware/celeryscript stuff
* Fix logger erros
* Fix migrations
* Fix compile errors
* Fix tzdata child_spec
2019-09-13 16:03:13 -07:00
Connor Rigby bdb11b2ea8
Lots of work on Farmware 2019-09-13 16:03:13 -07:00
connor rigby 66bb4e51f4
Update nerves_time 2019-09-13 16:03:13 -07:00
connor rigby d7ff13422c
Test Migration 2019-09-13 16:03:13 -07:00
connor rigby 32f46b5fd3
Start refactoring Farmware 2019-09-13 16:03:13 -07:00
connor rigby 358a1e209e
Update AMQP workers to reconnect on a 4 second timer
Pull in new csvm implementation

Update circleci config

Implement syncing and write_pin

add migration for syncing

add saftey to write_pin

Implement read_pin

Implement set_servo_angle

Implement more ast nodes

Implement e-stop and e-unlock

Fix a bunch of stuf

Fix missing assets on boot/init

It actually works

Rename csvm -> farmbot_celery_script; fix initial sync/dispatch

Fix a bunch of small bugs

Identify problem

Fix Sqlite bug and increase performance by 10 times!!

Fix sequences inside of sequences
2019-09-13 16:03:13 -07:00
connor rigby cf1ef23b17
Start splitting Farmbot Application Poncho Style
This begins splitting the monolithic Farmbot application
into separate, independent OTP applicatoins.
2019-09-13 16:03:13 -07:00