From b2c3bc1ff9fbe176f94c11de69f2299161f977f1 Mon Sep 17 00:00:00 2001 From: Rick Carlino Date: Tue, 2 Oct 2018 08:57:59 -0500 Subject: [PATCH] Shuffle files around better, remove more legacy references. Ready for one last QA round --- .gitignore | 4 ++-- DEPLOYMENT.md | 12 +++++----- LICENSE | 2 +- README.md | 23 ++++++++++--------- app/lib/key_gen.rb | 4 ++-- config/environments/production.rb | 6 +---- docker-compose.yml | 12 +++++----- .../api.Dockerfile | 0 .../rabbitmq.Dockerfile | 0 {mqtt => docker_configs}/rabbitmq.conf.erb | 0 .../rabbitmq_config_builder.rb | 11 +++++---- lib/tasks/api.rake | 14 ++++------- lib/tasks/mqtt.rake | 13 ----------- package.json | 2 +- spec/spec_helper.rb | 2 +- ubuntu_example.sh | 4 ++-- 16 files changed, 44 insertions(+), 65 deletions(-) rename docker-api.Dockerfile => docker_configs/api.Dockerfile (100%) rename mqtt/Dockerfile => docker_configs/rabbitmq.Dockerfile (100%) rename {mqtt => docker_configs}/rabbitmq.conf.erb (100%) rename mqtt/server.rb => docker_configs/rabbitmq_config_builder.rb (78%) delete mode 100644 lib/tasks/mqtt.rake diff --git a/.gitignore b/.gitignore index 91e8472fe..610c0a5b1 100755 --- a/.gitignore +++ b/.gitignore @@ -13,8 +13,8 @@ erd_diagram.png erd.pdf latest_corpus.ts log/*.log -mqtt/rabbitmq.conf -mqtt/rabbitmq.config +docker_configs/rabbitmq.conf +docker_configs/rabbitmq.config node_modules node_modules/ package-lock.json diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 444fd6c7a..e06826f89 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -4,23 +4,23 @@ Parts of this document may be out of date or reflect practices that are no longe Please raise an issue if you hit any issues during deployment. -# Run on a Local Machine (saves time) +# Run on a Local Machine If you want to run a server on a LAN for personal use, this is the easiest and cheapest option. -**Simplicity:** :heart::heart::heart: +**Simplicity:** :heart: :heart: :heart: **Reliability:** :broken_heart: -**Affordability:** :heart::heart::heart: +**Affordability:** :heart: :heart: :heart: 1. Follow the [developer setup guide](https://github.com/FarmBot/Farmbot-Web-App#q-how-do-i-setup-an-instance-locally). # Deployment Using Heroku (best reliability) -**Simplicity:** :heart::heart::heart::heart: +**Simplicity:** :heart: :heart: :heart: :heart: -**Reliability:** :heart::heart::heart::heart: +**Reliability:** :heart: :heart: :heart: :heart: **Affordability:** :broken_heart: @@ -28,4 +28,4 @@ If you want to run a server on a LAN for personal use, this is the easiest and c 2. Enable Dyno metadata: `heroku labs:enable runtime-dyno-metadata --app ` (we need this to know the version number of the web app). 3. (If emails are enabled) Enable the [Heroku scheduler](https://elements.heroku.com/addons/scheduler) and configure it to run `rake api:log_digest` every 10 minutes. This is required for Device log digests via email. -Don't forget to [set ENV vars](https://devcenter.heroku.com/articles/config-vars) as you would normally do with `application.yml` and run `heroku run rake db:setup`. +Don't forget to [set ENV vars](https://devcenter.heroku.com/articles/config-vars) and run `heroku run rake db:setup`. diff --git a/LICENSE b/LICENSE index 689aa206c..253fd74a7 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ The MIT License (MIT) -Copyright (c) 2015 Farmbot.io +Copyright (c) 2018 Farmbot.io Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index 4033ec40a..ff5eeedcd 100644 --- a/README.md +++ b/README.md @@ -11,12 +11,13 @@ This repository is intended for *software developers* who wish to modify the [Farmbot Web App](http://my.farmbot.io/). **If you are not a developer**, you are highly encouraged to use [the publicly available web app](http://my.farmbot.io/). Running a server is *a non-trivial task with security implications*. It requires an intermediate background in Ruby, SQL and Linux system administration. -If you are a developer interested in contributing or would like to provision your own server, you are in the right place. We do not have the resources available to help novice developers learn to setup servers, environments, configurations, or perform basic Linux command line instructions. If you raise an issue indicating that you haven't followed the setup instructions, looked through past issues, or done a cursory internet search for basic help, expect the issue to be closed and we'll point you to the setup instructions. *Again, if you do not have at least intermediate Linux and Ruby experience, please use the hosted version of the web app at my.farm.bot.* Running a self-hosted server is not easy! +If you are a developer interested in contributing or would like to provision your own server, you are in the right place. + # Q: Where do I report security issues? We take security seriously and value the input of independent researchers. Please see our [responsible disclosure guidelines](https://farm.bot/responsible-disclosure-of-security-vulnerabilities/). @@ -33,25 +34,21 @@ For a list of example API requests and responses, see our [reference documentati We provide example setup instructions for Ubuntu 18 [here](https://github.com/FarmBot/Farmbot-Web-App/blob/master/ubuntu_example.sh). -Installation was last tested against Ubuntu 18.04 in June of 2018 on an x86 based machine. +Installation was last tested against Ubuntu 18.04 in October of 2018 on an x86 based machine. Our ability to help individual users with private setup is limited. Using the public server at http://my.farm.bot is the recommended setup for end users. Please see the top of this document for more information. # Config Settings (important) -We try our best to follow the [12 Factor Methodology](https://12factor.net/). Part of that means using ENV variables as a means of [storing configuration](https://12factor.net/config). **Your server won't run without setting ENV variables first**. +We follow the [12 Factor Methodology](https://12factor.net/). As such, ENV variables are the primary means of [storing configuration](https://12factor.net/config). **Your server won't run without setting ENV variables first**. -You can accomplish this by setting the ENV variables directly from your shell / server management tool or by writing an `application.yml` file. +You can accomplish this by setting the ENV variables directly from your shell / server management tool or by writing an `.env` file in server's base directory. -See `config/application.example.yml` for a list of all the variables that must be set. +See `example.env` for a list of all the variables that must be set. -**Encryption keys**: Encryption keys will be auto-generated if not present. They can be reset using `rake keys:generate`. If `ENV['RSA_KEY']` is set, it will be used in place of the `*.pem` files. Useful for environments like Heroku, where file system access is not allowed. +**Encryption keys**: Encryption keys will be auto-generated if not present. They can be reset using `rake keys:generate`. If `ENV['RSA_KEY']` is set, it will be used in place of the `*.pem` files. This is useful for environments like Heroku and Docker, where file system access is not allowed. -# Want to Help? - -[Low Hanging Fruit](https://github.com/FarmBot/Farmbot-Web-App/search?utf8=%E2%9C%93&q=todo). [Raise an issue](https://github.com/FarmBot/Farmbot-Web-App/issues/new?title=Question%20about%20a%20TODO) if you have any questions. - -## Translating the web app into your language +# Translating the Web App into Your Language Thanks for your interest in internationalizing the FarmBot web app! To add translations: @@ -59,3 +56,7 @@ Thanks for your interest in internationalizing the FarmBot web app! To add trans 0. Navigate to `/public/app-resources/languages` and run the command `node _helper.js yy` where `yy` is your language's [language code](http://www.science.co.il/Language/Locale-codes.php). Eg: `ru` for Russian. 0. Edit the translations in the file created in the previous step: `"phrase": "translated phrase"`. 0. When you have updated or added new translations, commit/push your changes and submit a pull request. + +# Want to Help? + +[Low Hanging Fruit](https://github.com/FarmBot/Farmbot-Web-App/search?utf8=%E2%9C%93&q=todo). [Raise an issue](https://github.com/FarmBot/Farmbot-Web-App/issues/new?title=Question%20about%20a%20TODO) if you have any questions. diff --git a/app/lib/key_gen.rb b/app/lib/key_gen.rb index 7c85af78e..164f619ab 100644 --- a/app/lib/key_gen.rb +++ b/app/lib/key_gen.rb @@ -14,8 +14,8 @@ class KeyGen return rsa end - # Heroku / 12Factor users can't store stuff on the file system. Store your pem - # file in ENV['RSA_KEY'] if that applies to you. + # Heroku / 12Factor users can't store stuff on the file system. + # Store your pem file in ENV['RSA_KEY'] if that applies to you. def self.try_env OpenSSL::PKey::RSA.new(ENV['RSA_KEY']) if ENV['RSA_KEY'] end diff --git a/config/environments/production.rb b/config/environments/production.rb index b5ee6648f..5083f6b6d 100755 --- a/config/environments/production.rb +++ b/config/environments/production.rb @@ -13,11 +13,7 @@ FarmBot::Application.configure do config.log_level = :info config.perform_caching = false config.public_file_server.enabled = false - # config.after_initialize do - # Bullet.enable = true - # Bullet.console = true - # Bullet.rollbar = true if ENV["ROLLBAR_ACCESS_TOKEN"] - # end + # HACK AHEAD! Here's why: # 1. FarmBot Inc. Uses Sendgrid for email. # 2. FarmBot is an open source project that must be vendor neutral. diff --git a/docker-compose.yml b/docker-compose.yml index d313000a3..a7e1410d1 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,7 +1,7 @@ version: '3' services: mqtt: # ====================================================================== - build: ./mqtt + build: ./docker_configs/rabbitmq.Dockerfile ports: - "5672:5672" # AMQP (RabbitMQ) - "1883:1883" # MQTT @@ -17,7 +17,7 @@ services: web: # ====================================================================== build: context: . - dockerfile: docker-api.Dockerfile + dockerfile: docker_configs/api.Dockerfile # Add the `rm -f` part to avoid "server is still running..." errors: command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -e development -p ${API_PORT:-3000} -b 0.0.0.0" volumes: @@ -32,7 +32,7 @@ services: webpack: # =================================================================== build: context: . - dockerfile: docker-api.Dockerfile + dockerfile: docker_configs/api.Dockerfile command: ./node_modules/.bin/webpack-dev-server --config config/webpack.config.js volumes: - .:/farmbot @@ -46,7 +46,7 @@ services: delayed_job: # =============================================================== build: context: . - dockerfile: docker-api.Dockerfile + dockerfile: docker_configs/api.Dockerfile command: bundle exec rake jobs:work volumes: - .:/farmbot @@ -58,7 +58,7 @@ services: log_digests: # =============================================================== build: context: . - dockerfile: docker-api.Dockerfile + dockerfile: docker_configs/api.Dockerfile command: bundle exec rake api:log_digest volumes: - .:/farmbot @@ -70,7 +70,7 @@ services: rabbit_jobs: # =============================================================== build: context: . - dockerfile: docker-api.Dockerfile + dockerfile: docker_configs/api.Dockerfile command: bundle exec rails r lib/rabbit_workers.rb volumes: - .:/farmbot diff --git a/docker-api.Dockerfile b/docker_configs/api.Dockerfile similarity index 100% rename from docker-api.Dockerfile rename to docker_configs/api.Dockerfile diff --git a/mqtt/Dockerfile b/docker_configs/rabbitmq.Dockerfile similarity index 100% rename from mqtt/Dockerfile rename to docker_configs/rabbitmq.Dockerfile diff --git a/mqtt/rabbitmq.conf.erb b/docker_configs/rabbitmq.conf.erb similarity index 100% rename from mqtt/rabbitmq.conf.erb rename to docker_configs/rabbitmq.conf.erb diff --git a/mqtt/server.rb b/docker_configs/rabbitmq_config_builder.rb similarity index 78% rename from mqtt/server.rb rename to docker_configs/rabbitmq_config_builder.rb index bfff750c9..bfcc88ffa 100644 --- a/mqtt/server.rb +++ b/docker_configs/rabbitmq_config_builder.rb @@ -1,17 +1,17 @@ require 'erb' def needs_admin_password - raise "You must set an ADMIN_PASSWORD in application.yml." + raise "You must set an ADMIN_PASSWORD in `.env`." end puts "=== Setting config data" -CONFIG_PATH = "./mqtt" +CONFIG_PATH = "./docker_configs" CONFIG_FILENAME = "rabbitmq.conf" CONFIG_OUTPUT = "#{CONFIG_PATH}/#{CONFIG_FILENAME}" NO_API_HOST = "\nYou MUST set API_HOST to a real IP address or " + "domain name (not localhost).\n" + "API_PORT is also mandatory." -TEMPLATE_FILE = "./mqtt/rabbitmq.conf.erb" +TEMPLATE_FILE = "./docker_configs/rabbitmq.conf.erb" TEMPLATE = File.read(TEMPLATE_FILE) RENDERER = ERB.new(TEMPLATE) PROTO = ENV["FORCE_SSL"] ? "https:" : "http:" @@ -26,6 +26,7 @@ if !ENV["API_HOST"] || !ENV["API_PORT"] puts NO_API_HOST exit end - +ok = RENDERER.result(binding) # Write the config file. -File.write(CONFIG_OUTPUT, RENDERER.result(binding)) +File.write(CONFIG_OUTPUT, ok) +puts ok diff --git a/lib/tasks/api.rake b/lib/tasks/api.rake index 7c9d7958b..463ba8bdc 100644 --- a/lib/tasks/api.rake +++ b/lib/tasks/api.rake @@ -1,9 +1,9 @@ # Going to make an `update` AND `upgrade` task that do the same thing def same_thing sh "git pull https://github.com/FarmBot/Farmbot-Web-App.git master" - sh "bundle install" - sh "yarn install" - sh "rails db:migrate" + sh "sudo docker-compose run web bundle install" + sh "sudo docker-compose run web yarn install" + sh "sudo docker-compose run web rails db:migrate" end def check_for_digests @@ -31,12 +31,7 @@ namespace :api do desc "Run Webpack and Rails" task start: :environment do - sh "PORT=3000 bundle exec foreman start --procfile=Procfile.dev" - end - - desc "Run Rails _ONLY_. No Webpack." - task only: :environment do - sh "PORT=3000 bundle exec foreman start --procfile=Procfile.api_only" + puts "Perhaps you meant `sudo docker-compose up`?" end desc "Pull the latest Farmbot API version" @@ -44,5 +39,4 @@ namespace :api do desc "Pull the latest Farmbot API version" task(upgrade: :environment) { same_thing } - end diff --git a/lib/tasks/mqtt.rake b/lib/tasks/mqtt.rake deleted file mode 100644 index b38c45b37..000000000 --- a/lib/tasks/mqtt.rake +++ /dev/null @@ -1,13 +0,0 @@ -namespace :mqtt do - desc "Bootstraps the MQTT config file. To force a rebuild of *everything*, " + - "set FORCE_REBUILD='true'" - task start: :environment do - begin - require_relative '../../mqtt/server.rb' - rescue Errno::ECONNREFUSED => e - puts "API is not up yet. Waiting 5 seconds..." - sleep 5 - retry - end - end -end diff --git a/package.json b/package.json index 49438463e..745d7027e 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "coverage": "cat **/*lcov.info | ./node_modules/coveralls/bin/coveralls.js", "clean": "rm -rf public/dist && rm -rf public/webpack", "build": "TARGET=production bundle exec rake webpack:compile", - "start": "echo 'use `rails api:start`'", + "start": "echo 'use `sudo docker-compose up` instead.'", "heroku-postbuild": "webpack --config=./config/webpack.prod.js", "webpack": "./node_modules/.bin/webpack-dev-server --config config/webpack.dev.js", "test-slow": "jest --coverage --no-cache -w 4", diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index c1d516cb1..65fd8463a 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -79,7 +79,7 @@ RSpec.configure do |config| require "capybara/rails" require "capybara/rspec" require "selenium/webdriver" - # Be sure to run `RAILS_ENV=test rails api:start` and `rails mqtt:start`! + # Be sure to run the server in a seperate window! Capybara.run_server = false Capybara.app_host = "http://localhost:3000" Capybara.server_host = "localhost" diff --git a/ubuntu_example.sh b/ubuntu_example.sh index ffdaf7a7f..d6b65a2f0 100644 --- a/ubuntu_example.sh +++ b/ubuntu_example.sh @@ -38,9 +38,9 @@ sudo docker-compose run web npm install sudo docker-compose run web bundle exec rails db:setup # Generate a set of *.pem files for data encryption sudo docker-compose run web rake keys:generate -# Manually create the rabbitmq.conf file +# Manually create the rabbitmq.conf file. # TODO: Improve this step -RC 1 Oct 18 -sudo docker-compose run web bundle exec rails r mqtt/server.rb +sudo docker-compose run web bundle exec rails r docker_configs/rabbitmq_config_builder.rb # Build the UI assets via WebPack sudo docker-compose run web npm run build # Run the server! ٩(^‿^)۶