Relates to #233Fixes#364
More improvements to the transmitter submission process to auto-convert PPB to Hz and back, as well as popover.js tooltips from help_text in the model.
Signed-off-by: Corey Shields <cshields@gmail.com>
Re-implementing the case-on-type of the prior db UI.
To keep CSP happy along with the fact that we dynamically load the modal with django-bootstrap-modal, the javascript for these modals is in a new file, and that file's hash is kept in the CSP.
Also re-introduces child_src to test fixing a safari mapbox bug
Relates to #233Fixes#387
The intent of the "5 recent submitters" on the satellite page is to show the 5 most recent stations and their last submitted time, whereas this was showing the last 5 submissions regardless of the ground station.
Signed-off-by: Corey Shields <cshields@gmail.com>
Fixes regression where 'submit transmitter' link and modal were visible and partly accessible when not authenticated. Also tightens the satellite edit modal similarly.
Signed-off-by: Corey Shields <cshields@gmail.com>
Removes the field.html include - and the frequent inclusion of that in the transmitter cards.
Fix a bug in satellite.js where input not recognized as an integer
Add gitlab links to the version info
Removes the manually-entered kaitai struct field from view (will need removed from db in a later change) - replacing with a procedurally generated list of fields from the class imported through satnogsdecoders.
Fixes#385
Signed-off-by: Corey Shields <cshields@gmail.com>
Implement datatables for ground stations and satellites stat pages.
Fix ground stations icon
Improve datatables footer spacings all around
Tweak spacing around gravatar
Fixes#380
Signed-off-by: Corey Shields <cshields@gmail.com>
Bring transmitter uuid and copy-to-clipboard back to new UI.
Also fixes tooltips in new UI
Also fixes a typo in new 404 screen.
Fixes#382
Signed-off-by: Corey Shields <cshields@gmail.com>
Fixes#384 - by cleaning up the data export links
Fixes#379 - simple fontawesome change
Fixes#308 - adds 'most recent 5' observation submitters (regardless of number of submissions or staggering of submissions)
Changes the queryset for satellites to pull the properly 'approved'/valid transmitters as approved_transmitters which can be used to count transmitters with |length and reduce queries at the same time. As such:
Fixes#365Fixes#381
Thanks to @deckbsd and @adamkalis for their assistance on these!
Signed-off-by: Corey Shields <cshields@gmail.com>
Initial commit of new UI. There is still some work to be done before this goes into dev, but here is the work so far:
* Updated dependencies to latest 2.x django
* Updated to Bootstrap 4
* New home screen to display most recent satellite entries, most recent data, and contributors
* Adopted django-bootstrap-modal-forms for handling satellite and transmitter creation and update, with more of an emphasis on django's model/view/form model - and a dynamic flow where the modals and details are only loaded when the proper icon is clicked, reducing the overall page size
* Adopted AdminLTE 3.x framework atop Bootstrap 4
* Created reusable cards for satellite and transmitters
* Cards and Modals are organized into subdirectories for template includes and base templates, respectively
* New stats display widgets using BS4 and AdminLTE 3
* Satellite search is redesigned and now accessible from any page of the site
* Introduced datatables for an "All Satellites" view and a modification of the new "All Transmitters" view
* Focus on all UI scaling down to mobile devices
* New model created for Operator (/ Owner): name, names, description, website
* Added django-countries for support of CountryField
* Satellite model expanded to include: Operator, (satellite) website, countries, launched datetime, deployed datetime
* Transmitter suggestions can now be approved in the UI by superusers
* Satellite entries can now be edited in the UI by users with the change satellite permission
* Satellite page is now broken into 'tabbed' panels (Profile, Map, Transmitters, etc) - with the tab menu options appearing in the sidebar or at the top depending on screen size
* Other cleanup and changes that I'm missing for sure.
Signed-off-by: Corey Shields <cshields@gmail.com>
Signed-off-by: Julien Flawinne <jf.satnogs at protonmail dot com>
replace the for loop by one big query
Signed-off-by: Julien Flawinne <jf.satnogs at protonmail dot com>
Undo the modification made for test
Signed-off-by: Julien Flawinne <jf.satnogs at protonmail dot com>
API endpoints that are behind authentication check, are now
accessible by logged in users.
Signed-off-by: Alfredos-Panagiotis Damkalis <fredy@fredy.gr>
remove migration tests
Signed-off-by: Julien Flawinne <jf.satnogs at protonmail dot com>
use urlfield instead of texfield on sat model
Signed-off-by: Julien Flawinne <jf.satnogs at protonmail dot com>
remove the initialization of the url field
Signed-off-by: Julien Flawinne <jf.satnogs at protonmail dot com>
The last dependency update changed the required import order apparently
so the 'isort' command failed.
Signed-off-by: Fabian P. Schmidt <kerel@mailbox.org>
Signed-off-by: Flawinne Julien flawinne.julien at protonmail dot com
add uplink_mode field for transciever and transponder
Signed-off-by: Flawinne Julien flawinne.julien@protonmail.com
adapt unit test for transmitter model
Update migration for taking account of the invert value
update uplink_mode migration condition
Not request TLE with NORAD ID above 99000
Signed-off-by: Alfredos-Panagiotis Damkalis <fredy@fredy.gr>
rename mode transmitter field to downlink_mode
adapt unit test for transmitter model
update uplink_mode migration condition
add uplink_mode field for transciever and transponder transmitter type
Signed-off-by: Flawinne Julien flawinne.julien at protonmail dot com
keep mode id
add forgotten field mode id
add missing ,
fix yapf error
This change bumps DB up to django 2.2.6 and resolves dependencies. No migrations are necessary. I've tested in my own environment, time to test in db-dev!
Signed-off-by: Corey Shields <cshields@gmail.com>
This change bumps DB up to django 2.2.6 and resolves dependencies. No migrations are necessary. I've tested in my own environment, time to test in db-dev!
Signed-off-by: Corey Shields <cshields@gmail.com>
According to Django 1.11 documentation if there are 404.html and 500.html
pages in templates root directory then it loads them in case of 404 or
500 errors without any extra code needed. More details in:
https://docs.djangoproject.com/en/1.11/ref/views/#error-views
Signed-off-by: Alfredos-Panagiotis Damkalis <fredy@fredy.gr>
Updates:
* gulp to 4.0.2
* gulp-eslint to 6.0.0
* stylelint to 4.3.1
* bootstrap to 3.4.1
* chart.js to 2.8.0
* jquery to 3.4.1
* mapbox-gl to 1.3.1
* moment to 2.24.0
Removals:
* bootstrap-daterangepicker
* backbonejs(code parts that weren't removed)
* d3
* underscore
Signed-off-by: Alfredos-Panagiotis Damkalis <fredy@fredy.gr>
when linting db code I unintentionally swapped the return of mode_data with mode_label, causing NaN in stats.js when trying to calculate the mode doughnut.
Reverts this bug. Fixes#325
Signed-off-by: Corey Shields <cshields@gmail.com>
Fixes the cyclical dependency between celery.py and tasks.py when update_all_tle is called outside of celery by switching to autodiscovered tasks
Signed-off-by: Corey Shields <cshields@gmail.com>
[acinonyx@openwrt.gr: use @shared_task decorator and wrapper tasks to workaround celery bug]
Signed-off-by: Vasilis Tsiligiannis <acinonyx@openwrt.gr>
Fix some spacing issues and import syntax introduced by my linting changes.
The inline pylint disable for C0412 clashes with isort, there is no combination there that works for both.
Signed-off-by: Corey Shields <cshields@gmail.com>
Fixing these as part of W0511
There were some transmitter properties labeled "has_" (which implies a boolean) while returning a count. Renamed them and their instances to "_count" along with cleaning up some variables
Signed-off-by: Corey Shields <cshields@gmail.com>
There are clashes between the positioning and ordering of our imports between pylint and isort, just ignoring them on the pylint side for now..
Signed-off-by: Corey Shields <cshields@gmail.com>
Fixes a cyclical dependency between calling the celery app through an import in tasks.py (which then imports tasks.py) by retrieving celery through the class.
Tested in local dev
Signed-off-by: Corey Shields <cshields@gmail.com>
Ignoring cases of too many ancestors in API views.py as these are upstream, not much we can do about it.
Signed-off-by: Corey Shields <cshields@gmail.com>
Ignoring R0903 for tests.py where we do not need to care about public methods, and for django managers in models.py
This negates the need for MR !396
Signed-off-by: Corey Shields <cshields@gmail.com>
A couple of functions in tasks and utils were refactored with new helper functions to reduce the number of local variables per function.
In addition, fetch_data.py was flagged with R0914 but this is a relic of the original attempt at decoding data into DB, and is unused - so I am removing rather than fixing it.
Signed-off-by: Corey Shields <cshields@gmail.com>
Signed-off-by: Corey Shields <cshields@gmail.com>
Fix some spacing issues brought up in a full tox run (that were not identified in my prior linting flows)
too-many-nested-blocks - pulled a break point out for utils.py/decode_data to get rid of a nested block
Signed-off-by: Corey Shields <cshields@gmail.com>
broad-except, I added explicit exceptions where I felt safe doing so - but we do have a lot of broad exceptions accepted.
Signed-off-by: Corey Shields <cshields@gmail.com>
Linting W0613 led to the discovery that we filter on status entries that do not exist. Changing "functional" to "active" will fix https://db.satnogs.org/api/transmitters/?alive=true
fixes satnogs-db#298
Signed-off-by: Corey Shields <cshields@gmail.com>
we are going to ignore W0201 and W0221 around the TransmitterEntry save function. Reasons are in the code comments.
Signed-off-by: Corey Shields <cshields@gmail.com>
the check_celery function returns in a finally, which is unnecessary and is an antipattern. This works just the same dropping the finally statement and returning in the end.
Signed-off-by: Corey Shields <cshields@gmail.com>
display_decoded is an unused relic of the original decode method, and throws a W0105 warning.
removing this function completely
Signed-off-by: Corey Shields <cshields@gmail.com>
cleaning up C0111 exposed that we have some helpers initially added when django cache was implemented, but have since been abandoned.
Removing them as they are unused and unneeded
Signed-off-by: Corey Shields <cshields@gmail.com>
Found while documenting for C0111, this was old code for the initial "decode" functionality that was replaced by decode_data in utils.py
Signed-off-by: Corey Shields <cshields@gmail.com>
The special method __unicode__ is python2-only,
in python3 it's merged with __str__. In order to be py23-compatible
the python_2_unicode_decorator is used, see
https://docs.djangoproject.com/en/1.11/topics/python3/#str-and-unicode-methods
initial patch: Updated docstrings to show names
[kerel@mailbox.org: - reworded commit message,
- added similar change in demoddata
- added @python_2_unicode_compatible decorator]
Signed-off-by: Fabian P. Schmidt <kerel@mailbox.org>
Remove all bytestring references from existing migrations
Future migrations should avoid bytestrings with the future import we set.
Signed-off-by: Corey Shields <cshields@gmail.com>
With the introduction of 'satnogs-decoders' version saved within the
decoded data structure, there is no need to reset any data prior to
triggering a 'decode all' task. The newly decoded data can happily live
alongside old data until the administator of the data warehouse decides
to remove them, part of data janitor tasks. An additional advantage is
that if the decoder structure changes drastically, existing dashboards
can still be used to visualize the old data as long as they are
filtered by version and until new dashboards are prepared.
Signed-off-by: Vasilis Tsiligiannis <acinonyx@openwrt.gr>
We have a field in demoddata called "source" which stores the application source of the data (network, sids, manual). This causes a lot of confusion with the SiDS API format which has a "source" parameter (that we convert to "station")
As I'm working through documenting the API (and getting us ready for feature expansion), I'd like this field renamed to avoid future confusion.
Transmitter model has been refactored by changing the current schema
from Transmitter and Suggestion models to TransmitterEntry model and
two proxy models TransmitterSuggestion and Transmitter:
* TransmitterEntry model will hold all the entries and changes on
transmitter entries in db.
* TransmitterSuggestion model will be the proxy model of the unreviewed
transmitter entries.
* Transmitter model will be the proxy model of the latest approved and
reviewed transmitter entries.
There are also change on Transmitter model fields:
* alive field is now change to status field
* there are new fields: citation, user, reviewed and created
Signed-off-by: Corey Shields <cshields@gmail.com>
In django cache we are storing a full queryset of the satellites with their data count and last payload time. By trying to filter on this (sat_cache = sats_cache.filter(norad_cat_id=norad)) we cause django to re-evaluate the queries for that specific satellite. For CAS-4a this results in 2 queries that exceed 5 mins each.
This does not impact the /stats/ page as it iterates through the whole list of satellites in cache. This change mimics that behavior to find the satellite cache we care about in the template. Its a huge hack but its the only way we can show a count and bring the page view back for !237 quickly... This needs refactoring, big time..
I also pulled an unused has_telemetry_data which caused a db count
This leaves behind a query to pull the last payload to show who submitted the last payload, as without this query there is no connection between the cached satellite info and the actual observer (because the /stats/ page was not meant to make that connection). If this query continues to be prohibitive we may just need to remove the "last frame submitted by.." blob.
In this change we continue using the cache for the frame count, but instead of using the cache for the latest frame as discussed in IRC we grab the latest frame based on the ID. In my test environment this was a reasonable query time.
I'd rather we try this route before resorting to the cache for latest frame as it will be a better user experience.
Signed-off-by: Corey Shields <cshields@gmail.com>
This commit replaces the recursive 'dict' extraction functions with
'get_fields()' method which returns specific fields from Kaitai Struct
docstrings. The InfluxDB schema is also optimized to write multiple
fields per data point with the unique NORAD id used as the measurement
name.
Signed-off-by: Vasilis Tsiligiannis <acinonyx@openwrt.gr>
Due to Celestrak returning the TLE of 00005 when asked for TLEs for
satellites with NORAD IDs in the 999xx range, satellites with
temporary NORAD IDs get assigned a wrong TLE.
This commit adds a workaround by skipping satellites with temporary
NORAD IDs when updating TLEs.
This makes static files and templates source paths relative to the
application instead of current working directory.
Signed-off-by: Vasilis Tsiligiannis <acinonyx@openwrt.gr>
The code to preprocess the ELFIN telemetry frames to remove escape sequences throws
```
TypeError: Odd-length string
```
when executed in a python 2 environment.
Fixes#229
Introduces the following:
- decoding of DemodData through decoders that are compiled from kaitai structs and stored in base/decoders/*.py
- storage of the decoded data will either be in the payload_decoded field (default) or into influxdb if USE_INFLUX=True and the proper settings are met
- A scheduled task is set */15 to look for the last 4 hours of submitted frames for anything not decoded, and then decodes it. (we will want to tweak these numbers as we get a good feel for any race conditions that may occur)
- 2 new commands in the admin console, one that will reset all decoded data for a satellite, and another that will trigger decoding of all frames in the db for a satellite. These could be useful when making fixes or improvements to a decoder, or when a new decoder is initially added to the db.
Upon fixing CI we now have a slew of PEP8 722 issues (bare excepts). This MR fixes all of those by excepting Exception, which should catch anything the program generates while allowing systemexit exceptions.
deckbsd