Commit Graph

252 Commits (master)

Author SHA1 Message Date
Adeeb Shihadeh 8af20af66d
pylint: enforce indentation (W0311) (#24039)
* pylint: enforce indentation (W0311)

* few more
2022-03-24 23:23:29 -07:00
Adeeb Shihadeh 53909c1757
deleter: handle files (#24023) 2022-03-23 11:31:29 -07:00
Willem Melching da5a0c41a0
C3: detect metered Android hotspot (#23734)
* C3: detect metered networks

* show in ui

* fix text layout

* bump cereal

* revert ui changes

* set networkMetered

* add athena method

* add metered logging to uploader

* use in athena uploader

* remove param

* use networkmanager properties to set cell to unmetered

* fix indentation

* no need to check

* bump cereal

* review

* bump cereal
2022-03-09 11:36:52 +01:00
Adeeb Shihadeh 3243d1a81e
move lock clearing to uploader (#23826)
* move lock clearing to uploader

* test case
2022-02-22 22:30:11 -08:00
Willem Melching 2f00271ce6
omx encoder: move writing into thread (#23771)
* omx encoder: move writing to separate thread

* fix include

* pop

* log buffers sizes

* split copy and write
2022-02-22 13:02:50 +01:00
Willem Melching 724bab8ef3
Update raw_logger/framereader to new new ffmpeg api (#23799)
* Update raw_logger/framereader to new new ffmpeg api

* Can be removed now

* Handled by avcodec_free_context

* handle EAGAIN and clean up
2022-02-21 16:39:32 +01:00
Willem Melching 7765bc2166
improve logging around uploading (#23750)
* improve logging around uploading

* use int

* use raw

* duplicate
2022-02-11 12:58:41 +01:00
Shane Smiskol e5005b8f44
scons: fix duplicate environments with test flag (#23702)
* fix duplicate environments when built with --test

* lib

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-02-06 14:08:02 -08:00
Adeeb Shihadeh 652ef27a72 loggerd: less error logging 2022-01-24 21:15:14 -08:00
Dean Lee a4fefdcb94
RawLogger: downscale qcamera before encoding (#23607) 2022-01-24 13:41:44 -08:00
Ryan aa9e635311
Add more type hinting (#23595)
* Add more type hinting.

* Revert joystick_alert changes.

* Add typing to statsd.

* Update selfdrive/statsd.py

* Update selfdrive/test/test_fingerprints.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-01-21 15:11:16 -08:00
Willem Melching d3d10e014a
Ensure controlsd can engage again in process replay (#23556)
* hacks to make process replay engage again

* dont change that

* enable engagement check again

* chmod +x

* first working regen

* proper logMonoTime

* fix video framerate consistency

* mpc is valid again

* proper alive checks

* revert loggerd change

* ensure engaged in regen, fix managerState, peripheralState

* ubloxRaw is unused

* add submaster config for gm

* regen all services we can run without HW

* fix loggerd

* loggerd: matroska without the extension

* update ref
2022-01-21 13:13:34 +01:00
Dean Lee 22d43211d9
loggerd: make test_loggerd.py run on PC (#23574)
* run on PC

* set frameId

* remove todo

* rawlogger: remove suffix mkv

* remove unused import PC

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-01-19 13:55:02 -08:00
Robbe Derks 1b49ce6ec4
Statsd (#23413)
* device side of statsd

* need to start it

* enable in manager

* add sleep

* cleanup

* remove aggregates for now and standardize on industry terms

* manager needs main

* need to have a try/except

* atomic_write_on_fs_tmp does not work

* cleaner

* use dump

Co-authored-by: Willem Melching <willem.melching@gmail.com>

* one file at a time

* limit amount of files

* move to influx line protocol and cleanup

* needs to be a list

* fix timezone bug

* actually rate limit

* add to release

* normalized origin

* also log deviceType

* more stats

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2022-01-10 15:21:48 +01:00
Ewout ter Hoeven 332f568a82
Pyupgrade 3.6: Update syntax with Python 3.6+ features (#23305)
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default

Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>

Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
2021-12-24 11:18:39 -08:00
Ewout ter Hoeven 55390d273f
Convert format strings strings to f-strings (#23241)
* Convert all text strings to f-strings

Reformats all the text from the old "%-formatted" and .format(...) format to the newer f-string format, as defined in PEP 498. This requires Python 3.6+.

Flynt 0.69 was used to reformat the strings. 120 f-strings were created in 51 files.

F-strings are in general more readable, concise and performant. See also: https://www.python.org/dev/peps/pep-0498/#rationale

* revert pyextra changes

* revert ublox.py

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-12-16 14:58:17 +01:00
Dean Lee f042962933
put logcat in bootlog (#23217)
* put logcat in boot log

* continue

* use std::unique_ptr

* Revert "use std::unique_ptr"

This reverts commit c671f8ca71.
2021-12-14 11:54:54 -08:00
Dean Lee 4aa291c406
Use std::unique_ptr to manage memory (#23168)
* use unique_ptr

* trigger ci

* move down loggerdstate
2021-12-13 14:16:53 +01:00
Dean Lee cfad1f35e8
RawLogger: fix wrong pts (#23205) 2021-12-13 11:46:52 +01:00
Dean Lee 9decd3d8a2
util: move all functions into util namespace (#23203) 2021-12-12 14:42:23 -08:00
Adeeb Shihadeh 5f36b48199 loggerd: fix delayed exit from main thread 2021-12-07 18:22:00 -08:00
Willem Melching 2e0c73fc0c
Remove Raw Upload toggle (#23143)
* Remove Raw Upload toggle

* fix tests
2021-12-07 11:25:40 +01:00
Adeeb Shihadeh f71a03f967 increase loggerd per-cycle service limit to 200 2021-12-06 22:08:17 -08:00
Adeeb Shihadeh baf1a77ca2 loggerd: don't let a single service clog up the main thread 2021-12-06 19:31:11 -08:00
Dean Lee ddc612e9f6
LogCameraInfo: remove unused variables (#23142) 2021-12-06 14:25:28 +01:00
Dean Lee 480fb54926
move struct LogCameraInfo into loggerd.h (#23129) 2021-12-04 21:08:55 -08:00
grekiki 6951b3271d
Python: Replace more lists with generators (#23116)
* Replace lists with generators v2

* Replace set with {}

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Replace more set() with {}

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-12-03 22:57:19 -08:00
Dean Lee 348d2d2b0d
rename yuv streams (#23071)
* rename yuv streams

* bump cereal

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-11-29 18:14:59 -08:00
Willem Melching f7c46c6949
Do not run code on version.py import (#23063)
* Do not run code on version.py import

* fix athena
2021-11-29 19:38:55 +01:00
Willem Melching a2f32fd3e1
Clear loggerd locks once on boot (#23060)
* add test for clear_locks

* move to bootlog

* simplify test
2021-11-29 12:30:28 +01:00
Dean Lee 1d323e0fd6
loggerd: add test case for trigger_rotate (#23038)
* test rotate

* remove global LoggerdState
2021-11-26 14:53:09 +01:00
Dean Lee b5eb02181c
loggerd: add test case for sync_encoders (#23020) 2021-11-24 13:34:16 -08:00
Dean Lee 75dd0d6296
loggerd: split loggerd.cc into three files 2021-11-23 17:53:49 -08:00
Adeeb Shihadeh e13630dfee
faster jenkins (#22973)
Co-authored-by: Comma Device <device@comma.ai>
2021-11-18 23:43:09 -08:00
Willem Melching 23a1f47817
uploader: switch to upload_url v1.4 (#22966) 2021-11-18 16:23:50 +01:00
Willem Melching 77b334a698
swaglog.py: respect LOGPRINT env variable (#22950)
* swaglog.py: respect LOGPRINT env variable

* only change handler that prints

* keep usefull manager prints

* only print running list

* revert process changes

* fix test
2021-11-18 16:05:06 +01:00
Adeeb Shihadeh ea761cbbd5
loggerd: trigger rotate on frame id instead of frame count (#22848)
Co-authored-by: Comma Device <device@comma.ai>
2021-11-13 15:38:10 -08:00
Willem Melching fa0cbe5d40
Put journal and nvme stats in bootlog (#22849)
* Run commands and put output in bootlog

* log nvme

* fix indent

* cereal without event

* use regular array
2021-11-10 16:07:22 +01:00
Dean Lee 0bd9aed82c
loggerd/RawLogger: fix memory leak in encode_frame (#22826) 2021-11-08 15:27:51 -08:00
Willem Melching 45c506bb17 wshadow: fix comma three build 2021-11-02 17:27:34 +01:00
Mayfield 5246f0231e
enable wshadow (#22756)
* enable wshadow (#22714)

* fix replay

* more build fixes

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-11-02 17:08:53 +01:00
Dean Lee 2b4a477fbc
replay: improve segment download and merge (#22654)
* no-cache mode

* fix test cases build error

* space

* don't create cache dir in no-cache mode

* fix errors in test cases

* no_local_cache_

* set the number of connections by chunk_size

* use size_t instead of int64_t

* add test case for no-cache mode

* rename variables

* fix SIGSEGV

* cleanup

* faster decompressBZ2

* always decompress bz2

* add test cases

* prepare for python interface

* fix test cases build error

* continue

* camera_replay: cache remote file

* protected inheritance

* single option name

* TODO

* test_case for LogReader&FrameReader

* fix wrong require

* test case for FileReader

* cleanup test

* test:fix wrong filename

* check cached file's checksum

* fix mkdir permissions err

cleanup filereader

* remove initialize libav network libraries.

dd

* abort all loading if one failed

* cleanup tests

* use threadpool to limit concurrent downloads

* cache more segments

* merge 3 segments for replay

* one segment uses about 100M of memory

* use segments_need_merge.size()

* shutdown

* fix stuck if exit replay  before keyboard thread started

* load one segment at a time

* small cleanup

* cleanup filereader

* space

* tiny cleanup

* merge master

* cleanup test cases

* use util:create_directories

* cleanup framereader
2021-11-01 11:55:56 +01:00
Dean Lee 652c42da2c
util: add safe write functions (#22160)
* add safe write functions

* log error

* bzfile safe_fopen&safe_flush

* update test_case

* trigger ci

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-10-29 13:23:31 +02:00
Dean Lee 1f39d8cee6
util: add new function create_directories with unit tests (#21871)
* util::create_directories

* check bool ret

* don't assume mask

* rename with_umask to no_umask

* remove umask

* rebase master

* rebase master

* 0755
2021-10-29 12:27:35 +02:00
Dean Lee 18caf3535b
loggerd: fix concurrency issues in sync encoders (#22590)
* sync_encoders

* add variable camera_synced

* cleanup
2021-10-29 12:21:02 +02:00
Adeeb Shihadeh 5b331fd6f5
don't set core affinity on PC (#22706) 2021-10-26 21:51:46 -07:00
Willem Melching 89f311714c
loggerd: set encodeIdx.valid if frame id is correct (#22634)
* loggerd: set encodeIdx.valid if frame id is correct

* check valid flag in test

* test cleanup

* bump cereal
2021-10-26 16:00:43 +02:00
Adeeb Shihadeh 76bd932cf5
loggerd: always run encoders (#22649)
* loggerd: always run encoders

* fix raw logger

* bump loggerd cpu usage
2021-10-21 16:37:03 -07:00
Adeeb Shihadeh 84de248fa7
tici: affine loggerd to efficiency cores (#22581) 2021-10-18 13:21:24 -07:00
Dean Lee d28b98c602
replay : move utility functions into separate file (#22414)
* move functions into util

* read bz2 into stream

* pre-decompress log in the download thread

* cleanup logreader

* cache sha256 path

* use readBZ2file in test_logger

* Revert "cache sha256 path"

This reverts commit 60459d3ea09a2c80f4560cf95b1ce7d6af59f06d.

* use macro

* use ostringstream

* cleanup readBZ2File

* move precise_nano_sleep into util
2021-10-04 15:39:59 +02:00