Commit graph

5 commits

Author SHA1 Message Date
Adeeb Shihadeh 59d7c5e64a
move to github container registry + CI speedup (#19621) 2020-12-31 14:57:01 -08:00
Willem Melching 529e9a23ea
add check for partial response code (#2282) 2020-10-07 12:14:22 +02:00
Greg Hogan 2293a6a36c
eliminate round-trip reading entire file (#2275)
* eliminate round-trip reading entire file

* only if reading entire file

* print headers in errors

* fix end
2020-10-05 16:59:19 -07:00
grekiki 05b64257df
FileReader: cache files locally (#2170)
* framework for updates

* File caching

* update .gitignore

* update .gitignore

* Store in home tmp directory

* fix bugs in old code, add tests

* add test

* revert bug

* Fixed some bugs, more testing

* doing some style fixes

* Update caching with lenghts

* doing some style fixes

* doing some style fixes

* doing some style fixes

* subrepositories

* try to fix the tests

* change submodules back

* doing some style fixes

* if we don't cache, we don't cache length either

* fix curls

* make sure tests run in ci

* Cleanup tests

* Use int for cache flag

* fix linter

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2020-09-17 13:03:34 +02:00
Willem Melching c70700758d
Speedup URLFile (#1888)
* add parallel download support to URLFile

* make python 3.8 happy

* Fix chunk size

* Automatic number of threads

* No daemon threads in unlogger

* Cache length

* dont touch old filereader

* Remove debug info

* remove debug script

* Ignore type
2020-07-20 17:10:08 +02:00
Renamed from common/url_file.py (Browse further)