* Make `snapshot_manifest` methods `async`, so that they don't block async runtime
* Spawn `restore_shard_snapshot` task on `update` runtime instead of "current"
* Disable rocksdb compile time feature by default
* Also disable RocksDB feature in segment crate
* Enable RocksDB feature in all CI builds
* Remove extra job for testing non-RocksDB build, it's the default now
* Keep RocksDB structures in generated OpenAPI schema
* Fix obsolete --workspace flag breaking builds with explicit features
* Also build including RocksDB in e2e tests on CI
* After clearing WAL, flush segment
* Add debug log when WAL is cleared
* Clear WAL on consensus snapshot after writing state, truncate on start
* Apply consensus snapshot offset
* Fix off by one error
* Tweak debug assertion message
* Change WAL reconciliation condition, and fully clear WAL in this case
* Add debug assertion to prove Raft index and snapshot index are equal
* Add documentation to resolve bot nit
* Return error on WAL clear failure
* Fix typo
* Remove unused truncate functions
* Fix use the update runtime for update operations
* Enable IO on update runtime
FAILED tests/consensus_tests/test_collection_recovery_limits.py::test_collection_recovery_reach_limit - Exception: HTTP request to http://127.0.0.1:33891/collections/test_collection/points?wait=true&ordering=weak failed with status code 500 after 5.20299s with response body:
{'status': {'error': 'Service internal error: task 240 panicked with message "A Tokio 1.x context was found, but IO is disabled. Call `enable_io` on the runtime builder to enable IO."'}, 'time': 3.28820663, 'usage': {'hardware': {'cpu': 0, 'payload_io_read': 0, 'payload_io_write': 70315, 'payload_index_io_read': 0, 'payload_index_io_write': 0, 'vector_io_read': 0, 'vector_io_write': 1649102}, 'inference': None}}
* Wrap the tracker and bitmask in Arc
* Implement deferred flusher for gridstore, defer tracker writes
* Remove clone
* Dynamically adjust list of pending updates to drain what is flushed
* Return proper flusher, defer premature Gridstore flushing errors
* Use deferred Gridstore flusher across storages
* Remove all Arc<RwLock<_>> wrappers around Gridstore in storages
* Flush pages inside closure, also wrap them in a lock
* Remove direct flush function from Gridstore
* Add test to assert behavior of deferred flushing in Gridstore
* Add much more extensive test, including concurrent flushes and deletes
* Add test for draining value pointer, make sure to drop when flushed
* Feature gate RocksDB components
* Prematurely abort blocking task in `spawn_cancel_on_drop` on drop
These tasks are intended to be cancellable. Now we prematurely abort the
task if the future was dropped before the task is executed.
* Prematurely abort blocking task in `spawn_cancel_on_token` on cancel
These tasks are intended to be cancellable. Now we prematurely abort the
task if the cancellation token is triggered before the task is executed.
* Prematurely abort blocking task for fetching telemetry
* Prematurely abort stoppable task on drop, all are safe to abort early
* Make `move_dir` either move everything, or nothing at all
That is with the exception of file IO errors in which case data may be
partially moved.
Before this PR it was possible for the new target directory to be
created without moving all data into it. Now we either do all, or
nothing.
* Prematurely abort task for creating full snapshot
It is fine to either create it, or not at all.
* Prematurely abort blocking task for waiting on consensus leader
* Prematurely abort blocking cardinality estimation and shard info tasks
* Prematurely abort blocking point deduplication task
* Prematurely abort blocking task for checking available disk space
* Prematurely abort blocking shard read operations
All shard read operations, such as retrieve, scroll, facets and more can
be safely aborted prematurely.
Related to: <https://github.com/qdrant/qdrant/pull/7530>
* Prematurely abort blocking task for waiting on replica state
* Prematurely abort blocking task for waiting on transfer replica states
* Prematurely abort blocking task for loading segment
This can safely be aborted before the task is started
* Prematurely abort blocking task waiting for replica states
* Prematurely abort blocking task for creating snapshot file
Safe because it aborts before writing any snapshot files to disk
* add stopflag check into query_points of payload index
* propagate stopflag more
* fmt
* bench: compare better against no atomic
* use stoppable iter with IteratorExt
* clippy
* Add cancelled error helper
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
