* After clearing WAL, flush segment
* Add debug log when WAL is cleared
* Clear WAL on consensus snapshot after writing state, truncate on start
* Apply consensus snapshot offset
* Fix off by one error
* Tweak debug assertion message
* Change WAL reconciliation condition, and fully clear WAL in this case
* Add debug assertion to prove Raft index and snapshot index are equal
* Add documentation to resolve bot nit
* Return error on WAL clear failure
* Fix typo
* Remove unused truncate functions
* Fix use the update runtime for update operations
* Enable IO on update runtime
FAILED tests/consensus_tests/test_collection_recovery_limits.py::test_collection_recovery_reach_limit - Exception: HTTP request to http://127.0.0.1:33891/collections/test_collection/points?wait=true&ordering=weak failed with status code 500 after 5.20299s with response body:
{'status': {'error': 'Service internal error: task 240 panicked with message "A Tokio 1.x context was found, but IO is disabled. Call `enable_io` on the runtime builder to enable IO."'}, 'time': 3.28820663, 'usage': {'hardware': {'cpu': 0, 'payload_io_read': 0, 'payload_io_write': 70315, 'payload_index_io_read': 0, 'payload_index_io_write': 0, 'vector_io_read': 0, 'vector_io_write': 1649102}, 'inference': None}}
* Prematurely abort blocking task in `spawn_cancel_on_drop` on drop
These tasks are intended to be cancellable. Now we prematurely abort the
task if the future was dropped before the task is executed.
* Prematurely abort blocking task in `spawn_cancel_on_token` on cancel
These tasks are intended to be cancellable. Now we prematurely abort the
task if the cancellation token is triggered before the task is executed.
* Prematurely abort blocking task for fetching telemetry
* Prematurely abort stoppable task on drop, all are safe to abort early
* Make `move_dir` either move everything, or nothing at all
That is with the exception of file IO errors in which case data may be
partially moved.
Before this PR it was possible for the new target directory to be
created without moving all data into it. Now we either do all, or
nothing.
* Prematurely abort task for creating full snapshot
It is fine to either create it, or not at all.
* Prematurely abort blocking task for waiting on consensus leader
* Prematurely abort blocking cardinality estimation and shard info tasks
* Prematurely abort blocking point deduplication task
* Prematurely abort blocking task for checking available disk space
* Prematurely abort blocking shard read operations
All shard read operations, such as retrieve, scroll, facets and more can
be safely aborted prematurely.
Related to: <https://github.com/qdrant/qdrant/pull/7530>
* Prematurely abort blocking task for waiting on replica state
* Prematurely abort blocking task for waiting on transfer replica states
* Prematurely abort blocking task for loading segment
This can safely be aborted before the task is started
* Prematurely abort blocking task waiting for replica states
* Prematurely abort blocking task for creating snapshot file
Safe because it aborts before writing any snapshot files to disk
* Add collections_vector_total metric for backwards compatibility
Only this metric was removed since Qdrant version 1.15.5, and so this
adds it back for backwards compatibility. Though it has been marked as
deprecated now.
* Prefix replica metrics with collection, part of collection namespace
* Count running optimizations per collection
* Remove deprecation flag
* Minor optimization, preallocate vectors
* In metrics, aggregate page faults
* Include page fault of active children (recursively)
* type alias Pid: i32
* Code improvements
* Ignore already terminated processes
* Clippy
* Also count faults for joined descendant threads in children
* Import std::cmp::min
* Rework child fault recursion, remove second hash set
* Add simple test for iterating child processes
* Fix typo
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use limit values directly
* We don't include the parent PID
* Don't allocate hashmap when listing process children
* Extend process child PIDs test, fork recursively
---------
Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Add ReadActive replica state
* Add consensus function to switch from Partial to ReadActive and confirm
* On stream records to a different shard ID, switch to ReadActive first
* Update OpenAPI spec
* In stream records peer sync, require all peers to be up-to-date
* Review fixes
* Rename ReadActive to ActiveRead
* update openapi spec
* explicit state checking functions
* typo
* update fallback routing with usage of a new state
---------
Co-authored-by: KShivendu <kshivendu1@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Change page fault metrics from gauges to counters
* Change collection_vectors_total to collection_vectors
* Rename metric for page faults in joined child processes
* Rename some procfs metrics
* Show single limit for number of open files
* For number of file descriptors and mmaps follow convential names
* Rename dead_shards_total to dead_replicas
* Fix metric prefix for gRPC entries, add missing underscore
A underscore was missing if no global prefix was set
* make jwt with payload filter fail
* remove `whole` access requirement
* remove unnecessary mut
* fmt
* adjust test
* fix test
* remove more tests
* fix test
* fix test again
* fix test again
* Replace deprecated PayloadConstraint with JSON Value placeholder
* Fix expect message
---------
Co-authored-by: timvisee <tim@visee.me>
* Add vector count per vector-name to metrics API
* Add to metrics API
* Improve TinyMap::get_or_insert_default and add tests
* Minor improvements
* Update openapi
* Review remarks
* Remove `collection_vectors` since it can be calculated manually
* Dead shards in /metrics
* Update src/common/metrics.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add replica metrics
* Clippy
* Properly ignore shards with resharding status
* Include 0 as value in metrics
* Remove total_active_replicas and total_expected_replicas
* Improve minimum active replica count in case of resharding
* Reorder metrics calls min < max (to trigger CI)
* Fix metrics help
* Fix typo
---------
Co-authored-by: timvisee <tim@visee.me>
* Respect payload and hashring filters while replicating point
* Merge points instead of replacing for filtered stream records transfer
* cleanup
* Update utils
* Allow filters in transfer
* drop hashring filter since its not required when destination has single shard
* test comment
* Pass actual filter
* use exact=True
* Forward updates that satisfy the filter
* make test long enough for extra points to be upserted
* minor improvements
* fix test
* try with only inserting new points
* Temporarily forward points that match before or after
* Fix failing test
* Only do inserts
* Dont merge points and update old points in test
* Fix the bug in integration test
* fix read_batch_with_hashring fn name and docs
* changes after rebase
* add validation check
* apply suggestions
* Always trim clock tag when forwarding to a different shard ID
* review fixes
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Allow replicating points between shards
* Add integration test for replicating points between shards
* Drop filter from ForwardProxyShard to for now
* Clean up test
* fix assert failures
* Fix integration test
* Drop reliance on FilteredStreamRecords
* minor improvements
* Currently running optimizer count in metrics
* Clearly state the prerequisites of count_optimizers_running()
* Minor improvements
* improve metric naming
* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* Allow specifying initial state while creating shard key
* Dont wait for Active state if initial_state is set
* Generate gRPC docs and OpenAPI spec
* Apply suggestions
* Skip from scheamrs not serde
* Add integration test
* minor suggestions
* Disallow peer to join with URI that is already used
* Add test for rejecting peer join with duplicate URI
* Improve peer rejection logic
* Try to rejoin twice, we expect a consistent result
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* [strict-mode] Add max number of payload index count
* improve docs and validation for StrictModeConfig
* fix from coderabbit
* test blocking access to payload indexes
* polish test
* make sure that grpc::Vector can always use new format
* specify target version for ToDo
* Update lib/api/src/conversions/vectors.rs
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
