* add test for shard recovery after partial flush
* force flush with explicit fucntion
* Stop flush worker before explicitly flushing again
---------
Co-authored-by: timvisee <tim@visee.me>
* fix segment repair on load
* workaround for rocksdb
* Minor comment tweaks
* In in memory ID tracker, unset version first before dropping mapping
* rename cleanup_versions into fix inconsistencies
---------
Co-authored-by: timvisee <tim@visee.me>
* Allow specifying initial state while creating shard key
* Dont wait for Active state if initial_state is set
* Generate gRPC docs and OpenAPI spec
* Apply suggestions
* Skip from scheamrs not serde
* Add integration test
* minor suggestions
* WIP: make proxy segment non-appendable
* make deleted points and index changes private to Proxy Segment
* consistently remove empty unused segment from holder on unproxifying and the end of optimization
* remove/fix obsolete test
* Fix comment, some general adjustments
* Also don't remove non-original segments
* Use ahash for map of proxy index changes
* Inline format arguments
* Simplify has point branching
* Assert that proxy flushes to latest wrapped segment version
* make vector deletetion a CoW operation
* relax debug assertions
* Update lib/shard/src/proxy_segment/mod.rs
* fix test_partial_snapshot_empty
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* separate snapshot-related functions into a new file
* move out snapshot-related function in local shard
* move snapshot test to local shard level
* move snapshot_all_segments and proxy_all_segments_and_apply to local shard
* use former scroll update lock to prevent snapshot operations overlapping with updates
* review fixes: rename all scroll_lock, use write lock for unproxy instead of read
Shutdown `shard` explicitly at the end of benchmark to avoid error
```
A Tokio 1.x context was found, but it is being shutdown.
```
Co-authored-by: Ivan Boldyrev <ivan.boldyrev@qdrant.com>
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Fix non visible points during snapshots
* Update lib/shard/src/segment_holder/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* code review from mr.rabbit
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Remove deprecated vectors count from collection info
* Remove vectors count from shard info
* Update OpenAPI and gRPC spec
* Remove vectors count from example
* make sure that grpc::Vector can always use new format
* specify target version for ToDo
* Update lib/api/src/conversions/vectors.rs
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
