* flush all segments in one thread sequentially
* bonus: move flush-related functions into dedicated file
* Minor comment tweaks
* await for flush on segment holder level
* fmt
* Minor improvement, preallocate vector for payload index flushers
* Remove invalid comment
---------
Co-authored-by: timvisee <tim@visee.me>
* WIP: make proxy segment non-appendable
* make deleted points and index changes private to Proxy Segment
* consistently remove empty unused segment from holder on unproxifying and the end of optimization
* remove/fix obsolete test
* Fix comment, some general adjustments
* Also don't remove non-original segments
* Use ahash for map of proxy index changes
* Inline format arguments
* Simplify has point branching
* Assert that proxy flushes to latest wrapped segment version
* make vector deletetion a CoW operation
* relax debug assertions
* Update lib/shard/src/proxy_segment/mod.rs
* fix test_partial_snapshot_empty
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* make update batch smaller to prevent long blocks
* limit vector update operations as well
* limit update chunk for point upsertion
* Fix typo
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* separate snapshot-related functions into a new file
* move out snapshot-related function in local shard
* move snapshot test to local shard level
* move snapshot_all_segments and proxy_all_segments_and_apply to local shard
* use former scroll update lock to prevent snapshot operations overlapping with updates
* review fixes: rename all scroll_lock, use write lock for unproxy instead of read
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Fix non visible points during snapshots
* Update lib/shard/src/segment_holder/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* code review from mr.rabbit
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Bump segment versions on *-by-filter operations to acknowledge WAL
* Assert exactly one segment with the new operation version
* Review remarks
* Separate segment version number to not affect partial snapshots
* Adjust unit test
* Remove old bump_segment_version implementations
* Clarify max_persisted_segment_version_overwrite in flush_all
* Move trait implementations - Clippy
* Apply segment bumping to all *-by-filter functions
* Remove AtomicOptionU64
* Update function name and comments
* Remove max condition, start with overwrite value
* Add assertion, can never have zero segments
* Make max_persisted_segment_version_overwrite monotonic with fetch_max
---------
Co-authored-by: timvisee <tim@visee.me>
* make sure that grpc::Vector can always use new format
* specify target version for ToDo
* Update lib/api/src/conversions/vectors.rs
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Add unit test with proxy that propagates older delete to wrapped segment
* Link to pull request
* Disable faulty debug assertion, replace with comment noting semantics
* Minor tweaks
* Point version check in optimizer makes sense, assert no double segment
* In proxy segment, don't return deleted point versions
More specifically, don't return a point version from the wrapped segment
if that point is already marked for deletion within the proxy.
* Do get wrapped segment point version if it's newer than soft delete
* Flatten function
