* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Resume resharding driver on restart if resharding is active
* Update resharding state comment started text
Using since versus started. 'Since' better clarifies that the current
state has been active since that time. While 'started at' could lead to
confusion on whether that time is for starting the whole resharding
operation or just that state.
* Add resharding resumption test during migrating points
* Recover resharding hash ring after loading shards
* Update resharding resume test, interrupt at multiple stages, less points
* Remove unused parameter
* Update collection shard count at resharding shard creation/destruction
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Use debug_assert_eq
* Remove trailing comma
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Apply resharding filter only to existing shards, include shard selection
* Add test for stable exact point count during resharding
* Improve filtering, only two separate requests, filtered and non-filtered
* Make scrolling stable while resharding
* Add test for stable scroll during resharding
* Make search stable while resharding
* Add test for stable search during resharding
* Remove resharding post filter in retrieve
* Also assert cardinality point count in resharding test
* Fix typos and some tweaks
* Only clone filter if resharding is active
* Also minimize cloning with resharding filter on count request
* Add test for stable exact point count during resharding with indexing
* Update lib/collection/src/shards/shard_holder/mod.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/collection/src/collection/point_ops.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Also apply resharding filter to retrieve
* Restructure resharding filter usage in `Collection::retrieve`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add basic resharding test, asserting shard and point counts
* Add resharding replica balancing test
* Always propagate deletes with wait, fix first resharding test flakyness
* Don't count points in balance test for now
* Fix resharding balance test, initial shard may not be on first node
* Count points in resharding balance test
* Test resharding with concurrent updates
* Add point data consistency check to resharding tests
* Fix not using given collection name, list point counts in debug output
