* Improve telemetry logic and test
* Parametrize telemetry test
* Consistency hash peeer ID across telemetry
* clean test
* Use Option in segments telemetry
* updat openapi spec
* Avoid test failure on change in order of params
* Add in rest and grpc
* add to QueryEnum
* implement Query trait
* connect to scorer creation
* upd tests
* additional changes
* fmt
* gen openapi and grpc docs
* coderabbit fix
* add changes in async scorer
* test sum_scores in more places, refactor to remove repetition
* use f64 as internal expression score
* allow datetime as expression
* add test cases
* fix schema, generate openapi and grpc docs
* fmt
* clippy
* fix test precision
* graceful handling of too large numbers, more tests
* rename to DatetimeExpression for consistency
* use datetime's timestamp as seconds
* capture variable name when parsing
* homogenize DateTime into Datetime
* use interface distinction between datetime strings and payload keys
* fix rebase
* fix after rebase
* Add decay expressions
* remove code duplication in conversion
* test and fix lambda roundtrip
* Remove debug print in decay midpoint validation
* Improve error messages for decay midpoint and scale validation
* use similar lambda for gauss and exp decays
* avoid abs before squaring
* add comment about decay range
* add debug assertion
* Add payload index filtering IO measurements for some indices
* Add payload index metric to api and telemetry
* Also account for index access overhead
* Review remarks
* Anonymize new HardwareUsage field
* Fix tests
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* create initial strucutres
* clippy
* start field-query refactoring
* start field-query refactoring (2/N)
* start field-query refactoring (3/N): duplicate is_empty/null condiftions as field condition
* start field-query refactoring (4/N): re-instate is_empty fallback in case new index is not built yet
* filter for is_empty/is_null
* implement add/remove point
* upd schema
* open and create of null-index
* create null-index
* fix test
* Update lib/segment/src/index/query_optimization/condition_converter.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* unit test for null-index
* more unit tests
* add openapi tests
* fmt
* fix for integartion tests
* rabbit review fix
* make [null] non-empty
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add docs about running coverage tests locally
* Add link to CI coverage report
* higlight that we only cover unit tests
* Add CI coverage report image
* Add more steps
* Remove OOM instructions
* Add limits for filter and conditions
* clippy
* Review remarks + nested condition test
* Fix opnapi specs
* Improve error message by giving info about limits and usage
* First naive implementation of local shard clean task
* Add wait and timeout parameters to clean shard API
* Cancel shard cleaning and mark as dirty when changing hash rings
* Expose shards undergoing cleaning in telemetry
* Properly cancel shard clean task, add drop guard to enforce
* Ensure we have a local shard to clean
* Replace existing local shard cleanup API
* Refactor how we create, manage and await clean tasks
* Invalidate only affected shards when committing read hash ring
* Expose all local shard clean task statuses in telemetry
* Properly invalidate shards when aborting resharding
* Add review remark comments
* Log clean task errors, extract task in dedicated function
* Join task when invalidating so we wait for completion
* Invalidate shard clean tasks in a batch to more efficiently join them
* Expose clean progress with number of deleted points
* Annotate cancel safety
* Fix comment
* When calling clean endpoint, ensure we have the specified local shard
* Invalidate shard cleaning when a local shard is being dropped
* Prevent anonymous type in shard clean status telemetry
* Cancel shard clean task directly by dropping future, fix possible deadlock
Before this change trying to invalidate shard cleaning tasks could
deadlock. The actual task only had two cancel points inside a read lock
on the shard holder. The shard holder is constantly released and
relocked.
In some of the places we can trigger invalidation already hold a shard
holder write lock. The task itself would have to grab a read lock in
order to reach the cancel point, in which case the locks would be
fighting each other.
Because invalidation also joins the task and waits for it to abort this
could get stuck forever.
Now we don't have cancellation points anymore and simply drop the whole
future in case of cancellation. That'll prevent it getting stuck.
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Allow max-optimization-thread config to be set to null on update
* add new field. Support both, but prepare for deprecation.
* better openapi
* only introduce change in OptimizersConfigDiff
* move to `api::rest::schema`
* update openapi
* improve test, fix diff to config conversion
* upd grpc docs
* clippy
* remove schemars from common common
---------
Co-authored-by: Gulshan Kumar <kumargu@amazon.com>
* add on_disk option for bool index
* test that all files are covered
* generate openapi and docs
* clippy
* remove `populate` changes
* use `walkdir` crate
* Apply clippy suggestions
---------
Co-authored-by: timvisee <tim@visee.me>
* Strict mode config: Max collection size
* api specs
* Add tests + set/update payload check
* Improve function names and add comments
* rename config to separate vectors and payload
* fix tests
* Adjust configs docs
* add benchmark
* improve performance by caching shard info
* add bench for size_info() and fix tests
* Also limit the batch-size for vector updates (#5508)
* Also limit the batch-size for vector updates
* clippy
* add lost commit
* Load cache on collection initialization
* add unit type to parameter name
* fix renaming in test
* clearer error message
* fix test
* review remarks
* remove unused function for now
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
