* Add decay expressions
* remove code duplication in conversion
* test and fix lambda roundtrip
* Remove debug print in decay midpoint validation
* Improve error messages for decay midpoint and scale validation
* use similar lambda for gauss and exp decays
* avoid abs before squaring
* add comment about decay range
* add debug assertion
* Add payload index filtering IO measurements for some indices
* Add payload index metric to api and telemetry
* Also account for index access overhead
* Review remarks
* Anonymize new HardwareUsage field
* Fix tests
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* create initial strucutres
* clippy
* start field-query refactoring
* start field-query refactoring (2/N)
* start field-query refactoring (3/N): duplicate is_empty/null condiftions as field condition
* start field-query refactoring (4/N): re-instate is_empty fallback in case new index is not built yet
* filter for is_empty/is_null
* implement add/remove point
* upd schema
* open and create of null-index
* create null-index
* fix test
* Update lib/segment/src/index/query_optimization/condition_converter.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* unit test for null-index
* more unit tests
* add openapi tests
* fmt
* fix for integartion tests
* rabbit review fix
* make [null] non-empty
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add docs about running coverage tests locally
* Add link to CI coverage report
* higlight that we only cover unit tests
* Add CI coverage report image
* Add more steps
* Remove OOM instructions
* Add limits for filter and conditions
* clippy
* Review remarks + nested condition test
* Fix opnapi specs
* Improve error message by giving info about limits and usage
* First naive implementation of local shard clean task
* Add wait and timeout parameters to clean shard API
* Cancel shard cleaning and mark as dirty when changing hash rings
* Expose shards undergoing cleaning in telemetry
* Properly cancel shard clean task, add drop guard to enforce
* Ensure we have a local shard to clean
* Replace existing local shard cleanup API
* Refactor how we create, manage and await clean tasks
* Invalidate only affected shards when committing read hash ring
* Expose all local shard clean task statuses in telemetry
* Properly invalidate shards when aborting resharding
* Add review remark comments
* Log clean task errors, extract task in dedicated function
* Join task when invalidating so we wait for completion
* Invalidate shard clean tasks in a batch to more efficiently join them
* Expose clean progress with number of deleted points
* Annotate cancel safety
* Fix comment
* When calling clean endpoint, ensure we have the specified local shard
* Invalidate shard cleaning when a local shard is being dropped
* Prevent anonymous type in shard clean status telemetry
* Cancel shard clean task directly by dropping future, fix possible deadlock
Before this change trying to invalidate shard cleaning tasks could
deadlock. The actual task only had two cancel points inside a read lock
on the shard holder. The shard holder is constantly released and
relocked.
In some of the places we can trigger invalidation already hold a shard
holder write lock. The task itself would have to grab a read lock in
order to reach the cancel point, in which case the locks would be
fighting each other.
Because invalidation also joins the task and waits for it to abort this
could get stuck forever.
Now we don't have cancellation points anymore and simply drop the whole
future in case of cancellation. That'll prevent it getting stuck.
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Allow max-optimization-thread config to be set to null on update
* add new field. Support both, but prepare for deprecation.
* better openapi
* only introduce change in OptimizersConfigDiff
* move to `api::rest::schema`
* update openapi
* improve test, fix diff to config conversion
* upd grpc docs
* clippy
* remove schemars from common common
---------
Co-authored-by: Gulshan Kumar <kumargu@amazon.com>
* add on_disk option for bool index
* test that all files are covered
* generate openapi and docs
* clippy
* remove `populate` changes
* use `walkdir` crate
* Apply clippy suggestions
---------
Co-authored-by: timvisee <tim@visee.me>
* Strict mode config: Max collection size
* api specs
* Add tests + set/update payload check
* Improve function names and add comments
* rename config to separate vectors and payload
* fix tests
* Adjust configs docs
* add benchmark
* improve performance by caching shard info
* add bench for size_info() and fix tests
* Also limit the batch-size for vector updates (#5508)
* Also limit the batch-size for vector updates
* clippy
* add lost commit
* Load cache on collection initialization
* add unit type to parameter name
* fix renaming in test
* clearer error message
* fix test
* review remarks
* remove unused function for now
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* implement mmap sparse vector storage
* add to VectorStorageEnum
* clippy
* add tests, fix both simple and mmap storages
* smol correction on total_vector_count
* add sparse storage type to config
* fix reading config without storage type
* generate openapi
* use blob_store by path
* hidden setting to enable new storage
* validate existing path in `BlobStore::open()`
* use new dir for each sparse vector name
* fix and rename `max_point_offset`
Plus some extra refactors
* add storage compat test, to always check both storages work
* fix opening of storage + other misc fixes
* FIX!!!
`Unset` operations in the Tracker weren't updating the
`next_pointer_id`. So, when reopening the storage, those points wouldn't
get marked as deleted in the bitslice, thus creating the illusion that
they should exist, when they did not.
* refactor naming from `iter_*` to `for_each_*`
* fix checking for BlobStore existance
* fix typo
* fix error message
* better docs for open_or_create
* fix after rebase
* copy-paste blob_storage
* get rid of copied utils
* fix clippy
* fmt
* audit blob_store manifest
* move `Payload` into `fixtures.rs`
* fix clippy
* put csv in google storage
* ignore hm test
* fix codespell
* improve files test
* test different block sizes
* remove unused deps
* replace StdRng with SmallRng for faster tests
* Add UUID to collection...
...and recreate collection, when applying Raft snapshot, if UUID of collection is different
* fixup! Add UUID to collection...
Remove UUID field from gRPC and exclude it from OpenAPI spec 🤡
* fixup! fixup! Add UUID to collection...
Always generate collection UUID 🤦♀️
* Raft snapshot recreate collection no expose UUID (#5452)
* separate colleciton config structure from API
* fmt
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
