* First naive implementation of local shard clean task
* Add wait and timeout parameters to clean shard API
* Cancel shard cleaning and mark as dirty when changing hash rings
* Expose shards undergoing cleaning in telemetry
* Properly cancel shard clean task, add drop guard to enforce
* Ensure we have a local shard to clean
* Replace existing local shard cleanup API
* Refactor how we create, manage and await clean tasks
* Invalidate only affected shards when committing read hash ring
* Expose all local shard clean task statuses in telemetry
* Properly invalidate shards when aborting resharding
* Add review remark comments
* Log clean task errors, extract task in dedicated function
* Join task when invalidating so we wait for completion
* Invalidate shard clean tasks in a batch to more efficiently join them
* Expose clean progress with number of deleted points
* Annotate cancel safety
* Fix comment
* When calling clean endpoint, ensure we have the specified local shard
* Invalidate shard cleaning when a local shard is being dropped
* Prevent anonymous type in shard clean status telemetry
* Cancel shard clean task directly by dropping future, fix possible deadlock
Before this change trying to invalidate shard cleaning tasks could
deadlock. The actual task only had two cancel points inside a read lock
on the shard holder. The shard holder is constantly released and
relocked.
In some of the places we can trigger invalidation already hold a shard
holder write lock. The task itself would have to grab a read lock in
order to reach the cancel point, in which case the locks would be
fighting each other.
Because invalidation also joins the task and waits for it to abort this
could get stuck forever.
Now we don't have cancellation points anymore and simply drop the whole
future in case of cancellation. That'll prevent it getting stuck.
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Allow max-optimization-thread config to be set to null on update
* add new field. Support both, but prepare for deprecation.
* better openapi
* only introduce change in OptimizersConfigDiff
* move to `api::rest::schema`
* update openapi
* improve test, fix diff to config conversion
* upd grpc docs
* clippy
* remove schemars from common common
---------
Co-authored-by: Gulshan Kumar <kumargu@amazon.com>
* add on_disk option for bool index
* test that all files are covered
* generate openapi and docs
* clippy
* remove `populate` changes
* use `walkdir` crate
* Apply clippy suggestions
---------
Co-authored-by: timvisee <tim@visee.me>
* Strict mode config: Max collection size
* api specs
* Add tests + set/update payload check
* Improve function names and add comments
* rename config to separate vectors and payload
* fix tests
* Adjust configs docs
* add benchmark
* improve performance by caching shard info
* add bench for size_info() and fix tests
* Also limit the batch-size for vector updates (#5508)
* Also limit the batch-size for vector updates
* clippy
* add lost commit
* Load cache on collection initialization
* add unit type to parameter name
* fix renaming in test
* clearer error message
* fix test
* review remarks
* remove unused function for now
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* implement mmap sparse vector storage
* add to VectorStorageEnum
* clippy
* add tests, fix both simple and mmap storages
* smol correction on total_vector_count
* add sparse storage type to config
* fix reading config without storage type
* generate openapi
* use blob_store by path
* hidden setting to enable new storage
* validate existing path in `BlobStore::open()`
* use new dir for each sparse vector name
* fix and rename `max_point_offset`
Plus some extra refactors
* add storage compat test, to always check both storages work
* fix opening of storage + other misc fixes
* FIX!!!
`Unset` operations in the Tracker weren't updating the
`next_pointer_id`. So, when reopening the storage, those points wouldn't
get marked as deleted in the bitslice, thus creating the illusion that
they should exist, when they did not.
* refactor naming from `iter_*` to `for_each_*`
* fix checking for BlobStore existance
* fix typo
* fix error message
* better docs for open_or_create
* fix after rebase
* copy-paste blob_storage
* get rid of copied utils
* fix clippy
* fmt
* audit blob_store manifest
* move `Payload` into `fixtures.rs`
* fix clippy
* put csv in google storage
* ignore hm test
* fix codespell
* improve files test
* test different block sizes
* remove unused deps
* replace StdRng with SmallRng for faster tests
* Add UUID to collection...
...and recreate collection, when applying Raft snapshot, if UUID of collection is different
* fixup! Add UUID to collection...
Remove UUID field from gRPC and exclude it from OpenAPI spec 🤡
* fixup! fixup! Add UUID to collection...
Always generate collection UUID 🤦♀️
* Raft snapshot recreate collection no expose UUID (#5452)
* separate colleciton config structure from API
* fmt
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
* make the default `on_disk_payload` match the default config value
* remove duplicate impl, update docstrings
* change default `PayloadStorageType` too
* gen openapi
* add HwMeasurementAcc and API helpers+types
* add config to API to toggle hardware reporting in API responses
* add Default impl for HwMeasurementAcc
* comments
* review remarks
* include usage into API schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Track disk usage for each of the vectors and segments
* Dont extend vector_data field
* Remove comment and improve var name
* Introduce vectors_size_bytes and rename available_size_in_bytes
* Introduce vectors_size_bytes and rename available_size_in_bytes
* fix linter
* Avoid converting to u64
* calculate vectors_size_bytes for proxy segments
* Fix OpenAPI spec
* fix typo
* Restructure WAL delta logic, similar to what we had before the patch
* Tweak comments to elaborate on data race fix
* Refactor recovered function, not snapshot specific anymore
* Remove partial snapshot replica set state from schema
* include document & image objects into grpc API
* introduce image and object to rest api
* minor refactoring
* rename Vector -> VectorInternal
* decompose vector data structures
* add schema
* fmt
* grpc docs
* fix conversion
* fix clippy
* fix another conversion
* rename VectorInput -> VectorInputInternal
* replace grpc TryFrom with async functions
* fmt
* replace rest TryFrom with async functions
* add image and object into query rest
* separate inference related conversions
* move json-related conversions into a separate file
* move vector-related transformations into a separate file
* move more vector related-conversions into dedicated module
* Report CPU endianness in telemetry output
* Make endianness optional, don't report if anonymizing telemetry
* Update OpenAPI definition
* Switch back to single compile time endianness check
* Update OpenAPI specification
* Remove cpu-endian dependency
* De-anonymize CPU endianness
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* define mmap geo index
fix compilation
deleted flags
load new mmap
geo index tests
fix tests
fix build after rebase
add files list
* refactor get_stored_sub_regions output type
* review remanings
* mmap geo index api
* fix after rebase
