* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Anonymize ShardKey in ReplicaSetTelemetry
* Anonymize vector names in CollectionConfigInternal
* no-op: impl HardwareUsage for HardwareUsage (used in HardwareTelemetry)
* no-op: impl Anonymize for OptimizersStatus (used in OptimizerTelemetry and CollectionsAggregatedTelemetry)
* no-op: impl Anonymize for TrackerStatus in (used in TrackerTelemetry)
* no-op: Anonymize VectorParams in VectorsConfig
* no-op: Copy fields explicitly in OperationDurationStatistics
* no-op: `.clone().map(|x| x.anonymize())` -> `.anonymize()`
* no-op: impl Anonymize for ConsensusThreadStatus (used in PeerInfo and ClusterStatusTelemetry)
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* rename cpu_budget -> resource_budget
* clippy
* add io budget to resources
* fmt
* move budget structures into a separate file
* add extend permit function
* dont extend existing permit
* switch from IO to CPU permit
* do not release resource before aquiring an extension
* fmt
* Review remarks
* Improve resource permit number assertion
* Make resource permit replace_with only acquire extra needed permits
* Remove obsolete drop implementation
* allocate IO budget same as CPU
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
* Add `InternalUpdateParams` structure...
...to easier propagate `shard_id` and `clock_tag` internal API parameters
* fixup! Add `InternalUpdateParams` structure...
Use `InternalUpdateParams::default` instead of `Default::default`
* Remove now obsolete delinters, expect them where still applicable
---------
Co-authored-by: timvisee <tim@visee.me>
* Rename `UpdateParam` into `UpdateParams` and remove `Option`s from fields
* Use `common::inference::extract_token` in internal gRPC API
* Destructure `UpdateVectorsInternal`/`DeleteVectorsInternal` requests in internal gRPC API
* Add `extract_internal_request` helper for internal gRPC API
* fixup! Add `extract_internal_request` helper for internal gRPC API
Use `extract_internal_request` helper in `sync` API
* fixup! Rename `UpdateParam` into `UpdateParams` and remove `Option`s from fields
Remove `JsonSchema` derive from `UpdateParams`
* Retry-After on rate limiter hit for REST API
* fix tests
* Ceil retry after seconds number so clients don't retry too early
* Relax retry after time requirement a bit in case CI machines are slow
* Add rate limit test, test retry after if we replanish within a second
* Report time to wait in rate limit error
---------
Co-authored-by: timvisee <tim@visee.me>
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Measure read io for payload storage
* Add Hardware Counter to update functions
* Fix tests and benches
* Rename (some) *_measured functions back to original
* Add `resharding_enabled` option to `ClusterConfig`
If `resharding_enabled` is *not* set, `StartResharding` API returns an error
* Enable `resharding_enabled` option in development config
* fixup! Add `resharding_enabled` option to `ClusterConfig`
Fix tests
* fixup! Enable `resharding_enabled` option in development config
😅
* fixup! Enable `resharding_enabled` option in development config
🙄🙄🙄
* First naive implementation of local shard clean task
* Add wait and timeout parameters to clean shard API
* Cancel shard cleaning and mark as dirty when changing hash rings
* Expose shards undergoing cleaning in telemetry
* Properly cancel shard clean task, add drop guard to enforce
* Ensure we have a local shard to clean
* Replace existing local shard cleanup API
* Refactor how we create, manage and await clean tasks
* Invalidate only affected shards when committing read hash ring
* Expose all local shard clean task statuses in telemetry
* Properly invalidate shards when aborting resharding
* Add review remark comments
* Log clean task errors, extract task in dedicated function
* Join task when invalidating so we wait for completion
* Invalidate shard clean tasks in a batch to more efficiently join them
* Expose clean progress with number of deleted points
* Annotate cancel safety
* Fix comment
* When calling clean endpoint, ensure we have the specified local shard
* Invalidate shard cleaning when a local shard is being dropped
* Prevent anonymous type in shard clean status telemetry
* Cancel shard clean task directly by dropping future, fix possible deadlock
Before this change trying to invalidate shard cleaning tasks could
deadlock. The actual task only had two cancel points inside a read lock
on the shard holder. The shard holder is constantly released and
relocked.
In some of the places we can trigger invalidation already hold a shard
holder write lock. The task itself would have to grab a read lock in
order to reach the cancel point, in which case the locks would be
fighting each other.
Because invalidation also joins the task and waits for it to abort this
could get stuck forever.
Now we don't have cancellation points anymore and simply drop the whole
future in case of cancellation. That'll prevent it getting stuck.
* Support for json logging
* Rename enum name and verbose matches
* Use snake case formatting for log format variants
---------
Co-authored-by: Gulshan Kumar <kumargu@amazon.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Pass inference API token from request to inference service
---------
Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: generall <andrey@vasnetsov.com>
