* Deprecate init_from
* Mark memmap_threshold as deprecated
* Mark locks API as deprecated
* Mark RBAC collection access payload filter as deprecated
* Allow deprecations in tests and benchmarks
* Fix bug that causes all replicas to die if node is restarted during resharding
* fix recursive async problem
* Avoid write lock unless required
* avoid using &mut when & is sufficient
* Remove is_in_progress since check_abort_resharding exists (#6806)
* On resharding abort, only abort transfers related to current operation
* Fix resharding dead replicas improvements (#6881)
* Apply suggestions to fix resharding dead replicas
* fmt
---------
Co-authored-by: timvisee <tim@visee.me>
* add mmr as a ScoringQuery
* add mmr to CollectionQueryRequest
* handle mmr after collecting from shards
* use unwrap_or_else
* include mmr as a vector query
* ordering for MMR is None
* fix score threshold
* fmt + todo comment
* remove unused function
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Introduce new config to set default peer ID
* Add unit test to check persistence of default peer ID
* add missing update to main.rs
* fix clippy
* Log if using default peer ID
* Use non default peer id
* Move default_peer_id config to cluster config
* update main.rs
* minor var name refactor
* Add validation for default peer ID range
* Fix validation and ensure we dont generate Peer ID 0 randomly
* fmt
* Flag index selector
* Flag take_database_snapshot
* Flag field index builder
* Flag bool index
* Flag full text index
* Flag geo index
* Flag map index
* Feature flag numeric index
* Flag skip_rocksdb
* Flag payload storage
* Flag segment
* Flag builder
* Flag backup
* Make RocksDB crate optional
* Tweak feature flag gate
* Disable default segment crate features from all dependents
* Handle flag in collection crate correctly
* Flag payload storage types in tests
* Temporarily disable on-disk check for sparse vectors
* Flag sparse vector RocksDB in GPU code
* Fix compilation error since recent merge
* Flag payload storage types in new function
* remove todo
* Explicitly implement default to more clearly state difference
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Don't batch query if there's more segments than CPUs, limit threads
* Don't batch query if there's more segments than search threads
* Link to pull request
* Add note that we only consider local segments and global search threads
* Improve telemetry logic and test
* Parametrize telemetry test
* Consistency hash peeer ID across telemetry
* clean test
* Use Option in segments telemetry
* updat openapi spec
* Avoid test failure on change in order of params
* move problems.rs to collection crate
* change type on datetime_key expression
* extract from expression
* check for unindexed fields in formula
* clippy
* dedup code
* Only implement verification for `CollectionQuery` types (#6319)
* verify CollectionQuery's Prefetch
* remove implementation for REST query
* remove implementation for REST query groups
* add basic openapi test
* Add in rest and grpc
* add to QueryEnum
* implement Query trait
* connect to scorer creation
* upd tests
* additional changes
* fmt
* gen openapi and grpc docs
* coderabbit fix
* add changes in async scorer
* test sum_scores in more places, refactor to remove repetition
* Scroll lock: make sure no segments are modified between scroll and retrieve
* Take write lock in block_in_place to prevent blocking tokio runtime
* Use blocking write
* Recover collection snapshot on general runtime
* Make function const
---------
Co-authored-by: timvisee <tim@visee.me>
* Simplify `ShardKeyMapping` impl
* Automatically switch to new mappings format, if `Number` shard key is used
* Apply TODOs
* Add comment describing the issue, leave instructions for migration
* Cleanup
Remove unnecessary assignment
* Cleanup
Remove misleading comment
---------
Co-authored-by: timvisee <tim@visee.me>
* Propagate shard key mapping wrapper deeper
* Also bump reverse shard mapping when applying shard keys from snapshot
* Also apply new shard key mappings to existing replicas
* Apply shard key to replica set through apply_state directly
* Set both new shard key mappings through utility function
* Set shard key mappings directly during directory creation
* Rename get_key to key
* Update comment
* Use safe shard key wrapper type in consensus snapshot state
* Add shard ID iterator
* Produce net wrapped shard key mapping type directly
* Add useful comments
* Fix typo
* Add payload index filtering IO measurements for some indices
* Add payload index metric to api and telemetry
* Also account for index access overhead
* Review remarks
* Anonymize new HardwareUsage field
* Fix tests
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Anonymize ShardKey in ReplicaSetTelemetry
* Anonymize vector names in CollectionConfigInternal
* no-op: impl HardwareUsage for HardwareUsage (used in HardwareTelemetry)
* no-op: impl Anonymize for OptimizersStatus (used in OptimizerTelemetry and CollectionsAggregatedTelemetry)
* no-op: impl Anonymize for TrackerStatus in (used in TrackerTelemetry)
* no-op: Anonymize VectorParams in VectorsConfig
* no-op: Copy fields explicitly in OperationDurationStatistics
* no-op: `.clone().map(|x| x.anonymize())` -> `.anonymize()`
* no-op: impl Anonymize for ConsensusThreadStatus (used in PeerInfo and ClusterStatusTelemetry)
* Add retry-after header for gRPC
* propagate retry-after within inter node communication
* snipe
* tweak and test
* log all parsing errors
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* rename cpu_budget -> resource_budget
* clippy
* add io budget to resources
* fmt
* move budget structures into a separate file
* add extend permit function
* dont extend existing permit
* switch from IO to CPU permit
* do not release resource before aquiring an extension
* fmt
* Review remarks
* Improve resource permit number assertion
* Make resource permit replace_with only acquire extra needed permits
* Remove obsolete drop implementation
* allocate IO budget same as CPU
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
Arnaud Gourlay