* Add function to detect if storage is using RocksDB
* Add feature flag to toggle payload index migration
* Add naive payload index migration, rebuild from scratch if RocksDB
* Adjust payload index rebuild logs
* When rebuilding payload indices, clean up existing storages
* Update OpenAPI spec
* Use weak references to Gridstore in mutable payload index flushers
* Rework payload index clear/wipe functions, make them consistent
* Naively migrate RocksDB before loading indices
* Fix mmap payload index load, staged like others (#6837)
* Add staged loading of mmap numeric payload index
* Add staged loading of mmap map payload index
* Add staged loading of mmap geo payload index
* Add staged loading of mmap full text payload index
* Add staged loading of mmap bool and null payload index
* Minor tweaks
* Don't fail clearing file cache if file does not exist
---------
Co-authored-by: jojii <jojii@gmx.net>
* expose mmr in rest and grpc
* fix referenced vectors in mmr queries
* rename to MmrInput in grpc for consistency
* mmr is now an optional parameter of nearest query
* fix score_threshold at local shard
* fix conversion of rest -> collection
* fix and refactor planned query
* nits
* rename `lambda` to `diversity` in interface
* handle score_threshold within mmr calculation
* add openapi test
* fix candidate limit at collection level
* candidate_limit -> candidates_limit
* finish rename to candidates_limit
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Store payload index type in config
* Fix CI
* Use new function
* Review remarks
* Fix Ci
* Update lib/segment/src/index/struct_payload_index.rs
* Add is_appendable function to storage type
* Fix storage: store index type in payload index config (#6852)
* Restructure payload index config for backwards compatibility
See: <https://github.com/qdrant/qdrant/pull/6812#issuecomment-3057928785>
* Use mapping of combined schema and type, use storage type for serde
* Replace common functions with defer
* Minor adjustments
* Don't deserialize skip_rocksdb field in payload config if not set
* Rename all payload index types to snake_case format when serializing
* Fix clippy warnings when building without RocksDB
* Change type of FullPayloadIndexType from enum to struct
* Remove both from mutability, assume mutable there
* Also rename the payload index type to snake case
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* use vector statistics for scalar bq query
* fix minor error
* remove test_binary_scalar_internal test
* do NOT use special file for storing vector stats
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Enable Gridstore payload index storage, rename storage types
* Respect payload_index_skip_mutable_rocksdb feature flag
* Minor cleanup
* Fix map payload index Gridstore semantics, only create files if desired
The semantics of backing storages in payload indices are inconsistent.
This aligns the Gridstore backing storage to our RocksDB backing
storage, matching the create/open/load behavior. Now, it will only
create Gridstore files on disk if we request it to do so.
This commit only implements it on the map index. In following comments
the same will be implemented in all other payload indices too.
* Apply same Gridstore semantics to other payload index types
* Rename create parameter to create_if_missing
* Fix bench compilation errors
* add mmr as a ScoringQuery
* add mmr to CollectionQueryRequest
* handle mmr after collecting from shards
* use unwrap_or_else
* include mmr as a vector query
* ordering for MMR is None
* fix score threshold
* fmt + todo comment
* remove unused function
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Introduce new config to set default peer ID
* Add unit test to check persistence of default peer ID
* add missing update to main.rs
* fix clippy
* Log if using default peer ID
* Use non default peer id
* Move default_peer_id config to cluster config
* update main.rs
* minor var name refactor
* Add validation for default peer ID range
* Fix validation and ensure we dont generate Peer ID 0 randomly
* fmt
* WIP: add flag to break optimization loop on panic
* notification on manual budget change only
* move check for disk size before creting a temp segment
* # This is a combination of 7 commits.
* SameAsStorage default value
* fix coderabbit warnings
* neon for u8 bq
* sse for u8 bq
* fix windows build
* rename function
* add comments
* fmt
* fix arm build
* review remarks
* use sequential reads for building quantized vectors
* populate vector storages before indexing
* consistent naming
* Update lib/segment/src/fixtures/index_fixtures.rs
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* don't optimize when loading a mmap-backed immutable index
* clear cache after laoding text index from mmap
* Revert "clear cache after laoding text index from mmap"
This reverts commit 110199e8eb.
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Migrate RocksDB based payload storage into Gridstore
* Add migrate_rocksdb_payload_storage feature flag
* Minor tweaks
* Use iter to migrate all payloads
* Fix clippy warning
* Update OpenAPI spec
* Track if mmap payload storage is populated, set memory/disk storage type
* Add test
* Log amount of bytes we migrate instead of point count
* Remove TODO
* Render storage size migrated in human human readable format, not bytes
* On migration failure, drop full payload storage directory
* Fix missing import
* Add in-ram-mmap payload storage type
* When RocksDB is not compiled, select in-ram mmap payload storage
* Enable payload storage on-disk check again in config mismatch optimizer
* Add feature flag to build in-ram-mmap payload storage
* Tweak is_on_disk condition
* Bump version estimates in runtime feature flags
* check if we try to deactivate last initializing replica
* consider more cases
* Update lib/collection/src/shards/replica_set/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
