* put all flags impls in a single module
* use in appendable_dense_vector_storage
* use in appendable_mmap_multi_dense_storage
* use in mmap_sparse_vector_storage
* clippy
* Add comment we cannot rely on length of deleted vectors bitvec
---------
Co-authored-by: timvisee <tim@visee.me>
* separate flags buffering and in-memory container
* new bitvec flags
* fmt
* clippy
* make BufferedDynamicFlags public to crate only
* clear cache after loading
* clippyyy
* improve comment
* Use resize with a constant to eliminate a hot closure
* Add comment on why we take subslice of bitslice
---------
Co-authored-by: timvisee <tim@visee.me>
* TestRawScorerProducer to use VectorStorageEnum
* TestRawScorerProducer:🆕 accept distance as an argument
* Drop TMetric from TestRawScorerProducer
* Remove `impl VectorStorage for TestRawScorerProducer`
* Add TestRawScorerProducer::internal_scorer
* Remove TestRawScorerProducer::get_vector
* Rename TestRawScorerProducer::{get_scorer => scorer}
* Introduce BufferedDynamicFlags
* clippy and extra changes
* fix flushing
* add mutable index impl
* use MutableNullIndex instead of MmapNullIndex
* use BufferedDynamicFlags
* remove mmap_null_index
* take updates before returning the flusher
* bump total_point_count on add and remove point
* improve test case
* Only clean up mutable null index files if index exists
* Remove mention of cached in context of length
* refactor flags into a single `Option<Storage>`
---------
Co-authored-by: timvisee <tim@visee.me>
* Add test to catch bug
Bug: <https://github.com/qdrant/qdrant/issues/6880>
* Fix loading null index from wrong path
* Simplify test
* Also bump total point count when removing a point
* Add debug assertion, ensure we don't create null index in selector
* Update bug link
* Remove unused import
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* After payload index migration, delete RocksDB files
* Add comment on why it is safe to not clean up RocksDB files in some cases
* Update lib/segment/src/index/struct_payload_index.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Reformat
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Minor comment and log tweaks
* Persist payload index config when changed based on dirty flag
* Persist new index type when indices are rebuilt, they might have changed
* Still load RocksDB if some index still depends on it
* Fix typo
* Add function to detect if storage is using RocksDB
* Add feature flag to toggle payload index migration
* Add naive payload index migration, rebuild from scratch if RocksDB
* Adjust payload index rebuild logs
* When rebuilding payload indices, clean up existing storages
* Update OpenAPI spec
* Use weak references to Gridstore in mutable payload index flushers
* Rework payload index clear/wipe functions, make them consistent
* Naively migrate RocksDB before loading indices
* Fix mmap payload index load, staged like others (#6837)
* Add staged loading of mmap numeric payload index
* Add staged loading of mmap map payload index
* Add staged loading of mmap geo payload index
* Add staged loading of mmap full text payload index
* Add staged loading of mmap bool and null payload index
* Minor tweaks
* Don't fail clearing file cache if file does not exist
---------
Co-authored-by: jojii <jojii@gmx.net>
* Store payload index type in config
* Fix CI
* Use new function
* Review remarks
* Fix Ci
* Update lib/segment/src/index/struct_payload_index.rs
* Add is_appendable function to storage type
* Fix storage: store index type in payload index config (#6852)
* Restructure payload index config for backwards compatibility
See: <https://github.com/qdrant/qdrant/pull/6812#issuecomment-3057928785>
* Use mapping of combined schema and type, use storage type for serde
* Replace common functions with defer
* Minor adjustments
* Don't deserialize skip_rocksdb field in payload config if not set
* Rename all payload index types to snake_case format when serializing
* Fix clippy warnings when building without RocksDB
* Change type of FullPayloadIndexType from enum to struct
* Remove both from mutability, assume mutable there
* Also rename the payload index type to snake case
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Enable Gridstore payload index storage, rename storage types
* Respect payload_index_skip_mutable_rocksdb feature flag
* Minor cleanup
* Fix map payload index Gridstore semantics, only create files if desired
The semantics of backing storages in payload indices are inconsistent.
This aligns the Gridstore backing storage to our RocksDB backing
storage, matching the create/open/load behavior. Now, it will only
create Gridstore files on disk if we request it to do so.
This commit only implements it on the map index. In following comments
the same will be implemented in all other payload indices too.
* Apply same Gridstore semantics to other payload index types
* Rename create parameter to create_if_missing
* Fix bench compilation errors
* add mmr as a ScoringQuery
* add mmr to CollectionQueryRequest
* handle mmr after collecting from shards
* use unwrap_or_else
* include mmr as a vector query
* ordering for MMR is None
* fix score threshold
* fmt + todo comment
* remove unused function
---------
Co-authored-by: generall <andrey@vasnetsov.com>
