* Serialize vector of ecow strings using cbor
* Adjust trait bounds for Gridstore support in map index
* Add Gridstore backend to mutable map index
* Add Gridstore builder
* Enable new map index in index selector
* Add tests
* Use different Gridstore block size depending on map data type
* Fix review remarks
* Extend Gridstore blob trait, support basic primitives and vector of them
* Implement mutable numeric index on top of Gridstore
* Wrap Gridstore in rwlock, flush asynchronously
* Set custom Gridstore options tuned to numeric index
* Fix compilation errors
* Destruct RocksDB selector everywhere
* Add Gridstore payload index selector, build numeric, fallback others
* Add feature flag to Gridstore mutable payload index
* Don't store empty values in gridstore, it's not supported
* Reformat
* Initialize Gridstore during init of index builder
* Use owned and referenced paths correctly around Gridstore
* Change Gridstore page size in numeric index as suggested
* Make Gridstore block size dependent on size of numeric type
* Add tests
* Add separate fixed size blob trait for vec, prevent footgun
* Improve page size calculation
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Simplify Blob requirement for numeric payload index
* Add missing test cases
* Replace ref mut with &mut
* Use correct hardware counter
* Implement clear cache for numeric payload index on Gridstore
* The block size is dynamic
* Error if trying to load index from storage that doesn't match backend
* fix counter
* Add comment on why we change page size
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add function to open all appendable dense vector storage types
* Add function to find appendable dense storage files without opening it
* Add function to migrate RocksDB dense vector storage into mmap
* Add feature flag and actively migrate on load
* Log number of migrated points
* Add function to open all simple dense vector storage types
* Add test for simple dense vector storage migration
* Use correct column families
* Use more generic feature flag
* Also migrate vector deletes
* Fix review remarks
* rename `Document`->`TokenSet`
* add point_to_doc to mutable text index
introduce point_to_doc
add TODO for using the new field
handle removals
* remove ser/de from `TokenSet`
* clippy
* review fixes
* Gridstore config can be Copy
* Add test asserting wipe behavior
* Add clear function to Gridstore
* Make wipe take ownership, use new clear function everywhere
* Add comment to emphasize wipe behavior
* Propagate errors
* integrate generic posting list in full-text index
* improve intersection fn
improve iterator intersection
* make old module just for test
* recover old implementation of MmapPostings for tests
* add compatibility test
* fix compression
* clippy
* nits
* check if first id is already greater
protects against the case where the iterator's current element hasn't
been extracted yet
* update current element when there is no greater or equal
* reuse visitor in test
* optimize alignment
* bound checks and refactor is_in_range
* review
* edit comment
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Actively migrate RocksDB ID tracker to new format on segment load
* Extract ID tracker migration logic to function
* Feature flag ID tracker migration
* Simplify ID tracker migration by moving it deeper into load function
* Move migrate function to the bottom
* Add test to assert RocksDB to mutable ID tracker migration
* Assert new mutable ID tracker is empty
* Review remarks
* On RocksDB to mutable ID tracker migration failure, clean up files
* Demote empty mutable ID tracker to debug assertion
* Copy all point versions, including deleted, set known mappings
* read links and versions separatelly
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* use chunks_exact in compressed sparse posting list builder
* generalized impls
make both impls similar
new design
make impls similar
generalize impls
* separate into files in new crate
* additional builder fns
new fn for builder
* retrieve current iterator position
* simplify traits
* restructure
* add iterator
* clippy
add contains fn
* fix find_chunk
* move builders to builder
* revamp generics and traits
* add model test vs var-sized posting list
* fmt
* clippy
* improve traits and generics
* fix get_by_offset
* generalize tests
* improve test
* restructure view into a new file
* to and from components
* value handler takes closure to prevent perf penalty
* clippy
* remove unused deps
* revert changes in sparse index
* self review nits
* reword
* clippy
* edit comment
* lock CHUNK_LEN, but assert it is synced with BitPacker4x
* remove Sized constraint in iterator
* rename VarSized* to Unsized*
And add more from_components helpers
* avoid intermediate allocation when writing UnsizedValue
* checked u32 conversions
* fix test
* clippy
* Add debug assertion, assert that point ID is in range
* Bound check in get_by_offset
* improve tests to check against various lengths and offsets
* clippy 😤
---------
Co-authored-by: timvisee <tim@visee.me>
* Strict mode allows fullscan for multitenant payload index
* different error for missing payload index in multitenant case
* handle payload_m
* add simple test
* handle hnsw.m not set
* add another simple test
* fix test
* fallback to global HSNW config
* new error status code
* do not block on global HNSW and improver error reporting
* clearer error reporting
* review fixes
* fix test error messages
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add no-rocksdb feature flag, disable dense vector storage using RocksDB
* Disable multi dense vector storage using RocksDB
* Feature flag now unused function arguments
* Invert no-rocksdb, use rocksdb and enable by default
* RocksDB is required in some benches
* Feature flag RocksDB based sparse vector storage
* Propagate rocksdb feature flag to root crate
* Feature flag stored point, only used in context of RocksDB
* Mention what disabling RocksDB feature does in Cargo.toml
* Fix UnindexedField infra to handle lookup&range index requirements
* more generic approach
* better test
* restore
* small unit test for sanity
* review nits
* Do not recommend parametrized index
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Always run cancelletatiob handle for interrupted optimizations
* Stick to one way of checking cancelation
* Move segment optimization in sub function, cancel in single place (#6557)
* Move segment optimization into separate function, cancel in single place
* Don't use owned lock
* Unify cancellation check
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add congruence test for full text payload index
* Report what index variants we're testing
* Use string references
* deterministic keywords
* linting
* Use truncate instead of drain
* Don't check cardinality on deletion, indices behave differently
* Include index type in iterator
* Fix test compilation
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Fix unecessary unsafe in Clippy 1.87
* bump Rust version for intrinsics safe API
* ignore until we can bump the MSRV with cargo-chef
* remove unecessary change
* mark clearly as TODO
