* fix load of the graph links mmap
* decide compression based on filename rather than bit hacking
* fix test
* fix conversion: always create a new file for compressed links
* fix conversion: adjust compressed flag after conversion
* fix test (again)
* Fixes
* let GraphLayers manage its own files
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* use mmap bitslice and a metadata file to avoid loading
* fix and test reported files
* add new files to flusher
* update rocksdb+blob_store compatibility test
* switch to DynamicMmapFlags
* fix setting deleted to `false` on short bitslice
* remove `MmapBitSlice::extend`
* add some more tests to mmap sparse vector storage
* calculate deleted_count and next_point_offset on load
* get rid of metadata file
* cleanup
* fix test
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* get total sparse size from inverted index
* populate on load if not present
* use debug assertion
* correct size calculation
* r u happy clippy
* fix total sparse size in immutable ram
* don't use options. move size_of_available_vectors to other traits
* adjust test
* fix rebase
* add on_disk option for bool index
* test that all files are covered
* generate openapi and docs
* clippy
* remove `populate` changes
* use `walkdir` crate
* Apply clippy suggestions
---------
Co-authored-by: timvisee <tim@visee.me>
* allow to read related issues with collection-level access
* refactor to use requirements
* patch test
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* add `MmapBitSlice::extend` helper
* mmap bool index implementation
* unit test both implementations
* switch to `DynamicMmapFlags`
* tidy up
* recalculate `indexed_count` on load
* grow bitslice aligned to the mmap page size
* ergonomic get_slice_for
* fix for growing mmap
* use more `get_slice_for()`
* Strict mode config: Max collection size
* api specs
* Add tests + set/update payload check
* Improve function names and add comments
* rename config to separate vectors and payload
* fix tests
* Adjust configs docs
* add benchmark
* improve performance by caching shard info
* add bench for size_info() and fix tests
* Also limit the batch-size for vector updates (#5508)
* Also limit the batch-size for vector updates
* clippy
* add lost commit
* Load cache on collection initialization
* add unit type to parameter name
* fix renaming in test
* clearer error message
* fix test
* review remarks
* remove unused function for now
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Keep temporary segment if not empty
* Fix is empty condition
* Swap condition because it's cheaper
* Keep segment if no appendable segment or if not empty
* gpu hnsw construction
* fix build
* are you happy clippy
* are you happy clippy
* review remarks
* more comments
* external gpu vector storage
* decompose gpu inserter
* are you happy codespell
* are you happy clippy
* implement mmap sparse vector storage
* add to VectorStorageEnum
* clippy
* add tests, fix both simple and mmap storages
* smol correction on total_vector_count
* add sparse storage type to config
* fix reading config without storage type
* generate openapi
* use blob_store by path
* hidden setting to enable new storage
* validate existing path in `BlobStore::open()`
* use new dir for each sparse vector name
* fix and rename `max_point_offset`
Plus some extra refactors
* add storage compat test, to always check both storages work
* fix opening of storage + other misc fixes
* FIX!!!
`Unset` operations in the Tracker weren't updating the
`next_pointer_id`. So, when reopening the storage, those points wouldn't
get marked as deleted in the bitslice, thus creating the illusion that
they should exist, when they did not.
* refactor naming from `iter_*` to `for_each_*`
* fix checking for BlobStore existance
* fix typo
* fix error message
* better docs for open_or_create
* fix after rebase
* test to demonstrate inconsistency failure during optimization
* Fix apply points mixing point versions, only apply operation to latest (#5528)
* Patch apply points, only apply point operations to latest point version
* Simplify test, we don't need owned strings in JSON macro
* Use ahash in apply points because we map to simple numbers
* Minor improvements
* preserve the update version before doing move_if_exists in proxy (#5531)
* preserve the update version before doing move_if_exists in proxy
* Simplify move_if_exists with flat if statements and early return
* Fix inverted condition, consider deleted if delete is new
* Use hash map keys iterator as more idiomatic Rust
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
* parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012991 +0100
parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012977 +0100
parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012901 +0100
parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012766 +0100
parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012730 +0100
vulkan api wrapper crate
fmt
gpu is not a workspace member
exclude gpu from workspace
disable tests for gpu crate
ash version up
change collection for pipeline builder
gpu test
gpu is not optional
check device features
remove feature
make all fields private
more comments
are you happy fmt
qdrant team as authors
gpu vectors
move changes from magic branch
move storage updates
are you happy codespell
are you happy clippy
don't make storage_impl public
review remarks
are you happy clippy
are you happy fmt
* Update lib/segment/src/index/hnsw_index/gpu/gpu_vector_storage/tests.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* review remarks
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
