* Strict mode config: Max collection size
* api specs
* Add tests + set/update payload check
* Improve function names and add comments
* rename config to separate vectors and payload
* fix tests
* Adjust configs docs
* add benchmark
* improve performance by caching shard info
* add bench for size_info() and fix tests
* Also limit the batch-size for vector updates (#5508)
* Also limit the batch-size for vector updates
* clippy
* add lost commit
* Load cache on collection initialization
* add unit type to parameter name
* fix renaming in test
* clearer error message
* fix test
* review remarks
* remove unused function for now
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Keep temporary segment if not empty
* Fix is empty condition
* Swap condition because it's cheaper
* Keep segment if no appendable segment or if not empty
* gpu hnsw construction
* fix build
* are you happy clippy
* are you happy clippy
* review remarks
* more comments
* external gpu vector storage
* decompose gpu inserter
* are you happy codespell
* are you happy clippy
* implement mmap sparse vector storage
* add to VectorStorageEnum
* clippy
* add tests, fix both simple and mmap storages
* smol correction on total_vector_count
* add sparse storage type to config
* fix reading config without storage type
* generate openapi
* use blob_store by path
* hidden setting to enable new storage
* validate existing path in `BlobStore::open()`
* use new dir for each sparse vector name
* fix and rename `max_point_offset`
Plus some extra refactors
* add storage compat test, to always check both storages work
* fix opening of storage + other misc fixes
* FIX!!!
`Unset` operations in the Tracker weren't updating the
`next_pointer_id`. So, when reopening the storage, those points wouldn't
get marked as deleted in the bitslice, thus creating the illusion that
they should exist, when they did not.
* refactor naming from `iter_*` to `for_each_*`
* fix checking for BlobStore existance
* fix typo
* fix error message
* better docs for open_or_create
* fix after rebase
* test to demonstrate inconsistency failure during optimization
* Fix apply points mixing point versions, only apply operation to latest (#5528)
* Patch apply points, only apply point operations to latest point version
* Simplify test, we don't need owned strings in JSON macro
* Use ahash in apply points because we map to simple numbers
* Minor improvements
* preserve the update version before doing move_if_exists in proxy (#5531)
* preserve the update version before doing move_if_exists in proxy
* Simplify move_if_exists with flat if statements and early return
* Fix inverted condition, consider deleted if delete is new
* Use hash map keys iterator as more idiomatic Rust
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
* parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012991 +0100
parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012977 +0100
parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012901 +0100
parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012766 +0100
parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012730 +0100
vulkan api wrapper crate
fmt
gpu is not a workspace member
exclude gpu from workspace
disable tests for gpu crate
ash version up
change collection for pipeline builder
gpu test
gpu is not optional
check device features
remove feature
make all fields private
more comments
are you happy fmt
qdrant team as authors
gpu vectors
move changes from magic branch
move storage updates
are you happy codespell
are you happy clippy
don't make storage_impl public
review remarks
are you happy clippy
are you happy fmt
* Update lib/segment/src/index/hnsw_index/gpu/gpu_vector_storage/tests.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* review remarks
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Expose storage size of RocksDB payload storage
* no strict check as numbers vary on various OS
* loose assertions
* try again for Windows
* assert storage size range
* copy-paste blob_storage
* get rid of copied utils
* fix clippy
* fmt
* audit blob_store manifest
* move `Payload` into `fixtures.rs`
* fix clippy
* put csv in google storage
* ignore hm test
* fix codespell
* improve files test
* test different block sizes
* remove unused deps
* replace StdRng with SmallRng for faster tests
* Optimize mmap sequential access for large scoring
* use it
* make windows happy
* add missing CPU counters
* Will not need
* more unix build
* add benchmark
* fix and bench cmp
* larger batch size
* start appendable mmap implem
* reenable debug for bench
* remove old impl for comparison
* impl for chunk mmap
* fix windows
* debug assert
* errare human est
* windows cfg
* generalize peek_top_iter
* remove blanket implementation to force explicit decision
* less TODO
* account for size of T
* fmr
* simplify chunk impl
* relax constraint on contiguous ids
* remove madvice WILLNEED and clean batching infra
* some premature optimizations
* extra sequential mmap for MMapVectorStorage
* add read-only typed mmap
* add read-only typed mmap
* fix comments
* implement batch reads for chuncked mmap
* fix is_read_with_prefetch_efficient
* cleanup
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* make the default `on_disk_payload` match the default config value
* remove duplicate impl, update docstrings
* change default `PayloadStorageType` too
* gen openapi
