* Use `IdTrackerEnum` type instead of `dyn IdTracker`
It would allow to be more flexible on the IdTracker trait, making it
dyn-incompatible eventually.
Coauthored with Claude Code.
* Review fixes
* Add function to open all appendable dense vector storage types
* Add function to find appendable dense storage files without opening it
* Add function to migrate RocksDB dense vector storage into mmap
* Add feature flag and actively migrate on load
* Log number of migrated points
* Add function to open all simple dense vector storage types
* Add test for simple dense vector storage migration
* Use correct column families
* Use more generic feature flag
* Also migrate vector deletes
* Fix review remarks
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* include document & image objects into grpc API
* introduce image and object to rest api
* minor refactoring
* rename Vector -> VectorInternal
* decompose vector data structures
* add schema
* fmt
* grpc docs
* fix conversion
* fix clippy
* fix another conversion
* rename VectorInput -> VectorInputInternal
* replace grpc TryFrom with async functions
* fmt
* replace rest TryFrom with async functions
* add image and object into query rest
* separate inference related conversions
* move json-related conversions into a separate file
* move vector-related transformations into a separate file
* move more vector related-conversions into dedicated module
* fix hnsw full scan threshold
* add test
* are you happy clippy
* separate open_vector_storage
* remove public fields from builder
* wip: do not create segment in builder before build
* avoid arc in storage test and low-level loading functions
* WIP: remove internal segment from SegmentBuilder
* fmt
* finalize segment builder fixes
* Revert "are you happy clippy"
This reverts commit c04afa6989.
* Revert "add test"
This reverts commit 8e7ad6207e.
* Revert "fix hnsw full scan threshold"
This reverts commit 8904443fcb.
* remove _daatabse from builder
* fix optimizer test
* fix id tracker versions persistence
* do flush for segment components on build
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Integrate Multivector at segment level
* more tests - green with lower accuracy
* decrease expected accuracy to fix test
* cleanup test
* multivector can not be empty
* vary number of vector per multivec point
* make SimpleDenseVectorStorage generic against VectorElementType
* make generic loading of the simple dense storage
* move memmap_dense_vector_storage
* move mmap_dense_vectors
* move appendable_mmap_dense_vector_storage
* fmt
* move dynamic_mmap_flags
* move simple_dense_vector_storage
* move PrimitiveVectorElement
* fmt
* make MmapDenseVectors generic
* make MemmapDenseVectorStorage generic to data type
* fix UringReader on non-linux platform
* make ChunkedMmapVectors generic of the vector element type
* make AppendableMmapDenseVectorStorage generic of the vector element type
* make PrimitiveVectorElement trait even more global
* make Metric generic over vector element type and refactor it into GenericMetric
* make DenseVectorStorage generic over vector element
* remove temorary trait for migrating Metric
* make CustomQueryScorer generic against vector element type
* refactor PrimitiveVectorElement to use Cow and allow owned conversions
* Move score post-processing out of metric object
* naive implementation of metrics for byte vectors
* add enum for vector query on segment search
* rename newly introduced types
* fix: handle QueryVector on async scorer
* handle QueryVector in QuantizedVectors impl
* fix async scorer test after refactor
* rebase + refactor on queue_proxy_shard.rs
* constrain refactor propagation to segment_searcher
* fmt
* fix after rebase
* Use resize rather than while-push loop
* Add deleted flags to simple vector storage
* Add deleted flag to memmap vector storage
* Map BitSlice on mmap file for deleted flags
* Use vector specific deletion BitSlice in RawScorer
* Use BitSlice for deleted points, fix check point logic, clarify names
* Extract div_ceil function to shared module
* We can use unchecked set and replace because we just checked the length
* Add deleted count function to vector storage
* Add vector storage point deletion tests
* Keep deleted state in simple vector storage with update_from, add test
* Keep deleted state in memmap vector storage with update_from, add test
* Simplify div_ceil
* Improve deletion handling in update_from in mmap vector storage
* Improve performance, use trickery to get BitSlice view over deleted mmap
* Use BitSlice where possible, construct BitVec more efficiently
* Incorporate vector specific delete flags in quantized raw scorer
* Don't pin MmapMut, it is not required
* With quantization, keep mmap deleted flags in RAM for better performance
* Advice the kernel to prepare deleted flags mmap for faster future access
* Simplify deleted bitslice access, add bound check, remove unused function
* Fix compilation on Windows
* Cleanup
* Rename delete functions to delete_{point,vec} to prevent confusion
* Use then_some rather than match a boolean
* Lock deleted flags in memory only when quantization is available
* Add docs and stabilize issue link to dev_ceil
* Flush deleted mmap when closing segment
This requires us to to wrap the memory map struct in an Arc and Mutex.
Though this may look inefficient, it doesn't have a negative side effect
on deleted flag performance, because the flags are accessed through a
BitSlice that is separate and doesn't use locking.
* Rename some point functions to vec because that makes more sense
* Simplify delete flag fetching option, use deref func instead of asterisk
* Do not calculate slice size manually, use size_of_val
* remove test raw scorer
* use deref in check
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* remove deleted flags from vector storage
* remove deleted flags from mmap
* new simple vector storage format
* are you happy clippy
* remove id_tracker from raw_scorer
* revert vector storage format changes
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* many named vectors per point (segment-level)
* operation result for dim function
* beautifulized vector name
* fix naming bug
* segment version migration
* fmt
* add segment tests
* are you happy clippy
* fix build
* [WIP] many named vectors per point (collection-level) (#975)
* config and search
* fix placeholders for proxy segment move
* remove VectorType from collection
* are you happy fmt
* vectors in grps messages
* create collections with vectors
* segment holder fixes
* are you happy fmt
* remove default vector name placeholders
* are you happy fmt
* are you happy clippy
* fix build
* fix web api
* are you happy clippy
* are you happy fmt
* record vector&vectors
* openapi update
* fix openapi integration tests
* segment builder fix todo
* vector names for update from segment
* remove unwrap
* backward compatibility
* upd openapi
* backward compatible PointStruct
* upd openapi
* fix record back-comp
* fmt
* vector configuration backward compatibility
* fix vetor storage size estimation
* fmt
* multi-vec segment test + index test
* fmt
* api integration tests
* [WIP] Named vectors struct (#1002)
* move to separate file
* named vectors as struct
* use cow
* fix build
* keys iterator
* avoid copy in PointStruct -> get_vectors
* avoid another copy
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* [CLIPPY] Fixed the warning for references of the user defined types
* [CLIPPY] Fix module naming issue
* [CLIPPY] Fix the last set of warnings and enable clippy check during CI
* Moved cargo fmt and cargo clippy into it's own action
* raw points scorer
* raw point scorer for memmap storage
* search interface prepare
* graph binary saving + store PointOffsetId as u32
* WIP: entry points
* connect new link method
* update libs + search layer method + visited list + search context + update rust
* implement Euclid metric + always use MinHeap for priority queue
* small refactor
* search for 0 level entry
* update visited pool to be lock free and thread safe
* use ef_construct from graph layer struct + limit visited links to M
* add metric pre-processing before on vector upsert
* old hnsw heuristic
* save hnsw graph for export
* search method + tests
* small fixes
* add benchmark and profiler
* build time optimizations
* use SeaHash
* remove unsed benchmark
* merge hnsw graph function
* WIP:HNSW index build function
* HNSW build_index with additional indexing
* refactor fixtures
* graph save and load test
* test and fixes for filterable HNSW
* enable hnsw index for query planning
* fix cardinality estimation tests + remove query planner as class
* small refactor
* store full copy of collection settings with collection + allow partial override on creation #16
* API for updating collection parameters #16
* refactor: move collection error -> types
* report collection status in info API #17
* update OpenAPI Schema
