* WIP: introduce new vector store type
* handling of InRamMmap
* fmt
* feature-flag
* fmt
* Use if else
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/common/common/src/flags.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* also choose madvise for single-file in-ram-mmap
* simplify generics
* gpu fix
* fix bug
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add segment ID in telemetry response
* Migrate optimization log segment IDS to use UUIDs
* fmt
* clippy
* Bring back proxy segment SegmentEntry impl
* update openapi spec
* Bring back segment ID
* update comment
* Precompute segment uuid from filesyste while creating segment
* Update return types
* Update OpenAPI spec
* remove segment_ prefix from uuid field and use inline block
* Update OpenAPI spec
* Trigger CI
* clippy
* Improve comments to differentiate between segment id and uuid
* Remove duplicate function that shadowed segment entry function
---------
Co-authored-by: timvisee <tim@visee.me>
* implementation of async batch vectors reading
* EXPERIMENT: async-io for reading on scroll
* disable on non-linux
* make retrieve sequential for test
* wip: implement vector reading via callback
* simplify operations, remove duplicates
* use batch retrieve also for post-processing search results
* clippy
* fix tests
* review fixes
* Replace big match statement with simple option filter and equal check
* Inline format arguments
---------
Co-authored-by: timvisee <tim@visee.me>
* feat: Add enable_hnsw option for payload field indexes
Add optional enable_hnsw parameter to all payload index types to control
whether additional HNSW graph links are built for each indexed field.
- Add enable_hnsw field to all 8 payload index param types
- Update gRPC proto definitions and conversions
- Update OpenAPI schema
- Modify HNSW graph builder to respect enable_hnsw flag
- Add enable_hnsw() helper methods to PayloadSchemaParams and PayloadFieldSchema
- Update all tests to include new field (default: None)
When enable_hnsw is true and payload_M > 0, additional HNSW links will
be built for the payload field. Default value is true for backward compatibility.
* Fix Some format problems
* fix: address comment problem
---------
Co-authored-by: EC2 Default User <ec2-user@ip-10-78-171-148.ec2.internal>
* id tracker persisted mappings offset
* remove obsolete check
* review remarks
* review remarks
* When persisting mappings, truncate file that is larger than we expect
* If persisting mappings fails, truncate file to what we had before
This isn't necessary, but it is nice to clean up partial mappings.
* Fix typos
* Fix typos
* Just truncate the file, we don't have to seek anymore
* Rename length variable to mappings_expected_len
---------
Co-authored-by: timvisee <tim@visee.me>
* Remove call to Archive::set_sync
Also, this method was the last remaining part of our `tar-rs` fork,
so we can switch to the upstream version now.
* Do syncfs
* Replace `Option<Segment>` with `enum LoadSegmentOutcome`
* Replace some Path/PathBuf with str/String
* Rename field Segment::{current_path -> segment_path}
* safe_delete
* Use RwLock for pending changes in MmapBitSliceBufferedUpdateWrapper
* Use RwLock for pending operations in DatabaseColumnScheduledDeleteWrapper
* Use RwLock for pending updates in DatabaseColumnScheduledUpdateWrapper
* Drop alive guard before reconciliation, we don't touch files after
* Remove redundant clone
* Update comments
* in MmapSliceBufferedUpdateWrapper
* in MmapBitsliceBufferedUpdateWrapper
* in MutableIdTracker's versions updates
* in MutableIdTracker's mapping updates
* clone updates only when non-empty
* only lock for reconciling pending changes
* simpler reconciling
* use Mutex as argument to ensure we only lock within reconciliation
* Refactor `PyUpdateOperation` constructors
* Add default parameters to `PyVectorDataConfig::new`
* Add `Repr` trait and `WriteExt` helper
* Implement `__repr__` for config types
* fixup! Implement `__repr__` for config types
Use `Copy` instead of `Clone`
* fixup! Implement `__repr__` for config types
Add basic test
* Implement `__repr__` for `PyPointId`
* Implement `__repr__` for `PyVector`
* Implement `__repr__` for `PyVectorInternal`
* Implement `__repr__` for `PyPayload`
* Implement `__repr__` for `PyValue`
* Implement `__repr__` for `PyPoint`
* Implement `__repr__` for `PyPointVectors`
* Implement `__repr__` for `PyRecord`
* Move `PyScoredPoint` into a separate file
* Implement `__repr__` for `PyScoredPoint`
* Cleanup examples
* fixup! Implement `__repr__` for `PyScoredPoint`
* Move `PyOrderValue` into separate file
* Add `PyScoredPoint::order_value`
* Implement `pyclass_repr` proc-macro attribute
* Implement `__repr__` for config types using `pyclass_repr` attribute
* Implement `__repr__` for `PySparseVector` using `pyclass_repr` attribute
* Implement `__repr__` for `PyPoint` using `pyclass_repr` attribute
* Implement `__repr__` for `PyPointVectors` using `pyclass_repr` attribute
* Implement `__repr__` for `PyRecord` using `pyclass_repr` attribute
* Implement `__repr__` for `PyScoredPoint` using `pyclass_repr` attribute
* Minor fixes and cleanups
* fixup! Minor fixes and cleanups
* rollback copy for quantization config
* rollback copy for quantization config
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* introduce IsAliveLock
* add tests
* rename to mark_dead
* update gridstore comment
* Update some comments
* Add must_use attribute to lock_if_alive
* Rename mark_dead to blocking_mark_dead
* make the handle take a `Weak` reference
* make dropping explicit
* use `Mutex::lock_arc` instead
* less nesting
* clippy
---------
Co-authored-by: timvisee <tim@visee.me>
* Add progress_tracker.rs
* Pass progress tracker around
* Populate progress tracker with actual data
* Expose progress on `/collections/{name}/optimizations` endpoint
* Initial plan
* Update timeout error handling to use milliseconds for better precision
Co-authored-by: agourlay <606963+agourlay@users.noreply.github.com>
* Use Duration type and debug format for timeout errors
Changed timeout() methods to accept Duration directly instead of u128 milliseconds.
This uses Rust's built-in Duration debug formatting ({:?}) which provides clean output:
- Sub-second: "500ms"
- Exact seconds: "1s"
- Mixed: "2.5s"
Updated all call sites to pass Duration directly instead of timeout.as_millis().
Co-authored-by: agourlay <606963+agourlay@users.noreply.github.com>
* fmt/clippy
* make it consistent
* Inline some arguments and imports
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: agourlay <606963+agourlay@users.noreply.github.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
* On Windows ARM64 builds, disable usage of neon
* Also disable optimized popcount on Windows ARM64
* fix quantization build
* revert changes in BQ
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
