* refactor(field_index): make FieldIndexRead a supertrait of PayloadFieldIndexRead
Removes the `get_payload_field_index_read() -> &dyn PayloadFieldIndexRead`
bridge from `FieldIndexRead` and the five default impls that forwarded
through it. The overlapping read methods now come from the supertrait
directly, eliminating one layer of dynamic dispatch on the hot read path:
`FieldIndex::filter` → variant match → concrete typed-index method.
`FieldIndex` gains a direct `impl PayloadFieldIndexRead` block with
per-method match arms, mirroring the existing dispatch shape used by
`get_telemetry_data`, `values_count`, etc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: apply nightly rustfmt
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(numeric_index): split mod.rs into focused submodules (#8997)
* refactor(numeric_index): split mod.rs into focused submodules
`numeric_index/mod.rs` was 1308 lines mixing the storage-dispatch enum,
the public `NumericIndex<T, P>` wrapper, three builders, per-(T, P)
`ValueIndexer` impls, and the `Encodable`/`StreamRange` traits — hard
to navigate.
Split into:
- `mod.rs` keeps the shared traits (`Encodable`, `StreamRange`),
`Range<T>::as_index_key_bounds`, and re-exports.
- `wrapper.rs` — `NumericIndex<T, P>` + inherent impl + `NumericIndexIntoInnerValue` trait.
- `builders.rs` — `NumericIndexBuilder`, `NumericIndexMmapBuilder`, `NumericIndexGridstoreBuilder`.
- `value_indexer.rs` — `ValueIndexer` and per-(T, P) `value_retriever` inherent impls.
- `storage/` — the `NumericIndexInner` dispatch enum:
- `storage/mod.rs` — enum + simple match-and-forward (constructors, lifecycle, telemetry, per-point access).
- `storage/statistics.rs` — histogram-driven cardinality and point-count helpers.
- `storage/trait_impls.rs` — `PayloadFieldIndex`, `PayloadFieldIndexRead`, `StreamRange` impls.
Pure code reorganization — no behavior change. A few inherent methods
on `NumericIndexInner` had to widen from private to `pub(super)` /
`pub(in crate::index::field_index::numeric_index)` to remain reachable
across the new module boundaries (and from `tests.rs`); the new
constructors (`NumericIndexMmapBuilder::new`,
`NumericIndexGridstoreBuilder::new`) replace direct field
construction across files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(numeric_index): rename wrapper.rs -> numeric_index.rs, move point_ids_by_value out of statistics
- Renamed `wrapper.rs` to `numeric_index.rs`, matching the central
`NumericIndex` type and the surrounding module name.
- Moved `point_ids_by_value` from `storage/statistics.rs` to
`storage/mod.rs` next to `get_values`. It is an exact value->points
lookup primitive, not a cardinality estimate; the statistics module
is left to the histogram-driven helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(clippy): rename numeric_index submodule to index to fix module_inception lint
Clippy's module_inception rule disallows a module with the same name as
its containing module. Rename numeric_index.rs -> index.rs and update
the three internal references.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(field_index): push PayloadFieldIndexRead through NumericIndex and move special_check_condition per-variant (#8998)
Two related cleanups that share a theme — drop enum-level read dispatch
in favor of per-variant trait impls:
1. `NumericIndex<T, P>` now implements `PayloadFieldIndexRead` directly
(forwarding to its inner storage enum). The four numeric arms in
`FieldIndex`'s `impl PayloadFieldIndexRead` drop their `.inner()`
calls, so all eleven variants now use a uniform `idx.<method>(...)`
form.
2. `special_check_condition` moves from `FieldIndexRead` to
`PayloadFieldIndexRead` with a default `Ok(None)` body.
`FullTextIndex` (the only variant with non-trivial logic) overrides
it. `NumericIndex<T, P>` forwards through to the inner enum, and
the `FieldIndex` enum's match dispatch moves out of `FieldIndexRead`
into `PayloadFieldIndexRead` for consistency with the other five
trait methods. Removed the now-redundant declaration from
`FieldIndexRead` (inherited via the supertrait).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor: implement FieldIndexRead for FieldIndex
Move the 10 read-only inherent methods on FieldIndex (special_check_condition,
count_indexed_points, filter, estimate_cardinality, for_each_payload_block,
get_telemetry_data, values_count, values_is_empty, as_numeric, as_facet_index)
into impl FieldIndexRead for FieldIndex. Bodies are unchanged.
Write/lifecycle methods (add_point, remove_point, wipe, flusher, files,
immutable_files, ram_usage_bytes, is_on_disk, populate, clear_cache,
get_full_index_type) stay inherent on FieldIndex.
Call sites that hold &FieldIndex now resolve through the trait — added
`use ... FieldIndexRead;` imports where the compiler asked
(read_view/{payload_index_read,filtering}.rs, query_optimization/
condition_converter.rs and match_converter.rs, payload_storage/
query_checker.rs, two integration tests, full_text_index tests module).
StructPayloadIndex::get_facet_index keeps its concrete
OperationResult<FacetIndexEnum<'_>> return: the trait method as_facet_index
returns Option<impl FacetIndex + '_> (RPITIT) which can't be downcast back
to FacetIndexEnum, so the function inlines the variant match. Same shape as
before, just expressed locally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: remove unused StructPayloadIndex::get_facet_index
Confirmed dead code — the function had no callers anywhere in the workspace.
It was preserved across recent refactors (most recently #8967) but was never
referenced. Removing it also drops the now-unused JsonPath and FacetIndexEnum
imports in struct_payload_index/mod.rs.
This obsoletes the inline variant match introduced in the previous commit:
that match existed solely to keep get_facet_index returning the concrete
FacetIndexEnum<'_> when as_facet_index moved to the FieldIndexRead trait
with an opaque return. With get_facet_index gone, no workaround is needed.
The OperationError::MissingMapIndexForFacet variant stays — it is still used
by segment::read_view::facet and lib/collection/src/operations/types.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(index): drop PayloadIndex: PayloadIndexRead super-trait
`PayloadIndex` now only declares the mutating surface; reads live on
the sibling `PayloadIndexRead` trait. The previous super-trait
relationship forced any type that implemented `PayloadIndex` to also
implement `PayloadIndexRead`, blocking a future `PayloadIndexRead`-only
view that doesn't (and shouldn't) own the writable index machinery.
No behavioural change. Audit before committing showed no generic
bound site on `PayloadIndex` exists in the workspace, and every
caller that uses read methods already imports `PayloadIndexRead`
explicitly (the trait was already used as a generic bound on
`SegmentReadView`'s `TPayloadIndex` parameter and on
`iter_filtered_points`). The full workspace builds clean and all
segment / storage / collection tests pass without any consumer
update.
Doc comment on `PayloadIndex` updated to point readers at
`PayloadIndexRead` for the read surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(index): introduce StructPayloadIndexReadView<P, I, V> (#8970)
Move the read surface of `StructPayloadIndex` onto a new borrowed view
struct generic over `<P: PayloadStorageRead, I: IdTrackerRead, V: VectorStorageRead>`.
The view holds exactly the fields that `PayloadIndexRead` requires --
no more, no less:
pub struct StructPayloadIndexReadView<'a, P, I, V> {
payload: &'a Arc<AtomicRefCell<P>>,
id_tracker: &'a I,
vector_storages: &'a HashMap<VectorNameBuf, Arc<AtomicRefCell<V>>>,
field_indexes: &'a IndexesMap,
config: &'a PayloadConfig,
visited_pool: &'a VisitedPool,
}
`StructPayloadIndex` now exposes a `with_view(|v| ...)` accessor that
borrows `id_tracker` once at the top and constructs the view for the
closure scope. All read-method bodies move onto the view, which is the
sole `PayloadIndexRead` implementor for this index.
Why three generics
==================
- `P: PayloadStorageRead` -- already generic via PR #8968.
- `I: IdTrackerRead` -- direct method calls; held as `&I` (not
`&Arc<AtomicRefCell<I>>`) because the cell is collapsed at the
`with_view` boundary, saving a per-method `borrow()` atomic op.
`dyn IdTrackerRead` does not satisfy `I: IdTrackerRead` bounds in
Rust without an explicit blanket impl, so generic is the only
consistent option here.
- `V: VectorStorageRead` -- the only access site is
`available_vector_count()` for the `HasVector` cardinality branch
(`condition_cardinality` in `read_view/filtering.rs`).
Why `payload` keeps the `Arc`
=============================
`PayloadProvider<P>::new(...)` (introduced in PR #8968) takes
`Arc<AtomicRefCell<P>>` so that the returned `FormulaScorer<'q>` /
`Box<dyn FilterContext + 'a>` can outlive the caller frame. The view
therefore holds `&'a Arc<AtomicRefCell<P>>` (asymmetric vs the bare
`&I` for `id_tracker`). Switching to a borrow-based provider would
require reworking `formula_scorer` / `filter_context` to callback
style; deferred to a follow-up if needed.
What does NOT move
==================
- `build_field_indexes` and `clear_index_for_point` stay on
`StructPayloadIndex`. `build_field_indexes` is read-shaped but only
has write-side callers, and pulls in the `selector` machinery which
uses `path` + `storage_type`. Keeping it on the writable struct
means `path` and `is_appendable` do not need to leak into the view.
- The `selector` / `selector_with_type` helpers stay on the writable
struct for the same reason.
- The free helpers in `query_optimization/condition_converter.rs`
(range / geo / null / is-empty checkers) stay where they are; their
visibility is bumped from `fn` to `pub(in crate::index)` so the view
can still call them.
Module layout
=============
lib/segment/src/index/struct_payload_index/
mod.rs # owning struct + with_view
build.rs # write-side build coordination
payload_index.rs # impl PayloadIndex (mutating only)
tests.rs
read_view/
mod.rs # view struct + module wiring
payload_index_read.rs # impl PayloadIndexRead for view
filtering.rs # struct_filtered_context, condition_cardinality, query_field, estimate_field_condition
condition_converter.rs # impl block from query_optimization/
optimizer.rs # impl block from query_optimization/
value_retriever.rs # impl block from query_optimization/
tests.rs # smoke test that builds the view directly
Consumer migration
==================
- `Segment::with_view` nests the new `StructPayloadIndex::with_view`
inside it; `SegmentReadViewFor<'s>` uses the view as its
`TPayloadIndex` parameter.
- HNSW (`hnsw.rs`), sparse (`sparse_vector_index.rs`), plain
(`plain_vector_index.rs`) call sites wrap their read-method calls in
`payload_index.borrow().with_view(|v| ...)`.
- `Segment::get_indexed_fields`, `update_all_field_indices`, and
`SegmentBuilder::build` switch to `with_view` for `indexed_fields()`
/ `get_payload_sequential()`.
- Integration tests and benches similarly migrate.
- `set_payload` (still on `PayloadIndex` write impl) inlines its
former `self.get_payload(...)` call as `self.payload.borrow().get(...)`
to avoid going through `with_view` from a `&mut self` write path.
Smoke test (`read_view/tests.rs`) constructs the view directly over
`InMemoryPayloadStorage` + `InMemoryIdTracker` + an empty vector-storage
map, and exercises `indexed_fields()`, `query_points()`, and
`available_point_count()` -- proving the view is genuinely decoupled
from `StructPayloadIndex`. This is the abstraction PR 4 will use to
wire a read-only segment.
Verified
========
- `cargo build --workspace --tests --benches` -- green
- `cargo test -p segment --lib` -- 666 passed (665 + 1 new smoke test), 0 failed
- `cargo test -p segment --tests` -- 120 integration tests pass
- `cargo test -p storage --lib` -- 44 passed
- `cargo test -p collection --lib` -- 197 passed
- `cargo clippy -p segment --tests --benches` -- clean
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 5 leftover: with PayloadIndexRead.iter_filtered_points now on the
trait, filtered_read_by_index can move to the view alongside the other
three scroll helpers.
* `read_view/scroll.rs` gains `filtered_read_by_index` and the
`read_filtered` orchestrator.
* `segment/scroll.rs` is deleted entirely; `mod scroll;` removed from
`segment/mod.rs`.
* `Segment::read_filtered` collapses to a single
`with_view(|v| v.read_filtered(...))` delegator.
* `scroll_filtering_test.rs` integration test calls
`filtered_read_by_index` via `with_view`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 5 of the SegmentReadView migration.
New `read_view/scroll.rs` module hosts:
* `should_pre_filter` — payload-index cardinality estimation,
used by all three scroll-shaped trait methods (read_filtered,
read_ordered_filtered, read_random_filtered).
* `read_by_id_stream` — streamed enumeration of visible points.
* `filtered_read_by_id_stream` — streamed enumeration with a
payload-filter context applied.
`Segment`-side `read_filtered` collapses to a `with_view` orchestrator
(except for the `filtered_read_by_index` branch, see below).
`read_ordered_filtered` and `read_random_filtered` now route their
`should_pre_filter` calls through `with_view` (their own bodies migrate
in steps 6 and 7).
`filtered_read_by_index` stays on `Segment` for now: it depends on
`StructPayloadIndex::iter_filtered_points`, which is an inherent method
that takes the concrete `&IdTrackerEnum` and returns `impl Iterator`.
Migrating it cleanly requires extending `PayloadIndexRead` with a
trait-object-friendly version of that method, which is the same
prerequisite Steps 6/7/8 will need (sampling, order_by, facet all use
`iter_filtered_points`). I will do that as a focused pre-step before
Step 6.
`deferred_internal_id` / `deferred_deleted_count` view helpers bumped
from private to `pub(super)` so the new `scroll` module can call them.
`scroll_filtering_test.rs` integration test updated to call
`filtered_read_by_id_stream` via `with_view`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Read-only methods now live on a separate IdTrackerRead trait, with the
mutating IdTracker trait extending it. This lets read-only call sites
depend only on the read API.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
* measure filterable hnsw indexed vs unindexs + cpu vs gpu
* measure filterable hnsw indexed vs unindexs + cpu vs gpu
* limit GPU parallelism
* address review
* Propagate HardwareCounterCell through MmapMapIndex::get_values
Previously, `MmapMapIndex::get_values` used `ConditionedCounter::never()`,
which silently skipped all hardware IO counter tracking for mmap map index
value reads. This propagates a real `HardwareCounterCell` through the full
call chain so that disk IO from `values_iter` is properly measured.
Changes:
- `MmapMapIndex::get_values` now accepts `&HardwareCounterCell`, creates a
`ConditionedCounter` via `make_conditioned_counter`, and measures the
`deleted` bitvec access (matching `check_values_any` behavior).
- `MapIndex::get_values` forwards the counter to the `Mmap` variant.
- `FacetIndex::get_point_values` trait method now accepts `&HardwareCounterCell`,
propagated through `FacetIndexEnum` and both impls (MapIndex, BoolIndex).
- `indexed_variable_retriever` accepts and forwards the counter to
`IntMapIndex`, `KeywordIndex`, and `UuidMapIndex` calls.
- `SegmentBuilder::update` and `_get_ordering_value` accept and propagate
the counter instead of creating disposable instances.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add test
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* Don't insert deferred points into sparse index
# Conflicts:
# lib/segment/src/segment/tests.rs
# lib/segment/src/segment_constructor/segment_constructor_base.rs
# Conflicts:
# lib/segment/src/segment_constructor/segment_constructor_base.rs
* Clippy
* Assert consistency of deferred_internal_id var
* Return before SparseVector conversion in case of deferred point
* Use debug_assert instead
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* UIO traits: require Sized
I don't think we ever going to add unsized implementation. So, lets,
require it on the whole trait rather than on individual methods.
* Move `AccessPattern` to the `common` package
Was: segment::vector_storage::vector_storage_base::AccessPattern
Now: common::generic_consts::AccessPattern
* UIO traits: replace `bool` with `AccessPattern`
Instead of `IdTracker` to have `iter_*` methods, it now has a `point_mappings()` method, returning a guard to `PointMapping` value. It prepares for moving the point_mappings under a lock to allow parallel searches and deletions.
Also, an incorrect unsafe in `Segment::iter_points` is replaced by a safe version with `self_cell`.
Reduce test parameters across 5 areas to cut test runtime without
sacrificing coverage:
1. HNSW PQ tests: lower vector dimensionality from 131 to 64 for product
quantization variants, cutting PQ codebook training time (~31s -> ~8s)
2. HNSW index build: use 2 threads instead of 1 for index construction;
the tests only check accuracy above a threshold so determinism is not
required
3. Search attempts: reduce from 10 to 5 query vectors per test
4. Rescoring: skip the expensive re-upsert-all-as-zeros rescoring check
for PQ tests (already covered by scalar quantization variants)
5. Near-miss speedups:
- WAL: reduce QuickCheck iterations from 100 to 50
- Gridstore: halve operation counts, drop 64-byte block size case,
reduce proptest cases for gap search
- Continuous snapshot: reduce timeout from 20s to 10s
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add ignore_deferred flag for internal scroll API
* Ignore deferred points in ProxyShard::update
* Use enum instead of bool as type in function parameters
* Use consistent parameter name
* fmt
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* More explicit naming
* Add DeferredBehavior to `count()` and disable filtering in stream_records
* Don't filter `retrieve()` everywhere
* Use consistent filter behavior in read operations
* Add TODO for ignored parameter in remote_shard
* Rebase fixes
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Implement filtering: Search
* Add tests for search and read_filtered
* Implement filtering: read_ordered_filtered + Tests
* Codespell + Clippy
* Reduce code duplication in tests
* Use helper methods of Id tracker
* Implement filtering: Random Scroll
* fixes after rebase
* [Unittest] Only set deferred ID if deferred points exist
* [Unit test] Reduce combinations in segment creation test
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* Introduce EdgeShardConfig for edge shard
- Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs
- Segment config (vector_data, sparse_vector_data, payload_storage_type)
- Global hnsw_config and per-vector HNSW in segment config
- Optimizer params: deleted_threshold, vacuum_min_vector_number,
default_segment_number, max_segment_size, indexing_threshold,
prevent_unoptimized (excludes memmap_threshold, flush_interval_sec,
max_optimization_threads)
- Persist/load as edge_config.json in shard path
- EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>,
falls back to file or infer from segments; compatibility checked on load
- load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig)
- optimize() uses EdgeShardConfig for hnsw and optimizer thresholds
- Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config()
(update and persist)
- Python and examples use load_with_segment_config with existing config API
Made-with: Cursor
* Refactor EdgeShardConfig: user-facing params only, config module
- Replace SegmentConfig inside EdgeShardConfig with user-facing fields:
- on_disk_payload (bool) instead of payload_storage_type
- vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector,
no per-vector quantization; global quantization_config only
- sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk
- EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of
storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in
to_segment_config()
- Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs
- from_segment_config(&SegmentConfig) fills all inferrable params
- to_segment_config() builds SegmentConfig for segments and optimize()
- load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config
Made-with: Cursor
* Move optimizer threshold helpers to shard crate
- Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb,
get_deferred_points_threshold_bytes in shard::optimizers::config
- Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these
- Single place for threshold logic; collection and edge use shard helpers
Made-with: Cursor
* Use destructuring in config conversions to avoid missing new fields
- EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config
- EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config
- EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config
Adding new fields to source structs will now cause compile errors until conversions are updated.
Made-with: Cursor
* refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic
- PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap)
- VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap)
- QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable)
- collection: use from_on_disk_payload in non-rocksdb branch
- edge shard/vectors: use new helpers; remove duplicated conditionals
- shard optimizers: use from_on_disk and for_appendable_segment
Made-with: Cursor
* refactor(edge): use EdgeShardConfig directly, drop segment_config
- Add plain_segment_config() for create_appendable_segment (no HNSW)
- Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers
- Add vector_data_config(name) for query/MMR
- build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig
- create_appendable_segment: use plain_segment_config()
- search/query: use config().vectors and vector_data_config() instead of segment_config()
- Remove segment_config() from EdgeShardConfig and EdgeShard
- Add to_plain_vector_data_config on EdgeVectorParams
Made-with: Cursor
* [manual] review changes
* refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams
- PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.)
- PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types
- PyEdgeOptimizersConfig for optional optimizer settings
- EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate
- cargo fmt + clippy (remove map_identity)
Made-with: Cursor
* refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig
- Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType,
PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings
- Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs
- Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors,
EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig
- Update examples (common.py, repr.py) to use new config API
- Run cargo fmt
Made-with: Cursor
* [manual] review changes
* [manual] review changes
* [manual] fix test
* Address CodeRabbit review comments for PR 8322 (#8324)
* Address CodeRabbit review comments for PR 8322
- Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API
- HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings
- EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors
- EdgeShardConfig::load: use try_exists(), propagate I/O errors
- from_segment_config: infer hnsw_config from per-vector HNSW when all agree
- EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors
- Segment compat: prefix vector name in error messages; resolve None datatype to Float32
- max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides
- SegmentOptimizerConfig:🆕 build plain and optimizer maps in single pass
- config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into()
- vectors.rs: doc updates for per-vector quantization
Made-with: Cursor
* Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer
- Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this')
- Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist
- set_vector_hnsw_config: clone then mutate then write (fallible setter)
- max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl')
- max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count
Made-with: Cursor
* [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* update docstring and pyi
* fmt
* fmt
* clipy
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Use `IdTrackerEnum` type instead of `dyn IdTracker`
It would allow to be more flexible on the IdTracker trait, making it
dyn-incompatible eventually.
Coauthored with Claude Code.
* Review fixes
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Split `SegmentEntry`
Add `ImmutableSegmentEntry` for operations that can be applied to
immutable segments, and make the `SegmentEntry` as it subtrait.
* Rename to `NonAppendableSegmentEntry`
It differs semantically from `ImmutableSegmentEntry` by allowing point
deletion. Move point deletion to the trait too.
* Fix docstring
* use NonAppendableSegmentEntry where possible
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* download tar
* compute sha256 for stream download
* wip: propagate unpacking into down to the logic, todo: validation
* unpacked snapshot validation
* Minor tweaks
* Fix typo
* validation during unpack
* cancellation token
* update docstring
* remove redundant dep
* Rearrange unpack functions
- Rename `safe_unpack.rs` into `tar_unpack.rs` so it would be listed
near `tar_ext.rs` in IDEs.
- Replace calls like `ar = open_snapshot_archive(…); safe_unpack(ar, …);`
with a single call to `tar_unpack_file(…)`.
- Put calls to `Archive::new(); Archive::set_overwrite(false);` inside
`tar_unpack_reader` (was `safe_unpack`). So, now it is the only place
that does `set_overwrite`.
* Let clippy complain if tar::Archive::unpack used
* Mock snapshot download URL
Instead of downloading from storage.googleapis.com every time the test
runs, put small snapshot file to the repo.
The snapshot file is created using this command:
curl -s \
https://storage.googleapis.com/qdrant-benchmark-snapshots/test-shard.snapshot \
| tar \
--delete segments/4ea958d8-0b64-4312-9a53-0cd857e93535.tar \
--delete segments/65ac6276-8cca-4f5c-b767-9722190cee8b.tar \
> lib/storage/src/content_manager/snapshots/test-shard.snapshot
File contents:
$ tar tf lib/storage/src/content_manager/snapshots/test-shard.snapshot
wal/
wal/closed-255
newest_clocks.json
replica_state.json
shard_config.json
$ du -sh lib/storage/src/content_manager/snapshots/test-shard.snapshot
12K lib/storage/src/content_manager/snapshots/test-shard.snapshot
$ sha256sum < lib/storage/src/content_manager/snapshots/test-shard.snapshot
5d94eac5c1ede3994a28bc406120046c37370d5d45b489a0d2252531b4e3e1f2 -
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* WIP: introduce new vector store type
* handling of InRamMmap
* fmt
* feature-flag
* fmt
* Use if else
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/common/common/src/flags.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* also choose madvise for single-file in-ram-mmap
* simplify generics
* gpu fix
* fix bug
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* feat: Add enable_hnsw option for payload field indexes
Add optional enable_hnsw parameter to all payload index types to control
whether additional HNSW graph links are built for each indexed field.
- Add enable_hnsw field to all 8 payload index param types
- Update gRPC proto definitions and conversions
- Update OpenAPI schema
- Modify HNSW graph builder to respect enable_hnsw flag
- Add enable_hnsw() helper methods to PayloadSchemaParams and PayloadFieldSchema
- Update all tests to include new field (default: None)
When enable_hnsw is true and payload_M > 0, additional HNSW links will
be built for the payload field. Default value is true for backward compatibility.
* Fix Some format problems
* fix: address comment problem
---------
Co-authored-by: EC2 Default User <ec2-user@ip-10-78-171-148.ec2.internal>
* Replace `Option<Segment>` with `enum LoadSegmentOutcome`
* Replace some Path/PathBuf with str/String
* Rename field Segment::{current_path -> segment_path}
* safe_delete
* Add progress_tracker.rs
* Pass progress tracker around
* Populate progress tracker with actual data
* Expose progress on `/collections/{name}/optimizations` endpoint
