Commit Graph

1958 Commits

Author SHA1 Message Date
Luis Cossío
dcbbdeba8e [UIO] Genericize ChunkedVectors.status (#8928)
* genericize ChunkedVectors.status, remove `Sized` bound

* check exists with `UniversalReadFileOps`

* Inline UioChunkedVectors bound, drop the alias (#8952)

The empty trait + blanket impl was a stable-Rust trait-alias workaround
that hid a fairly short bound (UniversalWrite<T> + UniversalWrite<Status>
+ Send + 'static) at the cost of an indirection readers had to mentally
unwind. Spelling it out at the three sites that need it is shorter overall
and immediately tells the reader what is required.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 13:51:48 +02:00
Roman Titov
2a47997654 Merge pull request #8791
* Add `EncodedStorage::for_each_in_batch` method

* Implement `for_each_in_batch` method for `QuantizedChunkedMmapStorage`

* Add `EncodedVectors::for_each_in_batch` method

* Add `EncodedVectors::score` method

* Implement `score_stored_batch` for `QuantizedQueryScorer` and `Quanti…

* Remove `TElement` and `TMetric` type parameters from `QuantizedMultiQ…

* Use `QuantizedMultiQueryScorer` when building `raw_internal_scorer`...
2026-05-07 20:20:54 +02:00
Luis Cossío
0957f61072 match exhaustively on match variants (#8947) 2026-05-07 19:46:38 +02:00
xzfc
d8e49327f8 Split mmap_hashmap into multiple files (#8933)
Co-authored-by: generall <andrey@vasnetsov.com>
2026-05-07 14:35:15 +00:00
Ivan Pleshkov
491712424d tq remove data fit option (#8943)
* tq disable data fit option

* remove any mention in grpc
2026-05-07 15:17:31 +02:00
Ivan Pleshkov
d8218ed368 1-2 bits rescoring (#8941) 2026-05-07 14:46:09 +02:00
qdrant-cloud-bot
94db47bb5b Fix nested MatchTextAny falling back to substring matching (#8936) (#8939)
`special_check_condition` in `FieldIndex` did not handle
`Match::TextAny`, so nested-condition evaluation fell through to the
`ValueChecker` fallback which uses naive `String::contains` instead of
proper full-text tokenization. This caused "good" to match "goodness"
inside nested filters.

Add `check_payload_match_any` to `FullTextIndex` and wire it into
`special_check_condition`. Also replace the catch-all `_ => None` with
explicit match arms for all `Match` variants.

Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-07 12:42:19 +02:00
Arnaud Gourlay
74bbee0b5f Mmap payload index hardening (#8921) 2026-05-06 13:50:43 +02:00
Daniel Boros
dddc6e9b6f feat/vector-storage-read-only (#8889)
* feat: add vector storage read enum

* fix: linter

* chore: remove open functions

* fix: compiler error

* review fix 1: remove constructiors (for now), use UniversalReadFamily, implement 1st part of VectorStorageReadEnum

* fmt

* read only chucked_vector_storage

* fnt

* feat: add dense chunked variants to vector storage read enum (#8916)

* feat: add multi dense chunked (#8918)

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-05-06 13:31:23 +02:00
Andrey Vasnetsov
a10698ce94 Read-only ID tracker (#8887)
* wip: read-only id tracker for mutable data

* read-only id tracker
2026-05-06 12:18:22 +02:00
Arnaud Gourlay
8c354acc70 Text index hardening (#8920) 2026-05-06 11:25:20 +02:00
Ivan Pleshkov
665b91d587 Apply p square for TQ+ (dont use std+mean) (#8877)
* apply p square

* review remarks and reduce ram

* review remarks

* clean tmp logs

* review remarks

* review remarks

* trigger ci
2026-05-06 09:36:00 +02:00
xzfc
d5789bb78b Callback-based InvertedIndex::for_each_token_id (#8905)
* TokenizerTextKind

* IteratorExt::try_any

* Callback-based InvertedIndex::for_each_token_id
2026-05-05 22:25:13 +00:00
Jojii
d3ad1ac988 API Adjustments for TQ (#8914)
* API Adjustments for TQ

* Clippy
2026-05-05 22:57:11 +02:00
Luis Cossío
449381bca8 [UIO] Generic Roaring/Bitvec flags (#8896)
* propagate to BufferedDynamicFlags

* use duplicate for tests

* propagate to Bitvec/Roaring flags

* propagate to RoaringFlags

* fixup! propagate to BufferedDynamicFlags

* propagate to BitvecFlags
2026-05-05 11:20:28 -04:00
Andrey Vasnetsov
1822f104f8 refactor(segment): split ChunkedVectors into read-only core + write wrapper (#8915)
Extract the read-side API into ChunkedVectorsRead<T, S: UniversalRead<T>>
and rebuild the existing ChunkedVectors<T, S: UniversalWrite<T>> on top of
it via composition + Deref. Lets read-only consumers use the storage
without pulling in UniversalWrite, and avoids method duplication.

Reorganizes the file into a chunked_vectors/ module: config (constants,
ChunkedVectorsConfig, Status), chunks (read_chunks/create_chunk helpers),
read (ChunkedVectorsRead), write (ChunkedVectors wrapper). read_chunks
now takes a writeable flag so both open paths share it.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 16:57:37 +02:00
Luis Cossío
c1ba01ba39 [UIO] Generic dynamic flags (#8893)
* rename status

* impl simple StoredStruct

* use StoredStruct in `status`

* rename to `DynamicStoredFlags`

* Propagate `S` generic

* codespell nit
2026-05-05 10:46:10 -04:00
Arnaud Gourlay
4e19e153b9 Mmap map index hardening (#8913) 2026-05-05 16:01:00 +02:00
Arnaud Gourlay
b9be572ac3 Numeric index hardening (#8912) 2026-05-05 14:16:18 +02:00
Arnaud Gourlay
49134ec26b Geo index hardening (#8910) 2026-05-05 12:26:32 +02:00
Arnaud Gourlay
0d4e7c4d8d Fix immutable map index zero count (#8890)
* fix(segment): immutable map index skips values with no live points on load

When MmapMapIndex::open ORs the id-tracker's runtime deletion bitvec
into the on-disk one at open time, ImmutableMapIndex::open_mmap could
insert a zero-count entry into value_to_points for any value whose live
points were all deleted, then immediately trip its own post-build sort
assert. Skip such values, mirroring the runtime invariant maintained by
remove_idx_from_value_list which already removes entries when their
count drops to zero.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(segment): file-system immutability test for payload indices

Builds an immutable segment with all 8 PayloadSchemaType variants
indexed, snapshots every byte under payload_index/, then asserts
byte-for-byte equality plus per-field query correctness after
delete_point, flush, drop+reload, and a second deletes+flush on the
reloaded segment. Each query exercises a different read path of an
immutable index variant: map exact-match (keyword/uuid/integer),
numeric range (float), datetime range, geo bounding box, full-text
token match, bool match. Reproduces the regression fixed in the
previous commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 19:45:10 +02:00
Andrey Vasnetsov
fd4e25d6ad refactor(segment): split immutable_id_tracker into a module (#8888)
Mirrors the layout of mutable_id_tracker: storage helpers for the
mappings, versions, and deleted bitslice files live in their own
submodules, leaving mod.rs focused on the ImmutableIdTracker type and
its trait impls.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 15:49:05 +02:00
Andrey Vasnetsov
deb21b4683 feat(segment): add ReadOnlyPayloadStorage backed by GridstoreReader (#8886)
Introduce a read-only payload storage that wraps `GridstoreReader<Payload>`
and implements `PayloadStorageRead`. Also rename the parameter on
`PayloadStorageRead::get`/`get_sequential` from `point_id` to `point_offset`
for consistency with the underlying storage API.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 15:00:01 +02:00
Andrey Vasnetsov
81ea504a40 refactor(segment): migrate info/size_info/telemetry to SegmentReadView (step 11) (#8882)
* refactor(index): extend VectorIndexRead and PayloadIndexRead for telemetry/info

Two trait extensions (no defaults — every implementor must opt in):

* \`VectorIndexRead::is_index\` — distinguishes a real index from a plain
  full-scan one. Used by reporting code. Moved out of inherent
  \`VectorIndexEnum::is_index\` into the trait. Explicit impls on Plain
  (false), HNSW (true), Sparse* (true).
* \`PayloadIndexRead::get_telemetry_data\` — per-field-index telemetry.
  Moved out of inherent \`StructPayloadIndex::get_telemetry_data\` into
  the trait impl. \`PlainPayloadIndex\` returns an empty Vec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(segment): migrate info/size_info/telemetry to SegmentReadView

Step 11 of the SegmentReadView migration — the final logical step.

New \`read_view/info.rs\` exposes builder methods:
* \`build_size_info(uuid, segment_type, is_appendable)\`
* \`build_info(uuid, segment_type, is_appendable)\` — same plus
  \`index_schema\`
* \`build_telemetry(uuid, segment_type, is_appendable, config, detail)\`

The trivial segment-level fields (\`uuid\`, \`segment_type\`,
\`is_appendable\`, \`config\`) are passed in by the caller — they stay
direct on each segment-type rather than going through the view.
Everything else (vector data breakdown, payloads size, deferred
counts, vector-index telemetry, payload-field telemetry, …) is
computed once inside the view through the read traits.

\`Segment::size_info\`, \`info\`, and \`get_telemetry_data\` collapse to
one-line \`with_view\` delegators that pass in the trivial fields.

Cleanup: \`Segment::deferred_deleted_count\` is now unused (the view
has its own equivalent helper); deleted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
84744cb4a6 chore(index): drop unused imports in formula_scorer
Leftovers from moving \`StructPayloadIndex::formula_scorer\` out of this
file. Caught by clippy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
7e5c3a63b5 refactor(index): drop FormulaScorerRead, return concrete FormulaScorer
\`FormulaScorer<'a>\` is already self-contained — it holds the parsed
formula, prefetch scores, retrievers and condition checkers, all owned
or borrowed independently of any payload index. Wrapping it in a trait
adds nothing (a future ReadOnlySegment can construct one too).

* Remove the \`FormulaScorerRead\` trait. \`score(point_id)\` goes back
  to being an inherent method on \`FormulaScorer\`.
* \`PayloadIndexRead::formula_scorer\` returns
  \`OperationResult<FormulaScorer<'q>>\` directly.
* \`StructPayloadIndex\` and \`PlainPayloadIndex\` updated to match.
* \`PlainPayloadIndex\` no longer needs the turbofish placeholder —
  \`Err(...)\` is enough.
* View's \`formula_rescore.rs\` drops the \`FormulaScorerRead\` import.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
1e9e2c6c17 refactor(segment): migrate formula rescore to SegmentReadView
Step 10 of the SegmentReadView migration.

Move \`segment/formula_rescore.rs\` → \`read_view/formula_rescore.rs\`:
* \`do_rescore_with_formula\` (private helper).
* \`rescore_with_formula\` (\`ReadSegmentEntry\` orchestrator).

Both now use the trait-method \`PayloadIndexRead::formula_scorer\`
(added in the prior commit) and \`IdTrackerRead::internal_id\` instead
of inherent calls.

\`Segment::rescore_with_formula\` collapses to a single
\`with_view(|v| v.rescore_with_formula(...))\` delegator. The legacy
\`segment/formula_rescore.rs\` is deleted; \`mod formula_rescore;\`
removed from \`segment/mod.rs\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
a5b030f1ba refactor(index): abstract formula_scorer behind FormulaScorerRead trait
A read-only payload index implementation will have its own concrete
formula scorer type, so PayloadIndexRead can't return the appendable
\`FormulaScorer<'a>\` directly.

* New \`FormulaScorerRead\` trait next to \`FormulaScorer\` exposing only
  what the rescore code path consumes (\`score(point_id)\`). Implemented
  for \`FormulaScorer<'_>\` by moving its inherent \`score\` into the
  trait impl.
* \`PayloadIndexRead::formula_scorer\` returns
  \`OperationResult<impl FormulaScorerRead + 'q>\` (RPITIT).
* The inherent \`StructPayloadIndex::formula_scorer\` (which lived in
  \`formula_scorer.rs\`) is moved into the trait impl block in
  \`struct_payload_index.rs\`, with the body delegating to a new
  \`FormulaScorer::new\` constructor (fields stay private).
* \`PlainPayloadIndex\` always returns
  \`Err::<FormulaScorer<'q>, _>(...)\` — formula scoring is not
  supported there. The turbofish supplies the placeholder type tag.
* Re-export \`FormulaScorer\` and \`FormulaScorerRead\` from
  \`rescore_formula::mod\`. \`retrievers_map\` bumped from \`pub(super)\`
  to \`pub(crate)\` so the trait impl can call it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
df630e80fe refactor(index): drop default impl of fill_idf_statistics, force per-impl
A new index implementation should never silently skip IDF wiring. Make
\`VectorIndexRead::fill_idf_statistics\` a required method (no default)
and provide explicit no-op impls for the dense indexes:
* \`PlainVectorIndex\` — no-op.
* \`HNSWIndex\` — no-op.

\`SparseVectorIndex\` keeps the real implementation: moved from a
sibling \`pub fn\` (inherent) into the \`VectorIndexRead\` trait impl
block, so there is now exactly one definition.

\`VectorIndexEnum\` already overrides per-variant from the previous
commit; nothing changes there.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
09a02c0407 refactor(segment): migrate search, retrieve, and fill_query_context to SegmentReadView
Step 9 of the SegmentReadView migration.

New \`read_view/search.rs\` hosts:
* \`retrieve\` — full body (deferred filtering, vector enrichment, payload
  attachment).
* \`process_search_result\` — converts internal \`ScoredPointOffset\`s into
  user-facing \`ScoredPoint\`s; calls the new \`view.retrieve\`.
* \`search_batch\` — \`ReadSegmentEntry\` orchestrator.
* \`fill_query_context\` — uses the trait \`VectorIndexRead::fill_idf_statistics\`
  (added in the prior commit) and the existing \`indexed_vector_count\`.

The \`Segment\`-side trait method bodies for \`search_batch\`, \`retrieve\`,
and \`fill_query_context\` collapse to single \`with_view\` delegators.
\`rescore_with_formula\` (whose body still uses Segment-side
\`do_rescore_with_formula\`, migrated in step 11) routes its
\`process_search_result\` call through \`with_view\`.

\`segment/search.rs\` is trimmed to just the
\`#[cfg(feature = "testing")] pub fn search\` helper, which calls
\`search_batch\` (now a \`with_view\` delegator) — no migration needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
366c29b476 refactor(index): move VectorIndexEnum::fill_idf_statistics to trait, drop indexed_vectors
Adds \`fill_idf_statistics\` to the \`VectorIndexRead\` trait with a
no-op default (only sparse-vector indexes track IDF). \`VectorIndexEnum\`
overrides it with the per-variant dispatch that previously lived as an
inherent method.

The inherent \`indexed_vectors\` is removed: it returned the same value
as the trait method \`indexed_vector_count\` for every variant
(sparse's \`indexed_vector_count\` is itself
\`inverted_index.vector_count()\`). The remaining caller in
\`fill_query_context\` will use \`indexed_vector_count\` after the next
commit moves it to the view.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
da145e7403 refactor(index): drop FacetIndexRead, reuse FacetIndex on FacetIndexEnum
The introduced \`FacetIndexRead\` trait was almost identical to the
existing \`FacetIndex\` trait — the only difference was the augmented
\`for_each_value\` signature on \`FacetIndexEnum\` that took
\`hw_counter\` and \`deferred_internal_id\` to skip values whose only
points are deferred.

* Move that "skip deferred" logic onto \`FacetIndex\` itself as a default
  method \`for_each_visible_value\` — implemented in terms of the
  existing \`for_each_value\` and \`for_each_value_map\`. All
  \`FacetIndex\` impls (MapIndex, BoolIndex, FacetIndexEnum) get it for
  free.
* Make \`FacetIndexEnum\` impl \`FacetIndex\` directly (delegating to the
  inner index's \`FacetIndex\` impl). The augmented inherent
  \`for_each_value\` is gone.
* \`PayloadIndexRead::facet_index_for\` now returns
  \`Option<impl FacetIndex + '_>\` (no separate \`FacetIndexRead\` trait).
* View facet code uses \`for_each_visible_value\` instead of the old
  augmented \`for_each_value\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
320f9f8685 refactor(segment): migrate facets to SegmentReadView
Step 8 of the SegmentReadView migration.

Move \`segment/facet.rs\` → \`read_view/facet.rs\`:
* \`approximate_facet\`
* \`facet_values\`

Both now go through the trait-method \`facet_index_for\` (added in the
prior commit) and the existing trait-method \`filter_context\` instead
of the previous inherent \`get_facet_index\` and
\`struct_filtered_context\` calls.

\`Segment::unique_values\` and \`Segment::facet\` (the \`ReadSegmentEntry\`
trait orchestrators) collapse to single \`with_view\` delegators. The
legacy \`segment/facet.rs\` is deleted; \`mod facet;\` removed from
\`segment/mod.rs\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
6f31812d03 refactor(index): abstract facet index access behind FacetIndexRead trait
Same shape as the numeric_index_for change in the previous PR: a
read-only segment will have its own per-key facet-index type, so
PayloadIndexRead can't return the concrete \`FacetIndexEnum<'a>\`.

* New \`FacetIndexRead\` trait in \`field_index/facet_index.rs\` mirrors
  the inherent methods of \`FacetIndexEnum\` (\`for_points_values\`,
  \`for_each_value\`, \`for_each_value_map\`, \`for_each_count_per_value\`).
* Implemented for \`FacetIndexEnum<'a>\` by delegating to the inherent
  methods.
* \`PayloadIndexRead::facet_index_for(key) -> Option<impl FacetIndexRead + '_>\`
  added (RPITIT, no boxing). Each impl picks its own concrete return
  type:
  - \`StructPayloadIndex\` returns \`Option<FacetIndexEnum<'_>>\`.
  - \`PlainPayloadIndex\` always returns \`None\`; the explicit
    \`None::<FacetIndexEnum<'_>>\` turbofish supplies a placeholder
    type tag.
* \`FacetIndexRead\` re-exported from \`field_index/mod.rs\` (the
  \`facet_index\` submodule itself stays \`pub(super)\`).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
82a8210433 refactor(segment): migrate random sampling to SegmentReadView
Step 7 of the SegmentReadView migration.

Move \`segment/sampling.rs\` → \`read_view/sampling.rs\`:
* \`read_by_random_id\`
* \`filtered_read_by_index_shuffled\`
* \`filtered_read_by_random_stream\`
* \`read_random_filtered\` orchestrator

\`Segment::read_random_filtered\` collapses to a single
\`with_view(|v| v.read_random_filtered(...))\` delegator. The legacy
\`segment/sampling.rs\` is deleted; \`mod sampling;\` removed from
\`segment/mod.rs\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
3eef48279b refactor(common): use EitherVariant for NumericIndexInner::stream_range
Drop the bespoke 4-arm \`NumericRangeIter\` enum introduced in the prior
commit. The existing \`common::either_variant::EitherVariant\` is the
same shape and already has \`Iterator\` plus all the standard adapter
specializations.

Adds a \`DoubleEndedIterator\` impl to \`EitherVariant\` (\`next_back\`,
\`nth_back\`, \`rfold\`, \`rfind\`) so it can be used in this position.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
6e2ad02db6 refactor(index): replace Box<dyn> with RPITIT in StreamRange and NumericFieldIndexRead
Both \`StreamRange<T>::stream_range\` and the new
\`NumericFieldIndexRead\` trait now return \`impl Iterator\` /
\`impl DoubleEndedIterator\` directly. No more boxed trait-object
allocations in the order-by hot path.

Mechanism:
* \`NumericIndexInner::stream_range\` has 4-way branching
  (empty / mutable / immutable / mmap), all with different concrete
  iterator types. Unified via a new 4-arm \`NumericRangeIter\` enum that
  delegates \`Iterator\` and \`DoubleEndedIterator\` to its variants.
* \`NumericFieldIndex::stream_range\` and \`get_ordering_values\` have
  2-way branching (Int / Float). Unified via \`itertools::Either\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
e5df30a434 refactor(index): abstract numeric_index_for return behind NumericFieldIndexRead trait
A read-only payload index implementation will have its own concrete
numeric-index type that doesn't fit the appendable
\`NumericFieldIndex<'a>\` enum. So PayloadIndexRead can't return that
concrete type directly.

Introduce \`NumericFieldIndexRead\` trait exposing only what ordered
reads consume (\`get_ordering_values\`, \`stream_range\`). Implement it
for the existing \`NumericFieldIndex<'a>\` (delegates to the inherent
methods + \`StreamRange\`).

\`PayloadIndexRead::numeric_index_for\` now returns
\`Option<impl NumericFieldIndexRead + '_>\` (RPITIT). Each
implementation picks its own concrete return type:
- \`StructPayloadIndex\` returns \`Option<NumericFieldIndex<'_>>\`.
- \`PlainPayloadIndex\` always returns \`None\` (it has no field
  indexes); the explicit \`None::<NumericFieldIndex<'_>>\` turbofish
  just supplies a placeholder type tag — no value is constructed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
35c1a7c61c refactor(segment): finish scroll migration — move filtered_read_by_index to view
Step 5 leftover: with PayloadIndexRead.iter_filtered_points now on the
trait, filtered_read_by_index can move to the view alongside the other
three scroll helpers.

* `read_view/scroll.rs` gains `filtered_read_by_index` and the
  `read_filtered` orchestrator.
* `segment/scroll.rs` is deleted entirely; `mod scroll;` removed from
  `segment/mod.rs`.
* `Segment::read_filtered` collapses to a single
  `with_view(|v| v.read_filtered(...))` delegator.
* `scroll_filtering_test.rs` integration test calls
  `filtered_read_by_index` via `with_view`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
dab0005e5c refactor(segment): migrate ordered reads to SegmentReadView
Step 6 of the SegmentReadView migration.

Move `lib/segment/src/segment/order_by.rs` into
`read_view/order_by.rs`:
* `filtered_read_by_index_ordered` — uses
  `payload_index.iter_filtered_points` (now on trait, see prior commit)
  and `numeric_index_for` (also now on trait).
* `filtered_read_by_value_stream` — uses `numeric_index_for` and
  `filter_context` (already on trait).
* `read_ordered_filtered` — `ReadSegmentEntry` orchestrator, also moved
  to the view.

`Segment::read_ordered_filtered` collapses to a single
`with_view(|v| v.read_ordered_filtered(...))` delegator. The legacy
`segment/order_by.rs` file is deleted entirely; `mod order_by;` removed
from `segment/mod.rs`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
dc5c55ed5d refactor(index): extend PayloadIndexRead with iter_filtered_points and numeric_index_for
Prerequisite for migrating order_by, sampling, facet, and scroll's
`filtered_read_by_index` to `SegmentReadView`. Both methods existed only
as inherent methods / public field access on `StructPayloadIndex`; the
view holds a generic `&TPI: PayloadIndexRead` reference and so can
only call trait methods.

* `iter_filtered_points` is now a trait method, generic over
  `I: IdTrackerRead` for the id_tracker parameter and using `impl Iterator`
  in return position (RPITIT). Each impl keeps its own zero-cost concrete
  iterator chain — no boxed dyn-iterator allocation. The trait becomes
  non-object-safe, which is fine because nothing currently uses
  `dyn PayloadIndexRead`.
* `numeric_index_for(key) -> Option<NumericFieldIndex<'_>>` exposes the
  per-key numeric-index lookup that ordered reads rely on, replacing
  direct access to `StructPayloadIndex::field_indexes`.
* Implementations on `StructPayloadIndex` (real) and `PlainPayloadIndex`
  (the latter has no numeric indexes, so `numeric_index_for` returns
  `None`).
* Existing internal callers (hnsw, scroll, sampling, order_by, facet)
  now pass `&*id_tracker` to fully deref `AtomicRef<IdTrackerEnum>` to
  `&IdTrackerEnum` for the generic-param inference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
bc353f4113 refactor(segment): migrate scroll helpers (3 of 4) to SegmentReadView
Step 5 of the SegmentReadView migration.

New `read_view/scroll.rs` module hosts:
* `should_pre_filter` — payload-index cardinality estimation,
  used by all three scroll-shaped trait methods (read_filtered,
  read_ordered_filtered, read_random_filtered).
* `read_by_id_stream` — streamed enumeration of visible points.
* `filtered_read_by_id_stream` — streamed enumeration with a
  payload-filter context applied.

`Segment`-side `read_filtered` collapses to a `with_view` orchestrator
(except for the `filtered_read_by_index` branch, see below).
`read_ordered_filtered` and `read_random_filtered` now route their
`should_pre_filter` calls through `with_view` (their own bodies migrate
in steps 6 and 7).

`filtered_read_by_index` stays on `Segment` for now: it depends on
`StructPayloadIndex::iter_filtered_points`, which is an inherent method
that takes the concrete `&IdTrackerEnum` and returns `impl Iterator`.
Migrating it cleanly requires extending `PayloadIndexRead` with a
trait-object-friendly version of that method, which is the same
prerequisite Steps 6/7/8 will need (sampling, order_by, facet all use
`iter_filtered_points`). I will do that as a focused pre-step before
Step 6.

`deferred_internal_id` / `deferred_deleted_count` view helpers bumped
from private to `pub(super)` so the new `scroll` module can call them.

`scroll_filtering_test.rs` integration test updated to call
`filtered_read_by_id_stream` via `with_view`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
51dce33ebc refactor(segment): migrate payload-access methods to SegmentReadView
Step 4 of the SegmentReadView migration.

New `read_view/payload.rs` module hosts:
* `payload_by_offset` (was `pub(super)` on Segment in `segment_ops.rs`)
* `payload` (`ReadSegmentEntry` trait method, real composition:
  `lookup_internal_id` + `payload_by_offset`)
* `estimate_point_count` (`ReadSegmentEntry` trait method, composes
  filter handling, payload-index cardinality and deferred-point
  adjustment)

The corresponding Segment-side trait methods become `with_view`
delegators, and `Segment::payload_by_offset` is deleted.

`Segment::lookup_internal_id` (a transitional `with_view` delegator
introduced in step 1) is now unused after `vector` and `payload`
moved off of it; deleted as well. Tests that called it now go
through `with_view`.

`get_indexed_fields` stays direct on Segment — single-line
`payload_index.borrow().indexed_fields()`, no shared logic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
c52b78e565 refactor(segment): migrate vector-access methods to SegmentReadView
Step 3 of the SegmentReadView migration.

Move the vector-access logic onto the view in a new
`read_view/vectors.rs` module:
* `vector_by_offset`, `vectors_by_offsets` (was `pub(super)` on Segment)
* `read_vectors` (was `pub(crate)` on Segment in the now-deleted
  `segment/vectors.rs`)
* `vector` and `available_vectors_size_in_bytes` (`ReadSegmentEntry`
  trait methods)

`Segment::vector` and `Segment::available_vectors_size_in_bytes`
collapse to `with_view` delegators. The `retrieve` trait method now
calls `view.read_vectors(...)` inside per-name `with_view` closures.

Removed entirely:
* `segment/vectors.rs` (its only function moved to the view)
* `Segment::vector_by_offset` and `Segment::vectors_by_offsets`
  in `segment_ops.rs`

Kept on Segment:
* `vector_names` — trivial `vector_data.keys().cloned().collect()`,
  no shared logic to factor out.
* `all_vectors` — invariant `NamedVectors<'_>` lifetime fights the
  view borrow, and the body is a 6-line iteration over `self.vector`
  (which is itself now a view delegator). ReadOnlySegment will write
  the same trivial loop.

Tests in `segment/tests/mod.rs` that exercised
`segment.vector_by_offset(...)` now go through
`segment.with_view(|v| v.vector_by_offset(...))`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
a6f6d896d5 refactor(segment): extract deferred-point view methods into read_view/deferred.rs
Move the six deferred-point methods (deferred_internal_id /
deferred_deleted_count helpers + deferred_point_count, has_deferred_points,
point_is_deferred, deferred_point_ids, available_point_count_without_deferred)
out of read_view/segment_ops.rs into their own read_view/deferred.rs module.
Keeps each read_view file focused on a single concern.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
d8ab4e014c style(segment): rustfmt fixup for has_deferred_points
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
71f1d64f36 refactor(segment): migrate deferred-point logic to SegmentReadView
Step 2 of the SegmentReadView migration.

Move the five composition-style deferred-point methods onto the view:
`deferred_point_count`, `has_deferred_points`, `point_is_deferred`,
`deferred_point_ids`, `available_point_count_without_deferred`. Each
combines the deferred-point status with id-tracker reads, so they are
real shared logic — not trivial getters.

The Segment-side trait method bodies in `entry.rs` collapse to
`self.with_view(|v| v.foo(...))` delegators.

`deferred_internal_id` and `deferred_deleted_count` are one-line
field accessors on `DeferredPointStatus`, so they stay direct on
`Segment`. The view has its own private helpers with the same
one-liner shape — duplicated only at the access-path level.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
fd5b93357e refactor(segment): finalize SegmentReadView shape; migrate read_range and lookup_internal_id
Step 1 of the SegmentReadView migration.

* `SegmentReadView` is expanded to its final field set, parameterised over
  four read-only traits (`IdTrackerRead`, `PayloadIndexRead`,
  `PayloadStorageRead`, `VectorDataRead`). A `SegmentReadViewFor<'s>` type
  alias hides the verbosity at call sites for `Segment`. Steps 2-11 only
  add `impl` methods; the struct shape is now frozen.
* `Segment::with_view` borrows the three `AtomicRefCell`-wrapped storages
  up front and populates every view field; closure bound relaxed
  `Fn` -> `FnOnce`.
* `lookup_internal_id` (Option -> PointIdError conversion) and
  `read_range` (iterator + take_while) are migrated to the view. The
  previous Segment-side helpers become thin `with_view` delegators where
  external callers still rely on them.
* Trivial id-tracker getters (`has_point`, `is_empty`,
  `available_point_count`, `deleted_point_count`, `total_point_count`)
  stay implemented directly on `Segment` -- one-liners that
  `ReadOnlySegment` will reimplement just as cheaply, no shared logic to
  factor out.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
07fdfb8f82 missing file 2026-05-04 13:31:35 +02:00
generall
bf1f1d4461 some more refactoring for segment view 2026-05-04 13:31:35 +02:00