Commit Graph

3917 Commits

Author SHA1 Message Date
Luis Cossío
dcbbdeba8e [UIO] Genericize ChunkedVectors.status (#8928)
* genericize ChunkedVectors.status, remove `Sized` bound

* check exists with `UniversalReadFileOps`

* Inline UioChunkedVectors bound, drop the alias (#8952)

The empty trait + blanket impl was a stable-Rust trait-alias workaround
that hid a fairly short bound (UniversalWrite<T> + UniversalWrite<Status>
+ Send + 'static) at the cost of an indirection readers had to mentally
unwind. Spelling it out at the three sites that need it is shorter overall
and immediately tells the reader what is required.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 13:51:48 +02:00
Roman Titov
2a47997654 Merge pull request #8791
* Add `EncodedStorage::for_each_in_batch` method

* Implement `for_each_in_batch` method for `QuantizedChunkedMmapStorage`

* Add `EncodedVectors::for_each_in_batch` method

* Add `EncodedVectors::score` method

* Implement `score_stored_batch` for `QuantizedQueryScorer` and `Quanti…

* Remove `TElement` and `TMetric` type parameters from `QuantizedMultiQ…

* Use `QuantizedMultiQueryScorer` when building `raw_internal_scorer`...
2026-05-07 20:20:54 +02:00
Luis Cossío
0957f61072 match exhaustively on match variants (#8947) 2026-05-07 19:46:38 +02:00
xzfc
d8e49327f8 Split mmap_hashmap into multiple files (#8933)
Co-authored-by: generall <andrey@vasnetsov.com>
2026-05-07 14:35:15 +00:00
Ivan Pleshkov
491712424d tq remove data fit option (#8943)
* tq disable data fit option

* remove any mention in grpc
2026-05-07 15:17:31 +02:00
Ivan Pleshkov
d8218ed368 1-2 bits rescoring (#8941) 2026-05-07 14:46:09 +02:00
qdrant-cloud-bot
94db47bb5b Fix nested MatchTextAny falling back to substring matching (#8936) (#8939)
`special_check_condition` in `FieldIndex` did not handle
`Match::TextAny`, so nested-condition evaluation fell through to the
`ValueChecker` fallback which uses naive `String::contains` instead of
proper full-text tokenization. This caused "good" to match "goodness"
inside nested filters.

Add `check_payload_match_any` to `FullTextIndex` and wire it into
`special_check_condition`. Also replace the catch-all `_ => None` with
explicit match arms for all `Match` variants.

Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-07 12:42:19 +02:00
Arnaud Gourlay
8ab30ee089 Reinstate edge add_vector API (#8925) 2026-05-06 18:15:55 +02:00
Andrey Vasnetsov
fd9bc02696 Add ordering parameter to gRPC create/delete vector name (#8926)
Match the REST API by adding an optional `WriteOrdering` field to
`CreateVectorNameRequest` and `DeleteVectorNameRequest`, and propagate
it through the tonic handlers and remote-shard forwarding paths.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 15:54:47 +02:00
Arnaud Gourlay
cf6f743952 Fix strict-mode disable race (#8924) 2026-05-06 15:26:58 +02:00
Arnaud Gourlay
74bbee0b5f Mmap payload index hardening (#8921) 2026-05-06 13:50:43 +02:00
Daniel Boros
dddc6e9b6f feat/vector-storage-read-only (#8889)
* feat: add vector storage read enum

* fix: linter

* chore: remove open functions

* fix: compiler error

* review fix 1: remove constructiors (for now), use UniversalReadFamily, implement 1st part of VectorStorageReadEnum

* fmt

* read only chucked_vector_storage

* fnt

* feat: add dense chunked variants to vector storage read enum (#8916)

* feat: add multi dense chunked (#8918)

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-05-06 13:31:23 +02:00
Andrey Vasnetsov
a10698ce94 Read-only ID tracker (#8887)
* wip: read-only id tracker for mutable data

* read-only id tracker
2026-05-06 12:18:22 +02:00
Arnaud Gourlay
8c354acc70 Text index hardening (#8920) 2026-05-06 11:25:20 +02:00
Ivan Pleshkov
665b91d587 Apply p square for TQ+ (dont use std+mean) (#8877)
* apply p square

* review remarks and reduce ram

* review remarks

* clean tmp logs

* review remarks

* review remarks

* trigger ci
2026-05-06 09:36:00 +02:00
xzfc
d5789bb78b Callback-based InvertedIndex::for_each_token_id (#8905)
* TokenizerTextKind

* IteratorExt::try_any

* Callback-based InvertedIndex::for_each_token_id
2026-05-05 22:25:13 +00:00
Jojii
d3ad1ac988 API Adjustments for TQ (#8914)
* API Adjustments for TQ

* Clippy
2026-05-05 22:57:11 +02:00
Luis Cossío
449381bca8 [UIO] Generic Roaring/Bitvec flags (#8896)
* propagate to BufferedDynamicFlags

* use duplicate for tests

* propagate to Bitvec/Roaring flags

* propagate to RoaringFlags

* fixup! propagate to BufferedDynamicFlags

* propagate to BitvecFlags
2026-05-05 11:20:28 -04:00
Luis Cossío
fb051280a4 Properly handle O_DIRECT in io-uring (#8833)
* handle o_direct with custom buffer

* make separate impl

* use correct inner offset

* align buffer

* codespell

* review fixes

* use o_direct in tests

* clippy

* fix simple read impl

* simpler to_vec()

* use router function

* change todo comment
2026-05-05 11:07:36 -04:00
Andrey Vasnetsov
1822f104f8 refactor(segment): split ChunkedVectors into read-only core + write wrapper (#8915)
Extract the read-side API into ChunkedVectorsRead<T, S: UniversalRead<T>>
and rebuild the existing ChunkedVectors<T, S: UniversalWrite<T>> on top of
it via composition + Deref. Lets read-only consumers use the storage
without pulling in UniversalWrite, and avoids method duplication.

Reorganizes the file into a chunked_vectors/ module: config (constants,
ChunkedVectorsConfig, Status), chunks (read_chunks/create_chunk helpers),
read (ChunkedVectorsRead), write (ChunkedVectors wrapper). read_chunks
now takes a writeable flag so both open paths share it.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 16:57:37 +02:00
Luis Cossío
c1ba01ba39 [UIO] Generic dynamic flags (#8893)
* rename status

* impl simple StoredStruct

* use StoredStruct in `status`

* rename to `DynamicStoredFlags`

* Propagate `S` generic

* codespell nit
2026-05-05 10:46:10 -04:00
Arnaud Gourlay
4e19e153b9 Mmap map index hardening (#8913) 2026-05-05 16:01:00 +02:00
Arnaud Gourlay
b9be572ac3 Numeric index hardening (#8912) 2026-05-05 14:16:18 +02:00
Arnaud Gourlay
49134ec26b Geo index hardening (#8910) 2026-05-05 12:26:32 +02:00
Roman Titov
4ef7b1e4a5 io_uring pipelinemaxxing (#8897) 2026-05-05 11:17:56 +02:00
Andrey Vasnetsov
591d68dc2c test: fix flaky test_shard_transfer_includes_deferred_points[snapshot] (#8860)
* test: keep optimizers disabled during snapshot transfer in deferred test

The snapshot variant of test_shard_transfer_includes_deferred_points was
flaky because optimizers were enabled before the transfer, letting the
optimizer race ahead and fully index the segment before the snapshot was
captured (~1s of HNSW build for 500 small vectors fits comfortably before
the snapshot is taken). The deferred-state assertion then fails since all
points are already visible.

Only enable optimizers before the transfer for stream_records (which needs
them for its internal wait=true). For snapshot, leave optimizers disabled
through the transfer so deferred state is preserved on the wire, then
enable them afterwards for trigger_upsert_wait_true. The hung server-side
wait=true from the timeout-and-retry block does not block the snapshot —
wait_for_deferred_points_ready runs in a detached tokio::spawn.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: skip wait=true probe for snapshot variant

CI showed that with optimizers kept disabled through the snapshot transfer
(needed to preserve deferred state on the wire), the wait=true probe at
the start of the test leaves a hung server-side request: update_local
holds local.read() until the deferred wait resolves, and there is no
optimizer to resolve it. The subsequent shard transfer's apply path
deadlocks against that held read lock when queue_proxify_local tries to
take local.write().

For stream_records the config update later cancels the hung worker, so
the probe is fine there. Move the probe (and config update) under the
stream_records branch so the snapshot variant doesn't leave a hung
update around. The probe was auxiliary behaviour verification, not
central to the snapshot-of-deferred-points assertion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Revert "test: skip wait=true probe for snapshot variant"

This reverts commit d07aac78a263d7a91691e43444b9dae44e3d179f.

* test: add reproducer for deferred-wait shard-transfer deadlock

Adds test_shard_transfer_with_hung_deferred_wait_does_not_deadlock as a
focused reproducer for the engine bug surfaced by the snapshot variant
of test_shard_transfer_includes_deferred_points.

Lock-ordering chain:

1. With prevent_unoptimized=true and max_optimization_threads=0, a
   wait=true upsert on deferred points enters
   wait_for_deferred_points_ready (update_worker.rs:241), which loops
   on tokio::select over cancel and optimization_finished. The
   optimization_worker hits limit==0 and `continue`s without firing
   optimization_finished_sender (optimization_worker.rs:172-174), so
   neither branch of the select ever fires.

2. update_local (replica_set/update.rs:49) holds self.local.read()
   across the entire update await. actix-web does not cancel the
   response future on client disconnect, so the read guard stays alive
   even after the client's 5s timeout.

3. A subsequent snapshot transfer eventually calls queue_proxify_local
   (replica_set/shard_transfer.rs:122), which needs self.local.write().
   tokio::sync::RwLock is write-preferring: the queued writer blocks
   new readers, including is_local() calls on the consensus apply
   path itself (shard_transfer.rs:129-130). The apply never returns,
   the consensus broadcast never fires, POST /cluster times out with
   "Waiting for consensus operation commit failed".

The new test asserts the symptom (POST /cluster must return promptly)
without papering over the bug, so it stays red until the engine is
fixed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(replica_set): release local read guard around deferred-points wait (#8862)

* fix(replica_set): drop remotes read guard early in update_impl

`update_impl` was holding `self.remotes.read()` and `self.local.read()`
across the entire update await, including the deferred-points wait that
can park indefinitely under prevent_unoptimized + max_optimization_threads=0.

When a shard transfer is started concurrently with a parked wait=true
update, the consensus apply runs `add_remote`, which calls
`self.remotes.write().await`. tokio::sync::RwLock is write-preferring:
the queued writer is blocked behind the held read, the apply never
returns, and `POST /cluster` times out with "Waiting for consensus
operation commit failed".

Fix: snapshot updatable remote shards into owned `Vec<RemoteShard>` and
drop the read guard before the await. The remote_update futures now own
the cloned RemoteShards, so they no longer borrow from the guard.

The `local` guard is still held across the await (futures borrow
`&Shard` from it). Releasing it would unblock `queue_proxify_local`'s
`local.write()` too, but that requires wrapping `Shard` in `Arc` —
deferred to a follow-up. For the consensus-commit-timeout deadlock
exposed by `test_shard_transfer_with_hung_deferred_wait_does_not_deadlock`,
dropping `remotes` is sufficient.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(updater): wake deferred wait on caller-receiver drop

`wait_for_deferred_points_ready` parked on a `tokio::select` over
`cancel.cancelled()` and `optimization_finished_receiver.changed()`.
Under prevent_unoptimized + max_optimization_threads=0, neither fires:
optimization_worker.rs:171-174 hits `limit == 0` and `continue`s
without notifying, and the cancel token is the worker's lifecycle
token (only fired by stop_update_worker on config update / shutdown).
The top-of-loop `is_closed()` poll didn't help — the loop never
re-runs once the select parks.

Take `feedback_sender` by `&mut` and add `feedback_sender.closed()`
as a third select branch. When the matching `Receiver` is dropped
(by upstream cancellation, client-supplied timeout, or any future
cancellation), the detached task wakes immediately and exits with
WaitTimeout instead of staying parked until the next worker restart.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* [AI] split update operarion into submit and independent wait function

* [AI] refactor `update_local` to drop local shard lock after submitting update operation

* [AI] refactor `update_impl` for early release of the lock in case of local shard update

* fmt

* Apply suggestion from @generall

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 09:51:34 +02:00
dependabot[bot]
07378597e9 build(deps): bump siphasher from 1.0.2 to 1.0.3 (#8901)
Bumps [siphasher](https://github.com/jedisct1/rust-siphash) from 1.0.2 to 1.0.3.
- [Commits](https://github.com/jedisct1/rust-siphash/compare/1.0.2...1.0.3)

---
updated-dependencies:
- dependency-name: siphasher
  dependency-version: 1.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-05 06:56:08 +02:00
Roman Titov
cb34d95417 UniversalRead cleanup (#8894)
* Cleanup lifetimes and generic type parameters

- Rename read pipeline lifetime from `'a` into `'file`
- Use explicit `where` clauses everywhere

* Cleanup

* fixup! Cleanup lifetimes and generic type parameters
2026-05-04 23:03:08 +02:00
Jojii
8765e33515 TQ Hadamard SIMD (#8883)
* [ai+manual] hadamard SIMD

* [ai] Neon

* Minor refactor

* fix import
2026-05-04 21:31:27 +02:00
Roman Titov
6942b54bfb Simplify UniversalReadPipeline (#8850) 2026-05-04 19:57:29 +02:00
Arnaud Gourlay
0d4e7c4d8d Fix immutable map index zero count (#8890)
* fix(segment): immutable map index skips values with no live points on load

When MmapMapIndex::open ORs the id-tracker's runtime deletion bitvec
into the on-disk one at open time, ImmutableMapIndex::open_mmap could
insert a zero-count entry into value_to_points for any value whose live
points were all deleted, then immediately trip its own post-build sort
assert. Skip such values, mirroring the runtime invariant maintained by
remove_idx_from_value_list which already removes entries when their
count drops to zero.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(segment): file-system immutability test for payload indices

Builds an immutable segment with all 8 PayloadSchemaType variants
indexed, snapshots every byte under payload_index/, then asserts
byte-for-byte equality plus per-field query correctness after
delete_point, flush, drop+reload, and a second deletes+flush on the
reloaded segment. Each query exercises a different read path of an
immutable index variant: map exact-match (keyword/uuid/integer),
numeric range (float), datetime range, geo bounding box, full-text
token match, bool match. Reproduces the regression fixed in the
previous commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 19:45:10 +02:00
Andrey Vasnetsov
fd4e25d6ad refactor(segment): split immutable_id_tracker into a module (#8888)
Mirrors the layout of mutable_id_tracker: storage helpers for the
mappings, versions, and deleted bitslice files live in their own
submodules, leaving mod.rs focused on the ImmutableIdTracker type and
its trait impls.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 15:49:05 +02:00
Andrey Vasnetsov
deb21b4683 feat(segment): add ReadOnlyPayloadStorage backed by GridstoreReader (#8886)
Introduce a read-only payload storage that wraps `GridstoreReader<Payload>`
and implements `PayloadStorageRead`. Also rename the parameter on
`PayloadStorageRead::get`/`get_sequential` from `point_id` to `point_offset`
for consistency with the underlying storage API.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 15:00:01 +02:00
Andrey Vasnetsov
81ea504a40 refactor(segment): migrate info/size_info/telemetry to SegmentReadView (step 11) (#8882)
* refactor(index): extend VectorIndexRead and PayloadIndexRead for telemetry/info

Two trait extensions (no defaults — every implementor must opt in):

* \`VectorIndexRead::is_index\` — distinguishes a real index from a plain
  full-scan one. Used by reporting code. Moved out of inherent
  \`VectorIndexEnum::is_index\` into the trait. Explicit impls on Plain
  (false), HNSW (true), Sparse* (true).
* \`PayloadIndexRead::get_telemetry_data\` — per-field-index telemetry.
  Moved out of inherent \`StructPayloadIndex::get_telemetry_data\` into
  the trait impl. \`PlainPayloadIndex\` returns an empty Vec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(segment): migrate info/size_info/telemetry to SegmentReadView

Step 11 of the SegmentReadView migration — the final logical step.

New \`read_view/info.rs\` exposes builder methods:
* \`build_size_info(uuid, segment_type, is_appendable)\`
* \`build_info(uuid, segment_type, is_appendable)\` — same plus
  \`index_schema\`
* \`build_telemetry(uuid, segment_type, is_appendable, config, detail)\`

The trivial segment-level fields (\`uuid\`, \`segment_type\`,
\`is_appendable\`, \`config\`) are passed in by the caller — they stay
direct on each segment-type rather than going through the view.
Everything else (vector data breakdown, payloads size, deferred
counts, vector-index telemetry, payload-field telemetry, …) is
computed once inside the view through the read traits.

\`Segment::size_info\`, \`info\`, and \`get_telemetry_data\` collapse to
one-line \`with_view\` delegators that pass in the trivial fields.

Cleanup: \`Segment::deferred_deleted_count\` is now unused (the view
has its own equivalent helper); deleted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
84744cb4a6 chore(index): drop unused imports in formula_scorer
Leftovers from moving \`StructPayloadIndex::formula_scorer\` out of this
file. Caught by clippy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
7e5c3a63b5 refactor(index): drop FormulaScorerRead, return concrete FormulaScorer
\`FormulaScorer<'a>\` is already self-contained — it holds the parsed
formula, prefetch scores, retrievers and condition checkers, all owned
or borrowed independently of any payload index. Wrapping it in a trait
adds nothing (a future ReadOnlySegment can construct one too).

* Remove the \`FormulaScorerRead\` trait. \`score(point_id)\` goes back
  to being an inherent method on \`FormulaScorer\`.
* \`PayloadIndexRead::formula_scorer\` returns
  \`OperationResult<FormulaScorer<'q>>\` directly.
* \`StructPayloadIndex\` and \`PlainPayloadIndex\` updated to match.
* \`PlainPayloadIndex\` no longer needs the turbofish placeholder —
  \`Err(...)\` is enough.
* View's \`formula_rescore.rs\` drops the \`FormulaScorerRead\` import.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
1e9e2c6c17 refactor(segment): migrate formula rescore to SegmentReadView
Step 10 of the SegmentReadView migration.

Move \`segment/formula_rescore.rs\` → \`read_view/formula_rescore.rs\`:
* \`do_rescore_with_formula\` (private helper).
* \`rescore_with_formula\` (\`ReadSegmentEntry\` orchestrator).

Both now use the trait-method \`PayloadIndexRead::formula_scorer\`
(added in the prior commit) and \`IdTrackerRead::internal_id\` instead
of inherent calls.

\`Segment::rescore_with_formula\` collapses to a single
\`with_view(|v| v.rescore_with_formula(...))\` delegator. The legacy
\`segment/formula_rescore.rs\` is deleted; \`mod formula_rescore;\`
removed from \`segment/mod.rs\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
a5b030f1ba refactor(index): abstract formula_scorer behind FormulaScorerRead trait
A read-only payload index implementation will have its own concrete
formula scorer type, so PayloadIndexRead can't return the appendable
\`FormulaScorer<'a>\` directly.

* New \`FormulaScorerRead\` trait next to \`FormulaScorer\` exposing only
  what the rescore code path consumes (\`score(point_id)\`). Implemented
  for \`FormulaScorer<'_>\` by moving its inherent \`score\` into the
  trait impl.
* \`PayloadIndexRead::formula_scorer\` returns
  \`OperationResult<impl FormulaScorerRead + 'q>\` (RPITIT).
* The inherent \`StructPayloadIndex::formula_scorer\` (which lived in
  \`formula_scorer.rs\`) is moved into the trait impl block in
  \`struct_payload_index.rs\`, with the body delegating to a new
  \`FormulaScorer::new\` constructor (fields stay private).
* \`PlainPayloadIndex\` always returns
  \`Err::<FormulaScorer<'q>, _>(...)\` — formula scoring is not
  supported there. The turbofish supplies the placeholder type tag.
* Re-export \`FormulaScorer\` and \`FormulaScorerRead\` from
  \`rescore_formula::mod\`. \`retrievers_map\` bumped from \`pub(super)\`
  to \`pub(crate)\` so the trait impl can call it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
df630e80fe refactor(index): drop default impl of fill_idf_statistics, force per-impl
A new index implementation should never silently skip IDF wiring. Make
\`VectorIndexRead::fill_idf_statistics\` a required method (no default)
and provide explicit no-op impls for the dense indexes:
* \`PlainVectorIndex\` — no-op.
* \`HNSWIndex\` — no-op.

\`SparseVectorIndex\` keeps the real implementation: moved from a
sibling \`pub fn\` (inherent) into the \`VectorIndexRead\` trait impl
block, so there is now exactly one definition.

\`VectorIndexEnum\` already overrides per-variant from the previous
commit; nothing changes there.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
09a02c0407 refactor(segment): migrate search, retrieve, and fill_query_context to SegmentReadView
Step 9 of the SegmentReadView migration.

New \`read_view/search.rs\` hosts:
* \`retrieve\` — full body (deferred filtering, vector enrichment, payload
  attachment).
* \`process_search_result\` — converts internal \`ScoredPointOffset\`s into
  user-facing \`ScoredPoint\`s; calls the new \`view.retrieve\`.
* \`search_batch\` — \`ReadSegmentEntry\` orchestrator.
* \`fill_query_context\` — uses the trait \`VectorIndexRead::fill_idf_statistics\`
  (added in the prior commit) and the existing \`indexed_vector_count\`.

The \`Segment\`-side trait method bodies for \`search_batch\`, \`retrieve\`,
and \`fill_query_context\` collapse to single \`with_view\` delegators.
\`rescore_with_formula\` (whose body still uses Segment-side
\`do_rescore_with_formula\`, migrated in step 11) routes its
\`process_search_result\` call through \`with_view\`.

\`segment/search.rs\` is trimmed to just the
\`#[cfg(feature = "testing")] pub fn search\` helper, which calls
\`search_batch\` (now a \`with_view\` delegator) — no migration needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
366c29b476 refactor(index): move VectorIndexEnum::fill_idf_statistics to trait, drop indexed_vectors
Adds \`fill_idf_statistics\` to the \`VectorIndexRead\` trait with a
no-op default (only sparse-vector indexes track IDF). \`VectorIndexEnum\`
overrides it with the per-variant dispatch that previously lived as an
inherent method.

The inherent \`indexed_vectors\` is removed: it returned the same value
as the trait method \`indexed_vector_count\` for every variant
(sparse's \`indexed_vector_count\` is itself
\`inverted_index.vector_count()\`). The remaining caller in
\`fill_query_context\` will use \`indexed_vector_count\` after the next
commit moves it to the view.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
da145e7403 refactor(index): drop FacetIndexRead, reuse FacetIndex on FacetIndexEnum
The introduced \`FacetIndexRead\` trait was almost identical to the
existing \`FacetIndex\` trait — the only difference was the augmented
\`for_each_value\` signature on \`FacetIndexEnum\` that took
\`hw_counter\` and \`deferred_internal_id\` to skip values whose only
points are deferred.

* Move that "skip deferred" logic onto \`FacetIndex\` itself as a default
  method \`for_each_visible_value\` — implemented in terms of the
  existing \`for_each_value\` and \`for_each_value_map\`. All
  \`FacetIndex\` impls (MapIndex, BoolIndex, FacetIndexEnum) get it for
  free.
* Make \`FacetIndexEnum\` impl \`FacetIndex\` directly (delegating to the
  inner index's \`FacetIndex\` impl). The augmented inherent
  \`for_each_value\` is gone.
* \`PayloadIndexRead::facet_index_for\` now returns
  \`Option<impl FacetIndex + '_>\` (no separate \`FacetIndexRead\` trait).
* View facet code uses \`for_each_visible_value\` instead of the old
  augmented \`for_each_value\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
320f9f8685 refactor(segment): migrate facets to SegmentReadView
Step 8 of the SegmentReadView migration.

Move \`segment/facet.rs\` → \`read_view/facet.rs\`:
* \`approximate_facet\`
* \`facet_values\`

Both now go through the trait-method \`facet_index_for\` (added in the
prior commit) and the existing trait-method \`filter_context\` instead
of the previous inherent \`get_facet_index\` and
\`struct_filtered_context\` calls.

\`Segment::unique_values\` and \`Segment::facet\` (the \`ReadSegmentEntry\`
trait orchestrators) collapse to single \`with_view\` delegators. The
legacy \`segment/facet.rs\` is deleted; \`mod facet;\` removed from
\`segment/mod.rs\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
6f31812d03 refactor(index): abstract facet index access behind FacetIndexRead trait
Same shape as the numeric_index_for change in the previous PR: a
read-only segment will have its own per-key facet-index type, so
PayloadIndexRead can't return the concrete \`FacetIndexEnum<'a>\`.

* New \`FacetIndexRead\` trait in \`field_index/facet_index.rs\` mirrors
  the inherent methods of \`FacetIndexEnum\` (\`for_points_values\`,
  \`for_each_value\`, \`for_each_value_map\`, \`for_each_count_per_value\`).
* Implemented for \`FacetIndexEnum<'a>\` by delegating to the inherent
  methods.
* \`PayloadIndexRead::facet_index_for(key) -> Option<impl FacetIndexRead + '_>\`
  added (RPITIT, no boxing). Each impl picks its own concrete return
  type:
  - \`StructPayloadIndex\` returns \`Option<FacetIndexEnum<'_>>\`.
  - \`PlainPayloadIndex\` always returns \`None\`; the explicit
    \`None::<FacetIndexEnum<'_>>\` turbofish supplies a placeholder
    type tag.
* \`FacetIndexRead\` re-exported from \`field_index/mod.rs\` (the
  \`facet_index\` submodule itself stays \`pub(super)\`).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
82a8210433 refactor(segment): migrate random sampling to SegmentReadView
Step 7 of the SegmentReadView migration.

Move \`segment/sampling.rs\` → \`read_view/sampling.rs\`:
* \`read_by_random_id\`
* \`filtered_read_by_index_shuffled\`
* \`filtered_read_by_random_stream\`
* \`read_random_filtered\` orchestrator

\`Segment::read_random_filtered\` collapses to a single
\`with_view(|v| v.read_random_filtered(...))\` delegator. The legacy
\`segment/sampling.rs\` is deleted; \`mod sampling;\` removed from
\`segment/mod.rs\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
3eef48279b refactor(common): use EitherVariant for NumericIndexInner::stream_range
Drop the bespoke 4-arm \`NumericRangeIter\` enum introduced in the prior
commit. The existing \`common::either_variant::EitherVariant\` is the
same shape and already has \`Iterator\` plus all the standard adapter
specializations.

Adds a \`DoubleEndedIterator\` impl to \`EitherVariant\` (\`next_back\`,
\`nth_back\`, \`rfold\`, \`rfind\`) so it can be used in this position.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
6e2ad02db6 refactor(index): replace Box<dyn> with RPITIT in StreamRange and NumericFieldIndexRead
Both \`StreamRange<T>::stream_range\` and the new
\`NumericFieldIndexRead\` trait now return \`impl Iterator\` /
\`impl DoubleEndedIterator\` directly. No more boxed trait-object
allocations in the order-by hot path.

Mechanism:
* \`NumericIndexInner::stream_range\` has 4-way branching
  (empty / mutable / immutable / mmap), all with different concrete
  iterator types. Unified via a new 4-arm \`NumericRangeIter\` enum that
  delegates \`Iterator\` and \`DoubleEndedIterator\` to its variants.
* \`NumericFieldIndex::stream_range\` and \`get_ordering_values\` have
  2-way branching (Int / Float). Unified via \`itertools::Either\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
e5df30a434 refactor(index): abstract numeric_index_for return behind NumericFieldIndexRead trait
A read-only payload index implementation will have its own concrete
numeric-index type that doesn't fit the appendable
\`NumericFieldIndex<'a>\` enum. So PayloadIndexRead can't return that
concrete type directly.

Introduce \`NumericFieldIndexRead\` trait exposing only what ordered
reads consume (\`get_ordering_values\`, \`stream_range\`). Implement it
for the existing \`NumericFieldIndex<'a>\` (delegates to the inherent
methods + \`StreamRange\`).

\`PayloadIndexRead::numeric_index_for\` now returns
\`Option<impl NumericFieldIndexRead + '_>\` (RPITIT). Each
implementation picks its own concrete return type:
- \`StructPayloadIndex\` returns \`Option<NumericFieldIndex<'_>>\`.
- \`PlainPayloadIndex\` always returns \`None\` (it has no field
  indexes); the explicit \`None::<NumericFieldIndex<'_>>\` turbofish
  just supplies a placeholder type tag — no value is constructed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
35c1a7c61c refactor(segment): finish scroll migration — move filtered_read_by_index to view
Step 5 leftover: with PayloadIndexRead.iter_filtered_points now on the
trait, filtered_read_by_index can move to the view alongside the other
three scroll helpers.

* `read_view/scroll.rs` gains `filtered_read_by_index` and the
  `read_filtered` orchestrator.
* `segment/scroll.rs` is deleted entirely; `mod scroll;` removed from
  `segment/mod.rs`.
* `Segment::read_filtered` collapses to a single
  `with_view(|v| v.read_filtered(...))` delegator.
* `scroll_filtering_test.rs` integration test calls
  `filtered_read_by_index` via `with_view`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00
generall
dab0005e5c refactor(segment): migrate ordered reads to SegmentReadView
Step 6 of the SegmentReadView migration.

Move `lib/segment/src/segment/order_by.rs` into
`read_view/order_by.rs`:
* `filtered_read_by_index_ordered` — uses
  `payload_index.iter_filtered_points` (now on trait, see prior commit)
  and `numeric_index_for` (also now on trait).
* `filtered_read_by_value_stream` — uses `numeric_index_for` and
  `filter_context` (already on trait).
* `read_ordered_filtered` — `ReadSegmentEntry` orchestrator, also moved
  to the view.

`Segment::read_ordered_filtered` collapses to a single
`with_view(|v| v.read_ordered_filtered(...))` delegator. The legacy
`segment/order_by.rs` file is deleted entirely; `mod order_by;` removed
from `segment/mod.rs`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:31:35 +02:00