mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-03 00:20:57 -05:00
dev
235 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
xzfc
|
75385df69f |
Remove dead code (#10030)
* Remove dead code * Remove unused dependencies * `allow(dead_code)` -> `expect(dead_code)` * ast-grep: rule-tests/*-test.yml => tests/*-test.yml For brevity. * ast-grep: forbid allow(dead_code) |
||
|
xzfc
|
39547e3a67 | chore: bench_cache (#10028) | ||
|
Andrey Vasnetsov
|
0a16a62f99 |
feat: io_uring setting to control which components use the io_uring backend (#10008)
* feat: `io_uring` setting to control which components use the io_uring backend A few components have both an mmap and an io_uring variant reading the very same files: the immutable dense vector storages, the single-file TurboQuant storage, and the mmap payload storage. Until now the choice was a side effect of `async_scorer` — a vector-search knob — plus, for the payload storage, a feature flag that was parked off because io_uring is ~2x slower than mmap when the data fits the page cache (#9310, #9409). Add `storage.performance.io_uring`, optional, with two modes: - unset (default): unchanged behaviour. The vector storages keep following `async_scorer`; the payload storage stays on mmap. - `disabled`: no component uses io_uring. - `auto`: a component uses io_uring when its memory placement is `cold` (data is left on disk, so reads hit the disk and there is something to gain), its feature flag allows it, and the kernel supports io_uring. Components meant to sit in RAM keep using mmap. The decision lives in one place, `segment::common::io_uring::use_io_uring`, so the openers no longer each reach for the async-scorer global. Kernel support is now probed up front through `is_io_uring_supported()` instead of opening a file and falling back on error. `async_payload_storage` now defaults to on: it no longer decides anything by itself, it only lifts the ban, and the payload storage no longer follows `async_scorer` at all — so turning it on cannot silently move an existing `async_scorer: true` deployment onto the slower path. Which backend a component ended up on depends on the config, the placement and the kernel at once, so report it in `SegmentInfo`: `vector_data[name].io_backend` and `payload_storage_io_backend`, both `"mmap" | "io_uring"`, absent for components that have no such choice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Trim comments, drop trivial tests Two tests were only restating their own implementation: `test_mode_round_trip` round-tripped the encode/decode pair next to it, and `test_io_uring_config` checked that serde deserializes a two-variant enum. The mode matrix test stays, it is the one that pins the semantics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Flatten `IoBackend` in OpenAPI, derive `JsonSchema` for `IoUringMode` Per-variant doc comments on a plain string enum make schemars emit a `oneOf` of anonymous single-value objects instead of a flat `enum`. Move the variant descriptions into the enum doc, as `Memory` and friends already do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update lib/segment/src/vector_storage/turbo/turbo_vector_storage.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * upd openapi schema * Update lib/common/common/src/flags.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Require kernel io_uring support in the async-scorer fallback `use_io_uring` returned `get_async_scorer()` verbatim when the `io_uring` setting is unset, so an enabled async scorer on a kernel without io_uring opened the io_uring storage, failed, and fell back to mmap with an error log per segment. Gate that branch on `is_io_uring_supported()` too, like `Auto` already is, so the component just stays on mmap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * upd openapi schema --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> |
||
|
Tim Visée
|
be543561e5 |
Add Logstore and Blobstore wrapper (#9673)
* Gridstore: introduce storage operating mode in config Add a mode field to the gridstore config, selecting between the dynamic mode (current behavior, the default) and the upcoming serverless mode. The mode is specified through StorageOptions on creation, persisted in config.json, and read back first when opening so the correct variant can be selected automatically. Configs written before this field existed deserialize as dynamic. For now, selecting the serverless mode returns an error; the variant itself is added in follow-up commits. * Gridstore: move dynamic implementation into dedicated module Mechanical move of the current Gridstore implementation into gridstore/dynamic.rs as DynamicGridstore. The public Gridstore struct becomes a thin wrapper holding a mode variant enum, propagating every call into the selected variant. For now the enum only has the dynamic variant; the serverless variant is added in follow-up commits. No logic changes to the dynamic implementation itself: only visibility, the config parameter now passed into open (the wrapper reads it first to select the mode), and open_or_create staying on the wrapper. * Gridstore: add serverless tracker Add the append-only mapping tracker for the serverless storage mode. The tracker file is a plain array of 16-byte mapping entries without any header: the number of mappings is defined by the exact file length, and the entry index is the point offset. The file starts empty and only ever grows by appending, existing bytes are never rewritten. Mappings must be set in monotonically increasing point offset order; skipped offsets are backfilled as zeroed entries which decode as None. New mappings are buffered in memory and appended with a single write per flush. A flush with a stale target is a no-op so bytes are never written twice. A torn trailing entry (file length not a multiple of the entry size) is ignored when reading and truncated away when opening writable. Unlike the dynamic tracker, the file is read and written directly with positional file IO instead of memory mapping, as serverless environments do not handle memory mapped files well. * Gridstore: add serverless storage variant Add the append-only gridstore variant for serverless deployments, which restrict IO to appending to files: existing bytes can never be rewritten, and IO is expensive so as few files as possible are used. The variant stores all value data in a single page file next to the serverless tracker and the storage config, three files in total. Both data files start empty and only ever grow by appending; there is no preallocation, no used-block bitmask and no gap/region bookkeeping. Values are appended at put time at the next block aligned offset, with the zero padding included in the write so it lands exactly at the end of the file. Mappings are buffered and appended to the tracker with a single write per flush, after the page file is synced, so a mapping on disk never points at data that is not durable. Values cannot be updated or deleted, and must be put at monotonically increasing point offsets; violations are rejected before any data is written. Files are read and written directly, never memory mapped. The mode is selected through StorageOptions on creation and picked up automatically from the persisted config when opening. * Gridstore: serverless support in reader and view Extend the read-only GridstoreReader and the GridstoreView with the serverless mode, keeping both public types unchanged: like the writable Gridstore they now hold a mode variant internally, selected automatically from the persisted config when opening. The serverless reader holds the tracker and page directly and reads the files positionally, without memory mapping. A live reload re-reads the mapping count from the exact tracker file length (there is no size header), ignoring a torn trailing entry, and never truncates as it is read-only. Value reads always go directly to the file, so newly appended data is readable without remapping anything. * Gridstore: document storage operating modes * Gridstore: review fixes for the serverless mode Hardening and cleanup from a review pass over the new serverless storage variant: - Batch the reader side iteration like the writer already did, instead of materializing tracker mappings for the full range in one go, which could transiently allocate gigabytes on large storages. - Recover the append cursors when a positional write fails partway: truncate the file back to the tracked length so a retried append or flush never rewrites bytes that already landed in the file. - Validate page addressability before appending value data, a rejected put must not grow the page file. - Cross-check tracker and page consistency when opening: mappings that reference value data past the end of the page file (e.g. after a partial copy or restore) now fail fast instead of surfacing as opaque read errors per point. - Reject value pointers into any page other than page 0 on the serverless read path with PageNotFound, matching the dynamic mode contract, instead of silently reading from a wrong location. - Refresh the reported storage size on reader live reload even when no new mappings were flushed, unflushed value data may have grown the page file already. - Validate configs read from disk: a corrupt config with zero sized blocks, pages or regions is now rejected when opening instead of panicking on a division by zero later. - Classify rejected serverless puts as UnsupportedOperation, consistent with rejected deletes, so they don't surface as user-facing validation errors at the segment level. - Deduplicate the compression dispatch into Compression::compress and Compression::decompress, and the serverless file create/open patterns into shared direct IO helpers, so the two modes and files can't silently drift apart. * Gridstore: cover both operating modes in mode-agnostic tests Parameterize the gridstore tests that exercise mode-agnostic behavior over both the dynamic and serverless mode with rstest, using a single and bulk put/get roundtrips, storage files, basic persistence, corrupt config rejection, batched read congruence, reader live reload, and the different block sizes. Mode specific expectations branch inside the tests: expected file names, storage size semantics (whole blocks vs exactly packed bytes), value pointer layout (page spill over vs a single packed page), and gaps (created by deletes in dynamic mode, by skipped puts in serverless mode). Dynamic-only internals assertions are kept behind a mode check. Tests around updates, deletes, page spanning, block reuse and other dynamic-only behavior intentionally stay dynamic; the serverless specific format invariants remain covered by the dedicated serverless tests. * Gridstore: port serverless specific tests from sibling branch Source the serverless specific test cases that the serverless-gridstore-updates branch added, adapted to the dedicated variant implemented here (distinct file names, headerless tracker with 16 byte entries, a single packed page without trailing padding, and rejected re-puts): - writes only ever append: tracker and page files only grow and previously written bytes stay byte-for-byte untouched - new mappings land exactly at the end of the tracker file, which always covers the exact number of mappings - mapping gaps are zero-padded on disk and survive reopening - values are packed back to back at block aligned offsets, the page file ends exactly at the last value - serverless mode never creates nor reports block flag files - a flusher persists exactly the mappings that existed at its creation, later puts stay pending - a config claiming the wrong mode fails loudly in both directions instead of loading the incompatible file format of the other mode Tests around their mode switching, page spanning and tolerated deletes don't apply to this design and are intentionally not ported. * Gridstore: test serverless production risk scenarios Add tests for the operational aspects that matter before serverless mode goes to production, each covering a scenario that wasn't evaluated yet: - Replayed puts of already persisted offsets (a WAL redo after a crash where the flush completed but was never acknowledged) are rejected without appending anything, and max_point_offset is the exact offset a replay must resume at. - The accepted crash case of a tracker file extended with zeroed bytes: the entries count as permanent None mappings, can never be put again, and the storage stays consistent and writable past them. - The read-only reader never modifies the files: opening over a torn tracker tail, reading, iterating and live reloading leave both files byte-for-byte untouched. - A multi-round put/flush/reopen cycle always exposes exactly the flushed prefix, with the mapping count matching the exact tracker file length and unflushed offsets reusable. - An append beyond the maximum addressable block offset is rejected before writing anything, keeping retried puts from growing the page file unboundedly. * Gridstore: rename serverless mode to append-only, split into module Rename the mode after its defining characteristic instead of its deployment target: files only ever grow, existing bytes are never rewritten. Renames Mode::Serverless to Mode::AppendOnly (persisted as "mode": "append_only") and the on-disk file names to append_only_tracker.dat and append_only_page_0.dat. The serverless deployment motivation stays in the documentation. Also split the single 2300 line serverless.rs into an append_only module with dedicated files for the storage, page, view, reader and tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Use universal IO in Gridstore * Include upstream preopen logic in new Gridstore variant * Gridstore: buffer append-only value writes until flush In append-only mode, put previously wrote the value data to the page file right away, one write operation per put, while mappings were already buffered and batch persisted on flush. Buffer value writes the same way: both the value and its mapping now only land on disk once a flush cycle executes. This batches all new value data into a single write operation per flush, which is significantly more efficient on S3 based storage where every write is a costly operation. A flush now performs exactly two writes: one appending all buffered value data to the page file, one appending all pending mappings to the tracker file, in that order, so a mapping on disk never points at value data that is not durable. The page mirrors the tracker's pending mechanism: an in-memory buffer that is byte for byte the next append (zero padding between block aligned values included), a watermark captured at flusher creation so puts made during a flush stay buffered, a stale-flush no-op guard so appended bytes are never written twice, and truncate-back recovery on failed writes. Reads transparently serve buffered values from memory. As a side effect, a crash between flushes now leaves nothing on disk at all, where the write-through approach left orphaned value bytes in the page file. The buffered data is held in memory until the next flush, bounded by the flush cadence. Universal IO filesystem handles are now required to be Send + Sync, so the flusher closure can carry one to grow the page file at flush time; all existing backends already satisfied this. * Gridstore: rename inner DynamicGridstore to Gridstore The dynamic variant keeps the Gridstore name; the outer dispatching type will be renamed to Blobstore in a follow-up. Until then the inner type is referred to as dynamic::Gridstore to distinguish it from the outer type. * Gridstore: rename append-only variant to Arenastore The append-only variant stores all value data in a single ever-growing page, allocating space by appending, hence: arena store. * Gridstore: rename outer storage type to Blobstore The outer type dispatching between the two storage variants is now called Blobstore, being more generic than Gridstore. This frees up the Gridstore name, which now exclusively refers to the dynamic mode variant, next to Arenastore for the append-only variant. Storage components keep using the outer type, so they now use Blobstore. The gridstore crate name, GridstoreError, and the persisted names (config.json mode, payload config storage_type) are unchanged. * Gridstore: split Gridstore and Arenastore into dedicated modules The outer module is now blobstore, matching the Blobstore type it defines. The two storage variants each get their own submodule: the dynamic Gridstore moves from dynamic.rs into gridstore/ with its reader and view extracted from the shared files, mirroring the arenastore/ module (previously append_only/) which already had this layout. * Rename gridstore crate to blobstore The crate is named after the outer Blobstore storage type it provides. The gridstore name lives on in the dynamic mode variant. GridstoreError and the persisted names (config.json mode, payload config storage_type) are unchanged. * Arenastore: pack values back to back across multiple pages Drop the block alignment from the append-only mode: values are packed byte to byte, without blocks, and the tracker offset is now a plain byte offset within the page. Blocks and regions are dynamic mode concepts; their page size constraints no longer apply to append-only configs. Bring back support for multiple pages. Once appending a value would grow the current page beyond the configured page size, a new page is started, bounding the size of and the number of appends to each file: object stores like S3 Express limit the number of appends per object. A value larger than the page size gets a page of its own; values never span pages. A rollover creates the new, empty page file at put time; the value data itself stays buffered until the next flush, which appends to each touched page with a single write, using per-page watermarks captured at flusher creation. The reader scans for consecutively numbered page files when opening, validates the most recent mappings against them, and adopts pages created since on a live reload. * Blobstore: rename dynamic mode to mutable Rename Mode::Dynamic to Mode::Mutable, and the persisted config value with it: config.json now writes "mode": "mutable". There is no compatibility alias for "dynamic", released versions never wrote the mode field (a missing field still defaults to mutable), only unreleased storages did. The Gridstore type and module names for the mutable variant are unchanged. * Fix Edge compilation due to package rename * Review remarks * Extract Gridstore preopen into module * Rename Arenastore files * Use universal IO for append operations * Rename GridstoreError to BlobstoreError The error type belongs to the Blobstore crate and is shared by both the Gridstore and Arenastore variants, so it follows the crate naming. Also update the user-facing error messages that referred to the old name. * Split config into per-variant types * Rename Arenastore to Logstore Rename the Arenastore type to Logstore, including the reader, view, config, module and variant names. The storage file names follow: log_page_{n}.dat and log_tracker.dat. The persisted mode tag stays "append_only". * Move bitmask module into the Gridstore variant The bitmask tracks free blocks, which only exists in the mutable mode. Move the module from the crate root into the Gridstore variant that owns it. It stays re-exported at the crate root because the bitmask benchmark needs a public path. * Move pages module into the Gridstore variant Like the bitmask, the block based pages module is only used by the mutable mode. Move it from the crate root into the Gridstore variant that owns it. The Logstore variant has its own page implementation. * Use universal IO for every Logstore operation Replace the direct_io module with universal IO in the append-only tracker, making the whole Logstore go through a universal IO backend bounded by UniversalRead and UniversalAppend: - The tracker is generic over the backend now. Reads go through UniversalRead with the caller's access pattern, flushes land as one atomic append with the same offset compare-and-swap recovery as the pages: a retried append after a lost acknowledgement is adopted instead of appended twice. A torn trailing entry is still truncated away on writable open, through a fresh handle since shrinking is not supported through an open one. - The reader now schedules a prefetch for the tracker file too, it no longer bypasses the backend. - The config write, clear and wipe use the backend file operations instead of local filesystem calls, matching the Gridstore variant. * Batch reads in Logstore read_values Apply the same batching logic as the Gridstore variant: resolve all mappings first, then fetch the value data, both through the backend's read pipeline so async backends can serve the reads in parallel. The tracker gains a batched lookup mirroring the mutable tracker's iter, serving pending mappings and out of range point offsets directly from memory. The pages gain a batched value read; unflushed values are served from the in-memory buffers, and since values never span pages each value is a single read without reassembly. Like in the Gridstore variant, the callback may now be invoked in a different order than the requested point offsets. * Better describe logstore live reload ordering * use enum for options, swap `*Options`<->`*Config` naming * don't wrap enum in struct * ditch unused `StorageConfig`, make deserialization more ergonomic * rename `*Options`->`*Config` * make `preopen` non-blocking * fixup! ditch unused `StorageConfig`, make deserialization more ergonomic * fixup! use enum for options, swap `*Options`<->`*Config` naming * fixup! don't wrap enum in struct * fix rebase * use `populate` param in Logstore * test: failing repro of stale page after live reload across rollover A reader that live-reloads between a page rollover and the following flush adopts the new, still empty page. The previous page is then no longer the last one and is never reloaded again, so the tail that the next flush appends to it stays invisible to the reader forever: value pointer at byte 100 with length 100 is out of range AppendOnlyPages::live_reload only reloads the last held page, assuming earlier pages never change once a newer page exists. But the rollover creates the new page file eagerly at put time, while the previous page's buffered tail only lands at the next flush (see test_rollover_writes_no_value_data_before_flush), so a page can keep growing on disk after its successor exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: reload all pages that grew * use Fs in `open_or_create` * fix: publish tracker mappings only after the pages reload `AppendOnlyTracker::live_reload` observed the mapping count and made it visible in one step, before `LogstoreReader::live_reload` reloaded the pages. Every failure path in the page reload -- `list_files`, reopening a grown page, opening an adopted one, the truncation check -- therefore left the reader with mappings referencing value data it never loaded, so reads in the new offset range fail until a later reload happens to succeed. The edge refresh loop keeps a segment whose reload failed, expecting it to keep serving its pre-refresh state, which it then does not. Split observing from publishing: `reload_count` refreshes the handle and returns the count as a `PendingReload` token, `commit_reload` publishes it. The reader still observes the tracker first, as the writer persists pages before the mappings referencing them, but only commits once the pages are loaded. Reopening without committing is harmless: reads stay bounded by the unchanged count, and the bytes below it never change. A partial failure inside the page reload needs no unwinding, pages running ahead of the tracker is the safe direction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: batch the value reads in Logstore iteration `LogstoreView::iter_range`, the path behind `Logstore::iter` and `LogstoreReader::iter`, fetched the mappings for the whole range with a single read but then read the values themselves one at a time, serially. Gridstore routes its `iter` through `read_values` and pipelines both stages, so a full scan of an append-only storage was the one read path without batching -- one blocking round trip per value on the object store backends this variant exists for. It is reached by payload storage iteration and by the payload index build, which scans every payload. Feed the pointers into `read_batch_values` instead, keeping the single contiguous tracker read, which is better than the per-offset pipeline scheduling Gridstore does on that side. Values are now delivered through the read pipeline, so the callback may be invoked out of order, as it already could be for Gridstore's `iter` and for `read_values` in both variants. Both segment callers are order independent. Tests that happened to rely on the mmap backend completing reads in scheduling order now sort before comparing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: don't run the failed-page-reload test on Windows The test shrinks a page file out of band to make the page reload fail, but Windows refuses to resize a file while the reader holds it mapped, which it does by construction here: "the requested operation cannot be performed on a file with a user-mapped section open". The panic is on the injection itself, the code under test never runs. There is no portable injection. Truncating a page the reader holds is what the check under test detects, so the mapping cannot be avoided; failing the adopted page open instead needs a listed but unopenable file, and `local_list_files` descends into matching directories rather than listing them; failing the directory listing needs the storage directory removed, which Windows also refuses while pages are mapped. The storage itself is fine on Windows, its append path grows mapped pages there and every other Logstore test passes. The logic under test is platform independent and stays covered elsewhere, with the tracker half of the guarantee pinned by `test_live_reload`, which runs on every target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |
||
|
Ivan Pleshkov
|
7469834d9b |
[TQDT] TQ dense/multi vector storage consistency (#9953)
* [TQDT] Align TQ vector storage layout with the reference dense storage
Make the TurboQuant vector storage structurally mirror the reference dense
(and multi_dense) storages, down to file names and their contents:
- Rename files + structs to the dense convention:
- immutable.rs -> turbo_vector_storage.rs (ImmutableTurboVectorStorage ->
TurboVectorStorageImpl)
- appendable.rs -> appendable_turbo_vector_storage.rs
(AppendableTurboVectorStorage -> AppendableMmapTurboVectorStorage)
- multi.rs -> multi_turbo/appendable_mmap_multi_turbo_vector_storage.rs
(TurboMultiVectorStorage -> AppendableMmapMultiTurboVectorStorage)
- ReadOnlyTurboMultiVectorStorage -> ReadOnlyChunkedMultiTurboVectorStorage
- Thin out turbo/mod.rs to module declarations + re-exports: open_* fns move
into their storage files, consts + turbo_storage_roundtrip into shared.rs,
and TurboScoring / TurboMultiScoring join the other TQ traits in
vector_storage_base.rs.
- Split read_only/ into the chunked storage (read_only/) and the single-file
storage (read_only/immutable/), each with the mod/lifecycle/live_reload/
read_ops 4-file layout, mirroring dense/read_only/.
- Introduce multi_turbo/ mirroring multi_dense/, with its own read_only/
submodule holding ReadOnlyChunkedMultiTurboVectorStorage.
- Relocate the storage test suites to
tests/test_appendable_turbo_vector_storage.rs and
tests/test_appendable_multi_turbo_vector_storage.rs, paralleling the
dense/multi_dense integration test files (tests moved verbatim, no new
tests added).
- Fix a gpu-gated VectorStorageEnum match that referenced stale DenseTurbo /
DenseTurboAppendable variant names.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* are you happy fmt
* fix after rebase
* [TQDT] Address review feedback on TQ vector storage split
- gpu tests: use the real `VectorStorageEnum::DenseTurboAppendableMemmap`
variant (the old `DenseTurboAppendable` name never existed post-rename, so
the gpu-feature test failed to compile — missed because `cargo build
--features gpu` does not compile the `#[cfg(test)]` code).
- memory_reporter: report `DenseTurboUring` files as `FileStorageIntent::OnDisk`
like the other io_uring variants; io_uring never mmap-caches, so delegating
to `is_on_disk()` could wrongly report `Cached` for a populated backend.
- turbo_vector_storage: fix the misleading `insert_tq_bytes` doc comment — the
single-file backend rejects the upsert via `?`, so `set_deleted` is never
reached.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [TQDT] Fix clippy::wildcard_enum_match_arm in read-only routing test
Spell out the non-routing `VectorStorageType` variants instead of `_`, so a
future added variant fails the match rather than silently mapping to `false`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [TQDT] Fix stale Turbo4 storage-variant assertion in quantization test
The segment is built with the default (appendable/chunked) storage type, so a
Turbo4 datatype now lands in `DenseTurboAppendableMemmap`, not the single-file
`DenseTurboMemmap`. The assertion was left on the pre-split variant; align it
with the non-turbo branch, which already expects the appendable variants.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* are you happy fmt
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Luis Cossío
|
5c269b9525 |
Misc nits (#9894)
* use `WithVector::is_enabled` * suppress unused var lint * fix non linux "useless mut" lint |
||
|
Arnaud Gourlay
|
c196d2eb1a |
Benches: use SmallRng instead of ChaCha12-based generators (#9887)
* Benches: use SmallRng instead of ChaCha12-based generators All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength randomness, and several draw random values inside the timed closure, so cipher work was included in the measurement itself. Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW graph cache and sparse index cache by RNG algorithm so stale caches built from the old generator are not reused against newly generated vectors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Benches: replace free-function rand::random with local SmallRng Addresses review: rand::random draws from the thread RNG (ChaCha12), including inside the timed loop of the pq score benchmark. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jojii
|
6d6e88aff3 | IO uring for TQDT (#9852) | ||
|
Arnaud Gourlay
|
dd84044a3e |
refactor: replace StructPayloadIndex::open bool flags with StorageType and IndexLoadMode (#9754)
StructPayloadIndex::open took two adjacent bools (is_appendable, create)
and call sites passed every literal combination: (true, true),
(true, false) and (false, true) all exist. A transposed pair compiles
and silently yields e.g. non-appendable + create instead of
appendable + load-only.
The target enum already existed: open immediately converted the bool
into the private StorageType { Appendable, NonAppendable }, so the bool
survived only at the API boundary, exactly where the swap hazard lives.
Make StorageType public, take it directly, and introduce
IndexLoadMode { CreateIfMissing, LoadExisting } for the create flag.
create_segment had the same trailing create: bool with bare literals at
both callers, so its parameter is lifted to IndexLoadMode as well:
load_segment passes LoadExisting, build_segment passes CreateIfMissing.
No behavior change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Andrey Vasnetsov
|
fb681b7d9d |
Lazy roaring flags bitmap and bool index counts (#9749)
* [AI] make ReadOnlyRoaringFlags bitmap and bool index counts lazy
Opening a read-only segment scanned every flags file end to end:
`ReadOnlyRoaringFlags::open` materialized the whole RoaringBitmap via
`iter_ones()`. Every payload field carries a null index, so this was paid
per field per segment, for bitmaps most queries never touch.
Make the bitmap a `OnceLock`, filled by a scan on first access. Open now
reads only the tiny status file. `ReadOnlyBoolIndex`'s three eager count
fields collapse into one lazily-derived, cached `BoolCounts`; its
`live_reload` refreshes them in place when present and leaves them unset
otherwise, so reloading an index nothing queries stays scan-free.
Propagate the resulting `OperationResult` through `RoaringFlagsRead`,
`PayloadFieldIndexRead::count_indexed_points`, `FieldIndexRead`,
`PayloadIndexRead::{indexed_points, get_telemetry_data}`, `build_info` /
`build_telemetry` and `SegmentEntry::{info, get_telemetry_data}`, out
into shard, edge and collection.
`ram_usage_bytes` stays infallible: an unmaterialized bitmap holds no
RAM, so it reports 0 via the new `bitmap_if_materialized`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] correct `preopen` comment: `open` no longer scans the flags file
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] fix edge examples for fallible `info()`
`EdgeShardRead::info` now returns `OperationResult<ShardInfo>`. The
examples live in their own workspace (lib/edge/publish), so the main
`cargo check --workspace` never saw them.
Every call site sits in `fn main() -> Result<(), Box<dyn Error>>`, so
propagate with `?`. `bm25-search` compiled either way but would have
printed the `Result` rather than the `ShardInfo`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Andrey Vasnetsov
|
ab0d3ecc62 |
Add unified memory: cold|cached|pinned placement parameter for collection components (#9684)
* Add unified `memory: cold|cached|pinned` placement parameter for collection components
Introduce a single `memory` parameter that controls how each collection
component's data is held in RAM, replacing the inconsistent zoo of
`on_disk` / `always_ram` / `on_disk_payload` flags:
- `cold`: not pre-loaded from disk, cached with usage
- `cached`: pre-populated into page cache on load, evictable under pressure
- `pinned`: materialized on heap, never evicted by cache pressure
The parameter is available on dense vectors, HNSW config, all quantization
configs, the sparse index, all payload field index types, and payload
storage (as a new `payload: { memory }` sub-object on collection params).
When set, it overrides the deprecated legacy flag; when unset, behavior is
unchanged. Legacy flags are marked deprecated (Rust + proto) but keep
working; conflicts are resolved in favor of `memory` with a warning.
New capabilities enabled by the tri-state model:
- HNSW graph links can be pinned (first production caller of the existing
`GraphLinksResidency::Pinned`)
- sparse mmap index, quantized vectors and on-disk payload field indexes
gain a `cached` tier (mmap + populate on open)
`pinned` is rejected by API validation for components without a heap
variant (dense vector storage, payload storage). Low-memory mode degrades
placements at load time via `Memory::clamp_to_low_memory`, matching the
existing `prefer_disk`/`skip_populate` behavior. Effective-placement
comparison in the config-mismatch optimizer avoids spurious rebuilds when
the same placement is expressed through the new parameter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix gpu-gated tests for the new `memory` field
CI clippy runs with --all-features, which compiles the gpu-gated tests
that were missed locally: add the `memory` field to config literals and
allow deprecated placement params, same as in the rest of the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI tests for memory placement, keep sparse config downgrade-clean
- OpenAPI tests: create/update collections with `memory` on every component,
assert the parameters are echoed in collection info, assert legacy-only
collections expose no new fields, and assert `pinned` is rejected (422)
for dense vector storage and payload storage on both create and update.
- Persist only the explicitly requested `memory` parameter in
`sparse_index_config.json` instead of the legacy-resolved placement, so
configurations using only the deprecated `on_disk` flag keep byte-identical
files that older Qdrant versions load without unknown fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate collection meta ops at construction, not only in the API layer
The `memory: pinned` rejection for dense vectors and payload storage
lived in `Validate` impls on the internal request types, which only ran
through the REST actix extractor. gRPC validates just the proto message,
so a gRPC client could persist `pinned` where it is not supported and
have it silently treated as `cached`.
Run the derived validation in `CreateCollectionOperation::new` and
`UpdateCollectionOperation::new` instead: the constructors are the
common chokepoint for all API paths, before the operation is proposed
to consensus. This covers every validator on these types, not just the
`memory` checks, and keeps consensus-apply unaffected so mixed-version
clusters never reject already-committed operations.
`UpdateCollectionOperation::new` becomes fallible; `remove_replica` now
uses `new_empty` since it carries no user config. Regression tests drive
the gRPC conversion path and assert `InvalidArgument` for `pinned` on
create and update, with `cold`/`cached` accepted as a control.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Andrey Vasnetsov
|
94cdea1fe4 |
Explicit HNSW links residency: cold / cached / pinned (#9669)
* Explicit HNSW links residency: cold / cached / pinned Introduce GraphLinksResidency to make graph links memory residency an explicit choice instead of a side effect of the IO backend: - Cold: mmap without populate, pages fault in on demand (on_disk: true) - Cached: mmap with blocking populate, evictable (on_disk: false) - Pinned: materialized into anonymous heap, page cache evicted after the read; kept internal for now (no production caller selects it), non-borrowable universal-IO backends (io_uring, object stores) fall back to it by necessity Fixes along the way: - Freshly built non-on_disk indexes no longer pin links in heap: the builder now always serializes to disk and re-loads as mmap (Cold/Cached by on_disk), so a just-built index has the same single-copy residency as one loaded after restart, instead of a heap copy plus the freshly written file in page cache. - Materializing fallbacks evict the page cache after copying to heap (same hygiene as read_whole_via), so links are never resident twice. - Memory reporter now reports heap-materialized links as RAM with files as persistence-only, instead of claiming page-cache intent for data that never touches the page cache (previously such links were invisible: 0 RAM, 0 cached, full size as "expected cache"). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update lib/segment/src/index/hnsw_index/graph_links/links.rs Co-authored-by: Tim Visée <tim+github@visee.me> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Tim Visée <tim+github@visee.me> |
||
|
xzfc
|
785815e209 |
ConditionChecker fixups (#9559)
* Add NumericIndexValue * MapConditionChecker: get rid of `F: Fn() -> bool` bound To put this type into the upcoming `ConditionCheckerEnum`, it should be nameable. * filter_context: `Box<dyn ConditionChecker>` -> `OptimizedFilter` Removes one level of dyn indirection, so faster checks. * NullConditionChecker: merge IsEmpty/IsNull checkers into one So, less variants in the upcoming `ConditionCheckerEnum`. |
||
|
Andrey Vasnetsov
|
852ca200b5 |
feat: feature-flagged segment manifest for LocalShard (#9530)
* feat: feature-flagged segment manifest for LocalShard
Add an on-disk segment manifest (`segments/manifest.json`) that lists a
shard's segments and their state, so out-of-process readers (e.g. a
read-only follower, possibly over object storage) can discover segments
without scanning the filesystem. Gated by the new `write_segment_manifest`
feature flag (off by default).
shard: define the structure + helpers (`SegmentsManifest`,
`SegmentManifestState`, `from_segment_holder`) in a new `segment_manifest`
module, plus the `SEGMENT_MANIFEST_FILE` constant and path helper. The
manifest is a flat `{ "<uuid>": "<state>" }` map; only `active` is written
today, with `under_construction`/`retiring` defined so the format can be
extended without breaking compatibility.
collection: LocalShard owns the writing logic. The manifest is persisted
via `SaveOnDisk<SegmentsManifest>`, initialized from the live segment set
on load/build and refreshed by the optimization worker whenever the
segment set changes (the helper re-derives from the holder and no-ops when
unchanged). No changes to `lib/shard/src/optimize.rs` internals.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: make append_only_mutations a proper feature flag
Replace the debug-only `QDRANT_APPEND_ONLY_MUTATIONS=1` env-var escape
hatch in segment construction with a `FeatureFlags::append_only_mutations`
flag, so it works in release builds and is configurable like the other
flags (config / `QDRANT__FEATURE_FLAGS__APPEND_ONLY_MUTATIONS`).
Deliberately left out of `FeatureFlags::all()`: it changes mutation
semantics and `all` is enabled in dev and e2e configs, so it stays
explicit opt-in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* upd openapi schema
* feat: register segments in manifest via must-use token, holder builder
Wire segment-manifest maintenance through the segment lifecycle so a
newly created segment is registered as soon as it exists on disk, and
construction can't silently skip it:
- build_segment now returns a #[must_use] NewSegmentToken carrying the
new segment UUID; the lint forces callers to register or drop it.
- SegmentHolder owns the manifest and reconciles it on sync; new
segments are registered ASAP (even before being added to the holder)
via the token, before they can receive writes.
- SegmentHolderBuilder is the only way to obtain a shard's holder; its
build() wires up the manifest, so it can't be forgotten. init/set
manifest helpers are now private / test-only.
- Optimization registers the optimized segment before dropping the
superseded segments' data; deletion is intentionally lenient.
- Document the consistency assumptions on SegmentsManifest: it is a
superset-biased view that may list not-yet-finalized or already-deleted
segments, which readers must tolerate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Luis Cossío
|
8fc7b35d7a |
Explicit Populate in payload storage (and ` (#9514)
`GridstoreReader`) - Make the entrypoint of payload storage choose the `Populate` variant - Mutable payload indexes now populate gridstore blockingly before using it to load - Propagate `Populate` into `flags` module - Add `UniversalRead::populate_auto` to know whether a backend chooses to populate or not when `Populate::Auto` |
||
|
xzfc
|
bf1aa818c6 |
Sparse InvertedIndex: bring back generic methods (#9485)
* sparse InvertedIndex: bring back generics * nits --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |
||
|
Daniel Boros
|
4d6fb4e0ab |
feat: add open for read-only sparse vector index (#9435)
* feat: add open for read-only sparse vector index + enum sparse dispatcher * fix: universal-IO loads for read-only sparse index open * refactor: rename load_via/open_via to load_universal/open_universal * refactor: drop StorageVersion::load in favor of load_universal The regular-IO `load` duplicated `load_universal` over plain `File` IO. Remove it and route all callers through `load_universal(&MmapFs, ..)`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(sparse): decouple inverted index from concrete storage; split segment_constructor_base (#9461) Make the read-only index enum generic over storage `S` (no concrete MmapFile), and remove construction callbacks from the sparse index open paths. - InvertedIndex is now a pure read/search trait: `open`/`from_ram_index`/ `type Fs` moved off the trait to inherent methods on each concrete index type, so construction no longer requires `S::Fs: Default`. - SparseVectorIndex open split into a generic `plan` (load-vs-build decision + RAM-index build) and generic `finish` (assembly); callers do the concrete per-type construction, so no construction callbacks are needed. - ReadOnlySparseVectorIndex::open takes the already-constructed inverted index and caller-loaded config instead of a `load_inverted_index` callback. - VectorIndexReadEnum is generic over `S: UniversalRead`; sparse mmap variants hold `InvertedIndexCompressedMmap<_, S>` rather than a concrete `MmapFile`. - Split the 1182-line segment_constructor_base.rs into a module (paths, vector_storage, payload_storage, id_tracker, vector_index, sparse_vector_index, create_segment, segment, legacy_state); the sparse dispatcher's match arms collapse into three per-family helpers. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat: LiveReload (no-op) dispatch for read-only vector index enum (#9436) --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
xzfc
|
3d03dd4752 |
[UIO] condition checker: fallible checks (#9405)
* condition checker: fallible checks * Update |
||
|
Luis Cossío
|
da8a213ee1 | Add facets bench (#9456) | ||
|
Daniel Boros
|
d669a08c2b |
feat: add open to read-only HNSW index (#9427)
* feat: add open to read-only HNSW index * fix: dedicated universal-IO load for read-only graph * fix: report actual graph residency in read-only is_on_disk * refactor: rename load_via to load_universal * refactor: unify graph links loading on universal IO Replace the dual GraphLinks load paths (regular mmap + universal IO) with a single universal-IO loader, and the `Mmap` GraphLinksEnum variant with a `Universal` variant that keeps a type-erased `UniversalRead` handle alive behind `Box<dyn GraphLinksStorage>` (UniversalRead is not object-safe). - GraphLinksEnum::from_storage picks the variant from UniversalKind: borrowable (mmap-backed) kinds stay `Universal`, others are materialized into `Ram`, so the borrowability invariant in GraphLinksStorage::bytes holds by construction. - Loading takes a `Populate` parameter, derived from `hnsw_config.on_disk`: on_disk -> Populate::No (lazy), otherwise Populate::Blocking. - is_on_disk is reported from config again instead of the enum variant. - Split graph_links.rs into a graph_links/ module (format, vectors, storage, links, tests) with a relationship diagram in mod.rs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop LoadOption in favor of a Populate parameter After unifying the link load paths on universal IO, LoadOption only encoded a populate choice with a fs that was always MmapFs. Replace it with a `Populate` argument to GraphLayers::load, removing the enum, its constructors, the load_links helper, and the unused generic backend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ivan Pleshkov
|
6a20fc1df5 |
[TQDT] Vector storage update from without CowVector (#9325)
* [TQDT] Vector storage update_from without CowVector Move update_from to the kind-specific DenseVectorStorage, SparseVectorStorage and MultiVectorStorage traits in their native element type, and merge through batched_reader::merge_from. This drops the CowVector-based VectorStorageEnum::update_from shim: sources are copied in their native representation, with no f32 round-trip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * review remarks * remove obsolete comments --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Roman Titov
|
1e5013bcae | Add io_uring based payload storage type (#9310) | ||
|
Ivan Pleshkov
|
a8b5c1a243 |
[TQDT] Define update_from for each vector storage trait (#9357)
* define update from for each trait * are you happy clippy |
||
|
Luis Cossío
|
1a8d8dfdba |
Standardize on-disk geo index & friends (#9339)
* rename GeoMapIndex -> GeoIndex * rename on_disk module * fmt * remove `Box`ing * add separate immutable and on-disk read-only variants * propagate Populate, respect `on_disk`, remove `is_on_disk` * rename to `new_immutable` and fix tests * more geomap -> geo renaming * more adhoc `open_*` suffixes |
||
|
Roman Titov
|
7727bf3f35 | Use batched reads in vector and payload storage (#9113) | ||
|
Luis Cossío
|
66be8bd30a |
Rename indexes variants (#9281) + fixup for #9294
* rename IndexSelector variants * rename MapIndex::Mmap to MapIndex::OnDisk * rename NumericIndex::Mmap to NumericIndex::OnDisk * rename BoolIndex::Mmap to BoolIndex::Mutable * rename FullTextIndex::Mmap to FullTextIndex::OnDisk * rename GeoMapIndex::Storage to GeoMapIndex::OnDisk Also rename a few functions and test variants * Standardize numeric index (#9294) * remove `UniversalNumericIndex.is_on_disk` field * have separate Immutable and OnDisk variants for ReadOnlyNumericIndexInner * rename UniversalNumericIndex -> OnDiskNumericIndex * rename module to `on_disk_numeric_index` * actually load as immutable if on_disk=false * fixup! Standardize numeric index (#9294) Make builder also return proper Immutable variant * review nits |
||
|
Luis Cossío
|
fe28339d15 |
[UIO] Universal load numeric index (#9234)
* rename module, use read_via * propagate S to `UniversalNumericIndex<T, S>` |
||
|
xzfc
|
fff9c1f1c7 |
Migrate InvertedIndexCompressedMmap to UIO (#9144)
* Migrate InvertedIndexCompressedMmap to UniversalRead * Rehaul search_scratch.rs (was scores_memory_pool.rs) - Use `typed_arena::Arena` instead of `bumpalo::Bump`. Reason: bump don't drop. Drawback: `Arena::new()` is not free, and `Arena::clear()` doesn't exist; but this commit has workarounds. - Use names that make more sense. * Misc fixes * InvertedIndexCompressedMmap: explicit S type parameter |
||
|
Luis Cossío
|
5bc9847972 |
[UIO] Load GraphLinks with any backend (#9214)
* load with any universal io * Don't populate with sequential advice * use appropriate fs for `exists` * use `read_whole_via` more * TODO * clear ram cache after read_whole |
||
|
Andrey Vasnetsov
|
daa22f15e6 |
[UIO] Split UniversalReadFileOps into filesystem + file traits (#9151)
* [UIO] Split UniversalReadFileOps into filesystem + file traits
`UniversalReadFileOps` is now an instance-based trait describing a
filesystem handle (list/exists with `&self`, plus `from_context`). A new
`UniversalReadFs: UniversalReadFileOps` subtrait adds the
`open(&self, path, options) -> Self::File` capability with `type File:
UniversalRead`. `UniversalRead` no longer extends `UniversalReadFileOps`
and is purely a file-handle trait.
This separates "filesystem instance" from "file handle". Backends that
need per-instance configuration (S3 bucket name + credentials, mmap
default advice, io_uring runtime, block-cache controller `Arc`) gain a
typed home in `Self::ContextConfig`, and `list_files`/`exists`/`open`
become `&self` methods on the filesystem handle.
Concrete filesystem handles introduced for the three existing backends:
- `MmapFs` — unit struct; `ContextConfig = ()`; produces `MmapFile`.
- `IoUringFs` — carries `prevent_caching`; `ContextConfig =
IoUringConfigContext`; produces `IoUringFile`. `IoUringConfigContext`
becomes the construction input rather than a per-open argument.
- `BlockCacheFs` — carries `Arc<CacheController>`; `ContextConfig =
BlockCacheConfigContext`; produces `CachedSlice`.
`TConfigContext` (universal builder methods) is kept so generic-over-`Fs`
code can still set cross-backend knobs via
`Fs::ContextConfig::default().with_prevent_caching(true)`.
Wrappers (`ReadOnly<S>`, `TypedStorage<S, T>`, `StoredStruct<S, T>`),
higher-level storages (`StoredBitSlice<S>`, `UniversalHashMap<K, V, S>`)
keep `<S: UniversalRead>` parameterization but their `open(...)`
constructors now grow a `fs: &Fs` argument bound by
`Fs: UniversalReadFs<File = S>`. `read_json_via` becomes
`read_json_via(fs: &Fs, path)`.
All test code and benches in `common` updated to construct
`MmapFs`/`IoUringFs` inline as needed. `common` compiles cleanly with
tests and benches. `gridstore`, `segment`, and `tonic` caller updates
are in flight in subsequent commits.
* WIP: gridstore + segment caller sweep (partial)
Threads `fs: &Fs` through gridstore's `BitmaskGaps`, `Bitmask`, `Pages`,
and `Gridstore::new`/`open`/`create_new_page`. Most segment callers
have `OpenOptions { extra: ... }` removed and `S::open(path, opts, ctx)`
sites updated mechanically but the trait change is not yet propagated.
Does NOT compile yet. Tracker still has static `S::open` calls, segment
generic constructors (`MmapInvertedIndex<S>`, `UniversalMapIndex`,
`StoredGeoMapIndex`, etc.) still call `S::open`/`S::list_files`/`S::exists`
statically — they need an `fs: &Fs` parameter added. Tonic API
`StorageReadService<S>` also unconverted.
Committed as branch checkpoint; cascade continues in subsequent work.
* gridstore: thread `fs: &Fs` through Bitmask, BitmaskGaps, Pages, Tracker
Per the new `UniversalReadFs` shape, every constructor/method that opens
files takes an `fs: &Fs` parameter. Gridstore is currently mmap-only,
so the top-level `Gridstore` / `GridstoreReader::open` callers in the
crate pass `&MmapFs` inline. Tests do the same.
gridstore lib + tests now compile cleanly. Segment + tonic cascade
still pending.
* WIP: segment caller sweep — dynamic_stored_flags first
* WIP: segment cascade - id_tracker partial
* common benches: update to new UniversalReadFs::open shape (clippy clean)
* segment flags: thread Fs through BufferedDynamicFlags / Bitvec / Roaring
DynamicStoredFlags::set_len now takes `fs: &Fs`. BufferedDynamicFlags
stores an `Arc<Fs>` so the flusher closure can call `set_len` on resize.
BitvecFlags and RoaringFlags expose a new `Fs` type parameter and the
flag tests now pass `Fs::default()` (MmapFs/IoUringFs via duplicate_item).
Concrete consumers (bool/null index, mmap dense/multi/sparse storages)
pin `Fs = MmapFs` and pass `&MmapFs` to inner opens.
* segment: thread Fs through field-index lifecycle methods
Apply the new UniversalReadFs::open shape across:
- full_text_index (MmapInvertedIndex, MmapFullTextIndex, UniversalPostings)
- geo_index (StoredGeoMapIndex build/open + tests + builders)
- numeric_index lifecycle (UniversalNumericIndex build/open)
- map_index lifecycle (UniversalMapIndex build/open)
- stored_point_to_values (open / from_iter)
Concrete consumers pin Fs = MmapFs and pass &MmapFs inline; generic
open paths thread `fs: &Fs` where Fs: UniversalReadFs<File = S>.
* segment: thread Fs through chunked vectors and id-tracker callers
- ChunkedVectors gains `Fs` generic so add_chunk can call create_chunk
after open. ChunkedVectorsRead/load_config and chunks::{read_chunks,
create_chunk} take `fs: &Fs`. Concrete callers (dense / multi-dense /
sparse / quantized) pass MmapFs inline.
- DenseVectorStorageImpl stores `fs: Fs` so `update_from` can reopen
ImmutableDenseVectors. ImmutableDenseVectors::open takes `fs: &Fs`.
- VectorStorageEnum DenseUring* variants thread IoUringFs alongside
IoUringFile.
- segment_builder + segment_constructor_base pass MmapFs to
ImmutableIdTracker::{new, open}.
- QuantizedStorage::from_file takes `fs: &Fs`; quantized_vectors callers
pass &MmapFs.
* segment: apply nightly rustfmt after Fs refactor
cargo +nightly fmt --all over the segment crate after the
UniversalReadFs cascade. No semantic changes.
* segment: thread Fs through benches and id-tracker tests
Update the dynamic-mmap-flags and buffered-update-bitslice benches to
the new UniversalReadFs::open shape (pass `&MmapFs`). Update the
immutable-id-tracker test suite to forward `&MmapFs` to
`from_in_memory_tracker` / `open`.
* uio: pin Fs via UniversalRead::Fs assoc type; per-call OpenExtra
Two design changes that fall out of the per-instance Fs refactor:
1. Bidirectional Fs ↔ File pinning. `UniversalRead::Fs:
UniversalReadFs<File = Self>` lets generic-over-`S` code refer to
`S::Fs` directly instead of carrying an extra `<Fs: UniversalReadFs<File = S>>`
generic param. `ReadOnly<S>` wraps a file but has no natural
filesystem; a phantom `ReadOnlyFs<S::Fs>` satisfies the constraint
while inherent `ReadOnly::open` keeps taking `&S::Fs` directly.
2. `prevent_caching` moves from filesystem-instance state to per-call
`UniversalReadFs::OpenExtra: Default`. Was previously a knob on
`IoUringConfigContext` / `IoUringFs`, conflating "how this fs is
built" with "how this file is opened." Now `IoUringFs::OpenExtra =
IoUringOpenExtra { prevent_caching }`; mmap and block-cache use `()`.
`IoUringConfigContext` is gone, `TConfigContext` slims to a `Default`
marker.
Tonic StorageReadService holds `Arc<S::Fs>` (was `PhantomData<S>`); its
`new()` builds via `S::Fs::from_context(default)` and the spawn_blocking
closures clone the Arc to call instance methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: cfg-gate IoUringOpenExtra import for non-linux builds
The IoUringOpenExtra reexport from `universal_io` is gated on
`target_os = "linux"`. The previous commit left an unconditional import
in `persisted_hashmap/tests.rs`, breaking macOS/Windows CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: migrate simple_disk_cache to per-instance Fs API
PR #9097 (merged into dev concurrently with this branch) introduced a
`DiskCache<R: UniversalRead>` using the pre-refactor trait shape:
file-handle-as-filesystem (`R::open`, `R::list_files`), an
`OpenOptionsExtra` field on `OpenOptions`, and trait methods without
`&self`. The Fs-instance refactor on this branch removed all three.
Reshape `simple_disk_cache` to match the new design without breaking the
lazy mirror semantics:
- New `DiskCacheFs<R>` is the filesystem handle. Holds a clone of the
remote `R::Fs`; `list_files`/`exists` delegate; `from_context`
forwards to the inner Fs context. `open` constructs a `DiskCache<R>`
via the global `DiskCacheConfig`.
- `DiskCache<R>` now stores `remote_fs: R::Fs` + `remote_extra:
<R::Fs as UniversalReadFs>::OpenExtra`, so lazy remote opens go
through `self.remote_fs.open(path, options, extra)` instead of the
removed `R::open`. `open_with_config` takes the remote Fs + extra
explicitly (no more hard-coded `prevent_caching: true`; callers pass
the appropriate `OpenExtra`).
- `UniversalRead for DiskCache<R>` now declares `type Fs =
DiskCacheFs<R>` (no more `open` method on the file trait).
- Propagate the necessary bounds (`R::Fs: Clone`,
`<R::Fs as UniversalReadFs>::OpenExtra: Clone`,
`R::OwnedReadPipeline<u8, Range<u32>>: Send`) through `pipeline.rs`
free functions and impl blocks that reach into `DiskCache::remote` /
`local_state`.
- Drop the now-removed `extra: _` destructure in `LocalState::new`.
- Tests construct the remote Fs via `R::Fs::from_context(Default::default())`
and exercise `DiskCache::open_with_config`. 17 simple_disk_cache
tests pass; the 3 `empty_read_does_not_materialize_local_file`
failures pre-exist on dev (verified) and are unrelated.
`mold -run cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt on simple_disk_cache migration
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt for local_state imports after rebase
Co-authored-by: Cursor <cursoragent@cursor.com>
* uio: split DiskCacheFs into its own module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: replace TConfigContext with OpenExtra trait; move DiskCacheConfig onto DiskCacheFs
- Drop the empty TConfigContext marker; ContextConfig is now unconstrained
so backends can require explicit construction.
- Add OpenExtra trait with with_prevent_caching for backend-agnostic
per-call knobs; impl for () (no-op) and IoUringOpenExtra.
- DiskCacheFs now carries Arc<DiskCacheConfig> via the new
DiskCacheFsContext<C>; the prefill flow moves from the deleted
open_with_config into DiskCacheFs::open so Populate::Blocking /
PreferBackground work through the trait API.
- Remove the DiskCacheConfig global; callers must construct the context
explicitly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt after OpenExtra refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: fix spelling — Implementors → Implementers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: possible panic insetad of error propagation
* fix: missing cfg annotation
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
|
||
|
xzfc
|
905f3250b7 |
sparse: use zerocopy (#9140)
* sparse: use zerocopy for mmap reads * loaders::Csr: use zerocopy |
||
|
xzfc
|
899857c4f5 |
Make OpenOptions explicit (#9104)
* Refactor: make OpenOptions explicit * Refactor: remove unused OpenOptions::disk_parallel * Refactor: do not wrap `advice` in Option * Refactor: Add OpenOptionsExtra |
||
|
xzfc
|
faf2714f4b | Warn on clippy::wildcard_enum_match_arm (#9096) | ||
|
Andrey Vasnetsov
|
75a0a5dd0b |
refactor: move deferred-point ownership into ID tracker (#9062)
* refactor: move deferred-point ownership into the ID tracker Re-implements the idea from #8512 against current `dev`. Deferred-point state (`deferred_internal_id` + `deferred_deleted_count`) moves out of `Segment.deferred_point_status` and the cached `SparseVectorIndex.deferred_internal_id` field into `PointMappings`, exposed through `IdTrackerRead`. The threshold is set once at `MutableIdTracker::open` time; `PointMappings::drop` now maintains the deleted counter inline (with double-delete protection), removing the manual increment in `delete_point_internal` and the `calculate_deleted_deferred_point_count` rescan. Read paths consume the threshold through the id tracker: - The segment read view drops the `deferred_point_status` field and `with_view` no longer threads it in; `read_view/{deferred,info}.rs` call `self.id_tracker.deferred_*()` directly. - `SparseVectorIndex` no longer stores its own copy and its `update_vector` / search debug-assert read from `self.id_tracker.borrow().deferred_internal_id()`. - `VectorQueryContext.deferred_internal_id` and the `SegmentQueryContext::get_vector_context` parameter are gone; the three downstream readers (`plain_vector_index`, sparse search, sparse `update_vector`) consult their own id tracker. `PointMappingsRefEnum` centralises the dispatch: - `iter_internal_with_behavior(DeferredBehavior)` replaces ad-hoc branches in `iter_filtered_points` impls. - `external_iter_cutoff(DeferredBehavior)` covers iterators sourced outside the mapping (field-index outputs in `struct_payload_index::iter_filtered_points`). - The internal `deferred_internal_id()` accessor is private; the raw threshold no longer leaks to consumers. - `iter_from_visible` / `iter_random_visible` read the mapping's own threshold; callers that previously passed `DeferredBehavior::apply(...)` now branch on `deferred_behavior.include_all_points()` (scroll / order_by) or simply drop the argument (sampling / facet). `PayloadIndexRead::query_points` drops the now-redundant `deferred_internal_id` parameter; `iter_filtered_points` takes `DeferredBehavior` directly so HNSW build/search can request `IncludeAll` while normal reads request `Exclude`. RocksDB-related parts of the original PR are skipped — that tracker is already gone from `dev`. Tests adapted: sites that mutated `segment.deferred_point_status` directly now construct a parallel non-deferred segment via `create_deferred_segment(..., 0)` for comparison; `test_deleted_deferred_point_count` reads counters through the id tracker. See `docs/plans/deferred-points-owned-by-id-tracker.md` for the design write-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(benches): drop stale deferred_internal_id arg from query_points calls The boolean / range / conditional bench files weren't built by `cargo test -p segment`, so they slipped through. `cargo clippy --workspace --all-targets` catches them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: drop id_tracker / point_mappings args from iter_filtered_points Both impls already hold an id tracker on `self`: - `StructPayloadIndexReadView` carries `id_tracker: &'a I`, so `self.id_tracker.point_mappings()` borrows from `'a` and the lazy iterator chain keeps working unchanged. - `PlainPayloadIndex` carries `id_tracker: Arc<AtomicRefCell<...>>`, where the mapping borrow is local; collect into a `Vec` and return `into_iter()`. PlainPayloadIndex::iter_filtered_points has no direct callers — only `query_points` was using it — so eager collection is a non-issue. While here, take `self` by value on `iter_internal_visible`, `iter_from_visible`, `iter_random_visible`, `iter_internal_with_behavior`, and `external_iter_cutoff`. `PointMappingsRefEnum` is `Copy`; this matches the existing `iter_internal` / `iter_from` / `iter_random` shape and lets the iterator outlive a local `let point_mappings = ...;` binding. The HNSW `condition_points` helper drops its now-unused `id_tracker` parameter. All callers (sampling, scroll, order_by, facet ×2, hnsw build/search) just drop the two arguments. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: ignore /docs/plans/ and untrack the previously-committed plan `docs/plans/` is a scratch directory for per-feature planning notes — not something we want under source control. Add it to `.gitignore` and drop the deferred-points plan that slipped into history; the design is captured in the PR description. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: replace external_iter_cutoff with filter_deferred iterator wrapper Instead of exposing a raw `Option<PointOffsetType>` cutoff that every caller has to apply with their own `.filter(...)`, give `PointMappingsRefEnum` an iterator wrapper: fn filter_deferred<I: Iterator<Item = PointOffsetType>>( self, iter: I, deferred_behavior: DeferredBehavior, ) -> impl Iterator<Item = PointOffsetType> It returns the iterator unchanged for `IncludeAll` (or when the mapping has no threshold) and otherwise wraps it in a cutoff `.filter`, dispatched via `itertools::Either` so the no-cutoff path stays allocation-free. The struct payload index's `iter_filtered_points` swaps its open-coded filter for a single `point_mappings.filter_deferred(...)` call. The deferred threshold no longer leaks out of `PointMappingsRefEnum`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: move deferred wrapping out of peek_top_all, gate it as test-only `BatchFilteredSearcher::peek_top_all` baked the deferred cutoff into its iterator construction, which was the last place outside `PointMappingsRefEnum` that knew about the threshold. Split the deleted-iteration concern out into a new accessor: fn iter_not_deleted(&self) -> impl Iterator<Item = PointOffsetType> + 'a It borrows `&'a BitSlice` directly (not via `&self`), so callers can chain `filter_deferred` and then move `self` into `peek_top_iter` without lifetime conflicts. Sparse + plain vector index call sites now do: let iter = id_tracker .point_mappings() .filter_deferred(searcher.iter_not_deleted(), DeferredBehavior::Exclude); searcher.peek_top_iter(iter, &is_stopped) leaving `BatchFilteredSearcher` completely ignorant of deferred state. With deferred handling lifted out, `peek_top_all` itself is now used only by tests (3 inline `#[cfg(test)] mod tests`, 1 integration test, 1 bench) — gate it under `#[cfg(feature = "testing")]` to match `new_for_test`. Production code goes through the `iter_not_deleted` + `filter_deferred` + `peek_top_iter` composition. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: optimize Segment::retrieve and thread user data through read_vectors Two interlocking changes that together collapse the per-point lookups and intermediate allocations in `Segment::retrieve` down to one external-to-internal pass. ## `IdTrackerRead::resolve_external_ids` (new default trait method) Single-pass translation of a `&[PointIdType]` slice into two parallel vectors `(Vec<PointIdType>, Vec<PointOffsetType>)`. Folds deferred filtering (compare offset against the threshold inline — no separate `point_is_deferred` lookup) and missing-id errors (eager `PointIdError`) into resolution. Lives on the trait so the deferred threshold never leaks out of the id tracker; the parallel-vector shape lets a future batched payload / vector fetcher consume `&offsets` straight without unzipping. The `appendable_flag` guard previously in `point_is_deferred` is gone: non-appendable trackers always carry `deferred_internal_id() == None` (set only via `MutableIdTracker::open`, guarded by the segment constructor), so the check was load-bearing nowhere. ## User-data threading through `read_vectors` `VectorStorageRead::read_vectors` now takes `IntoIterator<Item = (U, PointOffsetType)>` and yields `(U, PointOffsetType, CowVector)`. The user-data tag rides alongside each offset all the way through, so callers can map results back into a parallel array without keeping a separate `offset → ...` lookup table. - Default trait impl: one-line per-key loop. - Dense impl: `unzip()` into parallel `(Vec<U>, Vec<PointOffsetType>)` in a single pass — same allocation count as before, just U riding alongside. - Enum delegations (`VectorStorageEnum`, `VectorStorageReadEnum`) forward unchanged. - `for_each_in_batch` and below stay untouched. `SegmentReadView::vectors_by_offsets<U: Copy>` becomes a lazy filter chain — no parallel `Vec<(orig_idx, offset)>` allocation. The dead `SegmentReadView::read_vectors` helper is removed. ## `Segment::retrieve` end-to-end Per N points / V vectors / payload: | Operation | Before | After | |----------------------------|---------------------|-------| | `id_tracker.internal_id` | N × (1 + V + 1) | N | | `id_tracker.external_id` | N × V | 0 | | `point_is_deferred` | N (when applicable) | 0 | | `offset_to_id` HashMap | N entries | none | | `Vec` in `vectors_by_offsets` | 1 | 0 | The vectors stage passes the external id as `read_vectors`'s user data — the callback gets `id` directly without any index lookup. The payload stage uses `payload_by_offset` against the already-resolved offsets. The shape is also batch-friendly: swapping in a future `IdTrackerRead::batch_internal_id` or `payload_index.batch_get_payload` needs no changes outside the two call sites. ## Behavioural notes - Missing-id now errors eagerly inside resolution, instead of in the vectors stage (`WithVector::Bool(true)` / `Selector`) or payload stage (`with_payload.enable`). The previous `WithVector::Bool(false)` + no-payload path silently inserted an empty record; that is now also an error. None of the existing callers (search post-processing, external retrieve API, the deferred-points test on tests/mod.rs:1179) pass non-existent ids. - Added a per-payload `check_stopped`; the vectors stage already had `stop_if` on its iterator chain. - `vector_by_offset` (the single-element helper) passes `()` as the no-op user data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: apply rustfmt to optimised retrieve / read_vectors paths Pre-push hook failure on the previous commit was rustfmt. Same content, formatted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * do not error out on missing points in retrieve --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
xzfc
|
bb8cdf6e5e | Remove unused InvertedIndexMmap / InvertedIndexImmutableRam (#9074) | ||
|
Andrey Vasnetsov
|
5436eb3bde |
refactor: read-only numeric index (#9038)
* refactor: split numeric index variants into dedicated modules Move MutableNumericIndex, ImmutableNumericIndex, and MmapNumericIndex into their own directories, each split into mod.rs (struct definitions), lifecycle.rs (open/build/wipe/mutations), and read_ops.rs (accessors), mirroring the map_index layout. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: drop single-variant Storage enums in mutable/immutable numeric index Replace `Storage<T>` wrappers around the only backing store with the store types directly: `Gridstore<Vec<T>>` for `MutableNumericIndex` and `Box<MmapNumericIndex<T>>` for `ImmutableNumericIndex`. Collapses the trivial single-arm matches into direct method calls. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: introduce NumericIndexRead trait Mirror `MapIndexRead` from the map_index refactor: define a `NumericIndexRead<T>` trait in `numeric_index/read_ops.rs` and implement it on each of the three storage variants (`MutableNumericIndex`, `ImmutableNumericIndex`, `MmapNumericIndex`). Trait signatures are unified across variants — in-memory variants accept and ignore the `hw_counter` argument that the mmap-backed variant uses for IO tracking, and `total_unique_values_count`, `values_range`, and `orderable_values_range` return `OperationResult` everywhere so the dispatcher in `NumericIndexInner` can call them generically. Variant-specific helpers that don't fit the shared shape stay as inherent methods: `MutableNumericIndex::map()`, `ImmutableNumericIndex::values_range_size()`, and `MmapNumericIndex::{values_range_size, is_on_disk}`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: add ReadOnlyAppendableNumericIndex Counterpart to `MutableNumericIndex`, mirroring `ReadOnlyAppendableMapIndex` from the map_index refactor. It reuses the shared `InMemoryNumericIndex` in-memory state but is backed by a `GridstoreReader` over generic `UniversalRead` instead of a writable `Gridstore`, and implements `NumericIndexRead` by forwarding to the in-memory index — no mutation surface. Loading / lifecycle (constructor, files, populate, clear_cache) will follow in a separate change; the storage field is held only to pin the on-disk layout for now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: rename MmapNumericIndex to UniversalNumericIndex, expose UniversalRead param Mirror `UniversalMapIndex`: the type is now generic over `S: UniversalRead` with a `MmapFile` default, so the index can be served from any `UniversalRead` backend (io_uring, disk-cache wrappers, …) rather than the hard-coded `MmapFile`. The `NumericIndexRead` impl and read-side helpers are generic over `S`; `build` / `open` and the other lifecycle methods stay `MmapFile`-only since they construct mmap-backed storage from a path. The `NumericIndexInner::Mmap` enum variant keeps its name and uses the default `S = MmapFile`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: split numeric_index/storage/mod.rs into lifecycle and read_ops `storage/mod.rs` now holds only the `NumericIndexInner` enum and module wiring. The variant-dispatch impls are split into sibling modules matching the layout of the individual storage variants: - `lifecycle.rs`: construction, persistence, file listing, cache control, and `remove_point`. - `read_ops.rs`: read-path forwarding — value lookups, telemetry, RAM accounting, `is_on_disk`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: implement NumericIndexRead for NumericIndexInner The enum-level read dispatch was a set of inherent methods scattered across storage/read_ops.rs and storage/statistics.rs with signatures that drifted from the variant trait (`values_count` returned `usize`, `max_values_per_point` vs `get_max_values_per_point`, a hand-rolled `get_telemetry_data`). Make `NumericIndexInner` implement `NumericIndexRead` directly so it shares one interface with the three storage variants. - All 12 trait methods are forwarded via match dispatch in storage/read_ops.rs; `values_range` / `orderable_values_range` box the per-variant iterators. - `get_histogram`, `get_points_count`, `total_unique_values_count` move out of statistics.rs into the trait impl; `values_is_empty` and `get_telemetry_data` now come from the trait defaults. - `point_ids_by_value` and `is_on_disk` stay as enum-only inherent helpers (not part of the shared trait). - Callers updated: `NumericIndex::values_count` unwraps the now `Option`-returning trait method; `filter` boxes `point_ids_by_value`; `field_index.rs` and `numeric_field_index.rs` import the trait. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: add ReadOnlyNumericIndexInner Read-only counterpart to `NumericIndexInner`, mirroring `ReadOnlyMapIndex` from the map_index refactor. Lives under `numeric_index/storage/read_only` and selects across the two read-only storage backends: - `Appendable(ReadOnlyAppendableNumericIndex<T, S>)` — loaded into RAM from the appendable Gridstore format. - `Immutable(UniversalNumericIndex<T, S>)` — served directly from the immutable stored format. Implements `NumericIndexRead` by forwarding each method to the active variant; `values_is_empty` / `get_telemetry_data` come from the trait defaults. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: rename numeric_index read_ops to numeric_index_read, split mod.rs Two changes: - Rename `numeric_index/read_ops.rs` (the `NumericIndexRead` trait definition) to `numeric_index_read.rs`, freeing the `read_ops` name. - Split the leftover content of `numeric_index/mod.rs` into sibling modules, matching the per-variant layout: - `lifecycle.rs`: the `Encodable` key-format trait + impls and the `HISTOGRAM_*` construction constants. - `read_ops.rs`: the `StreamRange` trait and the `Range` → index-key-bounds conversion. `mod.rs` now only wires modules and re-exports. `Encodable` and `StreamRange` keep their public paths via re-export; `tests.rs` gains explicit imports for the symbols it previously picked up through the `mod.rs` glob. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: dissolve numeric_index/index.rs into mod, lifecycle, read_ops `index.rs` only held the `NumericIndex` wrapper and its two impl blocks; spread them to match the per-module layout used elsewhere: - `NumericIndex` struct + `NumericIndexIntoInnerValue` trait → `mod.rs` (type definitions live with the module wiring). - The inherent `impl NumericIndex` (open / build / cache control / storage introspection) → `lifecycle.rs`, alongside the `HISTOGRAM_*` seed constants. - The `PayloadFieldIndexRead` impl → `read_ops.rs`. Also move the `Encodable` key-format trait out of `lifecycle.rs` into its own `encodable.rs`. `mod.rs` keeps re-exporting `Encodable` so its public path is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add ReadOnlyNumericIndex with NumericIndexRead + PayloadFieldIndexRead Read-only counterpart to `NumericIndex`, wrapping `ReadOnlyNumericIndexInner` plus the payload value type parameter `P`. Implements both `NumericIndexRead` and `PayloadFieldIndexRead` by forwarding to the inner storage-variant enum. To support `PayloadFieldIndexRead` without duplicating the query logic, the cardinality/filter/payload-block/condition-checker code is extracted into a new `query` module of generic free functions over `NumericIndexRead<T>`. `ReadOnlyNumericIndexInner` implements `PayloadFieldIndexRead` by plugging into those helpers; `ReadOnlyNumericIndex` delegates to its inner. The writable `NumericIndexInner` path is left untouched — its existing variant-specialized `estimate_points` heuristic stays in `storage`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: make query.rs the single source of truth for numeric index queries The generic `query` helpers and the per-`NumericIndexInner` impls in `storage/{trait_impls,statistics}.rs` had duplicated cardinality / filter / payload-block / condition-checker logic. Collapse them onto the shared `query` helpers: - `storage/trait_impls.rs`: `PayloadFieldIndexRead for NumericIndexInner` now forwards each method to `query::*` instead of carrying its own copy. - `storage/statistics.rs`: deleted — `range_cardinality` and `estimate_points` were duplicates of the `query` versions. - `estimate_points` needs a range size; add `values_range_size` to the `NumericIndexRead` trait with a default that counts `values_range`, overridden by the `Immutable` / `Mmap` variants with their `O(log n)` boundary search. The `MutableNumericIndex::map()` accessor (its only caller was the old `estimate_points`) is removed. - `values_range_size` takes `hw_counter` and threads it into `values_range` rather than fabricating a disposable counter. `tests.rs` calls `query::range_cardinality` directly now that the inherent method is gone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: integrate read-only numeric index into ReadOnlyFieldIndex Wire the four numeric variants (`IntIndex`, `DatetimeIndex`, `FloatIndex`, `UuidIndex`) into `ReadOnlyFieldIndex`, mirroring `FieldIndex`: - `PayloadFieldIndexRead` / `FieldIndexRead` dispatch covers the new variants — telemetry, value counts, value retrievers, `as_numeric`. - `ReadOnlyNumericFieldIndex` is the read-only counterpart of `NumericFieldIndex` (Int/Float order-by erasure over `ReadOnlyNumericIndexInner`); `as_numeric` returns it for the Int/Datetime/Float variants (UUIDs aren't numerically order-by-able, matching `FieldIndex`). - `ReadOnlyNumericIndex` gains per-`(T, P)` `value_retriever` methods (in `read_only/value_retriever.rs`) and an `inner()` accessor. `StreamRange` is now backed by a shared generic `query::stream_range` helper over `NumericIndexRead`, implemented for both `NumericIndexInner` and `ReadOnlyNumericIndexInner` — replacing the bespoke `EitherVariant` dispatch in `storage/trait_impls.rs`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: collapse ReadOnlyNumericIndex value retrievers onto one generic method The four per-`(T, P)` `value_retriever` methods were identical except for the per-value `T -> Value` conversion. Extract that conversion into a `NumericValueToJson` trait (one tiny impl per `(T, P)`) and keep a single generic `value_retriever` that builds the retriever closure once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fmt * refactor: dedup NumericFieldIndex / ReadOnlyNumericFieldIndex The two enums were structurally identical — same `StreamRange`, `get_ordering_values`, and `NumericFieldIndexRead` bodies — differing only in the backing storage type. Collapse them onto one generic `NumericFieldIndexView<'a, I, F>` with a single set of impls (over `I: NumericIndexRead<i64> + StreamRange<i64>` and the `f64` counterpart). `NumericFieldIndex` and `ReadOnlyNumericFieldIndex` are now type aliases of the generic view, so every existing call site is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
8de9c11a5d |
refactor(index): drop PayloadIndex: PayloadIndexRead super-trait (#8969)
* refactor(index): drop PayloadIndex: PayloadIndexRead super-trait `PayloadIndex` now only declares the mutating surface; reads live on the sibling `PayloadIndexRead` trait. The previous super-trait relationship forced any type that implemented `PayloadIndex` to also implement `PayloadIndexRead`, blocking a future `PayloadIndexRead`-only view that doesn't (and shouldn't) own the writable index machinery. No behavioural change. Audit before committing showed no generic bound site on `PayloadIndex` exists in the workspace, and every caller that uses read methods already imports `PayloadIndexRead` explicitly (the trait was already used as a generic bound on `SegmentReadView`'s `TPayloadIndex` parameter and on `iter_filtered_points`). The full workspace builds clean and all segment / storage / collection tests pass without any consumer update. Doc comment on `PayloadIndex` updated to point readers at `PayloadIndexRead` for the read surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(index): introduce StructPayloadIndexReadView<P, I, V> (#8970) Move the read surface of `StructPayloadIndex` onto a new borrowed view struct generic over `<P: PayloadStorageRead, I: IdTrackerRead, V: VectorStorageRead>`. The view holds exactly the fields that `PayloadIndexRead` requires -- no more, no less: pub struct StructPayloadIndexReadView<'a, P, I, V> { payload: &'a Arc<AtomicRefCell<P>>, id_tracker: &'a I, vector_storages: &'a HashMap<VectorNameBuf, Arc<AtomicRefCell<V>>>, field_indexes: &'a IndexesMap, config: &'a PayloadConfig, visited_pool: &'a VisitedPool, } `StructPayloadIndex` now exposes a `with_view(|v| ...)` accessor that borrows `id_tracker` once at the top and constructs the view for the closure scope. All read-method bodies move onto the view, which is the sole `PayloadIndexRead` implementor for this index. Why three generics ================== - `P: PayloadStorageRead` -- already generic via PR #8968. - `I: IdTrackerRead` -- direct method calls; held as `&I` (not `&Arc<AtomicRefCell<I>>`) because the cell is collapsed at the `with_view` boundary, saving a per-method `borrow()` atomic op. `dyn IdTrackerRead` does not satisfy `I: IdTrackerRead` bounds in Rust without an explicit blanket impl, so generic is the only consistent option here. - `V: VectorStorageRead` -- the only access site is `available_vector_count()` for the `HasVector` cardinality branch (`condition_cardinality` in `read_view/filtering.rs`). Why `payload` keeps the `Arc` ============================= `PayloadProvider<P>::new(...)` (introduced in PR #8968) takes `Arc<AtomicRefCell<P>>` so that the returned `FormulaScorer<'q>` / `Box<dyn FilterContext + 'a>` can outlive the caller frame. The view therefore holds `&'a Arc<AtomicRefCell<P>>` (asymmetric vs the bare `&I` for `id_tracker`). Switching to a borrow-based provider would require reworking `formula_scorer` / `filter_context` to callback style; deferred to a follow-up if needed. What does NOT move ================== - `build_field_indexes` and `clear_index_for_point` stay on `StructPayloadIndex`. `build_field_indexes` is read-shaped but only has write-side callers, and pulls in the `selector` machinery which uses `path` + `storage_type`. Keeping it on the writable struct means `path` and `is_appendable` do not need to leak into the view. - The `selector` / `selector_with_type` helpers stay on the writable struct for the same reason. - The free helpers in `query_optimization/condition_converter.rs` (range / geo / null / is-empty checkers) stay where they are; their visibility is bumped from `fn` to `pub(in crate::index)` so the view can still call them. Module layout ============= lib/segment/src/index/struct_payload_index/ mod.rs # owning struct + with_view build.rs # write-side build coordination payload_index.rs # impl PayloadIndex (mutating only) tests.rs read_view/ mod.rs # view struct + module wiring payload_index_read.rs # impl PayloadIndexRead for view filtering.rs # struct_filtered_context, condition_cardinality, query_field, estimate_field_condition condition_converter.rs # impl block from query_optimization/ optimizer.rs # impl block from query_optimization/ value_retriever.rs # impl block from query_optimization/ tests.rs # smoke test that builds the view directly Consumer migration ================== - `Segment::with_view` nests the new `StructPayloadIndex::with_view` inside it; `SegmentReadViewFor<'s>` uses the view as its `TPayloadIndex` parameter. - HNSW (`hnsw.rs`), sparse (`sparse_vector_index.rs`), plain (`plain_vector_index.rs`) call sites wrap their read-method calls in `payload_index.borrow().with_view(|v| ...)`. - `Segment::get_indexed_fields`, `update_all_field_indices`, and `SegmentBuilder::build` switch to `with_view` for `indexed_fields()` / `get_payload_sequential()`. - Integration tests and benches similarly migrate. - `set_payload` (still on `PayloadIndex` write impl) inlines its former `self.get_payload(...)` call as `self.payload.borrow().get(...)` to avoid going through `with_view` from a `&mut self` write path. Smoke test (`read_view/tests.rs`) constructs the view directly over `InMemoryPayloadStorage` + `InMemoryIdTracker` + an empty vector-storage map, and exercises `indexed_fields()`, `query_points()`, and `available_point_count()` -- proving the view is genuinely decoupled from `StructPayloadIndex`. This is the abstraction PR 4 will use to wire a read-only segment. Verified ======== - `cargo build --workspace --tests --benches` -- green - `cargo test -p segment --lib` -- 666 passed (665 + 1 new smoke test), 0 failed - `cargo test -p segment --tests` -- 120 integration tests pass - `cargo test -p storage --lib` -- 44 passed - `cargo test -p collection --lib` -- 197 passed - `cargo clippy -p segment --tests --benches` -- clean Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Luis Cossío
|
c1ba01ba39 |
[UIO] Generic dynamic flags (#8893)
* rename status * impl simple StoredStruct * use StoredStruct in `status` * rename to `DynamicStoredFlags` * Propagate `S` generic * codespell nit |
||
|
Daniel Boros
|
76031bd261 |
feat: split read only payload index (#8858)
* feat: split read only payload index * fix: imports * fix: linter |
||
|
Andrey Vasnetsov
|
483809294d |
split read only vector index (#8855)
* [AI] split trait for vector index into read only * fmt * gpu fix * fmt |
||
|
Andrey Vasnetsov
|
144528da31 |
split read only vector store (#8852)
* [AI] split trait for vector store into read only * fmt * fix: trait import --------- Co-authored-by: Daniel Boros <dancixx@gmail.com> |
||
|
Andrey Vasnetsov
|
c1597ac57e |
Split IdTracker trait into IdTrackerRead and IdTracker (#8826)
Read-only methods now live on a separate IdTrackerRead trait, with the mutating IdTracker trait extending it. This lets read-only call sites depend only on the read API. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Ivan Boldyrev
|
ca4ed7af69 |
Immutable storage for mmap numeric index (#8594)
* Immutable storage for mmap numeric index Do not use a deleted mmap-ed storage, instead, reconstruct a bitmask from the index and segment-level deleted mask. It effectively makes the mmap numeric index immutable, and immutable numeric index too as it delegates the storage to mmap numeric index. * Index reload tests * Make clippy happy * Restore deletion bitmask to reduce IO on index load It is more compact than count data we read before to check for cleared payloads. * immutable payload index storage 1 bis review (#8638) * use bitwise operations instead of loop * [AI] Only propagate bitslice * fmt * Fix deleted bitmask length It must be the same length as `point_to_values` length. * account for ram usage * Default missing external deletion bits to live, not deleted. * clear doc for flush --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> |
||
|
Luis Cossío
|
96c3bfa160 | [UIO] Dynamic mmap flags (#8760) | ||
|
Luis Cossío
|
5dc27660e5 |
[UIO, gridstore] generic Bitmask storage (#8501)
* [AI + manual] generalize BitmaskGaps storage * [AI + manual] use StoredBitSlice in Bitmask * use MmapFile as default storage * rename to read_all * duplicate import |
||
|
Roman Titov
|
9ba216b23f |
Remove AsyncRawScorer (#8685)
* Merge `scorer_mmap` and `vector_search` benchmarks * Move micro-batching logic from `RawScorerImpl::score_points` into `QueryScorer::score_stored_batch` * Propagate micro-batching logic from dense scorers... ...into `ImmutableDenseVectors`/`ChunkedVectors` * Implement io_uring-specialized `ImmutableDenseVectors::for_each_in_batch_async` * Implement io_uring-specialized `ChunkedVectors::for_each_in_batch_async` * Add `UniversalRead::type_id` method for runtime storage-type queries * fixup! Implement io_uring-specialized `ImmutableDenseVectors::for_each_in_batch_async` Enable io_uring-specialized scoring on Linux * fixup! Implement io_uring-specialized `ChunkedVectors::for_each_in_batch_async` Enable io_uring-specialized scoring on Linux * Refactor `DenseVectorStorageImpl::read_vectors`... ...to use `ImmutableDenseVectors::for_each_in_batch` instead of `read_vectors_async` * Remove `AsyncRawScorer` * fixup! Propagate micro-batching logic from dense scorers... Fix bugs * fixup! Propagate micro-batching logic from dense scorers... * fixup! Merge `scorer_mmap` and `vector_search` benchmarks Fix clippy 🙄 * fixup! Add `UniversalRead::type_id` method for runtime storage-type queries Change to `UniversalRead::kind` that returns `UniversalKind` enum * fixup! Implement io_uring-specialized `ImmutableDenseVectors::for_each_in_batch_async` Use `UniversalRead::kind` instead of `type_id` * fixup! Implement io_uring-specialized `ChunkedVectors::for_each_in_batch_async` Use `UniversalRead::kind` instead of `type_id` * review: rename point_id -> point_offset to match the type [skip-ci] --------- Co-authored-by: generall <andrey@vasnetsov.com> |
||
|
Tim Visée
|
e001a50dc4 |
Fix clippy warnings for Rust 1.95 (#8695)
* Remove redundant into_iter * Remove redundant type casting * Use if-branches in match * Use sort_by_key * Only iterate over values * Dismiss bench loop counter warning * done done --------- Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> |
||
|
Luis Cossío
|
ab49dbd188 |
[UIO] BufferedUpdateBitSlice (#8679)
* rename * migrate `MmapBitsliceBufferedUpdateWrapper` -> `BufferedUpdateBitSlice` |
||
|
dependabot[bot]
|
3acd7fffa9 |
build(deps): bump sha2 from 0.10.9 to 0.11.0 (#8558)
* build(deps): bump sha2 from 0.10.9 to 0.11.0 Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.10.9 to 0.11.0. - [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.10.9...sha2-v0.11.0) --- updated-dependencies: - dependency-name: sha2 dependency-version: 0.11.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix: adapt to sha2 0.11.0 API changes sha2 0.11.0 switched from GenericArray to hybrid_array::Array for hash output, which no longer implements LowerHex. Replace `format!("{:x}")` with explicit per-byte hex formatting. Made-with: Cursor * fix: replace write_all with update for sha2 0.11.0 compatibility sha2 0.11.0 removed the std::io::Write impl on hashers. Use Digest::update() instead of Write::write_all(). Made-with: Cursor --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <agent@cursor.com> |
||
|
Jojii
|
6fc6bcc5b3 |
Don't insert deferred points into sparse index (#8435)
* Don't insert deferred points into sparse index # Conflicts: # lib/segment/src/segment/tests.rs # lib/segment/src/segment_constructor/segment_constructor_base.rs # Conflicts: # lib/segment/src/segment_constructor/segment_constructor_base.rs * Clippy * Assert consistency of deferred_internal_id var * Return before SparseVector conversion in case of deferred point * Use debug_assert instead |