`PostingList::upsert` ends in `propagate_max_next_weight_to_the_left`, whose doc
comment states "If an entry has a weight larger than `max_next_weight`, the
propagation stops". It never stopped — the loop always walked the entire prefix.
Record ids ascend during an upload, so every insert lands at the end, walks the
whole list, and growing a posting list to length L costs O(L²). On the NeurIPS
2023 sparse base set (MS MARCO / SPLADE) the hottest dimension appears in ~66% of
documents, so at 1M points its posting list holds 660k elements and every new
point rewrites all of them.
Two changes to `PostingList`:
* restore the early exit — every entry satisfies the same recurrence the loop
walks, `max_next_weight[i] = max(max_next_weight[i + 1], weight[i + 1])`, so
once an entry already holds the value being written, every entry to its left is
correct too;
* add an append fast path — when the incoming record id is past the last stored
id, push directly instead of binary searching a list that can hold millions of
elements.
Neither changes what is stored. An index grown by `upsert` stays identical to one
built by `InvertedIndexBuilder`, `max_next_weight` included, which is what a
segment reload depends on; searches return identical results.
Building an `InvertedIndexRam` one point at a time, SPLADE vectors:
points before after
100k 2,216/s 124,016/s
1M 250/s 114,422/s
Uploading 1M of those points to a single node: 72.2s -> 23.7s with default
collection settings, and 839.8s -> 29.5s with `indexing_threshold: 0`, which
stops segments converting and is the documented way to speed up a bulk load.
Claude-Session: https://claude.ai/code/session_01HXjykBsMZaRNrP17o2fuEJ
Co-authored-by: Andrey Vasnetsov <andrey@qdrant.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [CachedFs] new `schedule` and `wait_all` primitives
* [AppendableIdTracker] don't reopen if just opened
* eager NotFound in `schedule_open`
* add traces for async reads
* finish `preopen`/`preload` with `wait_all`
* lock all segments in parallel for `live_reload`
* LIST before everything
to do: we don't have whole-fetch in async mode. to prevent sequential
`len`, we won't overlap static files with LIST.
* `wait_all` returns nothing
CompressedPostingBuilder stored the running byte offset with 'data_size as u32', which silently truncates if a posting list's compressed id-data ever exceeds u32::MAX bytes, producing a wrong offset and corrupt reads. Use u32::try_from(..).expect(..) so it fails loudly, matching the guarded sibling in the posting_list crate.
* Gridstore: introduce storage operating mode in config
Add a mode field to the gridstore config, selecting between the dynamic
mode (current behavior, the default) and the upcoming serverless mode.
The mode is specified through StorageOptions on creation, persisted in
config.json, and read back first when opening so the correct variant can
be selected automatically. Configs written before this field existed
deserialize as dynamic.
For now, selecting the serverless mode returns an error; the variant
itself is added in follow-up commits.
* Gridstore: move dynamic implementation into dedicated module
Mechanical move of the current Gridstore implementation into
gridstore/dynamic.rs as DynamicGridstore. The public Gridstore struct
becomes a thin wrapper holding a mode variant enum, propagating every
call into the selected variant. For now the enum only has the dynamic
variant; the serverless variant is added in follow-up commits.
No logic changes to the dynamic implementation itself: only visibility,
the config parameter now passed into open (the wrapper reads it first to
select the mode), and open_or_create staying on the wrapper.
* Gridstore: add serverless tracker
Add the append-only mapping tracker for the serverless storage mode.
The tracker file is a plain array of 16-byte mapping entries without any
header: the number of mappings is defined by the exact file length, and
the entry index is the point offset. The file starts empty and only ever
grows by appending, existing bytes are never rewritten. Mappings must be
set in monotonically increasing point offset order; skipped offsets are
backfilled as zeroed entries which decode as None.
New mappings are buffered in memory and appended with a single write per
flush. A flush with a stale target is a no-op so bytes are never written
twice. A torn trailing entry (file length not a multiple of the entry
size) is ignored when reading and truncated away when opening writable.
Unlike the dynamic tracker, the file is read and written directly with
positional file IO instead of memory mapping, as serverless environments
do not handle memory mapped files well.
* Gridstore: add serverless storage variant
Add the append-only gridstore variant for serverless deployments, which
restrict IO to appending to files: existing bytes can never be
rewritten, and IO is expensive so as few files as possible are used.
The variant stores all value data in a single page file next to the
serverless tracker and the storage config, three files in total. Both
data files start empty and only ever grow by appending; there is no
preallocation, no used-block bitmask and no gap/region bookkeeping.
Values are appended at put time at the next block aligned offset, with
the zero padding included in the write so it lands exactly at the end
of the file. Mappings are buffered and appended to the tracker with a
single write per flush, after the page file is synced, so a mapping on
disk never points at data that is not durable.
Values cannot be updated or deleted, and must be put at monotonically
increasing point offsets; violations are rejected before any data is
written. Files are read and written directly, never memory mapped.
The mode is selected through StorageOptions on creation and picked up
automatically from the persisted config when opening.
* Gridstore: serverless support in reader and view
Extend the read-only GridstoreReader and the GridstoreView with the
serverless mode, keeping both public types unchanged: like the writable
Gridstore they now hold a mode variant internally, selected
automatically from the persisted config when opening.
The serverless reader holds the tracker and page directly and reads the
files positionally, without memory mapping. A live reload re-reads the
mapping count from the exact tracker file length (there is no size
header), ignoring a torn trailing entry, and never truncates as it is
read-only. Value reads always go directly to the file, so newly
appended data is readable without remapping anything.
* Gridstore: document storage operating modes
* Gridstore: review fixes for the serverless mode
Hardening and cleanup from a review pass over the new serverless
storage variant:
- Batch the reader side iteration like the writer already did, instead
of materializing tracker mappings for the full range in one go, which
could transiently allocate gigabytes on large storages.
- Recover the append cursors when a positional write fails partway:
truncate the file back to the tracked length so a retried append or
flush never rewrites bytes that already landed in the file.
- Validate page addressability before appending value data, a rejected
put must not grow the page file.
- Cross-check tracker and page consistency when opening: mappings that
reference value data past the end of the page file (e.g. after a
partial copy or restore) now fail fast instead of surfacing as
opaque read errors per point.
- Reject value pointers into any page other than page 0 on the
serverless read path with PageNotFound, matching the dynamic mode
contract, instead of silently reading from a wrong location.
- Refresh the reported storage size on reader live reload even when no
new mappings were flushed, unflushed value data may have grown the
page file already.
- Validate configs read from disk: a corrupt config with zero sized
blocks, pages or regions is now rejected when opening instead of
panicking on a division by zero later.
- Classify rejected serverless puts as UnsupportedOperation, consistent
with rejected deletes, so they don't surface as user-facing
validation errors at the segment level.
- Deduplicate the compression dispatch into Compression::compress and
Compression::decompress, and the serverless file create/open patterns
into shared direct IO helpers, so the two modes and files can't
silently drift apart.
* Gridstore: cover both operating modes in mode-agnostic tests
Parameterize the gridstore tests that exercise mode-agnostic behavior
over both the dynamic and serverless mode with rstest, using a
single and bulk put/get roundtrips, storage files, basic persistence,
corrupt config rejection, batched read congruence, reader live reload,
and the different block sizes.
Mode specific expectations branch inside the tests: expected file
names, storage size semantics (whole blocks vs exactly packed bytes),
value pointer layout (page spill over vs a single packed page), and
gaps (created by deletes in dynamic mode, by skipped puts in serverless
mode). Dynamic-only internals assertions are kept behind a mode check.
Tests around updates, deletes, page spanning, block reuse and other
dynamic-only behavior intentionally stay dynamic; the serverless
specific format invariants remain covered by the dedicated serverless
tests.
* Gridstore: port serverless specific tests from sibling branch
Source the serverless specific test cases that the
serverless-gridstore-updates branch added, adapted to the dedicated
variant implemented here (distinct file names, headerless tracker
with 16 byte entries, a single packed page without trailing padding,
and rejected re-puts):
- writes only ever append: tracker and page files only grow and
previously written bytes stay byte-for-byte untouched
- new mappings land exactly at the end of the tracker file, which
always covers the exact number of mappings
- mapping gaps are zero-padded on disk and survive reopening
- values are packed back to back at block aligned offsets, the page
file ends exactly at the last value
- serverless mode never creates nor reports block flag files
- a flusher persists exactly the mappings that existed at its
creation, later puts stay pending
- a config claiming the wrong mode fails loudly in both directions
instead of loading the incompatible file format of the other mode
Tests around their mode switching, page spanning and tolerated deletes
don't apply to this design and are intentionally not ported.
* Gridstore: test serverless production risk scenarios
Add tests for the operational aspects that matter before serverless
mode goes to production, each covering a scenario that wasn't
evaluated yet:
- Replayed puts of already persisted offsets (a WAL redo after a
crash where the flush completed but was never acknowledged) are
rejected without appending anything, and max_point_offset is the
exact offset a replay must resume at.
- The accepted crash case of a tracker file extended with zeroed
bytes: the entries count as permanent None mappings, can never be
put again, and the storage stays consistent and writable past them.
- The read-only reader never modifies the files: opening over a torn
tracker tail, reading, iterating and live reloading leave both
files byte-for-byte untouched.
- A multi-round put/flush/reopen cycle always exposes exactly the
flushed prefix, with the mapping count matching the exact tracker
file length and unflushed offsets reusable.
- An append beyond the maximum addressable block offset is rejected
before writing anything, keeping retried puts from growing the page
file unboundedly.
* Gridstore: rename serverless mode to append-only, split into module
Rename the mode after its defining characteristic instead of its
deployment target: files only ever grow, existing bytes are never
rewritten. Renames Mode::Serverless to Mode::AppendOnly (persisted as
"mode": "append_only") and the on-disk file names to
append_only_tracker.dat and append_only_page_0.dat. The serverless
deployment motivation stays in the documentation.
Also split the single 2300 line serverless.rs into an append_only
module with dedicated files for the storage, page, view, reader and
tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use universal IO in Gridstore
* Include upstream preopen logic in new Gridstore variant
* Gridstore: buffer append-only value writes until flush
In append-only mode, put previously wrote the value data to the page
file right away, one write operation per put, while mappings were
already buffered and batch persisted on flush. Buffer value writes the
same way: both the value and its mapping now only land on disk once a
flush cycle executes.
This batches all new value data into a single write operation per
flush, which is significantly more efficient on S3 based storage where
every write is a costly operation. A flush now performs exactly two
writes: one appending all buffered value data to the page file, one
appending all pending mappings to the tracker file, in that order, so
a mapping on disk never points at value data that is not durable.
The page mirrors the tracker's pending mechanism: an in-memory buffer
that is byte for byte the next append (zero padding between block
aligned values included), a watermark captured at flusher creation so
puts made during a flush stay buffered, a stale-flush no-op guard so
appended bytes are never written twice, and truncate-back recovery on
failed writes. Reads transparently serve buffered values from memory.
As a side effect, a crash between flushes now leaves nothing on disk
at all, where the write-through approach left orphaned value bytes in
the page file. The buffered data is held in memory until the next
flush, bounded by the flush cadence.
Universal IO filesystem handles are now required to be Send + Sync, so
the flusher closure can carry one to grow the page file at flush time;
all existing backends already satisfied this.
* Gridstore: rename inner DynamicGridstore to Gridstore
The dynamic variant keeps the Gridstore name; the outer dispatching type
will be renamed to Blobstore in a follow-up. Until then the inner type is
referred to as dynamic::Gridstore to distinguish it from the outer type.
* Gridstore: rename append-only variant to Arenastore
The append-only variant stores all value data in a single ever-growing
page, allocating space by appending, hence: arena store.
* Gridstore: rename outer storage type to Blobstore
The outer type dispatching between the two storage variants is now called
Blobstore, being more generic than Gridstore. This frees up the Gridstore
name, which now exclusively refers to the dynamic mode variant, next to
Arenastore for the append-only variant. Storage components keep using the
outer type, so they now use Blobstore.
The gridstore crate name, GridstoreError, and the persisted names
(config.json mode, payload config storage_type) are unchanged.
* Gridstore: split Gridstore and Arenastore into dedicated modules
The outer module is now blobstore, matching the Blobstore type it
defines. The two storage variants each get their own submodule: the
dynamic Gridstore moves from dynamic.rs into gridstore/ with its reader
and view extracted from the shared files, mirroring the arenastore/
module (previously append_only/) which already had this layout.
* Rename gridstore crate to blobstore
The crate is named after the outer Blobstore storage type it provides.
The gridstore name lives on in the dynamic mode variant. GridstoreError
and the persisted names (config.json mode, payload config storage_type)
are unchanged.
* Arenastore: pack values back to back across multiple pages
Drop the block alignment from the append-only mode: values are packed
byte to byte, without blocks, and the tracker offset is now a plain byte
offset within the page. Blocks and regions are dynamic mode concepts;
their page size constraints no longer apply to append-only configs.
Bring back support for multiple pages. Once appending a value would
grow the current page beyond the configured page size, a new page is
started, bounding the size of and the number of appends to each file:
object stores like S3 Express limit the number of appends per object.
A value larger than the page size gets a page of its own; values never
span pages.
A rollover creates the new, empty page file at put time; the value data
itself stays buffered until the next flush, which appends to each
touched page with a single write, using per-page watermarks captured at
flusher creation. The reader scans for consecutively numbered page
files when opening, validates the most recent mappings against them,
and adopts pages created since on a live reload.
* Blobstore: rename dynamic mode to mutable
Rename Mode::Dynamic to Mode::Mutable, and the persisted config value
with it: config.json now writes "mode": "mutable". There is no
compatibility alias for "dynamic", released versions never wrote the
mode field (a missing field still defaults to mutable), only unreleased
storages did.
The Gridstore type and module names for the mutable variant are
unchanged.
* Fix Edge compilation due to package rename
* Review remarks
* Extract Gridstore preopen into module
* Rename Arenastore files
* Use universal IO for append operations
* Rename GridstoreError to BlobstoreError
The error type belongs to the Blobstore crate and is shared by both the
Gridstore and Arenastore variants, so it follows the crate naming. Also
update the user-facing error messages that referred to the old name.
* Split config into per-variant types
* Rename Arenastore to Logstore
Rename the Arenastore type to Logstore, including the reader, view,
config, module and variant names. The storage file names follow:
log_page_{n}.dat and log_tracker.dat. The persisted mode tag stays
"append_only".
* Move bitmask module into the Gridstore variant
The bitmask tracks free blocks, which only exists in the mutable mode.
Move the module from the crate root into the Gridstore variant that
owns it. It stays re-exported at the crate root because the bitmask
benchmark needs a public path.
* Move pages module into the Gridstore variant
Like the bitmask, the block based pages module is only used by the
mutable mode. Move it from the crate root into the Gridstore variant
that owns it. The Logstore variant has its own page implementation.
* Use universal IO for every Logstore operation
Replace the direct_io module with universal IO in the append-only
tracker, making the whole Logstore go through a universal IO backend
bounded by UniversalRead and UniversalAppend:
- The tracker is generic over the backend now. Reads go through
UniversalRead with the caller's access pattern, flushes land as one
atomic append with the same offset compare-and-swap recovery as the
pages: a retried append after a lost acknowledgement is adopted
instead of appended twice. A torn trailing entry is still truncated
away on writable open, through a fresh handle since shrinking is not
supported through an open one.
- The reader now schedules a prefetch for the tracker file too, it no
longer bypasses the backend.
- The config write, clear and wipe use the backend file operations
instead of local filesystem calls, matching the Gridstore variant.
* Batch reads in Logstore read_values
Apply the same batching logic as the Gridstore variant: resolve all
mappings first, then fetch the value data, both through the backend's
read pipeline so async backends can serve the reads in parallel.
The tracker gains a batched lookup mirroring the mutable tracker's
iter, serving pending mappings and out of range point offsets directly
from memory. The pages gain a batched value read; unflushed values are
served from the in-memory buffers, and since values never span pages
each value is a single read without reassembly.
Like in the Gridstore variant, the callback may now be invoked in a
different order than the requested point offsets.
* Better describe logstore live reload ordering
* use enum for options, swap `*Options`<->`*Config` naming
* don't wrap enum in struct
* ditch unused `StorageConfig`, make deserialization more ergonomic
* rename `*Options`->`*Config`
* make `preopen` non-blocking
* fixup! ditch unused `StorageConfig`, make deserialization more ergonomic
* fixup! use enum for options, swap `*Options`<->`*Config` naming
* fixup! don't wrap enum in struct
* fix rebase
* use `populate` param in Logstore
* test: failing repro of stale page after live reload across rollover
A reader that live-reloads between a page rollover and the following
flush adopts the new, still empty page. The previous page is then no
longer the last one and is never reloaded again, so the tail that the
next flush appends to it stays invisible to the reader forever:
value pointer at byte 100 with length 100 is out of range
AppendOnlyPages::live_reload only reloads the last held page, assuming
earlier pages never change once a newer page exists. But the rollover
creates the new page file eagerly at put time, while the previous
page's buffered tail only lands at the next flush (see
test_rollover_writes_no_value_data_before_flush), so a page can keep
growing on disk after its successor exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: reload all pages that grew
* use Fs in `open_or_create`
* fix: publish tracker mappings only after the pages reload
`AppendOnlyTracker::live_reload` observed the mapping count and made it
visible in one step, before `LogstoreReader::live_reload` reloaded the
pages. Every failure path in the page reload -- `list_files`, reopening a
grown page, opening an adopted one, the truncation check -- therefore left
the reader with mappings referencing value data it never loaded, so reads
in the new offset range fail until a later reload happens to succeed. The
edge refresh loop keeps a segment whose reload failed, expecting it to keep
serving its pre-refresh state, which it then does not.
Split observing from publishing: `reload_count` refreshes the handle and
returns the count as a `PendingReload` token, `commit_reload` publishes it.
The reader still observes the tracker first, as the writer persists pages
before the mappings referencing them, but only commits once the pages are
loaded. Reopening without committing is harmless: reads stay bounded by the
unchanged count, and the bytes below it never change.
A partial failure inside the page reload needs no unwinding, pages running
ahead of the tracker is the safe direction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: batch the value reads in Logstore iteration
`LogstoreView::iter_range`, the path behind `Logstore::iter` and
`LogstoreReader::iter`, fetched the mappings for the whole range with a
single read but then read the values themselves one at a time, serially.
Gridstore routes its `iter` through `read_values` and pipelines both stages,
so a full scan of an append-only storage was the one read path without
batching -- one blocking round trip per value on the object store backends
this variant exists for. It is reached by payload storage iteration and by
the payload index build, which scans every payload.
Feed the pointers into `read_batch_values` instead, keeping the single
contiguous tracker read, which is better than the per-offset pipeline
scheduling Gridstore does on that side.
Values are now delivered through the read pipeline, so the callback may be
invoked out of order, as it already could be for Gridstore's `iter` and for
`read_values` in both variants. Both segment callers are order independent.
Tests that happened to rely on the mmap backend completing reads in
scheduling order now sort before comparing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: don't run the failed-page-reload test on Windows
The test shrinks a page file out of band to make the page reload fail, but
Windows refuses to resize a file while the reader holds it mapped, which it
does by construction here: "the requested operation cannot be performed on a
file with a user-mapped section open". The panic is on the injection itself,
the code under test never runs.
There is no portable injection. Truncating a page the reader holds is what
the check under test detects, so the mapping cannot be avoided; failing the
adopted page open instead needs a listed but unopenable file, and
`local_list_files` descends into matching directories rather than listing
them; failing the directory listing needs the storage directory removed,
which Windows also refuses while pages are mapped.
The storage itself is fine on Windows, its append path grows mapped pages
there and every other Logstore test passes. The logic under test is platform
independent and stays covered elsewhere, with the tracker half of the
guarantee pinned by `test_live_reload`, which runs on every target.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* [UIO] implement VectorIndexReadEnum::preopen
Schedule background prefetch of the HNSW graph (config, graph data,
links) and sparse index (config, inverted index, version, indices
tracker) files, wired into the segment's first_preopen for every dense
and sparse vector.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Adapt index preopen to caller-controlled populate
Now that CachedFs respects the caller's populate: prefetch Cached HNSW
links with a background populate instead of the open's blocking one;
populate the immutable-RAM sparse index data (read in full on open)
while the mmap variant stays cold; apply the low-memory ImmutableRam
downgrade in preopen_sparse since it now changes populate behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Decide sparse index preopen populate from the index type
All readable sparse index variants share the compressed-mmap on-disk
format, so the datatype dispatch in preopen_sparse selected nothing.
Replace the per-TInvertedIndex preopen_ro trait machinery with one
populate-parameterized preopen in the sparse crate: the effective
index type alone decides whether the index data is warmed
(immutable-RAM reads it in full) or parked cold (mmap reads lazily).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Derive sparse index preopen populate from the segment config
The segment config's sparse_vector_data already carries the
SparseIndexConfig, so preopen_sparse doesn't need to read the persisted
copy at all: it derives populate from the segment-side index type and
merely schedules the config file for open_sparse to consume — still a
single fetch, without threading the parsed config through open_via.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Resolve index preopen placement via memory_placement
The HNSW read-only open and preopen used the deprecated on_disk flag
directly; resolve the graph residency like the writable open instead —
memory parameter with on_disk fallback, clamped by low-memory mode,
including the pinned placement. The sparse index preopen likewise
derives its populate from the effective memory placement, so the
cached mmap index is prefetched warm.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add CachedReadFs: prefetch-backed read-only universal-io filesystem
Snapshots the file listing at construction and serves opens from
explicitly prefetched handles (take-once, shared across clones via
Arc<Mutex>). A non-prefetched open falls back to a direct open on the
inner filesystem, panicking in debug builds and warning in release.
CachedFile is a transparent wrapper needed to satisfy the bidirectional
UniversalReadFs<File = Self> pinning, following the ReadOnly pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Serve read-only segment opens from CachedReadFs prefetch pool
ReadOnlySegment::open builds a per-segment CachedReadFs: the files known
in advance (version.info, segment.json) are scheduled before the listing
snapshot is taken so their fetch overlaps the listing round-trip, then
every remaining listed file is scheduled, running all fetches in parallel
instead of serializing them inside component opens. Existence checks and
format-detection probes are answered from the snapshot without touching
the inner filesystem.
Stored handles are taken out of the pool via the new
CachedReadFs::take_file, which returns the raw inner file — component
types stay over plain S, and CachedFile exists only transiently inside
open-read-discard helpers (read_json_via etc.) through the trait impl.
The read-only open path takes &CachedReadFs<S::Fs> concretely; storing
wrappers gained from-file constructors (StoredBitSlice::from_file,
UniversalHashMap::from_file, ReadOnly::from_file, gridstore
Tracker::open_cached / Pages::open_cached, read_chunks_cached).
Snapshot-less, CachedReadFs is a passthrough to the inner filesystem —
used by reload paths, writable build paths that reuse the on-disk index
opens, and tests, all of which keep their previous behavior. Components
that retain a filesystem for later reloads store the raw inner backend
(CachedReadFs::inner), never the stale snapshot.
Also: local_list_files now recurses into subdirectories, matching the
flat key-prefix semantics of object-store listings; the immutable id
tracker probes its defining file via exists (free on the snapshot)
instead of a probe-open that would consume the take-once handle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Group imports per nightly rustfmt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix sparse search bench for open_ro over CachedReadFs
CI clippy runs --all-targets; the bench target was missed by the
--tests sweep.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Match listing prefixes by path component, not by string
The recursive local_list_files compared whole path strings; on Windows a
joined prefix mixes `/` and `\` (`shard\index/chunk_`) while walked
entry paths use `\` throughout, so nothing ever matched (broke
list_files_returns_paths_relative_to_shard_dir on Windows CI).
Match the entry name at the prefix's final position against the
prefix's final component instead, then walk matched directories
exhaustively — same semantics, separator-agnostic. Apply the same
component-based matching to the CachedReadFs snapshot filter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* do not cache everything
* fmt
* dont unwrap files_info
* fix clippy
* relax debug assertion for now
* [AI] refactor into extension trait, relax Fs<->File requirement (#9725)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Remove from_iter_instead_of_collect from workspace lints
The lint was removed from clippy (beta) and now triggers
renamed_and_removed_lints warnings in every crate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::chunks_exact_to_as_chunks
Replace chunks_exact with a constant chunk size by as_chunks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::needless_late_init
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::useless_borrows_in_formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::uninlined_format_args
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::for_kv_map
Iterate map values directly instead of discarding keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_large_err on QueueProxyShard::new_from_version
The Err variant intentionally hands the LocalShard back to the caller.
Same pattern as the existing allow on ForwardProxyShard::new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_unit_err on wait_for_consensus_commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: add open for read-only sparse vector index + enum sparse dispatcher
* fix: universal-IO loads for read-only sparse index open
* refactor: rename load_via/open_via to load_universal/open_universal
* refactor: drop StorageVersion::load in favor of load_universal
The regular-IO `load` duplicated `load_universal` over plain `File` IO.
Remove it and route all callers through `load_universal(&MmapFs, ..)`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(sparse): decouple inverted index from concrete storage; split segment_constructor_base (#9461)
Make the read-only index enum generic over storage `S` (no concrete MmapFile),
and remove construction callbacks from the sparse index open paths.
- InvertedIndex is now a pure read/search trait: `open`/`from_ram_index`/
`type Fs` moved off the trait to inherent methods on each concrete index
type, so construction no longer requires `S::Fs: Default`.
- SparseVectorIndex open split into a generic `plan` (load-vs-build decision +
RAM-index build) and generic `finish` (assembly); callers do the concrete
per-type construction, so no construction callbacks are needed.
- ReadOnlySparseVectorIndex::open takes the already-constructed inverted index
and caller-loaded config instead of a `load_inverted_index` callback.
- VectorIndexReadEnum is generic over `S: UniversalRead`; sparse mmap variants
hold `InvertedIndexCompressedMmap<_, S>` rather than a concrete `MmapFile`.
- Split the 1182-line segment_constructor_base.rs into a module (paths,
vector_storage, payload_storage, id_tracker, vector_index,
sparse_vector_index, create_segment, segment, legacy_state); the sparse
dispatcher's match arms collapse into three per-family helpers.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat: LiveReload (no-op) dispatch for read-only vector index enum (#9436)
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Migrate InvertedIndexCompressedMmap to UniversalRead
* Rehaul search_scratch.rs (was scores_memory_pool.rs)
- Use `typed_arena::Arena` instead of `bumpalo::Bump`.
Reason: bump don't drop.
Drawback: `Arena::new()` is not free, and `Arena::clear()` doesn't
exist; but this commit has workarounds.
- Use names that make more sense.
* Misc fixes
* InvertedIndexCompressedMmap: explicit S type parameter
* dataset: add features to fs-err
Reason: fix `cargo check --all-targets -p dataset`. Feature unification
pulls `fs-err/debug` + `fs-err/tokio` without `fs-err/debug_tokio`.
* dataset: run `cargo metadata` to get target dir
Reason: avoid re-downloading in separate worktrees. The old
`cargo locate-project` ignores custom target dirs.
* sparse/benches/search: cache vectors/indices
* use pageout to clear mmap cache
* Also clear cache of deleted flags in mmap dense vector storage
* Add reference to madvise man pages for probe logic
* use deconstruct
---------
Co-authored-by: timvisee <tim@visee.me>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
* Don't insert deferred points into sparse index
# Conflicts:
# lib/segment/src/segment/tests.rs
# lib/segment/src/segment_constructor/segment_constructor_base.rs
# Conflicts:
# lib/segment/src/segment_constructor/segment_constructor_base.rs
* Clippy
* Assert consistency of deferred_internal_id var
* Return before SparseVector conversion in case of deferred point
* Use debug_assert instead
* Use array_windows when statically sized windows were used
* Bump MSRV to 1.94
* fix in gpu code
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Implement `Optional<T>` type
Define a type with the same presumed layout as `Option<T>`, but with defined behavior.
* Make `transmute_*` functions unsafe
The functions `memory::mmap_ops::transmute_*` are inherently unsafe, but
are not marked as are. Their usage is documented, but it is not always clear
if the code is correct.
* Add `CsrHeader` to resolve another unsoundness
Tuples have no defined layout.
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* wip: implement explicit populate and clear_cache functions for all components
* fmt
* implement clear and populate for vector storages
* fmt
* implement clear and populate for payload storage
* wip: implement explicit populate and clear_cache functions payload indexes
* implement explicit populate and clear_cache functions payload indexes
* fix clippy on CI
* only compile posix_fadvise on linux
* only compile posix_fadvise on linux
* implement explicit populate and clear_cache functions for quantized vectors
* fmt
* remove post-load prefault
* fix typo
* implement is-on-disk for payload indexes, implement clear on drop for segment, implement clear after segment build
* fmt
* also evict quantized vectors after optimization
* re-use and replace advise_dontneed
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
