* Benches: use SmallRng instead of ChaCha12-based generators
All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the
ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength
randomness, and several draw random values inside the timed closure, so
cipher work was included in the measurement itself.
Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW
graph cache and sparse index cache by RNG algorithm so stale caches built
from the old generator are not reused against newly generated vectors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Benches: replace free-function rand::random with local SmallRng
Addresses review: rand::random draws from the thread RNG (ChaCha12),
including inside the timed loop of the pq score benchmark.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [UIO] implement VectorIndexReadEnum::preopen
Schedule background prefetch of the HNSW graph (config, graph data,
links) and sparse index (config, inverted index, version, indices
tracker) files, wired into the segment's first_preopen for every dense
and sparse vector.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Adapt index preopen to caller-controlled populate
Now that CachedFs respects the caller's populate: prefetch Cached HNSW
links with a background populate instead of the open's blocking one;
populate the immutable-RAM sparse index data (read in full on open)
while the mmap variant stays cold; apply the low-memory ImmutableRam
downgrade in preopen_sparse since it now changes populate behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Decide sparse index preopen populate from the index type
All readable sparse index variants share the compressed-mmap on-disk
format, so the datatype dispatch in preopen_sparse selected nothing.
Replace the per-TInvertedIndex preopen_ro trait machinery with one
populate-parameterized preopen in the sparse crate: the effective
index type alone decides whether the index data is warmed
(immutable-RAM reads it in full) or parked cold (mmap reads lazily).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Derive sparse index preopen populate from the segment config
The segment config's sparse_vector_data already carries the
SparseIndexConfig, so preopen_sparse doesn't need to read the persisted
copy at all: it derives populate from the segment-side index type and
merely schedules the config file for open_sparse to consume — still a
single fetch, without threading the parsed config through open_via.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Resolve index preopen placement via memory_placement
The HNSW read-only open and preopen used the deprecated on_disk flag
directly; resolve the graph residency like the writable open instead —
memory parameter with on_disk fallback, clamped by low-memory mode,
including the pinned placement. The sparse index preopen likewise
derives its populate from the effective memory placement, so the
cached mmap index is prefetched warm.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add CachedReadFs: prefetch-backed read-only universal-io filesystem
Snapshots the file listing at construction and serves opens from
explicitly prefetched handles (take-once, shared across clones via
Arc<Mutex>). A non-prefetched open falls back to a direct open on the
inner filesystem, panicking in debug builds and warning in release.
CachedFile is a transparent wrapper needed to satisfy the bidirectional
UniversalReadFs<File = Self> pinning, following the ReadOnly pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Serve read-only segment opens from CachedReadFs prefetch pool
ReadOnlySegment::open builds a per-segment CachedReadFs: the files known
in advance (version.info, segment.json) are scheduled before the listing
snapshot is taken so their fetch overlaps the listing round-trip, then
every remaining listed file is scheduled, running all fetches in parallel
instead of serializing them inside component opens. Existence checks and
format-detection probes are answered from the snapshot without touching
the inner filesystem.
Stored handles are taken out of the pool via the new
CachedReadFs::take_file, which returns the raw inner file — component
types stay over plain S, and CachedFile exists only transiently inside
open-read-discard helpers (read_json_via etc.) through the trait impl.
The read-only open path takes &CachedReadFs<S::Fs> concretely; storing
wrappers gained from-file constructors (StoredBitSlice::from_file,
UniversalHashMap::from_file, ReadOnly::from_file, gridstore
Tracker::open_cached / Pages::open_cached, read_chunks_cached).
Snapshot-less, CachedReadFs is a passthrough to the inner filesystem —
used by reload paths, writable build paths that reuse the on-disk index
opens, and tests, all of which keep their previous behavior. Components
that retain a filesystem for later reloads store the raw inner backend
(CachedReadFs::inner), never the stale snapshot.
Also: local_list_files now recurses into subdirectories, matching the
flat key-prefix semantics of object-store listings; the immutable id
tracker probes its defining file via exists (free on the snapshot)
instead of a probe-open that would consume the take-once handle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Group imports per nightly rustfmt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix sparse search bench for open_ro over CachedReadFs
CI clippy runs --all-targets; the bench target was missed by the
--tests sweep.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Match listing prefixes by path component, not by string
The recursive local_list_files compared whole path strings; on Windows a
joined prefix mixes `/` and `\` (`shard\index/chunk_`) while walked
entry paths use `\` throughout, so nothing ever matched (broke
list_files_returns_paths_relative_to_shard_dir on Windows CI).
Match the entry name at the prefix's final position against the
prefix's final component instead, then walk matched directories
exhaustively — same semantics, separator-agnostic. Apply the same
component-based matching to the CachedReadFs snapshot filter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* do not cache everything
* fmt
* dont unwrap files_info
* fix clippy
* relax debug assertion for now
* [AI] refactor into extension trait, relax Fs<->File requirement (#9725)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Remove from_iter_instead_of_collect from workspace lints
The lint was removed from clippy (beta) and now triggers
renamed_and_removed_lints warnings in every crate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::chunks_exact_to_as_chunks
Replace chunks_exact with a constant chunk size by as_chunks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::needless_late_init
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::useless_borrows_in_formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::uninlined_format_args
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::for_kv_map
Iterate map values directly instead of discarding keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_large_err on QueueProxyShard::new_from_version
The Err variant intentionally hands the LocalShard back to the caller.
Same pattern as the existing allow on ForwardProxyShard::new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_unit_err on wait_for_consensus_commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: add open for read-only sparse vector index + enum sparse dispatcher
* fix: universal-IO loads for read-only sparse index open
* refactor: rename load_via/open_via to load_universal/open_universal
* refactor: drop StorageVersion::load in favor of load_universal
The regular-IO `load` duplicated `load_universal` over plain `File` IO.
Remove it and route all callers through `load_universal(&MmapFs, ..)`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(sparse): decouple inverted index from concrete storage; split segment_constructor_base (#9461)
Make the read-only index enum generic over storage `S` (no concrete MmapFile),
and remove construction callbacks from the sparse index open paths.
- InvertedIndex is now a pure read/search trait: `open`/`from_ram_index`/
`type Fs` moved off the trait to inherent methods on each concrete index
type, so construction no longer requires `S::Fs: Default`.
- SparseVectorIndex open split into a generic `plan` (load-vs-build decision +
RAM-index build) and generic `finish` (assembly); callers do the concrete
per-type construction, so no construction callbacks are needed.
- ReadOnlySparseVectorIndex::open takes the already-constructed inverted index
and caller-loaded config instead of a `load_inverted_index` callback.
- VectorIndexReadEnum is generic over `S: UniversalRead`; sparse mmap variants
hold `InvertedIndexCompressedMmap<_, S>` rather than a concrete `MmapFile`.
- Split the 1182-line segment_constructor_base.rs into a module (paths,
vector_storage, payload_storage, id_tracker, vector_index,
sparse_vector_index, create_segment, segment, legacy_state); the sparse
dispatcher's match arms collapse into three per-family helpers.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat: LiveReload (no-op) dispatch for read-only vector index enum (#9436)
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Migrate InvertedIndexCompressedMmap to UniversalRead
* Rehaul search_scratch.rs (was scores_memory_pool.rs)
- Use `typed_arena::Arena` instead of `bumpalo::Bump`.
Reason: bump don't drop.
Drawback: `Arena::new()` is not free, and `Arena::clear()` doesn't
exist; but this commit has workarounds.
- Use names that make more sense.
* Misc fixes
* InvertedIndexCompressedMmap: explicit S type parameter
* dataset: add features to fs-err
Reason: fix `cargo check --all-targets -p dataset`. Feature unification
pulls `fs-err/debug` + `fs-err/tokio` without `fs-err/debug_tokio`.
* dataset: run `cargo metadata` to get target dir
Reason: avoid re-downloading in separate worktrees. The old
`cargo locate-project` ignores custom target dirs.
* sparse/benches/search: cache vectors/indices
* use pageout to clear mmap cache
* Also clear cache of deleted flags in mmap dense vector storage
* Add reference to madvise man pages for probe logic
* use deconstruct
---------
Co-authored-by: timvisee <tim@visee.me>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
* Don't insert deferred points into sparse index
# Conflicts:
# lib/segment/src/segment/tests.rs
# lib/segment/src/segment_constructor/segment_constructor_base.rs
# Conflicts:
# lib/segment/src/segment_constructor/segment_constructor_base.rs
* Clippy
* Assert consistency of deferred_internal_id var
* Return before SparseVector conversion in case of deferred point
* Use debug_assert instead
* Use array_windows when statically sized windows were used
* Bump MSRV to 1.94
* fix in gpu code
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Implement `Optional<T>` type
Define a type with the same presumed layout as `Option<T>`, but with defined behavior.
* Make `transmute_*` functions unsafe
The functions `memory::mmap_ops::transmute_*` are inherently unsafe, but
are not marked as are. Their usage is documented, but it is not always clear
if the code is correct.
* Add `CsrHeader` to resolve another unsoundness
Tuples have no defined layout.
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* wip: implement explicit populate and clear_cache functions for all components
* fmt
* implement clear and populate for vector storages
* fmt
* implement clear and populate for payload storage
* wip: implement explicit populate and clear_cache functions payload indexes
* implement explicit populate and clear_cache functions payload indexes
* fix clippy on CI
* only compile posix_fadvise on linux
* only compile posix_fadvise on linux
* implement explicit populate and clear_cache functions for quantized vectors
* fmt
* remove post-load prefault
* fix typo
* implement is-on-disk for payload indexes, implement clear on drop for segment, implement clear after segment build
* fmt
* also evict quantized vectors after optimization
* re-use and replace advise_dontneed
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
