mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-31 07:00:56 -05:00
e1d901647e
* Add CachedReadFs: prefetch-backed read-only universal-io filesystem Snapshots the file listing at construction and serves opens from explicitly prefetched handles (take-once, shared across clones via Arc<Mutex>). A non-prefetched open falls back to a direct open on the inner filesystem, panicking in debug builds and warning in release. CachedFile is a transparent wrapper needed to satisfy the bidirectional UniversalReadFs<File = Self> pinning, following the ReadOnly pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serve read-only segment opens from CachedReadFs prefetch pool ReadOnlySegment::open builds a per-segment CachedReadFs: the files known in advance (version.info, segment.json) are scheduled before the listing snapshot is taken so their fetch overlaps the listing round-trip, then every remaining listed file is scheduled, running all fetches in parallel instead of serializing them inside component opens. Existence checks and format-detection probes are answered from the snapshot without touching the inner filesystem. Stored handles are taken out of the pool via the new CachedReadFs::take_file, which returns the raw inner file — component types stay over plain S, and CachedFile exists only transiently inside open-read-discard helpers (read_json_via etc.) through the trait impl. The read-only open path takes &CachedReadFs<S::Fs> concretely; storing wrappers gained from-file constructors (StoredBitSlice::from_file, UniversalHashMap::from_file, ReadOnly::from_file, gridstore Tracker::open_cached / Pages::open_cached, read_chunks_cached). Snapshot-less, CachedReadFs is a passthrough to the inner filesystem — used by reload paths, writable build paths that reuse the on-disk index opens, and tests, all of which keep their previous behavior. Components that retain a filesystem for later reloads store the raw inner backend (CachedReadFs::inner), never the stale snapshot. Also: local_list_files now recurses into subdirectories, matching the flat key-prefix semantics of object-store listings; the immutable id tracker probes its defining file via exists (free on the snapshot) instead of a probe-open that would consume the take-once handle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Group imports per nightly rustfmt Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix sparse search bench for open_ro over CachedReadFs CI clippy runs --all-targets; the bench target was missed by the --tests sweep. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Match listing prefixes by path component, not by string The recursive local_list_files compared whole path strings; on Windows a joined prefix mixes `/` and `\` (`shard\index/chunk_`) while walked entry paths use `\` throughout, so nothing ever matched (broke list_files_returns_paths_relative_to_shard_dir on Windows CI). Match the entry name at the prefix's final position against the prefix's final component instead, then walk matched directories exhaustively — same semantics, separator-agnostic. Apply the same component-based matching to the CachedReadFs snapshot filter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * do not cache everything * fmt * dont unwrap files_info * fix clippy * relax debug assertion for now * [AI] refactor into extension trait, relax Fs<->File requirement (#9725) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com>