* bytemuck::Pod already implies 'static
* no supertraits
* regions gaps with TypedStorage
* StoredBitSlice with TypedStorage
* bytemuck for PostingsHeader
* bytemuck for TrackerHeader
* bytemuck for MmapRange
* bytemuck for stored_point_to_values::Header
* Move element type from UniversalRead trait to method generics
Lifts the `T` parameter off `trait UniversalRead<T>` (and the matching
`UniversalWrite<T>`) and onto the read/write methods themselves. The
`ReadPipeline` associated type becomes a GAT over `T`. With per-method
generics, callers that need to read several element types from one storage
just write `S: UniversalRead` instead of stacking
`UniversalRead<u8> + UniversalRead<Counts> + ...`.
Removes the workarounds the old shape required:
- `TypedStorage<S, T>` newtype (sole purpose was disambiguating multi-bounds)
- `UniversalReadFamily` HKT shim
- `StoredGeoMapIndexStorage` four-bound trait alias
- `CachedSlice<T>` is now non-generic; `T` moves to `get_range`/`len`
No runtime behavior change: alignment in `IoUringRuntime` and `CachedSlice`
is preserved because `T` is still known at each call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restore TypedStorage as a typed-access fail-safe
Reintroduces `TypedStorage<S, T>` as a transparent wrapper around
`UniversalRead`/`UniversalWrite` storage that fixes the element type to
`T`. With per-method generics on the underlying traits, callers can
otherwise read or write any `T` from the same handle; this wrapper
binds it at the type level so accidental cross-type access fails to
compile.
The wrapper exposes inherent typed methods (`read::<P>`, `read_iter`,
`write`, `len`, …) that delegate to the inner storage with `T` fixed.
It does not implement `UniversalRead`/`UniversalWrite` itself — those
are intentionally avoided to prevent the typed binding from being
bypassed via the generic trait methods.
Restores the wrapping at the previous call sites: `StoredStruct`'s
inner storage, `ImmutableIdTracker`'s version mmap, the geo and
numeric index storages, the chunked-vectors chunks, and the
immutable dense vector storage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Silence clippy::len_without_is_empty on TypedStorage
`TypedStorage::len` returns `Result<u64>` (a fallible byte length from
the underlying storage), so an `is_empty` companion would also be
fallible and offer nothing over `len()? == 0`. Suppress the lint at
the impl block, matching how the underlying `UniversalRead::len` is
already exempted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fmt
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Cleanup lifetimes and generic type parameters
- Rename read pipeline lifetime from `'a` into `'file`
- Use explicit `where` clauses everywhere
* Cleanup
* fixup! Cleanup lifetimes and generic type parameters
Drop the bespoke 4-arm \`NumericRangeIter\` enum introduced in the prior
commit. The existing \`common::either_variant::EitherVariant\` is the
same shape and already has \`Iterator\` plus all the standard adapter
specializations.
Adds a \`DoubleEndedIterator\` impl to \`EitherVariant\` (\`next_back\`,
\`nth_back\`, \`rfold\`, \`rfind\`) so it can be used in this position.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: reproduce missed-wakeup race in SaveOnDisk::wait_for
Adds test_wait_for_no_missed_wakeup plus a #[cfg(test)]-only
test_set_pre_park_sleep_ms hook that injects a sleep inside wait_for
between releasing the read guard on data and parking on the condvar.
This widens the race window from nanoseconds to ~100ms so the writer
thread's notify_all reliably fires *before* the waiter parks, exposing
the bug as a hard failure (notification lost, wait_for hits timeout).
This commit only adds the test and the instrumentation; the bug is
still present, so the new test fails.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: prevent missed wakeup in SaveOnDisk::wait_for
write and write_optional were calling change_notification.notify_all
without holding notification_lock. A waiter that had released its read
guard on data but had not yet parked on the condvar would miss the
notification and wait the full timeout — even though the condition was
already true.
Acquire notification_lock around notify_all (in a new notify_change
helper). The waiter holds notification_lock across the release-and-park
gap, so notify_change blocks until the waiter has actually parked,
guaranteeing delivery.
This matches the standard parking_lot Condvar pattern where the mutex
that protects the predicate is held while signalling. The new
test_wait_for_no_missed_wakeup test (added in the previous commit) now
passes; manifests in production as the flaky
test_fix_reshard_down_without_shard_key consensus test, where
wait_for_shard_key_activation timed out waiting on a replica state
that had already become Active.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: remove missed-wakeup race instrumentation
Removes the #[cfg(test)] test_pre_park_sleep_ms hook and the
test_wait_for_no_missed_wakeup test that depended on it. The
instrumentation existed only to deterministically reproduce the
missed-wakeup race in the unfixed wait_for; with the fix in place
(notify_change holds notification_lock around notify_all) the race is
closed and the hook has no remaining purpose.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* UniversalReadPipeline
* Performance: cache pointers in MmapFile
The previous commit removed `MmapFile::read_batch` method override. So,
`MmapFile` now re-uses the default `UniversalRead::read_batch` impl,
which is implemented using `UniversalReadPipeline` interface.
Unsurprisingly, it caused a slowdown in the benchmarks, particularly
this one:
cargo bench -p common --bench universal_io -- mmap/8bytes/read_batch_full
This commit reclaims the performance back.
* remove unfulfilled lint
* Review suggestions
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Immutable storage for mmap numeric index
Do not use a deleted mmap-ed storage, instead, reconstruct a bitmask
from the index and segment-level deleted mask. It effectively makes
the mmap numeric index immutable, and immutable numeric index too as
it delegates the storage to mmap numeric index.
* Index reload tests
* Make clippy happy
* Restore deletion bitmask to reduce IO on index load
It is more compact than count data we read before to check for cleared
payloads.
* immutable payload index storage 1 bis review (#8638)
* use bitwise operations instead of loop
* [AI] Only propagate bitslice
* fmt
* Fix deleted bitmask length
It must be the same length as `point_to_values` length.
* account for ram usage
* Default missing external deletion bits to live, not deleted.
* clear doc for flush
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Gate `FsType` enum and impl with `#[cfg(fs_type_check_supported)]` to
suppress dead_code warnings on platforms where the fs type check is not
supported (e.g. Windows).
Move `CollectionError` import into the `#[cfg(unix)]` block where it is
actually used, fixing unused_imports on non-unix.
Add `#[cfg(unix)]` to `OTHER_COLLECTION_NAME` test constant that is
only referenced in unix-gated tests.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* [AI] strict mode parameter for limiting update requests if ram usage is over threshold
* opanAPI update
* [AI] end-to-end test
* fmt
* Fix e2e test: memory rejection check broken by string truncation
UnexpectedResponse.__str__() truncates the raw response body, cutting
off the `max_resident_memory_percent` hint at the end of the error
message. Use `resident memory usage` instead, which appears early
enough to survive the truncation.
Made-with: Cursor
* add grpc validation
* test check_resident_memory
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* rollback: iterator of errors into error of iterator
* [AI] ordering iterator
* [AI + manual] simplify geo-index iterators
* [AI] extra test for consistency
* fmt
* [AI] extend test for deleting points + fix the problem
* Update lib/segment/src/index/field_index/geo_index/mmap_geo_index.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Test if io_uring handles EINTR properly
* Fix unit test compilation after read_iter API change
Update test_io_uring_eintr_handling to match the new read_iter signature
that takes (Meta, ReadRange) tuples and returns Result<impl Iterator>.
Made-with: Cursor
* Install no-op SIGUSR1 handler in debug mode on Unix
Prevents SIGUSR1 from terminating the process with the default
disposition, so that io_uring EINTR tests can safely bombard
the process with signals.
Made-with: Cursor
* Enter tokio runtime context for SIGUSR1 handler, fix clippy
tokio::signal::unix::signal requires a reactor context, so enter
the runtime handle before installing the handler.
Also fix manual_let_else clippy warning in the EINTR unit test.
Made-with: Cursor
* Cleanup 🙄
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* use pageout to clear mmap cache
* Also clear cache of deleted flags in mmap dense vector storage
* Add reference to madvise man pages for probe logic
* use deconstruct
---------
Co-authored-by: timvisee <tim@visee.me>
* Add mincore-based memory stats to MmapFile
Add `resident_bytes()`, `disk_bytes()`, and `probe_memory_stats()` methods
to `MmapFile` for measuring page cache residency via `mincore(2)`. This is
the foundation for per-collection memory usage reporting.
Also extract `page_size()` as a public function in `mmap::advice`, replacing
the internal `PAGE_SIZE_MASK` with a direct page size cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] introduce trait for reporting memory usage per component
* [AI] memory reporter implementation for vector storage
* [AI] implement MemoryReporter for QuantizedVectors
* [AI] implement MemoryReporter for VectorIndexEnum
* Implement MemoryReporter for IdTrackerEnum with RAM estimation
Add ram_usage_bytes() to all ID tracker types and their data structures:
- PointMappings, CompressedPointMappings, CompressedVersions,
CompressedInternalToExternal, CompressedExternalToInternal
- MutableIdTracker, ImmutableIdTracker, InMemoryIdTracker
All ID trackers load their data into RAM (none use mmap for working data).
Files are reported as OnDisk (persistence only), actual RAM footprint
is reported via extra_ram_bytes. Uses struct destructuring to ensure
new fields trigger compile errors if not accounted for.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] segment-level memory usage report
* [AI] Block 3: Aggregation Layer and Data Model + internal api for remote shard
* [AI] REST API handler
* fmt
* [AI] clippy fixes
* [AI] macos fix + proxy segment fix
* [AI] make text index estimation a bit more correct
* fix is_on_disk reporting for dense_vector_storage
* fix after rebase
* [AI] deep account for quantized vectors RAM usage + unify chunk size + shring volatile storage after load
* remove debug log
* cache in test
* make manual test easier to run
* rollback chunk size diff, but keep it for test only
* review fixes
* Use exhaustive match
* Use div_ceil on bits everywhere
It does not seem to be strictly necessary because the number of bits
should already be a multiple of the used container size bytes. Still
it's good practice to be careful with this calculation.
* Improve heap size bytes for encoded product quantization vectors
* Include vector stats for binary quantized vectors
* In volatile chunked vectors, include heap allocated vector
* Include rest of heap allocated structures for mutable map index
* In mutable geo index, the hash map is also heap allocated
* Update tests/manual/test_memory_reporting.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Add empty placeholder vector storage types for named vector CRUD
Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as
placeholder storages for newly created named vectors on immutable
segments. These report all vectors as deleted, consume no disk space,
and are reconstructed from segment config on load via the new
VectorStorageType::Empty and SparseVectorStorageType::Empty variants.
Key design decisions:
- is_on_disk is derived from original user config, not hardcoded
- MultiVectorConfig is preserved for multi-vector support
- Config mismatch optimizer skips Empty storage to avoid false rebuilds
- Quantization delegates normally (handles 0 vectors gracefully)
- get_vector includes debug_assert to catch unexpected access
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] segment-level operations for creating and deleting anmed vectors
* [AI] implement named vector creation and deleting in proxy segment
* [AI] Step 3: Proxy Segment Handling for Named Vector Operations
* [AI] implement for Edge
* [AI] implement consensus operations for named vector operations
* [AI] refactor VectorNameConfig, remove VectorNameConfigInternal
* [AI] handle vector schema inconsistency in raft snapshot recovery
* [AI] rest + grpc API
* [AI] clippy
* [AI] generate openAPI schema
* fmt
* ci fixes
* [AI] fix jwt access test
* [AI] nop operation for awaiting of consensus-commited update ops
* [AI] move vector name operations into points service
* [AI] implement internal api for vector name operations
* [AI] change collection-level config along with segment level operation
* [AI] vector schema reconceliation instead of error
* fmt
* missing compile-time option
* [AI] integration test
* [AI] fix missing JWT tests
* [AI] remove NOP
* [AI] openapi test
* [AI] fix initialization of mutable segment
* [AI] more simple integration tests
* fmt
* [AI] make cluster test a bit harder
* [AI] make test less flacky
* [AI] rabbit comments
* [AI] check params compatibility before writing vector config
* [AI] make sure to register vector storages in structure payload index
* [AI] vector name validation
* lower vector length validation to 200 chars to account for prefix in filename
* [AI] proxy segment: prevent stale data leak through optimization
* fmt
* [AI] filter out removed vectors from proxy response
* [AI] handle vector name in proxy
* fmt
* adjust proxy info based on dropped vectors
* [AI] proxy segment: update filters to correct has_vector condition
* fmt
* clippy
* Fix consensus snapshot applicaiton for vector schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Return error instead of panicking for corrupted aliases file on startup
* common::fs::ops: provide file name in error messages
And drop FileStorageError in favor of std::io::Error, since we always
convert all kinds of errors into ServiceError anyway.
* TableOfContent:🆕 return errors instead of panics
Also, drop context strings. We use fs_err anyway, that should be enough.
---------
Co-authored-by: leohenon <77656081+lhenon999@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
* IoUringState: generic `RequestId`
* UniversalRead: generic `RequestId`
* Simplify `gridstore::Pages::get_page_value_ranges`
Now we don't need two separate `SmallVec`s as we can put `buffer_offset`
into `RequestId`.
* Better doc comment
* Rename `RequestId` -> `Meta`
