The HashSet bought nothing: names are unique by construction (map keys)
and all callers only iterate. A sorted Vec skips the hashing and set
allocation, and makes the iteration order deterministic.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* DiskIdTracker: RAM-resident is_uuid stored-bitmask sidecar + module split
Move the is_uuid flags out of the i2e file into a separate
id_tracker.is_uuid file in the compact StoredBitmask format (#9871),
loaded whole into RAM as a RoaringBitmap on open and prefetched in
preopen, so slot decoding never reads the flag from disk. Bump the
on-disk format version to 2 (DiskIdTracker is unreleased; no migration).
Add StoredBitmask::read_ones() in common to normalize any stored
encoding into a bitmap of set positions, with logical_len validated
against the u32 position space at open.
Restructure disk_id_tracker: read_only.rs becomes read_only/{mod,
lifecycle,live_reload,id_tracker_read} mirroring the immutable tracker,
and reader.rs becomes reader/{mod,lifecycle,lookup,iter}.
Also unbox the DiskMappingsRef iterators (impl Iterator instead of
Box<dyn>), and make IdTrackerRead::iter_internal_versions fallible so
the read-only disk tracker propagates storage errors instead of
silently truncating; its implementation now reads the versions file in
one pass (cleanup-on-open drains it anyway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Split disk_id_tracker mod.rs into lifecycle / read / write files
Mirror the read_only/ layout: mod.rs keeps the struct and resident-RAM
helpers, lifecycle.rs the build/open paths, id_tracker_read.rs the
DiskMappingsSource + IdTrackerRead impls, id_tracker.rs the mutable
IdTracker impl. Code moved verbatim; only imports and a field doc touched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Tighten disk_id_tracker docstrings around guarantees
State contracts (residency, laziness, error semantics, atomicity)
instead of narrating which structures hold or use what.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate plain and raw upsert drivers via PointToUpsert trait
upsert_points and upsert_points_raw were ~60-line clones differing only
in how the point struct writes itself into a segment. Extract a private
PointToUpsert trait with the two variation points — upsert_into (in-place
write) and write_moved (CoW-move record transform) — implemented for
PointStructPersisted and PointStructRawPersisted, and fold the chunked
driver into a single generic upsert_points_impl.
The public functions keep their names and signatures as thin wrappers.
The duplicated upsert_with_payload/upsert_raw_with_payload tails collapse
into one shared set_full_or_clear_payload helper, which also carries the
single has_point debug assertion.
No behavior changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate plain and raw sync drivers via PointToSync trait
sync_points and sync_points_raw were ~80-line clones of the same 5-step
algorithm, differing only in the retrieval call (retrieve vs retrieve_raw)
and the stored-record type compared against. Extend the upsert approach
with a PointToSync subtrait carrying an associated StoredRecord type,
retrieve_stored, and is_equal_to (delegating to the existing inherent
methods), and fold the drivers into a single generic sync_points_impl.
Step 5 calls upsert_points_impl directly; PointToUpsert and
upsert_points_impl become pub(super) to be visible within points/.
Public signatures unchanged.
No behavior changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lib/shard/src/update.rs grew to 2200+ lines. Split it into an update/
directory by operation kind, moving code verbatim:
- mod.rs: process_* dispatch entry points + re-exports (public API and
crate-internal paths are unchanged)
- points/: upsert.rs (plain, conditional and raw), delete.rs (by id and
by filter), sync.rs (plain and raw)
- vectors.rs, payload.rs, field_index.rs: per-kind apply functions
- helpers.rs: shared filter-based point selection (incl. the deferred
points corner case) and check_unprocessed_points
- tests.rs: the test module, unchanged
Only additions are per-file imports, re-export lists and two
pub(super) visibility bumps for now-cross-module helpers.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The WAL is only truncated past operations whose segment flush was
confirmed, so first_index is a durable lower bound on the applied
sequence. The persisted applied_seq can legitimately lag behind it by
more than one save interval: it is saved every 64 update-worker calls
from a counter that restarts at zero on process start, and synchronous
WAL replay never feeds it. A replay target computed from such a stale
applied_seq can then sit before first_index, tripping the debug_assert
from #8454 (flaky model_testing gate, #9844) and, in release builds,
enqueueing already-truncated indices that fail with spurious
"Operation not found in WAL" errors.
Clamp the replay target at first_index: nothing before it ever needs
replay.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Compact stored bitmask for on-disk field index deleted masks
Add StoredBitmask: a compact persisted bitmask written and read as a
whole. The payload is a roaring bitmap of whichever bit value is the
minority (mostly-0 and mostly-1 masks both stay tiny), falling back to
raw dense bits when roaring would not be smaller, so the file is never
larger than the dense representation. Files are replaced atomically via
UniversalWriteFileOps::atomic_save; there is no in-place mutation.
Use it for the write-once "no values" masks of the on-disk numeric,
geo, map and full-text indexes, replacing the raw dense bitslice files
sized at point_count/8 bytes regardless of content.
Writing the new format is gated by the compact_bitmask feature flag
(default off; enabled by `all` and `serverless_compatible`). Reading is
format-agnostic regardless of the flag: the compact deleted_mask.bin is
tried first, then the index-specific legacy file, so old segments stay
readable and flag-off builds produce byte-identical legacy files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Split save_bitmask into named helpers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Regenerate OpenAPI spec for compact_bitmask feature flag
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review findings on stored bitmask
- Avoid u64 overflow in the payload bound check when opening a mask
with a corrupted payload_len.
- Reject roaring payload positions beyond logical_len at read time,
enforcing the BitmaskContent range contract for corrupted files.
- Remove the opposite-format mask file after a successful save, so a
rebuild with a flipped compact_bitmask flag can't leave a stale
compact file shadowing the fresh legacy one (or an orphaned legacy
file next to a compact one).
- Make the compact-open numeric test tolerate builds that already
wrote the compact format.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every point id the workload draws now comes from an IdSpace pool
precomputed at startup. A --uuid-id-fraction (default 0.5) of the
--id-pool slots are well-formed v4 UUIDs built from the seeded rng via
uuid::Builder::from_random_bytes, the rest stay numeric. Precomputing
the pool keeps the id-reuse semantics (upserts overwrite live points,
deletes and retrieves hit them) that fresh per-op random UUIDs would
lose, and keeps runs seed-reproducible.
Sampling consumes a single range draw per id, exactly like the previous
NumId draw, so fraction 0 consumes no extra rng draws and reproduces
the numeric-only op stream byte-for-byte. The harness smoke tests run
with fraction 0.5, and the fraction is recorded in the trace header.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Align the workspace dependency pin with the already-resolved
Cargo.lock version (includes GHSA-7gcf-g7xr-8hxj fix).
Co-authored-by: Cursor <cursoragent@cursor.com>
Preparation for capping appendable segment growth in the update path
(#9158): a dedicated error for "all appendable segments reached
max_segment_size", so the update pipeline can recognize it and provision a
fresh appendable segment before re-applying the operation.
Maps to a transient service error at the collection level: if it ever
escapes recovery, failed-operation recovery re-applies the operation.
Part 1/5 of the appendable segment overflow fix.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every vector candidate now carries a distance metric instead of the
hardcoded Dot, threaded into the fixture schema, the CreateVectorName
generator and the read-back prediction.
The model predicts Cosine read-backs exactly by mirroring the engine's
ingestion preprocessing: metric_preprocess follows
NamedVectors::preprocess_dense_vector's per-datatype dispatch and calls
Distance::preprocess_vector itself, so predictions track the engine by
construction (including the identity preprocess of the byte metric,
which stores Uint8 vectors un-normalized). Stored vectors are
preprocessed exactly once (optimizer and CoW moves transfer raw bytes),
so predictions stay exact across moves, including Cosine + Float16.
New candidates: "e" (dense Cosine), "n" (multi-dense Cosine, per-row
normalization), "x" (dense Cosine + Float16), "o" (dense Cosine +
Turbo4, padding-free dim), "j" (dense Euclid), "k" (dense Manhattan).
Euclid/Manhattan preprocess is an identity, so their value is engine
side: Order::SmallBetter comparator coverage.
The startup predictability check now also rejects sparse + non-Dot
(sparse schemas carry no distance) and Turbo4 + Euclid/Manhattan (TQ's
L1/L2 modes store lengths differently from Dot/Cosine and their
copy-on-write re-quantization fixed point is not soak-validated yet).
Soak-validated on seeds 1/2/4/5/6/7/8 (30k ops), including two
restart runs (restart probability 0.002) with the optimizer enabled.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Profiling showed SipHash over JsonPath keys (RandomState::hash_one)
dominating field index lookups in
StructPayloadIndexReadView::estimate_field_condition: the map holds
only a handful of indexed fields, but it is queried per condition per
filter evaluation per query, and hashing the path is the entire
lookup cost.
Switch the field index lookup maps (IndexesMap, ReadOnlyIndexesMap,
the read view borrow, and the query_checker / condition_converter /
value_retriever signatures that receive them) from std HashMap to
AHashMap, already used throughout the crate. Serialized and API-level
schema maps keep std HashMap: they are cold and their type leaks into
conversions.
conditional_search bench (criterion, 100 samples, vs saved baseline):
- struct-conditional-search-query-points: 583.7 us -> 443.5 us (-23.1% median, p ~ 0)
- struct-conditional-search-context-check: 73.2 us -> 64.8 us (-12.6% median, p ~ 0)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add Ram variant, loads from vector storage
* impl live reload
* preopen sparse storage when using ram index
* Batch storage reads when building sparse RAM index
Rebuild `build_ram_index` around a single `read_vectors::<Sequential>`
pass instead of a per-point `get_vector_opt` loop, so Gridstore
coalesces the IO into block reads instead of a round-trip per point.
Benefits both the read-only mutable-RAM rebuild and the writable
`SparseVectorIndex::plan` build path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Skip the three slowest Windows rust-tests (>60s each): turbo multi
random-ops model tests and the turbo Manhattan HNSW quantization case.
Use rstest test_attr for the parametrized case since a top-level ignore
does not apply to all generated tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* universal_io: add UniversalAppend for atomic single-operation appends
Growing a file previously took a separate set_len + reopen + write dance
(bypassing universal_io and leaving a zero-filled window on crash), and
there was no way to express appends for backends without random-offset
writes.
UniversalAppend::append grows the file by writing at the current end of
file in one atomic grow+write operation and returns the offset at which
the data landed; append_batch lands multiple buffers contiguously in as
few operations as the backend allows. flusher() moves from
UniversalWrite into a new UniversalFlush supertrait so append-only
handles can require it without duplicating the method.
Local backends, both single-syscall:
- MmapFile appends through a dedicated O_APPEND fd (every write(2) /
writev(2) is an atomic grow+write at EOF), then remaps via reopen().
Its flusher also fdatasyncs after appends, since msync alone does not
persist file-size metadata.
- IoUringFile appends via pwritev2(RWF_APPEND). O_APPEND is not an
option there: on Linux, pwrite on an O_APPEND fd appends regardless
of the given offset, which would break positioned writes on clones
sharing the fd.
Concurrent appenders are out of contract (single logical writer);
object-store backends surface the new AppendOffsetConflict error and
recover via reopen() + retry.
* io_bridge: add AsyncWrite/AsyncAppend and appendable BlobFile
Add the write-side backend traits reserved next to AsyncRead: AsyncWrite
(create/remove/save) powers a UniversalWriteFileOps impl on BlobFs
(create-or-truncate put, delete, atomic whole-object save; directory ops
are no-ops), and AsyncAppend — a single-request append where the offset
must equal the current object size, acting as a compare-and-swap token —
powers UniversalAppend on BlobFile.
BlobFile caches the object size across appends (one HEAD for N appends;
a missing object counts as empty so the first append creates it),
concatenates batches into a single request, and drops the cache on
reopen() — the documented recovery path after AppendOffsetConflict. Its
flusher is a no-op: appends are durable once the backend acknowledges
them.
* io_bridge_object_store: native single-request S3 append
object_store has no append support, so issue the PutObject +
x-amz-write-offset-bytes request ourselves, reusing the store's
credential chain (AmazonS3::credentials) and object_store's SigV4
AwsAuthorizer, which signs every header present on the request — no
hand-rolled signing and no direct reqwest dependency. The offset doubles
as a compare-and-swap token: a mismatch (400 InvalidWriteOffset, or 412
on some S3-compatibles) maps to AppendOffsetConflict.
The write-offset append API exists on AWS S3 Express One Zone directory
buckets and compatible stores (e.g. MinIO AiStor) — plain S3 Standard
buckets reject it, and real Express zonal endpoints / session auth are
not verified yet; MinIO-AiStor-compatible stores are the primary target
for now. GCS and Azure sources simply do not implement AsyncAppend.
ObjectStoreSource carries an AppendContext (HTTP client + object URL
base + signing region) built per backend from its config, and gains a
generic AsyncWrite impl (single-put create/save, delete). A test-only
multi-request CAS emulation over InMemory exercises the BlobFile append
stack hermetically; an end-to-end flow against a real append-capable
store is gated behind S3_APPEND_INTEGRATION_TEST=1.
* simple_disk_cache: write-through UniversalAppend for DiskCache
Append to the remote (the single grow+write operation), then write the
same bytes through into the local mirror so tail reads do not re-fetch
what was just uploaded. LocalState::append_local keeps the fetched
bitmap accurate: blocks fully covered by the appended range are marked
fetched, and the pre-append partial tail block — which resize() drops
because set_len zero-fills its gap — is re-marked only when its prefix
was already fetched. If the mirror turns out stale (the remote grew
behind our back), append falls back to resize-only and lazy fetches
heal the gap on the next read.
Writeable opens are now allowed on DiskCacheFs solely to enable append;
DiskCache still never implements UniversalWrite. The writeable flag
propagates to the remote handle, which is opened buffered instead of
O_DIRECT: appends write through the page cache, which O_DIRECT reads on
the same fd would fight (and IoUringFile rejects appends on
prevent_caching handles). The remote-immutability docs are relaxed to
append-only with an immutable prefix, matching what reopen() already
assumed.
Includes a full-stack composition test: DiskCache write-through over
BlobFile offset tracking over an in-memory object store.
* io_bridge_object_store: build the append HTTP client lazily
Opening a source from an AwsConfig eagerly built the reqwest client
(TLS setup, connection pool) even when append was never used. Keep the
AppendContext construction to pure config (allow_http flag, object URL
base, signing region) and build the client on first append instead,
cached in an Arc<OnceLock> shared across clones of the source — and
thus across the file handles opened from it. Sources that never append
now pay nothing; client-construction errors surface on the first append
instead of at open.
* universal_io: test that append grows the regular file on disk
The conformance suite reads appended bytes back through universal-io
handles; also assert the underlying regular file itself — created
outside universal_io, verified with plain fs reads — for both local
backends.
* Mention why we use custom HTTP client, object_store crate has no support
* io_bridge_object_store: reject appends unconfirmed by the size header
A store without write-offset support may accept the signed PutObject as
a plain put — replacing the object with just the appended bytes — and
return 2xx (community MinIO did exactly this before 2025-05, commit
minio/minio@6d18dba9). The old success path fabricated the new length
when x-amz-object-size was missing, so the destruction stayed invisible
while every subsequent append repeated it.
Require the x-amz-object-size response header (returned by AWS and
MinIO AiStor appends) for any append at offset > 0 and fail loudly
without it. Offset-0 appends are equivalent to a whole-object write, so
they remain valid either way — a misconfigured store now fails on the
second append instead of never.
* io_bridge_object_store: honor endpoint/region env vars for appends
With AwsCredentials::Default the store is built via
AmazonS3Builder::from_env, which honors AWS_ENDPOINT_URL_S3,
AWS_ENDPOINT_URL, AWS_ENDPOINT, AWS_REGION and AWS_DEFAULT_REGION — but
append_context derived the append URL and SigV4 region only from the
typed config fields. An env-configured deployment would read from one
host while signing and sending appends to
https://{bucket}.s3.us-east-1.amazonaws.com.
Resolve the append endpoint and region the same way build_store does:
explicit config first, then (default credential chain only) the same
environment variables, with AWS_ENDPOINT_URL_S3 taking precedence as in
from_env. The resolution is a pure function over an injected env lookup
so the test does not touch process-global environment state.
* simple_disk_cache: delegate the append flusher to the remote
DiskCache's UniversalFlush impl was an unconditional no-op, justified by
object-store appends being durable on acknowledgement — but the impl is
generic over any appendable remote, and for local remotes (MmapFile,
IoUringFile, exactly the compositions the tests instantiate) that
silently dropped the fdatasync the UniversalAppend contract requires:
append, flush Ok, power loss, appended bytes gone.
Delegate to the remote's flusher once the cache is materialized: local
remotes get their sync, object-store flushers remain no-ops, and a
never-materialized cache has made no appends so a no-op stays correct.
* io_bridge_object_store: retry transient append failures
The append RPC was a single unretried HTTP attempt, while every other
request in this stack goes through object_store's retry layer — a
routine transient 503 SlowDown or connection reset failed the append
hard where a concurrent read would have silently recovered.
Retry connection errors, 5xx and 429 up to three attempts with a short
linear backoff, re-signing per attempt (the SigV4 signature embeds the
request date). Retrying is safe because the write offset is a
compare-and-swap; the one ambiguity — an attempt that landed but whose
acknowledgement was lost — surfaces as a write-offset conflict on the
retry, which is reconciled with a HEAD: under the single-writer
contract, an object size of exactly offset + data_len proves the tail
is ours, so the append reports success instead of a spurious conflict
(whose reopen-and-retry recovery would duplicate the record).
* universal_io: forward TypedStorage::flusher for any UniversalFlush
The flusher forwarding lived in TypedStorage's S: UniversalWrite impl
block, so append-only storages (DiskCache, BlobFile — UniversalAppend +
UniversalFlush but not UniversalWrite) offered append through the
wrapper while the durability flusher the append contract mandates was
unreachable without going through .inner.
Move it to an S: UniversalFlush block: UniversalWrite implies
UniversalFlush, so existing callers resolve unchanged, and duplicating
the method instead would have hit E0592 on backends implementing both —
the very ambiguity UniversalFlush was extracted to avoid.
* io_bridge_object_store: surface unbuildable append requests as errors
Request building could panic on two reachable paths: url accepts URIs
the http crate rejects (IPv6 zone identifiers, URIs beyond u16::MAX
bytes), and AppendContext::new is public so the object URL base is not
guaranteed to be a base URL. Both expects become S3Config errors, so a
configuration edge case fails the append instead of panicking the
thread driving the bridge runtime.
* simple_disk_cache: don't fail appends the remote already committed
append_impl committed to the remote first and returned Err when the
subsequent local-mirror update failed (e.g. ENOSPC on the cache volume)
— indistinguishable from "nothing was appended", so a retrying caller
would duplicate the record on the remote.
The mirror is cache maintenance, not part of the append: on a failed
write-through, log and degrade to bare growth so lazy fetches heal the
unmarked blocks (safe — blocks are only marked fetched after their
bytes landed). Only an unresizable mirror still surfaces an error, and
the UniversalAppend contract now documents that an append Err does not
guarantee nothing was appended: reopen() and re-check the length before
retrying.
* universal_io: bounds-check positioned io_uring writes against EOF
The UniversalAppend contract states that UniversalWrite::write beyond
the end-of-file fails and append is the only growth path — mmap
enforces it, but IoUringFile's write/write_batch/write_multi were
unchecked pwrites that silently extended the file with a zero-filled
hole, inflating subsequent append offsets.
Check every positioned write against the file length (fstat once per
call), matching mmap's OutOfBounds semantics, and generalize the
regression test to run on both local backends including the batched
path.
* io_bridge: reject appends on handles opened without writeable
BlobFs::open dropped OpenOptions entirely, so a BlobFile opened with
writeable: false still accepted appends — mmap and DiskCache enforce
the writeable requirement, the blob backend silently didn't, and a
stray append through a nominally read-only handle would mutate a shared
object.
Thread OpenOptions::writeable into BlobFile and reject appends with
PermissionDenied when it is unset, mirroring the other backends.
Directly-constructed handles (BlobFile::new/open, which take no
OpenOptions) remain writeable. With every backend now enforcing the
flag, the UniversalAppend contract drops its 'where the backend
enforces open modes' hedge.
* simple_disk_cache: answer empty appends from the mirror
An empty append still went through remote.append_batch, so it returned
the remote's live end-of-file — which can diverge from what this
handle's len() and reads observe when the remote grew behind our back —
while leaving the stale mirror unhealed (unlike a non-empty append in
the same state, which resizes).
Accept empty appends early: return the mirror length without touching
the remote at all, keeping the answer consistent with the handle's own
view. The trait contract now spells out that empty appends return the
handle's view of the end of file without growth I/O.
* universal_io: grow the mmap in place after appends
Every mmap append ended in a full reopen(): an open+fstat+close by path
just to learn the new length, and — on the non-Linux fallback, which
rebuilds the mapping with the open-time populate flag — a re-population
of the ENTIRE file per append, making appends O(file size) for handles
opened with Populate::Blocking. Populating after an append is pointless
anyway: we just touched the data we wrote.
Extract the remap machinery into remap_to() (reopen() keeps its exact
semantics, populate included) and add grow_mapping(): a stat-free grow
that never re-populates. Appends learn the new length from a single
fstat on the already-open O_APPEND fd — kept rather than trusting the
mapping length, which is stale exactly in the externally-grown-remote
scenario the disk cache heals through lazy fetches (the foreign-growth
tests catch the difference). The mirror's resize() passes the length it
just set_len'd, dropping its stat round-trip entirely.
Per small append this is write+fstat+mremap, down from
write+open+fstat+close+mremap, with no populate anywhere.
* universal_io: share the mmap append fd across clones
The flusher captured the per-clone append_file at flusher-creation
time, so a flusher obtained from a sibling clone — or created before
the handle's first append — msynced the shared mapping's data pages but
skipped the fdatasync that persists the appended file size: a
half-persist where a crash loses the acknowledged tail even though a
flusher ran after the appends (writer thread + long-lived flush-worker
clone is exactly the natural WAL shape).
Store the fd in an Arc<OnceLock> shared by all clones and read it at
flush time instead of capture time: any clone's append makes every
handle's flusher sync the size metadata, whatever the clone/flusher
creation order. Initialization races between clones keep exactly one
fd. No hot-path cost: reads and positioned writes never touch the cell,
and the append path pays one atomic load next to its syscalls. The
interior mutability also lets append_fd take &self.
* universal_io: document the clone remap hazard truthfully
The remap SAFETY comment claimed moving is safe "since we are holding
&mut self" — which says nothing about clones: they share the mapping
but keep their own raw ptr/len copies, so after a moving (or, on
non-Linux, replacing) remap a sibling clone's next read dereferences an
unmapped address. The trait contract understated the same hazard as a
concurrent-read constraint, while the UB persists after append returns.
State the contract once on MmapFile (clones must reopen before reading
after any growth; a stale clone read is undefined behavior, not a stale
view), correct the SAFETY argument to rely on it explicitly, annotate
the as_bytes unsafe blocks that depend on it, and sharpen the
UniversalAppend contract bullet accordingly. Making clones structurally
safe (resolving ptr/len through the shared Arc) is deliberately left as
a separate change.
* universal_io: share the vectored append machinery between backends
The IOV_MAX-chunking / EINTR-retry / WriteZero / advance_slices loop
existed twice — as local_file_ops::write_all_vectored (mmap) and
inlined around pwritev2 in IoUringFile::append_slices — along with a
verbatim collect/cast/filter-empties preamble in both append_batch
impls. Two copies of subtle short-write handling introduced by one
branch will diverge the first time only one of them gets a fix.
Add an io::Write adapter whose write_vectored issues
pwritev2(RWF_APPEND), letting the io_uring append delegate to the
shared write_all_vectored, and hoist the slice collection into
local_file_ops::collect_append_slices. IOV_MAX becomes private to the
one function enforcing it. No behavior change; the existing conformance
tests (including the beyond-IOV_MAX batch) cover both backends through
the shared path.
* io_bridge_object_store: add s3_express to the test config helper
The AwsConfig struct gained the s3_express field; update the
resolve-endpoint test helper accordingly.
* universal_io: run the append conformance suite over the S3 stack
Promote the backend-generic UniversalAppend battery (offsets, batches
across IOV_MAX, empty appends, read-after-append, reopen visibility,
flusher) from a private test into universal_io::conformance, exposed
under the testing feature so backend crates can run the identical
suite. mmap and io_uring keep running it as before; the object-store
bridge now runs it too, over BlobFs/BlobFile with the in-memory
offset-CAS append emulation — so local file system and S3 append
behavior are asserted by the same test. The real write-offset RPC
remains covered by the gated test_native_append_flow integration test.
* Swap order
* universal_io: disambiguate the io_uring crate import
The import reorder dropped the leading `::`, making `io_uring`
ambiguous with this very module (pulled into scope by the
`use super::*` glob) and breaking the build.
* io_bridge: don't materialize the mock object on rejected appends
MutableMockSource::append called get_or_insert_with before validating
the offset, so a rejected stale append against a missing object left an
empty entry behind (exists() flipping true) — a fidelity gap versus the
real backends, where a rejected append has no side effects. Check the
offset against the current length first and only materialize the buffer
on a match.
* universal_io: disambiguate the io_uring crate import
Restore the leading `::` on the io_uring crate import — without it the
name is ambiguous with this very module, which the `use super::*` glob
pulls into scope, and the crate fails to compile. Matches the sibling
files (pool.rs, runtime.rs), which already import via `::io_uring`.
* io_bridge_object_store: treat 404 under a nonzero append offset as a conflict
A missing object while the handle expected a nonzero end-of-file is a
stale view (the object was deleted behind our back) — the same
situation as an offset mismatch, with the same reopen-and-retry
recovery, and it is exactly what the in-memory emulation and the
io_bridge mock already report. The RPC path mapped every 404 to
NotFound instead, so the three implementations disagreed on the same
logical case. Keep NotFound for offset-0 appends, where a 404 is a
genuine missing-target error (e.g. a missing bucket) that retrying
cannot heal.
* Preallocate vector
* universal_io: disallow appends through the disk cache
Appends must go directly to the backing storage (mmap, io_uring, S3) —
the disk cache is strictly read-only again. Remove DiskCache's
UniversalAppend/UniversalFlush impls and the mirror write-through
machinery (LocalState::append_local), reject writeable opens at
DiskCacheFs::open, and drop the writeable/prevent_caching plumbing that
existed solely for cached appends, restoring read-only remote handles.
Attempting to append through the cache is now a compile-time error (the
trait impl no longer exists), and opening a cached handle writeable is
rejected at runtime, covered by a test in each backend variant.
* universal_io: make append idempotent via caller-supplied offset
Append now takes the byte offset where the data must land:
append(offset, data) -> Result<()>. Every backend validates that the
offset equals the current end of file before writing (mmap and io_uring
fstat the fd, object stores validate server-side via
x-amz-write-offset-bytes); on mismatch nothing is written and the append
fails with AppendOffsetConflict. Retrying an already-landed append
therefore conflicts instead of appending twice, and recovery is
re-deriving the offset from len().
BlobFile no longer tracks the object length locally; the store's own
offset check is the compare-and-swap.
* Review remarks
* Validate file length in S3 append response
* Fix linting, we don't mind a large enum variant on index builder
* universal_io: conformance-test stale-handle append conflict recovery
Promote the two-handle conflict scenario from the in-memory BlobFile
test into the backend-generic conformance battery: a second writeable
handle grows the file, the stale handle's append conflicts cleanly (the
offset check runs against the file, not the handle's view), and the
contract's documented recovery — reopen, re-check the length, append at
the real end — lands the data exactly once. Now exercised over mmap,
io_uring, and the object-store stack instead of only the in-memory
emulation.
* io_bridge_object_store: stub-server tests for append response handling
The native append's HTTP state machine was only exercised by the gated
live-store integration test (S3_APPEND_INTEGRATION_TEST=1), so none of
its branches ran in CI. Cover them hermetically against a minimal local
HTTP stub — one connection per canned response, no new dependencies:
- the signed write-offset PUT, and the new-size validation on success
(matching, mismatching, unparseable, and absent size headers — the
absent case at offset zero and past it);
- conflict mapping for 400 InvalidWriteOffset, 412, and 404 under a
nonzero offset, with 404 at offset zero staying NotFound, and a 400
without the conflict code staying a plain error;
- 429/5xx retries re-sending the same offset, giving up after
MAX_ATTEMPTS, and the lost-acknowledgement reconciliation via HEAD
(accepted when the object ends at offset + len, rejected otherwise);
- the status + body excerpt on unexpected failures.
* simple_disk_cache: statically assert the cache stays read-only
Disallowing appends through the disk cache made them a compile-time
error by removing the impls; pin that with assert_not_impl_any so the
UniversalAppend/UniversalFlush/UniversalWrite impls cannot quietly
return. Runtime rejection of writeable opens stays covered per backend
variant.
* Add Float16 and Uint8 storage datatypes to the model tester
Extend VectorCandidate with a datatype override and fold the DenseTurbo
kind into Dense + Some(Turbo4) so storage datatype has a single source
of truth. Two new initially-active candidates exercise half-precision
("h", dense 6) and unsigned-byte ("y", dense 4) storage; "c" carries an
explicit Some(Float32) to cover schema configs that spell the default
datatype out.
The model predicts lossy read-backs through the engine's own
PrimitiveVectorElement impls (as Turbo4 reuses turbo_storage_roundtrip)
and compares them exactly: both round-trips are deterministic and
idempotent, so they stay bit-stable across optimizer moves, WAL replay,
and reloads. Uint8 components are drawn from 0.0..256.0 since the
storage truncates with `x as u8` and unit-range draws would collapse to
zeros.
A compile-time assertion rejects datatype overrides on non-Dense
candidates: the fixture's sparse/multi-dense arms ignore the field and
multi-dense read-backs are compared without a round-trip prediction, so
a lossy multi-dense candidate would soak-panic with a false divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Plumb Float16 and Uint8 multi-dense support in the model tester
Multi-dense storage converts the flattened matrix component-wise
(from_float_multivector), so per-row round-trips through the same
PrimitiveVectorElement impls predict read-backs exactly. The fixture's
multi-dense arm now applies the candidate datatype (matching the
CreateVectorName path), model_vector predicts per-row, and two new
initially-active candidates exercise the combination: "w"
(MultiDense(5), Float16) and "z" (MultiDense(3), Uint8).
The compile-time candidate check narrows to the combinations that
remain unpredicted: Turbo4 multi-dense (the multivector quantization
path differs from the per-vector turbo_storage_roundtrip) and sparse
with any datatype override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Make datatype match exhaustive in random_dense_vec
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review findings on datatype plumbing
- Start candidate "c" active so the explicit Float32 schema path runs in
default soaks (CreateVectorName is FORCE_OFF by default)
- Single Float16/Uint8 roundtrip dispatch shared by the dense and
multi-dense arms of model_vector
- Hoist shared fixture builder plumbing into dense_params_builder
- Build one DenseVectorConfig literal in the CreateVectorName generator
- Inline single-caller datatype_of wrapper
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace const-eval candidate check with a startup assert
Const eval forbids iterators, forcing an index-based while loop. A plain
function called at the top of run() reads better, still fails before any
op is applied, and names the offending candidate in the panic message.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fold INITIAL_ACTIVE into an initially_active candidate field
The hand-maintained list duplicated ALL_CANDIDATES (11 of 12 names) and
had to be kept in sync when adding candidates; forgetting it was silent
since CreateVectorName is FORCE_OFF by default, so a forgotten name got
zero default-soak coverage. Each candidate now declares its activation
inline and the fixture and run() filter on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [AI] piggyback on in-flight requests
* cleanup tests
* Insert in-flight fetch through Slab's VacantEntry (#9840)
Replace the peek-key-then-insert pattern with slab's vacant_entry() API:
inserting through the reserved entry guarantees the fetch lands on the
key the remote read was tagged with, instead of relying on nothing
touching the slab between the peek and the insert. A failed remote
schedule still leaves no trace, as VacantEntry allocates nothing until
insert.
get_or_init_remote_pipeline now takes the field instead of &mut self so
the VacantEntry can hold a disjoint borrow of in_flight across the
schedule call.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Standalone usage reference for `edge-shard-query`: backends (S3 / GCS /
uio-grpc), connection and tuning flags, the scroll / search / search-sparse
sub-commands, filtering, and the live-reload diff mode.
Written to stand on its own, so it can be shared as a link without also
sharing the source.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The inverse Hadamard rotation is linear, so it distributes over
subtraction: |R'd1 - R'd2| = |R'(d1 - d2)|. Subtract the dequantized
vectors in rotated space and inverse-rotate the difference once,
instead of inverse-rotating both vectors per score call.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sparse nearest-neighbour search over a ReadOnlyEdgeShard, reusing the
same SearchRequest path as dense search with VectorInternal::Sparse.
The query vector is accepted as a JSON object or an index:value pair
list, sorted and validated before use. --vector is required (no random
fallback: the sparse vocabulary is not recoverable from the shard
config), and --hnsw-ef is omitted since it does not apply to the
sparse index.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add optional block-index sidecar to accelerate on-disk binary searches
Binary search directly over an unpopulated mmap costs O(log n) random
reads scattered across the whole file — one page fault (or remote range
read) per probe on high-latency storage. Introduce SortedBlockIndex: an
optional sidecar file storing the first element of every 16KiB block of
a sorted on-disk array, read whole into RAM at open. A lookup becomes an
in-RAM partition_point over the block firsts plus a single contiguous
read of one block, bisected in RAM.
Wire it into the three remaining scatter-probe sites:
- on-disk numeric index: binary_search_pairs over data.bin
- on-disk geo index: counts_of_hash over counts_per_hash.bin
- on-disk geo index: all_points start boundary over points_map.bin
The sidecar is backward and forward compatible: absent (old segments) or
failing validation, readers fall back to the existing plain binary
search; old code simply ignores the extra file.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate block-index sidecar snapshot recovery and preopen coverage
- snapshot round-trip test (Regular + Streamable): the sidecars written by
the on-disk numeric and geo indexes are collected via files() into the
snapshot tar and restored byte-for-byte, with correct query results on
the reloaded segment
- preopen tests for both indexes: after schedule_prefetch, open loads the
sidecar from the CachedFs prefetch pool even when the file is unlinked
from disk; an absent sidecar neither fails preopen nor open (fallback)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An empty update channel only means the last operation was received,
not that it finished applying in spawn_blocking. Use plunge_async as
a barrier before asserting point counts.
Fixes#9831
Co-authored-by: Cursor <cursoragent@cursor.com>
Drain the update worker queue after setup upserts with WaitUntil::Wal.
Wal only waits for the WAL write, so on slow CI (notably Windows) B could
time out while queued behind the setup backlog rather than because the
worker was blocked on A's deferred wait.
Fixes#9814
Co-authored-by: Cursor <cursoragent@cursor.com>
* Speed up TurboQuant Hadamard rotation by ~3x
Profiling apply_inverse showed 44% of cycles in the Fisher-Yates swap
replay (serial LCG plus a 64-bit modulo per element), not in the WHT.
- Materialize the shared permutations as index maps once in
HadamardRotation::new; each apply-time permutation becomes a flat
gather pass ping-ponging between the vector and a thread-local
scratch buffer. The replay path stays as the cfg(test) parity oracle.
- Fuse consecutive WHT outer butterfly stage pairs (h, 2h) into one
pass over the array (AVX2), halving memory traffic for h >= 16.
- Fuse the normalization multiply into the transform's final-stage
stores (wht_dispatch_scaled), removing the separate normalize pass.
Output is bit-identical on all paths: pinned by the existing
struct-vs-replay and SIMD-vs-scalar bit-equal tests plus a new
wht_dispatch_scaled parity test. NEON is unchanged.
Criterion hadamard bench (Zen 5): apply 2.8-3.7x faster across dims
128-4096 (1024: 6.98us -> 2.03us), apply_inverse 2.8-3.5x
(1024: 6.10us -> 2.05us).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: harden gather contract, small-dim parity, grow-only scratch
- Mark gather_permuted as unsafe fn: the get_unchecked justification relied
on a caller invariant (map indices in range) that the signature did not
surface; document it as a # Safety contract instead.
- Add dims 5 and 50 to static_rotation_matches_struct_and_roundtrips to pin
the map path against the replay oracle on degenerate chunk splits.
- Make the thread-local gather scratch grow-only, so threads alternating
between dims no longer shrink and re-zero the buffer on every call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Hard-assert input length in apply/apply_inverse
Fail fast at the public boundary: with debug_assert only, a release
build would WHT-normalize a wrong-length slice before the gather's
hard length asserts panic. Flagged by CodeRabbit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
drop_shard_key force-aborted any resharding on the key being dropped, but
swallowed a failed abort_resharding with a log-only error and dropped the
shards anyway. That could leave resharding_state.json referencing the
just-dropped key — a latent inconsistent load-time state.
Propagate the error with `?`. A ServiceError halts consensus and retries after
restart, which is safe because the abort and everything else in drop_shard_key
is replay-tolerant; a user error dismisses the entry before any shard is
dropped, which is equally consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A ServiceError returned from apply halts consensus on every peer (the entry can
never be applied), deterministically stalling the whole cluster. The transfer
Start validations that report a missing source or destination shard are
reachable — e.g. a committed Start racing a resharding-abort or shard-key-drop
that removed the shard — and all run before any durable write, so dismissing
the entry with a user error is safe and correct.
Convert five such sites to CollectionError::bad_request:
- validate_transfer: source shard missing, and destination shard missing in
both the resharding and filtered branches (helpers.rs);
- start_shard_transfer: the source and target get_shard lookups
(shard_transfer.rs).
The genuine "single node deployment" service_errors in collection_meta_ops.rs
are left untouched — those are real misconfiguration guards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Start's first durable write registers the transfer record; its last write sets
the destination replica state. On re-apply after a crash between the two,
check_transfer_conflicts found the operation's own half-applied transfer and
returned bad_request, which is dismissed and never retried — so the peer
permanently lacked the destination replica entry. Worse, a later Finish on
that peer then silently skips both the destination promotion and the source
removal, pinning a replica-set divergence.
Exclude the transfer's own key from the conflict scan so a replay falls through
and re-runs the (idempotent) start: register_start_shard_transfer is a
set-insert and the destination replica-state write is absolute, so re-running
reconciles the partial state instead of dismissing it. A genuinely conflicting
transfer (different key touching the same shard/peers) is still rejected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
start_resharding, finish_resharding and abort_resharding each saved the
updated collection config with a log-only `if let Err(err) = config.save(..)`
that swallowed the failure. A swallowed save let the operation report success
with a stale shard_number persisted on disk, arming a shard-dir/loader panic
(or a silently unloaded shard) at the next restart.
Propagate the error with `config.save(&self.path)?;` instead. The resulting IO
error is a ServiceError, so consensus halts and retries the entry after
restart. That is safe because in all three functions the config save is
value-idempotent (guarded by `shard_number != new_shard_number`) and every
step is replay-tolerant, so the retry converges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In the up-direction of abort_resharding, ShardHolder::abort_resharding drops
the new shard's directory, but the config.params.shard_number update was
persisted only at the very end of the function — after the transfers abort.
A crash anywhere in that wide window left shard_number pointing at an
already-deleted shard directory, which makes the auto-sharding loader panic
on the missing dir at startup (crash loop), before the consensus replay that
would reconcile the state can run.
Move the shard-count update block to before the shard_holder.abort_resharding
call. The block keeps its value-idempotence guard, so replay converges. The
config write lock is taken while holding the shard_holder write guard, matching
the shard_holder -> config ordering already used in finish_resharding, and is
released before abort_resharding. The reverse crash window (dir still present
but count already decremented) is benign and reconciled by replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In the down-direction of finish_resharding, the config.params.shard_number
update was persisted *after* drop_and_remove_shard. A crash between the two
left shard_number pointing at an already-deleted shard directory, which makes
the auto-sharding loader panic on the missing dir at startup (crash loop) —
before the consensus replay that would reconcile the state can run.
Move the shard-count update block to before drop_and_remove_shard (still after
remove_shard_from_key_mapping). The block keeps its value-idempotence guard, so
replay converges. The reverse crash window (dir still present but count already
decremented) is benign and reconciled by replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When Medium/Strong write-ordering forwards an update to the leader and it
fails with a transient error, the failure path proposed deactivating the
leader replica while passing its raw peer state as from_state. Unlike the
sibling deactivation site, this did not filter out transient/resharding
states, so a Resharding/ReshardingScaleDown leader could be proposed
Dead with from_state=Some(Resharding*).
On re-apply after a crash inside abort-resharding (which reverts the gated
replica to Active or removes it), the from_state gate no longer matches the
current state -> bad_input -> the entry is dismissed and never retried, so
the peer keeps Active while the rest of the cluster has Dead.
Filter the leader state with `.filter(|state| !state.is_partial_or_recovery())`,
matching the sibling site (PR #7849), so the proposal omits from_state for
transient/resharding states and converges on replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
StructPayloadIndex::open took two adjacent bools (is_appendable, create)
and call sites passed every literal combination: (true, true),
(true, false) and (false, true) all exist. A transposed pair compiles
and silently yields e.g. non-appendable + create instead of
appendable + load-only.
The target enum already existed: open immediately converted the bool
into the private StorageType { Appendable, NonAppendable }, so the bool
survived only at the API boundary, exactly where the swap hazard lives.
Make StorageType public, take it directly, and introduce
IndexLoadMode { CreateIfMissing, LoadExisting } for the create flag.
create_segment had the same trailing create: bool with bare literals at
both callers, so its parameter is lifted to IndexLoadMode as well:
load_segment passes LoadExisting, build_segment passes CreateIfMissing.
No behavior change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(recovery): advance newest clocks for WAL-replay tail on load
`load_from_wal` replays the WAL synchronously up to `to` and queues the
remaining `[to, last_wal_index)` tail to the update worker. Whether that
tail exists depends on how far the persisted `applied_seq` lags the WAL
end, independently of the `prevent_unoptimized` flag's value: the flag only
gates the worker's deferred-points wait, not whether the tail is queued.
The synchronous replay advances `newest_clocks` from each entry's clock tag,
but the queued tail did not: the update worker deserializes only the
operation and discards the clock tag.
As a result the newest-clocks recovery point regressed across a graceful
restart by up to the update-queue size, even though those operations are
durably in the WAL (which is exactly what the recovery point tracks). Worse,
the first post-restart updates would then be assigned clock ticks that
earlier WAL entries already carry for different operations, which corrupts
WAL-delta resolution in a cluster.
Advance `newest_clocks` over the queued tail during load, mirroring the
synchronous replay, before handing it to the worker. The range is
end-exclusive because `last_wal_index` is one past the last entry.
Regression dates to WAL replay honoring `applied_seq` (#8008), which
narrowed synchronous replay from the whole WAL to `[from, to)`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(model): assert clock recovery point survives close+reopen
Capture each shard's newest-clocks recovery point (flattened to
shard_id -> (peer_id, clock_id) -> tick, tokens dropped) on both sides of
every close+reopen in the model testing harness (mid-run restart and final
reload) and assert exact equality.
Both mismatch directions are bugs: a lost tick means clock durability broke,
a gained tick means the reload path over-advanced a clock. The check runs
after the existing model check so a lost WAL tail keeps its established
extra/missing-id postmortem signature, and a clocks-only divergence surfaces
distinctly.
This is what caught the WAL-replay-tail clock regression fixed in the
previous commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(recovery): tolerate unreadable WAL tail entries in clock advance
Review follow-ups for the deferred-tail clock advance:
- Log and skip a tail entry that fails to read instead of propagating
the error: the update worker tolerates the same failure when it
re-reads the entry, and failing here turns one bad tail record (or
applied_seq/truncation index skew) into a shard, and by default a
node, that cannot start. Add a red-green-verified regression test
that injects an undeserializable record into the deferred tail.
- Fix the pre-existing off-by-one in the send loop: last_wal_index is
one past the last entry, so `to..=last_wal_index` enqueued a phantom
op_num on every restart with a deferred tail.
- Reword new comments (em dashes, and the range-bound note is obsolete
now that both loops use the same exclusive bound; document the
two-pass shape instead: the WAL iterator is not Send across await).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reset first_voter and prune address book on first-peer --reinit
A peer removed from consensus and killed after `RemoveNode(self)` was
committed but before it was applied keeps the old cluster's
`first_voter` and peer addresses in `raft_state.json`. First-peer
`--reinit` reset `conf_state` to a single voter (itself) but left both
untouched (`first_voter` is in fact never reset, even when the removal
is applied).
Both values are served to peers bootstrapping onto the reinitialized
cluster. A joining peer seeds its initial `conf_state` with the
advertised `first_voter`, and Raft conf-changes are deltas on top of
that base - so a stale `first_voter` permanently corrupts the joining
peer's voter set: it ends up with {old first peer, itself}, missing the
actual leader. Its `/readyz` then treats the still-alive old peer as a
cluster member and waits for the old cluster's commit index, which its
own consensus never reaches.
This only manifests when the reinitialized leader replicates its log as
plain entries (nothing applied before the kill, so the log anchor is
index 0). If the leader sends a snapshot instead, the snapshot's full
`conf_state` heals the corrupted seed - which is why
test_reinit_removed_peer only failed sporadically on CI.
Fix first-peer `--reinit` to behave like founding a fresh cluster:
reset `first_voter` to this peer (not `None`, or `recover_first_voter`
would re-derive the old first voter from the retained Raft log) and
prune `peer_address_by_id` to this peer only.
The kill-before-apply state is now injected deterministically into
test_reinit_removed_peer, reproducing the exact CI failure against the
unfixed binary. Since `--reinit` now prunes the address book, the
stale-address injection in
test_reinit_removed_peer_readyz_ignores_old_cluster moved to a restart
without `--reinit`, so it keeps exercising the /readyz `conf_state`
membership filter from #9688.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix test_reinit_consensus expecting stale address book after --reinit
The test waited for cluster size 2 right after starting the reinitialized
first peer, before the second peer was even started. That only passed
because first-peer --reinit used to keep the old cluster's addresses in
the address book - the stale state the previous commit removes. A
reinitialized first peer is a fresh single-member cluster; the other
peers re-join and re-register their new addresses right after.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
