mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-29 01:17:56 -05:00
b60f298dea
* Add UpdateOnlyAppendableIdTracker, the append-only writer The write counterpart of `ReadOnlyAppendableIdTracker`, producing the two files that tracker already consumes — `mutable_id_tracker.mappings`, an append-only log of mapping changes, and `mutable_id_tracker.versions`, a dense array of one version per slot — through `UniversalAppend`, so the same code drives a local file and an object store. `insert_operations` records a batch of `MappingOperation`s in order: an insert claims the next slot above the highest one in use and reports it, a delete retires an external id and claims nothing. Nothing is rewritten in place, so re-inserting a live id moves it to a fresh slot and supersedes the old one — the update-only shape of an update. `set_internal_versions` extends the versions array. Ids may come in any order but must be exactly the slots the array does not cover yet: a slot below the end would need an in-place overwrite, and a hole would have to be zero-filled — and since "covered by the versions file" *is* the commit signal for readers, that would publish a slot as a live point of version 0 before its data exists. Both are rejected rather than written. Both methods append at an offset they probed for, never at an implicit end: the offset is a compare-and-swap token, so a file that has moved on since the probe is rejected instead of being written twice or in the wrong place. Both have persisted what they wrote when they return `Ok` — append, then run the handle's flusher — and nothing is buffered across calls. The order of the two calls, claim the slot then commit the version, is what makes a crash in between safe: readers ignore slots the versions array does not cover. Cleaning up the slots such a crash abandons is left to the opener, along with repairing a torn tail; the writer fails loudly rather than guessing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Share the versions file format between both write paths `set_internal_versions` was reimplementing what `versions_storage` already knew: entries are `SeqNumberType`-sized, slot `n` lives at `n * VERSION_ELEMENT_SIZE`, a file length that is not a whole number of entries has a torn tail. Every one of those facts existed in two to four places, spelled as inline `/`, `%` and `write_u64::<FileEndianess>`, so the append-only writer could drift away from the in-place one silently. Move them into `versions_storage`, which now owns the format for both writers and both readers: - `write_version` / `read_version`, the entry codec, with a static assertion tying its `u64` to `VERSION_ELEMENT_SIZE` so a change to `SeqNumberType` cannot silently shrink every offset; - `version_offset` and `versions_byte_len`, the slot arithmetic; - `VersionsLayout`, which splits a file length into committed entries and a partial tail. The two writers still react differently — the in-place one truncates the tail, the append-only one refuses it, because an append cannot — but they no longer each work out what the tail is. `store_version_changes`, `load_versions`, `set_internal_versions` and the read-only tracker's live reload all go through it. The write loops themselves stay separate: one seeks to sparse offsets, the other emits a validated consecutive run, and merging them would obscure both. What they share is where the bytes go, which is the part that must not diverge — and a new test pins it down by writing the same versions through both writers and comparing the files byte for byte. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Append mapping and version entries as batches Both writers built one concatenated buffer and handed it to `append`. `append_batch` takes the entries as separate buffers and places them in a single operation — a vectored write locally, one request on an object store — so the entry boundaries reach the backend instead of being flattened away first. Versions are fixed-size, so the entries are the payload's `chunks_exact`. Mapping changes are variable-length, so their bounds are recorded as they are serialized. Both keep the compare-and-swap offset, which `append_batch` validates the same way `append` does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test the writer against MutableIdTracker end to end The suite leaned on round-trips through the storage loaders, which only restated what the writer had just written. Four such tests are replaced by one that checks the property actually worth having: drive the append-only writer and `MutableIdTracker` with the same points, versions and deletes, open each segment through `ReadOnlyAppendableIdTracker`, and require the two views to be indistinguishable — counts, deleted state, external ids, live points' versions, and id resolution. Versions are compared for live points only. `MutableIdTracker::drop` overwrites the slot with `DELETED_POINT_VERSION`, which an append cannot do, so the append-only writer leaves the point's original version there; neither is observable for a point that is gone. The remaining tests keep what a round-trip cannot show: slot allocation across calls, instances and deletes (three tests folded into one), the rejection of holes and rewrites, and the byte-for-byte agreement of the two writers on the versions file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * better buffer names * fmt * Heal a torn versions tail instead of refusing to write A writer that dies mid-entry leaves the versions file ending inside a slot. The in-place writer already truncated that tail before writing; the append-only writer refused, which left the file unwritable forever since an append cannot truncate. Share the decision — what counts as torn, the healthy length, the warning — in `heal_versions_tail`, and let each writer supply the shrink its backend can do: `set_len` in place, or reading the committed prefix back and putting it in place as a whole file where there is no truncate. Dropping the tail loses nothing: the array covers a slot only once its whole entry is there, so a partial entry belongs to a slot no reader ever saw and no writer counted as committed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Append mappings at the end of the log, not the end of the file Mapping entries vary in length, so no length tells you whether the log ends on an entry boundary. Appending at the file's end therefore could not fail: a torn entry was silently appended after, and every entry from there on was framed off the stray bytes. Carry the boundary instead. `new` takes the offset just past the last complete entry — `ReadOnlyAppendableIdTracker::mappings_read_to`, from the same view that supplies `max_internal_id` — and appends there, which turns a file ending elsewhere into an append offset conflict. On that conflict `heal_mappings` cuts the file back to the log's end, the same read-prefix-and-rewrite the versions file heals with, and writes the batch again. A torn entry and a batch that landed unacknowledged are indistinguishable without parsing, and need not be told apart: neither `max_internal_id` nor `mappings_end` moves before an append is durable, so the retry writes the same bytes at the same offset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Move the healing functions into their own file `update_only/mod.rs` had grown to hold the writer, its two public write paths and the two repair routines they fall back on. Split the latter out: `heal_versions` and `heal_mappings` move verbatim into `update_only/heal.rs`, as a second impl block, following the layout the read-only half already uses (`lifecycle.rs`, `live_reload.rs`). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Retire inherited pending inserts when opening the writer A slot is spoken for from the moment the mappings log claims it — components may already have written data at it — so the writer must resume above every slot the log ever handed out. Deriving that bound from a reader's point set undercounts it twice over: a claimed slot whose version was never committed is not in the mapping, and one whose external id was deleted afterwards is not among the pending inserts either. Track it in the log instead, as `ReadOnlyAppendableIdTracker::max_claimed_internal_id`, bumped on every insert entry regardless of what becomes of the point, and take it as `UpdateOnlyAppendableIdTracker::new`'s bound. The points on those claimed-but-unversioned slots are the other half. They cannot be adopted: a writer stopped partway through them, so some components hold their data and others do not, and which is unknowable here. They cannot be left alone either, the versions array being dense — covering any slot above one of them publishes it, half-written. So `new` now takes the pending inserts explicitly and retires them, recording a `Delete` per id before the writer can be used at all, which is what makes it fallible. Doing it at construction rather than lazily on the first write means no write path can be added later that forgets to. `set_internal_versions` accordingly stops rejecting holes: it writes the whole run from the end of the array through the highest id given, covering skipped slots with `DELETED_POINT_VERSION` as the in-place writer's seek already does. It gains an upper bound in exchange — publishing a slot means covering every slot below it, so an id the log never claimed is refused. Live-reload semantics are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Skip UpdateOnlyAppendableIdTracker heal tests on Windows Healing replaces the file via atomic_save while an mmap handle is still open, which Windows denies with os error 5. * Drop mmap handles before healing ID tracker files atomic_save cannot replace a path while an mmap is still open on Windows. Copy the committed prefix, drop the handle, then rewrite and reopen. * Trim ID tracker docs and drop redundant helpers Condense the doc comments on the update-only tracker to the style of the sibling modules, merge a duplicate impl block, and remove `read_version` and a debug assert that restates its own operands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop the versions layout helpers and inline the arithmetic VersionsLayout, versions_byte_len and version_offset wrapped one divmod and one multiplication between them, and adopting the struct made the live-reload hunk longer than the line it replaced. Compute the committed length where it is needed instead, and let heal_versions_tail and heal_versions return unit, since no caller used the layout they handed back. Also drop the writer's unused max_claimed_internal_id accessor, and fold retires_inherited_pending_inserts_at_construction into retires_inherited_pending_inserts: the merged test asserts the retirement happened before the writer did anything, and commits a real version to the retired slot rather than letting it take the filler, so it still shows the Delete is what hides the point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>