mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-29 01:17:56 -05:00
master
312
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Luis Cossío
|
d7e41f35ed |
[UIO] Segment live_preload waits for all IO before returning (#10357)
* `LiveReload::live_preload` returns futures * await reopens and reloads concurrently |
||
|
Luis CossíoandClaude Fable 5
|
7cf6d2b2fe |
[UIO] make UniversalRead::live_preload async (#10356)
* rename `reopen`->`live_reload` and `schedule_reopen`->`live_preload` * `UniversalRead::live_preload` returns a shared future * assert snapshot-miss eagerly on `live_preload` `live_reload` cannot see the failed preload: its blocking fallback re-resolves the length from the remote and succeeds. The error surfaces at preload time, as callers (`ok_not_found`) expect. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Luis Cossío
|
9cfbe82b07 |
schedule_open returns nothing (#10355)
|
||
|
Luis Cossío
|
0e68451875 |
[UIO] renames + enforce LiveReload::live_preload (#10351)
* make LiveReload::live_preload required * rename `schedule_prefetch`->`schedule_open` * rename `reschedule_prefetch`->`reschedule_open` |
||
|
mikemikimike
|
750d999dfc | fix(segment): handle null values in arrays (#10101) | ||
|
|
5f952a62d9 |
[LiveReload] Preload ReadOnlyFlags (#10269)
* impl `live_preload` for `ReadOnlyFlags` * fix: apply CodeRabbit auto-fixes Fixed 1 file(s) based on 1 unresolved review comment. Co-authored-by: CodeRabbit <noreply@coderabbit.ai> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: CodeRabbit <noreply@coderabbit.ai> |
||
|
Luis Cossío
|
a9dcb654e0 | fix preload for InMemoryBitvecFlags (#10268) | ||
|
Luis Cossío
|
2489b5d462 |
[LiveReload] Preload chunked vectors (#10225)
* add InMemoryBitvecFlags::live_preload * impl live_preload for ReadOnlyChunkedVectors * heal-resilient reload * no panic, service error |
||
|
Tim Visée
|
34d52c4493 |
Integrate new bitflags structure (#10123)
* Add `FlagsMode::from_feature_flags`, the mode for newly created flags Compact in serverless-compatible deployments, dynamic otherwise. Only creation consults it; opening existing flags detects their mode from disk. * Support the compact mode in the read-only flags types Add `ReadOnlyFlags`, the mode-dispatching union of the two read-only counterparts, serving the shared `RoaringFlagsRead` surface. Teach `InMemoryBitvecFlags` to detect the mode it opens; its compact `reload_appended` decodes the whole (small) file, as the format has no random access. * Create flags through mode selection in storages and indexes Vector storage deleted flags and the bool/null indexes now open through `open_or_create` with the mode from the feature flags: serverless deployments create compact flags, dedicated ones keep creating dynamic flags, and existing flags are opened in their detected mode either way. * Read flags in either mode in the read-only bool and null indexes `ReadOnlyFlags` shares the `RoaringFlagsRead` surface and the lifecycle signatures of the roaring type it replaces, so the swap is a type rename. * Add TODO to not lock bitmask structure during flush * `MutableStoredBitmask::save` returns the number of bytes written Zero when the skip-clean save wrote nothing. Lets wrappers charge the actual write to a hardware counter. * Refuse to open compact flags in a dynamic-mode directory Creating the compact file next to dynamic files would leave a directory of both modes behind, which every later open rejects — refuse up front instead. Both production callers already rule the case out through `FlagsMode::detect`, so this only removes a foot-gun for future callers. The open-or-eagerly-create logic moves into `open_or_create_compact_mask`, shared with the update-only writer next. * Rewrite `UpdateOnlyStoredFlags` onto the compact bitmask The update-only flags writer now writes the compact mode — a single roaring-encoded `compact_flags.dat` through `MutableStoredBitmask` — instead of rewriting the whole padded dynamic file pair every batch. A flush with no effective changes now writes nothing at all, where the old writer rewrote the full mask on any `set`. This also fixes opening serverless-created segments: the old open eagerly wrote a `status.dat` into directories the writable side had created in the compact mode, leaving files of both modes behind and poisoning the directory for every later open. A directory already holding dynamic-mode flags is refused loudly rather than kept current or migrated; rebuild the segment to migrate its flags. Migration may come later. Drops the now-dead `InMemoryBitvecFlags::into_bitvec` and `DynamicFlagsStatus::new`, and demotes `file_size_for` to private. * Run edge tests with serverless feature flags The edge fixtures ran with default feature flags, building leader shards with dynamic-mode flags — a configuration edge never serves in production, and one the update-only flags writer now refuses. It also hid that the writer poisoned compact directories: no test exercised update-only writes over a serverless-created shard. Feature flags are process-global and first-init-wins, so every fixture in the binary initializes the same serverless set; the manifest test folds into it, since serverless implies `write_segment_manifest`. * Don't use sequencial mode for one shot reads |
||
|
Tim Visée
|
4a993d76d0 |
Add segment level type for serverless bitflags (#10121)
* Add `CompactStoredFlags`, segment wrapper over the mutable bitmask RAM-resident flags with a Flusher (skips the write when clean, cancels after drop) and files lister, backed by one compact stored-bitmask file rewritten whole on flush. Not integrated yet. * Add `FlagsMode`, detecting the storage mode of a flags directory `Dynamic` is the existing mmap stack for dedicated deployments, `Compact` the compact stored-bitmask file for serverless ones; detection probes which files are present. Also add the clippy allow the compact flags tests were missing. * Support the compact storage mode in `BitvecFlags` and `RoaringFlags` The wrappers keep their in-memory read state in both modes; the new `FlagsStorage` dispatches the write side between `BufferedDynamicFlags` and `CompactStoredFlags`. `open_or_create` opens existing flags in their detected mode and only applies `mode_if_create` to fresh ones — existing call sites keep constructing the dynamic stack through `new`. * Add `ReadOnlyCompactFlags`, read-only counterpart of compact flags Bound to `UniversalRead`: opens on the bitmask header alone, materializes the bitmap lazily on first query, and never creates a missing file. Implements `RoaringFlagsRead` for the shared query surface; `live_reload` reopens a fresh handle, as flushes replace the file whole but cached handles keep serving the bytes they were opened on. Not integrated yet. * Skip compact live-reload tests on Windows, which forbids the rename Both tests replace the compact flags file behind a reader whose disk cache keeps the "remote" file mapped. On Unix the rename-over succeeds and the mapping serves the old inode — the staleness under test — but Windows forbids renaming over a mapped file, failing the writer's flush with access denied. A limitation of the local-mmap remote stand-in, not of the reload logic, which stays covered on the other targets. * Don't check legacy flag file |
||
|
Luis Cossío
|
d0583725b5 |
[LiveReload] Prepare segment preload (#10221)
* genericize live_reload fs parameters * impl live_preload for ReadOnlySegment * split edge refresh into preload and apply passes * only rotate file infos after successful reload |
||
|
Andrey VasnetsovandClaude Fable 5
|
bcb7a45cc7 |
Add CachedBlobFile: cached reads + write-through appends for object stores (#10206)
* Add CachedBlobFile: cached reads + write-through appends for object stores Combine a DiskCache mirror (reads) with a BlobFile remote handle (appends) into CachedBlobFile/CachedBlobFs, the appendable universal-IO citizen for object stores. Appends perform the remote mutation inline and are durable at Ok: a native write-offset append in AppendMode::Native (with a soft limit on appends per object), or a whole-object rewrite in AppendMode::Rewrite for stores without native append. After a successful append the mirror length is advanced without extra IO; appended blocks fault in from the remote on first read. The multipart UploadPartCopy rewrite path (prefix >= 5 MiB) and the rewrite-required error classification are left as todo!() pending the AsyncRewrite backend capability. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Backend-advertised AppendMethod; reactive appended-block cap recovery Replace CachedBlobFile's stored AppendMode with AsyncAppend::supported_append: the backend advertises Native or PartialUpload, and append takes a matching AppendRequest variant, rejecting the ones it does not support. The multipart UploadPartCopy todo moves into the S3 backend's PartialUpload arm. Drop the native_appends soft-limit counter: it is per-handle in-memory state that resets on every restart, so it can never be the correctness mechanism and persisting it would not make it authoritative either. The store is the authority: hitting its appended-block cap now surfaces as the new UniversalIoError::AppendRewriteRequired (S3 400 TooManyParts), and CachedBlobFile recovers with a whole-object rewrite. Unrecognized errors stay hard errors instead of silently triggering rewrites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Per-store append strategies; server-side rewrites for plain S3 and GCS Replace the single AppendContext struct with an enum of strategy objects, one per store capability, each owning its append logic: - NativeAppend: the signed write-offset PutObject (S3 Express, MinIO AiStor; AwsConfig::native_append declares it for AiStor-like endpoints, s3_express implies it). - PartCopyAppend: plain S3 — appends land as one atomic multipart rewrite whose prefix parts are server-side UploadPartCopy requests; nothing but the appended data crosses the network. object_store keeps such provider-specific calls out of its portable surface, so the requests are hand-signed like the native append. - ComposeAppend: GCS — the appended data is uploaded as a temporary neighbor object and composed onto the destination server-side, conditional on the observed generation (a real compare-and-swap). AppendMethod is replaced by AppendSupport, which tells the caller the only thing it needs: when the store takes a direct append. Always (native, and compose: no part minimums, no block cap), AboveThreshold (part-copy: the copied prefix lands as non-last multipart parts, >= 5 MiB each), or Never. CachedBlobFile drops its hardcoded MIN_COPY_PREFIX and rewrites locally only below the backend-advertised threshold; AppendRequest::Rewrite now means only "append and rebuild as a single blob" — the appended-block cap recovery. The append module is split one file per strategy, with a shared SignedRequestContext transport and a test-only HTTP stub. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DiskCache tracks the remote object's etag Seeded from the new known_etag open extra (OpenExtra::with_known_etag), refreshed from FileInfo on schedule_reopen, and settable directly for callers that mutate the remote out of band. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Remove AppendRequest enum; appended-block cap recovery moves into the backend AsyncAppend::append takes plain (path, offset, data). A native S3 store that rejects an append with TooManyParts now falls back to the part-copy rewrite inside the dispatcher, instead of surfacing AppendRewriteRequired to CachedBlobFile for a second Rewrite request. The Rewrite variant was handled identically to Append everywhere except that one native path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Escalate to download+rewrite when the store rejects a part-copy rewrite The cap-recovery rewrite is chosen by the store's returned error, not a client-side threshold: a part-copy attempt rejected with EntityTooSmall (typed as UniversalIoError::AppendEntityTooSmall, parsed from the S3 error <Code>) falls back to downloading the sub-part-minimum prefix and PUTting the whole object back, guarded by a prefix-length offset check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix S3 Express appends: zonal endpoint + s3express SigV4 service Hand-issued appends targeted the standard endpoint and signed as "s3", so every append to a directory bucket got 404 NoSuchBucket, masked as AppendOffsetConflict by the 404 mapping. Derive the zonal {bucket}.s3express-{az}.{region} base from the mandatory --{az}--x-s3 bucket suffix (mirroring object_store's private derivation), carry the SigV4 service name in SignedRequestContext, and treat a 404 as a conflict only for NoSuchKey or bodiless responses — NoSuchBucket stays a loud error guarding the endpoint derivation. extract_xml_tag moves up to the context module and now tolerates tag attributes and pretty-printed bodies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Server-side etag precondition on appends; BlobFile loses UniversalAppend AsyncAppend::append carries an expected_etag that S3 part-copy rewrites attach as x-amz-copy-source-if-match (412 -> AppendEtagMismatch, a new typed error) and download_rewrite checks against the GET's own etag; native write-offset PUTs and GCS compose ignore it. BlobFile appends only through the inherent etag-aware append_bytes now — CachedBlobFile calls it directly with its DiskCache-tracked etag — and BlobFs's mutating ops become inherent, delegated from CachedBlobFs, per the standing TODOs. The append conformance battery runs over the CachedBlobFs stack, via new direct constructors that share one backend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Drop unfulfilled too_many_arguments expectation rewrite_parts has exactly seven parameters — at the clippy threshold, not over it — so the lint never fires and the expect fails CI under -D unfulfilled-lint-expectations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ivan PleshkovandClaude Opus 5
|
c6cf72a7c9 |
transfer: send raw payloads, behind feature flags (#10066)
A raw point can carry its payload as the byte blob it is stored as, mirroring `PointStructRaw.raw_payload` on the internal gRPC API. The blob travels from the sending node into the receiving node's WAL untouched, so the sender never parses the payload it read and neither node builds a protobuf value tree for it. It is parsed exactly once, where the operation is unpacked for apply (`process_point_operation`), because that is the first place the parsed form is actually needed: `set_full_payload` goes through the payload index, which cannot be updated from bytes. The gRPC boundary therefore only checks the encoding tag and rejects a point that sets both payload fields, the way the enclosing request already rejects both `points` and `raw_points`. Moving the parse onto the apply path makes its error classification load-bearing, so a malformed blob is reported as `OperationError::MalformedPayloadBlob` — the payload sibling of `MalformedVectorBlob`, mapped to `CollectionError::BadInput` for the same reason: a bad blob that reached the WAL has to be skipped on replay instead of crash-looping recovery. Three consequences of the blob living that long are handled explicitly rather than by convention: - `decode_payload_raw` takes the blob only once it has parsed, so a failure leaves the point holding it instead of holding neither representation. - `upsert_points_raw` and `sync_points_raw` refuse a point that still carries a blob. They read the parsed payload, so such a point would otherwise be stored with no payload at all, and a `debug_assert!` would not catch it in release. - `is_equal_to` compares blob to stored blob as bytes. A differing encoding costs a redundant upsert on sync, never a skipped one. The `raw_payload_transfer` bench measures the trade, per 100-point batch (one transfer batch) at payloads of ~200 B / ~700 B / ~7 KB: - Sender, storage bytes to wire: 16x / 37x / 113x faster. This is where the whole win is — no parse of the blob that was read, no value tree built. - WAL encode: 5x / 11x / 25x faster, writing a byte string instead of a map. - Receiver, wire to applicable point: 1.09x / 1.10x / 1.06x. Near neutral, as it swaps walking a prost value tree for a JSON parse. - Wire bytes: ~6% smaller. WAL bytes: 10-32% *larger*, because the blob is JSON while a parsed payload is written as a compact CBOR map. The WAL growth is accepted rather than fixed: decoding earlier to win those bytes back costs a second full deserialization, and would leave the receiving side with a `payload_raw` that is never populated. Making the blob itself compact belongs in the payload storage encoding (`RawPayloadEncoding` is the extension point for it), not here. Two flags, both off by default and both sender-only (nodes accept raw points and raw payloads regardless), read where the transfer batch is prepared: - `transfer_raw_points` transfers every collection as raw points, not only those whose vector storage would drift in a decode-encode round-trip. - `transfer_raw_payloads` ships the blob a raw read hands out; without it the prepared batch decodes it back into the parsed payload, and the wire message is exactly what it is today. Neither is enabled by `all`: a node only accepts them once it runs a version that understands them, so they can only be switched on a release later. Nothing enforces that yet — the transfer has no peer-version gate. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Luis Cossío
|
82ce673c5e |
[LiveReload] Add live_preload (#10036)
* LiveReload: change associated type, add `live_preload` * adjust existing trait implementations |
||
|
Luis Cossío
|
b06814f009 |
[CachedFs] Unchanged file open is no-op (#10050)
* add `UniversalIoError::UnchangedOpen` * add & impl `CachedReadFs::reschedule_prefetch` * clippy * add `OkUnchanged` helper * propagate scheduling errors immediately * drop lock before reacquiring it * clear prefetched_files on new snapshot * only avoid prefetch on full FileInfo equality * `UnchangedOpen` maps to `Cancelled` * match-all match |
||
|
Andrey VasnetsovandClaude Fable 5
|
15fe017cab |
[UpdateOnly] create Blobstore-backed storages append-only under a feature flag (#10154)
* [UpdateOnly] create Blobstore-backed storages append-only under a feature flag A new `append_only_storages` feature flag, enabled by `serverless_compatible`, switches every Blobstore creation site — the payload storage, the appendable field indexes (numeric, map, geo, full-text) and the sparse vector storage — to the append-only Logstore mode. One shared helper maps each site's Gridstore layout to its Logstore counterpart, carrying the page size and compression over; blocks and regions have no append-only equivalent. Only creation consults the flag: an existing storage keeps its persisted mode, both modes are always readable, so flipping the flag never strands data. Two changes make the flag usable rather than booby-trapped: `Logstore::delete_value` now succeeds trivially where nothing is stored, as mutable mode does, and errors only for a stored value. The ordinary write paths delete defensively — an index clears a slot before filling it, an empty value is stored as a deletion — and only ever hit occupied slots when something is genuinely mutated in place. A segment derives `append_only_storages` from the persisted payload storage mode when it opens — not from the flags, which may have changed since it was created — and it forces append-only mutation semantics on itself: every mutation clones to a fresh slot, and the same-operation slot-reuse shortcut is disabled, since the second step of a multi-step write would rewrite a payload row those storages cannot rewrite. The end-to-end test runs as its own binary (feature flags are process-global) with `serverless_compatible` on: the segment comes out holding Logstore storages, and upserts, updates of existing points, multi-step same-operation writes, deletes, an index build over existing points, a flush and a reload all run against them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] assume the flag pairing instead of deriving it, trim the docs Per review: `append_only_storages` without `append_only_mutations` is not a state to defend against — `init_feature_flags` forces the pairing, and the same-operation slot-reuse check reads the flag directly. That deletes the segment-side derivation: the `append_only_storages` segment field, the persisted-mode read at open, and the `is_append_only` accessor chain through `Blobstore`, `PayloadStorageImpl` and `PayloadStorageEnum`. Docstrings and comments trimmed to the guarantees; how the write paths use them is their own business. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] restructure the creation config around mode-neutral options `CreateOptions` in the blobstore crate holds what a caller actually decides — page size, block size, compression — and `into_config(append_only)` turns them into the config of either mode, each taking the fields it can express. The segment-side `storage_config` supplies only the mode, from the feature flag. That removes the misnomer chain the previous cut left behind: nothing named gridstore returns a config that might not be one, and no call site builds a `GridstoreConfig` just to have its fields repacked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] keep Logstore strict; fix the callers that deleted nothing Per review, `Logstore::delete_value` goes back to an unconditional error: a delete reaching an append-only storage is a caller bug to fix, not a case to absorb. The callers that issued vacuous deletes are fixed instead: - The numeric and geo indexes only delete from the storage when their in-memory index actually held values at the slot — the two are written in lockstep, so an empty slot has nothing stored either. The map and text indexes already worked this way. - The sparse storage skips the delete for keys at or past its end, where nothing was ever stored. Each removed call was wasted work in mutable mode too. The e2e test now also drives a numeric index and the sparse storage against append-only mode, and asserts that deleting a stored sparse vector fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * [UpdateOnly] drop the same-op slot reuse; upserts write the whole point at once The append-only path never needs a multi-step point write: the one real multi-stepper was the shard's upsert — `upsert_point` followed by a payload step under one operation number — and it now goes through `upsert_moved_point`, which writes vectors and payload as one operation and one slot. With that, the same-operation slot-reuse carve-out in `handle_point_mutate` has nothing to carry: on an append-only segment every mutating step clones to a fresh slot, unconditionally, and the `append_only_storages` special case disappears with it. The version gate skips only on strictly newer versions, so a caller that still multi-steps stays correct — it pays a slot per step. `PointToUpsert` now exposes the point's parts — raw vectors, decoded vectors, payload — and both write paths are provided from them: `upsert_into` hands the parts to `upsert_moved_point`, `write_moved` adapts them to the copy-on-write move callback. The two hand-written `upsert_into` bodies and the follow-up payload helper are gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Regenerate OpenAPI for the `append_only_storages` feature flag Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * one extra debug assertion * fmt --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 5
|
ff9c7e3078 |
[UpdateOnly] implement the appendable vector storages (#10151)
* [UpdateOnly] drop the `UniversalWrite` bound from `UpdateOnlyChunkedVectors` Nothing in it needs random-offset writes: the config, the chunk listing and the status file all go through `UniversalReadFs` / `UniversalWriteFileOps`, which `UniversalAppend` already provides. The bound excluded the object-store backend this writer exists for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] implement `UpdateOnlyDenseVectorStorage` The vectors go into `UpdateOnlyChunkedVectors` and the deleted flags into `UpdateOnlyStoredFlags`, both of which already append; what this adds is the directory layout and the rule for a point with no vector under this name. Such a point still takes its slot, holding a placeholder, and is flagged deleted — slots are shared across every named storage of the segment, so skipping one would shift every later vector of this storage against the id tracker. Only the missing ones are flagged: an unflagged slot reads as present, and the mask is explicitly allowed to be shorter than the vector count, so a batch where every point has a vector rewrites no mask at all. `VectorToStore` is the input, mirroring the two halves of `FullyQualifiedPoint`: vectors the batch decoded, and storage-native bytes carried over from a point's previous slot which are appended without a decode round-trip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] implement `UpdateOnlyMultiDenseVectorStorage` Rows here are not indexed by point slot — a point owns a run of them — so this writer tracks where the row space ends and places each run itself, reading the end from the chunked storage on open. A run that would straddle a chunk skips to the next one, as the writable side does, since a read of a multi-vector assumes its rows are contiguous within a chunk. The rows of a batch are therefore not always one span, and each span is appended on its own; the gap a skip leaves is zero-filled by the append that follows it. A point with no multi-vector here owns no rows at all: its offset entry says so. Unlike the single-vector storages there is no row to keep aligned, because the offsets are what map slot to rows. Adds `stored_len` and `remaining_chunk_keys` to `UpdateOnlyChunkedVectors` — the vector count read that #10114 dropped as unused now has a user. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] implement `UpdateOnlySparseVectorStorage` The vectors go into `UpdateOnlyBlobstore` — the sparse storage is the one that was already blob-backed — and the flags into `UpdateOnlyStoredFlags`. A point with no sparse vector stores nothing at all, since the storage is keyed by slot and an unwritten slot is already "no vector"; it is flagged instead. `UpdateOnlyStoredFlags::open` now materializes its directory rather than waiting for the first flag. Storages use that directory as the marker that they exist: `MmapSparseVectorStorage::open_or_create` takes its absence for "not created yet" and starts a fresh storage over the top of the old one. A batch that flags nothing must still leave it behind. Caught by the resume test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] implement the TurboQuant vector storages `UpdateOnlyTurboVectorStorage` and its multivector counterpart. The quantizer is rebuilt from the dimension and distance rather than read back — it carries no learned state, so the two sides encode identically, which the test asserts by comparing the encoded bytes against what the writable storage produces for the same vector. The multivector one places runs of rows exactly as the plain multivector storage does, skipping to the next chunk rather than straddling one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] add `UpdateOnlyVectorStorage`, the dispatch over the five families Selects the writer from the vector config the way the writable side selects the storage, and refuses a storage type an update-only segment cannot have: the mmap ones are built whole rather than appended to, and the empty placeholder has no files. Sparse gets its own opener, since sparse vectors are configured separately from dense ones rather than through `VectorDataConfig`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix clippy under `-D warnings` - `is_multiple_of` in place of the manual remainder checks in the two multivector writers. - Drop the `dead_code` expectations on `UpdateOnlyChunkedVectors`: the vector storages use it now, so the expectation no longer holds. - Drop a `TypedMultiDenseVectorRef::from` that converts to its own type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] fix a stale doc and an inconsistent guard The doc on `UpdateOnlyStoredFlags::open` still said nothing is created until the first flush, from before open started materializing the directory eagerly. And the span-merge guard in the multivector writer hedged with `dim.max(1)` while the same function divides by bare `dim` three lines up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] append a multivector batch's rows once, with the gaps as zero rows The chunk layer packs rows consecutively while runs must not straddle a chunk, so a batch's rows are not gapless. The old bridge grouped them into contiguous spans and appended each on its own, leaning on `ensure_chunk_lengths` — the repair path — to zero-fill the gap before every span, and saving the status once per span. Making the gaps explicit zero rows removes all of that: one append per batch, through the normal write path, one status save. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 5
|
c186f9d2e8 |
[UpdateOnly] implement UpdateOnlyFieldIndex for the appendable payload indexes (#10147)
* [UpdateOnly] implement `UpdateOnlyFieldIndex` for the appendable payload indexes The payload index write half for the update-only segment writer, over a backend that only appends. An appendable field index keeps two things: the values it persists per point, and the in-memory structure it answers queries from. Only the first is state — the second is rebuilt from it on every open, by the mutable index and by its read-only counterpart alike. A writer that never answers a query therefore holds nothing: it turns a point's payload into the values its index would persist, appends them at the point's slot, and is done. What differs between index types is only that translation, so that is all `UpdateOnlyIndexKind` captures; `UpdateOnlyValueIndex` is the storage around it, the same for all of them, and each kind lives next to the index it writes for as the read-only counterparts do. The extraction itself is taken from the index types' own `ValueIndexer` and `NumericIndexIntoInnerValue` impls rather than restated, so the two sides cannot drift apart. `UpdateOnlyFieldIndex` dispatches over the nine covered index types, mirroring `ReadOnlyFieldIndex`. What the writer emits is the append-only mode of the very same storage the mutable index writes, and `Blobstore` selects the mode from the persisted config, so the read side needs no change: every test here writes through the update-only writer and reads back through the ordinary appendable index, opened on the directory the writer produced. The boolean and null indexes are not covered and are refused loudly rather than skipped. They keep a bitmask over all points instead of values per point, and persist it through random-offset writes, which an append-only backend does not offer. A skipped index goes stale and then answers queries wrongly, and the null index complements every other index of every indexed field — so a caller that took a silent skip for "nothing to do" would leave every field it touched wrong. Covering them needs an append-only bitmask representation first. That is also why this stops short of the struct-payload-index fan-out: until bool and null can be written, a component that claims to keep a field's indexes current could not. `UpdateOnlyPayloadStorage` moves onto the shared `UpdateOnlyBlobstore` extracted here, which is what it already was. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] share the flattening, tokenization and flush guard Cleanup pass over the field-index writers, from a reuse/simplification review. Share what was restated: - `ValueIndexer::flatten_values` is now a provided method holding the loop that `add_point` had inlined. The update-only kinds call it instead of the free `extracted_values`, which built one throwaway `Vec` per input value on top. - `FullTextIndex::tokenize_document` and `serialize_stored_document` hold the sentinel placement and the phrase-matching order-vs-sort decision that the update-only kind had copied out of `MutableFullTextIndex::add_many`. Both sides call them, so a document written by one always matches the phrases the other would. Simplify: - `UpdateOnlyFieldIndex::open` matches on the index type alone and takes the text params via `TextIndexParams::try_from`, as `ReadOnlyFieldIndex::open` does. That drops the schema tuple, the nine-arm mismatch block and the `Option` return. - The `UuidIndex` variant is gone: that discriminant is historically map-backed, and both the writable selector and the read-only mirror already collapse it into `UuidMapIndex` — its `storage_dir` is `map_dir`, so a numeric-kind writer was writing into a directory everyone else opens as a map index. - Why bool and null cannot be written append-only now lives in `PayloadIndexType::is_append_only_writable`, next to `storage_dir`, so that whoever decides a field is update-only-serviceable can ask rather than rediscover it; `open` consults it as a backstop. - Dead `new()` constructors on the two zero-sized kinds. Skip the flush when nothing was buffered, in `UpdateOnlyBlobstore` rather than in one caller: a flush with nothing to write still syncs every page file of the storage, and for a field index an empty batch is the common case — every point that lacks the field, or holds a value the index rejects, stores nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] move each index kind under the appendable index it writes for `numeric_index/update_only.rs` and its three siblings sat at the index-type level, next to the enum over all three storage variants, although each writes for the appendable variant alone. They now live at `<index>/mutable_<index>/update_only/`, beside that variant's `read_only/` counterpart, which is the same split for the same reason. `mutable_text_index` is private, so the text kind is re-exported from `full_text_index` for the dispatch enum to name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] cover the bool and null indexes by rewriting their masks whole These two keep a bitmask over all points rather than values per point, so keeping one current means changing bytes in the middle of it — which an append-only backend cannot do. It can replace a file outright, and that is enough: `UpdateOnlyStoredFlags` reads the mask into memory on open, sets the batch's bits, and writes both files back whole, in the same format `DynamicStoredFlags` uses. A reader cannot tell which side produced them. The mask goes out before the length that publishes it, so a torn batch falls back to the shorter mask rather than to flags that were never written, and the whole-file write is charged to the hardware counter at flush — it is the write that actually happened, not the handful of bits the batch touched. `UpdateOnlyBoolIndex` and `UpdateOnlyNullIndex` sit on that, next to their mutable index like the other kinds. The null classification (which values count as present, which as null) moves into `classify_payload`, shared with `MutableNullIndex::add_point`, and the boolean one reuses that index's own `ValueIndexer`. Both are recorded for every point of a batch, including those whose field holds nothing: "this point has no value here" is precisely what these indexes are asked. With that, `UpdateOnlyFieldIndex` covers every index type `ReadOnlyFieldIndex` does, so the refusal and `PayloadIndexType::is_append_only_writable` are gone. The cost is that a batch rewrites the entire mask however few bits it touched — about 1.2 MiB per flag set for a segment of ten million points. Documented on the writer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] trim the doc comments on the field index writers Keep the guarantees and the non-obvious rationale, drop the restatements and the comments that narrate the next line. One code change: `values.contains(&true)` in place of `values.iter().any(|value| *value)` on the boolean index. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
xzfcandLuis Cossío
|
efd3bfd617 |
bitpacking_ordered: batched reads (#10038)
* bitpacking_ordered: batched reads * Replace ranges with pairs * "unsufficient" -> "insufficient" --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |
||
|
xzfc
|
ca3fe18f16 |
Remove dead code (#10030)
* Remove dead code * Remove unused dependencies * `allow(dead_code)` -> `expect(dead_code)` * ast-grep: rule-tests/*-test.yml => tests/*-test.yml For brevity. * ast-grep: forbid allow(dead_code) |
||
|
|
a11f8bb4ae |
feat: io_uring setting to control which components use the io_uring backend (#10008)
* feat: `io_uring` setting to control which components use the io_uring backend A few components have both an mmap and an io_uring variant reading the very same files: the immutable dense vector storages, the single-file TurboQuant storage, and the mmap payload storage. Until now the choice was a side effect of `async_scorer` — a vector-search knob — plus, for the payload storage, a feature flag that was parked off because io_uring is ~2x slower than mmap when the data fits the page cache (#9310, #9409). Add `storage.performance.io_uring`, optional, with two modes: - unset (default): unchanged behaviour. The vector storages keep following `async_scorer`; the payload storage stays on mmap. - `disabled`: no component uses io_uring. - `auto`: a component uses io_uring when its memory placement is `cold` (data is left on disk, so reads hit the disk and there is something to gain), its feature flag allows it, and the kernel supports io_uring. Components meant to sit in RAM keep using mmap. The decision lives in one place, `segment::common::io_uring::use_io_uring`, so the openers no longer each reach for the async-scorer global. Kernel support is now probed up front through `is_io_uring_supported()` instead of opening a file and falling back on error. `async_payload_storage` now defaults to on: it no longer decides anything by itself, it only lifts the ban, and the payload storage no longer follows `async_scorer` at all — so turning it on cannot silently move an existing `async_scorer: true` deployment onto the slower path. Which backend a component ended up on depends on the config, the placement and the kernel at once, so report it in `SegmentInfo`: `vector_data[name].io_backend` and `payload_storage_io_backend`, both `"mmap" | "io_uring"`, absent for components that have no such choice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Trim comments, drop trivial tests Two tests were only restating their own implementation: `test_mode_round_trip` round-tripped the encode/decode pair next to it, and `test_io_uring_config` checked that serde deserializes a two-variant enum. The mode matrix test stays, it is the one that pins the semantics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Flatten `IoBackend` in OpenAPI, derive `JsonSchema` for `IoUringMode` Per-variant doc comments on a plain string enum make schemars emit a `oneOf` of anonymous single-value objects instead of a flat `enum`. Move the variant descriptions into the enum doc, as `Memory` and friends already do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update lib/segment/src/vector_storage/turbo/turbo_vector_storage.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * upd openapi schema * Update lib/common/common/src/flags.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Require kernel io_uring support in the async-scorer fallback `use_io_uring` returned `get_async_scorer()` verbatim when the `io_uring` setting is unset, so an enabled async scorer on a kernel without io_uring opened the io_uring storage, failed, and fell back to mmap with an error log per segment. Gate that branch on `is_io_uring_supported()` too, like `Auto` already is, so the component just stays on mmap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * upd openapi schema --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> |
||
|
|
71d99b144c |
Add Logstore and Blobstore wrapper (#9673)
* Gridstore: introduce storage operating mode in config Add a mode field to the gridstore config, selecting between the dynamic mode (current behavior, the default) and the upcoming serverless mode. The mode is specified through StorageOptions on creation, persisted in config.json, and read back first when opening so the correct variant can be selected automatically. Configs written before this field existed deserialize as dynamic. For now, selecting the serverless mode returns an error; the variant itself is added in follow-up commits. * Gridstore: move dynamic implementation into dedicated module Mechanical move of the current Gridstore implementation into gridstore/dynamic.rs as DynamicGridstore. The public Gridstore struct becomes a thin wrapper holding a mode variant enum, propagating every call into the selected variant. For now the enum only has the dynamic variant; the serverless variant is added in follow-up commits. No logic changes to the dynamic implementation itself: only visibility, the config parameter now passed into open (the wrapper reads it first to select the mode), and open_or_create staying on the wrapper. * Gridstore: add serverless tracker Add the append-only mapping tracker for the serverless storage mode. The tracker file is a plain array of 16-byte mapping entries without any header: the number of mappings is defined by the exact file length, and the entry index is the point offset. The file starts empty and only ever grows by appending, existing bytes are never rewritten. Mappings must be set in monotonically increasing point offset order; skipped offsets are backfilled as zeroed entries which decode as None. New mappings are buffered in memory and appended with a single write per flush. A flush with a stale target is a no-op so bytes are never written twice. A torn trailing entry (file length not a multiple of the entry size) is ignored when reading and truncated away when opening writable. Unlike the dynamic tracker, the file is read and written directly with positional file IO instead of memory mapping, as serverless environments do not handle memory mapped files well. * Gridstore: add serverless storage variant Add the append-only gridstore variant for serverless deployments, which restrict IO to appending to files: existing bytes can never be rewritten, and IO is expensive so as few files as possible are used. The variant stores all value data in a single page file next to the serverless tracker and the storage config, three files in total. Both data files start empty and only ever grow by appending; there is no preallocation, no used-block bitmask and no gap/region bookkeeping. Values are appended at put time at the next block aligned offset, with the zero padding included in the write so it lands exactly at the end of the file. Mappings are buffered and appended to the tracker with a single write per flush, after the page file is synced, so a mapping on disk never points at data that is not durable. Values cannot be updated or deleted, and must be put at monotonically increasing point offsets; violations are rejected before any data is written. Files are read and written directly, never memory mapped. The mode is selected through StorageOptions on creation and picked up automatically from the persisted config when opening. * Gridstore: serverless support in reader and view Extend the read-only GridstoreReader and the GridstoreView with the serverless mode, keeping both public types unchanged: like the writable Gridstore they now hold a mode variant internally, selected automatically from the persisted config when opening. The serverless reader holds the tracker and page directly and reads the files positionally, without memory mapping. A live reload re-reads the mapping count from the exact tracker file length (there is no size header), ignoring a torn trailing entry, and never truncates as it is read-only. Value reads always go directly to the file, so newly appended data is readable without remapping anything. * Gridstore: document storage operating modes * Gridstore: review fixes for the serverless mode Hardening and cleanup from a review pass over the new serverless storage variant: - Batch the reader side iteration like the writer already did, instead of materializing tracker mappings for the full range in one go, which could transiently allocate gigabytes on large storages. - Recover the append cursors when a positional write fails partway: truncate the file back to the tracked length so a retried append or flush never rewrites bytes that already landed in the file. - Validate page addressability before appending value data, a rejected put must not grow the page file. - Cross-check tracker and page consistency when opening: mappings that reference value data past the end of the page file (e.g. after a partial copy or restore) now fail fast instead of surfacing as opaque read errors per point. - Reject value pointers into any page other than page 0 on the serverless read path with PageNotFound, matching the dynamic mode contract, instead of silently reading from a wrong location. - Refresh the reported storage size on reader live reload even when no new mappings were flushed, unflushed value data may have grown the page file already. - Validate configs read from disk: a corrupt config with zero sized blocks, pages or regions is now rejected when opening instead of panicking on a division by zero later. - Classify rejected serverless puts as UnsupportedOperation, consistent with rejected deletes, so they don't surface as user-facing validation errors at the segment level. - Deduplicate the compression dispatch into Compression::compress and Compression::decompress, and the serverless file create/open patterns into shared direct IO helpers, so the two modes and files can't silently drift apart. * Gridstore: cover both operating modes in mode-agnostic tests Parameterize the gridstore tests that exercise mode-agnostic behavior over both the dynamic and serverless mode with rstest, using a single and bulk put/get roundtrips, storage files, basic persistence, corrupt config rejection, batched read congruence, reader live reload, and the different block sizes. Mode specific expectations branch inside the tests: expected file names, storage size semantics (whole blocks vs exactly packed bytes), value pointer layout (page spill over vs a single packed page), and gaps (created by deletes in dynamic mode, by skipped puts in serverless mode). Dynamic-only internals assertions are kept behind a mode check. Tests around updates, deletes, page spanning, block reuse and other dynamic-only behavior intentionally stay dynamic; the serverless specific format invariants remain covered by the dedicated serverless tests. * Gridstore: port serverless specific tests from sibling branch Source the serverless specific test cases that the serverless-gridstore-updates branch added, adapted to the dedicated variant implemented here (distinct file names, headerless tracker with 16 byte entries, a single packed page without trailing padding, and rejected re-puts): - writes only ever append: tracker and page files only grow and previously written bytes stay byte-for-byte untouched - new mappings land exactly at the end of the tracker file, which always covers the exact number of mappings - mapping gaps are zero-padded on disk and survive reopening - values are packed back to back at block aligned offsets, the page file ends exactly at the last value - serverless mode never creates nor reports block flag files - a flusher persists exactly the mappings that existed at its creation, later puts stay pending - a config claiming the wrong mode fails loudly in both directions instead of loading the incompatible file format of the other mode Tests around their mode switching, page spanning and tolerated deletes don't apply to this design and are intentionally not ported. * Gridstore: test serverless production risk scenarios Add tests for the operational aspects that matter before serverless mode goes to production, each covering a scenario that wasn't evaluated yet: - Replayed puts of already persisted offsets (a WAL redo after a crash where the flush completed but was never acknowledged) are rejected without appending anything, and max_point_offset is the exact offset a replay must resume at. - The accepted crash case of a tracker file extended with zeroed bytes: the entries count as permanent None mappings, can never be put again, and the storage stays consistent and writable past them. - The read-only reader never modifies the files: opening over a torn tracker tail, reading, iterating and live reloading leave both files byte-for-byte untouched. - A multi-round put/flush/reopen cycle always exposes exactly the flushed prefix, with the mapping count matching the exact tracker file length and unflushed offsets reusable. - An append beyond the maximum addressable block offset is rejected before writing anything, keeping retried puts from growing the page file unboundedly. * Gridstore: rename serverless mode to append-only, split into module Rename the mode after its defining characteristic instead of its deployment target: files only ever grow, existing bytes are never rewritten. Renames Mode::Serverless to Mode::AppendOnly (persisted as "mode": "append_only") and the on-disk file names to append_only_tracker.dat and append_only_page_0.dat. The serverless deployment motivation stays in the documentation. Also split the single 2300 line serverless.rs into an append_only module with dedicated files for the storage, page, view, reader and tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Use universal IO in Gridstore * Include upstream preopen logic in new Gridstore variant * Gridstore: buffer append-only value writes until flush In append-only mode, put previously wrote the value data to the page file right away, one write operation per put, while mappings were already buffered and batch persisted on flush. Buffer value writes the same way: both the value and its mapping now only land on disk once a flush cycle executes. This batches all new value data into a single write operation per flush, which is significantly more efficient on S3 based storage where every write is a costly operation. A flush now performs exactly two writes: one appending all buffered value data to the page file, one appending all pending mappings to the tracker file, in that order, so a mapping on disk never points at value data that is not durable. The page mirrors the tracker's pending mechanism: an in-memory buffer that is byte for byte the next append (zero padding between block aligned values included), a watermark captured at flusher creation so puts made during a flush stay buffered, a stale-flush no-op guard so appended bytes are never written twice, and truncate-back recovery on failed writes. Reads transparently serve buffered values from memory. As a side effect, a crash between flushes now leaves nothing on disk at all, where the write-through approach left orphaned value bytes in the page file. The buffered data is held in memory until the next flush, bounded by the flush cadence. Universal IO filesystem handles are now required to be Send + Sync, so the flusher closure can carry one to grow the page file at flush time; all existing backends already satisfied this. * Gridstore: rename inner DynamicGridstore to Gridstore The dynamic variant keeps the Gridstore name; the outer dispatching type will be renamed to Blobstore in a follow-up. Until then the inner type is referred to as dynamic::Gridstore to distinguish it from the outer type. * Gridstore: rename append-only variant to Arenastore The append-only variant stores all value data in a single ever-growing page, allocating space by appending, hence: arena store. * Gridstore: rename outer storage type to Blobstore The outer type dispatching between the two storage variants is now called Blobstore, being more generic than Gridstore. This frees up the Gridstore name, which now exclusively refers to the dynamic mode variant, next to Arenastore for the append-only variant. Storage components keep using the outer type, so they now use Blobstore. The gridstore crate name, GridstoreError, and the persisted names (config.json mode, payload config storage_type) are unchanged. * Gridstore: split Gridstore and Arenastore into dedicated modules The outer module is now blobstore, matching the Blobstore type it defines. The two storage variants each get their own submodule: the dynamic Gridstore moves from dynamic.rs into gridstore/ with its reader and view extracted from the shared files, mirroring the arenastore/ module (previously append_only/) which already had this layout. * Rename gridstore crate to blobstore The crate is named after the outer Blobstore storage type it provides. The gridstore name lives on in the dynamic mode variant. GridstoreError and the persisted names (config.json mode, payload config storage_type) are unchanged. * Arenastore: pack values back to back across multiple pages Drop the block alignment from the append-only mode: values are packed byte to byte, without blocks, and the tracker offset is now a plain byte offset within the page. Blocks and regions are dynamic mode concepts; their page size constraints no longer apply to append-only configs. Bring back support for multiple pages. Once appending a value would grow the current page beyond the configured page size, a new page is started, bounding the size of and the number of appends to each file: object stores like S3 Express limit the number of appends per object. A value larger than the page size gets a page of its own; values never span pages. A rollover creates the new, empty page file at put time; the value data itself stays buffered until the next flush, which appends to each touched page with a single write, using per-page watermarks captured at flusher creation. The reader scans for consecutively numbered page files when opening, validates the most recent mappings against them, and adopts pages created since on a live reload. * Blobstore: rename dynamic mode to mutable Rename Mode::Dynamic to Mode::Mutable, and the persisted config value with it: config.json now writes "mode": "mutable". There is no compatibility alias for "dynamic", released versions never wrote the mode field (a missing field still defaults to mutable), only unreleased storages did. The Gridstore type and module names for the mutable variant are unchanged. * Fix Edge compilation due to package rename * Review remarks * Extract Gridstore preopen into module * Rename Arenastore files * Use universal IO for append operations * Rename GridstoreError to BlobstoreError The error type belongs to the Blobstore crate and is shared by both the Gridstore and Arenastore variants, so it follows the crate naming. Also update the user-facing error messages that referred to the old name. * Split config into per-variant types * Rename Arenastore to Logstore Rename the Arenastore type to Logstore, including the reader, view, config, module and variant names. The storage file names follow: log_page_{n}.dat and log_tracker.dat. The persisted mode tag stays "append_only". * Move bitmask module into the Gridstore variant The bitmask tracks free blocks, which only exists in the mutable mode. Move the module from the crate root into the Gridstore variant that owns it. It stays re-exported at the crate root because the bitmask benchmark needs a public path. * Move pages module into the Gridstore variant Like the bitmask, the block based pages module is only used by the mutable mode. Move it from the crate root into the Gridstore variant that owns it. The Logstore variant has its own page implementation. * Use universal IO for every Logstore operation Replace the direct_io module with universal IO in the append-only tracker, making the whole Logstore go through a universal IO backend bounded by UniversalRead and UniversalAppend: - The tracker is generic over the backend now. Reads go through UniversalRead with the caller's access pattern, flushes land as one atomic append with the same offset compare-and-swap recovery as the pages: a retried append after a lost acknowledgement is adopted instead of appended twice. A torn trailing entry is still truncated away on writable open, through a fresh handle since shrinking is not supported through an open one. - The reader now schedules a prefetch for the tracker file too, it no longer bypasses the backend. - The config write, clear and wipe use the backend file operations instead of local filesystem calls, matching the Gridstore variant. * Batch reads in Logstore read_values Apply the same batching logic as the Gridstore variant: resolve all mappings first, then fetch the value data, both through the backend's read pipeline so async backends can serve the reads in parallel. The tracker gains a batched lookup mirroring the mutable tracker's iter, serving pending mappings and out of range point offsets directly from memory. The pages gain a batched value read; unflushed values are served from the in-memory buffers, and since values never span pages each value is a single read without reassembly. Like in the Gridstore variant, the callback may now be invoked in a different order than the requested point offsets. * Better describe logstore live reload ordering * use enum for options, swap `*Options`<->`*Config` naming * don't wrap enum in struct * ditch unused `StorageConfig`, make deserialization more ergonomic * rename `*Options`->`*Config` * make `preopen` non-blocking * fixup! ditch unused `StorageConfig`, make deserialization more ergonomic * fixup! use enum for options, swap `*Options`<->`*Config` naming * fixup! don't wrap enum in struct * fix rebase * use `populate` param in Logstore * test: failing repro of stale page after live reload across rollover A reader that live-reloads between a page rollover and the following flush adopts the new, still empty page. The previous page is then no longer the last one and is never reloaded again, so the tail that the next flush appends to it stays invisible to the reader forever: value pointer at byte 100 with length 100 is out of range AppendOnlyPages::live_reload only reloads the last held page, assuming earlier pages never change once a newer page exists. But the rollover creates the new page file eagerly at put time, while the previous page's buffered tail only lands at the next flush (see test_rollover_writes_no_value_data_before_flush), so a page can keep growing on disk after its successor exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: reload all pages that grew * use Fs in `open_or_create` * fix: publish tracker mappings only after the pages reload `AppendOnlyTracker::live_reload` observed the mapping count and made it visible in one step, before `LogstoreReader::live_reload` reloaded the pages. Every failure path in the page reload -- `list_files`, reopening a grown page, opening an adopted one, the truncation check -- therefore left the reader with mappings referencing value data it never loaded, so reads in the new offset range fail until a later reload happens to succeed. The edge refresh loop keeps a segment whose reload failed, expecting it to keep serving its pre-refresh state, which it then does not. Split observing from publishing: `reload_count` refreshes the handle and returns the count as a `PendingReload` token, `commit_reload` publishes it. The reader still observes the tracker first, as the writer persists pages before the mappings referencing them, but only commits once the pages are loaded. Reopening without committing is harmless: reads stay bounded by the unchanged count, and the bytes below it never change. A partial failure inside the page reload needs no unwinding, pages running ahead of the tracker is the safe direction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: batch the value reads in Logstore iteration `LogstoreView::iter_range`, the path behind `Logstore::iter` and `LogstoreReader::iter`, fetched the mappings for the whole range with a single read but then read the values themselves one at a time, serially. Gridstore routes its `iter` through `read_values` and pipelines both stages, so a full scan of an append-only storage was the one read path without batching -- one blocking round trip per value on the object store backends this variant exists for. It is reached by payload storage iteration and by the payload index build, which scans every payload. Feed the pointers into `read_batch_values` instead, keeping the single contiguous tracker read, which is better than the per-offset pipeline scheduling Gridstore does on that side. Values are now delivered through the read pipeline, so the callback may be invoked out of order, as it already could be for Gridstore's `iter` and for `read_values` in both variants. Both segment callers are order independent. Tests that happened to rely on the mmap backend completing reads in scheduling order now sort before comparing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: don't run the failed-page-reload test on Windows The test shrinks a page file out of band to make the page reload fail, but Windows refuses to resize a file while the reader holds it mapped, which it does by construction here: "the requested operation cannot be performed on a file with a user-mapped section open". The panic is on the injection itself, the code under test never runs. There is no portable injection. Truncating a page the reader holds is what the check under test detects, so the mapping cannot be avoided; failing the adopted page open instead needs a listed but unopenable file, and `local_list_files` descends into matching directories rather than listing them; failing the directory listing needs the storage directory removed, which Windows also refuses while pages are mapped. The storage itself is fine on Windows, its append path grows mapped pages there and every other Logstore test passes. The logic under test is platform independent and stays covered elsewhere, with the tracker half of the guarantee pinned by `test_live_reload`, which runs on every target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |
||
|
Daniel Boros
|
3c55c33beb |
fix: fold appended points' deletion into read-only live_reload (#9948)
* fix: fold appended points' deletion into read-only live_reload * fix: batch appended-deletion reads in read-only live_reload * fix: clamp appended-deletion reload to persisted flag length |
||
|
Jojii
|
4efc5ead9f | Rename WrongVectorBytesSize -> MalformedVectorBlob (#9929) | ||
|
Ivan Pleshkov
|
7b14104111 | [TQDT] Fix/upsert raw malformed blob badinput (#9886) | ||
|
Andrey VasnetsovandClaude Fable 5
|
055bcd5c09 |
Add recoverable OutOfAppendableCapacity operation error (#9860)
Preparation for capping appendable segment growth in the update path (#9158): a dedicated error for "all appendable segments reached max_segment_size", so the update pipeline can recognize it and provision a fresh appendable segment before re-applying the operation. Maps to a transient service error at the collection level: if it ever escapes recovery, failed-operation recovery re-applies the operation. Part 1/5 of the appendable segment overflow fix. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Arnaud GourlayandClaude Fable 5
|
22d823cbff |
Use AHashMap for JsonPath-keyed field index lookups (#9854)
Profiling showed SipHash over JsonPath keys (RandomState::hash_one) dominating field index lookups in StructPayloadIndexReadView::estimate_field_condition: the map holds only a handful of indexed fields, but it is queried per condition per filter evaluation per query, and hashing the path is the entire lookup cost. Switch the field index lookup maps (IndexesMap, ReadOnlyIndexesMap, the read view borrow, and the query_checker / condition_converter / value_retriever signatures that receive them) from std HashMap to AHashMap, already used throughout the crate. Serialized and API-level schema maps keep std HashMap: they are cold and their type leaks into conversions. conditional_search bench (criterion, 100 samples, vs saved baseline): - struct-conditional-search-query-points: 583.7 us -> 443.5 us (-23.1% median, p ~ 0) - struct-conditional-search-context-check: 73.2 us -> 64.8 us (-12.6% median, p ~ 0) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Tim Visée
|
7333e6092a |
Universal IO: append to file (#9720)
* universal_io: add UniversalAppend for atomic single-operation appends Growing a file previously took a separate set_len + reopen + write dance (bypassing universal_io and leaving a zero-filled window on crash), and there was no way to express appends for backends without random-offset writes. UniversalAppend::append grows the file by writing at the current end of file in one atomic grow+write operation and returns the offset at which the data landed; append_batch lands multiple buffers contiguously in as few operations as the backend allows. flusher() moves from UniversalWrite into a new UniversalFlush supertrait so append-only handles can require it without duplicating the method. Local backends, both single-syscall: - MmapFile appends through a dedicated O_APPEND fd (every write(2) / writev(2) is an atomic grow+write at EOF), then remaps via reopen(). Its flusher also fdatasyncs after appends, since msync alone does not persist file-size metadata. - IoUringFile appends via pwritev2(RWF_APPEND). O_APPEND is not an option there: on Linux, pwrite on an O_APPEND fd appends regardless of the given offset, which would break positioned writes on clones sharing the fd. Concurrent appenders are out of contract (single logical writer); object-store backends surface the new AppendOffsetConflict error and recover via reopen() + retry. * io_bridge: add AsyncWrite/AsyncAppend and appendable BlobFile Add the write-side backend traits reserved next to AsyncRead: AsyncWrite (create/remove/save) powers a UniversalWriteFileOps impl on BlobFs (create-or-truncate put, delete, atomic whole-object save; directory ops are no-ops), and AsyncAppend — a single-request append where the offset must equal the current object size, acting as a compare-and-swap token — powers UniversalAppend on BlobFile. BlobFile caches the object size across appends (one HEAD for N appends; a missing object counts as empty so the first append creates it), concatenates batches into a single request, and drops the cache on reopen() — the documented recovery path after AppendOffsetConflict. Its flusher is a no-op: appends are durable once the backend acknowledges them. * io_bridge_object_store: native single-request S3 append object_store has no append support, so issue the PutObject + x-amz-write-offset-bytes request ourselves, reusing the store's credential chain (AmazonS3::credentials) and object_store's SigV4 AwsAuthorizer, which signs every header present on the request — no hand-rolled signing and no direct reqwest dependency. The offset doubles as a compare-and-swap token: a mismatch (400 InvalidWriteOffset, or 412 on some S3-compatibles) maps to AppendOffsetConflict. The write-offset append API exists on AWS S3 Express One Zone directory buckets and compatible stores (e.g. MinIO AiStor) — plain S3 Standard buckets reject it, and real Express zonal endpoints / session auth are not verified yet; MinIO-AiStor-compatible stores are the primary target for now. GCS and Azure sources simply do not implement AsyncAppend. ObjectStoreSource carries an AppendContext (HTTP client + object URL base + signing region) built per backend from its config, and gains a generic AsyncWrite impl (single-put create/save, delete). A test-only multi-request CAS emulation over InMemory exercises the BlobFile append stack hermetically; an end-to-end flow against a real append-capable store is gated behind S3_APPEND_INTEGRATION_TEST=1. * simple_disk_cache: write-through UniversalAppend for DiskCache Append to the remote (the single grow+write operation), then write the same bytes through into the local mirror so tail reads do not re-fetch what was just uploaded. LocalState::append_local keeps the fetched bitmap accurate: blocks fully covered by the appended range are marked fetched, and the pre-append partial tail block — which resize() drops because set_len zero-fills its gap — is re-marked only when its prefix was already fetched. If the mirror turns out stale (the remote grew behind our back), append falls back to resize-only and lazy fetches heal the gap on the next read. Writeable opens are now allowed on DiskCacheFs solely to enable append; DiskCache still never implements UniversalWrite. The writeable flag propagates to the remote handle, which is opened buffered instead of O_DIRECT: appends write through the page cache, which O_DIRECT reads on the same fd would fight (and IoUringFile rejects appends on prevent_caching handles). The remote-immutability docs are relaxed to append-only with an immutable prefix, matching what reopen() already assumed. Includes a full-stack composition test: DiskCache write-through over BlobFile offset tracking over an in-memory object store. * io_bridge_object_store: build the append HTTP client lazily Opening a source from an AwsConfig eagerly built the reqwest client (TLS setup, connection pool) even when append was never used. Keep the AppendContext construction to pure config (allow_http flag, object URL base, signing region) and build the client on first append instead, cached in an Arc<OnceLock> shared across clones of the source — and thus across the file handles opened from it. Sources that never append now pay nothing; client-construction errors surface on the first append instead of at open. * universal_io: test that append grows the regular file on disk The conformance suite reads appended bytes back through universal-io handles; also assert the underlying regular file itself — created outside universal_io, verified with plain fs reads — for both local backends. * Mention why we use custom HTTP client, object_store crate has no support * io_bridge_object_store: reject appends unconfirmed by the size header A store without write-offset support may accept the signed PutObject as a plain put — replacing the object with just the appended bytes — and return 2xx (community MinIO did exactly this before 2025-05, commit minio/minio@6d18dba9). The old success path fabricated the new length when x-amz-object-size was missing, so the destruction stayed invisible while every subsequent append repeated it. Require the x-amz-object-size response header (returned by AWS and MinIO AiStor appends) for any append at offset > 0 and fail loudly without it. Offset-0 appends are equivalent to a whole-object write, so they remain valid either way — a misconfigured store now fails on the second append instead of never. * io_bridge_object_store: honor endpoint/region env vars for appends With AwsCredentials::Default the store is built via AmazonS3Builder::from_env, which honors AWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL, AWS_ENDPOINT, AWS_REGION and AWS_DEFAULT_REGION — but append_context derived the append URL and SigV4 region only from the typed config fields. An env-configured deployment would read from one host while signing and sending appends to https://{bucket}.s3.us-east-1.amazonaws.com. Resolve the append endpoint and region the same way build_store does: explicit config first, then (default credential chain only) the same environment variables, with AWS_ENDPOINT_URL_S3 taking precedence as in from_env. The resolution is a pure function over an injected env lookup so the test does not touch process-global environment state. * simple_disk_cache: delegate the append flusher to the remote DiskCache's UniversalFlush impl was an unconditional no-op, justified by object-store appends being durable on acknowledgement — but the impl is generic over any appendable remote, and for local remotes (MmapFile, IoUringFile, exactly the compositions the tests instantiate) that silently dropped the fdatasync the UniversalAppend contract requires: append, flush Ok, power loss, appended bytes gone. Delegate to the remote's flusher once the cache is materialized: local remotes get their sync, object-store flushers remain no-ops, and a never-materialized cache has made no appends so a no-op stays correct. * io_bridge_object_store: retry transient append failures The append RPC was a single unretried HTTP attempt, while every other request in this stack goes through object_store's retry layer — a routine transient 503 SlowDown or connection reset failed the append hard where a concurrent read would have silently recovered. Retry connection errors, 5xx and 429 up to three attempts with a short linear backoff, re-signing per attempt (the SigV4 signature embeds the request date). Retrying is safe because the write offset is a compare-and-swap; the one ambiguity — an attempt that landed but whose acknowledgement was lost — surfaces as a write-offset conflict on the retry, which is reconciled with a HEAD: under the single-writer contract, an object size of exactly offset + data_len proves the tail is ours, so the append reports success instead of a spurious conflict (whose reopen-and-retry recovery would duplicate the record). * universal_io: forward TypedStorage::flusher for any UniversalFlush The flusher forwarding lived in TypedStorage's S: UniversalWrite impl block, so append-only storages (DiskCache, BlobFile — UniversalAppend + UniversalFlush but not UniversalWrite) offered append through the wrapper while the durability flusher the append contract mandates was unreachable without going through .inner. Move it to an S: UniversalFlush block: UniversalWrite implies UniversalFlush, so existing callers resolve unchanged, and duplicating the method instead would have hit E0592 on backends implementing both — the very ambiguity UniversalFlush was extracted to avoid. * io_bridge_object_store: surface unbuildable append requests as errors Request building could panic on two reachable paths: url accepts URIs the http crate rejects (IPv6 zone identifiers, URIs beyond u16::MAX bytes), and AppendContext::new is public so the object URL base is not guaranteed to be a base URL. Both expects become S3Config errors, so a configuration edge case fails the append instead of panicking the thread driving the bridge runtime. * simple_disk_cache: don't fail appends the remote already committed append_impl committed to the remote first and returned Err when the subsequent local-mirror update failed (e.g. ENOSPC on the cache volume) — indistinguishable from "nothing was appended", so a retrying caller would duplicate the record on the remote. The mirror is cache maintenance, not part of the append: on a failed write-through, log and degrade to bare growth so lazy fetches heal the unmarked blocks (safe — blocks are only marked fetched after their bytes landed). Only an unresizable mirror still surfaces an error, and the UniversalAppend contract now documents that an append Err does not guarantee nothing was appended: reopen() and re-check the length before retrying. * universal_io: bounds-check positioned io_uring writes against EOF The UniversalAppend contract states that UniversalWrite::write beyond the end-of-file fails and append is the only growth path — mmap enforces it, but IoUringFile's write/write_batch/write_multi were unchecked pwrites that silently extended the file with a zero-filled hole, inflating subsequent append offsets. Check every positioned write against the file length (fstat once per call), matching mmap's OutOfBounds semantics, and generalize the regression test to run on both local backends including the batched path. * io_bridge: reject appends on handles opened without writeable BlobFs::open dropped OpenOptions entirely, so a BlobFile opened with writeable: false still accepted appends — mmap and DiskCache enforce the writeable requirement, the blob backend silently didn't, and a stray append through a nominally read-only handle would mutate a shared object. Thread OpenOptions::writeable into BlobFile and reject appends with PermissionDenied when it is unset, mirroring the other backends. Directly-constructed handles (BlobFile::new/open, which take no OpenOptions) remain writeable. With every backend now enforcing the flag, the UniversalAppend contract drops its 'where the backend enforces open modes' hedge. * simple_disk_cache: answer empty appends from the mirror An empty append still went through remote.append_batch, so it returned the remote's live end-of-file — which can diverge from what this handle's len() and reads observe when the remote grew behind our back — while leaving the stale mirror unhealed (unlike a non-empty append in the same state, which resizes). Accept empty appends early: return the mirror length without touching the remote at all, keeping the answer consistent with the handle's own view. The trait contract now spells out that empty appends return the handle's view of the end of file without growth I/O. * universal_io: grow the mmap in place after appends Every mmap append ended in a full reopen(): an open+fstat+close by path just to learn the new length, and — on the non-Linux fallback, which rebuilds the mapping with the open-time populate flag — a re-population of the ENTIRE file per append, making appends O(file size) for handles opened with Populate::Blocking. Populating after an append is pointless anyway: we just touched the data we wrote. Extract the remap machinery into remap_to() (reopen() keeps its exact semantics, populate included) and add grow_mapping(): a stat-free grow that never re-populates. Appends learn the new length from a single fstat on the already-open O_APPEND fd — kept rather than trusting the mapping length, which is stale exactly in the externally-grown-remote scenario the disk cache heals through lazy fetches (the foreign-growth tests catch the difference). The mirror's resize() passes the length it just set_len'd, dropping its stat round-trip entirely. Per small append this is write+fstat+mremap, down from write+open+fstat+close+mremap, with no populate anywhere. * universal_io: share the mmap append fd across clones The flusher captured the per-clone append_file at flusher-creation time, so a flusher obtained from a sibling clone — or created before the handle's first append — msynced the shared mapping's data pages but skipped the fdatasync that persists the appended file size: a half-persist where a crash loses the acknowledged tail even though a flusher ran after the appends (writer thread + long-lived flush-worker clone is exactly the natural WAL shape). Store the fd in an Arc<OnceLock> shared by all clones and read it at flush time instead of capture time: any clone's append makes every handle's flusher sync the size metadata, whatever the clone/flusher creation order. Initialization races between clones keep exactly one fd. No hot-path cost: reads and positioned writes never touch the cell, and the append path pays one atomic load next to its syscalls. The interior mutability also lets append_fd take &self. * universal_io: document the clone remap hazard truthfully The remap SAFETY comment claimed moving is safe "since we are holding &mut self" — which says nothing about clones: they share the mapping but keep their own raw ptr/len copies, so after a moving (or, on non-Linux, replacing) remap a sibling clone's next read dereferences an unmapped address. The trait contract understated the same hazard as a concurrent-read constraint, while the UB persists after append returns. State the contract once on MmapFile (clones must reopen before reading after any growth; a stale clone read is undefined behavior, not a stale view), correct the SAFETY argument to rely on it explicitly, annotate the as_bytes unsafe blocks that depend on it, and sharpen the UniversalAppend contract bullet accordingly. Making clones structurally safe (resolving ptr/len through the shared Arc) is deliberately left as a separate change. * universal_io: share the vectored append machinery between backends The IOV_MAX-chunking / EINTR-retry / WriteZero / advance_slices loop existed twice — as local_file_ops::write_all_vectored (mmap) and inlined around pwritev2 in IoUringFile::append_slices — along with a verbatim collect/cast/filter-empties preamble in both append_batch impls. Two copies of subtle short-write handling introduced by one branch will diverge the first time only one of them gets a fix. Add an io::Write adapter whose write_vectored issues pwritev2(RWF_APPEND), letting the io_uring append delegate to the shared write_all_vectored, and hoist the slice collection into local_file_ops::collect_append_slices. IOV_MAX becomes private to the one function enforcing it. No behavior change; the existing conformance tests (including the beyond-IOV_MAX batch) cover both backends through the shared path. * io_bridge_object_store: add s3_express to the test config helper The AwsConfig struct gained the s3_express field; update the resolve-endpoint test helper accordingly. * universal_io: run the append conformance suite over the S3 stack Promote the backend-generic UniversalAppend battery (offsets, batches across IOV_MAX, empty appends, read-after-append, reopen visibility, flusher) from a private test into universal_io::conformance, exposed under the testing feature so backend crates can run the identical suite. mmap and io_uring keep running it as before; the object-store bridge now runs it too, over BlobFs/BlobFile with the in-memory offset-CAS append emulation — so local file system and S3 append behavior are asserted by the same test. The real write-offset RPC remains covered by the gated test_native_append_flow integration test. * Swap order * universal_io: disambiguate the io_uring crate import The import reorder dropped the leading `::`, making `io_uring` ambiguous with this very module (pulled into scope by the `use super::*` glob) and breaking the build. * io_bridge: don't materialize the mock object on rejected appends MutableMockSource::append called get_or_insert_with before validating the offset, so a rejected stale append against a missing object left an empty entry behind (exists() flipping true) — a fidelity gap versus the real backends, where a rejected append has no side effects. Check the offset against the current length first and only materialize the buffer on a match. * universal_io: disambiguate the io_uring crate import Restore the leading `::` on the io_uring crate import — without it the name is ambiguous with this very module, which the `use super::*` glob pulls into scope, and the crate fails to compile. Matches the sibling files (pool.rs, runtime.rs), which already import via `::io_uring`. * io_bridge_object_store: treat 404 under a nonzero append offset as a conflict A missing object while the handle expected a nonzero end-of-file is a stale view (the object was deleted behind our back) — the same situation as an offset mismatch, with the same reopen-and-retry recovery, and it is exactly what the in-memory emulation and the io_bridge mock already report. The RPC path mapped every 404 to NotFound instead, so the three implementations disagreed on the same logical case. Keep NotFound for offset-0 appends, where a 404 is a genuine missing-target error (e.g. a missing bucket) that retrying cannot heal. * Preallocate vector * universal_io: disallow appends through the disk cache Appends must go directly to the backing storage (mmap, io_uring, S3) — the disk cache is strictly read-only again. Remove DiskCache's UniversalAppend/UniversalFlush impls and the mirror write-through machinery (LocalState::append_local), reject writeable opens at DiskCacheFs::open, and drop the writeable/prevent_caching plumbing that existed solely for cached appends, restoring read-only remote handles. Attempting to append through the cache is now a compile-time error (the trait impl no longer exists), and opening a cached handle writeable is rejected at runtime, covered by a test in each backend variant. * universal_io: make append idempotent via caller-supplied offset Append now takes the byte offset where the data must land: append(offset, data) -> Result<()>. Every backend validates that the offset equals the current end of file before writing (mmap and io_uring fstat the fd, object stores validate server-side via x-amz-write-offset-bytes); on mismatch nothing is written and the append fails with AppendOffsetConflict. Retrying an already-landed append therefore conflicts instead of appending twice, and recovery is re-deriving the offset from len(). BlobFile no longer tracks the object length locally; the store's own offset check is the compare-and-swap. * Review remarks * Validate file length in S3 append response * Fix linting, we don't mind a large enum variant on index builder * universal_io: conformance-test stale-handle append conflict recovery Promote the two-handle conflict scenario from the in-memory BlobFile test into the backend-generic conformance battery: a second writeable handle grows the file, the stale handle's append conflicts cleanly (the offset check runs against the file, not the handle's view), and the contract's documented recovery — reopen, re-check the length, append at the real end — lands the data exactly once. Now exercised over mmap, io_uring, and the object-store stack instead of only the in-memory emulation. * io_bridge_object_store: stub-server tests for append response handling The native append's HTTP state machine was only exercised by the gated live-store integration test (S3_APPEND_INTEGRATION_TEST=1), so none of its branches ran in CI. Cover them hermetically against a minimal local HTTP stub — one connection per canned response, no new dependencies: - the signed write-offset PUT, and the new-size validation on success (matching, mismatching, unparseable, and absent size headers — the absent case at offset zero and past it); - conflict mapping for 400 InvalidWriteOffset, 412, and 404 under a nonzero offset, with 404 at offset zero staying NotFound, and a 400 without the conflict code staying a plain error; - 429/5xx retries re-sending the same offset, giving up after MAX_ATTEMPTS, and the lost-acknowledgement reconciliation via HEAD (accepted when the object ends at offset + len, rejected otherwise); - the status + body excerpt on unexpected failures. * simple_disk_cache: statically assert the cache stays read-only Disallowing appends through the disk cache made them a compile-time error by removing the impls; pin that with assert_not_impl_any so the UniversalAppend/UniversalFlush/UniversalWrite impls cannot quietly return. Runtime rejection of writeable opens stays covered per backend variant. |
||
|
Arnaud Gourlay
|
29836c65bd | Miscellaneous cleanups (#9849) | ||
|
|
e1a4f9867d |
Fix live-reload staleness of in-place-mutated files on caching backends (#9812)
* Mutate same-operation slots in place in append-only mode With append_only_mutations enabled, every mutation of an existing point clones it to a fresh internal id. Shard-level updates decompose one point write into several SegmentEntry steps (upsert_with_payload issues upsert_point plus set_full_payload/clear_payload), so a single upsert burned one slot per step, leaving a chain of immediately-dead clones. A slot whose version already equals op_num was written by an earlier step of the current operation. It cannot be durable yet — the segment write lock is held across the whole operation, so no flush (and no read-only follower) can have observed it, and versions flush last so a crash discards it and WAL replay re-applies the whole operation. Mutate such slots in place: one operation now allocates exactly one slot regardless of how many steps it decomposes into. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stamp payload storage version in write_point_parts overwrite_payload mutates payload storage, but the fused write never bumped version_tracker's payload version — the old CoW path did, via set_full_payload. The segment manifest would stamp payload files with a stale version, letting a partial snapshot skip payload storage that contains the moved point's row, so the restored id tracker would point at an offset with no payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync ReadOnlyRoaringFlags from disk on live-reload The flags file is preallocated to power-of-two capacity and mutated in place within its length, which the held handle's reopen() — an append-only-growth contract on caching backends — never picks up: bits the writer changes inside already-cached DiskCache blocks stayed stale forever on shard followers. Drop the `impl LiveReload for ReadOnlyRoaringFlags` altogether: this storage holds arbitrary flags with no notion of points, so a point-delta interface (deleted/new points) did not belong here — open never applied deleted points either. Replace it with an inherent live_reload(fs) that opens a fresh StoredBitSlice (a fresh open always mirrors the current remote bytes), resyncs the materialized bitmap from it, and swaps the handle. The on-disk flags are the sole source of truth. To make refresh-by-fresh-open safe while the old handle is still alive, every DiskCache open now mirrors into a uniquely-named local file (.{pid}-{counter} suffix) and removes it on drop. Mirrors were already truncated on every open (block validity is in-memory only), so the stable name carried no state. Also add trace logging for live-reload consumed mapping changes and pending deltas. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync immutable id tracker deleted bitmap via fresh handle on live-reload The immutable tracker's id_tracker.deleted file is a fixed-size bitmap whose bits the writer flips in place — its length never changes — so the held handle's reopen(), an append-only-growth contract on caching backends, never picked it up: the pre-deletion state cached at open was served forever and live-reload never reported deletions on shard followers. live_reload now takes the fs, opens a fresh StoredBitSlice (a fresh open always mirrors the current remote bytes), diffs it against the tracker's current state — mappings already reflect every previously reported deletion, so they are the baseline — and swaps the handle in. ReadOnlyIdTrackerEnum and the segment-level reload pass the fs through; the appendable and disk-resident variants keep their no-arg reloads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync disk id tracker deleted bitmap via fresh handle on live-reload Same staleness as the immutable tracker: the deleted file is a fixed-size bitmap mutated in place, so reopen() — append-only-growth on caching backends — served the pre-deletion state forever. live_reload now takes the fs, opens a fresh StoredBitSlice, and swaps it into deleted_file, so the per-point get_bit lookups read fresh state from then on too. The deleted_full take/diff/set baseline logic is unchanged. The enum's DiskResident arm passes the fs through. The regression test now covers both trackers over DiskCacheFs; the disk leg was verified to fail under the old reopen-based behavior. The disk tracker's versions file staleness is a separate, still-open case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Introduce header-stateless ReadOnlyTracker for gridstore live-reload The gridstore tracker file is preallocated and mutated in place (header rewrites, slot updates), which the reader's held handle never picked up on caching backends: Tracker::live_reload read the header through the stale handle — before reopen(), and reopen() would not have refreshed cached blocks anyway — concluded "no new pointers", and never reloaded pages. New points' payloads read as empty on shard followers. Replace the reader's tracker with a dedicated ReadOnlyTracker that holds nothing but the storage handle: reads don't need header state (slot addressing is positional, unwritten slots read as None in the zero-initialized file) and readers have no pending-updates buffer. Its live_reload opens a fresh handle and swaps it in. Tracker::live_reload is deleted. A new TrackerRead trait (max_point_offset/get/iter) is implemented by both the writable Tracker and ReadOnlyTracker, and GridstoreView is generic over it, so writer and reader share the read logic. max_point_offset reads the stored header count as plain data through the current handle (fresh as of the last reload) rather than deriving it from slot capacity: sparse vector storage builds total_vector_count from it, and the capacity of the preallocated file (1MB -> 65536 slots) would inflate every follower segment's sparse count. The method is now fallible; consumers propagate the error. GridstoreReader::live_reload refreshes tracker and pages unconditionally — the tracker carries no reliable cheap change signal. Making this incremental again is a deferred follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Re-open last held chunk on chunked-vectors live-reload Chunk files are preallocated to full size, so appended vectors are in-place writes within the existing file length. On caching backends a held handle keeps serving blocks fetched earlier — and a block fetched near the old tail extends into then-unwritten space — so vectors appended into that block read back as stale bytes after a reload. Mirror Pages::live_reload: on a len change (the status file is read through a fresh open every reload, so it stays a reliable gate), drop and re-open the last held chunk — the only one that can have gained vectors — alongside adopting newly created chunk files. Earlier, fully-filled chunks keep their handles; their contents cannot change. Covers dense, multi-dense, and quantized-chunked storages, which all delegate to ChunkedVectorsRead::live_reload. Avoiding the whole-chunk refetch per reload is a deferred follow-up, together with the analogous gridstore pages/tracker case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix lint: fs_err::create_dir_all in tests, drop unnecessary cast CI clippy runs with --all-targets and disallows std::fs methods in favor of fs_err; the new live-reload regression tests used std::fs::create_dir_all directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tombstone-only deletes on non-appendable segments in append-only mode The tombstone-only delete path existed but was gated on is_append_only(), which requires the segment to be appendable. Deletes landing on non-appendable segments — notably the CoW update deleting the point's old copy — still went through delete_point_internal, which clears the payload row in place before the id-tracker drop. Clearing the payload destroys committed state of an offset that stays visible to live-reload followers until the drop is flushed: a follower refreshing in that window resolves the point through its (unchanged) id-tracker view and reads an empty payload — observed as a one-refresh {} payload flicker on a point update. Writer-side flush ordering cannot close the window, since the follower samples the id tracker and the payload storage at different instants; the versions commit protocol protects inserts exactly because the marker is read first, and deletes have no marker. This is the same failure class for which vector-deletion propagation was disabled (see the comment in delete_point_internal). Gate deletes on the new is_append_only_delete() — append_only_mutations alone, no appendability requirement: tombstoning needs nothing from the segment but the id tracker. Normal deployments keep eager payload clearing and prompt space reuse; clone-based mutations keep requiring appendability via is_append_only(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: chunked vectors reload --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Daniel Boros <dancixx@gmail.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
9e6892e272 |
Preserve not-found classification on read-only reload/open paths (#9777)
Follow-up to #9763: an audit of the component live-reload and read-only open paths found four places where a not-found error lost its structured classification before reaching OperationError::FileNotFound: - MmapFile::reopen opened the file with a bare `?`, producing an unstructured Io(NotFound) — and this is the central call on the local follower's live-reload path, so no mid-reload segment removal would ever have classified on the mmap backend. Wrapped with extract_not_found. - BlockCacheFs::open and CachedSlice::reopen had the same bare-`?` pattern over CachedSlice::open's io::Result. Both wrapped. - ReadOnlySegment::open_via hand-rolled a service_error for a missing version file. The version file is written last, so its absence is exactly the vanished-mid-open signal; now FileNotFound { path }. - ReadOnlyRoaringFlags::read_status_len masked NotFound internally, so live_reload silently kept a stale len if the status file vanished mid-reload. The raw reader now propagates; open masks it explicitly with ok_not_found() (works on OperationResult since #9763), and live_reload requires the file. Everything else audited came back correct: lazily-created files (mutable id tracker, mutable-index gridstore) keep masking absence, optional-component probes at open keep ok_not_found, and the object-store bridge already maps missing objects to structured NotFound. New test vanished_segment_classifies_not_found pins both legs end-to-end: opening a missing segment directory and live-reloading a segment whose directory was removed both classify via is_not_found(). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
1bd3ee9b98 |
[UIO] implement VectorStorageReadEnum::preopen (#9780)
* [UIO] implement VectorStorageReadEnum::preopen Schedule background prefetch of every file the read-only vector storages open, wired into the segment's first_preopen between the id tracker and the payload indexes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * polish PR - fully populate deleted bitslices - don't use `.ok_not_found()` - section comments * don't overwrite `populate` in `CachedFs` --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |
||
|
|
138e24cd55 |
[UIO] Better populate of bool and null indexes (#9768)
* propagate `populate` to bool and null `preopen` * derive populate from payload schema * Fix broken doc reference: restore payload_schema identifier Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
e65a5c9ca5 |
Classify structured not-found errors as OperationError::FileNotFound (#9763)
Read-only edge-shard followers need to distinguish "essential segment file
missing because the leader removed the segment mid-reload" (re-check the
manifest, absorb) from real corruption (escalate). The classification
already exists at the universal-io layer (UniversalIoError::NotFound,
mmap MissingFile, both carrying the path), but died at the OperationError
boundary where everything collapsed into ServiceError strings.
- Add OperationError::FileNotFound { path } and route the structured
sources into it: UniversalIoError::NotFound, MmapError::MissingFile,
and GridstoreError::UniversalIo(NotFound) (the route payload-storage
live-reload errors take).
- Implement IsNotFound for OperationError so follower code can classify
(and OperationResult::ok_not_found() works where lazily-created files
are legitimately absent).
- Raw io::Error NotFound intentionally stays ServiceError: it has no
structured path; universal-io wraps not-found at the call site via
extract_not_found, so classification belongs at the source.
- CollectionError maps FileNotFound like a service error, keeping
external API behavior unchanged.
Groundwork for not-found handling in ReadOnlyEdgeShard live-reload.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Andrey VasnetsovandClaude Opus 4.8
|
7e5bb25814 |
Lazy roaring flags bitmap and bool index counts (#9749)
* [AI] make ReadOnlyRoaringFlags bitmap and bool index counts lazy
Opening a read-only segment scanned every flags file end to end:
`ReadOnlyRoaringFlags::open` materialized the whole RoaringBitmap via
`iter_ones()`. Every payload field carries a null index, so this was paid
per field per segment, for bitmaps most queries never touch.
Make the bitmap a `OnceLock`, filled by a scan on first access. Open now
reads only the tiny status file. `ReadOnlyBoolIndex`'s three eager count
fields collapse into one lazily-derived, cached `BoolCounts`; its
`live_reload` refreshes them in place when present and leaves them unset
otherwise, so reloading an index nothing queries stays scan-free.
Propagate the resulting `OperationResult` through `RoaringFlagsRead`,
`PayloadFieldIndexRead::count_indexed_points`, `FieldIndexRead`,
`PayloadIndexRead::{indexed_points, get_telemetry_data}`, `build_info` /
`build_telemetry` and `SegmentEntry::{info, get_telemetry_data}`, out
into shard, edge and collection.
`ram_usage_bytes` stays infallible: an unmaterialized bitmap holds no
RAM, so it reports 0 via the new `bitmap_if_materialized`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] correct `preopen` comment: `open` no longer scans the flags file
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] fix edge examples for fallible `info()`
`EdgeShardRead::info` now returns `OperationResult<ShardInfo>`. The
examples live in their own workspace (lib/edge/publish), so the main
`cargo check --workspace` never saw them.
Every call site sits in `fn main() -> Result<(), Box<dyn Error>>`, so
propagate with `?`. `bm25-search` compiled either way but would have
printed the `Result` rather than the `ShardInfo`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Andrey VasnetsovandClaude Opus 4.8
|
cf19e6bb0a |
[AI] implement ReadOnlyRoaringFlags::preopen (#9747)
Schedule background prefetch of the status and flags files that `ReadOnlyRoaringFlags::open` reads, probing existence through the status file the same way `open` does. Wire it up through the two roaring-flag backed leaves — bool and null index — whose arms in `ReadOnlyFieldIndex::preopen` were hardcoded `false` placeholders. Every field index carries a null index alongside it, so this puts the flags on the live segment-open prefetch path. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Luis Cossío
|
b5a12f1234 |
[AI + manual] cleanup (#9727)
- get rid of `from_file` abstractions - renames: - `CachedFs` to be the implementation - `CachedReadFs` to be the trait |
||
|
|
428e196f7d |
CachedReadFs: prefetch-backed read-only segment opens (#9712)
* Add CachedReadFs: prefetch-backed read-only universal-io filesystem Snapshots the file listing at construction and serves opens from explicitly prefetched handles (take-once, shared across clones via Arc<Mutex>). A non-prefetched open falls back to a direct open on the inner filesystem, panicking in debug builds and warning in release. CachedFile is a transparent wrapper needed to satisfy the bidirectional UniversalReadFs<File = Self> pinning, following the ReadOnly pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serve read-only segment opens from CachedReadFs prefetch pool ReadOnlySegment::open builds a per-segment CachedReadFs: the files known in advance (version.info, segment.json) are scheduled before the listing snapshot is taken so their fetch overlaps the listing round-trip, then every remaining listed file is scheduled, running all fetches in parallel instead of serializing them inside component opens. Existence checks and format-detection probes are answered from the snapshot without touching the inner filesystem. Stored handles are taken out of the pool via the new CachedReadFs::take_file, which returns the raw inner file — component types stay over plain S, and CachedFile exists only transiently inside open-read-discard helpers (read_json_via etc.) through the trait impl. The read-only open path takes &CachedReadFs<S::Fs> concretely; storing wrappers gained from-file constructors (StoredBitSlice::from_file, UniversalHashMap::from_file, ReadOnly::from_file, gridstore Tracker::open_cached / Pages::open_cached, read_chunks_cached). Snapshot-less, CachedReadFs is a passthrough to the inner filesystem — used by reload paths, writable build paths that reuse the on-disk index opens, and tests, all of which keep their previous behavior. Components that retain a filesystem for later reloads store the raw inner backend (CachedReadFs::inner), never the stale snapshot. Also: local_list_files now recurses into subdirectories, matching the flat key-prefix semantics of object-store listings; the immutable id tracker probes its defining file via exists (free on the snapshot) instead of a probe-open that would consume the take-once handle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Group imports per nightly rustfmt Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix sparse search bench for open_ro over CachedReadFs CI clippy runs --all-targets; the bench target was missed by the --tests sweep. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Match listing prefixes by path component, not by string The recursive local_list_files compared whole path strings; on Windows a joined prefix mixes `/` and `\` (`shard\index/chunk_`) while walked entry paths use `\` throughout, so nothing ever matched (broke list_files_returns_paths_relative_to_shard_dir on Windows CI). Match the entry name at the prefix's final position against the prefix's final component instead, then walk matched directories exhaustively — same semantics, separator-agnostic. Apply the same component-based matching to the CachedReadFs snapshot filter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * do not cache everything * fmt * dont unwrap files_info * fix clippy * relax debug assertion for now * [AI] refactor into extension trait, relax Fs<->File requirement (#9725) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |
||
|
xzfc
|
bc2c40d93c | Tidy up Debug impls (#9653) | ||
|
Luis Cossío
|
dd5271bc6d |
Explicit Populate in payload storage (and ` (#9514)
`GridstoreReader`) - Make the entrypoint of payload storage choose the `Populate` variant - Mutable payload indexes now populate gridstore blockingly before using it to load - Propagate `Populate` into `flags` module - Add `UniversalRead::populate_auto` to know whether a backend chooses to populate or not when `Populate::Auto` |
||
|
Daniel Boros
|
eb91244daa | feat: add live-reload for immutable dense vector (#9392) | ||
|
Luis Cossío
|
8beb3a08cf |
[UIO] Sorted offsets in LiveReload (#9385)
|
||
|
Daniel Boros
|
e0a1548fdb | feat: add in-memory bitvec flags (#9333) | ||
|
Thorsten Sommer
|
9b342cc120 | Fixed type parameters in anonymize_collection_values call (#9312) | ||
|
Daniel Boros
|
f682e9e396 | feat: implement LiveReload for bool index (#9299) | ||
|
Daniel Boros
|
fc3406ba41 |
feat: implement LiveReload for read-only roaring flags (#9329)
* feat: implement LiveReload for read-only roaring flags Make ReadOnlyRoaringFlags<S> implement the shared LiveReload trait (open retains the StoredBitSlice; reopen + read only the changed positions on reload), threading the backend S through the read-only bool/null index structs, the field-index enum and the facet enum. The per-index reload impls live in the bool/null branches stacked on top. * fix: review commits * feat: add remark comment |
||
|
Daniel Boros
|
05f6948cfa | refactor: move LiveReload trait to common module (#9314) | ||
|
Daniel Boros
|
0e244662d3 |
feat: detect not-found via error in bool, null and geo index open (#9265)
* feat: detect not-found via error in bool, null and geo index open * fix: error on inconsistent storage in bool and null index open * fix: hw counter write -> read |
||
|
Daniel Boros
|
0033a867cf |
feat: remove generic in Roaring flag & move into open method (#9236)
* feat: add readonly null index open * feat: add open method * refactor: split variants into modules * chore: remove duplicated impl * feat: remove generic in Roaring flag move into open method |
||
|
Daniel BorosandLuis Cossío
|
ada6e82157 |
feat: add readonly null index open (#9197)
* feat: add readonly null index open * feat: add get_mutability_type to ReadOnlyNullIndex * fix: pr reviews * simpler phantomdata --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |