mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-25 07:27:41 -05:00
Andrey VasnetsovandClaude Opus 5
c8261beaec
* [UpdateOnly] implement `UpdateOnlyPayloadStorage` The payload write half for the update-only segment writer: a short-lived storage opened for one batch and dropped with it, over a backend that only appends. Backed by a `Logstore` — the append-only mode of the same storage the writable `PayloadStorageImpl` uses — so a slot's payload is written once and never rewritten. `append_many` takes one payload per point at the slot the ID tracker claimed for it and flushes, so a batch is durable when the call returns and nothing is buffered across calls. Puts only buffer, so the flush is what touches the files: one append per touched page file plus one to the tracker, regardless of how many points the batch holds. A point with an empty payload is skipped, since an unwritten slot already reads back as an empty payload, and so is any gap between slots, which the tracker materializes as unmapped entries. `Logstore` had to leave the `Blobstore` facade for this: `Blobstore`'s type is bound at `UniversalWrite + UniversalAppend` for the sake of its `Gridstore` variant, so it cannot be named on a backend that only appends. Its cross-crate surface is `open_or_create`, `put_value` and `flusher`, nothing more; the new `open_or_create` mirrors `Blobstore`'s and rejects a storage created in mutable mode rather than opening it. Not wired into `AppendableSegment` yet — `store_points` stays `todo!()` until the vector storages and field indexes exist, as with `UpdateOnlyChunkedVectors`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] trim the doc comments on the payload storage writer Keep the guarantees and the non-obvious rationale, drop the restatements — the merged-baseline style of `UpdateOnlyChunkedVectors` and `AppendableSegment`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
blobstore
Storage for variable-sized values, provided by the outer Blobstore type.
Operates in one of two modes, specified when creating a storage and selected automatically when opening one, based on the persisted config:
- mutable (default, the
Gridstorevariant): read-write storage with in-place space reuse, backed by memory mapped files. - append-only (the
Logstorevariant): append-only storage for serverless deployments, one atomic append per file per flush.
Concepts shared by both modes:
- IDs (point offsets) are sequential integers, starting at 0.
- Value data is stored in page files.
- Values are compressed with lz4 (configurable).
- A tracker maps each point offset to page + offset within the page + value length. The offset counts blocks in mutable mode and bytes in append-only mode. The tracker is updated in-memory, and only persisted on flush.
- Supports multiple threads reading and single thread writing.
Mutable mode (Gridstore)
- The storage is divided into file pages of fixed size (32MB by default), mapped into memory using mmap and preallocated.
- Data units are blocks of fixed size (128 bytes by default); values span an integer number of contiguous blocks.
- Data can be written and read across multiple pages.
- Each block is mapped to a bit in the bitmask.
- A region is a fixed number of contiguous blocks.
- Gaps of free blocks in each region are tracked in a file.
- Deletes mark the block as deleted (in-memory) & updates their region
- Updates:
- not done in place, always a new value is inserted
- calculation of the new regions gaps is done on the fly
- One file per page, one file for tracker, one file for bitmask, one file for gaps, and one config file:
| file | content |
|---|---|
config.json |
storage config, "mode": "mutable" |
tracker.dat |
header with mapping count + mapping slots, preallocated |
page_{n}.dat |
value data, preallocated to the page size |
bitmask.dat |
one bit per block: used or free |
gaps.dat |
per-region free block gap summaries |
Append-only mode (Logstore)
Designed for serverless environments, which restrict IO: files can only be appended to, existing bytes can never be rewritten (preallocated zero padding cannot be filled in later), and IO is expensive so as few files as possible are used. All files are read and written through the configured universal IO backend.
- Values cannot be updated or deleted, and must be put at monotonically increasing point offsets. Violating puts and deletes are rejected.
- No blocks, bitmask, gaps or regions: space is never reused.
- Value data is packed back to back in page files, without alignment: each value starts right after the previous one. There is no preallocation and no padding, so a file's length always matches the end of its last appended value.
- Once appending a value would grow the current page beyond the configured page size, a new page is started, bounding the size of and the number of appends to each file (object stores limit appends per object). A value larger than the page size gets a page of its own; values never span pages.
- The tracker file is a plain array of 16 byte mapping entries without any
header: the entry index is the point offset, and the mapping count is
defined by the exact file length. Skipped point offsets are backfilled as
zeroed entries, which decode as
None. - Puts buffer both the value data and the mapping in memory; only a page rollover touches disk between flushes, by creating the new, empty page file. Reads transparently serve buffered values.
- Flushing appends all buffered value data to the page files with a single write per page and syncs them, then does the same for the pending mappings in the tracker file. A mapping on disk therefore never points at value data that is not durable. A flush with a stale target is a no-op, appended bytes are never written twice.
- A write may be torn. If the tracker file length is not a multiple of the entry size, the trailing partial entry is ignored when reading, and truncated away when opening writable.
- One file per page, one tracker file and one config file, with names distinct from the mutable mode so that one mode never attempts to load the incompatible file format of the other:
| file | content |
|---|---|
config.json |
storage config, "mode": "append_only" |
log_tracker.dat |
mapping entries, exact length = count * 16 |
log_page_{n}.dat |
value data, exact length = end of last value |
TODOs
- dictionary compression to optimize payload key repetition
- validate the usage with a block storage via HTTP range requests