mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-25 07:27:41 -05:00
Andrey VasnetsovandClaude Sonnet 5
309c64945a
* [UpdateOnly] wire every component into the appendable segment `AppendableSegment::store_points` sheds its `todo!()`: the id tracker claims a fresh slot per point, every component writes its data at those slots — each named vector storage, the payload storage, the payload indexes — and only then do the versions cover them, the step that makes the points visible to readers. A crash anywhere in between leaves claimed, unpublished slots, which the next writer to open the segment retires. Each vector comes from whichever half of `FullyQualifiedPoint` holds it: the batch's decoded vectors win over the bytes carried from the point's previous slot, and a name in neither still takes its slot as a vector the point does not have. The store components open lazily, on the first `store_points`. A batch that only deletes writes nothing but the mappings log, so it never pays for those opens — and it keeps working against segments whose payload storage was created in mutable mode, which the append-only writers refuse and which is all any leader builds today. The writer now also remembers what it stored, so `tombstone_points` skips a point this very batch wrote instead of retiring its fresh slot; the caller can hand over every slot a stored point used to occupy without holding that rule. `UpdateOnlySegmentEnum::open` takes the segment config, which is where the writer learns which vector storages exist. The end-to-end edge tests now run stores the whole way through: located and resolved through the `LookupSegment`s, appended by the writer, and read back through an ordinary follower — a new point with its payload, a rewrite winning over the old copy, a replayed batch skipping on the published versions, and a second writer resuming every component where the first ended. The leader still writes its payload storage in mutable mode, so the tests recreate it empty in append-only mode, standing in for segment creation wiring that does not exist yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] drop the stored-ids guard from `tombstone_points` The caller already never asks to retire a point its batch stored — it has to hold that rule regardless, since `preview` mirrors it to count outcomes — so the writer-side set was redundant state, and it made `tombstone_points` silently drop requests instead of honoring a stated contract. The contract is now stated: only points the batch deletes go here, because a delete addresses the external id and would take a stored point's fresh slot along with the stale one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] gate the store tests off Windows The leader's writable storage preallocates chunk files, and the append-only writer cuts them back to end at the data — its append offset is a compare-and-swap token, so a file longer than the data would make every append conflict. That cut replaces the file, which Windows refuses while the writer's own `LookupSegment`s hold it memory-mapped; on Linux the old inode simply lives on under the mappings. Nothing to fix in the writer: Windows cannot shrink a mapped file, and the production target is object storage, where neither preallocation nor mmap exists. The delete tests keep running everywhere; the store tests move into a `#[cfg(not(windows))]` module together with the imports and helpers only they use, so the Windows build carries no unused-import warnings. Cross-checked with `--target x86_64-pc-windows-msvc`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] wire the quantized overlay into StoreComponents Opens UpdateOnlyQuantizedVectors alongside each dense, non-multivector, non-Turbo4-datatype vector's raw storage, when the segment's quantization config supports incremental appends (Binary/Turbo). Multivector and Turbo4 combinations are out of scope (see UpdateOnlyQuantizedVectors' own doc comment) — such a vector simply has no quantized overlay entry and stays searchable exactly through its raw storage alone, same as before. store_points keeps the overlay's row count in exact lockstep with the raw storage: every point takes a row in both, in the same order, at the same id (start_slot + offset) — a decoded vector encoded for real, a Raw-bytes-carryover blob decoded back to f32 per its actual storage datatype (mirroring QuantizedVectors::create_impl's use of PrimitiveVectorElement::quantization_preprocess for the same purpose on the non-update-only path), and a Missing vector as an all-zero placeholder. Skipping a row for the latter two cases would silently misalign every later quantized lookup — scoring one point's vector against another's quantized copy — so this mirrors the raw storage's own "every point takes its slot" rule exactly rather than only handling the common decoded case. UpdateOnlyQuantizedVectors now retains its resolved QuantizedVectorsConfig (exposed via quantization_config()/dim()) rather than discarding it after opening storage, since a reopened overlay's persisted config is the source of truth for how to decode carried-over bytes — not necessarily identical to whatever live config the caller has to hand. Its now-unused flusher() is dropped: like every other update-only storage in this stack, a write is already durable when append_many/upsert_vector returns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This dir contains a Justfile with recipes to build/check/run examples for edge packages. Install https://github.com/casey/just to use it.
Rust Qdrant Edge package
Rust Qdrant Edge workspace lives in the publish directory. It's a separate workspace, not tied to the main workspace.
The qdrant-edge package is autogenerated by the amalgamate.py script and placed in the ./publish/qdrant-edge directory (gitignored).
If you need to make changes in the qdrant-edge package, edit the original packages in this repo (/lib in the repo root).
just rs-examples
# Or, manually:
./amalgamate.py
cargo check -p examples
cargo run -p examples --bin demo
cargo run -p examples --bin …
(see full binary list in publish/examples/src/bin)
How to publish Rust Package to crates.io
- Update the
VERSIONin publish/amalgamate.py. - Run Qdrant Edge Rust Release workflow.
Python Qdrant Edge package
It's in the python directory. It's part of the main workspace, unlike the Rust package.
just py-build
just py-examples
Or, manually:
# Setup environment
cd lib/edge/python
python -m venv .venv
source .venv/bin/activate
pip install --user maturin
# Build and install the package:
cd lib/edge/python
maturin develop --no-default-features
# Run example:
python examples/demo.py