mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-29 01:17:56 -05:00
Andrey VasnetsovandClaude Opus 5
1d18e46551
* [UpdateOnly] split UpdateOnlySegment into its lookup and writer phases Applying a batch runs in two phases that agree on almost nothing, and `UpdateOnlySegment` was both: `resolve.rs` used every field, `append.rs` used none of them and could not — a `ReadOnlyPayloadStorage` has no append path. The `fs` field existed only for the writes that were never wired up. Split along that line: * `LookupSegment` (was `UpdateOnlySegment`) is the read phase. Every segment of a shard is opened as one, on read-only bounds, and the phase above them aggregates. Loses the dead `fs` field. * `DeleteOnlySegment` and `AppendableSegment` are the write phase, one segment each, `UpdateOnlySegmentEnum` over the two. Opened for one batch and dropped with it, matching the append-only components, which buffer nothing across calls. The phases meet at `SegmentWriterState`, produced by `LookupSegment::writer_state` and consumed by `UpdateOnlySegmentEnum::open`. It carries the mappings-log tail an appendable writer resumes from, which `UpdateOnlyAppendableIdTracker::new` requires to come from one and the same read of that log. The writer kind follows the id-tracker format that was loaded, not the segment config: the format decides how a point is retired. That difference makes `tombstone_points` take both ids, `(external, slot)`; an immutable segment marks the slot in its deleted-points bitmask, an appendable one records a retirement for the id in its mappings log. The appendable half is implemented — deletes now run end-to-end. `store_points` and the immutable bitmask remain `todo!()`, still waiting on the append-only storages and field indexes. Two bugs surfaced while wiring it up: * A point stored into the write target must not have its old slot retired there: appending records a mapping that supersedes it, and retiring the id on top would take the new slot with it. * A second `apply_batch` through one writer resurrected deleted points. It resumed the log from the `mappings_end` its own first batch had moved past, and appending there cut that batch's entries off. Refused now; lifting it means reloading the segments after a batch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] one batch per writer, enforced by the type system Cleanup pass over the phase split. `apply_batch` now takes `self`. It could only ever serve one batch — the segments are read when the writer opens, and that read is both what a batch resolves against and what its writers resume from — and the runtime guard enforcing that cost a flag, its doc, two imports, a hand-maintained `writes_anything` condition, an error branch and a test. Consuming the writer makes the second call a compile error instead. Also: * drop `LookupSegment::uuid`, which nothing ever read, along with the two parameters and the argument that fed it; * `AppendableSegment::tombstone_points` was a copy of the tracker's own `retire_pending_inserts`; both now go through `delete_points`; * fold the duplicated "segment disappeared mid-batch" error into `LookupSegmentHolder::get`, and restore `write_target_uuid` as an `Option`, which is what two of its three callers wanted; * one fixture helper for the writer tests instead of three copies; * state the mappings-log co-read invariant once, with pointers, instead of three times. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] cut the writer surface down to what it does * `flush()` is gone from both writers and the enum. Both bodies were `Ok(())` and would stay that way: the id tracker persists what it writes before returning, and the deleted-points bitmask writer does not exist yet. The ordering it looked like it enforced — new slots durable before the tombstones retiring the old ones — falls out of call order, since every write is durable when it returns. Bring it back with the first storage that buffers. * `SegmentWriterState` was an enum of one unit variant and one payload, which is `Option`. `writer_state()` returns `Option<AppendableIdTrackerState>`, and `None` reads as what it means: no mappings log to resume, so a delete-only writer. * `LookupVectorData` wrapped a single `Arc<AtomicRefCell<_>>`; the map holds it directly now. * `appendable` joins the five `pub` fields around it, and `is_appendable()` goes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This dir contains a Justfile with recipes to build/check/run examples for edge packages. Install https://github.com/casey/just to use it.
Rust Qdrant Edge package
Rust Qdrant Edge workspace lives in the publish directory. It's a separate workspace, not tied to the main workspace.
The qdrant-edge package is autogenerated by the amalgamate.py script and placed in the ./publish/qdrant-edge directory (gitignored).
If you need to make changes in the qdrant-edge package, edit the original packages in this repo (/lib in the repo root).
just rs-examples
# Or, manually:
./amalgamate.py
cargo check -p examples
cargo run -p examples --bin demo
cargo run -p examples --bin …
(see full binary list in publish/examples/src/bin)
How to publish Rust Package to crates.io
- Update the
VERSIONin publish/amalgamate.py. - Run Qdrant Edge Rust Release workflow.
Python Qdrant Edge package
It's in the python directory. It's part of the main workspace, unlike the Rust package.
just py-build
just py-examples
Or, manually:
# Setup environment
cd lib/edge/python
python -m venv .venv
source .venv/bin/activate
pip install --user maturin
# Build and install the package:
cd lib/edge/python
maturin develop --no-default-features
# Run example:
python examples/demo.py