* Add shared qdrant-edge-ffi crate with UniFFI bindings
Introduce a shared FFI crate that wraps Qdrant Edge's core types with
UniFFI attributes so the same Rust source can power both the Swift and
Kotlin bindings. The crate lives at lib/edge/ffi/ and exposes ~60 public
types (EdgeShard, Point, Query, Filter, UpdateOperation, …) plus their
enum / record / sealed-union variants.
- lib/edge/ffi/src/ UniFFI-wrapped domain types (config, types,
filter, query, update, error, lib)
- lib/edge/ffi/bindgen/ Separate crate housing the uniffi-bindgen CLI
(needed so consumers of the `uniffi` runtime
don't have to depend on its CLI feature)
- lib/edge/ffi/uniffi.toml Sets the generated Kotlin package to
tech.qdrant.edge.ffi (Swift uses the crate
name verbatim)
Every public type carries Rust doc comments that UniFFI propagates to
Swift Quick Help and Kotlin KDoc, so the generated bindings ship with
first-class IDE documentation.
Workspace changes:
- Cargo.toml Adds the new crates as workspace members and
introduces a `release-mobile` profile
(thin LTO, codegen-units=1, strip=symbols,
panic=abort) for size-conscious mobile builds
- Cargo.lock Pins uniffi 0.31 and its transitive dependencies
Made-with: Cursor
* fix(edge-ffi): harden FFI boundary, add tests, quantization parity, optimize/HNSW
Builds on @ivan-afanasiev's qdrant-edge-ffi foundation (preceding commit) — takes
it to a tested, safe, reviewable state. Split out of #9359 per maintainer request
so the FFI crate can be reviewed in isolation; Swift/Android SDK PRs stack on top.
Boundary safety (host input → catchable error, never a process abort):
- release-mobile profile switched to `panic = "unwind"` so UniFFI's catch_unwind
turns a panic into a catchable error (abort would risk WAL/segment consistency
on an on-device DB).
- Fallible boundary conversions reject bad input (UUID, geo, JSON path, payload
JSON, contradictory match filters) with InvalidArgument instead of panicking.
- Host-supplied counts bounded: limit/offset (bounded_limit, 1 Mi cap), vector
size (1..=65536), HNSW params (m/payload_m ≤ 2048, ef_construct 4..=100000,
max_indexing_threads ≤ 1024) — these drive eager allocation / thread spawning
at optimize(), so unbounded values would abort uncatchably.
API:
- Quantization parity with the Python Edge SDK: all four strategies
(Scalar/Product/Binary/Turbo) accepted; HnswIndexConfig + optimize() exposed
(without optimize() search is brute-force).
- config() is an honest "as-requested" read-back (HNSW + quantization round-trip).
- EdgeError stays branchable (ShardClosed / InvalidArgument / OperationError;
field is `reason`, not `message`, to avoid the Kotlin Throwable collision).
edge core (required by the boundary):
- EdgeShard::flush is fallible (OperationResult) instead of panicking on lock
contention; Drop logs a flush error instead of aborting; python flush()? updated.
- scroll.rs drops a with_capacity(limit) pre-alloc a huge limit could turn into an
allocator abort (defense-in-depth alongside bounded_limit).
Tests (CI: cargo +nightly test -p qdrant-edge-ffi): 4 unit + 22 conversion +
18 integration — persistence, crash-recovery, payload round-trip, concurrency,
delete-reload, search ranking, scroll pagination, quantization accept + config
round-trip, HNSW optimize, boundary rejection.
* fix(edge-ffi): validate geo radius/rings and reject empty field conditions
Three boundary-validation gaps surfaced in review of #9374, all rejected
now with EdgeError::InvalidArgument instead of producing wrong results or
reaching a panic in the geo index:
- GeoRadius: a negative or non-finite radius passed straight through to
the geo index. Reject !is_finite() || < 0.0.
- GeoLineString rings (exterior + interiors of a GeoPolygon): the segment
type was built by direct struct literal, bypassing the engine's
validate_line_string (which only runs on the serde path). A malformed
ring (<4 points or unclosed) could panic in the geo index on indexed
payloads. Mirror validate_line_string at the single GeoLineString
conversion chokepoint, covering both exterior and interior rings.
- FieldCondition: a condition with no predicate set is a silent no-op
(matches every point). Reject it, mirroring the engine's
validate_field_condition. This is the engine/gRPC/REST/Python contract
of "at least one" predicate -- NOT "exactly one"; multiple predicates
remain valid and AND together. The doc comment is corrected accordingly.
Adds 9 conversion tests (geo radius negative/NaN/infinite/valid, ring
too-few/unclosed/bad-interior, field-condition no-predicate/multiple).
cargo +nightly test -p qdrant-edge-ffi: 53 green.
* fix(edge-ffi): adapt to memory/idf API and fallible info after rebase
Keep FlushMode::Sync from recent segment-holder changes while preserving
fallible flush. Fill newly required memory/idf fields (matching the Python
edge bindings) and propagate EdgeShard::info()'s OperationResult.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(edge): disable checkout credential persistence in edge-test
actions/checkout persists GITHUB_TOKEN into .git/config by default. This
workflow runs on pull_request from any branch and then builds and runs
repository-controlled code (Rust/Python examples), which could read or
exfiltrate that token. The job never pushes, so drop the persisted
credentials with persist-credentials: false.
Addresses a CodeRabbit security finding on the edge-ffi PR.
* fix(edge): block on lock in flush instead of failing on contention
flush() used try_lock()/try_read() and returned a 'lock busy' OperationError
when a concurrent update/optimize held the WAL or segment lock. That branch is
the wrong trade-off: callers of flush() expect their data persisted, and the
one spot it would fire on the direct Rust path is exactly when an in-flight
update is holding the WAL lock across its whole operation — i.e. when there is
unflushed data most worth persisting. At the FFI boundary it is moreover dead
code, since the outer Mutex<Option<EdgeShard>> already serializes every call.
Switch to blocking lock()/read(), matching the semantics update() and optimize()
already use on these same locks. flush() stays fallible so a genuine WAL/segment
I/O error is still surfaced rather than panicking. Drop cannot contend (it needs
&mut self, so no &self borrow can hold the locks), so it will not hang.
Addresses a CodeRabbit review nitpick on the edge-ffi PR.
* fix(edge-ffi): harden vector/query boundary from multi-agent review
Addresses findings from a multi-agent review of the FFI boundary:
- Multivector conversions were infallible: an empty outer Vec panicked in
release (MultiDenseVectorInternal::new_unchecked only debug_asserts) and a
ragged matrix was silently reshaped against row[0].len(), storing data the
host never sent. Make NamedVector/Vector -> persisted conversions TryFrom and
validate the matrix (non-empty, uniform non-zero row width) with the same
rules as the engine's try_from_matrix, returning InvalidArgument.
- Reject non-finite (NaN/inf) vector components at ingest, mirroring the geo
is_finite guard. The engine validates only dimensionality, so a poisoned
component would be stored and later serialized back as JSON null silently.
- unload() now returns Result: it flushes explicitly and surfaces a final fsync
failure instead of only reaching Drop's log line (no default log sink exists).
On error the shard stays loaded so the host can retry.
- Cap filter/prefetch nesting depth (MAX_QUERY_NESTING_DEPTH). Condition::Filter
and nested Prefetch are self-recursive; an unbounded host tree would overflow
the stack — a SIGABRT that panic=unwind cannot catch. Reject deeper trees as
InvalidArgument via depth-threaded conversion helpers.
- Replace the '""'-on-serialization-failure fallback in payload/vector JSON
encoding with .expect (serialization is infallible; fail loud, not silent
invalid JSON).
- Docs: correct the flush() # Errors (can return OperationError), the edge-core
flush() caller enumeration, upsert_points/update_vectors # Errors (vector
validation), and reword the fictional lib/edge/VERSION / version-sync comment
in ffi/Cargo.toml to reflect that no automated check exists yet.
* test(edge-ffi): add behavior coverage for vectors, filters, query, updates
Adds 18 integration tests closing gaps a multi-agent review flagged (the suite
proved type conversion but not behavior):
Safety (back the new boundary validation):
- multi_vector_invalid_matrices_rejected — empty/ragged/zero-dim multi-vectors
- non_finite_vector_components_rejected — NaN/inf across single/named/multi/sparse
- finite_vectors_accepted_by_upsert_constructor — over-rejection guard
- deeply_nested_filter_rejected_shallow_accepted, deeply_nested_prefetch_rejected
— depth cap rejects >64, accepts shallow
Behavior:
- filter_restricts_count_scroll_and_search — a filter actually narrows the result
set across count/scroll/search (not just that conversion succeeds)
- flush_under_concurrent_upserts — flush() blocks under a concurrent update loop,
no panic, final count == successful upserts
- vector_content_round_trips_through_retrieve_and_search
- cosine_distance_ranks_by_direction, euclid_and_manhattan_rank_nearest_first
- delete_points_by_filter / update_vectors / delete_vectors / delete_payload /
clear_payload — the five previously-untested update ops
- multivector_round_trips, sparse_vector_round_trips
- rrf_fusion_over_prefetches_returns_fused_set
Suite: 4 unit + 32 conversion + 36 integration = 72, all green.
Not covered (blocked by FFI surface, tracked for follow-up): facet() and OrderBy
scroll both need a payload index, and UpdateOperation exposes no create-index
constructor.
* fix(edge): avoid lost-update TOCTOU in set_vector_hnsw_config
set_vector_hnsw_config read().clone()'d the config, mutated the clone, then
write(|c| *c = cfg) overwrote the whole config. The read lock is released before
the write, so a concurrent config update between the two is silently discarded
(lost-update TOCTOU). Use SaveOnDisk::write_optional to run the fallible mutation
on a clone inside the held lock, returning None on failure to abort persist+swap
without overwriting — the atomic pattern the repo's SaveOnDisk learning
prescribes for fallible config mutations.
Addresses a CodeRabbit critical finding on the edge-ffi PR.
* fix(edge-ffi): adapt read requests to edge::* structs after #9901 rebase
#9901 (Edge: request-specific structures for EdgeShardRead) moved the read API
off the shard/core request types onto edge-owned request structs. Retarget the
FFI conversions accordingly:
- QueryRequest/SearchRequest/CountRequest/ScrollRequest/FacetRequest/Prefetch
now convert into edge::{QueryRequest,SearchRequest,CountRequest,ScrollRequest,
FacetRequest,Prefetch} (were ShardQueryRequest/CoreSearchRequest/*Internal).
Field shapes are identical except score_threshold, which is a plain ScoreType
(f32) on the edge structs, not OrderedFloat — drop the wrap. Nesting-depth
guard and bounded_limit validation preserved.
- retrieve() builds an edge::RetrieveRequest and calls the new single-argument
EdgeShardRead::retrieve (was a 3-arg call).
- Cargo.lock reconciled onto dev's lockfile + the FFI/uniffi deps (dev advanced
23 commits incl. dependency bumps).
cargo +nightly test --locked -p qdrant-edge-ffi: 4 unit + 32 conversion + 18
integration all green.
* feat(edge-ffi): full engine coverage — restructure, all update ops, all scoring queries, missing shard methods
Restructure the crate to mirror the edge crate's layout: shard.rs owns the
EdgeShard object and lifecycle, ops/ has one file per read operation
(request/response records + conversions + exported method together), and
update construction stays in update.rs. Multiple #[uniffi::export] impl
blocks merge cleanly in the generated bindings.
Interface modernization:
- retrieve() takes a RetrieveRequest record mirroring edge::RetrieveRequest
- config surface moves off the deprecated always_ram/on_disk booleans to a
Memory placement enum (cold/cached/pinned); read-back resolves legacy
flags via memory_placement(), None-preserving for HNSW
- EdgeShard.inner is RwLock<Option<...>>: operations take the read half and
run in parallel; unload/update_from_snapshot take the write half and
drain in-flight requests
- request/config records carry #[uniffi(default = ...)], so generated
Swift/Kotlin constructors get default arguments (count exact=true, facet
limit=10, HNSW 16/100/10000/0, everything optional defaults to nil/null)
- all conversions destructure their source exhaustively; intentionally
unexposed internal fields are named `field: _` with a why-comment
Full update-operation coverage (Python SDK parity):
- upsert_points gains condition/update_mode (conditional upsert),
update_vectors gains condition, set_payload gains a JSON-path key
- new: delete_vectors_by_filter, set/overwrite/delete/clear payload
by-filter forms, overwrite_payload, create/delete_field_index (with a
PayloadSchemaType enum), create_dense_vector, create_sparse_vector,
delete_vector_name
Full scoring-query coverage:
- Query::Nearest takes a NamedVector (dense/sparse/multi-vector search)
- new Query variants: Recommend (BestScore/SumScores), Discover, Context,
Feedback; new ScoringQuery variants: Formula (recursive Expression
object with validating, depth-capped constructors) and Mmr;
Fusion::Rrf gains weights
Missing shard methods: query_groups, search_matrix, create, path,
snapshot_manifest, update_from_snapshot (full + partial recovery),
set_hnsw_config, set_vector_hnsw_config, set_optimizers_config.
Two compile-time coverage maps (update.rs, ops/query.rs) exhaustively
match the engine's operation/query enum trees with no wildcard arms, so a
new engine variant fails compilation in this crate until the FFI surface
decision is recorded. A staging passthrough feature keeps them exhaustive
when shard/staging is enabled. The scoring-query map immediately caught
the otherwise-missed Mmr variant.
Tests: 82 total (4 unit + 32 conversion + 46 integration), including new
end-to-end coverage for field-index-enabled facet, conditional upsert,
overwrite/clear-by-filter, recommend/discover/context/MMR, formula
re-scoring, grouped queries, search matrix, vector-name ops, and the
lifecycle additions. Kotlin+Swift binding generation verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(edge-ffi): cover the remaining enum boundaries — filter tree, formula, schema, selections
Add coverage maps for the four construction-only enum families that had no
exhaustiveness guard, and expose everything they revealed as missing:
Filter surface (the big one — filter.rs map over Condition/Match/
RangeInterface/AnyVariants/ValueVariants):
- Condition::Slice — deterministic id-space slice filter, with the
serde-path total/index validation re-applied at the boundary
- Condition::Nested — nested-object array filters, depth-counted against
the recursion cap
- Match::TextAny / Match::Phrase / Match::Prefix — the three previously
unreachable text-match modes
- FieldCondition.datetime_range — RFC 3339 datetime ranges (mutually
exclusive with the float range, matching the engine's single
RangeInterface slot)
- Filter.min_should
- WithPayload::Exclude — exclude-style payload selection (types.rs map
over WithPayloadInterface/PayloadSelector/WithVector)
- OrderBy.start_from — Integer/Float/Datetime cursor (mapped in the
scoring-query coverage map)
- CustomIdChecker recorded as intentionally unexposed (serde-skipped,
runtime-internal)
Formula (formula.rs map over ExpressionInternal + DecayKind): all 17
expression variants were already constructible; the map now forces a
decision when the engine grows a new one.
Field-index schema and vector-name config (update.rs map extended):
PayloadFieldSchema::FieldType covered per schema type; the per-type
FieldParams forms recorded as not exposed yet; VectorNameConfig
Dense/Sparse tied to their constructors.
Tests: 91 total (+7 conversion, +2 integration: slice partitioning and
payload-exclude retrieval). Kotlin binding generation verified for the
new types.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(edge-ffi): resolve clippy warnings (inline format arg, large enum variant)
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge-ffi): address multi-agent review of the full-coverage expansion
Fixes findings from a multi-agent review (each independently re-verified),
targeting the new surface added in the full-engine-coverage expansion. All
verdicts CONFIRMED except query_groups (refuted → no change). 100 tests green.
- formula: cap Expression node count (MAX_FORMULA_NODES = 10_000), not just
depth. Expression is a host-held Arc; combinators eagerly deep-clone their
children's inner tree, so reusing one handle as several children
(sum([e, e]) repeated) grows node count as 2^depth while depth stays under
the 64 cap — an eager multi-GB clone that aborts the process (uncatchable
under panic=unwind). Track a saturating size in node() and decay(); the
depth guard stays (it protects the stack for narrow-but-deep chains).
- sparse: validate host sparse vectors at the boundary via
validate_sparse_vector_impl (indices.len == values.len, unique indices).
Without it a length mismatch reached an out-of-bounds panic at insert/scoring
(opaque caught-panic, not a typed error) and duplicate indices silently
double-counted. Route both sparse arms through a fallible helper; the
infallible From<SparseVector> is removed so no path can bypass validation.
- order_value: surface it on ScoredPoint/Record (new OrderValue { Int, Float },
matching the Python edge SDK). It was dropped behind a comment claiming "no
order-by-scored surface" — false: order-by query/scroll and OrderBy.start_from
are exposed, and ordered results carry a constant score and no next_offset, so
order_value is the only way to resume ordered pagination when payload is off.
- search_matrix: drop from v1. It is an O(n^2) analytics op the reference Python
edge SDK does not expose; its flat 1-Mi bounds are the wrong shape (one call
hangs/OOM-crashes the device on a normal shard). Removing pre-publish is free;
re-adding later with proper caps is non-breaking, the reverse is not.
- tests: node-count reject, sparse len/dup reject, order_value populated +
start_from resume, and snapshot negative tests (bad path / corrupt archive /
post-unload surface OperationError, not a panic, and the shard survives).
* fix(edge-ffi): reject oversized formula before the eager clone; close test gaps
Follow-up from a re-review of the fix commit.
- formula: the node-count check ran AFTER the eager `inner` deep-clone, because
the clone was an argument to `node()` (evaluated before the function body). A
host could build one accepted handle and reuse it as many children in one call
(sum(vec![e; N])), materializing N x e.size nodes before the size check could
reject it — the same uncatchable OOM abort the cap was meant to prevent, in two
calls. `node()` now takes the children handles plus a build closure and runs
both caps BEFORE invoking it, so a rejected tree never clones (decay() already
did this). All combinators route through it.
- tests: decay node-count reject + happy path (decay has its own guard, was
untested); wide-fanout reject (regression for the ordering fix); query-side
ScoringQuery::OrderBy -> ScoredPoint.order_value (only the scroll/Record side
was covered); real multi-page order-by continuation via StartFrom (was a
single-page degenerate case); sparse values-longer reject; float OrderValue;
with_payload=false omits payload. Suite: 107 green (4 + 39 + 64).
* feat(edge-ffi): keep search_matrix behind an off-by-default `matrix` feature
Reconsidered the outright removal: the FFI crate is a general UniFFI boundary,
not mobile-only, so non-mobile consumers (desktop/server/Rust) may want the
distance-matrix op. Instead of deleting it, gate the whole `ops/matrix.rs`
module behind a new off-by-default `matrix` Cargo feature.
- The mobile Swift/Kotlin bindings build without the feature, so search_matrix
stays off the mobile surface (verified: default `cargo test` excludes it and
its test; `--features matrix` includes both).
- The O(n^2) DoS is documented, not capped here: the op is opt-in and off the
constrained mobile surface, so bounding sample_size/limit_per_sample for a
given deployment is the enabling consumer's / SDK layer's responsibility. The
module doc spells this out; the per-field bounded_limit only stops a lone
u64::MAX value, not the quadratic compute.
- CI: add an `--all-features` test leg so the feature-gated surface (matrix +
the pre-existing staging passthrough) can't bit-rot — this also closes the
build-publish review note that `staging` had no CI coverage.
* docs+test(edge-ffi): tighten matrix doc wording; pin formula node-count boundary
Non-blocking polish from a final all-reviewer pass (6/6 APPROVE):
- matrix.rs / Cargo.toml docs: (1) the per-field bounded_limit caps at
MAX_RESULT_COUNT (1,048,576), not merely a lone u64::MAX — say so, since 1 Mi
is itself catastrophic for an O(n^2) op; (2) the DoS bound is owned by the
opting-in consumer, not an "SDK/wrapper layer" that need not exist for a raw
UniFFI consumer; (3) the mobile bindgen (a follow-up PR) must build with
default features to keep the op off the mobile surface — stated as intent, not
present-tense fact (no swift/android dirs exist yet).
- formula_node_count_exact_boundary: pins the exact MAX_FORMULA_NODES threshold
(10_000 accepted, 10_001 rejected) — the existing tests jumped to ~16k, so the
precise cap edge was unverified.
Suite: 108 default / 109 --all-features, all green.
* feat(edge-ffi): make the matrix feature on by default
Flip `matrix` to on-by-default (`default = ["matrix"]`). General (desktop /
server-side / Rust) UniFFI consumers now get `search_matrix` without opting in;
the mobile Swift/Kotlin bindgen builds with `--no-default-features` to drop the
O(n^2) analytics op from the mobile binding surface.
CI now covers all three shapes: default (matrix on), `--no-default-features`
(mobile surface, matrix excluded — guards the crate still compiles/passes
without it), and `--all-features` (matrix + staging). Verified: default 66 /
no-default 65 / all-features 66 integration tests, all green; no Cargo.lock
drift.
* fix(edge-ffi): rustfmt the sparse validator; correct stale matrix-feature docs
From a full all-reviewer pass of the review fixes:
- types.rs: rustfmt the `to_internal_sparse` `.map_err` chain. It failed
`cargo +nightly fmt --all -- --check` (the rust-lint.yml CI gate) — the
edge-test legs only run `cargo test`, so it slipped through locally.
- ops/mod.rs + integration.rs: fix two doc comments that still said the `matrix`
feature is 'off-by-default' after it was flipped on-by-default. Left uncorrected
they'd mislead the follow-up mobile-bindgen author into skipping
`--no-default-features` and shipping the O(n^2) op to phones.
Design decisions (recorded): matrix stays on-by-default; the O(n^2) DoS is
documented, not capped — a static cap can't know the target device (compute cost
is device-dependent; the caller owns it via async/timeout). No cap added.
Reviewers: 3 APPROVE, 3 CHANGES-REQUESTED, all CR items were these two doc/fmt
misses plus the recorded design calls. All 3 feature configs green (66/65/66).
* fix(edge-ffi): resolve clippy errors in FFI tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge): reject conflicting vector-name re-create instead of desyncing config
The segment-level `create_vector_name` is idempotent: re-creating an existing
name is a no-op that leaves storage untouched. `EdgeShard::update` treated any
`Ok` from that no-op as success and unconditionally re-applied the requested
params to the shard config, so a second `create_dense_vector("v", 8, Cosine)`
after `("v", 4, Dot)` left `config()` advertising 8/Cosine while storage kept
4/Dot — a shape the shard then rejects on upsert. The empty name "" (the
default vector) hit the same path against a single-vector shard's primary field.
Reject a conflicting re-create up front (before the WAL, so it can't brick
replay) with a clear error, accept an identical re-create as idempotent, and
only re-apply config when storage actually changed.
* fix(edge-ffi): harden FFI boundary from an adversarial per-search-type review
Boundary-validation and doc fixes surfaced by fuzzing each query type with
executed repros:
- Reject non-finite floats that JSON cannot represent but the raw-f64 FFI can:
order-by `StartFrom::Float` (a NaN panicked on a float-indexed field and
silently truncated an integer scan), and formula `decay` midpoint/scale +
`div` by_zero_default (a NaN evaded the engine's comparison-based range
checks → debug panic across the boundary / silent all-zero rescore).
- Drop the `key` parameter from `overwrite_payload`/`overwrite_payload_by_filter`:
the engine has no payload selector on the overwrite path (the server discards
it too), so a keyed overwrite silently replaced the whole payload.
- Reject a `FieldCondition` carrying more than one predicate: the engine has no
defined semantics for it (it evaluates one, index-dependent), so passing
several through diverged silently from a Qdrant server. Callers AND predicates
via separate `must` conditions.
- Doc fixes: FeedbackCoefficients b/c (exponent/multiplier, not weight/margin),
RecommendStrategy::BestScore default, div-by-zero behavior, retrieve
duplicate-ID collapsing, non-atomic partial-snapshot recovery.
Also fix the clippy `--all-targets -D warnings` lints in the test files that
reddened CI's `lint` job (uninlined_format_args, err_expect, disallowed
std::fs::write) and add regression tests for the validations above.
* fix(edge): compare only vector identity fields when detecting re-create conflicts
The conflicting-re-create guard added in the previous commit compared the full
`EdgeVectorParams`/`EdgeSparseVectorParams` via `PartialEq`, but a vector
declared in the initial `EdgeConfig` is stored with `on_disk: Some(false)` (from
`from_vector_data_config`) while a `CreateVectorName` op leaves it `None`. So an
identical re-create of a construction-defined vector — or of any vector after
`set_vector_hnsw_config` — was falsely rejected as a conflict, a regression on
the idempotent path.
Compare only the identity fields the op actually defines (dense: size, distance,
multivector_config, datatype; sparse: modifier, datatype); the storage/tuning
fields it cannot express are set from the config, the optimizer, or
`set_*_config` and must not trigger a false conflict. Adds a regression test
re-creating the config-defined "vec".
* style: apply cargo fmt
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(edge-ffi): expose payload_schema in info() and parameterized index creation
Close the last interface gap in the FFI surface: `info()` previously
elided the engine's `payload_schema`, and `create_field_index` only
accepted a bare type.
- New `payload_index` module mirroring the full `PayloadSchemaParams`
family (all 8 index types incl. tokenizer/stopwords/stemmer options)
with bidirectional conversions.
- `ShardInfo.payload_schema` reports each index's type, creation params,
and indexed point count.
- New `UpdateOperation::create_field_index_with_params` constructor;
coverage map updated accordingly.
- Boundary normalizations, following the quantization-config precedent:
deprecated `on_disk` folds into the reported `memory` placement, and
contradictory integer params (lookup+range both off) reject with
`InvalidArgument`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ivan Afanasiev <ivan.afanasiev@yahoo.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The publish/ directory is a separate Cargo workspace, so it does not
inherit the root workspace's [profile.dev] debug = "line-tables-only"
override. Example binaries were built with full debug info, each
statically linking qdrant-edge at ~700 MB per binary. Linking several
of them concurrently ran the runner out of disk, surfacing as
"mold: failed to write to an output file. Disk full?" + SIGBUS.
Mirror the debug info override in the publish workspace and free
~20 GB of unused preinstalled software on Linux runners as headroom.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Building the full qdrant-edge wheel matrix (15 build jobs + 8 test jobs
across Linux/macOS/Windows, glibc/musl, CPython/PyPy) on every push to
dev is too expensive. Dev pushes now build only the most popular flavor:
x86_64 manylinux_2_17 CPython. Thanks to abi3-py310 that single wheel
covers every CPython >= 3.10, and it is tested on the latest stable
Python (3.14) on ubuntu-latest.
The full platform matrix still runs on manual workflow_dispatch
releases, and their test matrix is extended from {3.10, pypy3.11} to
all stable CPythons (3.10-3.14) plus PyPy on all four OS runners.
Since macOS/Windows builds are skipped on push, edge-py-test and
merge-artifacts switch to !cancelled()-based conditions so they are not
skip-propagated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The recover-remote-concurrent step can hang indefinitely with S3 storage;
cap CI steps at 15 minutes and curl requests at 120 seconds so stuck runs
do not consume the full 6-hour job timeout.
Co-authored-by: Cursor <cursoragent@cursor.com>
The previous fix only updated the Linux block because its old_string
matched a longer comment that the macOS/Windows blocks never had, so
those two jobs kept the original PEP 440 `.dev<N>` syntax without the
patch bump. Result: latest CI on `dev` (run 28509950366) still failed
with `unexpected character '.' after patch version number` on both
macOS and Windows, showing `version = "0.7.2.dev55"`.
Bring both blocks in line with Linux: SemVer `-dev<N>` suffix and the
patch-component bump. Same awk logic on all three, so maturin sees a
Cargo-valid version on every runner.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cargo rejects PEP 440 `.dev<N>` syntax because it isn't valid SemVer, so
build the SemVer form `-dev<N>` instead. maturin translates the pre-release
identifier back to PEP 440 `.dev<N>` when constructing the wheel filename
and metadata, so the published version is unchanged from the intent.
Fixes the "unexpected character '.' after patch version number" error
seen on the first push of this branch.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: publish qdrant-edge-py pre-releases on merge to dev
Trigger the edge-py release workflow on push to `dev` and rewrite the
Cargo.toml package version to `X.Y.Z.dev<run_number>` before building.
PEP 440 treats the `.devN` suffix as a pre-release, so `pip install
qdrant-edge-py` continues to install the latest stable version while
users who want to try dev builds can pin explicitly:
pip install qdrant-edge-py==0.7.2.dev137
pip install --pre qdrant-edge-py
Manual `workflow_dispatch` runs are unchanged and still use the exact
Cargo.toml version.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: publish dev wheels to GCS instead of PyPI
On push to `dev`, upload the built wheels to gs://qdrant-debug/edge-py-dev/
(same bucket as the debug binaries, different sub-folder) and regenerate a
pip-compatible HTML index of every wheel currently in the bucket.
Users install a specific dev version with:
pip install --find-links \
https://storage.googleapis.com/qdrant-debug/edge-py-dev/qdrant-edge-py.html \
qdrant-edge-py==0.7.2.dev137
The PyPI publish job is now gated back to `inputs.publish`, so manual
workflow_dispatch runs still target PyPI for stable releases.
Reuses the existing GCS_ACCESS_KEY_ID / GCS_SECRET_ACCESS_KEY secrets and
the AWS CLI + S3-compatible XML API pattern established by debug-tools.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: keep only latest dev version in GCS bucket
After uploading the freshly built wheels, delete every wheel in the
bucket that doesn't belong to the current version. Order matters:
upload happens first so there's never a window with zero wheels
available in the index.
Current version is extracted from the wheel filename (PEP 427
guarantees version is the second `-`-separated field, and PEP 440
versions can't contain `-`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: auto-bump patch component for dev wheel versions
Extend the version-patch step to bump the patch component of the SemVer
version in `Cargo.toml` before appending `.dev<run_number>`. For example
`0.7.2` in Cargo.toml becomes `0.7.3.dev137` in the built wheel.
This keeps every dev build strictly above the latest published stable in
PEP 440 ordering, so consumers can install "the current dev" without
knowing the run number:
pip install --find-links <index-url> qdrant-edge-py --pre
Without the bump, `0.7.2.dev137` would sort below stable `0.7.2` and
`--pre` would silently prefer the stable release.
Manual `workflow_dispatch` runs are unchanged and still publish the
exact Cargo.toml version to PyPI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After `--package qdrant`, cargo scopes `--bin` to the qdrant crate only.
Pass `--package edge-shard-query --bin edge-shard-query` so the edge tool
is actually produced for collection/upload.
Co-authored-by: Cursor <cursoragent@cursor.com>
Without an explicit qdrant package, cargo applies --features service_debug
to every -p target including edge-shard-query, which does not define it.
Co-authored-by: Cursor <cursoragent@cursor.com>
edge-shard-query does not define service_debug; use qdrant/service_debug
when building qdrant bins and edge tools in one cargo invocation.
Co-authored-by: Cursor <cursoragent@cursor.com>
Use the perf profile instead of release, merge qdrant and edge tool builds
into one cargo invocation, and add sccache so shared deps compile once and
rustc outputs are cached across runs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Turn the single-purpose `edge-s3-scroll` tool into a sub-command-based
`edge-shard-query` that can run different read requests against a
ReadOnlyEdgeShard opened over object storage.
- Add `scroll` and `search` sub-commands; shared connection/cache args
live at the top level.
- Accept arbitrary payload filters as JSON via `--filter` (curl
`--data` style: literal JSON, `@file`, or `@-` for stdin), parsed
straight into the filter DSL. Keep `--filter-key`/`--filter-value`
as a shortcut.
- `search` accepts a query vector (JSON array, comma list, or
`@file`/`@-`), named vector, offset, score threshold, and HNSW
params.
- Rename the package/binary/dir (`s3_scroll` -> `shard_query`) since
the tool is no longer scroll-only nor S3-only.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the edge-s3-scroll tool (lib/edge/tools/s3_scroll) to the debug
tools build/upload. It's a standalone workspace crate without the
qdrant `service_debug` feature, so it's built in a separate `cargo build
-p edge-s3-scroll` invocation and then collected/uploaded alongside the
rest.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: add nightly model_testing workflow
Run the model_testing binary nightly on dev with two sequential passes
(async-scorer and without io_uring), reusing a single built binary.
Note: a temporary pull_request trigger is included to test-drive the
workflow on the PR; it is to be removed before merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: fix checkout ref for pull_request event
github.ref_name resolves to <pr>/merge on pull_request, which checkout
treats as a branch and fails to fetch. Use the default ref outside the
nightly schedule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: drop mold/clang linker from model_testing workflow
free-disk-space (large-packages) removes the llvm/clang packages, so the
clang linker used for mold no longer exists on the runner. Build time is
dominated by the long model-testing runs, so the default linker is fine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: random seed, constants, both-run, backtrace, dispatch inputs, failure issue
- Random seed per run (logged for repro), shared by both passes
- OP_NUM/SHARD_COUNT/RESTART_PROBABILITY/ID_POOL as env constants
- no-io_uring run uses if: !cancelled() so both passes always run
- RUST_BACKTRACE=1 for debuggable panics
- workflow_dispatch inputs to override op_num/seed
- rust-cache saves on the nightly (schedule) run, not the never-matching dev ref
- open/update a GitHub issue on scheduled failures
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run model_testing with 2 shards by default
Exercises the cross-shard reload/WAL-replay path (the 2-shard-only revert
class) instead of single-shard only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: bump model_testing OP_NUM to 200000
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: remove temporary pull_request trigger from model_testing workflow
Test-drive validated on the PR; the nightly now runs only on schedule and
workflow_dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build the debug tools for x86_64-unknown-linux-musl so the resulting
binaries are statically linked and run on any Linux host regardless of
distro / libc. Mirrors the musl setup used in release-artifacts.yml
(apt deps + taiki-e/setup-cross-toolchain-action) and drops the
gnu-specific mold linker hack, which the cross toolchain replaces.
Cache is keyed per target so the musl artifacts persist across runs
without colliding with gnu caches.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Recent AWS CLI versions add default CRC32 integrity checksums on
uploads, which GCS's S3-compatible UploadPart rejects with
SignatureDoesNotMatch. This only affected the larger debug binaries that
go through multipart upload. Opt out via AWS_REQUEST_CHECKSUM_CALCULATION
/ AWS_RESPONSE_CHECKSUM_VALIDATION=when_required.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-add the push-to-dev trigger so debug tools are rebuilt automatically
on every merge to dev, while keeping manual dispatch with a branch
input. The branch used for checkout and the object prefix falls back to
the pushed branch when no input is given.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: build debug tools and publish to GHCR via ORAS
Adds a `Build debug tools` workflow that compiles the `service_debug`
helper binaries (wal_inspector, wal_pop, segment_inspector,
schema_generator, model_testing) and pushes them to GHCR as an OCI
artifact using ORAS.
These are debug-only tools built between releases and are intentionally
kept out of the release artifacts. Publishing them as an OCI artifact
gives a versioned, browsable, registry-backed location alongside our
Docker images instead of burying them in per-run CI artifacts.
Pull with:
oras pull ghcr.io/qdrant/qdrant/debug-tools:dev
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: upload debug tools to S3 instead of GHCR
Switch the debug-tools workflow from publishing an OCI artifact via ORAS
to uploading the `service_debug` binaries to S3 with public-read, giving
plain HTTPS download links and no client tooling beyond curl.
Binaries are uploaded under a moving `<branch>/` prefix (latest) and an
immutable `<branch>-<sha>/` prefix (history); direct links are written
to the job summary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: target GCS (S3-compatible) instead of AWS S3
Point the debug-tools upload at GCS via its S3-compatible XML API using
AWS-style HMAC interoperability keys. Drops per-object ACLs (GCS buckets
use uniform access; public read is granted via IAM) and uses GCS
path-style public URLs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: set debug-tools bucket to qdrant-debug
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run debug-tools manually with a branch input
Drop the push trigger so the workflow only runs on explicit dispatch
from the UI, and add a `branch` input selecting which branch to build.
The chosen branch (and its actual HEAD commit) form the object prefix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: fix rust-cache running before checkout in shard-snapshot job
The `test-shard-snapshot-api-s3-minio` job ran `Swatinem/rust-cache`
before `actions/checkout`, so there was no `Cargo.lock`/source present
when the cache action ran. As a result the action no-opped and the job
never actually restored or saved a Rust build cache. Move the cache
step to run after checkout.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: remove dead rust-cache steps from docker-only jobs
These jobs only build Docker images via `docker buildx`/docker build
and never invoke cargo directly, so the `Swatinem/rust-cache` step
never populates or restores anything useful:
- integration-tests.yml: `test-consistency` (docker buildx + shell
consistency checks only)
- docker-image.yml: `build` and `build-gpu` (docker buildx only)
Drop the unused cache steps.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: drop rust-cache from coverage and release workflows
These workflows run on rare triggers (coverage: nightly schedule;
release-artifacts: on release publish), so by the time they run again
their cache is almost always already evicted from the 10 GB Actions
cache budget. They mostly just consume cache space that would otherwise
be useful to the frequently-run PR builds, so remove the rust-cache
steps entirely:
- coverage.yml: unit-coverage and integration-coverage
- release-artifacts.yml: build-windows-binaries
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: drop explicit clippy cache key in rust-lint
The lint job has no compatible cache peer to share with: clippy
produces clippy-driver metadata that cargo fingerprints separately
from normal build/test artifacts, and its `--all-features` invocation
pulls in the GPU dependency crates, so it matches neither the
rust-tests nor the gpu group. With nothing to share, the explicit
`key: clippy` is equivalent to the default per-job key, so drop it for
consistency with the rest of the workflows.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: dedicated rust-cache keys with single writer per group
Give related jobs explicit cache keys and let only the broadest build
in each group save the cache (`save-if: "false"` on the others), so a
narrower build can't overwrite it with fewer compiled dependencies
first.
- `rust-tests` (rust.yml): explicit `shared-key: rust-tests`. It builds
`--workspace --tests`, which pulls in dev-dependency features that the
integration-tests builds (plain `cargo build`) do not, so it is kept
as its own dependency cache rather than merged with integration-tests.
- `integration-tests`: writer = integration-tests; readers =
integration-tests-consensus, test-shard-snapshot-api-s3-minio.
- `edge`: writer = edge-test (clippy + examples build); reader =
edge-rust-check. Both build the amalgamated qdrant-edge `examples`
package.
rust-gpu keeps its own default cache (the `gpu` feature adds the
ash/gpu-allocator/shaderc dependency crates).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Reuse `integration-tests` build cache for `io-bridge-object-store-tests` workflow
* Cleanup Docker cache in GHA workflows
* ci: only save rust-cache on the integration branch
Under the 10 GB Actions cache budget there's no practical benefit to
saving feature-branch build caches: they'd be evicted before being
reused. Scope every rust-cache *writer* to save only on `dev` (the
branch PRs target and restore from) via `save-if`:
- rust.yml, rust-lint.yml, edge-test.yml, integration-tests.yml
Reader jobs already have `save-if: "false"` and are unchanged.
Also drop rust-cache from rust-gpu.yml entirely: it only runs on
`master` pushes (releases, a few times a month at most), so any cache
would always be evicted between runs.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Potential fix for code scanning alert no. 7: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for code scanning alert no. 9: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for code scanning alert no. 10: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for code scanning alert no. 19: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for code scanning alert no. 20: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Set permissions in GitHub workflow jobs
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* feat: add io_bridge
* fix: cached dispatcher
* feat: add open_with_handle
* fix: naming
* fix: wording
* fix: rebase io_bridge onto split BorrowedReadPipeline/OwnedReadPipeline
* fix: linter
* feat: add S3 backend
* chore: remove io_bridge
* fix: linter
* fix: linter
* fix: read handle
* chore: simplify S3Source
* fix: s3 test
* feat: support multi runtime
* fix: clippy errors
* fix: review comments
* feat: add io design
* feat: add S3 backend
* chore: fix docs
* fix: dev changes
* chore: add some docs
* chore: remove explicit type
* feat: add new methods
* [WIP] review refactor
* fmt
* fix: bytes alignment
* fix: linter
* feat: remove Bytes
* fix: ci/cd
* fix: tests
* fix: tests
* refactor: simplify io_bridge pipeline to Handle-based dispatch
Replace the BridgeRuntime worker thread + request channel + boxed
BridgeRequest with direct tokio Handle usage:
- BridgeRuntime is now just an Arc<Runtime>; schedule() spawns the read
future via Handle::spawn instead of routing it through a dispatcher
thread. Removes BridgeRequest and the now-unreachable S3RuntimeShutDown
error variant.
- Guard against a panicking read task hanging wait() forever: the spawned
task catches unwinds and converts them into a TaskPanicked error reply,
so every scheduled slot is always answered.
- Encapsulate slot bookkeeping in PendingSlots, exposing only the needed
operations instead of a public map + counter.
- Split the grown pipeline.rs into a pipeline/ module (slots / inner /
borrowed / owned), de-duplicating the shared read-future construction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: move pipeline buffer ownership into the read future
Instead of the pipeline owning the destination Vec<T> in a slot map and
the future writing through a SendBytePtr raw pointer, let the future
allocate the buffer itself and return it through BridgeResponse. The
buffer crosses the worker-thread boundary as a normal move via the reply
channel, wrapped in a SendableVec<T> newtype that asserts Send for
T: bytemuck::Pod only.
This removes the entire unsafe SendBytePtr apparatus from the pipeline:
no raw pointer, no unsafe fn, no per-call-site unsafe blocks, no
heap-stability invariants. The only remaining unsafe in the crate is one
bounded `unsafe impl<T: Pod> Send for SendableVec<T>` with a trivially
true invariant (Pod types are plain bytes).
PendingSlots collapses back to PendingSlots<U>: slots no longer carry
buffers. AlignedBufWriter::from_raw_bytes (used only by the SendBytePtr
path) and its test are removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: drop SendableVec wrapper now that Item: Send
UniversalRead's element type is now bound to `Item` (`Pod + Send`),
so the io_bridge pipeline no longer needs a hand-rolled `Send` wrapper
around `Vec<T>` to ship buffers through the reply channel. Replace
`SendableVec<T>` with `Vec<T>` end-to-end and tighten the local impls
to `T: Item`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: implement UniversalRead::reopen for BlobFile
BlobFile has no cached file metadata or mapping — `len()` queries the
object store fresh on each call — so reopen is a no-op, matching the
io_uring impl.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add new fs impl for Blob
* fix: is_in_ram_or_mmap for S3
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: pin just version to 1.50.0 in edge workflows
Without an explicit just-version, the setup-just action resolves "latest"
on every run, making CI implicitly depend on whatever was released most
recently. Pin to 1.50.0 so just upgrades are an intentional change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: centralize just setup in a composite action
Add .github/actions/setup-just wrapping extractions/setup-just@v4.0.0 with
just-version pinned to 1.50.0, so the version is declared once instead of
duplicated across every edge workflow.
Also replaces the xzfc/setup-just fork in edge-rust-release: the reason
for the fork (missing just-version pinning support in extractions/
setup-just) was resolved in v4.0.0, so the workaround is no longer needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: centralize protoc setup in a composite action
Add .github/actions/setup-protoc that pins both the arduino/setup-protoc
SHA and the protoc version (34.1) in one place, so future bumps no longer
require touching every workflow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: replace arduino/setup-protoc with direct release-zip download
The arduino action queries the GitHub API on every invocation even when
the version is fully pinned, which is unnecessary work. Inline a small
shell step that downloads the protoc release zip directly using
RUNNER_OS/RUNNER_ARCH for asset selection. Supports Linux, macOS, and
Windows (x64 + arm64 where protobuf publishes assets).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: parallelize consensus tests with pytest-xdist
Enable pytest-xdist for consensus_tests to run tests across multiple
workers in parallel, significantly reducing CI wall time (~20min → ~5-7min).
Changes:
- Add `-n auto --dist=loadfile` to the consensus test pytest invocation
- Remove hardcoded port_seed from tests that don't need fixed ports for
restart/rejoin (test_order_by, test_consensus_compaction,
test_named_vector_crud, test_listener_node)
- Give test_cluster_rejoin its own PORT_SEED=15000 to avoid port
conflicts with auth tests (PORT_SEED=10000)
- Derive restart ports from killed PeerProcess objects instead of
hardcoded arithmetic where possible
- Add xdist_group("auth") marker to auth test files to ensure they
run on the same worker (they share PORT_SEED=10000)
Made-with: Cursor
* fix: remove remaining hardcoded port_seed=20000 causing parallel test conflicts
8 test files were using port_seed=20000 as a positional argument to
start_cluster(), which was missed in the initial change. When running
in parallel with pytest-xdist, multiple workers would try to bind to
the same port range (20000-20x02), causing port conflicts and cascading
test failures.
Also remove port_seed=23000 from test_snapshot_recovery_kill.py since
it doesn't need fixed ports for restart.
Made-with: Cursor
* fix: use saved port for restart in test_two_follower_nodes_down
The test was restarting killed peers on hardcoded ports (20200/20100)
that previously matched port_seed=20000. After switching to random
ports, the restart ports no longer match the original peer ports,
causing raft state URI mismatches and peer startup failures.
Save the p2p_port from the killed PeerProcess and reuse it for restart.
Made-with: Cursor
* Reuse p2p ports when restarting killed peers in consensus tests
When a peer is killed and restarted with random ports, it gets a new
consensus URI. The cluster needs a Raft operation to update this URI,
which under CPU contention from parallel test workers can exceed the
30-second timeout. Fix by capturing each peer's p2p_port before killing
and reusing it on restart, so the URI stays the same and no consensus
update is needed.
Made-with: Cursor
* A few improvements for parallel runs (#8731)
* fix: make auth tests' PORT_SEED per-worker to avoid port collisions
* ci: improve failure visibility for parallel consensus tests
* Three small changes to make hangs, interleaved output, and coverage runs behave predictably under pytest-xdist
* fix: two test bugs surfaced by parallel runs and revert drop PR_SET_PDEATHSIG helper
* fix: wait for count convergence in test_triple_replication
* fix: clean leaked peer processes at test start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
