* feat: add io_bridge
* fix: cached dispatcher
* feat: add open_with_handle
* fix: naming
* fix: wording
* fix: rebase io_bridge onto split BorrowedReadPipeline/OwnedReadPipeline
* fix: linter
* feat: add S3 backend
* chore: remove io_bridge
* fix: linter
* fix: linter
* fix: read handle
* chore: simplify S3Source
* fix: s3 test
* feat: support multi runtime
* fix: clippy errors
* fix: review comments
* feat: add io design
* feat: add S3 backend
* chore: fix docs
* fix: dev changes
* chore: add some docs
* chore: remove explicit type
* feat: add new methods
* [WIP] review refactor
* fmt
* fix: bytes alignment
* fix: linter
* feat: remove Bytes
* fix: ci/cd
* fix: tests
* fix: tests
* refactor: simplify io_bridge pipeline to Handle-based dispatch
Replace the BridgeRuntime worker thread + request channel + boxed
BridgeRequest with direct tokio Handle usage:
- BridgeRuntime is now just an Arc<Runtime>; schedule() spawns the read
future via Handle::spawn instead of routing it through a dispatcher
thread. Removes BridgeRequest and the now-unreachable S3RuntimeShutDown
error variant.
- Guard against a panicking read task hanging wait() forever: the spawned
task catches unwinds and converts them into a TaskPanicked error reply,
so every scheduled slot is always answered.
- Encapsulate slot bookkeeping in PendingSlots, exposing only the needed
operations instead of a public map + counter.
- Split the grown pipeline.rs into a pipeline/ module (slots / inner /
borrowed / owned), de-duplicating the shared read-future construction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: move pipeline buffer ownership into the read future
Instead of the pipeline owning the destination Vec<T> in a slot map and
the future writing through a SendBytePtr raw pointer, let the future
allocate the buffer itself and return it through BridgeResponse. The
buffer crosses the worker-thread boundary as a normal move via the reply
channel, wrapped in a SendableVec<T> newtype that asserts Send for
T: bytemuck::Pod only.
This removes the entire unsafe SendBytePtr apparatus from the pipeline:
no raw pointer, no unsafe fn, no per-call-site unsafe blocks, no
heap-stability invariants. The only remaining unsafe in the crate is one
bounded `unsafe impl<T: Pod> Send for SendableVec<T>` with a trivially
true invariant (Pod types are plain bytes).
PendingSlots collapses back to PendingSlots<U>: slots no longer carry
buffers. AlignedBufWriter::from_raw_bytes (used only by the SendBytePtr
path) and its test are removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: drop SendableVec wrapper now that Item: Send
UniversalRead's element type is now bound to `Item` (`Pod + Send`),
so the io_bridge pipeline no longer needs a hand-rolled `Send` wrapper
around `Vec<T>` to ship buffers through the reply channel. Replace
`SendableVec<T>` with `Vec<T>` end-to-end and tighten the local impls
to `T: Item`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: implement UniversalRead::reopen for BlobFile
BlobFile has no cached file metadata or mapping — `len()` queries the
object store fresh on each call — so reopen is a no-op, matching the
io_uring impl.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add new fs impl for Blob
* fix: is_in_ram_or_mmap for S3
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: pin just version to 1.50.0 in edge workflows
Without an explicit just-version, the setup-just action resolves "latest"
on every run, making CI implicitly depend on whatever was released most
recently. Pin to 1.50.0 so just upgrades are an intentional change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: centralize just setup in a composite action
Add .github/actions/setup-just wrapping extractions/setup-just@v4.0.0 with
just-version pinned to 1.50.0, so the version is declared once instead of
duplicated across every edge workflow.
Also replaces the xzfc/setup-just fork in edge-rust-release: the reason
for the fork (missing just-version pinning support in extractions/
setup-just) was resolved in v4.0.0, so the workaround is no longer needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: centralize protoc setup in a composite action
Add .github/actions/setup-protoc that pins both the arduino/setup-protoc
SHA and the protoc version (34.1) in one place, so future bumps no longer
require touching every workflow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: replace arduino/setup-protoc with direct release-zip download
The arduino action queries the GitHub API on every invocation even when
the version is fully pinned, which is unnecessary work. Inline a small
shell step that downloads the protoc release zip directly using
RUNNER_OS/RUNNER_ARCH for asset selection. Supports Linux, macOS, and
Windows (x64 + arm64 where protobuf publishes assets).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: parallelize consensus tests with pytest-xdist
Enable pytest-xdist for consensus_tests to run tests across multiple
workers in parallel, significantly reducing CI wall time (~20min → ~5-7min).
Changes:
- Add `-n auto --dist=loadfile` to the consensus test pytest invocation
- Remove hardcoded port_seed from tests that don't need fixed ports for
restart/rejoin (test_order_by, test_consensus_compaction,
test_named_vector_crud, test_listener_node)
- Give test_cluster_rejoin its own PORT_SEED=15000 to avoid port
conflicts with auth tests (PORT_SEED=10000)
- Derive restart ports from killed PeerProcess objects instead of
hardcoded arithmetic where possible
- Add xdist_group("auth") marker to auth test files to ensure they
run on the same worker (they share PORT_SEED=10000)
Made-with: Cursor
* fix: remove remaining hardcoded port_seed=20000 causing parallel test conflicts
8 test files were using port_seed=20000 as a positional argument to
start_cluster(), which was missed in the initial change. When running
in parallel with pytest-xdist, multiple workers would try to bind to
the same port range (20000-20x02), causing port conflicts and cascading
test failures.
Also remove port_seed=23000 from test_snapshot_recovery_kill.py since
it doesn't need fixed ports for restart.
Made-with: Cursor
* fix: use saved port for restart in test_two_follower_nodes_down
The test was restarting killed peers on hardcoded ports (20200/20100)
that previously matched port_seed=20000. After switching to random
ports, the restart ports no longer match the original peer ports,
causing raft state URI mismatches and peer startup failures.
Save the p2p_port from the killed PeerProcess and reuse it for restart.
Made-with: Cursor
* Reuse p2p ports when restarting killed peers in consensus tests
When a peer is killed and restarted with random ports, it gets a new
consensus URI. The cluster needs a Raft operation to update this URI,
which under CPU contention from parallel test workers can exceed the
30-second timeout. Fix by capturing each peer's p2p_port before killing
and reusing it on restart, so the URI stays the same and no consensus
update is needed.
Made-with: Cursor
* A few improvements for parallel runs (#8731)
* fix: make auth tests' PORT_SEED per-worker to avoid port collisions
* ci: improve failure visibility for parallel consensus tests
* Three small changes to make hangs, interleaved output, and coverage runs behave predictably under pytest-xdist
* fix: two test bugs surfaced by parallel runs and revert drop PR_SET_PDEATHSIG helper
* fix: wait for count convergence in test_triple_replication
* fix: clean leaked peer processes at test start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* lib/edge/publish/cargo: improve target directory discovery
Don't hardcode workspace root, ask cargo instead.
* Fix warning in qdrant-edge amalgamation
`cargo check` in `lib/edge/publish` produces the following warning:
warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending`
* lib/edge/publish/examples: fix compiler warnings
* refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs
To match python examples that have the same constant in `common.py`.
* Use `lib/edge/data` dir for both Python and Rust examples
Before this commit, `lib/edge/publish/data` and `lib/edge/python/data`
were used. I'd like Python and Rust to use the same prepared resources
in the future.
* ❗Add lib/edge/Justfile with recipes to build/run examples
I constantly keep forgetting how to compile and run edge stuff.
Also, this would be used in CI in subsequent commits.
* ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes
Also now we can rename `creates-io-readme.md` to `README.md` like
reasonable people.
* ❗doc: Add qdrant-edge-py README.md
Since the previous commit removed qdrant-edge-py README.md, we need to
add something in its place instead.
The new README is adapted from Rust edge README.
And the previous README wasn't good anyway, see [1], it mentions maturin
which is irrelevant for users.
[1]: https://pypi.org/project/qdrant-edge-py/0.5.0/
* GHA: Add "Setup Qdrant" composite action
We need running Qdrant instance to test edge examples on CI. We can't
use docker for this because we want to run tests on Windows and macOS.
OTOH, this action downloads binaries for all platforms provided in
https://github.com/qdrant/qdrant/releases/tag/v1.17.0
* ❗GHA: Use single worflow for python and rust edge examples
Merge workflows to reuse build cache. Also, modernize it by using `uv`,
and recently introduced `setup-qdrant` and Justfile. Also, make it run
on three platforms (only on merges).
* ❗GHA: edge-{py,rust}-release: also run examples before publishing
Use similar approach as in the previous commit. But note that this time
we build/test also on ARM Linux and Windows.
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* amalgamate.py: specify encoding="utf-8"
For legacy platforms that still default to cp1252.
* amalgamate.py: use `shutil.which`
In case if `ast-grep` is a cmd wrapper.
* amalgamate.py: write rules to temporary file
Windows runner don't like inline-rules.
* amalgamate.py: Add decription to package
Required for publishing to crates.io.
* amalgamate.py: package build scripts for `common` and `segment`
Also, add more comments.
* amalgamate.py: keep generated module order deterministic
Not an issue, but nice to have. Suggested by CodeRabbit.
* edge-rust-release.yml: also check on arm machines
* edge-rust-release.yml: Use cargo build instead of cargo check
To catch linker errors.
* edge-rust-release.yml: disable fail-fast (for easier debugging)
* edge-rust-release.yml: add --no-verify to cargo publish
The previous job `edge-rust-check` already builds it, no need to build
it again in the `publish` job.
* Fixups of amalgamator
Fix issues that break `qdrant-edge` build process:
- `use … as segment;` - this causes `ast-grep` rules to replace wrong
paths. So, rename to avoid collisions.
- `#[macro_use]` and `extern crate` required be in the top-level
`lib.rs`.
- `format!("…", crate::something::…)` - `ast-grep` can't fix paths
inside macros. Fixed by moving `crate::something::…` out of the macro.
* Add lib/edge/publish workspace and amalgamation script
* Move `lib/edge/examples` into `lib/edge/publish/` workspace
And fix them to use the generated `qdrant-edge` crate.
* Add github workflow
* Cleanup `qdrant-edge` public API
Removes empty modules. Checked by `cargo doc`.
* Move pyproject.toml to root
* Migrate pyproject.toml from Poetry to uv
* Update GH workflows
* Update test script, doc and nix to use uv
* Use latest uv
* Fix uv.lock
* Cleanup shell.nix
* Cleanup
- Explicit `uv sync` is not required, `uv run` will install deps
automatically.
- We don't provide a python package, so the `[build-system]` section
is not needed.
* Fix UV_VERSION inconsistency
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
