Commit Graph
5019 Commits
Author SHA1 Message Date
Tim Visée f58f779b91 Add a user agent to HTTP clients (#7623)
* Add user agent to add HTTP clients

* Move user agent constant to defaults.rs
2025-12-03 10:18:53 +01:00
Roman Titov 1de85b956e Qdrant Edge Python bindings improvements (#7561)
* Use anonymous lifetime in `FromPyObject` implementations

* Use `PyResult` in `IntoPyObject` implementations

* Cleanup imports and derives

* Cleanup `filter` conversions

* Add `PointVectors` getters

* Move `config` module into sub-directory

* Split `config` into sub-modules

* Simplify enum bindings

* Add zero-cost conversions for `PyVectorDataConfig` and `PySparseVectorDataConfig`

* Add getters to config structures

* fixup! Add getters to config structures

More zero-cost conversions for `PyVector*DataConfig`

* Implement `PyHnswIndexConfig`

* Implement `PyQuantizationConfig`

* fixup! Simplify enum bindings

* fixup! Implement `PyHnswIndexConfig`

* fixup! Implement `PyHnswIndexConfig`

* fixup! Implement `PyHnswIndexConfig`

* Implement `PySparseVectorDataConfig`

* fixup! Implement `PySparseVectorDataConfig`

* fixup! Implement `PySparseVectorDataConfig`
2025-12-03 10:18:53 +01:00
Tim Visée d25d877b3b Change minimum version for batch WAL transfer into constant (#7619) 2025-12-03 10:18:52 +01:00
tellet-q 362db71f2d Build dev with new feature flag (#7612)
* Build dev images with new feature flag

* Disable feature for master and tags
2025-12-03 10:18:52 +01:00
Arnaud Gourlay f638d9f24e Document storage compatibility test for v1.15.0 (#7601)
* Enable storage compatibility test for v1.15.0

* proper explanation
2025-12-03 10:18:52 +01:00
tellet-q 808aadae92 Increase timeout for e2e tests (#7611) 2025-12-03 10:18:52 +01:00
Tim Visée b0202cd4b3 Disable RocksDB features in local development builds (#7552)
* Disable rocksdb compile time feature by default

* Also disable RocksDB feature in segment crate

* Enable RocksDB feature in all CI builds

* Remove extra job for testing non-RocksDB build, it's the default now

* Keep RocksDB structures in generated OpenAPI schema

* Fix obsolete --workspace flag breaking builds with explicit features

* Also build including RocksDB in e2e tests on CI
2025-12-03 10:17:54 +01:00
Tim Visée 25055e6297 Bump rocksdb dependency to 0.24.0 (#7605) 2025-12-03 10:17:41 +01:00
Tim Visée e293ab1f08 Fix cosign error on release image (#7606) v1.16.1 2025-11-25 16:58:56 +01:00
Tim Visée 4a66b7cf2b Bump version to 1.16.1 (#7603)
* Bump version to 1.16.1

* Update missed cherry picks
2025-11-25 14:59:57 +01:00
Daniel Borosandgenerall 8b7c9f051f fix/inference timeout error (#7598)
* fix: inference timeout error

* chore: cleanup timeout from inference config

* cleanup timeout from InferenceConfig

* chore: revert config.toml changes

* inference_token -> infrence_params for timeout propagation

* always use at least some timeout

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-11-25 14:25:39 +01:00
Arnaud Gourlayandtimvisee 70b8623bb1 Add timeout to cancel metrics and telemetry calls (#7579)
* Add timeout to cancel metrics and telemetry calls

* relax constraints

* Configure timeout default in OpenAPI schema

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-25 12:56:45 +01:00
dependabot[bot] 9a21aefe89 build(deps): bump integer-encoding from 4.0.2 to 4.1.0 (#7595)
Bumps [integer-encoding](https://github.com/dermesser/integer-encoding-rs) from 4.0.2 to 4.1.0.
- [Commits](https://github.com/dermesser/integer-encoding-rs/commits/v4.1.0)

---
updated-dependencies:
- dependency-name: integer-encoding
  dependency-version: 4.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 11:16:11 +01:00
dependabot[bot] a8833762a7 build(deps): bump quote from 1.0.41 to 1.0.42 (#7591)
Bumps [quote](https://github.com/dtolnay/quote) from 1.0.41 to 1.0.42.
- [Release notes](https://github.com/dtolnay/quote/releases)
- [Commits](https://github.com/dtolnay/quote/compare/1.0.41...1.0.42)

---
updated-dependencies:
- dependency-name: quote
  dependency-version: 1.0.42
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 11:16:11 +01:00
dependabot[bot] 990828d044 build(deps): bump actions/checkout from 5 to 6 (#7589)
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 11:16:11 +01:00
dependabot[bot] a62baf792f build(deps): bump clap from 4.5.49 to 4.5.53 (#7590)
Bumps [clap](https://github.com/clap-rs/clap) from 4.5.49 to 4.5.53.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.49...clap_complete-v4.5.53)

---
updated-dependencies:
- dependency-name: clap
  dependency-version: 4.5.53
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 11:16:10 +01:00
dependabot[bot] 963ac92937 build(deps): bump cc from 1.2.41 to 1.2.47 (#7593)
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.2.41 to 1.2.47.
- [Release notes](https://github.com/rust-lang/cc-rs/releases)
- [Changelog](https://github.com/rust-lang/cc-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/cc-rs/compare/cc-v1.2.41...cc-v1.2.47)

---
updated-dependencies:
- dependency-name: cc
  dependency-version: 1.2.47
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 11:16:10 +01:00
dependabot[bot] 4be5aedf9d build(deps): bump csv from 1.3.1 to 1.4.0 (#7594)
Bumps [csv](https://github.com/BurntSushi/rust-csv) from 1.3.1 to 1.4.0.
- [Commits](https://github.com/BurntSushi/rust-csv/compare/1.3.1...1.4.0)

---
updated-dependencies:
- dependency-name: csv
  dependency-version: 1.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 11:16:10 +01:00
dependabot[bot] 6ab32f9c46 build(deps): bump indicatif from 0.18.0 to 0.18.3 (#7597) 2025-11-25 11:16:10 +01:00
Andrey Vasnetsov e85329e94a remove unused function (#7586) 2025-11-25 11:16:01 +01:00
Tim Visée 745f0d6d15 Bump wal to fix flush problems (#7587) 2025-11-25 11:16:01 +01:00
Tim Visée a4dd7b3fce Fix WAL handling on consensus snapshot (#7577)
* After clearing WAL, flush segment

* Add debug log when WAL is cleared

* Clear WAL on consensus snapshot after writing state, truncate on start

* Apply consensus snapshot offset

* Fix off by one error

* Tweak debug assertion message

* Change WAL reconciliation condition, and fully clear WAL in this case

* Add debug assertion to prove Raft index and snapshot index are equal

* Add documentation to resolve bot nit

* Return error on WAL clear failure

* Fix typo

* Remove unused truncate functions
2025-11-25 11:16:01 +01:00
Roman Titov 8da26c0ded Minor tweaks to WAL compaction logs (#7580) 2025-11-25 11:16:01 +01:00
Arnaud Gourlayandtellet-q 263f987106 Storage compat. tests all supported versions (#7566)
* Storage compat. tests all supported versions

* v1.15.0 creates problems

* less parallelism to not blow the disk

* Add xdist group (#7568)

* Add session fixture and xdist group

* Increase number of workers back to 4

* Revert caching

* Remove folders with docker_client

* Fix --dist=loadgroup

* Disable v1.15.0

---------

Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
2025-11-25 11:16:01 +01:00
Ivan Pleshkovandxzfc 9ec813e0da P-Square one pass quantile estimation method (#7520)
* p square one pass method

* are you happy fmt

* fix n=9 case

* marker struct

* refactor

* proper tests

* add bench

* better namings

* are you happy clippy

* are you happy clippy

* fix additional markers order

* use ArrayVec instead of SmallVec

* use orderer float to sort f64

* Update lib/quantization/src/p_square.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* Update lib/quantization/src/p_square.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* mispelling

* review remarks

* fix typo

* change float checks order

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2025-11-25 11:15:51 +01:00
Andrey Vasnetsov f852aac448 rename common -> qdrant_common to avoid name with other python packages (#7571) 2025-11-25 11:15:50 +01:00
Ivan Boldyrev c74abf5346 Use batch searcher for sparse vector search (#7572)
It allows to remove `peek_*` methods from `FilteredScorer`, simplifying
code.
2025-11-25 11:15:50 +01:00
Arnaud Gourlay c724ca5135 Fix: use the search runtime for prepare_query_context (#7575) 2025-11-25 11:15:36 +01:00
Arnaud Gourlay dda47e8388 Fix use the update runtime for update operations (#7569)
* Fix use the update runtime for update operations

* Enable IO on update runtime

FAILED tests/consensus_tests/test_collection_recovery_limits.py::test_collection_recovery_reach_limit - Exception: HTTP request to http://127.0.0.1:33891/collections/test_collection/points?wait=true&ordering=weak failed with status code 500 after 5.20299s with response body:
{'status': {'error': 'Service internal error: task 240 panicked with message "A Tokio 1.x context was found, but IO is disabled. Call `enable_io` on the runtime builder to enable IO."'}, 'time': 3.28820663, 'usage': {'hardware': {'cpu': 0, 'payload_io_read': 0, 'payload_io_write': 70315, 'payload_index_io_read': 0, 'payload_index_io_write': 0, 'vector_io_read': 0, 'vector_io_write': 1649102}, 'inference': None}}
2025-11-25 11:15:36 +01:00
Andrey Vasnetsov 7d9ffe4553 fix default rescoring param choice for quantization rescoring (#7570) 2025-11-25 11:15:36 +01:00
Ivan Boldyrev 267fdd286c Batched scans for dense vectors (#7514)
* Batched iteration for plain `HNSWIndex` searches
* Batched iteration for `PlainVectorIndex` search
2025-11-25 11:15:36 +01:00
Andrey Vasnetsov 90a4a1f422 await for consensus before unproxifying (#7558) 2025-11-25 11:15:22 +01:00
Tim Visée 8d3e7b3534 Fix panic on startup with old storage, reenable old shard key format (#7564)
* Revert "Remove old shard key mapping format (#7047)"

This reverts commit 783c128c1a.

This reverts pull request <https://github.com/qdrant/qdrant/pull/7047>.

* Bump TODOs
2025-11-25 11:15:22 +01:00
Arnaud Gourlay a1d81f1508 Bump storage compat. data v1.16.0 (#7560)
* Bump storage compat. data v1.16.0

* forgot about v1.15.5
2025-11-25 11:15:22 +01:00
Tim Visée 8c73c3a794 Add deferred flusher to Gridstore (#7446)
* Wrap the tracker and bitmask in Arc

* Implement deferred flusher for gridstore, defer tracker writes

* Remove clone

* Dynamically adjust list of pending updates to drain what is flushed

* Return proper flusher, defer premature Gridstore flushing errors

* Use deferred Gridstore flusher across storages

* Remove all Arc<RwLock<_>> wrappers around Gridstore in storages

* Flush pages inside closure, also wrap them in a lock

* Remove direct flush function from Gridstore

* Add test to assert behavior of deferred flushing in Gridstore

* Add much more extensive test, including concurrent flushes and deletes

* Add test for draining value pointer, make sure to drop when flushed

* Feature gate RocksDB components
2025-11-25 11:15:16 +01:00
dependabot[bot]andtimvisee fb85951bd1 build(deps): bump sigstore/cosign-installer from 3.10.0 to 4.0.0 (#7431)
* build(deps): bump sigstore/cosign-installer from 3.10.0 to 4.0.0

Bumps [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) from 3.10.0 to 4.0.0.
- [Release notes](https://github.com/sigstore/cosign-installer/releases)
- [Commits](https://github.com/sigstore/cosign-installer/compare/v3.10.0...v4.0.0)

---
updated-dependencies:
- dependency-name: sigstore/cosign-installer
  dependency-version: 4.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Keep old bundle format for Harbor compatibility

* Remove trailing white spaces

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
2025-11-25 11:15:15 +01:00
Arnaud Gourlay 2b7f25b21f Missing validations for shard transfers (#7557)
* Missing validations for shard transfers

* some more

* fix test
2025-11-25 11:15:15 +01:00
Tim ViséeandRoman Titov 5cfe7ee4ee Adjust log warning about trying to read lock a segment (#7556)
* Fix log warning about trying to read lock a segment

* Fix typo

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

---------

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
2025-11-25 11:15:15 +01:00
dependabot[bot] de2f73af93 build(deps): bump lukemathwalker/cargo-chef (#7554)
Bumps lukemathwalker/cargo-chef from latest-rust-1.90-bookworm to latest-rust-1.91.1-bookworm.

---
updated-dependencies:
- dependency-name: lukemathwalker/cargo-chef
  dependency-version: latest-rust-1.91.1-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 11:15:15 +01:00
Tim Visée 4da01d375d Actively migrate away from RocksDB for vector/payload storage and index (#7551)
* Actively migrate away from RocksDB for vector/payload storage and index

* Update OpenAPI spec
2025-11-25 11:15:15 +01:00
Tim Visée bdbbfcad15 Bump filter stream records minimum version to 1.16.0 (#7549) 2025-11-25 11:14:55 +01:00
Arnaud Gourlay 9cb4e0a887 Bump rustls 0.23.35 (#7547) 2025-11-25 11:14:54 +01:00
Tim Visée 0d79528705 Bump version to 1.16.0 (#7535)
* Bump version to 1.16.0

* Update missed cherry picks

* Update redoc to 1.16.0
v1.16.0
2025-11-17 12:10:37 +01:00
Andrey Vasnetsov adcda00405 avoid docstring comments for enum fields to make openapi nicer (#7538)
* avoid docstring comments for enum fields to make openapi nicer

* add more spaces
2025-11-14 20:45:33 +01:00
Andrey Vasnetsov 85852faeb4 remove java_outer_classname (#7539) 2025-11-14 20:45:28 +01:00
Arnaud Gourlay c3470d0948 Fix consensus test port race condition (#7537)
Do not release the port before the process is really gone
otherwise it can be wrongly reused.

E.g.

2025-11-14T10:39:20.156573Z ERROR qdrant::startup: Panic occurred in file src/tonic/mod.rs at line 358:
called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))
2025-11-14 20:45:23 +01:00
Andrey Vasnetsov fbcdc0cf13 Implement ReplicatePoints in grpc (#7536)
* implement ReplicatePoints in grpc

* move filter into a separate file
2025-11-14 16:21:05 +01:00
Tim Visée 6e11c74073 Audit all spawn blocking calls, prematurely abort them (#7533)
* Prematurely abort blocking task in `spawn_cancel_on_drop` on drop

These tasks are intended to be cancellable. Now we prematurely abort the
task if the future was dropped before the task is executed.

* Prematurely abort blocking task in `spawn_cancel_on_token` on cancel

These tasks are intended to be cancellable. Now we prematurely abort the
task if the cancellation token is triggered before the task is executed.

* Prematurely abort blocking task for fetching telemetry

* Prematurely abort stoppable task on drop, all are safe to abort early

* Make `move_dir` either move everything, or nothing at all

That is with the exception of file IO errors in which case data may be
partially moved.

Before this PR it was possible for the new target directory to be
created without moving all data into it. Now we either do all, or
nothing.

* Prematurely abort task for creating full snapshot

It is fine to either create it, or not at all.

* Prematurely abort blocking task for waiting on consensus leader

* Prematurely abort blocking cardinality estimation and shard info tasks

* Prematurely abort blocking point deduplication task

* Prematurely abort blocking task for checking available disk space

* Prematurely abort blocking shard read operations

All shard read operations, such as retrieve, scroll, facets and more can
be safely aborted prematurely.

Related to: <https://github.com/qdrant/qdrant/pull/7530>

* Prematurely abort blocking task for waiting on replica state

* Prematurely abort blocking task for waiting on transfer replica states

* Prematurely abort blocking task for loading segment

This can safely be aborted before the task is started

* Prematurely abort blocking task waiting for replica states

* Prematurely abort blocking task for creating snapshot file

Safe because it aborts before writing any snapshot files to disk
2025-11-14 13:31:04 +01:00
Andrey Vasnetsov 20826e0dc7 wal_pop: binary to remove latest entry of WAL (#7526)
* binary to remove latest entry of WAL

* fix clippy
2025-11-14 12:31:08 +01:00
217875dd61 more stop flags (#7531)
* add stopflag check into query_points of payload index

* propagate stopflag more

* fmt

* bench: compare better against no atomic

* use stoppable iter with IteratorExt

* clippy

* Add cancelled error helper

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
2025-11-14 12:31:08 +01:00