Commit Graph

5465 Commits

Author SHA1 Message Date
Daniel Boros
ec6cce8291 fix/inference timeout error (#7598)
* fix: inference timeout error

* chore: cleanup timeout from inference config

* cleanup timeout from InferenceConfig

* chore: revert config.toml changes

* inference_token -> infrence_params for timeout propagation

* always use at least some timeout

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-11-25 14:25:15 +01:00
Arnaud Gourlay
82929710fd Add timeout to cancel metrics and telemetry calls (#7579)
* Add timeout to cancel metrics and telemetry calls

* relax constraints

* Configure timeout default in OpenAPI schema

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-25 12:56:05 +01:00
dependabot[bot]
871c9e0709 build(deps): bump integer-encoding from 4.0.2 to 4.1.0 (#7595)
Bumps [integer-encoding](https://github.com/dermesser/integer-encoding-rs) from 4.0.2 to 4.1.0.
- [Commits](https://github.com/dermesser/integer-encoding-rs/commits/v4.1.0)

---
updated-dependencies:
- dependency-name: integer-encoding
  dependency-version: 4.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 10:20:54 +01:00
dependabot[bot]
2a15fce109 build(deps): bump quote from 1.0.41 to 1.0.42 (#7591)
Bumps [quote](https://github.com/dtolnay/quote) from 1.0.41 to 1.0.42.
- [Release notes](https://github.com/dtolnay/quote/releases)
- [Commits](https://github.com/dtolnay/quote/compare/1.0.41...1.0.42)

---
updated-dependencies:
- dependency-name: quote
  dependency-version: 1.0.42
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-25 10:20:45 +01:00
dependabot[bot]
ec72bcb5cf build(deps): bump actions/checkout from 5 to 6 (#7589)
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-24 21:08:43 -03:00
dependabot[bot]
117fe59948 build(deps): bump clap from 4.5.49 to 4.5.53 (#7590)
Bumps [clap](https://github.com/clap-rs/clap) from 4.5.49 to 4.5.53.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.49...clap_complete-v4.5.53)

---
updated-dependencies:
- dependency-name: clap
  dependency-version: 4.5.53
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-24 21:06:48 -03:00
dependabot[bot]
c18180ea3a build(deps): bump cc from 1.2.41 to 1.2.47 (#7593)
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.2.41 to 1.2.47.
- [Release notes](https://github.com/rust-lang/cc-rs/releases)
- [Changelog](https://github.com/rust-lang/cc-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/cc-rs/compare/cc-v1.2.41...cc-v1.2.47)

---
updated-dependencies:
- dependency-name: cc
  dependency-version: 1.2.47
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-24 21:05:59 -03:00
dependabot[bot]
7f7d9981a2 build(deps): bump csv from 1.3.1 to 1.4.0 (#7594)
Bumps [csv](https://github.com/BurntSushi/rust-csv) from 1.3.1 to 1.4.0.
- [Commits](https://github.com/BurntSushi/rust-csv/compare/1.3.1...1.4.0)

---
updated-dependencies:
- dependency-name: csv
  dependency-version: 1.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-24 21:04:23 -03:00
dependabot[bot]
1f29a4ec41 build(deps): bump indicatif from 0.18.0 to 0.18.3 (#7597) 2025-11-24 23:59:10 +00:00
Andrey Vasnetsov
1a92c6904a remove unused function (#7586) 2025-11-24 17:00:45 +01:00
Tim Visée
2e14c3ea01 Bump wal to fix flush problems (#7587) 2025-11-24 16:56:03 +01:00
Tim Visée
549f13fbeb Fix WAL handling on consensus snapshot (#7577)
* After clearing WAL, flush segment

* Add debug log when WAL is cleared

* Clear WAL on consensus snapshot after writing state, truncate on start

* Apply consensus snapshot offset

* Fix off by one error

* Tweak debug assertion message

* Change WAL reconciliation condition, and fully clear WAL in this case

* Add debug assertion to prove Raft index and snapshot index are equal

* Add documentation to resolve bot nit

* Return error on WAL clear failure

* Fix typo

* Remove unused truncate functions
2025-11-24 15:49:32 +01:00
Roman Titov
f85911d3ae Minor tweaks to WAL compaction logs (#7580) 2025-11-24 15:13:32 +01:00
Arnaud Gourlay
0f2e415a72 Storage compat. tests all supported versions (#7566)
* Storage compat. tests all supported versions

* v1.15.0 creates problems

* less parallelism to not blow the disk

* Add xdist group (#7568)

* Add session fixture and xdist group

* Increase number of workers back to 4

* Revert caching

* Remove folders with docker_client

* Fix --dist=loadgroup

* Disable v1.15.0

---------

Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
2025-11-24 12:45:26 +01:00
Ivan Pleshkov
c737d9d087 P-Square one pass quantile estimation method (#7520)
* p square one pass method

* are you happy fmt

* fix n=9 case

* marker struct

* refactor

* proper tests

* add bench

* better namings

* are you happy clippy

* are you happy clippy

* fix additional markers order

* use ArrayVec instead of SmallVec

* use orderer float to sort f64

* Update lib/quantization/src/p_square.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* Update lib/quantization/src/p_square.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* mispelling

* review remarks

* fix typo

* change float checks order

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2025-11-21 20:21:32 +03:00
Andrey Vasnetsov
59ee168dcd rename common -> qdrant_common to avoid name with other python packages (#7571) 2025-11-21 15:01:54 +01:00
Ivan Boldyrev
b5825c1245 Use batch searcher for sparse vector search (#7572)
It allows to remove `peek_*` methods from `FilteredScorer`, simplifying
code.
2025-11-21 20:29:24 +07:00
Arnaud Gourlay
9147ccdfce Fix: use the search runtime for prepare_query_context (#7575) 2025-11-20 16:00:50 +01:00
Arnaud Gourlay
50cf48b818 Fix use the update runtime for update operations (#7569)
* Fix use the update runtime for update operations

* Enable IO on update runtime

FAILED tests/consensus_tests/test_collection_recovery_limits.py::test_collection_recovery_reach_limit - Exception: HTTP request to http://127.0.0.1:33891/collections/test_collection/points?wait=true&ordering=weak failed with status code 500 after 5.20299s with response body:
{'status': {'error': 'Service internal error: task 240 panicked with message "A Tokio 1.x context was found, but IO is disabled. Call `enable_io` on the runtime builder to enable IO."'}, 'time': 3.28820663, 'usage': {'hardware': {'cpu': 0, 'payload_io_read': 0, 'payload_io_write': 70315, 'payload_index_io_read': 0, 'payload_index_io_write': 0, 'vector_io_read': 0, 'vector_io_write': 1649102}, 'inference': None}}
2025-11-20 12:33:39 +01:00
Andrey Vasnetsov
43f0c73bf2 fix default rescoring param choice for quantization rescoring (#7570) 2025-11-20 12:21:28 +01:00
Ivan Boldyrev
428f1e76d4 Batched scans for dense vectors (#7514)
* Batched iteration for plain `HNSWIndex` searches
* Batched iteration for `PlainVectorIndex` search
2025-11-20 17:34:02 +07:00
Andrey Vasnetsov
113e8aab54 await for consensus before unproxifying (#7558) 2025-11-19 14:14:49 +01:00
Tim Visée
84b2fb8793 Fix panic on startup with old storage, reenable old shard key format (#7564)
* Revert "Remove old shard key mapping format (#7047)"

This reverts commit 783c128c1a.

This reverts pull request <https://github.com/qdrant/qdrant/pull/7047>.

* Bump TODOs
2025-11-19 13:40:22 +01:00
Arnaud Gourlay
af86a3b02b Bump storage compat. data v1.16.0 (#7560)
* Bump storage compat. data v1.16.0

* forgot about v1.15.5
2025-11-19 10:53:39 +01:00
Tim Visée
8bbeda5783 Add deferred flusher to Gridstore (#7446)
* Wrap the tracker and bitmask in Arc

* Implement deferred flusher for gridstore, defer tracker writes

* Remove clone

* Dynamically adjust list of pending updates to drain what is flushed

* Return proper flusher, defer premature Gridstore flushing errors

* Use deferred Gridstore flusher across storages

* Remove all Arc<RwLock<_>> wrappers around Gridstore in storages

* Flush pages inside closure, also wrap them in a lock

* Remove direct flush function from Gridstore

* Add test to assert behavior of deferred flushing in Gridstore

* Add much more extensive test, including concurrent flushes and deletes

* Add test for draining value pointer, make sure to drop when flushed

* Feature gate RocksDB components
2025-11-18 14:21:43 +01:00
dependabot[bot]
8383bc6471 build(deps): bump sigstore/cosign-installer from 3.10.0 to 4.0.0 (#7431)
* build(deps): bump sigstore/cosign-installer from 3.10.0 to 4.0.0

Bumps [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) from 3.10.0 to 4.0.0.
- [Release notes](https://github.com/sigstore/cosign-installer/releases)
- [Commits](https://github.com/sigstore/cosign-installer/compare/v3.10.0...v4.0.0)

---
updated-dependencies:
- dependency-name: sigstore/cosign-installer
  dependency-version: 4.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Keep old bundle format for Harbor compatibility

* Remove trailing white spaces

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
2025-11-18 13:24:48 +01:00
Arnaud Gourlay
b784799b69 Missing validations for shard transfers (#7557)
* Missing validations for shard transfers

* some more

* fix test
2025-11-18 13:08:51 +01:00
Tim Visée
986a55a5e0 Adjust log warning about trying to read lock a segment (#7556)
* Fix log warning about trying to read lock a segment

* Fix typo

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

---------

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
2025-11-18 12:40:34 +01:00
dependabot[bot]
88fd3c1de6 build(deps): bump lukemathwalker/cargo-chef (#7554)
Bumps lukemathwalker/cargo-chef from latest-rust-1.90-bookworm to latest-rust-1.91.1-bookworm.

---
updated-dependencies:
- dependency-name: lukemathwalker/cargo-chef
  dependency-version: latest-rust-1.91.1-bookworm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-18 11:19:26 +01:00
Tim Visée
fca10199b4 Actively migrate away from RocksDB for vector/payload storage and index (#7551)
* Actively migrate away from RocksDB for vector/payload storage and index

* Update OpenAPI spec
2025-11-18 11:18:46 +01:00
Tim Visée
b687408230 Bump filter stream records minimum version to 1.16.0 (#7549) 2025-11-17 16:54:17 +01:00
Arnaud Gourlay
5da8f41918 Bump rustls 0.23.35 (#7547) 2025-11-17 16:04:01 +01:00
Tim Visée
77cd5962c2 Bump dev version to v1.16.1-dev (#7548) 2025-11-17 15:45:10 +01:00
Andrey Vasnetsov
411223d7f7 avoid docstring comments for enum fields to make openapi nicer (#7538)
* avoid docstring comments for enum fields to make openapi nicer

* add more spaces
2025-11-14 20:31:33 +01:00
Andrey Vasnetsov
355cf3624c remove java_outer_classname (#7539) 2025-11-14 18:24:58 +01:00
Arnaud Gourlay
ed006a6ef9 Fix consensus test port race condition (#7537)
Do not release the port before the process is really gone
otherwise it can be wrongly reused.

E.g.

2025-11-14T10:39:20.156573Z ERROR qdrant::startup: Panic occurred in file src/tonic/mod.rs at line 358:
called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))
2025-11-14 17:24:02 +01:00
Andrey Vasnetsov
4420165464 Implement ReplicatePoints in grpc (#7536)
* implement ReplicatePoints in grpc

* move filter into a separate file
2025-11-14 16:12:29 +01:00
Tim Visée
1b6e52554a Audit all spawn blocking calls, prematurely abort them (#7533)
* Prematurely abort blocking task in `spawn_cancel_on_drop` on drop

These tasks are intended to be cancellable. Now we prematurely abort the
task if the future was dropped before the task is executed.

* Prematurely abort blocking task in `spawn_cancel_on_token` on cancel

These tasks are intended to be cancellable. Now we prematurely abort the
task if the cancellation token is triggered before the task is executed.

* Prematurely abort blocking task for fetching telemetry

* Prematurely abort stoppable task on drop, all are safe to abort early

* Make `move_dir` either move everything, or nothing at all

That is with the exception of file IO errors in which case data may be
partially moved.

Before this PR it was possible for the new target directory to be
created without moving all data into it. Now we either do all, or
nothing.

* Prematurely abort task for creating full snapshot

It is fine to either create it, or not at all.

* Prematurely abort blocking task for waiting on consensus leader

* Prematurely abort blocking cardinality estimation and shard info tasks

* Prematurely abort blocking point deduplication task

* Prematurely abort blocking task for checking available disk space

* Prematurely abort blocking shard read operations

All shard read operations, such as retrieve, scroll, facets and more can
be safely aborted prematurely.

Related to: <https://github.com/qdrant/qdrant/pull/7530>

* Prematurely abort blocking task for waiting on replica state

* Prematurely abort blocking task for waiting on transfer replica states

* Prematurely abort blocking task for loading segment

This can safely be aborted before the task is started

* Prematurely abort blocking task waiting for replica states

* Prematurely abort blocking task for creating snapshot file

Safe because it aborts before writing any snapshot files to disk
2025-11-14 13:29:06 +01:00
Andrey Vasnetsov
47b2e539a4 wal_pop: binary to remove latest entry of WAL (#7526)
* binary to remove latest entry of WAL

* fix clippy
2025-11-14 11:04:33 +01:00
Andrey Vasnetsov
ccabe47a08 more stop flags (#7531)
* add stopflag check into query_points of payload index

* propagate stopflag more

* fmt

* bench: compare better against no atomic

* use stoppable iter with IteratorExt

* clippy

* Add cancelled error helper

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
2025-11-14 10:58:04 +01:00
Tim Visée
c86aa1a000 Abort pending blocking search tasks when we drop/invalidate the search (#7530)
* Cancel pending blocking search tasks when we drop/invalidate the search

* Mention PR with explanation in comment
2025-11-14 08:49:33 +01:00
Roman Titov
80ae57f0b3 Allow payload operations on dummy shard (#7521)
Co-authored-by: Tim Visée <tim+github@visee.me>
2025-11-13 18:31:41 +01:00
Roman Titov
190f1cd726 Restrict /logger API (#7527) 2025-11-13 16:21:50 +01:00
Roman Titov
b15bc53dce Qdrant Edge CI (#7525) 2025-11-13 15:23:06 +01:00
Andrey Vasnetsov
dee409b928 allow initial_state in API (#7528) 2025-11-13 14:12:07 +01:00
Arnaud Gourlay
204a27f3bc Fix missing version debug log (#7524)
* Fix missing version debug log

* we do not need curly braces where we are going
2025-11-12 17:18:44 +01:00
Arnaud Gourlay
b7da071199 Clarify multivector inline storage warning (#7523)
* Clarify multivector inline storage warning

* do what I actually wanted to do in first place

* fix test
2025-11-12 15:52:04 +01:00
Kyamran
b217d091c9 Fix docs/DEVELOPMENT.md (#7518) 2025-11-12 15:26:22 +01:00
Andrey Vasnetsov
e34a6e126e Mark legacy output vector fields deprecated (#7183)
* mark legacy output vector fields deprecated

* grpc docs

* Also mark legacy vector input fields as deprecated

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-12 15:21:54 +01:00
Andrey Vasnetsov
65312a1e82 implement shard key init state in grpc (#7522)
* implement shard key init state in grpc

* Update lib/api/src/grpc/proto/collections.proto

Co-authored-by: Tim Visée <tim+github@visee.me>

* sync grpc autogen

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2025-11-12 15:05:17 +01:00