Commit Graph

544 Commits

Author SHA1 Message Date
Arnaud Gourlay
af86a3b02b Bump storage compat. data v1.16.0 (#7560)
* Bump storage compat. data v1.16.0

* forgot about v1.15.5
2025-11-19 10:53:39 +01:00
Arnaud Gourlay
b784799b69 Missing validations for shard transfers (#7557)
* Missing validations for shard transfers

* some more

* fix test
2025-11-18 13:08:51 +01:00
Arnaud Gourlay
ed006a6ef9 Fix consensus test port race condition (#7537)
Do not release the port before the process is really gone
otherwise it can be wrongly reused.

E.g.

2025-11-14T10:39:20.156573Z ERROR qdrant::startup: Panic occurred in file src/tonic/mod.rs at line 358:
called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))
2025-11-14 17:24:02 +01:00
Roman Titov
190f1cd726 Restrict /logger API (#7527) 2025-11-13 16:21:50 +01:00
Arnaud Gourlay
b7da071199 Clarify multivector inline storage warning (#7523)
* Clarify multivector inline storage warning

* do what I actually wanted to do in first place

* fix test
2025-11-12 15:52:04 +01:00
Tim Visée
c3d2d5d291 Fix hash ring initialization for custom sharding, prevent resharding panic (#7517)
* Correctly rebuild hash rings, don't use default with custom sharding

* Correctly initialize hash ring map with custom sharding

Don't load the non key ring by default

* Validate shard key for resharding more aggressively

When using auto sharding, require no shard key. When using custom
sharding, require a shard key.

* Only pass sharding method into shard holder construction

* Remove now obsolete debug assertion, we use same source config

* Correctly reconstruct hash rings, also add shard IDs in auto mode

* Add test, do not break consensus on invalid resharding down operation

* Add tests for resharding down shard count precondition

* Minor test tweaks

* Improve debug assertion

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

---------

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
2025-11-12 11:25:24 +01:00
Jojii
f6b715bd62 Metrics for snapshots (#7497)
* Add atomic counter for running snapshots

* Refactor collection telemetry + export prometheus metric

* Handle collection in ToC + current recovery measurements

* Fix openapi tests

* Fix tests

* Add snapshot metrics for streaming and partial snapshots.

* Fix typo in metrics name

* Adjust metrics names

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-11 15:00:11 +01:00
Arnaud Gourlay
c0a6f86af7 Fix flaky test consensus compaction (#7513) 2025-11-11 12:48:45 +01:00
Andrey Vasnetsov
bf87170db3 promotion integration test (#7511)
* test with background update process

* fmt

* remove unused imports

* Remove unused import

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-11 11:50:13 +01:00
Andrey Vasnetsov
fe06b8559d simple integration test for tenant promotion (#7503)
* simple integration test for tenant promotion

* add cleanup after promotion

* When switching to ReadActive state, set it for the correct peer ID

* fix test

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-10 17:42:05 +01:00
Andrey Vasnetsov
7994816a1e remove filter level jwt (#7450)
* make jwt with payload filter fail

* remove `whole` access requirement

* remove unnecessary mut

* fmt

* adjust test

* fix test

* remove more tests

* fix test

* fix test again

* fix test again

* Replace deprecated PayloadConstraint with JSON Value placeholder

* Fix expect message

---------

Co-authored-by: timvisee <tim@visee.me>
2025-10-31 10:03:04 +01:00
Andrey Vasnetsov
925d5127c8 remove init_from (#7454)
* wip: remove init_from

* remove init_from tests

* upd schema

* rm test
2025-10-29 17:37:02 +01:00
Andrey Vasnetsov
c1af720d0d remove lock api (#7449)
* remove lock api

* fix api count + remove tests

* remove another test

* remove more tests
2025-10-29 17:36:54 +01:00
Jojii
48d61f1719 Metrics vectors by name per collection (#7441)
* Add vector count per vector-name to metrics API

* Add to metrics API

* Improve TinyMap::get_or_insert_default and add tests

* Minor improvements

* Update openapi

* Review remarks

* Remove `collection_vectors` since it can be calculated manually
2025-10-29 17:00:53 +01:00
Kumar Shivendu
9f0779ed16 Respect payload filters while replicating point (#7362)
* Respect payload and hashring filters while replicating point

* Merge points instead of replacing for filtered stream records transfer

* cleanup

* Update utils

* Allow filters in transfer

* drop hashring filter since its not required when destination has single shard

* test comment

* Pass actual filter

* use exact=True

* Forward updates that satisfy the filter

* make test long enough for extra points to be upserted

* minor improvements

* fix test

* try with only inserting new points

* Temporarily forward points that match before or after

* Fix failing test

* Only do inserts

* Dont merge points and update old points in test

* Fix the bug in integration test

* fix read_batch_with_hashring fn name and docs

* changes after rebase

* add validation check

* apply suggestions

* Always trim clock tag when forwarding to a different shard ID

* review fixes

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-10-29 19:41:55 +05:30
Tim Visée
851d30a679 build(deps): bump starlette from 0.47.2 to 0.49.1 in /tests (#7470)
Bumps [starlette](https://github.com/Kludex/starlette) from 0.47.2 to 0.49.1.
- [Release notes](https://github.com/Kludex/starlette/releases)
- [Changelog](https://github.com/Kludex/starlette/blob/main/docs/release-notes.md)
- [Commits](https://github.com/Kludex/starlette/compare/0.47.2...0.49.1)

---
updated-dependencies:
- dependency-name: starlette
  dependency-version: 0.49.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-29 13:10:29 +01:00
xzfc
962c036b49 Drop docs/grpc/docs.md (#7453) 2025-10-27 11:23:26 +00:00
Kumar Shivendu
addb74cc28 Implement logic to replicate points between shards (#7361)
* Allow replicating points between shards

* Add integration test for replicating points between shards

* Drop filter from ForwardProxyShard to for now

* Clean up test

* fix assert failures

* Fix integration test

* Drop reliance on FilteredStreamRecords

* minor improvements
2025-10-24 16:39:04 +05:30
Tim Visée
cafe4aa482 Remove trailing whitespaces (#7435) 2025-10-21 14:05:57 +02:00
Kumar Shivendu
c0f580bddd Shard key initial state (#7371)
* Allow specifying initial state while creating shard key

* Dont wait for Active state if initial_state is set

* Generate gRPC docs and OpenAPI spec

* Apply suggestions

* Skip from scheamrs not serde

* Add integration test

* minor suggestions
2025-10-13 17:58:34 +05:30
Tim Visée
b9243c5af4 Forbid peer to join cluster with URI that is already used (#7375)
* Disallow peer to join with URI that is already used

* Add test for rejecting peer join with duplicate URI

* Improve peer rejection logic

* Try to rejoin twice, we expect a consistent result
2025-10-13 12:32:51 +02:00
xzfc
95b4bbf978 Rename HnswConfig::{copy_vectors -> inline_storage} (#7389) 2025-10-13 10:08:06 +00:00
Andrey Vasnetsov
34596530d1 remove default explicit config for strict mode (#7369)
* remove default explicit config for strict mode

* Don't prefix commented out properties with space to remain consistent

* Don't crash test if strict mode config is not set at all

---------

Co-authored-by: timvisee <tim@visee.me>
2025-10-09 12:00:01 +02:00
Arnaud Gourlay
a24d380ee0 Investigate flaky snapshot transfer test (#7339) 2025-10-09 10:47:23 +02:00
Arnaud Gourlay
fe94fdc5ce Bump Python 3.13 on CI (#7365)
* Bump Python 3.13 on CI

* Regenerate tls certificates

---------

Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
2025-10-08 12:50:34 +02:00
tellet-q
c3ca182423 Refactor bash tests: storage-compatibility (#7299)
* Add e2e test for compatibility
2025-10-08 08:33:55 +02:00
Andrey Vasnetsov
8dd915c0b1 non appendable proxies (#7345)
* WIP: make proxy segment non-appendable

* make deleted points and index changes private to Proxy Segment

* consistently remove empty unused segment from holder on unproxifying and the end of optimization

* remove/fix obsolete test

* Fix comment, some general adjustments

* Also don't remove non-original segments

* Use ahash for map of proxy index changes

* Inline format arguments

* Simplify has point branching

* Assert that proxy flushes to latest wrapped segment version

* make vector deletetion a CoW operation

* relax debug assertions

* Update lib/shard/src/proxy_segment/mod.rs

* fix test_partial_snapshot_empty

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
2025-10-08 00:19:41 +02:00
Tim Visée
a1efafbbf3 Only wait for green collection status in empty snapshot test (#7360) 2025-10-07 18:12:59 +02:00
Tim Visée
3856ff6bdb Before doing actual partial snapshot tests, wait for green status (#7358) 2025-10-07 15:57:46 +02:00
xzfc
780d406ff8 Add CollectionInfo::warnings field (#7293)
* refactor: swap DiffConfig parmeters

was: diff.update(&config)
now: config.update(diff)

* refactor: introduce DiffConfig::update_opt

* feat: add CollectionInfo::configuration_status field

* rename ConfigurationStatus to CollectionWarning
2025-09-29 18:17:03 +00:00
Arnaud Gourlay
f5d824508e Regen storage compat. v1.15.4 (#7321)
* Regen storage compat. 1.14 & 1.15

* bump prev patch to v1.15.4

* skip v1.14.1

* cleaner

* sleepy
2025-09-29 16:37:16 +02:00
Arnaud Gourlay
04d8d05aa5 Fix deadlock on Actix RT in Collection Info (#7277) 2025-09-22 11:14:27 +02:00
Arnaud Gourlay
8a31160297 Always run Telemetry on a blocking thread (#7269)
* Always run Telemetry on a blocking thread

* run Telemetry on general runtime instead of the blocking Actix one

* Polish test
2025-09-19 10:28:09 +02:00
Arnaud Gourlay
f66dc31452 Fix deadlock on Actix RT in Cluster Info (#7267) 2025-09-17 15:54:11 +02:00
Arnaud Gourlay
5238d16909 Fix deadlock on Actix RT in Telemetry (#7265)
* Deadlock on Actix RT

* Spawn blocking task for sync lock
2025-09-17 10:57:00 +02:00
Andrey Vasnetsov
bff9cb1a50 fix strict mode validation for nested filters (#7264)
* fix strict mode validation for nested filters

* fix test
2025-09-16 12:03:54 +02:00
Arnaud Gourlay
e79100a2f2 Fix phrase match to not match with unknown tokens (#7252)
* Fix phrase match to not match with unknown tokens

* add tests

* spelling
2025-09-16 11:16:17 +02:00
Arnaud Gourlay
bc13f01680 [strict-mode] Add max number of payload index count (#7222)
* [strict-mode] Add max number of payload index count

* improve docs and validation for StrictModeConfig

* fix from coderabbit

* test blocking access to payload indexes

* polish test
2025-09-15 13:12:40 +02:00
Arnaud Gourlay
04d1c194e6 Fix RecommendExample conversion to support new format (#7234) 2025-09-11 17:36:29 +02:00
Roman Titov
a3ce716590 Fix deadlock during streaming shard snapshot (#7241)
* Add `test_shard_snapshot_deadlock` integration test

* Fix deadlock during streaming shard snapshot
2025-09-11 14:55:03 +02:00
Arnaud Gourlay
f30e7f7273 Fix missing REST API validation for point batch update (#7219)
* Fix missing REST API validation for point batch update

* use a valid vector
2025-09-05 18:25:37 +02:00
tellet-q
23a20261cf Add e2e test for snapshots (#7066)
* Add e2e test that mimics 1 crasher's scenario
2025-09-05 09:14:59 +02:00
Tim Visée
d78290b08c build(deps): bump h2 from 4.2.0 to 4.3.0 in /tests (#7167)
Bumps [h2](https://github.com/python-hyper/h2) from 4.2.0 to 4.3.0.
- [Changelog](https://github.com/python-hyper/h2/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/python-hyper/h2/compare/v4.2.0...v4.3.0)

---
updated-dependencies:
- dependency-name: h2
  dependency-version: 4.3.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-28 15:36:21 +02:00
tellet-q
5936926c88 Refactor bash test for snapshots recovery (#7008)
* Refactor bash test for snapshots recovery

* Address review: minio check

* Address review
2025-08-27 15:25:27 +02:00
tellet-q
6c75566ac4 Add tests for immutable text index (#7153)
* Add unit test

* Flush text index deletes before reopening them

* Fix unused warning if compiling without RocksDB

* Add e2e test

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-27 12:18:58 +02:00
Andrey Vasnetsov
e744bd6014 implement custom collection metadata (#7123)
* implement custom collection metadata

* persist metadata change

* Also unset a key in the test

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-26 11:49:17 +02:00
Andrey Vasnetsov
bf60e96619 filter text any (#7100)
* internal implementation for filter any

* implement api for match `text_any`

* allow naive implementation of text match any

* dedup after kmerge + renames

* improve `expected_should_estimation`

* congruence test

* openapi test

* setup index in each test

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-08-24 12:09:05 +02:00
Andrey Vasnetsov
c188f5b7c4 Update if (#7006)
* implement condition parameter for upsert operation

* fmt

* update api schema

* implement conditional update for update-vectors op

* rename to update_filter for consistency

* add tests

* explicilty ignore strict mode for conditional updates

* rabbit review fix

* Also assert vector element before, confirm we don't normalize

* Simplify filter creation using new helper

* Rename merge_with_ids to with_point_ids

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
2025-08-15 15:47:43 +02:00
Jojii
290f9e2675 Always load InferenceService (#7050)
* Always load InferenceService

* Add BM25 openapi test

* Add warning to development.yaml config (#7051)
2025-08-13 16:59:42 +02:00
tellet-q
f0612d6680 Refactor bash tests: tls (#6995)
* Refactor tls bash test

* Update codespell

* Address review
2025-08-11 10:05:47 +02:00