Commit Graph
541 Commits
Author SHA1 Message Date
Andrey Vasnetsov d16c62c76c Expose shard snapshot API (#8132)
* [AI] generate openapi schema for stream download of shard snapshot

* bump number of APIs

* cover API with access test

* fix test

* fix test
2026-02-17 11:00:47 +01:00
Arnaud Gourlay cc3f578b5c Remove update queue opnum from CollectionInfo (#8127) 2026-02-16 10:17:29 +01:00
Arnaud Gourlay 6b139a823d uv dependencies upgrade (#8124) 2026-02-16 10:17:28 +01:00
Roman Titov bdd4bb5180 Ensure WAL and shard clocks consistency when creating shard snapshot (#8104) 2026-02-13 11:52:23 +01:00
9fdc92e615 Update Python dependencies: urllib3, protobuf, werkzeug, pyasn1 (#8119)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: agourlay <606963+agourlay@users.noreply.github.com>
2026-02-13 09:55:50 +01:00
Andrey Vasnetsov 7e1a8325dc Ignore cacert for internal requests if not configured (#8099)
* ignore cert verification if not configured

* also ignore hostnames

* script for manually testing tls clsuter
2026-02-13 09:55:46 +01:00
Tim Visée caa5a14866 Fix stream records transfer data race, losing pending updates (#8103)
* Add integration test to assert all queued updates are also transferred

* Plunge update queue in stream records transfers

* Migrate existing plunger usages to new plunge helper

* Skip test if not compiled with staging flag

* Only send delay operation when staging feature is enabled

* Reformat

* Fix review remarks
2026-02-13 09:55:37 +01:00
Luis CossíoandArnaud Gourlay bc6246df5a [relevance feedback] add rest and grpc interfaces (#7399)
* add rest and grpc interfaces

Also handle inference for this query

* follow refactor from base branch

* renaming from Ms Cooper

* get started on validation testing

* more validations

* test equivalence with query when less than 2 feedback elements

* rename feedback query to relevance_feedback query

* fix extraction of context pairs

* rename feedback vector to example

* make coderabbit happier

* upd test

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2026-02-13 09:55:37 +01:00
tellet-q 22fb1c5bc4 Add a test to check if creating snapshots blocks other operations (#7510)
* Add a test to check if creating snapshots blocks other operations
2026-02-13 09:55:36 +01:00
tellet-q 4cf0bbd63c Add test for consensus' freeze during streaming snapshot (#8075) 2026-02-13 09:55:36 +01:00
Luis Cossío b88a1d76ce fix: handle score_threshold in Formula queries (#8097)
* handle `score_threshold` in Formula queries

* AI: add openapi test

prompt: Upload 4 points with a numeric payload, use the payload as the
score, and set a score threshold. We can assert which ids should be in
the result and which shouldn't
2026-02-13 09:55:18 +01:00
Andrey Vasnetsov 037600e8af Weighted rrf (#8063)
* weighted rrf implementation

* test

* fmt

* fix edge

* validate number of sources and number of weights

* do not partial match

* upd schema

* review fixes

* update formula

* remove calcualtions from tests

* update comment, because AI have OCD

* fmt
2026-02-10 00:04:33 +01:00
tellet-qandtimvisee 79d0678440 Add more stages for shard transfer profiling (#8027)
* Improve stages reporting

* Remove transferring records info from snapshot transfer's comment

* Add detailed profiling for ReshardingStreamRecords

* Address AI review comment

* Remove some nesting

---------

Co-authored-by: timvisee <tim@visee.me>
2026-02-10 00:03:26 +01:00
Ivan Pleshkov 78207079dd Update queue status in metrics and telemetry (#8060)
* update queue status in metrics and telemetry

* fix python tests

* update openapi
2026-02-10 00:01:27 +01:00
Arnaud Gourlayandxzfc aeeafe4a0e Update queue info into CollectionInfo (#8010)
* Update queue info into CollectionInfo

* basic test

* fix

* cleaner

* Update lib/collection/src/shards/local_shard/mod.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* fix the fix

* rename to op_num

* skip_serializing_if just in case

* first step

* test queue length with staging feature

* gate integration test based on binary feature

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2026-02-09 23:39:24 +01:00
Tim Visée 7d2a9657ac Use local telemetry for peer version check, not cluster level telemetry (#8016) 2026-02-09 23:39:17 +01:00
Andrey Vasnetsov c755e6bfe8 more reliable version check (#8002) 2026-02-09 23:37:14 +01:00
tellet-q 55969c7e97 Track shard transfer stages and duration (#8001)
* Track shard transfer stages and duration
2026-02-09 23:36:50 +01:00
Andrey Vasnetsov 9099a7ac1b introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert (#7963)
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert

* add test

* upd dockstring

* require resharding once all peers have updated version

* use service error

* fix clippy again

* wait for same version before resharding in tests
2026-02-09 23:28:45 +01:00
xzfc 8fb203eeb5 Update GET collections/{}/optimizations (#7946)
* refactor(SegmentOptimizer::name): make it `&'static`

* refactor(TrackerStatus): add is_running()

* refactor: add TrackerSegmentInfo

* feat: rework optimizations endpoint

* feat(collections/{}/optimizations): trim spaces
2026-02-09 23:28:03 +01:00
Daniel Borosandgenerall 00e0a7fc24 feat/test-timeout (#7920)
* feat: add timeout tests

* fix: timeout test

* fix: api urls

* fix: test

* feat: add staging api def

* feat: add parallel executor for tests

* fix: imports

* fix: ci/cd

* rollback openapi schema change

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-02-09 23:15:44 +01:00
Andrey Vasnetsov ac2c18e1fe Validate vector name in matrix request (#7923)
* validate `using` param

* fix slop
2026-02-09 23:08:14 +01:00
Kumar Shivendu a036211a0f Test behaviour when having no shards in custom sharding (#7867)
* Test behaviour when having no shards in custom sharding

* minor suggestion
2026-02-09 22:56:17 +01:00
SapphireandEC2 Default User 37de1f219d Add enable_hnsw option for payload field schema (#7887)
* feat: Add enable_hnsw option for payload field indexes

Add optional enable_hnsw parameter to all payload index types to control
whether additional HNSW graph links are built for each indexed field.

- Add enable_hnsw field to all 8 payload index param types
- Update gRPC proto definitions and conversions
- Update OpenAPI schema
- Modify HNSW graph builder to respect enable_hnsw flag
- Add enable_hnsw() helper methods to PayloadSchemaParams and PayloadFieldSchema
- Update all tests to include new field (default: None)

When enable_hnsw is true and payload_M > 0, additional HNSW links will
be built for the payload field. Default value is true for backward compatibility.

* Fix Some format problems

* fix: address comment problem

---------

Co-authored-by: EC2 Default User <ec2-user@ip-10-78-171-148.ec2.internal>
2026-02-09 22:53:47 +01:00
tellet-qandgenerall 30a34642b9 Add a test for recovery after kill during Partial (#7762)
* Add a test for recovery after kill during Partial

* Address AI review

* simplify test

* Merge pull request #7829

* introduce a new state

* switch to ManualRecovery for user-initiated snapshot operations

* fmt

* fix test and regen api

* test fix: instead of force recovery on update failed snapshots now re…

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-02-09 22:35:29 +01:00
Andrey Vasnetsov a21b641d26 Merge pull request #7835
* alt api key

* rm typo

* fix key length check logic

* differnt dirs for tests

* fix tests

* rm unused
2026-02-09 22:35:18 +01:00
Luis Cossío d889ff90ee allow collection-level jwt access (#7758) 2026-02-09 22:31:53 +01:00
Luis Cossío 45c560b5bc [cluster telemetry] Expose endpoint (#7729)
* expose cluster telemetry endpoint

* add endpoint to openapi

* add jwt test

* remove dead_code expectation

* fix consistency check

* fix jwt access test

* use response(reference(..)) in openapi
2026-02-09 22:31:32 +01:00
tellet-q f1d54ca889 Move test dependencies to tests (#7793)
* Move pyproject and uv.lock to tests folder

* Make tests working-dir agnostic

* Fix test

* Address review
2025-12-18 17:29:18 +01:00
Tim Visée e35d452043 Add test for broken WAL delta after stream records abort (#7787)
* Add test to reproduce broken WAL delta after aborting stream records

* Add staging env var to slow down stream records transfers for test

* Tweak test formatting and utilities a bit

* Add comment to test, link to PR describing bug

* Update test so it still succeeds with patched behavior

* Fix broken WAL delta after stream records abort (#7791)

* Make set_replica_state async

* Add function called when active state of local replica changes

* Add snapshot for newest clocks

* Bump newest clocks snapshot on replica deactivation

* Use newest clocks snapshot during recovery

* Add enum for specifying whether to take or clear clocks snapshot

* Store clock snapshot inside clock map, removing extra file

This greatly simplifies state handling. It also prevent any kind of
desynchronization because all newest clocks are always persisted
atomically.

* Immediately persist clocks after taking snapshot

* Always update snapshot, only take if missing

* Take clock snapshots through each shard flavor, including proxies

* Propagate dedicated functions for taking and clearing clocks snapshot

* Only persist clocks immediately if changed on snapshot/clear

* Simplify recovery point logic, always take clocks snapshot if exists

* Remove unwrap

* Fix typo

* Fix doc comment

* Transfer driver is async, use Tokio sleep

* Reduce visibility
2025-12-18 17:29:16 +01:00
b234b55a4c Migrate Python to uv (#7790)
* Move pyproject.toml to root

* Migrate pyproject.toml from Poetry to uv

* Update GH workflows

* Update test script, doc and nix to use uv

* Use latest uv

* Fix uv.lock

* Cleanup shell.nix

* Cleanup

- Explicit `uv sync` is not required, `uv run` will install deps
  automatically.
- We don't provide a python package, so the `[build-system]` section
  is not needed.

* Fix UV_VERSION inconsistency

---------

Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
2025-12-18 17:28:50 +01:00
Arnaud Gourlay 7a4c6fd9cb Bump misc. integration test deps (#7746) 2025-12-18 17:27:42 +01:00
tellet-q c31d165601 Download archives once (#7711) 2025-12-18 17:27:28 +01:00
Andrey Vasnetsov 2101fbdcec ignore shards with no active replicas (#7707) 2025-12-18 17:27:20 +01:00
xzfc 8013c73978 Ignore deprecation warnings for schemathesis (#7730) 2025-12-18 17:27:18 +01:00
xzfc 6c0d10abe7 Indexing progress (#7625)
* Add progress_tracker.rs

* Pass progress tracker around

* Populate progress tracker with actual data

* Expose progress on `/collections/{name}/optimizations` endpoint
2025-12-18 17:27:18 +01:00
Roman Titov e33b4672e5 Fix test_collection_recovery test (#7700) 2025-12-18 17:26:37 +01:00
Roman Titov d2834de0b5 Mark *new* local replicas as locally-disabled, when recovering Raft snapshot (#7684)
* Remove faulty mark-as-dead condition when recovering Raft snapshot 😅

* Mark new local replicas as locally-disabled when recovering Raft snapshot

* Add test
2025-12-04 08:44:02 +01:00
Tim Viséeanddependabot[bot] ec0a3389f1 build(deps): bump werkzeug from 3.1.3 to 3.1.4 in /tests (#7671)
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.1.3 to 3.1.4.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/werkzeug/compare/3.1.3...3.1.4)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-version: 3.1.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-03 10:23:55 +01:00
Jojii 3a19714f49 Make indexed-only-excluded-points metric default to 0 (#7649)
* Make indexed_only_excluded in metrics default to 0

* Fix tests
2025-12-03 10:23:55 +01:00
tellet-q c6f6224de7 Long running e2e test on many collections (#7576)
* Add test for many collections

* Reduce number of collections
2025-12-03 10:23:25 +01:00
Arnaud Gourlay 504c7156fa Test storage compat. v1.6.1 (#7648) 2025-12-03 10:23:25 +01:00
AMIR 8eec2046d7 Add list shard keys API (#7615) 2025-12-03 10:20:40 +01:00
Arnaud Gourlay 9bfeed0292 Fix Openapi schema for DELETE issues (#7634) 2025-12-03 10:20:34 +01:00
Arnaud Gourlay f638d9f24e Document storage compatibility test for v1.15.0 (#7601)
* Enable storage compatibility test for v1.15.0

* proper explanation
2025-12-03 10:18:52 +01:00
Arnaud Gourlayandtellet-q 263f987106 Storage compat. tests all supported versions (#7566)
* Storage compat. tests all supported versions

* v1.15.0 creates problems

* less parallelism to not blow the disk

* Add xdist group (#7568)

* Add session fixture and xdist group

* Increase number of workers back to 4

* Revert caching

* Remove folders with docker_client

* Fix --dist=loadgroup

* Disable v1.15.0

---------

Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
2025-11-25 11:16:01 +01:00
Arnaud Gourlay a1d81f1508 Bump storage compat. data v1.16.0 (#7560)
* Bump storage compat. data v1.16.0

* forgot about v1.15.5
2025-11-25 11:15:22 +01:00
Arnaud Gourlay 2b7f25b21f Missing validations for shard transfers (#7557)
* Missing validations for shard transfers

* some more

* fix test
2025-11-25 11:15:15 +01:00
Arnaud Gourlay c3470d0948 Fix consensus test port race condition (#7537)
Do not release the port before the process is really gone
otherwise it can be wrongly reused.

E.g.

2025-11-14T10:39:20.156573Z ERROR qdrant::startup: Panic occurred in file src/tonic/mod.rs at line 358:
called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))
2025-11-14 20:45:23 +01:00
Roman Titov 32b7fdfb7f Restrict /logger API (#7527) 2025-11-14 12:31:08 +01:00