upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-09-21 13:37:46 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,263 Commits 438 Branches 119 Tags
07f12e1277281ea7ea139a3d84bd860f5e6b9637
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Roman Titov 3b2cdd6b9b Always Copy-on-Write when updating payload in immutable segments (#7952) 2026-02-09 23:17:56 +01:00
Roman Titov da70bf13c6 Update qdrant-edge-py version to 0.4.0 (#7926) 
* Update `qdrant-edge-py` version to 0.4.0

* fixup! Update `qdrant-edge-py` version to 0.4.0

🙄🙄🙄
 2026-02-09 23:08:27 +01:00
Roman Titov 3184b53a32 Rename edge::Shard into edge::EdgeShard (#7925) 2026-02-09 23:08:22 +01:00
Roman Titov 1a15be85ef Flush edge::Shard on Drop (#7911) 
And add explicit `flush` and `close` methods
 2026-02-09 23:07:20 +01:00
Roman TitovandAndrey Vasnetsov 4c7ed0de44 Implement info request for Qdrant Edge (#7890) 
* Implement `edge::Shard::info`

* Add `ReprStr` marker-trait

* Add Python bindings for payload index types

* Implement `PyShard::info`

* fixup! Add Python bindings for payload index types

Add `enable_hnsw` fields

* add __repr__ and exted tests

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2026-02-09 23:02:57 +01:00
Roman Titovandgenerall fd18177766 Implement scroll and count requests for Qdrant Edge (#7880) 
* Cleanup `shard` crate module declarations

* Move `ScrollRequestInternal` into `shard` crate

* fixup! Move `ScrollRequestInternal` into `shard` crate

Fix imports

* fixup! Move `ScrollRequestInternal` into `shard` crate

`const fn default_*`

* Implement `edge::Shard::scroll`

* fixup! Implement `edge::Shard::scroll`

Re-export `OrderByInterface`

* Cleanup `edge` module declarations

* Cleanup `qdrant-edge-py` module declarations

* Move `PyWithPayload` and `PyWithVector` into `types::query`

* Add `PyScrollRequest` type

* Implement `PyShard::scroll`

* Move `CountRequestInternal` into `shard` crate

* fixup! Move `CountRequestInternal` into `shard` crate

Fix imports

* fixup! Move `CountRequestInternal` into `shard` crate

Rename `default_exact_count` into `CountRequestInternal::default_exact`

* Implement `edge::Shard::count`

* Implement `PyShard::count`

* review: offset for scroll, default values, examples

* ai review

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2026-02-09 22:55:21 +01:00
Roman Titov 5846411fb3 Add explicit SharedShardHolder type instead of LockedShardHolder type-alias (#7837) 2026-02-09 22:32:50 +01:00
Roman Titov 7715a54f5d Cleanup Dockerfile (#7814) 2026-02-09 22:27:38 +01:00
Roman Titov df20fdbb74 Replace lazy_static with std::sync::LazyLock (#7808) 2026-02-09 22:25:33 +01:00
Roman Titov bc6e25969e Fix CI 😬 (#7796) 2025-12-18 17:29:17 +01:00
Roman Titov 51bed3387b Qdrant Edge Python release CI workflow (#7562) 2025-12-18 17:29:17 +01:00
Roman Titov 962070c686 Implement __repr__ for Qdrant Edge types (Part 3) (#7744) 2025-12-18 17:28:49 +01:00
Roman Titov d0d1c3ded4 Implement __repr__ for Qdrant Edge types (Part 2) (#7742) 
* Implement `__repr__` for `PyJsonPath`

* Implement `__repr__` for `PyFilter`

* Implement `__repr__` for `PyQuery`

* Implement `__repr__` for `PyQueryRequest`

* Implement `__repr__` for `PySearchRequest`

* Implement `__repr__` for `PyFilter` using `pyclass_repr` attribute

* Implement `__repr__` for `PyQuery` using `pyclass_repr` attribute

* Implement `__repr__` for `PyQueryRequest` using `pyclass_repr` attribute

* Implement `__repr__` for `PySearchRequest` using `pyclass_repr` attribute
 2025-12-18 17:28:22 +01:00
Roman Titovandgenerall 0e7727c2fe Implement __repr__ for Qdrant Edge types (#7695) 
* Refactor `PyUpdateOperation` constructors

* Add default parameters to `PyVectorDataConfig::new`

* Add `Repr` trait and `WriteExt` helper

* Implement `__repr__` for config types

* fixup! Implement `__repr__` for config types

Use `Copy` instead of `Clone`

* fixup! Implement `__repr__` for config types

Add basic test

* Implement `__repr__` for `PyPointId`

* Implement `__repr__` for `PyVector`

* Implement `__repr__` for `PyVectorInternal`

* Implement `__repr__` for `PyPayload`

* Implement `__repr__` for `PyValue`

* Implement `__repr__` for `PyPoint`

* Implement `__repr__` for `PyPointVectors`

* Implement `__repr__` for `PyRecord`

* Move `PyScoredPoint` into a separate file

* Implement `__repr__` for `PyScoredPoint`

* Cleanup examples

* fixup! Implement `__repr__` for `PyScoredPoint`

* Move `PyOrderValue` into separate file

* Add `PyScoredPoint::order_value`

* Implement `pyclass_repr` proc-macro attribute

* Implement `__repr__` for config types using `pyclass_repr` attribute

* Implement `__repr__` for `PySparseVector` using `pyclass_repr` attribute

* Implement `__repr__` for `PyPoint` using `pyclass_repr` attribute

* Implement `__repr__` for `PyPointVectors` using `pyclass_repr` attribute

* Implement `__repr__` for `PyRecord` using `pyclass_repr` attribute

* Implement `__repr__` for `PyScoredPoint` using `pyclass_repr` attribute

* Minor fixes and cleanups

* fixup! Minor fixes and cleanups

* rollback copy for quantization config

* rollback copy for quantization config

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2025-12-18 17:28:22 +01:00
Roman Titov e33b4672e5 Fix test_collection_recovery test (#7700) 2025-12-18 17:26:37 +01:00
Roman Titov 38f5ac2bae Qdrant Edge Python bindings improvements (Part 2) (#7639) 2025-12-18 17:26:07 +01:00
Roman Titov d2834de0b5 Mark *new* local replicas as locally-disabled, when recovering Raft snapshot (#7684) 
* Remove faulty mark-as-dead condition when recovering Raft snapshot 😅

* Mark new local replicas as locally-disabled when recovering Raft snapshot

* Add test
 2025-12-04 08:44:02 +01:00
Roman Titov 106731bdf5 Make snapshot_manifest methods async (#7629) 
* Make `snapshot_manifest` methods `async`, so that they don't block async runtime

* Spawn `restore_shard_snapshot` task on `update` runtime instead of "current"
 2025-12-03 10:20:12 +01:00
Roman Titov 1de85b956e Qdrant Edge Python bindings improvements (#7561) 
* Use anonymous lifetime in `FromPyObject` implementations

* Use `PyResult` in `IntoPyObject` implementations

* Cleanup imports and derives

* Cleanup `filter` conversions

* Add `PointVectors` getters

* Move `config` module into sub-directory

* Split `config` into sub-modules

* Simplify enum bindings

* Add zero-cost conversions for `PyVectorDataConfig` and `PySparseVectorDataConfig`

* Add getters to config structures

* fixup! Add getters to config structures

More zero-cost conversions for `PyVector*DataConfig`

* Implement `PyHnswIndexConfig`

* Implement `PyQuantizationConfig`

* fixup! Simplify enum bindings

* fixup! Implement `PyHnswIndexConfig`

* fixup! Implement `PyHnswIndexConfig`

* fixup! Implement `PyHnswIndexConfig`

* Implement `PySparseVectorDataConfig`

* fixup! Implement `PySparseVectorDataConfig`

* fixup! Implement `PySparseVectorDataConfig`
 2025-12-03 10:18:53 +01:00
Roman Titov 8da26c0ded Minor tweaks to WAL compaction logs (#7580) 2025-11-25 11:16:01 +01:00
Roman TitovandTim Visée 7bdab0cf00 Allow payload operations on dummy shard (#7521) 
Co-authored-by: Tim Visée <tim+github@visee.me>
 2025-11-14 12:31:08 +01:00
Roman Titov 32b7fdfb7f Restrict /logger API (#7527) 2025-11-14 12:31:08 +01:00
Roman Titov 9d2ab514eb Qdrant Edge CI (#7525) 2025-11-14 12:31:07 +01:00
Roman Titov f5fcd9096f Allow partial recovery of dummy shard (#7512) 2025-11-14 12:31:05 +01:00
Roman Titovandgenerall e175c138dc Cleanup edge_py::types::filter module (#7490) 
Co-authored-by: generall <andrey@vasnetsov.com>
 2025-11-14 12:31:05 +01:00
Roman Titov 5b5b5b8625 Implement PyQuery conversions (#7481) 
* Rename `PyVectorType` into `PyNamedVector` 😠

* Move `PyQuery` into `types::query`

* Implement `PyQuery` conversions

* Use `bytemuck` to safely transmute between `edge-py` wrapper types (#7488)
 2025-11-14 12:31:04 +01:00
Roman Titovandtimvisee 59b6c5db77 Fix clippy 🙄 (#7486) 
Co-authored-by: timvisee <tim@visee.me>
 2025-11-14 12:31:03 +01:00
Roman Titov 233c1a0ccd Improvements to Qdrant Edge Python bindings (#7473) 
* Cleanup `examples/qdrant-edge.py`

* Cleanup `edge_py::types::vector` module

* Cleanup `edge_py::types::filter::condition` module

* Extend `PyWithPayload` type

* Use `from` instead of `into` for conversions

* Assert variants for enum conversions
 2025-11-14 12:31:03 +01:00
Roman Titov f010382630 Implement edge_py::Shard::query (#7465) 2025-11-14 12:30:14 +01:00
Roman Titovandgenerall d977f8e2ce Implement edge::Shard::query (#7440) 
Co-authored-by: generall <andrey@vasnetsov.com>
 2025-11-14 12:30:12 +01:00
Roman Titov 058222c3d3 Cleanups for shard crate and related stuff (#7445) 2025-11-14 12:30:11 +01:00
Roman Titov 67814b39c5 Move mmr into shard crate (#7429) 2025-11-14 12:30:11 +01:00
Roman Titov 329b223326 Implement edge::Shard::query_scroll (#7426) 2025-11-14 12:29:35 +01:00
Roman Titov 6d5ed0a8f2 Move PlannedQuery and QueryScrollRequestInternal into shard crate (#7417) 2025-11-14 12:29:33 +01:00
Roman Titov c7c8c606bf Move ShardQueryRequest into shard crate (#7410) 2025-11-14 12:29:32 +01:00
Roman Titov af6f8b4826 Move FormulaInternal into shard crate (#7409) 2025-11-14 12:29:32 +01:00
Roman Titov 8ee2d6e61c More type conversion for Qdrant on Edge Python bindings (#7372) 
* Cleanup `edge::Shard::retrieve`

* Cleanup Python errors

* Cleanup `PointId` conversions

* Rename `ids` into `point_ids` in `PyShard::retrieve`

* WIP: Prototype native conversions for more types
 2025-11-14 12:28:14 +01:00
Roman TitovandAndrey Vasnetsov 144dc3811e Type conversions into native Python types for Qdrant on Edge (#7367) 
* Cleanup README.md

* Rename `PyResult` into `Result`

* Rename `interface` module into `types` and simplify imports

`wildcard::goes::brrr::*`

* Add `PointId` conversion

* Add `Payload` conversion

* fixup! Add `Payload` conversion

Avoid cloning `Payload` for output conversion

* fixup! Add `PointId` conversion

Fix example

* review fixes #1: fix uuid parsing, avoid .into, use custom types

* review fixes #2: rewrite point id to avoid chain if

* review fixes #3: allow ot use uuid + remove comment

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2025-11-14 12:28:14 +01:00
Roman Titov 87289eb507 WIP: Extend Qdrant Edge Python bindings (#7343) 2025-11-14 12:27:28 +01:00
Roman Titovandgenerall 7977dbf9e7 Python bindings for Qdrant on Edge (#7312) 
* Qdrant Edge Python binding 🐍⛓️‍💥

* fixup! Qdrant Edge Python binding 🐍⛓️‍💥

Fix example + cleanup

* review: move QueryEnum + mark search as deprecated

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2025-11-14 12:27:28 +01:00
Roman Titov eee6719f20 Cleanup fs-err error messages in Edge (#7325) 2025-11-14 12:26:52 +01:00
Roman Titov 81834ec8a0 Qdrant on Edge prototype (#7170) 2025-11-14 12:26:52 +01:00
Roman Titov ad6a1ec220 Fix deadlock during streaming shard snapshot (#7241) 
* Add `test_shard_snapshot_deadlock` integration test

* Fix deadlock during streaming shard snapshot
 2025-09-29 11:29:22 +02:00
Roman Titov 4dd9d9bf4d Use Path instead of str in ConsensusWal::new (#7205) 2025-09-29 11:26:37 +02:00
Roman Titov e41a480c14 Move CoreSearchRequest and QueryEnum from collection into shard crate (#7169) 2025-09-29 11:25:30 +02:00
Roman Titov 2d33b13429 Move segments_updater from collection into shard crate (#7163) 2025-09-29 11:24:57 +02:00
Roman Titov 1192d4ce42 Make update_worker_fn a blocking task, instead of async (#7015) 2025-09-29 11:24:40 +02:00
Roman Titov 2e70b4c5f1 Fixes to UpdateHandler::flush_worker (#7151) 
- rename `wal_flash_job` to `wal_flush_job`
- correctly handle/report WAL flush errors
 2025-09-29 11:24:40 +02:00
Roman Titov 2f315b9b8e Move SerdeWal from collection into shard crate (#7146) 2025-08-26 16:40:11 +02:00
Roman Titov de07bf63e1 Move SegmentHolder into shard crate (#7115) 2025-08-26 13:18:36 +02:00
Roman Titov 804c7568e0 Move UpdateTracker from SegmentHolder into UpdateHandler/CollectionUpdater (#7112) 2025-08-26 13:18:16 +02:00
Roman Titov d4a46aab17 Move PayloadIndexSchema into shard crate (#7108) 2025-08-26 13:17:59 +02:00
Roman Titov c3e76dd04b Move SaveOnDisk into common crate (#7105) 2025-08-26 13:17:55 +02:00
Roman Titov e1d79f186e Refactor ShardHolder to use SegmentConfig instead of CollectionParams (#7098) 2025-08-26 13:17:46 +02:00
Roman Titov bea8551952 Remove async from SegmentHolder::deduplicate_points (#7094) 2025-08-26 13:17:45 +02:00
Roman Titov d4607bf3ac Move ProxySegment from collection into shard crate (#7093) 2025-08-26 13:17:20 +02:00
Roman Titov 258f9553f1 Cleanup collection::operations and shard::operations modules (#7064) 2025-08-26 13:16:39 +02:00
Roman Titov 98a5c79314 Extract CollectionUpdateOperations into separate shard crate (#7054) 2025-08-26 13:16:39 +02:00
Roman Titov f03aa08259 Move SegmentHolder::scroll_read_lock to LocalShard::scroll_read_lock (#7053) 2025-08-26 13:16:38 +02:00
Roman Titov 990eee11b4 Refactor Collection::restore_shard_snapshot (#6988) 2025-08-11 13:17:22 +02:00
Roman Titov 31738d3b85 Track payload index schema version (#6819) 2025-08-11 13:17:19 +02:00
Roman Titov 2938660e92 Fix test_create_shard_key_read_availability flakiness (#6902) 2025-08-11 13:17:11 +02:00
Roman Titov 343628b87c Cleanup partial snapshot stuff (#6925) 2025-08-11 13:16:07 +02:00
Roman Titov bf319a0b40 Avoid creating partial snapshot if snapshot manifest contains no changes (#6924) 2025-08-11 13:15:39 +02:00
Roman Titov fdc5366884 Fix DateTimeWrapper parsing (#6952) 2025-08-11 13:15:27 +02:00
Roman Titov e6bdcb2520 Update collection-level payload index schema after recovering partial snapshot (#6886) 2025-08-11 13:15:23 +02:00
Roman Titov 262a31efc1 Avoid blocking async runtime when loading local shard (#6862) 2025-07-17 13:58:13 +02:00
Roman Titovgeneralltimvisee
 		eceefa780e Wait until replicas are activated, when creating custom shards (#6817) 
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
 2025-07-17 13:49:48 +02:00
Roman Titov b8ed6b8807 Upgrade indicatif to 0.18.0 (#6808) 2025-07-17 13:47:25 +02:00
Roman Titov bda74ceae2 Create custom shards in Initializing state instead of Active (#6778) 2025-07-17 13:39:59 +02:00
Roman TitovandTim Visée 16368853da Improve read request handling during partial-snapshot recovery (#6725) 
* Fix read request handling during partial snapshot recovery

* fixup! Fix read request handling during partial snapshot recovery

Just use a second lock 😔

* fixup! Fix read request handling during partial snapshot recovery

Remove accidental `\`

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
 2025-07-17 13:30:27 +02:00
Roman Titov 082f101988 Track version of mutable files in vector and payload storage (#6652) 2025-07-17 13:14:45 +02:00
Roman Titov 422bce5ef0 Implement immutable_files method for segment sub-structures (#6637) 2025-07-17 13:10:08 +02:00
Roman Titov 7497d01bf1 Tweak partial snapshot recover and recover_from response (#6610) 2025-07-17 13:07:13 +02:00
Roman Titov 9dd3b6ff9e Simpler partial snapshots implementation based on immutable_files (#6599) 2025-07-17 13:04:15 +02:00
Roman Titov 07319a1ab1 Unify strict mode checks between REST and gRPC APIs (#6616) 2025-07-17 11:38:00 +02:00
Roman Titovandgenerall 2aafd0174d Add partial snapshot create requests count and recovery timestamp to telemetry (#6545) 
Co-authored-by: generall <andrey@vasnetsov.com>
 2025-05-22 23:27:14 +02:00
Roman Titovandgenerall 86d7c02528 Add partial snapshot recovery and search locks (#6497) 
Co-authored-by: generall <andrey@vasnetsov.com>
 2025-05-22 23:27:01 +02:00
Roman Titov 168ba380e2 Add recover_partial_snapshot_from endpoint (#6468) 2025-05-22 23:13:35 +02:00
Roman Titov c0bff6e5b1 Simplify tar_ext module (#5873) 
- Replace `WriteSeekBox*` with `SeekWrapper`
- Fix paste-bugs in tests
 2025-05-22 23:09:10 +02:00
Roman Titov 1bcf466b17 Make segment RocksDB optional, only initialize if required (#6441) 2025-05-22 22:53:41 +02:00
Roman Titov e98d74020c Generate less debug symbols in dev profile (#6443) 
* Generate less debug symbols in `dev` profile

* Retain `integration-tests-consensus` logs for 10 days (instead of 90)
 2025-05-22 22:52:35 +02:00
Roman Titov 5483581064 Add partial snapshot API endpoints (#6304) 2025-05-22 22:49:26 +02:00
Roman Titov 6e181db333 Optimize GitHub Actions workflows (#6362) 
* Add shared rust-cache key for integration test jobs

* Disable debug symbols

* Remove non-necessary setup steps...

...from `test-low-resources` and `test-snapshot-operations-s3-minio` jobs

* Always upload `test-consensus`/`integration-tests-consensus` logs...

...even when tests passed successfully

* Reduce poll interval for `wait_for` checks in integration tests
 2025-04-21 00:07:25 +02:00
Roman Titov 5d2ede9719 Rename some GHA jobs, so that default rust-cache keys are unique for all workflows/jobs (#6280) 2025-04-21 00:02:53 +02:00
Roman Titov 90340a098c Implement partial snapshot recovery (#6206) 
- Fix creating partial snapshot without RocksDB backup
- Fix `segment_manifest.json` file placement
- Implement partial snapshot recovery
- Allow restoring partial snapshot for all shard types
- Validate segment manifests when recovering partial snapshot
- Log error, if streaming shard/partial snapshot failed
- Contextualize errors in `snapshot_files` function
- Tweak log messages in `Segment::take_snapshot` method
- Remove unnecessary lifetimes from `SegmentHolder` iter methods
 2025-04-20 23:46:47 +02:00
Roman Titov f9cc4e06f4 Merge take_snapshot and take_partial_snapshot methods (#6232) 2025-03-31 16:20:45 +02:00
Roman Titovandtimvisee c4ec15db9a Simplify ShardKeyMapping implementation (#6220) 
* Simplify `ShardKeyMapping` impl

* Automatically switch to new mappings format, if `Number` shard key is used

* Apply TODOs

* Add comment describing the issue, leave instructions for migration

* Cleanup

Remove unnecessary assignment

* Cleanup

Remove misleading comment

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-03-31 16:19:06 +02:00
Roman Titovandtimvisee c5bbcf8a9d Extend partial segment snapshot to support ProxySegments (#5978) 
* Add `SegmentManifest::segment_id` field

* Tweak `SegmentEntry::take_partial_snapshot` to support `ProxySegment`

* Track segments added into partial snapshot archive

* Debug assert that segment IDs are a valid UUID

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-03-21 12:00:43 +01:00
Roman Titov 03dc95c74c Check use_mutable_id_tracker_without_rocksdb feature flag when building new segment (#6211) 2025-03-21 11:46:51 +01:00
Roman Titov e05e3b775e Include ClusterMetadata into Raft snapshot 😬 (#6014) 2025-03-21 11:36:48 +01:00
Roman Titov 8f4d4beff3 Add common::update::update function (#5977) 2025-03-21 11:36:43 +01:00
Roman Titovandgenerall 4a26134bec Prototype partial segment snapshots and SegmentManifest format (#5928) 
Co-authored-by: generall <andrey@vasnetsov.com>
 2025-03-21 11:36:39 +01:00
Roman Titov 01acabc0c6 Move UpdateParams structure into common::update... (#5974) 
...to easier propagate `wait` and `ordering` API parameters
 2025-02-17 10:20:47 +01:00
Roman Titovandtimvisee 363c5412a0 Add InternalUpdateParams structure (#5968) 
* Add `InternalUpdateParams` structure...

...to easier propagate `shard_id` and `clock_tag` internal API parameters

* fixup! Add `InternalUpdateParams` structure...

Use `InternalUpdateParams::default` instead of `Default::default`

* Remove now obsolete delinters, expect them where still applicable

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-02-17 10:20:37 +01:00
Roman Titov fa13607f76 Tiny refactoring to reduce some boilerplate in update API (#5953) 
* Rename `UpdateParam` into `UpdateParams` and remove `Option`s from fields

* Use `common::inference::extract_token` in internal gRPC API

* Destructure `UpdateVectorsInternal`/`DeleteVectorsInternal` requests in internal gRPC API

* Add `extract_internal_request` helper for internal gRPC API

* fixup! Add `extract_internal_request` helper for internal gRPC API

Use `extract_internal_request` helper in `sync` API

* fixup! Rename `UpdateParam` into `UpdateParams` and remove `Option`s from fields

Remove `JsonSchema` derive from `UpdateParams`
 2025-02-11 14:42:04 +01:00
Roman Titov a5ce94248d Compact Raft WAL until last *applied* entry, not until first *unapplied* (#5900) 2025-02-11 14:36:36 +01:00
Roman Titov 8afe42da3e Optimize consensus loop (#5728) 2025-02-11 14:35:51 +01:00
Roman Titov 0d16911607 Split tonic::api::points_common module (#5865) 2025-01-28 10:44:07 +01:00
Roman Titov 8868ecc9b2 Split qdrant::common::points module (#5862) 2025-01-27 18:26:02 +01:00