upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-10-03 03:17:43 -05:00
Code Issues Packages Projects Releases Wiki Activity
2,826 Commits 457 Branches 119 Tags
83b93b951002a75845bc00ec00b50dafae86bfc8
Commit Graph
2826 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrey Vasnetsov 83b93b9510 use new openapi documentation page (#4444) 2024-06-11 15:21:57 +02:00
Arnaud Gourlay 3782792a5d Shortcut HNSW build on empty storages (#4441) 2024-06-11 14:23:02 +02:00
Tim Visée 0223a0b7ca When selecting a segment for writing, select the smallest one (#4440) 
* Preallocate list of entires to prevent some unnecessary reallocations

* Implement Copy for OptimizerThresholds

* Add shard holder function get smallest segment

* Take the smallest segment in the segments updater

* Add test to assert inserting into smallest segment

* Fix compilation warnings
 2024-06-11 12:59:05 +02:00
Ivan Pleshkov 6b35507770 Hnsw update entry point after linking (#4442) 
* hnsw update entry point after linking

* are you happy fmt
 2024-06-11 12:48:15 +02:00
dependabot[bot] 65fcd66123 Bump actix-files from 0.6.5 to 0.6.6 (#4437) 
Bumps [actix-files](https://github.com/actix/actix-web) from 0.6.5 to 0.6.6.
- [Release notes](https://github.com/actix/actix-web/releases)
- [Changelog](https://github.com/actix/actix-web/blob/v0.6.6/CHANGES.md)
- [Commits](https://github.com/actix/actix-web/compare/v0.6.5...v0.6.6)

---
updated-dependencies:
- dependency-name: actix-files
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-11 10:59:14 +02:00
Andrey Vasnetsov 301a105deb better default max segment size estimation (#4428) 2024-06-11 10:58:58 +02:00
dependabot[bot] 1da61e298a Bump tar from 0.4.40 to 0.4.41 (#4434) 
Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.40 to 0.4.41.
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.40...0.4.41)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-11 09:27:37 +02:00
dependabot[bot] 6ae01ed860 Bump actix-multipart from 0.6.1 to 0.6.2 (#4431) 
Bumps [actix-multipart](https://github.com/actix/actix-web) from 0.6.1 to 0.6.2.
- [Release notes](https://github.com/actix/actix-web/releases)
- [Changelog](https://github.com/actix/actix-web/blob/v0.6.2/CHANGES.md)
- [Commits](https://github.com/actix/actix-web/compare/v0.6.1...v0.6.2)

---
updated-dependencies:
- dependency-name: actix-multipart
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-11 09:27:28 +02:00
dependabot[bot] cc39a1aa26 Bump sealed_test from 1.0.0 to 1.1.0 (#4435) 
Bumps [sealed_test](https://github.com/oknozor/sealed-test) from 1.0.0 to 1.1.0.
- [Release notes](https://github.com/oknozor/sealed-test/releases)
- [Commits](https://github.com/oknozor/sealed-test/commits/1.1.0)

---
updated-dependencies:
- dependency-name: sealed_test
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-11 09:26:48 +02:00
dependabot[bot] 645ca1ef32 Bump actix-web from 4.6.0 to 4.7.0 (#4432) 
Bumps [actix-web](https://github.com/actix/actix-web) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/actix/actix-web/releases)
- [Changelog](https://github.com/actix/actix-web/blob/master/CHANGES.md)
- [Commits](https://github.com/actix/actix-web/compare/web-v4.6.0...web-v4.7.0)

---
updated-dependencies:
- dependency-name: actix-web
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-11 09:26:28 +02:00
dependabot[bot] f4198b1082 Bump url from 2.5.0 to 2.5.1 (#4429) 
Bumps [url](https://github.com/servo/rust-url) from 2.5.0 to 2.5.1.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.5.0...v2.5.1)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-11 09:25:57 +02:00
dependabot[bot] e83324707e Bump cc from 1.0.98 to 1.0.99 (#4436) 
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.0.98 to 1.0.99.
- [Release notes](https://github.com/rust-lang/cc-rs/releases)
- [Commits](https://github.com/rust-lang/cc-rs/compare/1.0.98...1.0.99)

---
updated-dependencies:
- dependency-name: cc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-11 09:12:51 +02:00
Luis Cossío f62119c670 universal-query: Basic integration tests (#4409) 
Adds basic comparison to current existing endpoints
 2024-06-10 19:50:33 -04:00
Luis Cossío 2f060d568d universal-query: Fix lost offset in planned query conversion (#4408) 
I realized we were sometimes ignoring `offset` when crafting the `PlannedQuery`, specifically in the case of root-level scroll-like requests. This PR fixes this and propagates the root level offset into all the limits in prefetches
 2024-06-10 19:29:35 -04:00
Luis Cossío 883cbdc7aa universal-query: Reject having prefetches without a rescoring query (#4407) 
Defining `prefetch`es without defining a rescoring `query` does not make much sense, and makes handling ambiguous. So we rather reject this combination.
 2024-06-10 19:07:37 -04:00
Luis Cossío 8dc1eb235b universal-query: some interface renaming/edits (#4406) 
in grpc:
- restructure `ContextInput`, so that it becomes easy to handle both pairs and positive/negative lists in the future

in rest: 
- restructure context queries so that they don't repeat the label in order to be used . E.g. turn `{ query: { context: { context: [ ... ] } } }` into `{ query: { context: [ ... ] } }` 
- fixes fusion so that `{query: { fusion: "rrf" } }` actually works

both:
- renames `RecommendInput` positives and negatives to singular (same as in reco api)
 2024-06-10 18:47:40 -04:00
Luis Cossío 8b884c646a universal-query: Exclude referenced ids (#4405) 
In the Query API, all vector inputs can be ids, for convenience, but we were missing to exclude those ids from the response.
 2024-06-10 18:27:29 -04:00
Luis Cossío fbb4c2740d universal-query: Fix local shard ordering (#4404) 
- adds an `order` field in `planned_query::ResultsMerge` to indicate the desired direction of merging, and merges accordingly
  - converting from `ShardQueryRequest` to `PlannedQuery` now needs the collection params to determine the required order depending on the vector config
- handles previously unhandled `filter` and `score_threshold` in `LocalShard::merge_prefetches()`, either by passing it to the core_search/scroll rescoring or during regular merging
 2024-06-10 18:07:13 -04:00
dependabot[bot] 90a6b8acde Bump clap from 4.5.4 to 4.5.7 (#4433) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.5.4 to 4.5.7.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.4...v4.5.7)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-10 17:04:53 -04:00
Tim Viséeandgenerall 84a2dd8e95 Ensure we have any segment within capacity, otherwise add new one (#4416) 
* Extract logic for creating thresholds config

* Put collection params and threshold config in update handler

* Add function to add a new appendable segment if all are over capacity

* Make new method static, call it before each optimization loop

* Update error message formatting

* Use exact point count in replication consensus test

* Add a test to assert segment creation when all are over capacity

* Suffix optimizer thresholds with _kb to clarify unit

* Move segment capacity check logic, run if optimizers are disabled

* fix: add -> mul

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-06-10 18:45:53 +02:00
Arnaud Gourlay c53a5470a9 Fix multivector simple storage metadata (#4424) 
* Fix multivector simple storage metadata

* code review naming
 2024-06-10 18:13:13 +02:00
Luis Cossío 6d0d542356 universal-query: Expose REST endpoint (#4380) 
Exposes `POST collections/{collection_name}/points/query` for single queries
 2024-06-10 11:55:46 -04:00
Luis Cossío d35319c663 universal-query: validate REST query request (#4388) 
Implements `Validate` for REST-facing `QueryRequest`
 2024-06-10 11:27:28 -04:00
Arnaud Gourlay 17aa55dded Remove unused chaos-test feature (#4422) 2024-06-10 17:12:32 +02:00
xzfc d2ef556237 Make sparse vectors generic (#4364) 
* Make sparse vector generic

* Generic search_context tests
 2024-06-10 15:04:32 +00:00
xzfc c36b4793e0 Allow failures on Windows CI (#4421) 2024-06-10 15:49:02 +02:00
Roman Titov 535f4e206c Dockerfile updates (#4414) 2024-06-07 23:24:17 +02:00
Luis Cossío ff77ed14b2 universal-query: finish implementation of query() fn in collection (#4362) 
- reuse merging from shards in both `query()` and `query_internal()`
- check filters for issues if it was a slow request
 2024-06-07 15:49:13 -04:00
Luis Cossío 9ec1266dca universal-query: convert CollectionQueryRequest into ShardQueryRequest (resolve ids into vectors) (#4360) 
- make `VectorQuery` generic
- retrieve ids from `CollectionQueryRequest` and convert into vectors
- implement `RetrieveRequest` trait for it
- calculate avg vector with logic from `recommendations.rs`
- convert `CollectionQueryRequest` into `ShardQueryRequest`
- begin implementing `query()` fn in collection
 2024-06-07 12:58:47 -04:00
Arnaud Gourlay bd3a107ae0 Enable RocksDB WAL for data consistency on crash (#4411) 2024-06-07 15:22:53 +02:00
Tim Visée b5a691eba4 Replicate new shard to match replication factor in resharding driver (#4381) 
* Add replication stage for resharding

* Use consistent naming

* Simplify awaiting shard transfer success, add periodic sanity check

* Fix typos
 2024-06-07 12:16:35 +02:00
Tim Visée 60551bf10c Fix deadlock caused by concurrent snapshot and optimization (#4402) 
* Rename segment addition functions, clarify this generates a new ID

* Don't randomize segment IDs, auto increment to prevent duplicates

* Rename swap to swap_new

* On snapshot unproxy, put segments back with their original segment ID

* Add sanity check to optimizer unproxy, must swap same number of segments

* Clean up

* Extend snapshot test, assert we end up with the same segment IDs
 2024-06-06 20:11:00 +02:00
Arnaud GourlayandTim Visée 4db2b7a0de Add get optional vector for all storages (#4368) 
* Add get optional vector for dense simple storage

* more bound checks

* remove storage get_opt blanket implementation

* remove duplication

* remove duplication

* Update lib/segment/src/vector_storage/multi_dense/simple_multi_dense_vector_storage.rs

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
 2024-06-06 18:07:06 +02:00
Lokesh Kumar 50c7a7e380 doc: added link for opening new issue on GitHub (#4400) 2024-06-06 09:55:03 +02:00
Ivan Pleshkov 61cfb8bcb5 Test fix segment builder for sparse (#4397) 
* test fix segment builder for sparse

* are you happy fmt
 2024-06-05 13:52:08 +02:00
Andrey Vasnetsov 308f611fe0 initialize sparse vectors in the segment builder (#4396) 2024-06-05 12:01:20 +02:00
dependabot[bot]andIvan Pleshkov 95edc18c45 Bump quantization from 14f42f9 to 0caf67d (#4383) 
* Bump quantization from `14f42f9` to `0caf67d`

Bumps [quantization](https://github.com/qdrant/quantization) from `14f42f9` to `0caf67d`.
- [Commits](https://github.com/qdrant/quantization/compare/14f42f944358b4e95c482892dc4ead019809a448...0caf67d96f022a792bda2e41fa878ba1e113113f)

---
updated-dependencies:
- dependency-name: quantization
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix build

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
 2024-06-05 11:39:40 +02:00
Lokesh Kumar 5844d23dea doc: update links to qdrant integrations in README.md (#4387) 2024-06-04 11:05:44 -04:00
Tim Visée c50da3210e Fix incorrect log message, not strictly related to snapshot recovery (#4391) 2024-06-04 14:04:08 +02:00
Tim Visée 28e8cfd3b8 Persist state in resharding driver (#4379) 
* Allow SaveOnDisk to initialize with non-default state

* Persist resharding driver state
 2024-06-04 11:35:28 +02:00
Ivan Pleshkovandgenerall a629fc4d4c Fix hnsw full scan threshold (#4369) 
* fix hnsw full scan threshold

* add test

* are you happy clippy

* separate open_vector_storage

* remove public fields from builder

* wip: do not create segment in builder before build

* avoid arc in storage test and low-level loading functions

* WIP: remove internal segment from SegmentBuilder

* fmt

* finalize segment builder fixes

* Revert "are you happy clippy"

This reverts commit c04afa6989.

* Revert "add test"

This reverts commit 8e7ad6207e.

* Revert "fix hnsw full scan threshold"

This reverts commit 8904443fcb.

* remove _daatabse from builder

* fix optimizer test

* fix id tracker versions persistence

* do flush for segment components on build

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-06-04 11:16:11 +02:00
Tim Visée a71f522fb5 Migrate points to target shard in resharding driver (#4373) 
* Fix incorrect error message

* Simplify shard transfer retrieval function

* Add consensus dispatcher function add current peer ID

* Add simple resharding driver state

* Add consensus dispatcher function to start shard transfer

* Add basic implementation for driving resharding migration transfers

* Add migrate points documentation

* Use consistent naming

* Add logic to wait for a shard transfer to end, either finish or abort

* Add logic for waiting on shard transfer end, either finish or abort

* Start awaiting transfer end before we propose one

* Add channel to broadcast shard transfer state updates

* Do not keep shard holder read lock while waiting for shard transfer end

* On shard transfer abort through consensus snapshot, notify on channel

* If point migration transfer timed out, abort it to clean up

* Fix typo, update comments

* Mark TODOs for resharding
 2024-06-04 11:11:29 +02:00
dependabot[bot] 387c6265fd Bump serde from 1.0.202 to 1.0.203 (#4384) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.202 to 1.0.203.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.202...v1.0.203)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-04 09:42:24 +02:00
dependabot[bot] e76ba29bd6 Bump tokio from 1.37.0 to 1.38.0 (#4386) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.37.0 to 1.38.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.37.0...tokio-1.38.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-04 09:12:07 +02:00
dependabot[bot] d67baa4767 Bump rustls from 0.23.8 to 0.23.9 (#4382) 
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.8 to 0.23.9.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.23.8...v/0.23.9)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-04 08:47:05 +02:00
dependabot[bot] 39f41183d8 Bump rstest from 0.19.0 to 0.21.0 (#4385) 
Bumps [rstest](https://github.com/la10736/rstest) from 0.19.0 to 0.21.0.
- [Release notes](https://github.com/la10736/rstest/releases)
- [Changelog](https://github.com/la10736/rstest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/la10736/rstest/compare/v0.19.0...v0.21.0)

---
updated-dependencies:
- dependency-name: rstest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-03 18:52:12 -04:00
Arnaud Gourlay d766d2dddc Fix Versions persistence atomicity (#4376) 2024-06-01 20:29:18 +02:00
Tim Visée eb738d06cc Deduplicate collection ID in shard transfers/resharding (#4372) 
* Deduplicate collection ID in shard transfers, name is the same

* Deduplicate collection ID in resharding

* Use collection ID in consensus dispatcher
 2024-05-31 17:05:16 +02:00
Roman Titov abebd9dd03 Add ReshardingOperation::CommitHashRing consensus operation (#4344) 2024-05-31 17:05:07 +02:00
Arnaud Gourlay 603efeda05 Rocksdb remove should use the same options as put (#4370) 2024-05-31 15:45:55 +02:00