upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-09-28 00:47:32 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,633 Commits 437 Branches 119 Tags
fix-api-spec
Commit Graph
999 Commits
Author SHA1 Message Date
Tim ViséeLuis Cossíogenerall
 		4827a362e8 Update collection: HNSW config (#2083) 
* Add field to REST collection update to change HNSW config

* Add field to gRPC collection update to change HNSW config

* Update OpenAPI specification

* Update gRPC docs

* Improve optimizer filtering for excluded segment IDs

* Add config mismatch optimizer detecting basic HNSW mismatches

* Trigger optimizers when changing collection HNSW config

* Make config mismatch optimizer aware of vector specific HNSW configs

* Extract triggering optimizers after collection update into function

* Add config mismatch optimizer test for changing HNSW config

* Simplify config mismatch optimizer test

* Add config mismatch optimizer test for vector specific HNSW change

* Reformat

* Update collection HNSW params in integration test

* Validate HNSW config in collection update gRPC endpoint

* Improve description of update collection request

* Do not require to rebuild HNSW when on_disk flag changes

Ref: https://github.com/qdrant/qdrant/pull/2083#discussion_r1231002072

* Do rebuild segment with on_disk change

* Trigger optimizers in parallel

* Recreate optimizers only once on collection update

* Reformat

* Fix incorrect usage of self

* Fix deadlock in collection state config update

* Also rebuild index on full scan threshold change

* decompose worst_segment condition check

* review refactor

* Update lib/storage/src/content_manager/collection_meta_ops.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

* Update lib/api/src/grpc/proto/collections.proto

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

* upd grpc docs

* upd grpc docs

* update openapi

---------

Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
 2023-07-18 14:32:13 +02:00
dependabot[bot] 3154a33ff7 Bump uuid from 1.4.0 to 1.4.1 (#2281) 
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.4.0 to 1.4.1.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/1.4.0...1.4.1)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-18 08:53:14 +02:00
dependabot[bot] ed76952b5d Bump anyhow from 1.0.71 to 1.0.72 (#2276) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.71...1.0.72)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-18 00:26:52 -04:00
dependabot[bot] 617456e76e Bump hashring from 0.3.0 to 0.3.1 (#2280) 
Bumps [hashring](https://github.com/jeromefroe/hashring-rs) from 0.3.0 to 0.3.1.
- [Changelog](https://github.com/jeromefroe/hashring-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeromefroe/hashring-rs/compare/0.3.0...0.3.1)

---
updated-dependencies:
- dependency-name: hashring
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-18 00:26:10 -04:00
dependabot[bot] 46e0e50747 Bump semver from 1.0.17 to 1.0.18 (#2278) 
Bumps [semver](https://github.com/dtolnay/semver) from 1.0.17 to 1.0.18.
- [Release notes](https://github.com/dtolnay/semver/releases)
- [Commits](https://github.com/dtolnay/semver/compare/1.0.17...1.0.18)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-18 00:23:52 -04:00
Arnaud Gourlay 855cb32270 Try to fix flaky test_task_stop (#2271) 
* Try to fix flaky test

* less strict
 2023-07-18 01:25:48 +02:00
Andrey Vasnetsov dda8330e15 detect file extension better (#2272) 
* detect file extension better

* fix clippy

* refactor + test
 2023-07-17 17:10:52 +02:00
Zein Wen 79475ea972 Add geo_polygon filter to proto interface, complete conversion fn, and add an integration test (#2188) 2023-07-17 12:36:50 +02:00
dependabot[bot] 054ccc10a7 Bump tar from 0.4.38 to 0.4.39 (#2259) 
Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.38 to 0.4.39.
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.38...0.4.39)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 10:59:34 +02:00
dependabot[bot] c44b05b39b Bump geo from 0.25.1 to 0.26.0 (#2263) 
Bumps [geo](https://github.com/georust/geo) from 0.25.1 to 0.26.0.
- [Changelog](https://github.com/georust/geo/blob/main/CHANGES.md)
- [Commits](https://github.com/georust/geo/commits)

---
updated-dependencies:
- dependency-name: geo
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 10:14:09 +02:00
Andrey Vasnetsov f4943a8267 try to fix flacky macos test (#2261) 
* try to fix flacky macos test

* try more threads

* fix conditions for stopable task

* fix test
 2023-07-14 14:24:19 +02:00
Tim Visée 69df274d22 Improve flaky stopable task test (#2257) 
* Use duration constant

* Loosen requirements for stoppable task

* Add test for 64 concurrent stoppable tasks

* Reformat

* Improve stopping many stoppable tasks a bit

* Change stoppable task test requirements
 2023-07-13 21:55:27 +02:00
二手掉包工程师 a0f5f895f7 Align the fmt command with .rusty-hook.toml (#2241) 
* Align the fmt command with `.rusty-hook.toml`

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

* Address #[warn(clippy::useless_conversion)]

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

* Address #[warn(clippy::drain_collect)]

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

---------

Signed-off-by: hi-rustin <rustin.liu@gmail.com>
 2023-07-13 10:52:01 +02:00
Yaroslav Halchenko 6624c95afd codespell: workflow, config, typos fixed (#2248) 
* Add github action to codespell master on push and PRs

* Add rudimentary codespell config

* some skips

* fix some ambigous typos

* [DATALAD RUNCMD] run codespell throughout

=== Do not change lines below ===
{
 "chain": [],
 "cmd": "codespell -w",
 "exit": 0,
 "extra_inputs": [],
 "inputs": [],
 "outputs": [],
 "pwd": "."
}
^^^ Do not change lines above ^^^

* Add dev branch as target for the workflow
 2023-07-12 16:30:42 +02:00
Tim Visée a0a3620942 Fix stack overflow in segment spinlock (#2250) 2023-07-12 15:36:29 +02:00
Andrey Vasnetsov 38d35361aa use search runtime to do reads in shard transfer (#2252) 2023-07-12 15:36:18 +02:00
Andrey Vasnetsov d7209fd1e8 limit transfers scheduling on sync-local-state (#2249) 2023-07-12 10:51:50 +02:00
Taras Tsugrii 3e252492c2 [bench] Reserve vector storage to avoid reallocs. (#2245) 
This is particularly useful for microbenchmarks since
1) this cost shouldn't affect benchmark results
2) memory allocations are the largest source of measurement noise
 2023-07-11 09:18:38 +02:00
Arnaud Gourlay 8fd32e8d28 SegmentSearcher search-in-segment is not async (#2211) 
* SegmentSearcher search_in_segment is not async

* keep it iterator style
 2023-07-10 15:00:04 +02:00
Andrey Vasnetsov b0aae6d35a upd wal deps (#2237) 2023-07-10 14:36:35 +02:00
Arnaud Gourlay 600e16055a Non blocking Scroll and parallel segments runs (#2221) 
* Non blocking Scroll and parallel segments runs

* set max number of working threads used by spawn_blocking
 2023-07-10 13:41:17 +02:00
Arnaud Gourlay d984d751d7 Fix set payload index to handle type change (#2235) 2023-07-10 11:24:14 +02:00
Roman TitovTim ViséeAndrey Vasnetsov
 		a0ed771409 Fix optimizers stop lock (#2222) 
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-07-07 16:06:26 +02:00
Arnaud Gourlay 17b7731a89 Add message to unimplemented! (#2230) 
* Add message to unimplemented!

* yield str id as well

* modern syntax
 2023-07-07 12:09:50 +02:00
Luis Cossío a77d6775b2 remove unused Serialize derives in build.rs (#2225) 2023-07-07 10:49:41 +02:00
Luis Cossío e5510e3d30 remove unused "no_bracket_syntax" error code (#2223) 2023-07-06 15:59:14 -04:00
dependabot[bot] 1f31b31ac1 Bump async-trait from 0.1.70 to 0.1.71 (#2217) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.70 to 0.1.71.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.70...0.1.71)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-06 08:39:29 +02:00
Luis Cossío 33afc616ce Expose & benchmark binary index (#2174) 
* expose boolean index

* benchmark against keyword and no index

* add integration tests

* rename benchmark groups
 2023-07-05 09:29:34 -04:00
Arnaud Gourlay 2164850c98 SegmentSearcher retrieve is not async (#2210) 2023-07-05 15:24:10 +02:00
dependabot[bot] 65d10e3fab Bump async-trait from 0.1.69 to 0.1.70 (#2205) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.69 to 0.1.70.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.69...0.1.70)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-05 08:43:45 +02:00
Ivan Pleshkovandgenerall 12423f910d Add missed vector preprocess (#2203) 
* test missed preprocess after segment update

* missed preprocess

* remove preprocess_named_vectors fn

* are you happy clippy

* fix integration tests

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2023-07-05 00:30:15 +02:00
dependabot[bot] b310eae791 Bump pprof from 0.11.1 to 0.12.0 (#2197) 
Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.11.1 to 0.12.0.
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-04 08:51:31 +02:00
dependabot[bot] 4fda29a5fc Bump async-trait from 0.1.68 to 0.1.69 (#2196) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.68 to 0.1.69.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.68...0.1.69)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-04 08:22:27 +02:00
Luis Cossío 4527f1db81 Small refactor: remove duplicated indexed_points() function (#2103) 
* remove duplicated `indexed_points()` function

* update for binary index
 2023-07-03 13:25:54 -04:00
Luis CossíotimviseeAndrey Vasnetsov
 		cacc942b36 Add binary index (#2107) 
* nitpick: use CardinalityEstimation::exact() instead of explicit struct

* wip: implement binary index

* persist binary index

* add tests, refactor binary memory

* use bitflags for BinaryItem, use early return from @timvisee 's review

* fix iterator's end +1

* fix indexed_count, keep track of falses and trues counts

* Fix true/false counts if flag for point was set already

* Remove match in filter structure

* - rename `*_count()`
- fix set_or_insert to actually set
- add more tests

* review suggestions for binary index (#2186)

* small taste refactors

* move memory into its own module

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-07-03 12:49:53 -04:00
Zein Wen a009257884 Add fn polygon_hashes and its helper functions (#2162) 
This is one of a series of commits for the new feature Geo Filter by Polygon(#795) that add function polygon_hashes that retrieves as-high-as-possible with maximum of max_regions number of geo-hash guaranteed to contain the whole polygon.

It also includes two helper functions: 1) check_polygon_intersection and 2) minimum_bounding_rectangle_for_polygon.

Test cases validate the functionality for different scenarios.
 2023-07-03 18:14:49 +02:00
Arnaud Gourlay 27d4705411 Revert "TableOfContent now accepts Handle, not Runtime (#2096)" (#2191) 
This reverts commit fd87ca2c6d.
 2023-07-03 14:05:47 +02:00
Andrey Vasnetsov 8e1f228349 handle inconsistent vector storage in sync operation (#2185) 2023-07-03 12:38:18 +02:00
Andrey Vasnetsov d47e2d7cb7 clear collection directory before creating a new collection (#2189) 2023-07-03 12:35:41 +02:00
dependabot[bot] 91112ceaa4 Bump num_cpus from 1.15.0 to 1.16.0 (#2182) 
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.15.0 to 1.16.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.15.0...v1.16.0)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-03 09:52:50 +02:00
dependabot[bot] 12e9271263 Bump num-derive from 0.3.3 to 0.4.0 (#2179) 
Bumps [num-derive](https://github.com/rust-num/num-derive) from 0.3.3 to 0.4.0.
- [Changelog](https://github.com/rust-num/num-derive/blob/master/RELEASES.md)
- [Commits](https://github.com/rust-num/num-derive/compare/num-derive-0.3.3...num-derive-0.4.0)

---
updated-dependencies:
- dependency-name: num-derive
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-03 09:51:29 +02:00
dependabot[bot] 5fdc81cfe9 Bump tokio from 1.29.0 to 1.29.1 (#2177) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.29.0 to 1.29.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.29.0...tokio-1.29.1)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-03 09:50:19 +02:00
Denis Bazhenovandtimvisee fd87ca2c6d TableOfContent now accepts Handle, not Runtime (#2096) 
* TableOfContent now accepts Handle, not Runtime

Runtime is left within main() and dropped there to shutdown the system

* Correct termination of consensus loop

Now consensus loop is joined by main thread and
correctly terminated between iterations of the loop

* Add clippy #[allow]

* Move complicated type into definition

* Remove duplicate on_consensus_stopped call

* wait_unwrap() is now time bounded

Drop is happening only if Arc<TableOfContent> is droped by all threads
in 30 seconds to limit shutdown time

---------

Co-authored-by: timvisee <tim@visee.me>
 2023-07-02 15:28:47 +02:00
Luis Cossío dcbddebd0b fix condition to select duration measurer (#2183) 2023-07-02 13:10:32 +02:00
Luis Cossío 047774a7b4 Small group_by improvements (#2074) 
* improve group_by
- improve termination condition check
- add all_ids to aggregator to avoid flat_map
 2023-06-30 09:42:04 -04:00
Andrey Vasnetsov 4cd11e2134 add create-collection lock + remove shard versioning function (#2161) 
* add create-collection lock + remove shard versioning function

* review fix
 2023-06-30 13:53:54 +02:00
Tim Visée 9a4acf8c29 Use configured temporary directory in actix multipart file uploads (#2156) 2023-06-30 11:49:11 +02:00
Andrey Vasnetsovandtimvisee 24b887ca89 Fix double-read deadlock in the get-collection (#2171) 
Co-authored-by: timvisee <tim@visee.me>
 2023-06-29 14:13:11 +02:00
dependabot[bot] 47eedc3aaa Bump tokio from 1.28.2 to 1.29.0 (#2170) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.28.2 to 1.29.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.2...tokio-1.29.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 22:39:26 +02:00
Andrey Vasnetsov 247ea20a4e split search_existing_point_on_level and regular search_on_level functions (#2167) 2023-06-28 18:11:21 +02:00