upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-06 18:10:58 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,465 Commits 346 Branches 118 Tags
94de0e2d2dcf997bf26bec34dad407d8de9a9abb
Commit Graph

1780 Commits

Author SHA1 Message Date
Arnaud Gourlay
464d9c1f52 Fix Clippy lints 1.73 (#2760) 2023-10-04 12:00:32 +02:00
Zein Wen
5c7b57c970 Extend GeoPolygon to support interiors (#2315) 
* Extend GeoPolygon to support interiors (#2315)

Per GeoJson, we should support polygon with exterior and interiors (holes on the surface) in Geo Filter by Polygon(#795). This commit extend current GeoPolygon filter to accept interiors. It includes:

1. changes to proto and internal GeoPolygon struct, and validation fn
2. add and refactor some tests
3. add integration test

* add gRPC geo_polygon validation

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2023-10-04 11:57:51 +02:00
Arnaud Gourlay
252c1026c6 Validate empty filter condition (#2745) 
* Validate empty filter condition

* Validate filter objects in other requests

* Propagate nested filter validation

* add gRPC validation

* make match exhaustive with explicit variants

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
 2023-10-04 08:37:32 +02:00
Ivan Pleshkov
0623196eff Mutable numeric index (#2653) 
* mutable numeric index

* mutable numeric index

* apply changes from dev

* fix build

* remove obsolete allow
 2023-10-03 11:48:42 +02:00
dependabot[bot]
fac083f715 Bump urllib3 from 2.0.3 to 2.0.6 in /tests/storage-compat/populate_db (#2757) 
* Bump urllib3 from 2.0.3 to 2.0.6 in /tests/storage-compat/populate_db

Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.3 to 2.0.6.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.0.3...2.0.6)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix spelling

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
 2023-10-03 09:16:25 +02:00
Ivan Pleshkov
05806c8bdb remove fake metric from benches (#2754) 2023-10-02 13:30:04 +02:00
Luis Cossío
a67168bb6f Include binary index on match value (#2744) 2023-10-02 09:56:44 +02:00
Arnaud Gourlay
2e4729a1f3 Promote operation error to dedicated file (#2736) 2023-09-29 16:23:24 +02:00
Andrey Vasnetsov
c3657fd9c7 better check for boundaries in range request (#2742) 2023-09-29 09:41:54 +02:00
Roman Titov
bad92e132d Fix ARM-specific import after moving ScoreType to common (#2735) 2023-09-28 14:11:59 +02:00
Arnaud Gourlay
5bb80fbc77 Move ScoredPointOffset into common (#2734) 2023-09-28 12:11:29 +02:00
Eugene Tolbakov
ef9e4e5971 Refactor: replace expect with ServiceError, remove repetive code (#2701) 
* Refactor: replace expect with ServiceError, remove repetive code

* chore: remove unnecessary code

* chore: apply cr suggestions

* chore: apply clippy recommendations

* Improve constructing errors

* Improve test assertions

* fix: replace max_regions assertion with operation error

* fix: adjust according to CR

* chore: replace Option with OperationResult for estimate_cardinality

* chore: replace Option with OperationResult for filter

* Better handling of transforming none into an error

* Replace for loop with simple iterator, simply other iterator

---------

Co-authored-by: timvisee <tim@visee.me>
 2023-09-25 15:10:45 +02:00
Arnaud Gourlay
840248f6fc Introduce Memory common module (#2712) 
* Introduce Memory common module

* fix Windaube build

* move mmap_ops as well
 2023-09-25 11:30:50 +02:00
Ivan Pleshkov
3ffaa76d8c Search rescoring modes (#2628) 
* search rescoring modes

* fix tests build

* use bool option in rest api

* minor refactoring

* fmt

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2023-09-24 16:27:05 +02:00
Luis Cossío
e98a3c7d3d Expose new recommend scorer via recommend strategy (#2662) 2023-09-22 17:18:25 -03:00
Luis Cossío
4e24ce2219 Increase acceptable stopping delay, better debug info in test_building_cancellation (#2713) 2023-09-22 09:42:28 -03:00
Arnaud Gourlay
74980c0439 Fix cardinality estimation for Any matcher (#2710) 2023-09-21 19:42:03 +02:00
Arnaud Gourlay
b2145eeb5d RocksDB deletes obsolete file more aggressively (#2709) 2023-09-21 13:26:14 +02:00
Arnaud Gourlay
94699183fc introduce IO common module (#2704) 2023-09-21 09:13:19 +02:00
Roman Titov
abf533671a Refactor lib/common to support adding sub-crates to "common" (#2551) 2023-09-20 13:13:32 +02:00
Tim Visée
b62ab26f1f Fix test_building_cancellation flakiness with different delays (#2697) 
* Reduce flakiness of `test_building_cancellation` with 5k base segment

* use fixed acceptable stopping delay, and relative cancelling delay

* Remove debug statement

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
 2023-09-20 10:56:33 +02:00
dependabot[bot]
7329b16386 Bump chrono from 0.4.30 to 0.4.31 (#2691) 
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.30 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.30...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-09-20 08:38:46 +02:00
dependabot[bot]
baa349e49b Bump io-uring from 0.6.1 to 0.6.2 (#2692) 
Bumps [io-uring](https://github.com/tokio-rs/io-uring) from 0.6.1 to 0.6.2.
- [Commits](https://github.com/tokio-rs/io-uring/commits)

---
updated-dependencies:
- dependency-name: io-uring
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-09-19 08:39:50 +02:00
dependabot[bot]
c468611059 Bump schemars from 0.8.13 to 0.8.15 (#2687) 
Bumps [schemars](https://github.com/GREsau/schemars) from 0.8.13 to 0.8.15.
- [Release notes](https://github.com/GREsau/schemars/releases)
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.13...v0.8.15)

---
updated-dependencies:
- dependency-name: schemars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-09-19 08:38:27 +02:00
Tim Visée
9b848e5929 Reduce flakiness of test_building_cancellation with larger base seg… (#2682) 
* Reduce flakiness of `test_building_cancellation` with larger base segment

* Change number of points to 2k, increase timeouts a bit
 2023-09-18 14:31:57 -03:00
Arnaud Gourlay
5fe82cb6d7 Sparse vector index (#2318) 
* Sparse vector index

* better name

* fill properly postings

* conflict

* easy reviews

* use unstable sort

* use optional instead of bool flag

* clearer score type and future direction

* NonZeroUsize length for FixedLengthPriorityQueue

* rename id to record_id

* cleanup dimension inverted index

* standard Iterator

* code review

* Shift comment

* early return if no matching posting lists

* sprinkle iterators

* privatize the means of production

---------

Co-authored-by: timvisee <tim@visee.me>
 2023-09-15 16:30:11 +02:00
Luis Cossío
117518b1b7 Recommendation scorer internals (#2538) 
* use enum_dispatch on vector_index_base.rs

* add reco_query_scorer.rs

* smol refactor on reco_query_scorer

* add PositiveNegative variant

* recommend query scorer internals

- missing tests

* add test for RecoQuery

* Revert "use enum_dispatch on vector_index_base.rs"

This reverts commit 016e831d00d67ef01e64b4f7a76854e555cd9697.

* remove enum_dispatch usage

* disable score_internal implementation for QuantizedRecoQueryScorer

* refactor test

* refactor with latest preprocess changes

* add reco scorer for async scorer

* add iter_all() helper for RecoQuery

* rename PositiveNegative -> Recommend,
change variable names,
refactor quantized query scorer

* refactor new raw scorer

* add tests for comparison raw against async and u8 quant

* smol test improvements

* fix error after rebase

* fmt

* finish score equivalency test, remove dbgs

* make scorer builder for quantized storage

* move query preprocessing to `new()` of quantized query scorers

* self review

* review suggestions

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2023-09-14 12:37:14 -03:00
Josh Soref
a0bbc40bc9 Spelling (#2658) 
* spelling: accumulating

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: and

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: back

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: batching

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: been

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: benchmark

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: collections

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: confusion

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: consensus

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: decrease

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: equal

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: github

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: minimal

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: nonexistent

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: oversampling

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: paths

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: points

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: prevent

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: protobuf

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: proxied

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: randomness

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: recover

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

---------

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2023-09-14 11:38:23 +02:00
Arnaud Gourlay
30a440e2b0 Misc. improvements to GraphLinks (#2651) 2023-09-13 12:19:05 +02:00
dependabot[bot]
3eafe92722 Bump chrono from 0.4.28 to 0.4.30 (#2634) 
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.28 to 0.4.30.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.28...v0.4.30)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-09-12 08:19:28 +02:00
dependabot[bot]
eba8610948 Bump walkdir from 2.3.3 to 2.4.0 (#2633) 
Bumps [walkdir](https://github.com/BurntSushi/walkdir) from 2.3.3 to 2.4.0.
- [Commits](https://github.com/BurntSushi/walkdir/compare/2.3.3...2.4.0)

---
updated-dependencies:
- dependency-name: walkdir
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-09-12 08:17:35 +02:00
Ivan Pleshkov
7d9662d19a Fix flaky test in quantization oversampling (#2629) 2023-09-11 14:56:43 +02:00
Luis Cossío
562b3aa070 fix clippy errors on segment_builder_test.rs (#2606) 2023-09-06 12:32:10 -03:00
Luis Cossío
3558c71285 fix test_building_cancellation (#2094) 2023-09-06 11:34:01 -03:00
Luis Cossío
b25fc4c8ea Refactor batch search to allow different scorers (#2529) 
* add enum for vector query on segment search

* rename newly introduced types

* fix: handle QueryVector on async scorer

* handle QueryVector in QuantizedVectors impl

* fix async scorer test after refactor

* rebase + refactor on queue_proxy_shard.rs

* constrain refactor propagation to segment_searcher

* fmt

* fix after rebase
 2023-09-05 09:26:24 -03:00
dependabot[bot]
21f352a3ef Bump schemars from 0.8.12 to 0.8.13 (#2582) 
Bumps [schemars](https://github.com/GREsau/schemars) from 0.8.12 to 0.8.13.
- [Release notes](https://github.com/GREsau/schemars/releases)
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.12...v0.8.13)

---
updated-dependencies:
- dependency-name: schemars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-09-05 09:47:11 +02:00
dependabot[bot]
632fca1658 Bump chrono from 0.4.26 to 0.4.28 (#2581) 
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.28.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.26...v0.4.28)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-09-05 09:46:45 +02:00
Ivan Pleshkov
2a29cf0241 immutable map index integration (#2524) 
* immutable map index integration

* remove wipe

* fix unit tests

* get appendable flag from config

* minor refactoring

* fix chunked mmap appendable flag

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2023-09-04 15:24:52 +02:00
Tim Visée
1d05b9f3ef Centralize common validation functions (#2548) 
* Centralize generic range validation

* Centralize validate not empty

* Centralize collection name validation

* Add geo polygon validation in REST API and centralize validation

* Add tests for common validation functions

* Don't use reference in generic range validation function

* Other improvements
 2023-09-04 10:57:24 +02:00
zpp12354321
51b5a8d187 fix: remove replaced id when db corrupted (#2510) 2023-09-03 14:00:34 +02:00
Luis Cossío
1f0fcfecb6 refactor: Make preprocess() own its vector argument (#2553) 
* make preprocess own its vector argument

* update doc comment
 2023-09-01 08:55:22 +02:00
Ivan Pleshkov
032f760c5a Immutable map index (#2453) 
* immutable map index

* use (super) instead of (crate)

* more comments

* decompose remove_point

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2023-08-31 15:20:55 +02:00
Arnaud Gourlay
6fd37d04f2 Delete temp snapshot files on error (#2528) 
* Delete temp snapshot files on error

* shorter resulting path to please max length on Windows

* Remove one layer of tmp folder for shards

Unicity of segment tmp folder provided by Uuid
 2023-08-31 12:09:59 +02:00
Tim Visée
c2faf6c729 Fix missing points, vectors and payload (#2514) 
* On segment flush, read-lock all segments to prevent CoW between flushes

For example, we have a point on an immutable segment. If we use a
set-payload operation, we do copy-on-write. The point from immutable
segment A is deleted, the updated point is stored on appendable segment
B.

Because of flush ordering segment B (appendable) is flushed before
segment A (not-appendable). If the copy-on-write operation happens in
between, the point is deleted from A but the new point in B is not
persisted. We cannot recover this by replaying the WAL in case of a
crash because the point in A does not exist anymore, making
copy-on-write impossible.

Locking all segments prevents copy-on-write operations from occurring in
between flushes.

* Return proper status on set payload operations in segment

* Disable propagating point deletions to its vectors, it looses vectors

* Update vector tests after disabling point delete propagation to vectors

* Implement retry with exponential backoff for read-locking all segments

* Use try_read_for rather than sleeping after lock attempts

* Simplify locking many, just lock one by one

Refs: <https://github.com/qdrant/qdrant/pull/2527>
Co-authored-by: generall <andrey@vasnetsov.com>

* Comment out now obsolete test

* Error handling in method getting segment locks by ID

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2023-08-31 10:03:51 +02:00
Arnaud Gourlay
790103f888 Fix lints for Clippy 1.72 (#2476) 2023-08-24 16:43:34 +02:00
dependabot[bot]
b500f6e446 Bump charabia from 0.8.2 to 0.8.3 (#2490) 
Bumps [charabia](https://github.com/meilisearch/charabia) from 0.8.2 to 0.8.3.
- [Release notes](https://github.com/meilisearch/charabia/releases)
- [Commits](https://github.com/meilisearch/charabia/compare/v0.8.2...v0.8.3)

---
updated-dependencies:
- dependency-name: charabia
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-23 12:05:59 +02:00
Ivan Pleshkov
fef89e84b9 fix flaky oversampling test (#2494) 
* fix unstable oversampling test

* review fixes

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2023-08-23 11:26:54 +02:00
dependabot[bot]
1214dab600 Bump tempfile from 3.7.1 to 3.8.0 (#2480) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.7.1 to 3.8.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.7.1...v3.8.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-22 15:47:11 +02:00
Ivan Pleshkov
d14d2c2e36 binary quantization (#2470) 
* binary quantization

* update quantization version

* fix quantizing bug

* are you happy fmt
 2023-08-21 21:16:27 +02:00
Ivan Pleshkov
db3f712b4d fix quantization oversampling (#2478) 2023-08-21 18:04:38 +02:00