upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-06 10:00:58 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,465 Commits 344 Branches 118 Tags
94de0e2d2dcf997bf26bec34dad407d8de9a9abb
Commit Graph

1780 Commits

Author SHA1 Message Date
Ivan Pleshkov
22fd3a3963 minor doc and api in tests updates (#1022) 2022-09-15 13:37:38 +04:00
dependabot[bot]
1508ce83be Bump criterion from 0.3.6 to 0.4.0 (#1004) 
Bumps [criterion](https://github.com/bheisler/criterion.rs) from 0.3.6 to 0.4.0.
- [Release notes](https://github.com/bheisler/criterion.rs/releases)
- [Changelog](https://github.com/bheisler/criterion.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bheisler/criterion.rs/compare/0.3.6...0.4.0)

---
updated-dependencies:
- dependency-name: criterion
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-09-13 10:18:56 +02:00
Andrey Vasnetsov
ba26e2f85e Faster filtered scroll (#1003) 
* faster filtered scroll for low cardinality filters

* add test

* scroll strategy heuristics
 2022-09-12 17:55:56 +02:00
Andrey Vasnetsov
4ae6552379 tiny-map + benchmark (#1001) 
* tiny-map + benchmark

* are you happy clippy

* lower default capacity

* fmt

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
 2022-09-12 01:56:47 +02:00
Ivan Pleshkov
f6b2186193 [WIP] Many named vectors per point (#958) 
* many named vectors per point (segment-level)

* operation result for dim function

* beautifulized vector name

* fix naming bug

* segment version migration

* fmt

* add segment tests

* are you happy clippy

* fix build

* [WIP] many named vectors per point (collection-level) (#975)

* config and search

* fix placeholders for proxy segment move

* remove VectorType from collection

* are you happy fmt

* vectors in grps messages

* create collections with vectors

* segment holder fixes

* are you happy fmt

* remove default vector name placeholders

* are you happy fmt

* are you happy clippy

* fix build

* fix web api

* are you happy clippy

* are you happy fmt

* record vector&vectors

* openapi update

* fix openapi integration tests

* segment builder fix todo

* vector names for update from segment

* remove unwrap

* backward compatibility

* upd openapi

* backward compatible PointStruct

* upd openapi

* fix record back-comp

* fmt

* vector configuration backward compatibility

* fix vetor storage size estimation

* fmt

* multi-vec segment test + index test

* fmt

* api integration tests

* [WIP] Named vectors struct (#1002)

* move to separate file

* named vectors as struct

* use cow

* fix build

* keys iterator

* avoid copy in PointStruct -> get_vectors

* avoid another copy

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-09-11 20:59:23 +02:00
Andrey Vasnetsov
24ac939b4a Sync Points API (#985) 
* shard sync operation

* fnt

* tests

* fix test

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* match payload after vector only

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-05 15:09:39 +02:00
Andrey Vasnetsov
b9eee55a9f Full text search (#963) 
* allow additional params for payload field index

* fmt

* wip: full text index building

* fmt

* text search request

* text search request

* full text index persitance and loading

* fmt

* enable fts index in mapping

* clippy

* fix tests + add integration test

* review fixes: extend payload index test

* revert incedental change
 2022-09-01 12:50:12 +02:00
Andrey Vasnetsov
be38254ee8 small refactoring (#746) 
* small refactoring

* fix tests
 2022-09-01 12:36:28 +02:00
dependabot[bot]
4dd9d53b8d Bump geo from 0.22.1 to 0.23.0 (#964) 
Bumps [geo](https://github.com/georust/geo) from 0.22.1 to 0.23.0.
- [Release notes](https://github.com/georust/geo/releases)
- [Changelog](https://github.com/georust/geo/blob/main/CHANGES.md)
- [Commits](https://github.com/georust/geo/compare/geo-0.22.1...geo-0.23.0)

---
updated-dependencies:
- dependency-name: geo
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-08-29 12:13:32 +02:00
Arnaud Gourlay
95dcc333b0 properly format doc links for cargo doc (#956) 2022-08-23 09:09:34 +02:00
Ivan Pleshkov
f9fb0777a0 Wrap rocksdb column usages (#951) 
* wrap rocksdb column usages

* remove obsolete comments

* are you happy clippy
 2022-08-22 09:41:08 +02:00
Andrey Vasnetsov
a2acca0345 Segment batch search (#813) 
* batch search benchmark

* collect filter iterator in indexed search

* fmt

* fix

* fix

* fmt

* use new tempfile create

* auto batching

* Clippy fixes

* REST, gRPC and internal APIs

* fix bugs & less duplication

* two steps payload retrieval mechanism & fix duplication

* add proxy_segment implementation & tests

* add gRPC docs

* remove unused code (#950)

* only filter ids within a batch

* add more equivalence tests

* add integration test search vs batch

* assert more search options in tests

* cleanup assertions

* fix offset panic

* rename search batch API

* openapi spec

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-08-18 14:48:17 +02:00
Ivan Pleshkov
f357bd5d9b Allow to flush segment in separate thread (#927) 
* allow to flush segment in separate thread

* flush as separate function (#928)

* flush as separate function

* review suggestion

* reduce locks during vector scoring

* fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* don't run background flush twice

* Update lib/segment/src/segment.rs

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* increase flush interval

* Update lib/segment/src/segment.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* are you happy fmt

* test background flush

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-08-15 13:47:52 +04:00
Arnaud Gourlay
38c8097fc8 Clippy next (#941) 
* Clippy derive_partial_eq_without_eq

* Clippy  explicit_auto_deref

* Clippy single_match

* Clippy manual_find_map

* Clippy unnecessary_to_owned

* Clippy derive_partial_eq_without_eq

* Clippy get_first
 2022-08-12 09:38:31 +02:00
Russ Cam
8e1f2ca353 Use tempfile (#922) 
This commit replaces tempdir with tempfile.
tempdir is archived.

Closes #544
 2022-08-10 09:39:21 +02:00
dependabot[bot]
9713267358 Bump rocksdb from 0.18.0 to 0.19.0 (#920) 
* Bump rocksdb from 0.18.0 to 0.19.0

Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb) from 0.18.0 to 0.19.0.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases)
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.18.0...v0.19.0)

---
updated-dependencies:
- dependency-name: rocksdb
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* iterator_cf returns a Result

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-08-08 11:12:22 +02:00
Andrey Vasnetsov
70317d1bbc check for segment integrity by checking the version, skip on failure (#904) 2022-08-03 17:39:13 +02:00
Andrey Vasnetsov
88c672475e Fix integration test (#903) 
* fix integration test problems + overflow

* fmt
 2022-08-03 16:45:34 +02:00
Andrey Vasnetsov
826165cfcf fix enum conversion in gRPC (#898) 2022-08-03 09:48:30 +02:00
Ivan Pleshkov
5728a0e670 optimizers and indices telemetry (#892) 2022-08-02 13:30:31 +04:00
Arnaud Gourlay
dc4cccde3d Prevent Vector size zero (#876) 2022-07-31 13:03:53 +02:00
Ivan Pleshkov
ddbd190aaa Grpc telemetry (#873) 
* grpc telemetry

* fix avg calc

* separate telemetry collectors

* update openapi

* use parking lot for actix

* are you happy fmt

* fix miskate in naming
 2022-07-30 12:33:15 +04:00
Andrey Vasnetsov
52e0c53e36 Fix hnsw graph backward compatibility for 0 8 5 (#865) 
* try to load legacy graph format if primary failed

* fmt
 2022-07-26 19:14:50 +02:00
Andrey Vasnetsov
1ceb9307d2 Fix payload index overwrite vals (#849) 
* force removing of exisisting values on payload index upsert

* clippy
 2022-07-25 11:38:41 +02:00
Ivan Pleshkov
42e930ab8f Segment telemetry (#814) 
* segment telemetry

* anonymize trait

* fix build

* are you happy fmt

* anonimyze implementations

* sliding window avg (#826)

* Actix web telemetry (#828)

* actix web telemetry

* small as move

* use tokio mutex instead of std

* add comments

* are you happy fmt

* use u16 as http status code

* telemetry structs rename

* fix build

* using parking lot mutex

* telemetry web api (#842)

* telemetry web api

* telemetry openapi (#843)

* use async mutex for telemetry collector

* use tokio mutex for telemetry collector

* are you happy fmt
 2022-07-22 19:27:07 +04:00
Egor Ivkov
026bd040b0 Add import formatting rules (#820) 
* Add import formatting rules

* Review fix: update rusty hook
 2022-07-15 15:42:25 +03:00
Egor Ivkov
ef3c6c5e15 Fair hash ring (#816) 
* Fair hash ring

* Python test processes removal fix
 2022-07-14 19:27:53 +02:00
dependabot[bot]
80213f8af2 Bump bitvec from 1.0.0 to 1.0.1 (#802) 
Bumps [bitvec](https://github.com/bitvecto-rs/bitvec) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/bitvecto-rs/bitvec/releases)
- [Changelog](https://github.com/bitvecto-rs/bitvec/blob/main/CHANGELOG.md)
- [Commits](https://github.com/bitvecto-rs/bitvec/commits)

---
updated-dependencies:
- dependency-name: bitvec
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-07-11 20:50:44 +02:00
Ivan Pleshkov
ff236410df Graph layers remove code duplications (#796) 
* remove add link points

* remove level factor

* remove use heuristic

* remove code duplications in graph layers

* are you happy fmt

* restore unit tests

* remove obsolete comment
 2022-07-11 18:29:34 +04:00
Andrey Vasnetsov
098bc4c751 Count api (#777) 
* count api

* test for approx counting

* fmt + clippy

* unit test

* add warning
 2022-07-05 09:33:38 +02:00
dependabot[bot]
ebb5b3da3c Bump geo from 0.22.0 to 0.22.1 (#779) 
Bumps [geo](https://github.com/georust/geo) from 0.22.0 to 0.22.1.
- [Release notes](https://github.com/georust/geo/releases)
- [Changelog](https://github.com/georust/geo/blob/main/CHANGES.md)
- [Commits](https://github.com/georust/geo/compare/geo-0.22.0...geo-0.22.1)

---
updated-dependencies:
- dependency-name: geo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-07-04 21:58:11 +02:00
Andrey Vasnetsov
e983b07a15 Parallel hnsw building (#773) 
* parallel hnsw building

* improve hnsw payload blocks condition

* update indexing optimizer condition

* fmt
 2022-07-03 01:14:05 +02:00
Andrey Vasnetsov
fac87018c4 Snapshoting API (#764) 
* wip: rest api for snapshots

* fmt

* fix tests

* fmt

* wip: collection snapshoting and recovery

* fmt

* remote shard snapshots

* remote shard test

* fmt

* extend proxy snapshot test + fix double read lock

* fmt + clippy

* openapi schema

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* reduce writes on snapshots location

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-07-01 13:21:57 +02:00
Arnaud Gourlay
cb82b7be01 Fix clippy 1.62 (#771) 
* clippy let_unit_value

error: this let-binding has unit value
   --> lib/segment/src/vector_storage/simple_vector_storage.rs:397:9
    |
397 | /         let _top_idx = match closest.get(0) {
398 | |             Some(scored_point) => {
399 | |                 assert_ne!(scored_point.idx, 2);
400 | |                 assert_eq!(&raw_res1[scored_point.idx as usize], scored_point);
...   |
404 | |             }
405 | |         };
    | |__________^
    |
    = note: `-D clippy::let-unit-value` implied by `-D warnings`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#let_unit_value
help: omit the `let` binding
    |
397 ~         match closest.get(0) {
398 +             Some(scored_point) => {
399 +                 assert_ne!(scored_point.idx, 2);
400 +                 assert_eq!(&raw_res1[scored_point.idx as usize], scored_point);
401 +             }
402 +             None => {

* clippy format_push_string

error: `format!(..)` appended to existing `String`
   --> src/main.rs:224:21
    |
224 |                     error.push_str(&format!("Deadlock #{}\n", i));
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `-D clippy::format-push-string` implied by `-D warnings`
    = help: consider using `write!` to avoid the extra allocation
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#format_push_string

error: `format!(..)` appended to existing `String`
   --> src/main.rs:226:25
    |
226 | /                         error.push_str(&format!(
227 | |                             "Thread Id {:#?}\n{:#?}\n",
228 | |                             t.thread_id(),
229 | |                             t.backtrace()
230 | |                         ));
    | |__________________________^
    |
    = help: consider using `write!` to avoid the extra allocation
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#format_push_string

error: could not compile `qdrant` due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
error: could not compile `qdrant` due to 2 previous errors
 2022-07-01 01:59:49 +02:00
Arnaud Gourlay
db9e5a3abb SegmentHolder snapshot all segments (#759) 
* SegmentHolder snapshot all segments

* snapshot proxy segment

* release segment resources

* use write_version in points deletion
 2022-06-30 13:50:15 +02:00
dependabot[bot]
4abc9d1689 Bump pprof from 0.9.1 to 0.10.0 (#754) 
Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.9.1 to 0.10.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases)
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/compare/v0.9.1...v0.10.0)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-06-28 09:43:39 +02:00
Andrey Vasnetsov
92c05bb56f set max number of open files for rocksdb 2022-06-28 02:22:08 +02:00
Andrey Vasnetsov
8ac3eef4c7 fix db flush ordering to prevent persisting versions for un-persisted points 2022-06-28 00:38:10 +02:00
Arnaud Gourlay
9129a3d6eb introduce segment-level snapshots (#751) 2022-06-27 18:58:39 +02:00
Andrey Vasnetsov
850e937c2a Storage points tracking refactoring (#750) 
* segment refactoring

* rm points iterator

* fmt
 2022-06-27 15:17:09 +02:00
Andrey Vasnetsov
83c21450c3 Pagination (#743) 
* offset param for search and recomendation

* test and fix for pagination

* clippy + fmt

* upd schemas

* fix grpc requests in test
 2022-06-27 13:22:02 +02:00
dependabot[bot]
b7a2e6e001 Bump geo from 0.21.0 to 0.22.0 (#742) 
Bumps [geo](https://github.com/georust/geo) from 0.21.0 to 0.22.0.
- [Release notes](https://github.com/georust/geo/releases)
- [Changelog](https://github.com/georust/geo/blob/main/CHANGES.md)
- [Commits](https://github.com/georust/geo/compare/geo-0.21.0...geo-0.22.0)

---
updated-dependencies:
- dependency-name: geo
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-06-24 21:12:53 +02:00
Andrey Vasnetsov
40245c5e30 Cifs dir delete (#705) 
* consume objects before removing the data

* fmt

* rm debug code
 2022-06-16 12:38:43 +02:00
dependabot[bot]
e3c58b47fa Bump geo from 0.20.1 to 0.21.0 (#683) 
Bumps [geo](https://github.com/georust/geo) from 0.20.1 to 0.21.0.
- [Release notes](https://github.com/georust/geo/releases)
- [Changelog](https://github.com/georust/geo/blob/main/CHANGES.md)
- [Commits](https://github.com/georust/geo/compare/geo-0.20.1...geo-0.21.0)

---
updated-dependencies:
- dependency-name: geo
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-06-14 08:40:12 +02:00
Andrey Vasnetsov
d5ed8050b7 bug fix: prevent division by zero if empty payload 2022-06-08 02:39:58 +02:00
Andrey Vasnetsov
8e52c255ab add consensus configuration + lower log level + fix flickering test 2022-06-06 21:44:03 +02:00
Gabriel Velo
c15981092a [WIP] [real-time index] Implement payloadstorage for structpayloadindex (#642) 
* [real-time index] Extend FieldIndex enum and StructPayloadIndex with method from PayloadStorage

* [real-time index] add missing remove_point methods

* [real-time index] add new index to FieldIndex enum

* fix compile

* are you happy fmt

* merge load and remove

* fix test generics

* decrement points count

* remove from histogram

* simplify histogram usage

* [real-time index] remove old tests and fix clippy warnings

* histogram: method to derive range by size (#657)

* [real-time index] add histogram based payload_blocks implementation.

* payload blocks

* fmt

* clippy

* [real-time index] refactor Segment to use PayloadIndex instead of PayloadStorage.

* fix tests

* fmt

* clippy

* rename indexes

* remove redundent params

* add struct payload deletion test + fix delete payload in map index

* remove payload threshold

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-06-06 17:14:20 +02:00
Andrey Vasnetsov
7c254a0de0 preserve fields order in OpenAPI schema (#655) 2022-06-02 19:12:30 +02:00
Andrey Vasnetsov
2601c017de Smarter defaults (#637) 
* auto segments number

* auto segments number

* replace vector number limits with vector size limits

* fmt
 2022-06-01 18:09:38 +02:00
Andrey Vasnetsov
1b458780eb On disk payload storage (#634) 
* implement on-disk payload storage

* fmt + clippy

* config param for on-disk payload storage

* upd openapi definitions

* add integration test with on-disk payload

* fix clippy

* review fixes

* fmt
 2022-06-01 17:23:34 +02:00