upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-04 17:10:59 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,206 Commits 345 Branches 118 Tags
b6b704955f9687d69ec00e499b6afdaba3cc89f0
Commit Graph

185 Commits

Author SHA1 Message Date
Andrey Vasnetsov
d7379f0706 Fix recover snapshot peer (#1534) 
* allow to recover distributed snapshot on local intance + allow recover non-existing collections

* async snapshot recovery

* fix tests

* be polite earlier
 2023-03-15 16:27:10 +01:00
Ivan Pleshkov
128e49fcc3 integrate quantized data to storages (#1311) 
* integrate quantized data to storages

* revert gitignore

* are you happy clippy

* quantize in optimizer

* provide flag

* fix segfault

* skip quantization flag, update scores

* use quantization flag

* are you happy fmt

* use quantization flag

* quantized search test

* are you happy fmt

* refactor test, refactor scorer choosing

* are you happy fmt

* run quantization on segment builder

* decrease testing parameters

* simplify segment

* update version

* remove use_quantization flag

* provide quantization config

* quantization version up

* euclid dist

* add euclid test

* saveload

* fix initialization bugs

* quantization lib version up

* fix arm build

* refactor scorer selecting

* quant lib version up

* are you happy fmt

* are you happy fmt

* are you happy clippy

* add save/load test for simple storage

* add comments

* quantiles

* quantization mmap

* remove f32

* mmap test

* fix mmap slice

* fix mmap test

* use chunks for quantization storage

* fix build

* are you happy fmt

* update quantization library

* update quantization lib

* update quantization lib

* integrate api changes

* are you happy fmt

* change quantization api

* additional checks in tests

* update quantization version

* fix unit tests

* add quantization to storage config

* use quantization for all cardinality search cases

* Integrate quantization suggestions 2 (#1520)

* review api

* wip: refactor quantization integrations

* wip: refactor quantization integrations

* wip: fmt

* include quantization into snapshot

* fmt

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:26:00 +01:00
Luis Cossío
c414a77105 feat: add "any" to "match" statements (#1466) 
* feat: add "any" to "match" statements

Adds support for using "any" to match on a list of different possible values

* enable proper counting in cardinality estimation

* fmt

* upd OpenAPI

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:20:11 +01:00
Roman Titov
5149d39146 Add /cluster/recover API endpoint (#1451) 
- Add required internal `raft::raw_node::RawNode::report_snapshot` calls
- Add methods to propagate `raft::raw_node::RawNode::request_snapshot` call to the consensus thread
- Add `/cluster/recover` API endpooint
- Add `test_collection_recovery.py` integration test (disabled on CI; documents current, incorrect behavior)
 2023-03-15 16:19:59 +01:00
Andrey Vasnetsov
8c067bc977 upd to v1.0.3 (#1523) 2023-03-05 21:11:40 +01:00
Andrey Vasnetsov
99d94cbebe upd version (#1486) 2023-02-21 18:36:21 +01:00
Andrey Vasnetsov
486737d7b5 v1.0.1 2023-02-08 14:00:55 +01:00
Arnaud Gourlay
f66031ffdf Update to v1.0.0 (#1434) 
* Update to v1.0.0

* resync v1.0.0 openapi
 2023-02-08 09:12:02 +01:00
Andrey Vasnetsov
1687e71f74 remove /collections/aliases (#1440) 2023-02-07 08:51:32 +01:00
Andrey Vasnetsov
baed8cfd95 RFC for 2023 roadmap (#1309) 
* RFC for 2023 roadmap

* upd bage

* upd titles
 2023-02-06 15:23:52 +01:00
Andrey Vasnetsov
3ba631b9bd remove deprecate point ids param in gprc (#1305) 
* remove deprecate point ids param in gprc

* upd docs
 2023-02-06 15:22:14 +01:00
Roman Titov
34bdbc2fe6 Add read_consistency parameter to the APIs (#1371) (#1407) 
* WIP: Add `read_consistency` parameter to the APIs

* WIP: Add `read_consistency` parameter to the APIs

TODO:
- Add documentation

* `cargo fmt`

* Add gRPC documentation

* Add OpenAPI documentation

* Cleanup

* fixup! Add OpenAPI documentation

* fixup! Add gRPC documentation

Who would have known there's `generate_grpc_docs.sh`!? 🥲🙈🤦‍♀️

* generate openapi

* Fix `read_consistency` query parameter deserialization

* Further improve `read_consistency` query parameter deserialization

* `cargo clippy`

* Fix `Payload` comparison during read operation result resolving

* Fix grammar

* rename `read_consistency` -> `consistency` and add integration test

* use majority for test

* fix tests

* Fix tests

* fixup! Fix tests

Apply the same fix to `ScoredPoint`

* Remove an `unwrap`

* fixup! Fix tests

Gotta love those negative conditions, or how a missed `!` can ruin your day... 🤦‍♀️

* Make internal API calls strictly "local-shard only"

* Implement a few basic traits for `ResolverRecord`

* fixup! Implement a few basic traits for `ResolverRecord`

* Revert "Make internal API calls strictly "local-shard only""

This reverts commit 25378e61ff.

* Fix `Record::payload` and `ScoredPoint::payload` serialization

* Revert "Fix `Record::payload` and `ScoredPoint::payload` serialization"

This reverts commit b566bea49b.

* Fix `Record::payload` and `ScoredPoint::payload` visibility

* fixup! Fix `Record::payload` and `ScoredPoint::payload` visibility

Remove `todo!()`

* refactoring

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-02-06 15:21:43 +01:00
Andrey Vasnetsov
90c6b784f5 add init state to replica set (#1404) 
* add init state to replica set

* update openapi

* disallow activating shard from non-initialize state

* fmt

* remove useless operation confirm on collection creation

* clippy fixes

* fix test
 2023-02-06 15:20:20 +01:00
Andrey Vasnetsov
80af394d32 add write consistency param to rest and grpc api (#1397) 
* add write consistency param to rest and grpc api

* fix grpcurl + refactor grpc for better extencibility

* upd grpc docs

* fix conversion
 2023-02-06 15:18:58 +01:00
Andrey Vasnetsov
8cd4fe15ed Init collection from (#1364) 
* WIP

* WIP: add shards selection

* fmt

* WIP: collection init from other collection

* fmt

* fix bugs

* add collection write lock + use collection manager runtime instead of search runtime

* fmt

* integration test

* better names ?

* payload indexes transfer
 2023-02-06 15:17:01 +01:00
Arnaud Gourlay
fe06844c48 Delete snapshots API (#1377) 
* Delete snapshots API

* remove unecessary plurals
 2023-02-06 15:16:33 +01:00
Arnaud Gourlay
25d1383496 List aliases API (rest & gRPC) (#1363) 
* List aliases API (rest & gRPC)

* rework api to avoid conflicts
 2023-02-06 15:15:52 +01:00
Andrey Vasnetsov
313fa9fe80 upd to v0.11.7 (#1342) 2023-01-13 16:03:56 +01:00
Andrey Vasnetsov
7b59d2b48f up to v0.11.6 - 1.0 release candidate (#1304) 2022-12-27 13:06:48 +01:00
Andrey Vasnetsov
a1f651fbd6 Recommendation with other collection (#1283) 
* WIP: move recommendation API out of collection

* recomendations with external source of vectors

* fix test

* fmt

* fixes + test

* review fixes
 2022-12-25 22:51:36 +01:00
Andrey Vasnetsov
06be5d20bc update to v0.11.5 (#1269) 2022-12-06 17:59:55 +01:00
Andrey Vasnetsov
4e084a6852 allow set different M param for default HNSW and payload-related links (#1257) 
* allow set different M param for default HNSW and payload-related links

* review fixes
 2022-12-06 15:04:19 +01:00
Andrey Vasnetsov
5cac1b2ab5 add parameter that defines a source of truth for snapshot recovery (#1262) 
* add parameter that defines a source of truth for snapshot recovery

* review fixes
 2022-12-06 15:04:09 +01:00
Andrey Vasnetsov
469c098ca4 Update points by filter (#1249) 
* allow to set remove payload by selector

* fmt
 2022-12-06 15:02:33 +01:00
Andrey Vasnetsov
0d130c395a Full payload update (#1245) 
* implement api to fully overwrite payload of the point

* fmt
 2022-12-06 15:02:02 +01:00
Andrey Vasnetsov
c60cb27276 v0.11.4 (#1244) 2022-11-25 14:53:22 +01:00
Andrey Vasnetsov
d702dd7a58 update openapi - telemetry + max_indexing_threads 2022-11-25 13:28:42 +01:00
Ivan Pleshkov
5e3416e2ce Hnsw links memmap (#1211) 
* add on disk key

* remove obsolete graph initialization

* remove obsolete max level

* update openapi

* graph links trait

* use mmap option

* same format for ram and mmap

* fix segment unit tests

* are you happy fmt

* are you happy clippy

* fix ci and add mmap test

* review fixes

* remove unused try-from

* fix version compatibility

* avoid loading from disk during conversion

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-11-25 12:17:57 +01:00
Andrey Vasnetsov
84fc500b91 v0.11.3 (#1217) 2022-11-15 16:12:21 +01:00
Andrey Vasnetsov
ec25e973fb Recover distributed collection (#1214) 
* WIP: shards recovering API: download snapshot

* fmt

* snapshot recovery API

* fmt

* snapshot recovery integration test + fixes

* review fixes

* review fixes 2

* Update lib/storage/src/content_manager/snapshots/recover.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-11-15 13:33:22 +01:00
Andrey Vasnetsov
e027db4d27 V0.11.2 (#1209) 
* Update: unmaintained crate memmap -> memmap2 (#559) (#1160)

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* Consensus q n a (#1169)

* Expand comments and minor refactoring for raft state

* fmt

* add comments to consensus.rs

* fix "Consensus q n a" compatibility

* Use less ram for id tracker (#1176)

* use less ram for id tracker

* are you happy clippy

* use vec for internals

* use versions for internal ids

* keys test

* Use less ram for id tracker fixes (#1182)

* WIP: internal_to_version

* fmt

* fix unit tests

* add comment

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* remove suggesting changes in replications on replication factor change (#1177)

* Bump actix-cors from 0.6.3 to 0.6.4 (#1185)

Bumps [actix-cors](https://github.com/actix/actix-extras) from 0.6.3 to 0.6.4.
- [Release notes](https://github.com/actix/actix-extras/releases)
- [Commits](https://github.com/actix/actix-extras/compare/cors-v0.6.3...cors-v0.6.4)

---
updated-dependencies:
- dependency-name: actix-cors
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* enable HTTP compression middleware (#1184)

* Use systematically assert_http_ok for better error reporting (#1183)

* Use systematically assert_http_ok for better error reporting

* extraction assertion to use it outside of pytest

* Bump pprof from 0.10.1 to 0.11.0 (#1188)

Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.10.1 to 0.11.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases)
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Cosine dist zero vectors (#1198)

* skip pre-processing of zero-length vector for cosine distance

* fmt

* Bump env_logger from 0.9.1 to 0.9.3 (#1201)

Bumps [env_logger](https://github.com/env-logger-rs/env_logger) from 0.9.1 to 0.9.3.
- [Release notes](https://github.com/env-logger-rs/env_logger/releases)
- [Changelog](https://github.com/env-logger-rs/env_logger/blob/main/CHANGELOG.md)
- [Commits](https://github.com/env-logger-rs/env_logger/compare/v0.9.1...v0.9.3)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump indicatif from 0.17.1 to 0.17.2 (#1202)

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.17.1 to 0.17.2.
- [Release notes](https://github.com/console-rs/indicatif/releases)
- [Commits](https://github.com/console-rs/indicatif/compare/0.17.1...0.17.2)

---
updated-dependencies:
- dependency-name: indicatif
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump ordered-float from 3.3.0 to 3.4.0 (#1204)

Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.3.0...v3.4.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump clap from 4.0.18 to 4.0.22 (#1205)

Bumps [clap](https://github.com/clap-rs/clap) from 4.0.18 to 4.0.22.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.18...v4.0.22)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump num_cpus from 1.13.1 to 1.14.0 (#1203)

* wait for state deactivation on replica update failure (#1200)

* wait for state deactivation on replica update failure

* review fixes

* upd version to 0.11.2

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: erare-humanum <116254494+erare-humanum@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-11-09 14:03:49 +01:00
Arnaud Gourlay
c627e04c64 Update to v0.11.1 (#1191) 
* update to v0.11.1

* also default version

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-11-04 23:10:13 +01:00
Andrey Vasnetsov
144ecbdadd Complete feature road-map 2022-10-26 20:21:58 +02:00
Andrey Vasnetsov
194a19e0ad upd to v0.11.0 2022-10-26 10:40:02 +02:00
Andrey Vasnetsov
664d9bd93b implement concern-factor, forbig disabling the last node, improve tes… (#1168) 
* implement concern-factor, forbig disabling the last node, improve test stability

* upd docs

* improve test_recover_dead_node

* rename to write_consistency_factor

* Fix bug: applied index higher than commit on restart (#1172)

* create collection with default status = dead + await for activation on create_collecton_op_submit

* fmt

* fix unit tests

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
 2022-10-25 17:37:15 +02:00
Andrey Vasnetsov
65a76c6b61 update and improve openapi definitions 2022-10-24 16:03:28 +02:00
Andrey Vasnetsov
21754ef2ef report indexed payload points in info api + other fixes (#1164) 
* report indexed payload points in info api + other fixes

* rollback debug changes

* clippy

* clippy
 2022-10-24 09:43:02 +02:00
Ivan Pleshkov
516dcd7020 Telemetry level of detail (#1049) 
* telemetry level of detail

* rename duration aggregator

* are you happy fmt

* move total searches sum

* separate levels

* optional bucket size

* search telemetry improvements

* separate web telemetry into methods

* tonic telemetry methods

* merge optimizations

* are you happy fmt

* better rounding

* qdrant configs on level 1

* provide collection params

* add peers count

* collection points count

* update openapi

* use pattern in actix telemetry

* are you happy fmt

* merge dev

* are you happy fmt

* fix merge conflicts

* update openapi

* fix build

* are you happy fmt

* add exact searches statistics

* process replica set

* update openapi

* fix wrong name

* fix naming

* fix unwrap

* review

* fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-23 00:48:55 +02:00
Egor Ivkov
a96f92cbfe Record message send failures and stop consensus on service error (#1152) 
* Record message send failures and stop consensus on service error

* Generate openapi

* Review fixes: log first error and remove on success
 2022-10-22 23:23:37 +02:00
Ivan Pleshkov
44626bc80a exact search parameter (#1150) 
* exact search parameter

* exact search test

* decrease attempts count because of test speed

* minor review fixes

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-20 15:39:21 +04:00
Egor Ivkov
d3849d712b Allow peer force removal (#1139) 
* Allow peer force removal

* Remove shards
 2022-10-19 17:01:13 +02:00
Andrey Vasnetsov
7f8c088711 Replication transfer fixes (#1142) 
* fix validation and task assignment during replica transfer

* fmt

* upd openapi

* extend integration test
 2022-10-19 11:38:26 +02:00
Ivan Pleshkov
f50f1f4c71 #1119 disable updating feature (#1126) 
* disable updating feature

* customize error message

* update namings

* Update src/actix/api/service_api.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* fix build

* update api

* update api and allow deletion

* update openapi

* replace Bad Request with Forbidden

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-18 12:58:38 +02:00
Andrey Vasnetsov
cf29b67640 v0.10.5 2022-10-18 10:49:43 +02:00
Arnaud Gourlay
9324be1d60 [Replication] Add replicas (#1085) 
* Add replicas to ReplicaSet

* unproxify shard & miscs

* no exlusive write locking while adding replicas

* on_optimizer_config_update on Shard with async. recursion

* no exlusive write locking while removing replicas

* shortcut replica set propagation #1101

* remove unused field

* make RequestShardTransfer callback sync.

* promote local & remote to replica state

* fixes for replica sync api (#1123)

* code review

* fix replica set update - fail only if all failed

* Add replica redesign (#1131)

* refactor shard/mod.rs

* wip

* fmt

* it compiles

* temporary disable replica placemeant change on replication factor

* fmt

* finish todos

* small refactoring

* remove change::add

* replica-set -> shard-replica-set

* fmt

* upd openapi

* fix finish transfer logic

* fix existing integration tests

* shard transfer validation

* fmt

* review fixes

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-17 20:12:10 +02:00
Andrey Vasnetsov
411ddabe49 v0.10.4 2022-10-13 09:14:11 +02:00
Andrey Vasnetsov
68a46a0f2e v0.10.3 2022-10-13 09:14:11 +02:00
Andrey Vasnetsov
b275d452a8 shard replica manipulation API (#1121) 2022-10-12 15:19:43 +02:00
Egor Ivkov
ca01b8ac8e Fix: Do not rely on empty message for is_leader_established flag (#1103) 
* Fix: Do not rely on empty message for `is_leader_established` flag

* Record last consensus update

* Update OpenAPI
 2022-10-05 18:10:33 +03:00
Andrey Vasnetsov
935f069d53 upd version to v0.10.2 2022-09-30 11:08:16 +02:00