upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-07 02:20:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,116 Commits 346 Branches 118 Tags
c3e88a33e8ed2a429cb48973cfedbfef387147cd
Commit Graph

247 Commits

Author SHA1 Message Date
Andrey Vasnetsov
8cd4fe15ed Init collection from (#1364) 
* WIP

* WIP: add shards selection

* fmt

* WIP: collection init from other collection

* fmt

* fix bugs

* add collection write lock + use collection manager runtime instead of search runtime

* fmt

* integration test

* better names ?

* payload indexes transfer
 2023-02-06 15:17:01 +01:00
Arnaud Gourlay
fe06844c48 Delete snapshots API (#1377) 
* Delete snapshots API

* remove unecessary plurals
 2023-02-06 15:16:33 +01:00
Arnaud Gourlay
25d1383496 List aliases API (rest & gRPC) (#1363) 
* List aliases API (rest & gRPC)

* rework api to avoid conflicts
 2023-02-06 15:15:52 +01:00
Arnaud Gourlay
6669f950b9 Stop on fail consensus recovery (#1347) 2023-02-06 15:15:16 +01:00
Arnaud Gourlay
95ba6d27d7 Retry recovering from known peers (#1338) 2023-01-13 12:14:53 +01:00
Arnaud Gourlay
5525897d05 Add logging to node recovery (#1337) 2023-01-13 12:14:47 +01:00
Hozan
5c4a133451 Add the option to load alternative configuration file (#1328) 
* add the option to load alternative configuration file

* minor fix & pass cargo clippy warning
 2023-01-13 12:14:19 +01:00
Roman Titov
3ef66f1e96 Add configuration option to select "memory advice" used for memmapped storage 
- Add a simple abstraction/wrapper over `memmap2::Advice` and a global switch
  to select "memory advice" used for memmapped storage to the `segment` crate
- Add `mmap_advice` configuration option to the `storage::types::StorageConfig`
- Implement `search-points` benchmark
 2023-01-13 12:13:34 +01:00
Andrey Vasnetsov
ac57478ba4 Fixing missed leader inconsistency (#1286) 
* WIP: introduce local state

* WIP: sync local state with consensus in idle

* fmt

* rm extensive debug

* update triple replication test

* update triple replication test

* rm debug logs

* fix check for established leader + only sync local if no proposals

* remove unused file

* test compatible with python 3.8

* test compatible with python 3.8

* longer wait for consensus

* longer wait for consensus

* extra sleep in test

* remove extra sleep

* Fixing missed leader inconsistency - transfers (#1298)

* explicit request timeout

* explicit request timeout

* explicit request timeout

* explicit request timeout

* explicit request timeout

* prevent double handelling of the transfer termination

* kill the process

* log on inconsistency

* log on inconsistency

* log on inconsistency

* debug

* revert debug in test

* forward updates to partial shards, abort transactions on dead node report

* fmt

* disable retry of transfer, if the transfer was cancelled + allow predictable ports in test

* fix import
 2022-12-25 22:51:47 +01:00
Andrey Vasnetsov
a1f651fbd6 Recommendation with other collection (#1283) 
* WIP: move recommendation API out of collection

* recomendations with external source of vectors

* fix test

* fmt

* fixes + test

* review fixes
 2022-12-25 22:51:36 +01:00
Arnaud Gourlay
f9ffa3c797 update tonic 0.8 & prost 0.11 (#1285) 
* update tonic 0.8 & prost 0.11

* install protoc via package manager
 2022-12-25 22:47:44 +01:00
Arnaud Gourlay
9702054127 Clippy for Rust 1.66 (#1284) 2022-12-25 22:46:53 +01:00
Arnaud Gourlay
9ff03ca5d5 do not block grpc async threads (#1252) 2022-12-06 15:02:55 +01:00
Andrey Vasnetsov
469c098ca4 Update points by filter (#1249) 
* allow to set remove payload by selector

* fmt
 2022-12-06 15:02:33 +01:00
Andrey Vasnetsov
0d130c395a Full payload update (#1245) 
* implement api to fully overwrite payload of the point

* fmt
 2022-12-06 15:02:02 +01:00
Andrey Vasnetsov
16d3bba233 Fix consensus freezes (#1241) 
* do not await for replica deactivation if there is a logical error

* use async mutex in raft service + integration test for deadlock

* fmt
 2022-11-25 12:17:34 +01:00
Andrey Vasnetsov
6405bbb074 Extend telemetry (#1230) 
* WIP: extend telemetry api for faster cluster manager

* fmt

* clippy fix
 2022-11-25 12:16:23 +01:00
Arnaud Gourlay
e83bfef8b2 enable compression for gRPC (#1186) 
* enable compression for gRPC

* add benchmarks

* fix grpc benchmark

* discardResponseBodies to reduce memory usage

* improve & split benchmarks

* do not use gzip for inter node communication as benchmarks are not conclusive
 2022-11-25 12:15:26 +01:00
Andrey Vasnetsov
ec25e973fb Recover distributed collection (#1214) 
* WIP: shards recovering API: download snapshot

* fmt

* snapshot recovery API

* fmt

* snapshot recovery integration test + fixes

* review fixes

* review fixes 2

* Update lib/storage/src/content_manager/snapshots/recover.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-11-15 13:33:22 +01:00
Andrey Vasnetsov
e027db4d27 V0.11.2 (#1209) 
* Update: unmaintained crate memmap -> memmap2 (#559) (#1160)

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* Consensus q n a (#1169)

* Expand comments and minor refactoring for raft state

* fmt

* add comments to consensus.rs

* fix "Consensus q n a" compatibility

* Use less ram for id tracker (#1176)

* use less ram for id tracker

* are you happy clippy

* use vec for internals

* use versions for internal ids

* keys test

* Use less ram for id tracker fixes (#1182)

* WIP: internal_to_version

* fmt

* fix unit tests

* add comment

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* remove suggesting changes in replications on replication factor change (#1177)

* Bump actix-cors from 0.6.3 to 0.6.4 (#1185)

Bumps [actix-cors](https://github.com/actix/actix-extras) from 0.6.3 to 0.6.4.
- [Release notes](https://github.com/actix/actix-extras/releases)
- [Commits](https://github.com/actix/actix-extras/compare/cors-v0.6.3...cors-v0.6.4)

---
updated-dependencies:
- dependency-name: actix-cors
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* enable HTTP compression middleware (#1184)

* Use systematically assert_http_ok for better error reporting (#1183)

* Use systematically assert_http_ok for better error reporting

* extraction assertion to use it outside of pytest

* Bump pprof from 0.10.1 to 0.11.0 (#1188)

Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.10.1 to 0.11.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases)
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Cosine dist zero vectors (#1198)

* skip pre-processing of zero-length vector for cosine distance

* fmt

* Bump env_logger from 0.9.1 to 0.9.3 (#1201)

Bumps [env_logger](https://github.com/env-logger-rs/env_logger) from 0.9.1 to 0.9.3.
- [Release notes](https://github.com/env-logger-rs/env_logger/releases)
- [Changelog](https://github.com/env-logger-rs/env_logger/blob/main/CHANGELOG.md)
- [Commits](https://github.com/env-logger-rs/env_logger/compare/v0.9.1...v0.9.3)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump indicatif from 0.17.1 to 0.17.2 (#1202)

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.17.1 to 0.17.2.
- [Release notes](https://github.com/console-rs/indicatif/releases)
- [Commits](https://github.com/console-rs/indicatif/compare/0.17.1...0.17.2)

---
updated-dependencies:
- dependency-name: indicatif
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump ordered-float from 3.3.0 to 3.4.0 (#1204)

Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.3.0...v3.4.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump clap from 4.0.18 to 4.0.22 (#1205)

Bumps [clap](https://github.com/clap-rs/clap) from 4.0.18 to 4.0.22.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.18...v4.0.22)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump num_cpus from 1.13.1 to 1.14.0 (#1203)

* wait for state deactivation on replica update failure (#1200)

* wait for state deactivation on replica update failure

* review fixes

* upd version to 0.11.2

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: erare-humanum <116254494+erare-humanum@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-11-09 14:03:49 +01:00
Andrey Vasnetsov
63b59c1044 fix colletion params update (#1208) 2022-11-09 12:58:06 +01:00
Andrey Vasnetsov
8b0b282079 handle collections transition from single-node to cluster (#1189) 2022-11-04 12:52:51 +01:00
Andrey Vasnetsov
38bc2eb0f2 update rust version and fix clippy complains (#1190) 2022-11-04 11:53:28 +01:00
Andrey Vasnetsov
664d9bd93b implement concern-factor, forbig disabling the last node, improve tes… (#1168) 
* implement concern-factor, forbig disabling the last node, improve test stability

* upd docs

* improve test_recover_dead_node

* rename to write_consistency_factor

* Fix bug: applied index higher than commit on restart (#1172)

* create collection with default status = dead + await for activation on create_collecton_op_submit

* fmt

* fix unit tests

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
 2022-10-25 17:37:15 +02:00
Ivan Pleshkov
516dcd7020 Telemetry level of detail (#1049) 
* telemetry level of detail

* rename duration aggregator

* are you happy fmt

* move total searches sum

* separate levels

* optional bucket size

* search telemetry improvements

* separate web telemetry into methods

* tonic telemetry methods

* merge optimizations

* are you happy fmt

* better rounding

* qdrant configs on level 1

* provide collection params

* add peers count

* collection points count

* update openapi

* use pattern in actix telemetry

* are you happy fmt

* merge dev

* are you happy fmt

* fix merge conflicts

* update openapi

* fix build

* are you happy fmt

* add exact searches statistics

* process replica set

* update openapi

* fix wrong name

* fix naming

* fix unwrap

* review

* fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-23 00:48:55 +02:00
Andrey Vasnetsov
387b52d7fa propagate peer url to consensus if specified uri is does not match a … (#1156) 
* propagate peer url to consensus if specified uri is does not match a stored one + tests

* review fixes
 2022-10-23 00:07:39 +02:00
Egor Ivkov
a96f92cbfe Record message send failures and stop consensus on service error (#1152) 
* Record message send failures and stop consensus on service error

* Generate openapi

* Review fixes: log first error and remove on success
 2022-10-22 23:23:37 +02:00
Andrey Vasnetsov
658b185b24 exclude / from actix logger middleware (#1157) 2022-10-21 09:28:22 +02:00
Andrey Vasnetsov
75563b20b4 Consensus fault tollerance improvements (#1149) 
* consensus fault tollerance improvements

* fmt
 2022-10-20 11:22:22 +02:00
Andrey Vasnetsov
123ae1607d Await for collection created for sure (#1144) 
* run plunger operation after user request to ensure that the majority of the consensus actually applied previous operation

* fmt

* fix test

* ensure all operations are unique in consensus queue

* add extra timeout for CI tests

* fix timout param in delete peer + do not explicilty await on peer removal

* fmt
 2022-10-19 17:59:16 +02:00
Egor Ivkov
d3849d712b Allow peer force removal (#1139) 
* Allow peer force removal

* Remove shards
 2022-10-19 17:01:13 +02:00
Andrey Vasnetsov
7f8c088711 Replication transfer fixes (#1142) 
* fix validation and task assignment during replica transfer

* fmt

* upd openapi

* extend integration test
 2022-10-19 11:38:26 +02:00
Ivan Pleshkov
f50f1f4c71 #1119 disable updating feature (#1126) 
* disable updating feature

* customize error message

* update namings

* Update src/actix/api/service_api.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* fix build

* update api

* update api and allow deletion

* update openapi

* replace Bad Request with Forbidden

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-18 12:58:38 +02:00
Arnaud Gourlay
9324be1d60 [Replication] Add replicas (#1085) 
* Add replicas to ReplicaSet

* unproxify shard & miscs

* no exlusive write locking while adding replicas

* on_optimizer_config_update on Shard with async. recursion

* no exlusive write locking while removing replicas

* shortcut replica set propagation #1101

* remove unused field

* make RequestShardTransfer callback sync.

* promote local & remote to replica state

* fixes for replica sync api (#1123)

* code review

* fix replica set update - fail only if all failed

* Add replica redesign (#1131)

* refactor shard/mod.rs

* wip

* fmt

* it compiles

* temporary disable replica placemeant change on replication factor

* fmt

* finish todos

* small refactoring

* remove change::add

* replica-set -> shard-replica-set

* fmt

* upd openapi

* fix finish transfer logic

* fix existing integration tests

* shard transfer validation

* fmt

* review fixes

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-17 20:12:10 +02:00
Andrey Vasnetsov
b275d452a8 shard replica manipulation API (#1121) 2022-10-12 15:19:43 +02:00
Egor Ivkov
03ec804bef Adds panic hook (#1106) 2022-10-07 19:17:08 +03:00
Egor Ivkov
ca01b8ac8e Fix: Do not rely on empty message for is_leader_established flag (#1103) 
* Fix: Do not rely on empty message for `is_leader_established` flag

* Record last consensus update

* Update OpenAPI
 2022-10-05 18:10:33 +03:00
Egor Ivkov
db39c07d05 Add sync and sync_source for transfers (#1073) 
* Add `sync` and `sync_source` for transfers
 2022-09-29 18:06:47 +03:00
dependabot[bot]
a662ec430a Bump clap from 3.2.22 to 4.0.0 (#1072) 
* Bump clap from 3.2.22 to 4.0.0

Bumps [clap](https://github.com/clap-rs/clap) from 3.2.22 to 4.0.0.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.22...clap_complete-v4.0.0)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix clap 4.x

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-29 15:05:49 +02:00
Egor Ivkov
c6c0afa781 [replication] Suggest replica set changes on repl_factor change (#1059) 
* Suggest replica set changes on repl_factor change

* Review fix: account for replication_factor being 1

* Update lib/collection/src/collection.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update lib/storage/src/content_manager/toc.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Review fix: accept higher repl factor than n peers

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-28 17:05:11 +03:00
Egor Ivkov
9a3fad44b2 Revert "Revert replication changes (#1055)" (#1056) 
This reverts commit a2ed0cf5db.
 2022-09-23 16:46:06 +03:00
Egor Ivkov
a2ed0cf5db Revert replication changes (#1055) 
* Revert "Replication set `Change` enum"

This reverts commit 28e8487542.

* Revert changes (#1053)

* Revert "Generate API - replication follow up (#1043)"

This reverts commit 183ea9e5e3.

* Revert "Replication related API changes (#1037)"

This reverts commit d690b700ff.

* Revert "Persist replica state (#1035)"

This reverts commit 31c3ae482d.

* Revert "[replication] Account for replicas in shard distribution (#1009)"

This reverts commit e332d482b0.

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-09-23 16:33:07 +03:00
Egor Ivkov
d690b700ff Replication related API changes (#1037) 
* Replication related API changes

* minor proto file fix
 2022-09-21 18:49:47 +03:00
Andrey Vasnetsov
dc07b01e1f remove deprecated fields from API (#1030) 
* remove depricated fields from API

* fmt

* upd openapi and integration tests

* fix grpc test

* regenerate storage reference data

* improve docs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-19 13:51:03 +02:00
Andrey Vasnetsov
94883e7062 allow empty with_vector 2022-09-16 01:37:34 +02:00
Andrey Vasnetsov
278b3def9b use structure for better openapi generator compatibility (#1020) 2022-09-15 10:51:46 +02:00
Ivan Pleshkov
f6b2186193 [WIP] Many named vectors per point (#958) 
* many named vectors per point (segment-level)

* operation result for dim function

* beautifulized vector name

* fix naming bug

* segment version migration

* fmt

* add segment tests

* are you happy clippy

* fix build

* [WIP] many named vectors per point (collection-level) (#975)

* config and search

* fix placeholders for proxy segment move

* remove VectorType from collection

* are you happy fmt

* vectors in grps messages

* create collections with vectors

* segment holder fixes

* are you happy fmt

* remove default vector name placeholders

* are you happy fmt

* are you happy clippy

* fix build

* fix web api

* are you happy clippy

* are you happy fmt

* record vector&vectors

* openapi update

* fix openapi integration tests

* segment builder fix todo

* vector names for update from segment

* remove unwrap

* backward compatibility

* upd openapi

* backward compatible PointStruct

* upd openapi

* fix record back-comp

* fmt

* vector configuration backward compatibility

* fix vetor storage size estimation

* fmt

* multi-vec segment test + index test

* fmt

* api integration tests

* [WIP] Named vectors struct (#1002)

* move to separate file

* named vectors as struct

* use cow

* fix build

* keys iterator

* avoid copy in PointStruct -> get_vectors

* avoid another copy

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-09-11 20:59:23 +02:00
Ivan Pleshkov
c32375a6ae fix telemetry web init (#990) 2022-09-06 14:25:37 +04:00
Andrey Vasnetsov
24ac939b4a Sync Points API (#985) 
* shard sync operation

* fnt

* tests

* fix test

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* match payload after vector only

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-05 15:09:39 +02:00
Andrey Vasnetsov
b9eee55a9f Full text search (#963) 
* allow additional params for payload field index

* fmt

* wip: full text index building

* fmt

* text search request

* text search request

* full text index persitance and loading

* fmt

* enable fts index in mapping

* clippy

* fix tests + add integration test

* review fixes: extend payload index test

* revert incedental change
 2022-09-01 12:50:12 +02:00