upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-05 01:20:53 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,206 Commits 349 Branches 118 Tags
b6b704955f9687d69ec00e499b6afdaba3cc89f0
Commit Graph

98 Commits

Author SHA1 Message Date
Andrey Vasnetsov
d7379f0706 Fix recover snapshot peer (#1534) 
* allow to recover distributed snapshot on local intance + allow recover non-existing collections

* async snapshot recovery

* fix tests

* be polite earlier
 2023-03-15 16:27:10 +01:00
Arnaud Gourlay
a93a55e900 Fix condition check for values_count (#1502) 2023-03-15 16:24:25 +01:00
Arnaud Gourlay
3d8b5131bd Nested payload filters (#1487) 
* Nested payload filters

* close ToDo + add parsing of multuiple array values

* fmt

* improve testing nested arrays

* fix NumericIndex to accumulate points_to_values mapping

* revert numberic index + strict array field access

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:24:04 +01:00
Luis Cossío
c414a77105 feat: add "any" to "match" statements (#1466) 
* feat: add "any" to "match" statements

Adds support for using "any" to match on a list of different possible values

* enable proper counting in cardinality estimation

* fmt

* upd OpenAPI

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:20:11 +01:00
Roman Titov
5149d39146 Add /cluster/recover API endpoint (#1451) 
- Add required internal `raft::raw_node::RawNode::report_snapshot` calls
- Add methods to propagate `raft::raw_node::RawNode::request_snapshot` call to the consensus thread
- Add `/cluster/recover` API endpooint
- Add `test_collection_recovery.py` integration test (disabled on CI; documents current, incorrect behavior)
 2023-03-15 16:19:59 +01:00
Andrey Vasnetsov
1687e71f74 remove /collections/aliases (#1440) 2023-02-07 08:51:32 +01:00
Roman Titov
34bdbc2fe6 Add read_consistency parameter to the APIs (#1371) (#1407) 
* WIP: Add `read_consistency` parameter to the APIs

* WIP: Add `read_consistency` parameter to the APIs

TODO:
- Add documentation

* `cargo fmt`

* Add gRPC documentation

* Add OpenAPI documentation

* Cleanup

* fixup! Add OpenAPI documentation

* fixup! Add gRPC documentation

Who would have known there's `generate_grpc_docs.sh`!? 🥲🙈🤦‍♀️

* generate openapi

* Fix `read_consistency` query parameter deserialization

* Further improve `read_consistency` query parameter deserialization

* `cargo clippy`

* Fix `Payload` comparison during read operation result resolving

* Fix grammar

* rename `read_consistency` -> `consistency` and add integration test

* use majority for test

* fix tests

* Fix tests

* fixup! Fix tests

Apply the same fix to `ScoredPoint`

* Remove an `unwrap`

* fixup! Fix tests

Gotta love those negative conditions, or how a missed `!` can ruin your day... 🤦‍♀️

* Make internal API calls strictly "local-shard only"

* Implement a few basic traits for `ResolverRecord`

* fixup! Implement a few basic traits for `ResolverRecord`

* Revert "Make internal API calls strictly "local-shard only""

This reverts commit 25378e61ff.

* Fix `Record::payload` and `ScoredPoint::payload` serialization

* Revert "Fix `Record::payload` and `ScoredPoint::payload` serialization"

This reverts commit b566bea49b.

* Fix `Record::payload` and `ScoredPoint::payload` visibility

* fixup! Fix `Record::payload` and `ScoredPoint::payload` visibility

Remove `todo!()`

* refactoring

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-02-06 15:21:43 +01:00
Andrey Vasnetsov
80af394d32 add write consistency param to rest and grpc api (#1397) 
* add write consistency param to rest and grpc api

* fix grpcurl + refactor grpc for better extencibility

* upd grpc docs

* fix conversion
 2023-02-06 15:18:58 +01:00
Arnaud Gourlay
fe06844c48 Delete snapshots API (#1377) 
* Delete snapshots API

* remove unecessary plurals
 2023-02-06 15:16:33 +01:00
Arnaud Gourlay
25d1383496 List aliases API (rest & gRPC) (#1363) 
* List aliases API (rest & gRPC)

* rework api to avoid conflicts
 2023-02-06 15:15:52 +01:00
Andrey Vasnetsov
a1f651fbd6 Recommendation with other collection (#1283) 
* WIP: move recommendation API out of collection

* recomendations with external source of vectors

* fix test

* fmt

* fixes + test

* review fixes
 2022-12-25 22:51:36 +01:00
dependabot[bot]
f6bfb78fdc Bump certifi from 2021.10.8 to 2022.12.7 in /openapi/tests (#1272) 
Bumps [certifi](https://github.com/certifi/python-certifi) from 2021.10.8 to 2022.12.7.
- [Release notes](https://github.com/certifi/python-certifi/releases)
- [Commits](https://github.com/certifi/python-certifi/compare/2021.10.08...2022.12.07)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-14 09:04:32 +01:00
Andrey Vasnetsov
469c098ca4 Update points by filter (#1249) 
* allow to set remove payload by selector

* fmt
 2022-12-06 15:02:33 +01:00
Andrey Vasnetsov
0d130c395a Full payload update (#1245) 
* implement api to fully overwrite payload of the point

* fmt
 2022-12-06 15:02:02 +01:00
Andrey Vasnetsov
ec25e973fb Recover distributed collection (#1214) 
* WIP: shards recovering API: download snapshot

* fmt

* snapshot recovery API

* fmt

* snapshot recovery integration test + fixes

* review fixes

* review fixes 2

* Update lib/storage/src/content_manager/snapshots/recover.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-11-15 13:33:22 +01:00
Egor Ivkov
d3849d712b Allow peer force removal (#1139) 
* Allow peer force removal

* Remove shards
 2022-10-19 17:01:13 +02:00
Andrey Vasnetsov
2b4ab27506 Write lock for user requests (#1143) 
* lock update operations only

* fmt

* fix

* lock openapi integration test
 2022-10-19 12:57:52 +02:00
Ivan Pleshkov
f50f1f4c71 #1119 disable updating feature (#1126) 
* disable updating feature

* customize error message

* update namings

* Update src/actix/api/service_api.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* fix build

* update api

* update api and allow deletion

* update openapi

* replace Bad Request with Forbidden

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-18 12:58:38 +02:00
Ivan Pleshkov
8eedcf3735 add empty security to openapi (#1079) 2022-09-30 10:02:40 +02:00
Andrey Vasnetsov
dc07b01e1f remove deprecated fields from API (#1030) 
* remove depricated fields from API

* fmt

* upd openapi and integration tests

* fix grpc test

* regenerate storage reference data

* improve docs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-19 13:51:03 +02:00
Ivan Pleshkov
4d3b4b132d More multivector api updates (#1025) 
* rest multivector api updates

* update openapi

* update grpc doc
 2022-09-16 12:25:42 +04:00
Ivan Pleshkov
22fd3a3963 minor doc and api in tests updates (#1022) 2022-09-15 13:37:38 +04:00
Ivan Pleshkov
f6b2186193 [WIP] Many named vectors per point (#958) 
* many named vectors per point (segment-level)

* operation result for dim function

* beautifulized vector name

* fix naming bug

* segment version migration

* fmt

* add segment tests

* are you happy clippy

* fix build

* [WIP] many named vectors per point (collection-level) (#975)

* config and search

* fix placeholders for proxy segment move

* remove VectorType from collection

* are you happy fmt

* vectors in grps messages

* create collections with vectors

* segment holder fixes

* are you happy fmt

* remove default vector name placeholders

* are you happy fmt

* are you happy clippy

* fix build

* fix web api

* are you happy clippy

* are you happy fmt

* record vector&vectors

* openapi update

* fix openapi integration tests

* segment builder fix todo

* vector names for update from segment

* remove unwrap

* backward compatibility

* upd openapi

* backward compatible PointStruct

* upd openapi

* fix record back-comp

* fmt

* vector configuration backward compatibility

* fix vetor storage size estimation

* fmt

* multi-vec segment test + index test

* fmt

* api integration tests

* [WIP] Named vectors struct (#1002)

* move to separate file

* named vectors as struct

* use cow

* fix build

* keys iterator

* avoid copy in PointStruct -> get_vectors

* avoid another copy

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-09-11 20:59:23 +02:00
Andrey Vasnetsov
b9eee55a9f Full text search (#963) 
* allow additional params for payload field index

* fmt

* wip: full text index building

* fmt

* text search request

* text search request

* full text index persitance and loading

* fmt

* enable fts index in mapping

* clippy

* fix tests + add integration test

* review fixes: extend payload index test

* revert incedental change
 2022-09-01 12:50:12 +02:00
Arnaud Gourlay
03c10fc310 Batch recommendation API (#954) 2022-08-22 09:41:22 +02:00
Andrey Vasnetsov
a2acca0345 Segment batch search (#813) 
* batch search benchmark

* collect filter iterator in indexed search

* fmt

* fix

* fix

* fmt

* use new tempfile create

* auto batching

* Clippy fixes

* REST, gRPC and internal APIs

* fix bugs & less duplication

* two steps payload retrieval mechanism & fix duplication

* add proxy_segment implementation & tests

* add gRPC docs

* remove unused code (#950)

* only filter ids within a batch

* add more equivalence tests

* add integration test search vs batch

* assert more search options in tests

* cleanup assertions

* fix offset panic

* rename search batch API

* openapi spec

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-08-18 14:48:17 +02:00
Egor Ivkov
0a634453e3 Peer removal API (#894) 
* Remove peer api

* Keep the same cluster status api for compatibility
 2022-08-02 15:56:41 +03:00
Arnaud Gourlay
320b220b01 Move shard API (#869) 
* Move shard API

* add missing collection_name to openapi spec

* add basic shard transfer info to collection cluster API

* improve testing
 2022-07-28 20:14:15 +02:00
Ivan Pleshkov
42e930ab8f Segment telemetry (#814) 
* segment telemetry

* anonymize trait

* fix build

* are you happy fmt

* anonimyze implementations

* sliding window avg (#826)

* Actix web telemetry (#828)

* actix web telemetry

* small as move

* use tokio mutex instead of std

* add comments

* are you happy fmt

* use u16 as http status code

* telemetry structs rename

* fix build

* using parking lot mutex

* telemetry web api (#842)

* telemetry web api

* telemetry openapi (#843)

* use async mutex for telemetry collector

* use tokio mutex for telemetry collector

* are you happy fmt
 2022-07-22 19:27:07 +04:00
Andrey Vasnetsov
89da942cc8 upd tags in openapi 2022-07-18 18:12:40 +02:00
Arnaud Gourlay
9b8b5b6376 [sharding] Collection cluster info API (#822) 
* Collection cluster info API

* better output format

* update openapi spec

* local shard info reports number of points

* rebase
 2022-07-18 17:57:34 +02:00
Andrey Vasnetsov
123332c867 Full snapshot (#824) 
* full snapshot rest api

* openapi for snapshots

* openapi snapshot api

* fmt + clippy

* fix recovery + implement grpc methods

* fmt

* same aliases to full storage snapshot

* fmt
 2022-07-18 17:11:23 +02:00
Andrey Vasnetsov
098bc4c751 Count api (#777) 
* count api

* test for approx counting

* fmt + clippy

* unit test

* add warning
 2022-07-05 09:33:38 +02:00
Andrey Vasnetsov
fac87018c4 Snapshoting API (#764) 
* wip: rest api for snapshots

* fmt

* fix tests

* fmt

* wip: collection snapshoting and recovery

* fmt

* remote shard snapshots

* remote shard test

* fmt

* extend proxy snapshot test + fix double read lock

* fmt + clippy

* openapi schema

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* reduce writes on snapshots location

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-07-01 13:21:57 +02:00
Andrey Vasnetsov
83c21450c3 Pagination (#743) 
* offset param for search and recomendation

* test and fix for pagination

* clippy + fmt

* upd schemas

* fix grpc requests in test
 2022-06-27 13:22:02 +02:00
Andrey Vasnetsov
ddf940c491 allow server selection in OpenAPI schema (#682) 2022-06-14 09:54:47 +02:00
Gabriel Velo
c15981092a [WIP] [real-time index] Implement payloadstorage for structpayloadindex (#642) 
* [real-time index] Extend FieldIndex enum and StructPayloadIndex with method from PayloadStorage

* [real-time index] add missing remove_point methods

* [real-time index] add new index to FieldIndex enum

* fix compile

* are you happy fmt

* merge load and remove

* fix test generics

* decrement points count

* remove from histogram

* simplify histogram usage

* [real-time index] remove old tests and fix clippy warnings

* histogram: method to derive range by size (#657)

* [real-time index] add histogram based payload_blocks implementation.

* payload blocks

* fmt

* clippy

* [real-time index] refactor Segment to use PayloadIndex instead of PayloadStorage.

* fix tests

* fmt

* clippy

* rename indexes

* remove redundent params

* add struct payload deletion test + fix delete payload in map index

* remove payload threshold

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-06-06 17:14:20 +02:00
Andrey Vasnetsov
1b458780eb On disk payload storage (#634) 
* implement on-disk payload storage

* fmt + clippy

* config param for on-disk payload storage

* upd openapi definitions

* add integration test with on-disk payload

* fix clippy

* review fixes

* fmt
 2022-06-01 17:23:34 +02:00
Andrey Vasnetsov
c286bde562 Cluster api (#631) 
* cluster status API

* add cluster api to OpenAPI specification
 2022-05-27 14:52:51 +02:00
Andrey Vasnetsov
036c186e0d Better error reporting in enums (#587) 
* remove conflicting deprecated apis

* implement custom JsonSchema for the PointInsertOperations

* fmt

* clippy

* revert unnesessare changes
 2022-05-23 13:28:17 +02:00
Andrey Vasnetsov
5800319edb refactor distances + add score_threshold + fix negative euclid distance (#569) 
* refactor distances + add score_threshold + fix negative euclid distance

* generate docs

* fix clippy

* Update lib/segment/src/spaces/tools.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-05-09 17:32:48 +02:00
Andrey Vasnetsov
6af867e037 fix and extend interation test for scrolling over points (#494) 2022-04-20 09:52:59 +02:00
Egor Ivkov
ee4cd768ea Allows specifying collection meta operation commit timeout in API (#444) 
* Allows specifying collection meta operation commit timeout in API

Also restructures a part of gRPC collection API

* Review: link to dev docs

* Review: Timeout error description

Also committed generated protobuf code that somehow was left out

* Review: seconds in API and minor proto fixes
 2022-04-08 14:42:38 +03:00
Andrey Vasnetsov
b07428f620 Is empty condition (#423) 
* is-empty condition

* fmt

* better assert

* fmt
 2022-04-03 16:55:51 +02:00
Andrey Vasnetsov
adc1f4ad97 Bool filter (#421) 
* bool match condition

* use generic values for match requests

* fmt

* upd grpc interface

* upd grpc docs
 2022-04-03 16:08:34 +02:00
Andrey Vasnetsov
6fc8d0affe upd openAPI schema names (#416) 2022-03-29 14:12:52 +02:00
Andrey Vasnetsov
8dcb900113 fix intergation test: freeze python deps on working versions 2022-03-28 23:56:08 +02:00
Arnaud Gourlay
2a81fc9142 Remove sled for alias mappings (#402) 
* improve alias tests

* introduce new alias persistence and remove Sled

* introduce FileStorageError

* use new error in AliasMapping

* aliases are kept in memory and saved to disk when modified

* better naming

* make alias renaming atomic

* fmt

* use correct alias and docs

* code review: cleaner error conversion
 2022-03-23 14:21:04 +01:00
Arnaud Gourlay
cfed400240 gitignore __pycache__/ (#401) 2022-03-21 17:39:40 +01:00
Gabriel Velo
f69a7b740f json as payload (#306) 
add json as payload
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-03-21 07:09:10 -03:00