Commit Graph

369 Commits

Author SHA1 Message Date
Andrey Vasnetsov
cbd33324ce Revert "revert skip-serializing (#4527)" (#4529)
This reverts commit cb68dfc2f2.
2024-06-27 15:23:43 +02:00
Ivan Pleshkov
5584ed0435 Fix colberts TODOs (#4566)
* Fix colberts TODOs

* are you happy clippy
2024-06-27 10:50:02 +02:00
Tim Visée
1474f4844f Fix two documentation mistakes in our gRPC definitions (#4567) 2024-06-26 14:27:43 +02:00
Tim Visée
1c38b0849d Bump dev version to 1.9.8-dev (#4559) 2024-06-26 08:56:50 +02:00
generall
371471eeac fix openapi generation for Fusion 2024-06-24 22:38:32 +02:00
Arnaud Gourlay
7c4809e1d3 universal-query: Add validations to query input (#4536)
* universal-query: Add validations to query input

* update openapi spec

* use raw request to by-pass local validations

* add validation for order_by and factorize
2024-06-24 16:56:36 +02:00
Luis Cossío
46b9ea44ec order_by: Begin migration to order_value in Record [v1.10] (#4526)
* begin migration to order_value

* gen openapi and grpc docs

* cargo clippy --fix

* fixup
2024-06-24 14:25:50 +02:00
Andrey Vasnetsov
d4807dcc8b Api consistency update (#4533)
* rename search_params -> params

* rename multivector_config + generate schema

* upd tests
2024-06-23 23:56:42 +02:00
Luis Cossío
cb68dfc2f2 revert skip-serializing (#4527) 2024-06-21 20:01:20 +02:00
Arnaud Gourlay
9e8609afc5 Fixing various API docs (#4520)
* Fixing various API docs

* more fixes
2024-06-21 18:24:12 +02:00
Arnaud Gourlay
1d6f3549aa universal-query: gRPC batch API (#4513)
* universal-query: gRPC batch API

* regen gRPC docs

* enable jwt access test
2024-06-20 14:10:27 +02:00
Tim Visée
7ede41c778 Fix the field indices in the abort shard transfer message (#4506) 2024-06-19 15:36:59 +02:00
Andrey Vasnetsov
e5facdba19 fix: replace vector_count with dim (#4505) 2024-06-19 15:18:48 +02:00
Roman Titov
d4b32012eb Rework CommitHashRing consensus message into CommitRead/CommitWrite/Finish (#4417)
* Refactor `CommitHashRing` into `CommitRead`/`CommitWrite`/`Finish`

* Add `Resharding` filter condition

* Filter "resharded" points from search, scroll by, count and retrieve request results

* fixup! Refactor `CommitHashRing` into `CommitRead`/`CommitWrite`/`Finish`

`cargo clippy --fix`

* Apply suggestions from code review

* fixup! Filter "resharded" points from search, scroll by, count and retrieve request results

Add `Condition::is_local_only` method

* fixup! Add `Resharding` filter condition

* fixup! Filter "resharded" points from search, scroll by, count and retrieve request results

Clarified a few `TODO`s

* Fix clippy suggestions

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
2024-06-19 12:14:26 +02:00
Luis Cossío
d107aee6b7 Add layer of response structure specific to query (#4498) 2024-06-19 08:31:39 +02:00
Luis Cossío
f6d6b7a713 universal-query: Expose batch query in REST (#4497)
Exposes the ability to query in batch
2024-06-18 18:47:08 -04:00
Arnaud Gourlay
d4b8e482ff universal-query: gRPC query API (#4495)
* universal-query: gRPC query API

* misc fixes

* more review

* add custom validation on QueryPoints.limit

* fix jwt_access test for gRPC query
2024-06-18 21:14:30 +02:00
Andrey Vasnetsov
1878713679 Fix multivector for unnamed vectors (#4482)
* minor conversion improvement

* use NamedVectors in update_vectors

* remove merge from VectorStruct

* rename Multi -> Named in vector struct

* add multi-dense vectors option into VectorStruct

* generate openapi

* rename VectorStruct -> VectorStructInternal

* add conversion for anonymous multivec in grpc

* renames for BatchVectorStruct

* implement multi-dense for batch

* allow multi-dense in batch upserts

* test and fixes
2024-06-18 20:38:24 +02:00
Tim Visée
3255719e27 Properly integrate replica set state switching in resharding transfers (#4480)
* Add serde/validate attributes to resharding operations, matching others

* Fix broken comment

* Add debug message for resharding driver entering stages

* Fix shard transfer start setting state of wrong replica for resharding

* Remove obsolete clones

* Add target shard ID to shard key, add relevant gRPC types

* Move target shard ID below source shard ID field

* Rename collection_name to collection_id

* Reformat

* Transferring point batches must merge points in case of resharding

* In resharding state, sync list of peers on start

* Add logic for setting replica set state through consensus dispatcher

* Properly start resharding transfer

* Properly finish resharding transfers, set shard state correctly

* Fix shard transfer initialisation with different target shard

* Fix shard state handling with resharding on all nodes on transfer start

* Don't reset locally disabled state if only existing shard is resharding

* Add important TODOs

* Update OpenAPI and gRPC specification

* Elaborate on some logic in code with comments

* Use user configured shard transfer method for replication

* Add debug assert, on transfer start we should not replace existing shard

* On transfer start, be aware of different sender/receiver local states

This fixes transfers where we might not have a replica on all nodes

* Fix shard transfer not setting cutoff point on target shard

* While resharding, migrate shards in numerical order

* On shard transfer initialisation, ensure transfer targets given shard
2024-06-18 17:51:13 +02:00
Luis Cossío
9bc1fd7563 universal-query: use batch implementation everywhere (#4492)
* use batch version everywhere, get rid of single query version

* extra renames
2024-06-18 11:39:06 -04:00
Andrey Vasnetsov
54fa9e2870 fix generation of query types in openapi (#4494) 2024-06-18 10:55:50 -04:00
xzfc
53e883ea48 Add datatype option for sparse vector index (#4454)
* Add SparseVectorIndexDatatype

* Update tests

* Review fixes

* review

* gen openAPI

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-06-18 13:32:06 +00:00
Luis Cossío
8839711686 Skip serializing null fields in Record and ScoredPoint (REST) (#4394)
* skip serializing null fields in Record and ScoredPoint

* update tests

* fix consensus tests
2024-06-18 09:27:19 -04:00
Arnaud Gourlay
1cafdacc9e universal-query: add lookups to query API definition (#4479)
* universal-query: add with_lookup to query API definition

* fix consistency doc check

* lookup_from and not lookup_with

* clean import

* improve doc for lookup location wrt using

* Suggestions for descriptions

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2024-06-18 07:54:59 +02:00
Luis Cossío
ec663de5eb Merge pull request #4473
* add batch query handling for internal service

* cargo clippy --fix
2024-06-16 16:49:17 +02:00
Arnaud Gourlay
8cd131d5fc universal-query: timeout support (#4449) 2024-06-12 15:16:26 +02:00
Tim Visée
b0b21fc0a8 Bump dev version to 1.9.6-dev (#4445) 2024-06-11 16:29:21 +02:00
Luis Cossío
8dc1eb235b universal-query: some interface renaming/edits (#4406)
in grpc:
- restructure `ContextInput`, so that it becomes easy to handle both pairs and positive/negative lists in the future

in rest: 
- restructure context queries so that they don't repeat the label in order to be used . E.g. turn `{ query: { context: { context: [ ... ] } } }` into `{ query: { context: [ ... ] } }` 
- fixes fusion so that `{query: { fusion: "rrf" } }` actually works

both:
- renames `RecommendInput` positives and negatives to singular (same as in reco api)
2024-06-10 18:47:40 -04:00
Luis Cossío
6d0d542356 universal-query: Expose REST endpoint (#4380)
Exposes `POST collections/{collection_name}/points/query` for single queries
2024-06-10 11:55:46 -04:00
Luis Cossío
d35319c663 universal-query: validate REST query request (#4388)
Implements `Validate` for REST-facing `QueryRequest`
2024-06-10 11:27:28 -04:00
Tim Visée
81cd8f312b Bump dev version to 1.9.5-dev (#4345) 2024-05-28 16:29:37 +02:00
Luis Cossío
c8e82e75a8 universal-query: Converge API types into CollectionQueryRequest (#4304)
* rest -> collection

* grpc -> collection

* fix rebase

* use constants for defaults
2024-05-27 13:28:08 -04:00
Luis Cossío
b14d0b2463 universal-query: user-facing gRPC and REST request types (#4303)
* add grpc `QueryPoints`

* add rest types

* move to REST types to `api::rest::schema`, decouple `OrderByInterface` and `RecommendStrategy`

* openapi changes

* add score_threshold param

* decouple `ShardKeySelector`, add shard key param

* remove commented-out code

* new codespell
2024-05-24 08:23:12 -04:00
Arnaud Gourlay
cbcdefb310 Cleanup unused dependencies (#4306) 2024-05-23 10:06:44 +02:00
Luis Cossío
2a82ef41a0 universal-query: Introduce order_value field in ScoredPoint (#4291)
* - Fix ordering in group-by for custom scoring
- Rename `OrderingValue` -> `OrderedValue`
- Introduce `order_value` field in `ScoredPoint`

* modify `Ord` implementation

* rename to `OrderValue`

* `has_custom_scoring` -> `is_distance_scored`

* regen apis

* flip bools in `is_distance_scored`
2024-05-22 09:56:31 -04:00
Andrey Vasnetsov
24df71785c [universal-search] refactoring suggestions (#4262)
* rename some structures

* use enum for fusion
2024-05-18 19:09:36 +02:00
Luis Cossío
f3f8c3bc9e Merge pull request #4261
* add `OrderBy` scoring, and add `scrolls` to `PlannedQuery`
2024-05-18 13:07:12 +02:00
Luis Cossío
7dcdae04c7 universal-query: Test and fix PlannedQuery conversion (#4242)
* test and fix PlannedQuery conversion

* add `ScoringQuery::RRF`

* add more test cases, fix more
2024-05-16 10:33:00 -04:00
Luis Cossío
4fd5df9f60 universal-query: Connect grpc internal query service (#4223)
* add grpc -> internal conversions

* return `Status` instead of `CollectionError` in grpc -> internal conversions

* sketch points internal api connection of `query`
2024-05-15 11:53:33 -04:00
Luis Cossío
bfa72bb6d8 universal-query: Add query() to ShardOperation trait (#4210)
* add `query` to shard trait

* add missing conversions for query

* update grpc docs

* Query response has intermediate results

* add ShardQueryResponse description

* move pub use to the top, keep only one way of reaching reexports
2024-05-15 09:36:46 -04:00
Ivan Pleshkov
e80844969f Float16 integration and API (#4234)
* f16 integration

tests

api

fix test

are you happy clippy

* fix build
2024-05-15 10:36:55 +02:00
Luis Cossío
4132d9a226 universal-query: Introduce new RawVector and VectorInput messages (#4209)
* Introduce `RawVector` and `VectorInput` messages

* gen grpc docs

* remove extra `optional`

* gen grpc docs
2024-05-14 11:40:11 -04:00
Luis Cossío
88bb625efd universal-query: Sketch request types pipeline (#4198)
* sketch types pipeline

* separate query, scroll, and search implementations into own files
2024-05-13 12:04:36 -04:00
Roman Titov
dae311e5e8 Add basic resharding types (#4216) 2024-05-13 13:31:03 +02:00
Tim Visée
ac2dd4da1b Bump dev version to 1.9.3-dev (#4208) 2024-05-10 11:28:12 +02:00
xzfc
4f3239f27c Misc fixes (#4212)
* SparseVectorIndex: replace pub with feature-gated getters

* Fixup dev-dependencies

* Fixup rust doc comments
2024-05-10 08:16:28 +00:00
Jojii
2bbf491204 make grpc service type enums unique for UpdateCollectionClusterSetupRequest and PointsUpdateOperation (#4205)
* make grpc service type enums unique
2024-05-09 13:41:06 +02:00
Arnaud Gourlay
b602cb7daa gRPC API multivector for write and retrieve (#4179)
* gRPC API multivector for write and retrieve

* improve naming

* validate multiv + sparse

* code review

* regen gRPC docs

* remove incorrect validation
2024-05-08 19:28:47 +02:00
Luis Cossío
b2538b2a12 remove deprecated internal search api (#4178) 2024-05-07 08:06:54 -04:00
Arnaud Gourlay
4f797c1bb4 REST API multivector for write and retrieve (#4117)
* REST API multivector for write and retrieve

* add test showing search is not supported

* use helper

* debug asserts

* better naming

* iterator style

* better naming

* add validation test for checking all inner vectors have the same size
2024-05-03 10:55:17 +02:00