* universal-query: Add validations to query input
* update openapi spec
* use raw request to by-pass local validations
* add validation for order_by and factorize
* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
* Add serde/validate attributes to resharding operations, matching others
* Fix broken comment
* Add debug message for resharding driver entering stages
* Fix shard transfer start setting state of wrong replica for resharding
* Remove obsolete clones
* Add target shard ID to shard key, add relevant gRPC types
* Move target shard ID below source shard ID field
* Rename collection_name to collection_id
* Reformat
* Transferring point batches must merge points in case of resharding
* In resharding state, sync list of peers on start
* Add logic for setting replica set state through consensus dispatcher
* Properly start resharding transfer
* Properly finish resharding transfers, set shard state correctly
* Fix shard transfer initialisation with different target shard
* Fix shard state handling with resharding on all nodes on transfer start
* Don't reset locally disabled state if only existing shard is resharding
* Add important TODOs
* Update OpenAPI and gRPC specification
* Elaborate on some logic in code with comments
* Use user configured shard transfer method for replication
* Add debug assert, on transfer start we should not replace existing shard
* On transfer start, be aware of different sender/receiver local states
This fixes transfers where we might not have a replica on all nodes
* Fix shard transfer not setting cutoff point on target shard
* While resharding, migrate shards in numerical order
* On shard transfer initialisation, ensure transfer targets given shard
* universal-query: add with_lookup to query API definition
* fix consistency doc check
* lookup_from and not lookup_with
* clean import
* improve doc for lookup location wrt using
* Suggestions for descriptions
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
in grpc:
- restructure `ContextInput`, so that it becomes easy to handle both pairs and positive/negative lists in the future
in rest:
- restructure context queries so that they don't repeat the label in order to be used . E.g. turn `{ query: { context: { context: [ ... ] } } }` into `{ query: { context: [ ... ] } }`
- fixes fusion so that `{query: { fusion: "rrf" } }` actually works
both:
- renames `RecommendInput` positives and negatives to singular (same as in reco api)
* add `query` to shard trait
* add missing conversions for query
* update grpc docs
* Query response has intermediate results
* add ShardQueryResponse description
* move pub use to the top, keep only one way of reaching reexports
* REST API multivector for write and retrieve
* add test showing search is not supported
* use helper
* debug asserts
* better naming
* iterator style
* better naming
* add validation test for checking all inner vectors have the same size
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* byte storage with quantization
raw scorer integration
config and test
are you happy fmt
fn renamings
cow refactor
use quantization branch
quantization update
* are you happy clippy
* don't use distance in quantized scorers
* fix build
* add fn quantization_preprocess
* apply preprocessing for only cosine float metric
* fix sparse vectors tests
* update openapi
* more complicated integration test
* update openapi comment
* mmap byte storages support
* fix async test
* move .unwrap closer to the actual check of the vector presence
* fmt
* remove distance similarity function
* avoid copying data while working with cow
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Decouple NamedVectorStruct from REST API
* add NamedVectorStruct::MultiDense to segment
* update doc
* fully qualified segment types
* no need to ser/de anymore
* sprinkle required PartialEq
* new rbac code needs to convert
* Multivec knob for SegmentConfig
* regen openapi
* add TODO for next step
* introduce multivecconfig to support more similarity aggregation
* update openapi
* Add grey color for collection info status
* Restructure locks in local shard info method
* Set collection status to grey if we have pending optimizations
* Update OpenAPI specification and gRPC documentation
* Set optimizer status instead of color for compatibility reasons
* Only set and check for grey status if there are no other statuses
