* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
* Add serde/validate attributes to resharding operations, matching others
* Fix broken comment
* Add debug message for resharding driver entering stages
* Fix shard transfer start setting state of wrong replica for resharding
* Remove obsolete clones
* Add target shard ID to shard key, add relevant gRPC types
* Move target shard ID below source shard ID field
* Rename collection_name to collection_id
* Reformat
* Transferring point batches must merge points in case of resharding
* In resharding state, sync list of peers on start
* Add logic for setting replica set state through consensus dispatcher
* Properly start resharding transfer
* Properly finish resharding transfers, set shard state correctly
* Fix shard transfer initialisation with different target shard
* Fix shard state handling with resharding on all nodes on transfer start
* Don't reset locally disabled state if only existing shard is resharding
* Add important TODOs
* Update OpenAPI and gRPC specification
* Elaborate on some logic in code with comments
* Use user configured shard transfer method for replication
* Add debug assert, on transfer start we should not replace existing shard
* On transfer start, be aware of different sender/receiver local states
This fixes transfers where we might not have a replica on all nodes
* Fix shard transfer not setting cutoff point on target shard
* While resharding, migrate shards in numerical order
* On shard transfer initialisation, ensure transfer targets given shard
* universal-query: add with_lookup to query API definition
* fix consistency doc check
* lookup_from and not lookup_with
* clean import
* improve doc for lookup location wrt using
* Suggestions for descriptions
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
in grpc:
- restructure `ContextInput`, so that it becomes easy to handle both pairs and positive/negative lists in the future
in rest:
- restructure context queries so that they don't repeat the label in order to be used . E.g. turn `{ query: { context: { context: [ ... ] } } }` into `{ query: { context: [ ... ] } }`
- fixes fusion so that `{query: { fusion: "rrf" } }` actually works
both:
- renames `RecommendInput` positives and negatives to singular (same as in reco api)
* add `query` to shard trait
* add missing conversions for query
* update grpc docs
* Query response has intermediate results
* add ShardQueryResponse description
* move pub use to the top, keep only one way of reaching reexports
* REST API multivector for write and retrieve
* add test showing search is not supported
* use helper
* debug asserts
* better naming
* iterator style
* better naming
* add validation test for checking all inner vectors have the same size
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* byte storage with quantization
raw scorer integration
config and test
are you happy fmt
fn renamings
cow refactor
use quantization branch
quantization update
* are you happy clippy
* don't use distance in quantized scorers
* fix build
* add fn quantization_preprocess
* apply preprocessing for only cosine float metric
* fix sparse vectors tests
* update openapi
* more complicated integration test
* update openapi comment
* mmap byte storages support
* fix async test
* move .unwrap closer to the actual check of the vector presence
* fmt
* remove distance similarity function
* avoid copying data while working with cow
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Decouple NamedVectorStruct from REST API
* add NamedVectorStruct::MultiDense to segment
* update doc
* fully qualified segment types
* no need to ser/de anymore
* sprinkle required PartialEq
* new rbac code needs to convert
* Multivec knob for SegmentConfig
* regen openapi
* add TODO for next step
* introduce multivecconfig to support more similarity aggregation
* update openapi
* Add grey color for collection info status
* Restructure locks in local shard info method
* Set collection status to grey if we have pending optimizations
* Update OpenAPI specification and gRPC documentation
* Set optimizer status instead of color for compatibility reasons
* Only set and check for grey status if there are no other statuses
* Change `ClockMap` to always reject tick 0
* Enable rejecting operations in `LocalShard::update`...
...and propagate updated `clock_tick` back to the caller
* WIP: Retry operation on *all* nodes (with new tick)...
...if *any* node rejected the operation
* fixup! WIP: Retry operation on *all* nodes (with new tick)...
Keep the same `clock` for the whole duration of `update`
* Add update response status for clock rejection (#3725)
* Add clock rejected update status
* Don't hide internal items in docs
* Propagate clock rejected status if operation clock is too old
* Fix test in which we manually define clock ticks
* Expect forward proxy rejection, node already has that operation
* Add constant for first valid clock tick, init clocks in recovery point
* In testing, advance all zero clocks to start ticking from 1
* Attempt to fix clock_set_clock_map_workflow
* Rename WalError::Rejected to ClockRejected
* Limit retry attempts when operation is rejected for an old clock tick
* Don't log rejected clock tag if tick was 0
* Cleanup `ClockMap`/`RecoveryPoint`
* Cleanup `ClockSet` and `ShardReplicaSet::update`
* Fix `clock_set_clock_map_workflow` test
* add debug assertions
* fmt
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* When restarting shard transfer, use sync if old or new one had sync set
* refactor restart transfer models
---------
Co-authored-by: generall <andrey@vasnetsov.com>
