in grpc:
- restructure `ContextInput`, so that it becomes easy to handle both pairs and positive/negative lists in the future
in rest:
- restructure context queries so that they don't repeat the label in order to be used . E.g. turn `{ query: { context: { context: [ ... ] } } }` into `{ query: { context: [ ... ] } }`
- fixes fusion so that `{query: { fusion: "rrf" } }` actually works
both:
- renames `RecommendInput` positives and negatives to singular (same as in reco api)
* add `query` to shard trait
* add missing conversions for query
* update grpc docs
* Query response has intermediate results
* add ShardQueryResponse description
* move pub use to the top, keep only one way of reaching reexports
* REST API multivector for write and retrieve
* add test showing search is not supported
* use helper
* debug asserts
* better naming
* iterator style
* better naming
* add validation test for checking all inner vectors have the same size
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* byte storage with quantization
raw scorer integration
config and test
are you happy fmt
fn renamings
cow refactor
use quantization branch
quantization update
* are you happy clippy
* don't use distance in quantized scorers
* fix build
* add fn quantization_preprocess
* apply preprocessing for only cosine float metric
* fix sparse vectors tests
* update openapi
* more complicated integration test
* update openapi comment
* mmap byte storages support
* fix async test
* move .unwrap closer to the actual check of the vector presence
* fmt
* remove distance similarity function
* avoid copying data while working with cow
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Decouple NamedVectorStruct from REST API
* add NamedVectorStruct::MultiDense to segment
* update doc
* fully qualified segment types
* no need to ser/de anymore
* sprinkle required PartialEq
* new rbac code needs to convert
* Multivec knob for SegmentConfig
* regen openapi
* add TODO for next step
* introduce multivecconfig to support more similarity aggregation
* update openapi
* Add grey color for collection info status
* Restructure locks in local shard info method
* Set collection status to grey if we have pending optimizations
* Update OpenAPI specification and gRPC documentation
* Set optimizer status instead of color for compatibility reasons
* Only set and check for grey status if there are no other statuses
* Change `ClockMap` to always reject tick 0
* Enable rejecting operations in `LocalShard::update`...
...and propagate updated `clock_tick` back to the caller
* WIP: Retry operation on *all* nodes (with new tick)...
...if *any* node rejected the operation
* fixup! WIP: Retry operation on *all* nodes (with new tick)...
Keep the same `clock` for the whole duration of `update`
* Add update response status for clock rejection (#3725)
* Add clock rejected update status
* Don't hide internal items in docs
* Propagate clock rejected status if operation clock is too old
* Fix test in which we manually define clock ticks
* Expect forward proxy rejection, node already has that operation
* Add constant for first valid clock tick, init clocks in recovery point
* In testing, advance all zero clocks to start ticking from 1
* Attempt to fix clock_set_clock_map_workflow
* Rename WalError::Rejected to ClockRejected
* Limit retry attempts when operation is rejected for an old clock tick
* Don't log rejected clock tag if tick was 0
* Cleanup `ClockMap`/`RecoveryPoint`
* Cleanup `ClockSet` and `ShardReplicaSet::update`
* Fix `clock_set_clock_map_workflow` test
* add debug assertions
* fmt
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* When restarting shard transfer, use sync if old or new one had sync set
* refactor restart transfer models
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add consensus operation to restart shard transfer with different config
* Require shard transfer restart to have a changed configuration
* implement api
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add first stubs for WAL delta shard transfer method
* Repurpose queue proxy, use it for transferring WAL diff as well
* Integrate WAL delta transfer is transfer selection logic
* Add WalDelta shard transfer type which is not exposed in public API
* Basic implementation of falling back to stream records transfer
* Share await_consensus_sync function
* Ask remote shard for recovery point
* During WAL delta transfer, resolve shard diff locally for recovery point
* Rebase on latest dev, support empty WAL diff
* Rebase on latest dev, support empty WAL diff
* Use partial snapshot state for WAL delta transfer
* Set cutoff point on remote shard after shard WAL delta transfer
* Set cutoff point on remote shard after stream records transfer
* During WAL delta transfer, set s tate from partial snapshot to partial
* Describe WAL delta transfer in a comment
* Allow updating cutoff point in stream records transfer to fail
* Do not set cutoff point on remote shard on WAL delta transfer
* Make await consensus sync logic easier to read and reason about
* Fix fallback to other shard transfer method on WAL delta transfer fail
* Various minor improvements
* Add TODO for just ignoring API unimplemented errors
* Only allow stream records cutoff point error if remote is older version
* Allow switching to partial to fail when falling back
* Only change shard state to partial if not in partial state already
* Change default shard transfer method back to stream records
* Add important TODO back
* Add WAL delta shard transfer method in gRPC
* Prefer configured shard transfer method as default
* Extract shard transfer fallback logic into separate function
* Add recovery shard replica set state
* Accept forced operations in partial snapshot state
* Fix switching into wrong state
* Add some helpful comments
* Deduplication in match statement
* Get rid of Arc in SegmentOptimizer::get_telemetry_counter()
* Get rid of SegmentOptimizer::get_telemetry_data
* Prometheus histogram support
* Fixes, and sparse buckets
* Preallocate in convert_histogram, merge_histograms
* debug_assert to check boundaries are sorted
* Generate histograms when details_level >= 3 or in /metrics
* Support set by key in low level.
* Rename key field.
* Format.
* Pass key.
* Format.
* Test.
* Clippy.
* Fix ci lint.
* Check grpc consistency.
* Update openapi.
* Fix empty key test case.
* Support array index.
* Format.
* Add test for non exists key.
* Clippy fix.
* Add idempotence test.
* Update index by updated payload.
* Add ut for utils.
* Add ut for 1 level key.
* Fix ut.
* Support no exits key.
* Fix test result.
* Fix after rebase
* handle wildcart insertion into non-existing array
* avoid double read of payload during update
* fix missing removing data from index in case if set_payload removes indexed field
---------
Co-authored-by: Shylock Hg <shylock@DESKTOP-40I855A>
Co-authored-by: Albert Safin <xzfcpw@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
