in grpc:
- restructure `ContextInput`, so that it becomes easy to handle both pairs and positive/negative lists in the future
in rest:
- restructure context queries so that they don't repeat the label in order to be used . E.g. turn `{ query: { context: { context: [ ... ] } } }` into `{ query: { context: [ ... ] } }`
- fixes fusion so that `{query: { fusion: "rrf" } }` actually works
both:
- renames `RecommendInput` positives and negatives to singular (same as in reco api)
* add `query` to shard trait
* add missing conversions for query
* update grpc docs
* Query response has intermediate results
* add ShardQueryResponse description
* move pub use to the top, keep only one way of reaching reexports
* REST API multivector for write and retrieve
* add test showing search is not supported
* use helper
* debug asserts
* better naming
* iterator style
* better naming
* add validation test for checking all inner vectors have the same size
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Decouple NamedVectorStruct from REST API
* add NamedVectorStruct::MultiDense to segment
* update doc
* fully qualified segment types
* no need to ser/de anymore
* sprinkle required PartialEq
* new rbac code needs to convert
* Add grey color for collection info status
* Restructure locks in local shard info method
* Set collection status to grey if we have pending optimizations
* Update OpenAPI specification and gRPC documentation
* Set optimizer status instead of color for compatibility reasons
* Only set and check for grey status if there are no other statuses
* Add peer metadata state to consensus, update on sync local state
* Minor formatting improvements
* Use Qdrant version from constant
* Initialize empty metadata if not present on disk
* Refactor old parameter name
* Logging improvements based on review feedback
* Assert that Qdrant version matches crate version
* Do not use a custom (de)serializer for peer metadata
* Use semver directly for handling Qdrant versions
* Rename is_outdated to is_different_version
* Assert Qdrant version in build.rs
* Disable updating peer metadata until Qdrant 1.9
We do this, because if a node is running 1.8, there may still be nodes
running 1.7. Those nodes do not support this operation.
* clippy
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add unique token to clock tags
* Consider last token in clock map
* Mark potential problem with TODO, we might want to keep clock token
* Represent UUID in gRPC as bytes, keep token when converting gRPC to REST
* Do not panic on invalid UUID, propagate parse error
* Use helper function for clock tag conversion with error handling
* Fix accept/reject logic for clocks, accept higher/equal, update token
* Add test for accepting exactly the same last operation multiple times
* Update advance method description to elaborate on what is accepted
* Keep unique clock tag token in recovery point
* Fix WAL delta tests, don't expect rejection from forward proxy anymore
* Extend missing clocks in recovery point in a more efficient way
* Fix clock tick comparison logic in recovery point
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* use u64 as token
* Remove failable clock tag conversions, use direct conversion
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Change `ClockMap` to always reject tick 0
* Enable rejecting operations in `LocalShard::update`...
...and propagate updated `clock_tick` back to the caller
* WIP: Retry operation on *all* nodes (with new tick)...
...if *any* node rejected the operation
* fixup! WIP: Retry operation on *all* nodes (with new tick)...
Keep the same `clock` for the whole duration of `update`
* Add update response status for clock rejection (#3725)
* Add clock rejected update status
* Don't hide internal items in docs
* Propagate clock rejected status if operation clock is too old
* Fix test in which we manually define clock ticks
* Expect forward proxy rejection, node already has that operation
* Add constant for first valid clock tick, init clocks in recovery point
* In testing, advance all zero clocks to start ticking from 1
* Attempt to fix clock_set_clock_map_workflow
* Rename WalError::Rejected to ClockRejected
* Limit retry attempts when operation is rejected for an old clock tick
* Don't log rejected clock tag if tick was 0
* Cleanup `ClockMap`/`RecoveryPoint`
* Cleanup `ClockSet` and `ShardReplicaSet::update`
* Fix `clock_set_clock_map_workflow` test
* add debug assertions
* fmt
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* When restarting shard transfer, use sync if old or new one had sync set
* refactor restart transfer models
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add consensus operation to restart shard transfer with different config
* Require shard transfer restart to have a changed configuration
* implement api
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add first stubs for WAL delta shard transfer method
* Repurpose queue proxy, use it for transferring WAL diff as well
* Integrate WAL delta transfer is transfer selection logic
* Add WalDelta shard transfer type which is not exposed in public API
* Basic implementation of falling back to stream records transfer
* Share await_consensus_sync function
* Ask remote shard for recovery point
* During WAL delta transfer, resolve shard diff locally for recovery point
* Rebase on latest dev, support empty WAL diff
* Rebase on latest dev, support empty WAL diff
* Use partial snapshot state for WAL delta transfer
* Set cutoff point on remote shard after shard WAL delta transfer
* Set cutoff point on remote shard after stream records transfer
* During WAL delta transfer, set s tate from partial snapshot to partial
* Describe WAL delta transfer in a comment
* Allow updating cutoff point in stream records transfer to fail
* Do not set cutoff point on remote shard on WAL delta transfer
* Make await consensus sync logic easier to read and reason about
* Fix fallback to other shard transfer method on WAL delta transfer fail
* Various minor improvements
* Add TODO for just ignoring API unimplemented errors
* Only allow stream records cutoff point error if remote is older version
* Allow switching to partial to fail when falling back
* Only change shard state to partial if not in partial state already
* Change default shard transfer method back to stream records
* Add important TODO back
* Add WAL delta shard transfer method in gRPC
* Prefer configured shard transfer method as default
* Extract shard transfer fallback logic into separate function
