* add rest and grpc interfaces
Also handle inference for this query
* follow refactor from base branch
* renaming from Ms Cooper
* get started on validation testing
* more validations
* test equivalence with query when less than 2 feedback elements
* rename feedback query to relevance_feedback query
* fix extraction of context pairs
* rename feedback vector to example
* make coderabbit happier
* upd test
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* [strict-mode] Add max number of payload index count
* improve docs and validation for StrictModeConfig
* fix from coderabbit
* test blocking access to payload indexes
* polish test
* allow custom K parameter for RRF
* generate grpc docs and openapi
* use tagged type approach for parametrized fusions
* use params approach in grpc
* simplify api structure
* upd schema
* nits
* rest: parameterized rrf as query variant
* consistency in doc comments
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Retain more closed segments
* Add back rocksdb
* Update WalOptions/WalConfig across the code
* Use NonZeroUsize
* Expose via APIs
* Update gRPC docs
* Fix stoarge compat test
* recompile openapi.json with rocksdb
* default wal retain closed fn
* update openapi.json
* Use qdrant/wal latest commit and remove from config.yaml
* expose mmr in rest and grpc
* fix referenced vectors in mmr queries
* rename to MmrInput in grpc for consistency
* mmr is now an optional parameter of nearest query
* fix score_threshold at local shard
* fix conversion of rest -> collection
* fix and refactor planned query
* nits
* rename `lambda` to `diversity` in interface
* handle score_threshold within mmr calculation
* add openapi test
* fix candidate limit at collection level
* candidate_limit -> candidates_limit
* finish rename to candidates_limit
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add function for validating legacy collection names
* Apply legacy collection name validation everywhere
* Add more validation to gRPC endpoints
* Adjust tests
* gRPC API for Distance Matrix
* generate API docs
* apply correct limit to sample
* add JWT tests
* pluralitiy
* fix COO naming
* drops COO as it trips codespell and is redundant
* clarify docs
* Add peer metadata state to consensus, update on sync local state
* Minor formatting improvements
* Use Qdrant version from constant
* Initialize empty metadata if not present on disk
* Refactor old parameter name
* Logging improvements based on review feedback
* Assert that Qdrant version matches crate version
* Do not use a custom (de)serializer for peer metadata
* Use semver directly for handling Qdrant versions
* Rename is_outdated to is_different_version
* Assert Qdrant version in build.rs
* Disable updating peer metadata until Qdrant 1.9
We do this, because if a node is running 1.8, there may still be nodes
running 1.7. Those nodes do not support this operation.
* clippy
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* feat: Print commit id in dev container build workflow
* fix: Keep .git while building Qdrant binary in docker
* fix: Remove redundant printing of git commit id
* fix: Copy only git files first
* fix: Docker build should have commit id if present
* refactor: Use fewer lines of code
* feat: Use git commit id from build arg
* refactor: Use cleaner code
* Fix string reference issue
* fix: Dont print info log if commit is not set
* fix: Include git commit id in release containers
---------
Co-authored-by: timvisee <tim@visee.me>
* Add functions to propagate updating cutoff point from collection level
* Add gRPC endpoint to set cutoff point
* Lock highest and cutoff clock maps separately
* Add initial gRPC call for requesting WAL recovery point for remote shard
* Add remote shard method to request WAL recovery point
* Add recovery point type in gRPC, use it in recovery point functions
* Add function to extend recovery point with missing clocks from clock map
* Add new gRPC type for recovery point clocks
* Remove atomic loading, because we use regular integers now
* feat: Print commit id in dev container build workflow
* fix: Keep .git while building Qdrant binary in docker
* fix: Remove redundant printing of git commit id
* fix: Copy only git files first
* fix: Docker build should have commit id if present
* refactor: Use fewer lines of code
* feat: Expose git commit id in the health check endpoint
* fix: CI errors
* test: Add test for health check api
* feat: Add / endpoint to openapi schema
* Make git commit hash optional
* ci: Enable debugging setup-protoc action
* Install later protobuf compiler through GitHub Action
* Disable debug mode for setup-protoc job
* refactor: Use commit instead of commit_id
* fix: Use commit instead of commit_id gRPC docs
* test: Update ping API test
* refactor: Rename ping api to root api
---------
Co-authored-by: timvisee <tim@visee.me>
* create and connect discovery http and grpc interfaces
* add openapi tests
* fix bad rebase
* Add better descriptions
* remove numpy from openapi tests
* fix rebase artifact
* remove already addressed TODO
* add more tests
* 🤡🔫 (cfg batch handler)
* add timeout query param for discover requests
* More gRPC validation
* make fields pydantic_openapi_generator_v3 friendly
* `context_pairs` -> `context` with struct for pairs
* discovery api is only discovery or context,
move struct description to fields
---------
Co-authored-by: timvisee <tim@visee.me>
* add timeout query param for search requests
* enable timeout for recommend requests
* Add query timeout for group by requests
* update openapi models
* Don't decrease timeout after recommend preprocessing
* Add openapi test
* code review
* add timeout to individual group by requests, non-decreasing
* handle timeout for discover
* Update timeout field tag in SearchBatchPoints
message
* Add `WaitForShardState` gRPC call definition
* Implement logic for `WaitForShardState` call
* In next shard snapshot transfer stage, wait for remote to reach `Partial`
* In shard snapshot transfer, synchronize consensus as last step
We don't have to synchronize consensus right away. Instead we just
confirm that the remote shard has reached `Partial` state. Then we
transform the queue proxy shard into the forward proxy.
Right before we finalize the transfer we do want to synchronize
consensus. First make sure the shard has reached `Partial` state in our
local replica set. Then synchronize all other nodes to make sure they
reach at least the same consensus state.
* Reformat internal collection service definition
* Fix paste-bugs in `snapshot_service.proto`
* Add shard snapshot gRCP API definition
* Add validation to shard snapshot gRPC API definition
* Implement conversions between gRPC and `collection` types
* Extract shard snapshot API implementation into common sub-module
* Implement shard snapshot gRPC API
* Generate gRPC docs
* Refactor `ShardSnapshots` gRPC service to be internal API only
* fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Move `ShardSnapshotRecoverResponse` to `shard_snapshots_service.proto`
* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Update `api/src/grpc/qdrant.rs`
* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Update gRPC docs
* Switch `ShardSnapshots` gRPC service to use `validate_and_log` instead of `validate`
