* first PR implementation (#2865)
- fetch offset id
- restructure tests
- only let order_by with numeric
- introduce order_by interface
cargo fmt
update openapi
calculate range to fetch using offset + limit, do some cleanup
enable index validation, fix test
Fix pagination
add e2e tests
make test a little more strict
select numeric index on read_ordered_filtered
add filtering test 🫨
fix filtering on order-by
fix pip requirements
add grpc interface, make read_ordered_filtered fallible
fmt
small optimization of `with_payload` and `with_vector`
refactor common logic of point_ops and local_shard_operations
Make filtering test harder and fix limit for worst case
update openapi
small clarity refactor
avoid extra allocation when sorting with offset
stream from numeric index btree instead of calculating range
use payload to store order-by value, instead of modifying Record interface
various fixes:
- fix ordering at collection level, when merging shard results
- fix offset at segment level, to take into account also value offset
- make rust tests pass
remove unused histogram changes
fix error messages and make has_range_index exhaustive
remove unused From impl
Move OrderBy and Direction to segment::data_types::order_by
Refactor normal scroll_by in local_shard_operations.rs
More cleanup + rename OrderableRead to StreamWithValue
empty commit
optimization for merging results from shards and segments
fix case of multi-valued fields
fix IntegerIndexParams name after rebase
precompute offset key
use extracted `read_by_id_stream`
Expose value_offset to user
- rename offset -> value_offset
- extract offset value fetching logic
* remove offset functionality when using order_by
* include order_by in ForwardProxyShard
* extra nits
* remove histogram changes
* more nits
* self review
* resolve conflicts after rebase, not enable order-by with datetime index schema
* make grpc start_from value extendable
* gen grpc docs
---------
Co-authored-by: kwkr <kawka.maciej.93@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Merge serde attributes
* Remove obsolete conversion
* Add integer type with parameters
* Make integer lookup and range parameters non-optional
* Add parameterized integer index types test
Co-authored-by: Di Zhao <diz@twitter.com>
* Cleanup
---------
Co-authored-by: Di Zhao <diz@twitter.com>
* add checksum to SnapshotDescription
* implement storing snapshot checksums in a file
* Don't serialize checksum if it's None for backwards compatibility
* Remove hex dependency, use Rust std formatter for this
* Do not error if we cannot remove checksum file for snapshot
Some snapshots may not have a corresponding checksum file. Maybe it was
created in an older Qdrant version that didn't have support for this, or
a user hasn't provided any.
* Add debug message when hashing snapshot, can be expensive on large files
* Inline debug messages
* Add checksum to shard snapshots
* If creating snapshot fails, delete snapshot target and checksum file
* Use Rust idiomatic ok() and improve debug messages
* Use correct snapshot checksum paths, clean up after shard snapshot
* Use better path type in get_checksum_path
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* feat: Expose git commit id in the health check endpoint
* fix: CI errors
* test: Add test for health check api
* feat: Add / endpoint to openapi schema
* Make git commit hash optional
* ci: Enable debugging setup-protoc action
* Install later protobuf compiler through GitHub Action
* Disable debug mode for setup-protoc job
* refactor: Use commit instead of commit_id
* fix: Use commit instead of commit_id gRPC docs
* test: Update ping API test
* refactor: Rename ping api to root api
---------
Co-authored-by: timvisee <tim@visee.me>
* Make point, indexed point and vector counts optional in collection info
* Update OpenAPI specification and gRPC docs
* Use functional entry API
* Mark point/vector count fields as deprecated
* Use internal collection info structure to remove internal optionals
* Don't deprecate, but mark point counts as approximate
* Read-only API keys
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Correct placement of OpenAPI security
Place regex dep with actix/tonic
* Read-only API keys
* Replace with pytests
* API Key tests run on the same job
* Drop allow dead-code
* Rename setting key
* Containerized tests
* No special config files
* DRY
* refactor: re-use can_write method
* refactor: replace static by constants
* refactor: get PID from `$!`
* refactor: use explicit brackets on boolean condition
* style: fix identation
* small fixes + account for new APIs
* specify security in openapi
* small fix + chmod for .sh testfile
* add best-efford check for api consistency
---------
Co-authored-by: Amr Hassan <amr.hassan@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* WIP: collection-level storage for payload indexe scheme
* introduce consensus-level operation for creating payload index
* make operation_id optional in the UpdateResult
* set payload index in newly created shards
* upd api definitions
* include payload index schema into collection consensus state
* include payload index schema into shard snapshot
* review fixes
* create and connect discovery http and grpc interfaces
* add openapi tests
* fix bad rebase
* Add better descriptions
* remove numpy from openapi tests
* fix rebase artifact
* remove already addressed TODO
* add more tests
* 🤡🔫 (cfg batch handler)
* add timeout query param for discover requests
* More gRPC validation
* make fields pydantic_openapi_generator_v3 friendly
* `context_pairs` -> `context` with struct for pairs
* discovery api is only discovery or context,
move struct description to fields
---------
Co-authored-by: timvisee <tim@visee.me>
* SparseVector implements VectorIndex
* dedicated telemetry
* conflict
* easy code reviews
* simplify tracking indexed points count
* move telemetry conversion to sparse file
* move max_result_count out of inverted index trait
* unify sparse vector fixtures
* simpler conversion
* add todo regarding OOM potential
* reuse check deleted from raw scorer with TODO
* change new to open to handle mmap index
* add timeout query param for search requests
* enable timeout for recommend requests
* Add query timeout for group by requests
* update openapi models
* Don't decrease timeout after recommend preprocessing
* Add openapi test
* code review
* add timeout to individual group by requests, non-decreasing
* handle timeout for discover
* Update timeout field tag in SearchBatchPoints
message
* wip: implement consensus operations to create and delete shard keys
* Fix typo in variable name
* wip: methods to add shard keys
* fmt
* api for submitting creating and deleting shard keys operations into consensus
* fmt
* apply shard mapping in with the collection state transfer
* handle single-to-cluster transition
* fix api structure and bugs
* fmt
* rename param
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
* Clone inside blocks
* Add shard transfer method to distinguish between batching and snapshots
* Add stub method to drive snapshot transfer
* Store remote shard in forward proxy, merge unproxy methods
* On snapshot shard transfer, create a shard snapshot
* Unify logic for unproxifying forward and queue proxy
* Error snapshot transfer if shard is not a queue proxy
* Add remote shard function to request remote HTTP port
* Handle all specific shard types when proxifying
* Allow queue proxy for some shard holder snapshot methods
* Bring local and remote shard snapshot transfer URLs into transfer logic
* Expose optional shard transfer method parameter in REST and gRPC API
* Expose shard transfer method in list of active transfers
* Fix off-by-one error in queue proxy shard batch transfer logic
* Do not set max ack version for WAL twice, already set when finalizing
* Merge comment for two similar calls
* Use reqwest client to transfer and recover shard snapshot on remote
Using the reqwest client should be temporary. We better switch to a gRPC
call here eventually to use our existing channels. That way we don't
require an extra HTTP client (and dependency) just for this.
* Send queue proxy updates to remote when shard is transferred
* On shard queue transfer, set max WAL ack to last transferred
* Add safe queue proxy destructor, skip destructing in error
This adds a finalize method to safely destruct a queue proxy shard. It
ensures that all remaining updates are transferred to the remote, and
that the max acknowledged version for our WAL is released. Only then is
the queue proxy shard destructed unwrapping the inner local shard.
Our unproxify logic now ensures that the queue proxy shard remains if
transferring the updates fails.
* Clean up method driving shard snapshot transfer a bit
* Change default shard transfer method to stream records
This changes the default transfer method to stream records rather than
using a snaphsot transfer. We can switch this once snapshot transfer is
fully integrated.
* Improve error handling, don't panic but return proper error
* Do not unwrap in type conversions
* Update OpenAPI and gRPC specification
* Resolve and remove some TODOs
* During shard snapshot transfer, use REST port from config
* Always release max acknowledged WAL version on queue proxy finalize
* Rework queue unproxying, transform into forward proxy to handle errors
When a queue or forward proxy shard needs to be unproxified into a local
shard again we typically don't have room to handle errors. A queue proxy
shard may error if it fails to send updates to the remote shard, while a
forward proxy does not fail at all when transforming.
We now transfer queued updates before a shard is unproxified. This
allows for proper error handling. After everything is transferred the
shard is transformed into a forward proxy which can eventually be safely
unproxified later.
* Add trace logging for transferring queue proxy updates in batch
* Simplify snapshot method conversion from gRPC
* Remove remote shard parameter
* Add safe guard to queue proxy handler, panic in debug if not finalized
* Improve safety and architecture of queue proxy shard
Switch from an explicit finalized flag to an outer-inner architecture.
This improves the interface and robustness of the type.
* Do not panic on drop if already unwinding
* Make REST port interface in channel service for local node explicitly
* Recover shard on remote over gRPC, remove reqwest client
* Use shard transfer priority for shard snapshot recovery
* Remove obsolete comment
* Simplify qualified path with use
* Don't construct URLs ourselves as a string, use `parse` and `set_port`
* Use `set_path` when building shard download URL
* Fix error handling in queue to forward proxy transformation
Before, we didn't handle finalization errors properly. If this failed,
tie shard would be lost. With this change the queue proxy shard is put
back.
* Set default shard transfer method to stream records, eliminate panics
* Fix shard snapshot transfer not correctly aborting due to queue proxy
When a shard transfer fails (for any reason), the transfer is aborted.
If we still have a queue proxy shard it should also be reverted, and
collected updates should be forgotten. Before this change it would try
to send all collected updates to the remote, even if the transfer
failed.
* Review fixes
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Review fixes
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Initiate forward and queue proxy shard in specialized transfer methods
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add consensus interface to shard transfer, repurpose dispatcher (#2873)
* Add shard transfer consensus interface
* Integrate shard transfer consensus interface into toc and transfer logic
* Repurpose dispatcher for getting consensus into shard transfer
* Derive clone
* Mark consensus as unused for now
* Use custom dispatcher with weak ref to prevent Arc cycle for ToC
* Add comment on why a weak reference is used
* Do exhaustive match in shard unproxy logic
* Restructure match statement, use match if
* When queue proxifying shard, allow forward proxy state if same remote
* Before retrying a shard transfer after error, destruct queue proxy
* Synchronize consensus across all nodes for shard snapshot transfer (#2874)
* Move await consensus commit functions into channel service
* Add shard consensus method to synchronize consensus across all nodes
* Move transfer config, channels and local address into snapshot transfer
* Await other nodes to reach consensus before finalizing shard transfer
* Do not fail right away awaiting consensus if still on older term
Instead, give the node time to reach the same term.
* Fix `await_commit_on_all_peers` not catching peer errors properly
* Change return type of `wait_for_consensus_commit` to `Result`
This is of course more conventional, and automatically sets `must_use`.
* Explicitly note number of peers when awaiting consensus
* Before consensus sync, wait for local shard to reach partial state
* Fix timeout error handling when waiting for replica set state
* Wait for replica set to have remote in partial state instead
* Set `(Partial)Snapshot` states for shard snapshot transfer through consensus (#2881)
* When doing a shard snapshot transfer, set shard to `PartialSnapshot`
* Add shard transfer method to set shard state to partial
It currently uses a naive implementation. Using a custom consensus
operation to also confirm a transfer is still active will be implemented
later.
* Add consensus snapshot transfer operation to change shard to partial
The operation `ShardTransferOperations::SnapshotRecovered` is called
after the shard snapshot is recovered on the remote and it progresses
the transfer further.
The operation sets the shard state from `PartialSnapshot` to `Partial`
and ensures the transfer is still active.
* Confirm consensus put shard into partial state, retry 3 times
* Get replica set once
* Add extensive shard snapshot transfer process docs, clean up function
* Fix typo
* Review suggestion
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Add delay between consensus confirmation retries
* Rename retry timeout to retry delay
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* On replicate shard, remember specified method
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Fix paste-bugs in `snapshot_service.proto`
* Add shard snapshot gRCP API definition
* Add validation to shard snapshot gRPC API definition
* Implement conversions between gRPC and `collection` types
* Extract shard snapshot API implementation into common sub-module
* Implement shard snapshot gRPC API
* Generate gRPC docs
* Refactor `ShardSnapshots` gRPC service to be internal API only
* fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Move `ShardSnapshotRecoverResponse` to `shard_snapshots_service.proto`
* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Update `api/src/grpc/qdrant.rs`
* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Update gRPC docs
* Switch `ShardSnapshots` gRPC service to use `validate_and_log` instead of `validate`
* Extend GeoPolygon to support interiors (#2315)
Per GeoJson, we should support polygon with exterior and interiors (holes on the surface) in Geo Filter by Polygon(#795). This commit extend current GeoPolygon filter to accept interiors. It includes:
1. changes to proto and internal GeoPolygon struct, and validation fn
2. add and refactor some tests
3. add integration test
* add gRPC geo_polygon validation
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
