* first PR implementation (#2865)
- fetch offset id
- restructure tests
- only let order_by with numeric
- introduce order_by interface
cargo fmt
update openapi
calculate range to fetch using offset + limit, do some cleanup
enable index validation, fix test
Fix pagination
add e2e tests
make test a little more strict
select numeric index on read_ordered_filtered
add filtering test 🫨
fix filtering on order-by
fix pip requirements
add grpc interface, make read_ordered_filtered fallible
fmt
small optimization of `with_payload` and `with_vector`
refactor common logic of point_ops and local_shard_operations
Make filtering test harder and fix limit for worst case
update openapi
small clarity refactor
avoid extra allocation when sorting with offset
stream from numeric index btree instead of calculating range
use payload to store order-by value, instead of modifying Record interface
various fixes:
- fix ordering at collection level, when merging shard results
- fix offset at segment level, to take into account also value offset
- make rust tests pass
remove unused histogram changes
fix error messages and make has_range_index exhaustive
remove unused From impl
Move OrderBy and Direction to segment::data_types::order_by
Refactor normal scroll_by in local_shard_operations.rs
More cleanup + rename OrderableRead to StreamWithValue
empty commit
optimization for merging results from shards and segments
fix case of multi-valued fields
fix IntegerIndexParams name after rebase
precompute offset key
use extracted `read_by_id_stream`
Expose value_offset to user
- rename offset -> value_offset
- extract offset value fetching logic
* remove offset functionality when using order_by
* include order_by in ForwardProxyShard
* extra nits
* remove histogram changes
* more nits
* self review
* resolve conflicts after rebase, not enable order-by with datetime index schema
* make grpc start_from value extendable
* gen grpc docs
---------
Co-authored-by: kwkr <kawka.maciej.93@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Move CPU count function to common, fix wrong CPU count in visited list
* Change default number of rayon threads to 8
* Use CPU budget and CPU permits for optimizer tasks to limit utilization
* Respect configured thread limits, use new sane defaults in config
* Fix spelling issues
* Fix test compilation error
* Improve breaking if there is no CPU budget
* Block optimizations until CPU budget, fix potentially getting stuck
Our optimization worker now blocks until CPU budget is available to
perform the task.
Fix potential issue where optimization worker could get stuck. This
would happen if no optimization task is started because there's no
available CPU budget. This ensures the worker is woken up again to
retry.
* Utilize n-1 CPUs with optimization tasks
* Better handle situations where CPU budget is drained
* Dynamically scale rayon CPU count based on CPU size
* Fix incorrect default for max_indexing_threads conversion
* Respect max_indexing_threads for collection
* Make max_indexing_threads optional, use none to set no limit
* Update property documentation and comments
* Property max_optimization_threads is per shard, not per collection
* If we reached shard optimization limit, skip further checks
* Add remaining TODOs
* Fix spelling mistake
* Align gRPC comment blocks
* Fix compilation errors since last rebase
* Make tests aware of CPU budget
* Use new CPU budget calculation function everywhere
* Make CPU budget configurable in settings, move static budget to common
* Do not use static CPU budget, instance it and pass it through
* Update CPU budget description
* Move heuristic into defaults
* Fix spelling issues
* Move cpu_budget property to a better place
* Move some things around
* Minor review improvements
* Use range match statement for CPU count heuristics
* Systems with 1 or 2 CPUs do not keep cores unallocated by default
* Fix compilation errors since last rebase
* Update lib/segment/src/types.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/storage/src/content_manager/toc/transfer.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename cpu_budget to optimizer_cpu_budget
* Update OpenAPI specification
* Require at least half of the desired CPUs for optimizers
This prevents running optimizations with just one CPU, which could be
very slow.
* Don't use wildcard in CPU heuristic match statements
* Rename cpu_budget setting to optimizer_cpu_budget
* Update CPU budget comments
* Spell acquire correctly
* Change if-else into match
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename max_rayon_threads to num_rayon_threads, add explanation
* Explain limit in update handler
* Remove numbers for automatic selection of indexing threads
* Inline max_workers variable
* Remove CPU budget from ShardTransferConsensus trait, it is in collection
* small allow(dead_code) => cfg(test)
* Remove now obsolete lazy_static
* Fix incorrect CPU calculation in CPU saturation test
* Make waiting for CPU budget async, don't block current thread
* Prevent deadlock on optimizer signal channel
Do not block the optimization worker task anymore to wait for CPU budget
to be available. That prevents our optimizer signal channel from being
drained, blocking incoming updates because the cannot send another
optimizer signal. Now, prevent blocking this task all together and
retrigger the optimizers separately when CPU budget is available again.
* Fix incorrect CPU calculation in optimization cancel test
* Rename CPU budget wait function to notify
* Detach API changes from CPU saturation internals
This allows us to merge into a patch version of Qdrant. We can
reintroduce the API changes in the upcoming minor release to make all of
it fully functional.
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* feat: Print commit id in dev container build workflow
* fix: Keep .git while building Qdrant binary in docker
* fix: Remove redundant printing of git commit id
* fix: Copy only git files first
* fix: Docker build should have commit id if present
* refactor: Use fewer lines of code
* Merge serde attributes
* Remove obsolete conversion
* Add integer type with parameters
* Make integer lookup and range parameters non-optional
* Add parameterized integer index types test
Co-authored-by: Di Zhao <diz@twitter.com>
* Cleanup
---------
Co-authored-by: Di Zhao <diz@twitter.com>
* add checksum to SnapshotDescription
* implement storing snapshot checksums in a file
* Don't serialize checksum if it's None for backwards compatibility
* Remove hex dependency, use Rust std formatter for this
* Do not error if we cannot remove checksum file for snapshot
Some snapshots may not have a corresponding checksum file. Maybe it was
created in an older Qdrant version that didn't have support for this, or
a user hasn't provided any.
* Add debug message when hashing snapshot, can be expensive on large files
* Inline debug messages
* Add checksum to shard snapshots
* If creating snapshot fails, delete snapshot target and checksum file
* Use Rust idiomatic ok() and improve debug messages
* Use correct snapshot checksum paths, clean up after shard snapshot
* Use better path type in get_checksum_path
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* feat: Expose git commit id in the health check endpoint
* fix: CI errors
* test: Add test for health check api
* feat: Add / endpoint to openapi schema
* Make git commit hash optional
* ci: Enable debugging setup-protoc action
* Install later protobuf compiler through GitHub Action
* Disable debug mode for setup-protoc job
* refactor: Use commit instead of commit_id
* fix: Use commit instead of commit_id gRPC docs
* test: Update ping API test
* refactor: Rename ping api to root api
---------
Co-authored-by: timvisee <tim@visee.me>
* Make point, indexed point and vector counts optional in collection info
* Update OpenAPI specification and gRPC docs
* Use functional entry API
* Mark point/vector count fields as deprecated
* Use internal collection info structure to remove internal optionals
* Don't deprecate, but mark point counts as approximate
* WIP: collection-level storage for payload indexe scheme
* introduce consensus-level operation for creating payload index
* make operation_id optional in the UpdateResult
* set payload index in newly created shards
* upd api definitions
* include payload index schema into collection consensus state
* include payload index schema into shard snapshot
* review fixes
* create and connect discovery http and grpc interfaces
* add openapi tests
* fix bad rebase
* Add better descriptions
* remove numpy from openapi tests
* fix rebase artifact
* remove already addressed TODO
* add more tests
* 🤡🔫 (cfg batch handler)
* add timeout query param for discover requests
* More gRPC validation
* make fields pydantic_openapi_generator_v3 friendly
* `context_pairs` -> `context` with struct for pairs
* discovery api is only discovery or context,
move struct description to fields
---------
Co-authored-by: timvisee <tim@visee.me>
This commit adds the csharp_namespace option to proto files, allowing
the C# gRPC client generated from the protos to be generated with
the namespace used in the official .NET client, without needing to
patch the values in.
Closesqdrant/qdrant-dotnet#3
* add timeout query param for search requests
* enable timeout for recommend requests
* Add query timeout for group by requests
* update openapi models
* Don't decrease timeout after recommend preprocessing
* Add openapi test
* code review
* add timeout to individual group by requests, non-decreasing
* handle timeout for discover
* Update timeout field tag in SearchBatchPoints
message
* Use 24 hour timeout for shard snapshot transfer and recovery on remote
This allows the snapshot transfer and recovery process to take much
longer than before. Now it allows to take up to 24 hours for very large
snapshots over slow connections, while before it only allowed this
process to take 60 seconds.
* Downgrade timeout into `recover_shard_snapshot_from_url`, remove retries
* Remove custom timeout implementation:
- Set very high internal channel timeout
- Intercept requests to set default timeout if not already set
Co-authored-by: Tim Visée <tim+github@visee.me>
* improve From<Status> for CollectionError
* Increase maximum gRPC channel timeout to 24 hours
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
* reproduce reported from with gprc panic in batch requests
* Fix validation extension on lists, just return first validation error
The public validator crate interface does not support creating
`ValidationErrors` struct with non-static strings, nor with multiple
errors for a single field. We can therefore not collect multiple errors
from multiple items and must return early on the first error.
* Update test, expect validation error response
---------
Co-authored-by: timvisee <tim@visee.me>
* Add `WaitForShardState` gRPC call definition
* Implement logic for `WaitForShardState` call
* In next shard snapshot transfer stage, wait for remote to reach `Partial`
* In shard snapshot transfer, synchronize consensus as last step
We don't have to synchronize consensus right away. Instead we just
confirm that the remote shard has reached `Partial` state. Then we
transform the queue proxy shard into the forward proxy.
Right before we finalize the transfer we do want to synchronize
consensus. First make sure the shard has reached `Partial` state in our
local replica set. Then synchronize all other nodes to make sure they
reach at least the same consensus state.
* Reformat internal collection service definition
* Clone inside blocks
* Add shard transfer method to distinguish between batching and snapshots
* Add stub method to drive snapshot transfer
* Store remote shard in forward proxy, merge unproxy methods
* On snapshot shard transfer, create a shard snapshot
* Unify logic for unproxifying forward and queue proxy
* Error snapshot transfer if shard is not a queue proxy
* Add remote shard function to request remote HTTP port
* Handle all specific shard types when proxifying
* Allow queue proxy for some shard holder snapshot methods
* Bring local and remote shard snapshot transfer URLs into transfer logic
* Expose optional shard transfer method parameter in REST and gRPC API
* Expose shard transfer method in list of active transfers
* Fix off-by-one error in queue proxy shard batch transfer logic
* Do not set max ack version for WAL twice, already set when finalizing
* Merge comment for two similar calls
* Use reqwest client to transfer and recover shard snapshot on remote
Using the reqwest client should be temporary. We better switch to a gRPC
call here eventually to use our existing channels. That way we don't
require an extra HTTP client (and dependency) just for this.
* Send queue proxy updates to remote when shard is transferred
* On shard queue transfer, set max WAL ack to last transferred
* Add safe queue proxy destructor, skip destructing in error
This adds a finalize method to safely destruct a queue proxy shard. It
ensures that all remaining updates are transferred to the remote, and
that the max acknowledged version for our WAL is released. Only then is
the queue proxy shard destructed unwrapping the inner local shard.
Our unproxify logic now ensures that the queue proxy shard remains if
transferring the updates fails.
* Clean up method driving shard snapshot transfer a bit
* Change default shard transfer method to stream records
This changes the default transfer method to stream records rather than
using a snaphsot transfer. We can switch this once snapshot transfer is
fully integrated.
* Improve error handling, don't panic but return proper error
* Do not unwrap in type conversions
* Update OpenAPI and gRPC specification
* Resolve and remove some TODOs
* During shard snapshot transfer, use REST port from config
* Always release max acknowledged WAL version on queue proxy finalize
* Rework queue unproxying, transform into forward proxy to handle errors
When a queue or forward proxy shard needs to be unproxified into a local
shard again we typically don't have room to handle errors. A queue proxy
shard may error if it fails to send updates to the remote shard, while a
forward proxy does not fail at all when transforming.
We now transfer queued updates before a shard is unproxified. This
allows for proper error handling. After everything is transferred the
shard is transformed into a forward proxy which can eventually be safely
unproxified later.
* Add trace logging for transferring queue proxy updates in batch
* Simplify snapshot method conversion from gRPC
* Remove remote shard parameter
* Add safe guard to queue proxy handler, panic in debug if not finalized
* Improve safety and architecture of queue proxy shard
Switch from an explicit finalized flag to an outer-inner architecture.
This improves the interface and robustness of the type.
* Do not panic on drop if already unwinding
* Make REST port interface in channel service for local node explicitly
* Recover shard on remote over gRPC, remove reqwest client
* Use shard transfer priority for shard snapshot recovery
* Remove obsolete comment
* Simplify qualified path with use
* Don't construct URLs ourselves as a string, use `parse` and `set_port`
* Use `set_path` when building shard download URL
* Fix error handling in queue to forward proxy transformation
Before, we didn't handle finalization errors properly. If this failed,
tie shard would be lost. With this change the queue proxy shard is put
back.
* Set default shard transfer method to stream records, eliminate panics
* Fix shard snapshot transfer not correctly aborting due to queue proxy
When a shard transfer fails (for any reason), the transfer is aborted.
If we still have a queue proxy shard it should also be reverted, and
collected updates should be forgotten. Before this change it would try
to send all collected updates to the remote, even if the transfer
failed.
* Review fixes
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Review fixes
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Initiate forward and queue proxy shard in specialized transfer methods
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add consensus interface to shard transfer, repurpose dispatcher (#2873)
* Add shard transfer consensus interface
* Integrate shard transfer consensus interface into toc and transfer logic
* Repurpose dispatcher for getting consensus into shard transfer
* Derive clone
* Mark consensus as unused for now
* Use custom dispatcher with weak ref to prevent Arc cycle for ToC
* Add comment on why a weak reference is used
* Do exhaustive match in shard unproxy logic
* Restructure match statement, use match if
* When queue proxifying shard, allow forward proxy state if same remote
* Before retrying a shard transfer after error, destruct queue proxy
* Synchronize consensus across all nodes for shard snapshot transfer (#2874)
* Move await consensus commit functions into channel service
* Add shard consensus method to synchronize consensus across all nodes
* Move transfer config, channels and local address into snapshot transfer
* Await other nodes to reach consensus before finalizing shard transfer
* Do not fail right away awaiting consensus if still on older term
Instead, give the node time to reach the same term.
* Fix `await_commit_on_all_peers` not catching peer errors properly
* Change return type of `wait_for_consensus_commit` to `Result`
This is of course more conventional, and automatically sets `must_use`.
* Explicitly note number of peers when awaiting consensus
* Before consensus sync, wait for local shard to reach partial state
* Fix timeout error handling when waiting for replica set state
* Wait for replica set to have remote in partial state instead
* Set `(Partial)Snapshot` states for shard snapshot transfer through consensus (#2881)
* When doing a shard snapshot transfer, set shard to `PartialSnapshot`
* Add shard transfer method to set shard state to partial
It currently uses a naive implementation. Using a custom consensus
operation to also confirm a transfer is still active will be implemented
later.
* Add consensus snapshot transfer operation to change shard to partial
The operation `ShardTransferOperations::SnapshotRecovered` is called
after the shard snapshot is recovered on the remote and it progresses
the transfer further.
The operation sets the shard state from `PartialSnapshot` to `Partial`
and ensures the transfer is still active.
* Confirm consensus put shard into partial state, retry 3 times
* Get replica set once
* Add extensive shard snapshot transfer process docs, clean up function
* Fix typo
* Review suggestion
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Add delay between consensus confirmation retries
* Rename retry timeout to retry delay
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* On replicate shard, remember specified method
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Fix paste-bugs in `snapshot_service.proto`
* Add shard snapshot gRCP API definition
* Add validation to shard snapshot gRPC API definition
* Implement conversions between gRPC and `collection` types
* Extract shard snapshot API implementation into common sub-module
* Implement shard snapshot gRPC API
* Generate gRPC docs
* Refactor `ShardSnapshots` gRPC service to be internal API only
* fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Move `ShardSnapshotRecoverResponse` to `shard_snapshots_service.proto`
* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Update `api/src/grpc/qdrant.rs`
* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Update gRPC docs
* Switch `ShardSnapshots` gRPC service to use `validate_and_log` instead of `validate`
* Properly route validation for cluster operation types
* Add `must_not_match` validation type
* Validate move shard, source and target address must not be the same
* Improve validation error message if source and target shard are the same
* Set up validation routing for shard operations in gRPC
* Validate shard operation timeout to be 1 or higher
* Validate move shard in gRPC, source/target address must not be the same
* Add test checking we cannot move shard to ourselves
* Unify different peer validation for move shard request
* Extend GeoPolygon to support interiors (#2315)
Per GeoJson, we should support polygon with exterior and interiors (holes on the surface) in Geo Filter by Polygon(#795). This commit extend current GeoPolygon filter to accept interiors. It includes:
1. changes to proto and internal GeoPolygon struct, and validation fn
2. add and refactor some tests
3. add integration test
* add gRPC geo_polygon validation
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
