* Add peer metadata state to consensus, update on sync local state
* Minor formatting improvements
* Use Qdrant version from constant
* Initialize empty metadata if not present on disk
* Refactor old parameter name
* Logging improvements based on review feedback
* Assert that Qdrant version matches crate version
* Do not use a custom (de)serializer for peer metadata
* Use semver directly for handling Qdrant versions
* Rename is_outdated to is_different_version
* Assert Qdrant version in build.rs
* Disable updating peer metadata until Qdrant 1.9
We do this, because if a node is running 1.8, there may still be nodes
running 1.7. Those nodes do not support this operation.
* clippy
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add unique token to clock tags
* Consider last token in clock map
* Mark potential problem with TODO, we might want to keep clock token
* Represent UUID in gRPC as bytes, keep token when converting gRPC to REST
* Do not panic on invalid UUID, propagate parse error
* Use helper function for clock tag conversion with error handling
* Fix accept/reject logic for clocks, accept higher/equal, update token
* Add test for accepting exactly the same last operation multiple times
* Update advance method description to elaborate on what is accepted
* Keep unique clock tag token in recovery point
* Fix WAL delta tests, don't expect rejection from forward proxy anymore
* Extend missing clocks in recovery point in a more efficient way
* Fix clock tick comparison logic in recovery point
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* use u64 as token
* Remove failable clock tag conversions, use direct conversion
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Change `ClockMap` to always reject tick 0
* Enable rejecting operations in `LocalShard::update`...
...and propagate updated `clock_tick` back to the caller
* WIP: Retry operation on *all* nodes (with new tick)...
...if *any* node rejected the operation
* fixup! WIP: Retry operation on *all* nodes (with new tick)...
Keep the same `clock` for the whole duration of `update`
* Add update response status for clock rejection (#3725)
* Add clock rejected update status
* Don't hide internal items in docs
* Propagate clock rejected status if operation clock is too old
* Fix test in which we manually define clock ticks
* Expect forward proxy rejection, node already has that operation
* Add constant for first valid clock tick, init clocks in recovery point
* In testing, advance all zero clocks to start ticking from 1
* Attempt to fix clock_set_clock_map_workflow
* Rename WalError::Rejected to ClockRejected
* Limit retry attempts when operation is rejected for an old clock tick
* Don't log rejected clock tag if tick was 0
* Cleanup `ClockMap`/`RecoveryPoint`
* Cleanup `ClockSet` and `ShardReplicaSet::update`
* Fix `clock_set_clock_map_workflow` test
* add debug assertions
* fmt
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* When restarting shard transfer, use sync if old or new one had sync set
* refactor restart transfer models
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add consensus operation to restart shard transfer with different config
* Require shard transfer restart to have a changed configuration
* implement api
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add first stubs for WAL delta shard transfer method
* Repurpose queue proxy, use it for transferring WAL diff as well
* Integrate WAL delta transfer is transfer selection logic
* Add WalDelta shard transfer type which is not exposed in public API
* Basic implementation of falling back to stream records transfer
* Share await_consensus_sync function
* Ask remote shard for recovery point
* During WAL delta transfer, resolve shard diff locally for recovery point
* Rebase on latest dev, support empty WAL diff
* Rebase on latest dev, support empty WAL diff
* Use partial snapshot state for WAL delta transfer
* Set cutoff point on remote shard after shard WAL delta transfer
* Set cutoff point on remote shard after stream records transfer
* During WAL delta transfer, set s tate from partial snapshot to partial
* Describe WAL delta transfer in a comment
* Allow updating cutoff point in stream records transfer to fail
* Do not set cutoff point on remote shard on WAL delta transfer
* Make await consensus sync logic easier to read and reason about
* Fix fallback to other shard transfer method on WAL delta transfer fail
* Various minor improvements
* Add TODO for just ignoring API unimplemented errors
* Only allow stream records cutoff point error if remote is older version
* Allow switching to partial to fail when falling back
* Only change shard state to partial if not in partial state already
* Change default shard transfer method back to stream records
* Add important TODO back
* Add WAL delta shard transfer method in gRPC
* Prefer configured shard transfer method as default
* Extract shard transfer fallback logic into separate function
* feat: Print commit id in dev container build workflow
* fix: Keep .git while building Qdrant binary in docker
* fix: Remove redundant printing of git commit id
* fix: Copy only git files first
* fix: Docker build should have commit id if present
* refactor: Use fewer lines of code
* feat: Use git commit id from build arg
* refactor: Use cleaner code
* Fix string reference issue
* fix: Dont print info log if commit is not set
* fix: Include git commit id in release containers
---------
Co-authored-by: timvisee <tim@visee.me>
* Add functions to propagate updating cutoff point from collection level
* Add gRPC endpoint to set cutoff point
* Lock highest and cutoff clock maps separately
* Add recovery shard replica set state
* Accept forced operations in partial snapshot state
* Fix switching into wrong state
* Add some helpful comments
* Deduplication in match statement
* Support set by key in low level.
* Rename key field.
* Format.
* Pass key.
* Format.
* Test.
* Clippy.
* Fix ci lint.
* Check grpc consistency.
* Update openapi.
* Fix empty key test case.
* Support array index.
* Format.
* Add test for non exists key.
* Clippy fix.
* Add idempotence test.
* Update index by updated payload.
* Add ut for utils.
* Add ut for 1 level key.
* Fix ut.
* Support no exits key.
* Fix test result.
* Fix after rebase
* handle wildcart insertion into non-existing array
* avoid double read of payload during update
* fix missing removing data from index in case if set_payload removes indexed field
---------
Co-authored-by: Shylock Hg <shylock@DESKTOP-40I855A>
Co-authored-by: Albert Safin <xzfcpw@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* docs, cardinality estimation, grpc not implemented yet
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* Impl min_should clause in REST API
* perform cardinality estimation by estimating cardinalities of intersection and combining as union
* add openapi spec with docs update
* add integration test
* Impl min_should clause in gRPC
* Cargo fmt & clippy
* Fix minor comments
* add equivalence test between min_should and must
* shortcut at min_count matches
* use `Filter::new_*` whenever possible
* Add missing min_should field
* Fix gRPC field ordering & remove deny_unknown_fields
* Empty commit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Add initial gRPC call for requesting WAL recovery point for remote shard
* Add remote shard method to request WAL recovery point
* Add recovery point type in gRPC, use it in recovery point functions
* Add function to extend recovery point with missing clocks from clock map
* Add new gRPC type for recovery point clocks
* Remove atomic loading, because we use regular integers now
* support \`start_from\`: DateTime
* add order by datetime test
* generate openapi models and grpc docs
* fixup after rebase
* allow string representation of datetime in grpc
* add TODO
* fix `.start_from()`
* use custom deserialization on datetime
* add benches for large MatchAny
* use HashSet for MatchAny
* use fnv hash
* make fnv workspace level dependency; apply clippy
* remove SmolStr from Keyword; Improve performance
* add bench for small number of keywords
* fix openapi
* fix performance issue
* apply integer optimization; create magic number constant
* merge range and datetime range into one interface
* remove separate datetime_range from FieldCondition
* review fixes, remove code duplication. Thanks @xzfc
* update openapi test
* first PR implementation (#2865)
- fetch offset id
- restructure tests
- only let order_by with numeric
- introduce order_by interface
cargo fmt
update openapi
calculate range to fetch using offset + limit, do some cleanup
enable index validation, fix test
Fix pagination
add e2e tests
make test a little more strict
select numeric index on read_ordered_filtered
add filtering test 🫨
fix filtering on order-by
fix pip requirements
add grpc interface, make read_ordered_filtered fallible
fmt
small optimization of `with_payload` and `with_vector`
refactor common logic of point_ops and local_shard_operations
Make filtering test harder and fix limit for worst case
update openapi
small clarity refactor
avoid extra allocation when sorting with offset
stream from numeric index btree instead of calculating range
use payload to store order-by value, instead of modifying Record interface
various fixes:
- fix ordering at collection level, when merging shard results
- fix offset at segment level, to take into account also value offset
- make rust tests pass
remove unused histogram changes
fix error messages and make has_range_index exhaustive
remove unused From impl
Move OrderBy and Direction to segment::data_types::order_by
Refactor normal scroll_by in local_shard_operations.rs
More cleanup + rename OrderableRead to StreamWithValue
empty commit
optimization for merging results from shards and segments
fix case of multi-valued fields
fix IntegerIndexParams name after rebase
precompute offset key
use extracted `read_by_id_stream`
Expose value_offset to user
- rename offset -> value_offset
- extract offset value fetching logic
* remove offset functionality when using order_by
* include order_by in ForwardProxyShard
* extra nits
* remove histogram changes
* more nits
* self review
* resolve conflicts after rebase, not enable order-by with datetime index schema
* make grpc start_from value extendable
* gen grpc docs
---------
Co-authored-by: kwkr <kawka.maciej.93@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Move CPU count function to common, fix wrong CPU count in visited list
* Change default number of rayon threads to 8
* Use CPU budget and CPU permits for optimizer tasks to limit utilization
* Respect configured thread limits, use new sane defaults in config
* Fix spelling issues
* Fix test compilation error
* Improve breaking if there is no CPU budget
* Block optimizations until CPU budget, fix potentially getting stuck
Our optimization worker now blocks until CPU budget is available to
perform the task.
Fix potential issue where optimization worker could get stuck. This
would happen if no optimization task is started because there's no
available CPU budget. This ensures the worker is woken up again to
retry.
* Utilize n-1 CPUs with optimization tasks
* Better handle situations where CPU budget is drained
* Dynamically scale rayon CPU count based on CPU size
* Fix incorrect default for max_indexing_threads conversion
* Respect max_indexing_threads for collection
* Make max_indexing_threads optional, use none to set no limit
* Update property documentation and comments
* Property max_optimization_threads is per shard, not per collection
* If we reached shard optimization limit, skip further checks
* Add remaining TODOs
* Fix spelling mistake
* Align gRPC comment blocks
* Fix compilation errors since last rebase
* Make tests aware of CPU budget
* Use new CPU budget calculation function everywhere
* Make CPU budget configurable in settings, move static budget to common
* Do not use static CPU budget, instance it and pass it through
* Update CPU budget description
* Move heuristic into defaults
* Fix spelling issues
* Move cpu_budget property to a better place
* Move some things around
* Minor review improvements
* Use range match statement for CPU count heuristics
* Systems with 1 or 2 CPUs do not keep cores unallocated by default
* Fix compilation errors since last rebase
* Update lib/segment/src/types.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/storage/src/content_manager/toc/transfer.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename cpu_budget to optimizer_cpu_budget
* Update OpenAPI specification
* Require at least half of the desired CPUs for optimizers
This prevents running optimizations with just one CPU, which could be
very slow.
* Don't use wildcard in CPU heuristic match statements
* Rename cpu_budget setting to optimizer_cpu_budget
* Update CPU budget comments
* Spell acquire correctly
* Change if-else into match
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename max_rayon_threads to num_rayon_threads, add explanation
* Explain limit in update handler
* Remove numbers for automatic selection of indexing threads
* Inline max_workers variable
* Remove CPU budget from ShardTransferConsensus trait, it is in collection
* small allow(dead_code) => cfg(test)
* Remove now obsolete lazy_static
* Fix incorrect CPU calculation in CPU saturation test
* Make waiting for CPU budget async, don't block current thread
* Prevent deadlock on optimizer signal channel
Do not block the optimization worker task anymore to wait for CPU budget
to be available. That prevents our optimizer signal channel from being
drained, blocking incoming updates because the cannot send another
optimizer signal. Now, prevent blocking this task all together and
retrigger the optimizers separately when CPU budget is available again.
* Fix incorrect CPU calculation in optimization cancel test
* Rename CPU budget wait function to notify
* Detach API changes from CPU saturation internals
This allows us to merge into a patch version of Qdrant. We can
reintroduce the API changes in the upcoming minor release to make all of
it fully functional.
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* feat: Print commit id in dev container build workflow
* fix: Keep .git while building Qdrant binary in docker
* fix: Remove redundant printing of git commit id
* fix: Copy only git files first
* fix: Docker build should have commit id if present
* refactor: Use fewer lines of code
* Merge serde attributes
* Remove obsolete conversion
* Add integer type with parameters
* Make integer lookup and range parameters non-optional
* Add parameterized integer index types test
Co-authored-by: Di Zhao <diz@twitter.com>
* Cleanup
---------
Co-authored-by: Di Zhao <diz@twitter.com>
* add checksum to SnapshotDescription
* implement storing snapshot checksums in a file
* Don't serialize checksum if it's None for backwards compatibility
* Remove hex dependency, use Rust std formatter for this
* Do not error if we cannot remove checksum file for snapshot
Some snapshots may not have a corresponding checksum file. Maybe it was
created in an older Qdrant version that didn't have support for this, or
a user hasn't provided any.
* Add debug message when hashing snapshot, can be expensive on large files
* Inline debug messages
* Add checksum to shard snapshots
* If creating snapshot fails, delete snapshot target and checksum file
* Use Rust idiomatic ok() and improve debug messages
* Use correct snapshot checksum paths, clean up after shard snapshot
* Use better path type in get_checksum_path
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* feat: Expose git commit id in the health check endpoint
* fix: CI errors
* test: Add test for health check api
* feat: Add / endpoint to openapi schema
* Make git commit hash optional
* ci: Enable debugging setup-protoc action
* Install later protobuf compiler through GitHub Action
* Disable debug mode for setup-protoc job
* refactor: Use commit instead of commit_id
* fix: Use commit instead of commit_id gRPC docs
* test: Update ping API test
* refactor: Rename ping api to root api
---------
Co-authored-by: timvisee <tim@visee.me>
* Make point, indexed point and vector counts optional in collection info
* Update OpenAPI specification and gRPC docs
* Use functional entry API
* Mark point/vector count fields as deprecated
* Use internal collection info structure to remove internal optionals
* Don't deprecate, but mark point counts as approximate
