* Add healthz, livez and readyz endpoints
* Describe healthz, livez and readyz endpoints in OpenAPI definition
* Add integration test for healthz, livez and readyz endpoints
* Fix typo
* On segment flush, read-lock all segments to prevent CoW between flushes
For example, we have a point on an immutable segment. If we use a
set-payload operation, we do copy-on-write. The point from immutable
segment A is deleted, the updated point is stored on appendable segment
B.
Because of flush ordering segment B (appendable) is flushed before
segment A (not-appendable). If the copy-on-write operation happens in
between, the point is deleted from A but the new point in B is not
persisted. We cannot recover this by replaying the WAL in case of a
crash because the point in A does not exist anymore, making
copy-on-write impossible.
Locking all segments prevents copy-on-write operations from occurring in
between flushes.
* Return proper status on set payload operations in segment
* Disable propagating point deletions to its vectors, it looses vectors
* Update vector tests after disabling point delete propagation to vectors
* Implement retry with exponential backoff for read-locking all segments
* Use try_read_for rather than sleeping after lock attempts
* Simplify locking many, just lock one by one
Refs: <https://github.com/qdrant/qdrant/pull/2527>
Co-authored-by: generall <andrey@vasnetsov.com>
* Comment out now obsolete test
* Error handling in method getting segment locks by ID
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Clean up imports
* Clean up whitespace
* Use drop collection function from helpers
* Use parameterized integration tests to test on_disk variations
* Promote on_disk_vectors parameter into dedicated fixture
* Merge basic retrieve test with parameterization
* Flatten functions
* Split validation test functions
* Don't parameterize on_disk states when testing validation
* Parameterize init_from_collection test sequence
* Parameterize on_disk_payload in tests with fixture
* Parameterize new batch update tests
* Fix integration test collection setup distance after incorrect merge
* Reoptimize segments on quantization mismatch
* Add collection quantization params to collection update REST endpoint
* Do not require rebuild of quantization for on_disk change
* Add collection quantization params to collection update gRPC endpoint
* Add option to update vector specific quantization config to REST API
* Add option to update vector specific quantization config to gRPC API
* Update OpenAPI and gRPC specification
* Fix quantization mismatch detecting not working as expected
* Fix config mismatch optimizer not using vector specific quantization
* Add unit test for quantization in config mismatch optimizer
* Add some quantization params to collection update integration test
* Reformat
* Apply suggestions from code review
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Remove if-statement, simply return
* Move quantization params in named vector in integration test
* Test updating quantization params in multivec integration test
* Fix gRPC docs
* Fix quantization rebuild on changing enabled state on indexed segment
* allow disabling quantization config
* fmt
* clippy
* compare all params of the quantization for rebuild
* explicitly use params on update
* test for disabling quantization
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* unit-test for updating multivec config
* Remove single vector support from vector params update in REST API
With this change, a user is always required to specify vector names when
updating their parameters. Updating vector parameters in a collection
with a single vector is still possible by providing an empty name. This
is very reasonable as updating vector parameters on a single vector
collection is a bad pattern, since parameters can be set on the
collection itself.
* Update integration tests to reflect REST API change
* Update OpenAPI specification
* Remove obsolete VectorsConfigDiff functions, update docs
* Update OpenAPI specification
* Add name validation to update collection endpoint for vector params
---------
Co-authored-by: timvisee <tim@visee.me>
* Add update vector config type
* Add mutable getter to update vector config
* Add update vector config to update collection REST operation, reorder
* Add logic to update vector HNSW in collection
* Update OpenAPI specification
* If HNSW diff is empty, consider it as unset
* Update descriptions to clarify an empty HNSW diff unsets
* Make new vector HNSW diff update existing diff
This means that the existing vector HNSW diff is kept intact. Specified
fields are updated in the existing diff. An empty HNSW diff object may
be provided to unset the diff.
* Add update vector config to update collection gRPC operation
* Update gRPC docs
* Use vector specific HNSW config in integration test
* Extract & improve gRPC type conversions, fix collection update with None
* Validate vector specific HNSW config in collection update gRPC endpoint
* Explicitly test we do not rebuild on on_disk change
* Use new method to recreate optimizers
* Don't test on_disk change anymore
* Reuse collection write lock to save, do not relock
* Fix invalid update collection request body in OpenAPI test
* Simplify update collection test
* Add update collection test for HNSW parameters
* Extract vector param update logic into local functions
* Reformat
* Transform UpdateVectorParams into VectorParamsDiff
* Transform UpdateVectorsConfig into VectorsConfigDiff
* Combine serde attributes
* Make HNSW diff empty check generic over DiffConfig types
* Move DiffConfig implementation
* Fix typo
* Add github action to codespell master on push and PRs
* Add rudimentary codespell config
* some skips
* fix some ambigous typos
* [DATALAD RUNCMD] run codespell throughout
=== Do not change lines below ===
{
"chain": [],
"cmd": "codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Add dev branch as target for the workflow
* rename `Lookup` -> `RetrievedLookup`
* feat: expose lookup in groups
- make `with_vectors` and `with_payload` optional
- include in grpc
- move GroupId to inside of PointGroup message
- include `WithLookup` inside of `GroupRequest`, doesn't make sense to keep them separated
* chore: remove allow(dead_code) from builder
* feat: introduce `WithLookupInterface` to allow specifying only the name of the lookup collection
* fix: WithLookupInterface with collection name only brings all payload and vectors
* fix: update grpc docs
* add tests and set better defaults
* remove `RetrievedLookup` from proto
* short-circuit `fill_search_result_with_payload` when it's not needed
* cargo fmt
* proto: revert putting `GroupId` inside `PointGroup`
* update grpc docs
* Remove WithLookupInterface sugar from grpc
- set with_vectors of lookup default to false
- set with_payload of lookup default to true
* update integration tests
* use singular lookup
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Dependabot couldn't find the original pull request head commit, 7f8b1be2d712f4552c8b4cd2377149c8971b1516.
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* one more test for optional vectors
* remove check
* clippy
* Add test retrieving non existing vector name
---------
Co-authored-by: timvisee <tim@visee.me>
* Flatten points selector in delete vectors REST endpoint
This makes the vector deletion API consistent with other endpoints using
the point selector.
* Update tests for flattened delete vectors point selector
* make api consistent with existing ones
* update openapi
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* nested object filter
* code review
* add support for must_not in nested
* extract functions
* support and test must_not in SimpleConditionChecker
* add index matching unit test (to be continued)
* remove extra clone
* test with should
* WIP: Nested object filter suggestions (#1855)
* switch to bitvec
* fix clippy
* more tests
* fmt
* fix some tests
* add test with text
* support for nested should
* do not rely on indexes for nested queries & fix test
* use index to make index-aware checks in nested payload
* fix value-count tests
* re-fa-cto-ring
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add integration test for optional named vector update/delete endpoints
* Update optional named vector integration tests for API change
* Fix update vectors endpoint not validating inner vectors
* Fix failing test due to unspecified order
* Add update/delete REST endpoints
* Add update/delete vector REST endpoints to OpenAPI specification
* Fix update vector with no changes not triggering a validation error
* Update OpenAPI specification for updated request type
* feat: use 0 as disable for indexing and memmap
* amend: edit test, update doc comments, remove validation
* fix: use constant instead of literal, reword doc comments
* test: update test for loop and timeout
* fix: remove validation on OptimizerConfigDiff and fix test_validatioin.py
* fix: remove validation from build.rs, remove internal hack from doccomments
* fix: update grpc docs
* revert max_optimization_threads validation in test
* update config comments and minor test change
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add QuantizationConfigDiff type
* Add quantization config diff to vector parameters
* Prefer vector config over collection config for quantization
* Update OpenAPI specification
* Validate quantization configuration quantile in 0.5-1.0 range
As per https://github.com/qdrant/qdrant/pull/1681
* Add test if check if vector specific quantization config is persisted
* Alias quantization to quantization_config in vector parameters
* Remove quantization config diff, use full vector specific config instead
* Regenerate OpenAPI specification and gRPC docs
* Fix compilation error
* Add error handling to quantization config conversions
* Fix quantization integration test, make HNSW test stricter
* Validate VectorConfig/VectorParams, remove obsolete validation
* Add HNSW config diff to vector parameters
* Validate params in collection config
* Add HNSW config to segment vector data config
* Add VectorsConfig params iterator for more elegant conversions
* Prefer vector HNSW config over collection config for building HNSW index
* Base segment vector param HNSW config on collection config
* General improvements
* Rewrite HNSW ef_construct extract function to also consider vector configs
* Update OpenAPI specification
* Add test to check if vector specific HNSW config is persisted
* review changes
* review changes
* Regenerate gRPC docs
* Fix test on Windows
* Regenerate OpenAPI specification
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* feat: add wait parm for all snapshot API
* update openAPI
* avoid unwrap
* remove all unwarp
* return HTTP 202 when no waiting on snapshot creation
* return 202 when non wait
* update open API
* update param on python test
* update and test api
* optimize the test
* return 202 when non-wait delete
* recover fixes
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* actix validation
* add check for memmap/indexing_threshold
* fix actix json settings
* Validate settings configuration on start, print pretty warnings on fail
* Add more validation rules for settings and nested types
* Move shared validation logic into collection/operations
* Show validation warning in log when loading some internal configs
* Show prettier actix JSON validation errors
* Stubs for pretty handling of query errors, reformat validation errors
* Use crate flatten function, the hard work was already done for us
We don't have to flatten validation errors into their qualified field
names ourselves because there is a utility function for this.
* Configure actix path validator
* Actix endpoints don't require public
* Extend validation to more actix types
* Validate all remaining actix path and query properties
* Rephrase range validation messages to clearly describe they're inclusive
* Validate all query params to respond with pretty deserialize errors
* Nicely format JSON payload deserialize error responses
* Improve error reporting for upsert point batches
* Add basic validation test that checks a path, query and payload value
* Add some validation constraints
* Add simple validation error render test
* Update Cargo.lock
---------
Co-authored-by: timvisee <tim+github@visee.me>
* add minimal working is_null filter
* add is_null condition to grpc api (backward compatible)
* add unit tests is_null and is_empty conditions
* add is_null to points.proto file
* add some failing OpenAPI tests
* fix a failing test due to change in collection data
* refactor MultiValue's check for is_null
* fix is_empty condition not picking up "key":[]
* remove duplicate OpenAPI integration test
* reuse same variable in condition checker tests
* update grpc docs
* fix is_null cardinality estimation to match is_empty
* update openapi specs
* remove unused debug statements
* add new test points to original test_collection
* fix failing tests according to newly added points
* add the `"key":[null]` test_case
* Fix incorrect metrics value for cluster commit
* Rewrite metrics logic, don't use registry, write values directly
* Only report REST timings for requests having HTTP 200 response
* Limit metrics reporting of endpoints to whitelist
The whitelist contains a selection of search, recommend and upsert endpoints.
* Add MetricsParam, remove detail level, keep anonymize
* Request metrics in basic API test
* Specify content type for metrics endpoint
* Add OpenAPI test for metrics endpoint, remove from basic API test
This test probes for some strings that must exist in the output
* Add note that metrics endpoint whitelist must be sorted
