* first PR implementation (#2865)
- fetch offset id
- restructure tests
- only let order_by with numeric
- introduce order_by interface
cargo fmt
update openapi
calculate range to fetch using offset + limit, do some cleanup
enable index validation, fix test
Fix pagination
add e2e tests
make test a little more strict
select numeric index on read_ordered_filtered
add filtering test 🫨
fix filtering on order-by
fix pip requirements
add grpc interface, make read_ordered_filtered fallible
fmt
small optimization of `with_payload` and `with_vector`
refactor common logic of point_ops and local_shard_operations
Make filtering test harder and fix limit for worst case
update openapi
small clarity refactor
avoid extra allocation when sorting with offset
stream from numeric index btree instead of calculating range
use payload to store order-by value, instead of modifying Record interface
various fixes:
- fix ordering at collection level, when merging shard results
- fix offset at segment level, to take into account also value offset
- make rust tests pass
remove unused histogram changes
fix error messages and make has_range_index exhaustive
remove unused From impl
Move OrderBy and Direction to segment::data_types::order_by
Refactor normal scroll_by in local_shard_operations.rs
More cleanup + rename OrderableRead to StreamWithValue
empty commit
optimization for merging results from shards and segments
fix case of multi-valued fields
fix IntegerIndexParams name after rebase
precompute offset key
use extracted `read_by_id_stream`
Expose value_offset to user
- rename offset -> value_offset
- extract offset value fetching logic
* remove offset functionality when using order_by
* include order_by in ForwardProxyShard
* extra nits
* remove histogram changes
* more nits
* self review
* resolve conflicts after rebase, not enable order-by with datetime index schema
* make grpc start_from value extendable
* gen grpc docs
---------
Co-authored-by: kwkr <kawka.maciej.93@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* This changes the returning of StorageError::BadInput to StorageError::AlreadyExists.
* This adds AlreadyExists error type to StorageError enum
* This implements already_exists function to handle AlreadyExists StorageError.
* This adds error to status code for StorageError::AlreadyExists using tonic::Code::AlreadyExists
* This adds Error type for actix to handle AlreadyExists Storage error using Error::CONFLICT
* This adds using HttpResponse::Conflict for building HttpResp in case of StorageError
* This adds StatusCode and description for HttpError caused by StorageError
* fix integration test
* rename is_collection_exists -> collection_exists
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Fix merge of vector operations, don't drop earlier updates on same point
* Add unit test for vector struct merging
* Add integration test for fix, also covering bug report
* feat: Expose git commit id in the health check endpoint
* fix: CI errors
* test: Add test for health check api
* feat: Add / endpoint to openapi schema
* Make git commit hash optional
* ci: Enable debugging setup-protoc action
* Install later protobuf compiler through GitHub Action
* Disable debug mode for setup-protoc job
* refactor: Use commit instead of commit_id
* fix: Use commit instead of commit_id gRPC docs
* test: Update ping API test
* refactor: Rename ping api to root api
---------
Co-authored-by: timvisee <tim@visee.me>
* fix missing indexing values on set-payload operation
* Update function documentation, use conventional warning
* Don't deep match statements
* test: Update of payload on already indexed payload should work (#3253)
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
* sparse vectors with large indices support
* fix unit tests
* are you happy clippy
* review remarks; acc test for inverted index
* Test large index for sparse vectors
set unique name to the test collection
Revert "fix consensus tests"
This reverts commit 93afe59f6ba663d5a89bbf25e3418b873fd72b16.
fix rebase mistake
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* create and connect discovery http and grpc interfaces
* add openapi tests
* fix bad rebase
* Add better descriptions
* remove numpy from openapi tests
* fix rebase artifact
* remove already addressed TODO
* add more tests
* 🤡🔫 (cfg batch handler)
* add timeout query param for discover requests
* More gRPC validation
* make fields pydantic_openapi_generator_v3 friendly
* `context_pairs` -> `context` with struct for pairs
* discovery api is only discovery or context,
move struct description to fields
---------
Co-authored-by: timvisee <tim@visee.me>
* add timeout query param for search requests
* enable timeout for recommend requests
* Add query timeout for group by requests
* update openapi models
* Don't decrease timeout after recommend preprocessing
* Add openapi test
* code review
* add timeout to individual group by requests, non-decreasing
* handle timeout for discover
* Update timeout field tag in SearchBatchPoints
message
* Extend GeoPolygon to support interiors (#2315)
Per GeoJson, we should support polygon with exterior and interiors (holes on the surface) in Geo Filter by Polygon(#795). This commit extend current GeoPolygon filter to accept interiors. It includes:
1. changes to proto and internal GeoPolygon struct, and validation fn
2. add and refactor some tests
3. add integration test
* add gRPC geo_polygon validation
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* allow raw vectors in recommend
* add openapi test
* keep point id input separately forever in grpc
* minor fix
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add healthz, livez and readyz endpoints
* Describe healthz, livez and readyz endpoints in OpenAPI definition
* Add integration test for healthz, livez and readyz endpoints
* Fix typo
* On segment flush, read-lock all segments to prevent CoW between flushes
For example, we have a point on an immutable segment. If we use a
set-payload operation, we do copy-on-write. The point from immutable
segment A is deleted, the updated point is stored on appendable segment
B.
Because of flush ordering segment B (appendable) is flushed before
segment A (not-appendable). If the copy-on-write operation happens in
between, the point is deleted from A but the new point in B is not
persisted. We cannot recover this by replaying the WAL in case of a
crash because the point in A does not exist anymore, making
copy-on-write impossible.
Locking all segments prevents copy-on-write operations from occurring in
between flushes.
* Return proper status on set payload operations in segment
* Disable propagating point deletions to its vectors, it looses vectors
* Update vector tests after disabling point delete propagation to vectors
* Implement retry with exponential backoff for read-locking all segments
* Use try_read_for rather than sleeping after lock attempts
* Simplify locking many, just lock one by one
Refs: <https://github.com/qdrant/qdrant/pull/2527>
Co-authored-by: generall <andrey@vasnetsov.com>
* Comment out now obsolete test
* Error handling in method getting segment locks by ID
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Clean up imports
* Clean up whitespace
* Use drop collection function from helpers
* Use parameterized integration tests to test on_disk variations
* Promote on_disk_vectors parameter into dedicated fixture
* Merge basic retrieve test with parameterization
* Flatten functions
* Split validation test functions
* Don't parameterize on_disk states when testing validation
* Parameterize init_from_collection test sequence
* Parameterize on_disk_payload in tests with fixture
* Parameterize new batch update tests
* Fix integration test collection setup distance after incorrect merge
* Reoptimize segments on quantization mismatch
* Add collection quantization params to collection update REST endpoint
* Do not require rebuild of quantization for on_disk change
* Add collection quantization params to collection update gRPC endpoint
* Add option to update vector specific quantization config to REST API
* Add option to update vector specific quantization config to gRPC API
* Update OpenAPI and gRPC specification
* Fix quantization mismatch detecting not working as expected
* Fix config mismatch optimizer not using vector specific quantization
* Add unit test for quantization in config mismatch optimizer
* Add some quantization params to collection update integration test
* Reformat
* Apply suggestions from code review
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Remove if-statement, simply return
* Move quantization params in named vector in integration test
* Test updating quantization params in multivec integration test
* Fix gRPC docs
* Fix quantization rebuild on changing enabled state on indexed segment
* allow disabling quantization config
* fmt
* clippy
* compare all params of the quantization for rebuild
* explicitly use params on update
* test for disabling quantization
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* unit-test for updating multivec config
* Remove single vector support from vector params update in REST API
With this change, a user is always required to specify vector names when
updating their parameters. Updating vector parameters in a collection
with a single vector is still possible by providing an empty name. This
is very reasonable as updating vector parameters on a single vector
collection is a bad pattern, since parameters can be set on the
collection itself.
* Update integration tests to reflect REST API change
* Update OpenAPI specification
* Remove obsolete VectorsConfigDiff functions, update docs
* Update OpenAPI specification
* Add name validation to update collection endpoint for vector params
---------
Co-authored-by: timvisee <tim@visee.me>
* Add update vector config type
* Add mutable getter to update vector config
* Add update vector config to update collection REST operation, reorder
* Add logic to update vector HNSW in collection
* Update OpenAPI specification
* If HNSW diff is empty, consider it as unset
* Update descriptions to clarify an empty HNSW diff unsets
* Make new vector HNSW diff update existing diff
This means that the existing vector HNSW diff is kept intact. Specified
fields are updated in the existing diff. An empty HNSW diff object may
be provided to unset the diff.
* Add update vector config to update collection gRPC operation
* Update gRPC docs
* Use vector specific HNSW config in integration test
* Extract & improve gRPC type conversions, fix collection update with None
* Validate vector specific HNSW config in collection update gRPC endpoint
* Explicitly test we do not rebuild on on_disk change
* Use new method to recreate optimizers
* Don't test on_disk change anymore
* Reuse collection write lock to save, do not relock
* Fix invalid update collection request body in OpenAPI test
* Simplify update collection test
* Add update collection test for HNSW parameters
* Extract vector param update logic into local functions
* Reformat
* Transform UpdateVectorParams into VectorParamsDiff
* Transform UpdateVectorsConfig into VectorsConfigDiff
* Combine serde attributes
* Make HNSW diff empty check generic over DiffConfig types
* Move DiffConfig implementation
* Fix typo
* Add github action to codespell master on push and PRs
* Add rudimentary codespell config
* some skips
* fix some ambigous typos
* [DATALAD RUNCMD] run codespell throughout
=== Do not change lines below ===
{
"chain": [],
"cmd": "codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Add dev branch as target for the workflow
* rename `Lookup` -> `RetrievedLookup`
* feat: expose lookup in groups
- make `with_vectors` and `with_payload` optional
- include in grpc
- move GroupId to inside of PointGroup message
- include `WithLookup` inside of `GroupRequest`, doesn't make sense to keep them separated
* chore: remove allow(dead_code) from builder
* feat: introduce `WithLookupInterface` to allow specifying only the name of the lookup collection
* fix: WithLookupInterface with collection name only brings all payload and vectors
* fix: update grpc docs
* add tests and set better defaults
* remove `RetrievedLookup` from proto
* short-circuit `fill_search_result_with_payload` when it's not needed
* cargo fmt
* proto: revert putting `GroupId` inside `PointGroup`
* update grpc docs
* Remove WithLookupInterface sugar from grpc
- set with_vectors of lookup default to false
- set with_payload of lookup default to true
* update integration tests
* use singular lookup
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
