* fix default opeapi file path
* better capture of process id in integration-tests.sh
* add pycache to gitignore
* add extra case for order_by test
* parse dates without timezone but with decimals correctly
* update payload_indexing test
* add extra test for space-separated date
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* docs, cardinality estimation, grpc not implemented yet
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* Impl min_should clause in REST API
* perform cardinality estimation by estimating cardinalities of intersection and combining as union
* add openapi spec with docs update
* add integration test
* Impl min_should clause in gRPC
* Cargo fmt & clippy
* Fix minor comments
* add equivalence test between min_should and must
* shortcut at min_count matches
* use `Filter::new_*` whenever possible
* Add missing min_should field
* Fix gRPC field ordering & remove deny_unknown_fields
* Empty commit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* support \`start_from\`: DateTime
* add order by datetime test
* generate openapi models and grpc docs
* fixup after rebase
* allow string representation of datetime in grpc
* add TODO
* fix `.start_from()`
* use custom deserialization on datetime
* Extend datetime formats in REST API (#3529)
* Fix linter problem
* Support indexing and condition checking and optimize tests
* Update lib/segment/src/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Update lib/segment/src/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Update lib/segment/src/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add some more tests for wrong or unsupported datetime format
* Reformat
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
* merge range and datetime range into one interface
* remove separate datetime_range from FieldCondition
* review fixes, remove code duplication. Thanks @xzfc
* update openapi test
* first PR implementation (#2865)
- fetch offset id
- restructure tests
- only let order_by with numeric
- introduce order_by interface
cargo fmt
update openapi
calculate range to fetch using offset + limit, do some cleanup
enable index validation, fix test
Fix pagination
add e2e tests
make test a little more strict
select numeric index on read_ordered_filtered
add filtering test 🫨
fix filtering on order-by
fix pip requirements
add grpc interface, make read_ordered_filtered fallible
fmt
small optimization of `with_payload` and `with_vector`
refactor common logic of point_ops and local_shard_operations
Make filtering test harder and fix limit for worst case
update openapi
small clarity refactor
avoid extra allocation when sorting with offset
stream from numeric index btree instead of calculating range
use payload to store order-by value, instead of modifying Record interface
various fixes:
- fix ordering at collection level, when merging shard results
- fix offset at segment level, to take into account also value offset
- make rust tests pass
remove unused histogram changes
fix error messages and make has_range_index exhaustive
remove unused From impl
Move OrderBy and Direction to segment::data_types::order_by
Refactor normal scroll_by in local_shard_operations.rs
More cleanup + rename OrderableRead to StreamWithValue
empty commit
optimization for merging results from shards and segments
fix case of multi-valued fields
fix IntegerIndexParams name after rebase
precompute offset key
use extracted `read_by_id_stream`
Expose value_offset to user
- rename offset -> value_offset
- extract offset value fetching logic
* remove offset functionality when using order_by
* include order_by in ForwardProxyShard
* extra nits
* remove histogram changes
* more nits
* self review
* resolve conflicts after rebase, not enable order-by with datetime index schema
* make grpc start_from value extendable
* gen grpc docs
---------
Co-authored-by: kwkr <kawka.maciej.93@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* This changes the returning of StorageError::BadInput to StorageError::AlreadyExists.
* This adds AlreadyExists error type to StorageError enum
* This implements already_exists function to handle AlreadyExists StorageError.
* This adds error to status code for StorageError::AlreadyExists using tonic::Code::AlreadyExists
* This adds Error type for actix to handle AlreadyExists Storage error using Error::CONFLICT
* This adds using HttpResponse::Conflict for building HttpResp in case of StorageError
* This adds StatusCode and description for HttpError caused by StorageError
* fix integration test
* rename is_collection_exists -> collection_exists
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Fix merge of vector operations, don't drop earlier updates on same point
* Add unit test for vector struct merging
* Add integration test for fix, also covering bug report
* feat: Expose git commit id in the health check endpoint
* fix: CI errors
* test: Add test for health check api
* feat: Add / endpoint to openapi schema
* Make git commit hash optional
* ci: Enable debugging setup-protoc action
* Install later protobuf compiler through GitHub Action
* Disable debug mode for setup-protoc job
* refactor: Use commit instead of commit_id
* fix: Use commit instead of commit_id gRPC docs
* test: Update ping API test
* refactor: Rename ping api to root api
---------
Co-authored-by: timvisee <tim@visee.me>
* fix missing indexing values on set-payload operation
* Update function documentation, use conventional warning
* Don't deep match statements
* test: Update of payload on already indexed payload should work (#3253)
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
* sparse vectors with large indices support
* fix unit tests
* are you happy clippy
* review remarks; acc test for inverted index
* Test large index for sparse vectors
set unique name to the test collection
Revert "fix consensus tests"
This reverts commit 93afe59f6ba663d5a89bbf25e3418b873fd72b16.
fix rebase mistake
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* Read-only API keys
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Correct placement of OpenAPI security
Place regex dep with actix/tonic
* Read-only API keys
* Replace with pytests
* API Key tests run on the same job
* Drop allow dead-code
* Rename setting key
* Containerized tests
* No special config files
* DRY
* refactor: re-use can_write method
* refactor: replace static by constants
* refactor: get PID from `$!`
* refactor: use explicit brackets on boolean condition
* style: fix identation
* small fixes + account for new APIs
* specify security in openapi
* small fix + chmod for .sh testfile
* add best-efford check for api consistency
---------
Co-authored-by: Amr Hassan <amr.hassan@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* create and connect discovery http and grpc interfaces
* add openapi tests
* fix bad rebase
* Add better descriptions
* remove numpy from openapi tests
* fix rebase artifact
* remove already addressed TODO
* add more tests
* 🤡🔫 (cfg batch handler)
* add timeout query param for discover requests
* More gRPC validation
* make fields pydantic_openapi_generator_v3 friendly
* `context_pairs` -> `context` with struct for pairs
* discovery api is only discovery or context,
move struct description to fields
---------
Co-authored-by: timvisee <tim@visee.me>
* add timeout query param for search requests
* enable timeout for recommend requests
* Add query timeout for group by requests
* update openapi models
* Don't decrease timeout after recommend preprocessing
* Add openapi test
* code review
* add timeout to individual group by requests, non-decreasing
* handle timeout for discover
* Update timeout field tag in SearchBatchPoints
message
* Extend GeoPolygon to support interiors (#2315)
Per GeoJson, we should support polygon with exterior and interiors (holes on the surface) in Geo Filter by Polygon(#795). This commit extend current GeoPolygon filter to accept interiors. It includes:
1. changes to proto and internal GeoPolygon struct, and validation fn
2. add and refactor some tests
3. add integration test
* add gRPC geo_polygon validation
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* allow raw vectors in recommend
* add openapi test
* keep point id input separately forever in grpc
* minor fix
---------
Co-authored-by: generall <andrey@vasnetsov.com>
