* Strict mode config: Max collection size
* api specs
* Add tests + set/update payload check
* Improve function names and add comments
* rename config to separate vectors and payload
* fix tests
* Adjust configs docs
* add benchmark
* improve performance by caching shard info
* add bench for size_info() and fix tests
* Also limit the batch-size for vector updates (#5508)
* Also limit the batch-size for vector updates
* clippy
* add lost commit
* Load cache on collection initialization
* add unit type to parameter name
* fix renaming in test
* clearer error message
* fix test
* review remarks
* remove unused function for now
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Limit maximum preallocation on search results aggregator hash set
Prevents an allocation failure and panic if a user specifies a huge
search limit.
* Prevent overflow on fixed length priority queue limit with u64::MAX
* Fix overflow with high limit in search API
* Add two basic tests covering high search limit
Collection name under test is equal to the test module name, without `.py` suffix.
* Helps by debugging/tracing failed tests and find relevant logs lines in qdrant log files
* Opens up a possibility to run tests in parallel, given that there are no data sharing
between test modules
Change details:
* defined module scoped `collection_name` fixture in `conftest.py`
* removed `collection_name` module variable
* each test signature modified to declare the dependency to `collection_name` fixture
* `@pytest.mark.parametrize` migrated to `@pytest-cases.parametrize` in cases when
`collection_name` was used as the value
Although test logic passes the assertions, the test might fail
if during the teardown the created collection is not deleted,
because the client could not reach the database due to networking issue.
In order to make test executions more robust, collection deletes
are retried in case of networking issues.
* Fix search and delete using subslice index, rather than container intex
* Add sanity checks in debug assertions
* test
---------
Co-authored-by: generall <andrey@vasnetsov.com>
If `wait=true` param is not set for delete vector API calls, a test asserting later
that data are gonna can become flaky, because at the assertion time the data could be
still there.
The following files contain such tests at the moment:
* `test_multi_vector_uint8.py`
* `test_multi_vector.py`
* `test_multi_vector_unnamed.py`
* `test_optional_vectors.py`
In order to fix these and prevent the flakiness of future test,
`request_with_validation` helper add `wait=true` param, if it is not set
for delete vector calls.
* `QDRANT_HOST` variable is used by all openapi tests, removed its copies from
`test_multi_vector_uint8.py`, `test_multi_vector_unnamed.py`, `test_query.py`,
`test_shard_snapshot.py`, and `test_snapshot.py`
* Added optional `QDRANT_HOST_HEADERS` env variable to set custom HTTP headers
in order to reach Qdrant instance behind a reverse proxy. The content of the
variable is JSON payload, e.g. `{"host": "qdrant.local"}`
* Adapted `./tests/basic_*.sh` scripts to aware of `QDRANT_HOST_HEADERS` env variable
* Add on_disk:True for uuid index schema check
* Add tests for uuid payload index
* Fix test
* Add test comparing filtered queries between keyword index and uuid index
* Remove order by case
* Clean up
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
* Refactor: Convert app and cluster info metrics from counters to gauges
- Change app_info, app_status_recovery_mode, and cluster_enabled metrics from COUNTER to GAUGE
- Update metric values to use boolean as f64 for consistency and clarity
- Improve metric type accuracy for better representation of state information
* fix: Update docs regarding metrics type
* openapi: Update app_info metric type
* metrics: Update test to check for the right metric type
* expose random sampling query
* fix random query rescoring
* only ignore offset when there are no prefetches
* fix offset and with_payload/with_vector
* gen grpc docs
* make tests less flaky
* gen openapi schema
* Remove suffixed spaces
---------
Co-authored-by: timvisee <tim@visee.me>
* universal-query: Grouping REST API
* add API docs
* bump API count
* add jwt validation test
* fix API
* better test
* stay on CoreSearchRequest where possible
* use existing scoring helper
* push broken test to illustrate issue
* track best score per point_id to enable sort on payload
* add example test for query discover with groups
* track ScoredPoint instead of PointId
* add default values for smoother ux
* Use poetry for openapi tests
* Use poetry for consensus tests
* Use poetry for gen_storage_compat_data/populate_db.py
* Make consensus_tests scripts executable
* Use poetry for test-consensus-compose
* Cleanup
* Don't call poetry run in scripts
* universal-query: lookup_from implementation
* pushin test demonstrating broken equivalence
* more tests and fixes
* validate fusion not use with 'using' field
* cleanup collection info propagation
* do not clone the query in CollectionQueryResolveRequest
* cleanup
* address last code review
* universal-query: Add validations to query input
* update openapi spec
* use raw request to by-pass local validations
* add validation for order_by and factorize
* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
* include old vector into update function and clean posting lists accordingly
* add integration test
* fix counter
* also remove old vector if the insertion is empty
* clippy
* borrow once
* fix max_next_weight correcton on delete + test
* vector index responsible for updating vector storage
* review fixes
* add debug assert
* Allow multivec search on legacy REST API
* show that it works for gRPC as well
* better error message
* update error assertion
* show validation on REST as well
* remove unecessary test
* fix conversion - dim is not vec count
* fmt
* Use TypedMultiDenseVectorRef everywhere (#4224)
* Use TypedMultiDenseVectorRef everywhere
* remove obsolete test
* fix codespell
* fix build
* test single dense vector expansion on upsert
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* REST API multivector for write and retrieve
* add test showing search is not supported
* use helper
* debug asserts
* better naming
* iterator style
* better naming
* add validation test for checking all inner vectors have the same size
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
