* wip: add rest param
* WIP: Refactor `search_with_graph` to use oversampling
* fixup! WIP: Refactor `search_with_graph` to use oversampling
Fix `search_with_graph` quantization config handling
* fixup! WIP: Refactor `search_with_graph` to use oversampling
Use `truncate` instead of `shrink_to` 🤦♀️
* schrink -> truncate
* protection against wrong oversampling value
* Revert second "fixup! WIP: Refactor `search_with_graph` to use oversampling"
Accidentally commited the wrong file 🤦♀️
* fix max method notation
* rollbacK: max notation
* Add `oversampling` field to `QuantizationSearchParams` gRPC type
* Simplify `oversampled_top` calculation
* `./tools/generate_grpc_docs.sh`
* `./tools/generate_openapi_models.sh`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* rename `Lookup` -> `RetrievedLookup`
* feat: expose lookup in groups
- make `with_vectors` and `with_payload` optional
- include in grpc
- move GroupId to inside of PointGroup message
- include `WithLookup` inside of `GroupRequest`, doesn't make sense to keep them separated
* chore: remove allow(dead_code) from builder
* feat: introduce `WithLookupInterface` to allow specifying only the name of the lookup collection
* fix: WithLookupInterface with collection name only brings all payload and vectors
* fix: update grpc docs
* add tests and set better defaults
* remove `RetrievedLookup` from proto
* short-circuit `fill_search_result_with_payload` when it's not needed
* cargo fmt
* proto: revert putting `GroupId` inside `PointGroup`
* update grpc docs
* Remove WithLookupInterface sugar from grpc
- set with_vectors of lookup default to false
- set with_payload of lookup default to true
* update integration tests
* use singular lookup
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* nested object filter
* code review
* add support for must_not in nested
* extract functions
* support and test must_not in SimpleConditionChecker
* add index matching unit test (to be continued)
* remove extra clone
* test with should
* WIP: Nested object filter suggestions (#1855)
* switch to bitvec
* fix clippy
* more tests
* fmt
* fix some tests
* add test with text
* support for nested should
* do not rely on indexes for nested queries & fix test
* use index to make index-aware checks in nested payload
* fix value-count tests
* re-fa-cto-ring
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* wip: chunked mmap
* Fix typo
* insert and get methods
* dynamic bitvec
* clippy
* wip: vector storage
* wip: fmt
* wip: mmap chunks
* wip: mmap problems
* Share transmuted mutable reference over mmap
* option to enable appendable mmap vectors
* fmt
* rename storage status file
* update tests
* fix get deleted value range
* add recovery to vector storage tests
* add flush to tests
* fix transmute from immutable to mutable
* make transmuted pointer private
* remove unused unsafe functions
* force WAL flush if wait=true
* move wal flush into updater thread
* remove flush from update api
* Minimize pub visibility for specialized/dangerous functions
* Allocate vector with predefined capacity
* Inline format parameters
* Assert we have multiple chunks while testing, test is useless otherwise
* Remove unnecessary scope
* Remove unnecessary dereference
* Random bool has 0.5 as standard distribution, use iter::repeat_with
* Replace RemovableMmap::new with Default derive
* Rename len to num_flags
* Use Option replace as it is convention alongside take
* Add FileId enum to replace error prone manual ID rotating
* Use debug_assert_eq where applicable
* Refactor drop and set to replace
* Change default chunk size for chunked mmap vectors to 32MB
This change is made as per GitHub review, because allocating a few
storages with 128MB would take a significant amount of time and storage.
See: https://github.com/qdrant/qdrant/pull/1838#discussion_r1187215475
* Replace for-loops with iterators
* Draft: add typed mmap to improve code safety (#1860)
* Add typed mmap
* Replace some crude mmap usages with typed mmap
* Use typed mmap for deleted flags
* Simplify dynamic mmap flags a lot with new typed mmap, remove flags option
* Reformat
* Remove old mmap functions that are now unused
* Reimplement mmap locking for mmap_vectors
* Add MmapBitSlice tests
* Replace MmapChunk with new typed mmap
* Update docs
* Clean-up
* Disable alignment assertions on Windows for now
* Rename mmap lock to mlock to prevent confusion with lockable types
* one more small test
* Some review fixes
* Add aliasing note
* Add basic error handling in typed mmap constructors
* Use typed mmap error handling throughout project
* Move mmap type module to common
* Fix transmute functions being unsound
See https://github.com/qdrant/qdrant/pull/1860#discussion_r1188593854
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add segment entry function to update named vectors
* Use already available function to update existing vectors
We already had a segment function to update existing named vectors. This
change ensure we use that instead of separating it separately. As a
bonus, this adds support for setting multiple named vectors at once.
* Update set vectors ourselves, don't drop omitted vectors
* Refactor vector updating functions, separate update and replace
* Add basic vector ops, add update/delete functionality to segment updater
* Add internal and public gRPC types and actions for vectors
* Add gRPC API actions
* Reformat
* Add VectorOperations to vector ops, add basic validation
* Validate gRPC vector types
* Validate vector operation structs
* Construct PointIdsList through From trait
* Update gRPC docs
* Use VectorsSelector for vector deletions in gRPC
* Add support for updating multiple points/vectors in update vectors API
* Update gRPC docs
* Fix incorrect gRPC type numbering
* Return point ID error from vector update/delete functions if not found
* Fix disbalanced vectors test
* feat: use 0 as disable for indexing and memmap
* amend: edit test, update doc comments, remove validation
* fix: use constant instead of literal, reword doc comments
* test: update test for loop and timeout
* fix: remove validation on OptimizerConfigDiff and fix test_validatioin.py
* fix: remove validation from build.rs, remove internal hack from doccomments
* fix: update grpc docs
* revert max_optimization_threads validation in test
* update config comments and minor test change
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add QuantizationConfigDiff type
* Add quantization config diff to vector parameters
* Prefer vector config over collection config for quantization
* Update OpenAPI specification
* Validate quantization configuration quantile in 0.5-1.0 range
As per https://github.com/qdrant/qdrant/pull/1681
* Add test if check if vector specific quantization config is persisted
* Alias quantization to quantization_config in vector parameters
* Remove quantization config diff, use full vector specific config instead
* Regenerate OpenAPI specification and gRPC docs
* Fix compilation error
* Add error handling to quantization config conversions
* Fix quantization integration test, make HNSW test stricter
* Validate VectorConfig/VectorParams, remove obsolete validation
* Add HNSW config diff to vector parameters
* Validate params in collection config
* Add HNSW config to segment vector data config
* Add VectorsConfig params iterator for more elegant conversions
* Prefer vector HNSW config over collection config for building HNSW index
* Base segment vector param HNSW config on collection config
* General improvements
* Rewrite HNSW ef_construct extract function to also consider vector configs
* Update OpenAPI specification
* Add test to check if vector specific HNSW config is persisted
* review changes
* review changes
* Regenerate gRPC docs
* Fix test on Windows
* Regenerate OpenAPI specification
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* add minimal working is_null filter
* add is_null condition to grpc api (backward compatible)
* add unit tests is_null and is_empty conditions
* add is_null to points.proto file
* add some failing OpenAPI tests
* fix a failing test due to change in collection data
* refactor MultiValue's check for is_null
* fix is_empty condition not picking up "key":[]
* remove duplicate OpenAPI integration test
* reuse same variable in condition checker tests
* update grpc docs
* fix is_null cardinality estimation to match is_empty
* update openapi specs
* remove unused debug statements
* add new test points to original test_collection
* fix failing tests according to newly added points
* add the `"key":[null]` test_case
* feat: add "any" to "match" statements
Adds support for using "any" to match on a list of different possible values
* enable proper counting in cardinality estimation
* fmt
* upd OpenAPI
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* add on disk key
* remove obsolete graph initialization
* remove obsolete max level
* update openapi
* graph links trait
* use mmap option
* same format for ram and mmap
* fix segment unit tests
* are you happy fmt
* are you happy clippy
* fix ci and add mmap test
* review fixes
* remove unused try-from
* fix version compatibility
* avoid loading from disk during conversion
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Revert "Replication set `Change` enum"
This reverts commit 28e8487542.
* Revert changes (#1053)
* Revert "Generate API - replication follow up (#1043)"
This reverts commit 183ea9e5e3.
* Revert "Replication related API changes (#1037)"
This reverts commit d690b700ff.
* Revert "Persist replica state (#1035)"
This reverts commit 31c3ae482d.
* Revert "[replication] Account for replicas in shard distribution (#1009)"
This reverts commit e332d482b0.
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* include number of indexed points into the info api
* fmt
* await for a certain level of indexing progressing before promoting shard
* log before waiting
* await for index fixes
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
