* nested object filter
* code review
* add support for must_not in nested
* extract functions
* support and test must_not in SimpleConditionChecker
* add index matching unit test (to be continued)
* remove extra clone
* test with should
* WIP: Nested object filter suggestions (#1855)
* switch to bitvec
* fix clippy
* more tests
* fmt
* fix some tests
* add test with text
* support for nested should
* do not rely on indexes for nested queries & fix test
* use index to make index-aware checks in nested payload
* fix value-count tests
* re-fa-cto-ring
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* wip: chunked mmap
* Fix typo
* insert and get methods
* dynamic bitvec
* clippy
* wip: vector storage
* wip: fmt
* wip: mmap chunks
* wip: mmap problems
* Share transmuted mutable reference over mmap
* option to enable appendable mmap vectors
* fmt
* rename storage status file
* update tests
* fix get deleted value range
* add recovery to vector storage tests
* add flush to tests
* fix transmute from immutable to mutable
* make transmuted pointer private
* remove unused unsafe functions
* force WAL flush if wait=true
* move wal flush into updater thread
* remove flush from update api
* Minimize pub visibility for specialized/dangerous functions
* Allocate vector with predefined capacity
* Inline format parameters
* Assert we have multiple chunks while testing, test is useless otherwise
* Remove unnecessary scope
* Remove unnecessary dereference
* Random bool has 0.5 as standard distribution, use iter::repeat_with
* Replace RemovableMmap::new with Default derive
* Rename len to num_flags
* Use Option replace as it is convention alongside take
* Add FileId enum to replace error prone manual ID rotating
* Use debug_assert_eq where applicable
* Refactor drop and set to replace
* Change default chunk size for chunked mmap vectors to 32MB
This change is made as per GitHub review, because allocating a few
storages with 128MB would take a significant amount of time and storage.
See: https://github.com/qdrant/qdrant/pull/1838#discussion_r1187215475
* Replace for-loops with iterators
* Draft: add typed mmap to improve code safety (#1860)
* Add typed mmap
* Replace some crude mmap usages with typed mmap
* Use typed mmap for deleted flags
* Simplify dynamic mmap flags a lot with new typed mmap, remove flags option
* Reformat
* Remove old mmap functions that are now unused
* Reimplement mmap locking for mmap_vectors
* Add MmapBitSlice tests
* Replace MmapChunk with new typed mmap
* Update docs
* Clean-up
* Disable alignment assertions on Windows for now
* Rename mmap lock to mlock to prevent confusion with lockable types
* one more small test
* Some review fixes
* Add aliasing note
* Add basic error handling in typed mmap constructors
* Use typed mmap error handling throughout project
* Move mmap type module to common
* Fix transmute functions being unsound
See https://github.com/qdrant/qdrant/pull/1860#discussion_r1188593854
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add segment entry function to update named vectors
* Use already available function to update existing vectors
We already had a segment function to update existing named vectors. This
change ensure we use that instead of separating it separately. As a
bonus, this adds support for setting multiple named vectors at once.
* Update set vectors ourselves, don't drop omitted vectors
* Refactor vector updating functions, separate update and replace
* Add basic vector ops, add update/delete functionality to segment updater
* Add internal and public gRPC types and actions for vectors
* Add gRPC API actions
* Reformat
* Add VectorOperations to vector ops, add basic validation
* Validate gRPC vector types
* Validate vector operation structs
* Construct PointIdsList through From trait
* Update gRPC docs
* Use VectorsSelector for vector deletions in gRPC
* Add support for updating multiple points/vectors in update vectors API
* Update gRPC docs
* Fix incorrect gRPC type numbering
* Return point ID error from vector update/delete functions if not found
* Fix disbalanced vectors test
* feat: use 0 as disable for indexing and memmap
* amend: edit test, update doc comments, remove validation
* fix: use constant instead of literal, reword doc comments
* test: update test for loop and timeout
* fix: remove validation on OptimizerConfigDiff and fix test_validatioin.py
* fix: remove validation from build.rs, remove internal hack from doccomments
* fix: update grpc docs
* revert max_optimization_threads validation in test
* update config comments and minor test change
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add QuantizationConfigDiff type
* Add quantization config diff to vector parameters
* Prefer vector config over collection config for quantization
* Update OpenAPI specification
* Validate quantization configuration quantile in 0.5-1.0 range
As per https://github.com/qdrant/qdrant/pull/1681
* Add test if check if vector specific quantization config is persisted
* Alias quantization to quantization_config in vector parameters
* Remove quantization config diff, use full vector specific config instead
* Regenerate OpenAPI specification and gRPC docs
* Fix compilation error
* Add error handling to quantization config conversions
* Fix quantization integration test, make HNSW test stricter
* Validate VectorConfig/VectorParams, remove obsolete validation
* Add HNSW config diff to vector parameters
* Validate params in collection config
* Add HNSW config to segment vector data config
* Add VectorsConfig params iterator for more elegant conversions
* Prefer vector HNSW config over collection config for building HNSW index
* Base segment vector param HNSW config on collection config
* General improvements
* Rewrite HNSW ef_construct extract function to also consider vector configs
* Update OpenAPI specification
* Add test to check if vector specific HNSW config is persisted
* review changes
* review changes
* Regenerate gRPC docs
* Fix test on Windows
* Regenerate OpenAPI specification
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Implement validation on generated gRPC point types
* Validate gRPC point API requests
* Nicely describe validation errors
* Add validation extensions for special types used in gRPC
* Validate gRPC collection API requests
* Validate gRPC Raft and snapshot API requests
* Simplify validation builder, guess derives from fields, fix two fields
* Remove point recommending limit constraint to fix gRPC tests
* Clean up validation logic for gRPC in build.rs
* Validate internal gRPC requests, don't error but show warnings in log
* Optimize gRPC validation extensions, remove unused bits
* Validate gRPC collection creation configuration
* Add custom validators for special gRPC types
* Value is automatically attached to validation error
* Cleanup
* Remove obsolete tests
* Add basic gRPC validation test
* Add some gRPC request payload validation tests
* Set max collection name length to 255 in validation to match Actix
* use common channel pool for healthcheck requests
* add dynamic channel pool
* fmt
* lower parallel connections num
* fmt
* fix clippy
* fix clippy
* better status error message
* smarter channel dropping conditions
* per-channel stats
* refactor pool
* rollback dockerfile
* health-check timeout
* prevent reporting last active peer as dead
* introduce update rate limit
* only rate-limit client requests
* ability to save snapshot without locking wal (#1685)
* ability to save snapshot without locking wal
* fix empty wal saving
* skip waiting on local shard in listener mode even if it is a direct request to listener shard
* snapshot recovery test + better handling of segment versions
* use latest wal
* review changes
* feat: add wait parm for all snapshot API
* update openAPI
* avoid unwrap
* remove all unwarp
* return HTTP 202 when no waiting on snapshot creation
* return 202 when non wait
* update open API
* update param on python test
* update and test api
* optimize the test
* return 202 when non-wait delete
* recover fixes
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* add minimal working is_null filter
* add is_null condition to grpc api (backward compatible)
* add unit tests is_null and is_empty conditions
* add is_null to points.proto file
* add some failing OpenAPI tests
* fix a failing test due to change in collection data
* refactor MultiValue's check for is_null
* fix is_empty condition not picking up "key":[]
* remove duplicate OpenAPI integration test
* reuse same variable in condition checker tests
* update grpc docs
* fix is_null cardinality estimation to match is_empty
* update openapi specs
* remove unused debug statements
* add new test points to original test_collection
* fix failing tests according to newly added points
* add the `"key":[null]` test_case
* integrate quantized data to storages
* revert gitignore
* are you happy clippy
* quantize in optimizer
* provide flag
* fix segfault
* skip quantization flag, update scores
* use quantization flag
* are you happy fmt
* use quantization flag
* quantized search test
* are you happy fmt
* refactor test, refactor scorer choosing
* are you happy fmt
* run quantization on segment builder
* decrease testing parameters
* simplify segment
* update version
* remove use_quantization flag
* provide quantization config
* quantization version up
* euclid dist
* add euclid test
* saveload
* fix initialization bugs
* quantization lib version up
* fix arm build
* refactor scorer selecting
* quant lib version up
* are you happy fmt
* are you happy fmt
* are you happy clippy
* add save/load test for simple storage
* add comments
* quantiles
* quantization mmap
* remove f32
* mmap test
* fix mmap slice
* fix mmap test
* use chunks for quantization storage
* fix build
* are you happy fmt
* update quantization library
* update quantization lib
* update quantization lib
* integrate api changes
* are you happy fmt
* change quantization api
* additional checks in tests
* update quantization version
* fix unit tests
* add quantization to storage config
* use quantization for all cardinality search cases
* Integrate quantization suggestions 2 (#1520)
* review api
* wip: refactor quantization integrations
* wip: refactor quantization integrations
* wip: fmt
* include quantization into snapshot
* fmt
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* feat: add "any" to "match" statements
Adds support for using "any" to match on a list of different possible values
* enable proper counting in cardinality estimation
* fmt
* upd OpenAPI
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Forward write request according to write consistency
* use gRPC internal update API by making shard_id optional
* improve and test
* mark leader peer as failed in case of service error during forward
* forward write ordering param
