* internal implementation for filter any
* implement api for match `text_any`
* allow naive implementation of text match any
* dedup after kmerge + renames
* improve `expected_should_estimation`
* congruence test
* openapi test
* setup index in each test
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* allow custom K parameter for RRF
* generate grpc docs and openapi
* use tagged type approach for parametrized fusions
* use params approach in grpc
* simplify api structure
* upd schema
* nits
* rest: parameterized rrf as query variant
* consistency in doc comments
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* make bm25 parameters compatible with default fastembed params
* bm25 params in openapi schema
* fmt
* improve bm25 config deserialization
* more explicit docstring for disabling english
* Retain more closed segments
* Add back rocksdb
* Update WalOptions/WalConfig across the code
* Use NonZeroUsize
* Expose via APIs
* Update gRPC docs
* Fix stoarge compat test
* recompile openapi.json with rocksdb
* default wal retain closed fn
* update openapi.json
* Use qdrant/wal latest commit and remove from config.yaml
* Deprecate init_from
* Mark memmap_threshold as deprecated
* Mark locks API as deprecated
* Mark RBAC collection access payload filter as deprecated
* Allow deprecations in tests and benchmarks
* Enable migration from RocksDB to mutable ID tracker by default
We planned to enable this by default in the Qdrant 1.15.0 release.
* Synchronize enable at version TODOs for other flags with planning
* Update OpenAPI spec
* Add function to detect if storage is using RocksDB
* Add feature flag to toggle payload index migration
* Add naive payload index migration, rebuild from scratch if RocksDB
* Adjust payload index rebuild logs
* When rebuilding payload indices, clean up existing storages
* Update OpenAPI spec
* Use weak references to Gridstore in mutable payload index flushers
* Rework payload index clear/wipe functions, make them consistent
* Naively migrate RocksDB before loading indices
* Fix mmap payload index load, staged like others (#6837)
* Add staged loading of mmap numeric payload index
* Add staged loading of mmap map payload index
* Add staged loading of mmap geo payload index
* Add staged loading of mmap full text payload index
* Add staged loading of mmap bool and null payload index
* Minor tweaks
* Don't fail clearing file cache if file does not exist
---------
Co-authored-by: jojii <jojii@gmx.net>
* expose mmr in rest and grpc
* fix referenced vectors in mmr queries
* rename to MmrInput in grpc for consistency
* mmr is now an optional parameter of nearest query
* fix score_threshold at local shard
* fix conversion of rest -> collection
* fix and refactor planned query
* nits
* rename `lambda` to `diversity` in interface
* handle score_threshold within mmr calculation
* add openapi test
* fix candidate limit at collection level
* candidate_limit -> candidates_limit
* finish rename to candidates_limit
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Migrate RocksDB based payload storage into Gridstore
* Add migrate_rocksdb_payload_storage feature flag
* Minor tweaks
* Use iter to migrate all payloads
* Fix clippy warning
* Update OpenAPI spec
* Track if mmap payload storage is populated, set memory/disk storage type
* Add test
* Log amount of bytes we migrate instead of point count
* Remove TODO
* Render storage size migrated in human human readable format, not bytes
* On migration failure, drop full payload storage directory
* Fix missing import
* Add in-ram-mmap payload storage type
* When RocksDB is not compiled, select in-ram mmap payload storage
* Enable payload storage on-disk check again in config mismatch optimizer
* Add feature flag to build in-ram-mmap payload storage
* Tweak is_on_disk condition
* Bump version estimates in runtime feature flags
* add `"match": { "phrase": ... }` condition
* gen grpc and openapi
* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)
* expose setting in rest and grpc
* phrase matching openapi test
* regen openapi
* [phrase matching] Text index fixes (#6730)
* allow rocksdb-based immutable text index
* fix repeated-token phrases
* fmt
* Update OpenAPI spec
---------
Co-authored-by: timvisee <tim@visee.me>
* add repeated word case in openapi test
* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)
* allow phrase filter when index is present
* prettier error message
* clippppppy
---------
Co-authored-by: timvisee <tim@visee.me>
* bq encodings
* are you happy clippy
* are you happy clippy
* are you happy clippy
* are you happy clippy
* gpu tests
* update models
* are you happy fmt
* move additional bits to the end
* fix tests
* Welford's Algorithm
* review remarks
* are you happy clippy
* remove debug println in test
* coderabit nitpicks
* remove unnecessary clone and partialeq
* Use f64 for Welford's Algorithm
* try fix ci
* revert cargo-nextest
* add debug assertions
* Add stopwords support
* Update codespell configuration and improve stopword assertion message
* Add stopwords schema to OpenAPI definition
* Rename StopwordsOption to StopwordsInterface
* Add stopwords support for multiple languages
* Update codespell configuration to skip all Rust files in stop_words directory
* Normalize stopwords to lowercase during insertion
* Add language aliases for stopwords
* + grpc
* Unspecified Language
* clippy fmt
* openapi
* Refactor to use BTreeSet
* update tokenizer test to include custom stopwords with contractions
* simplified grpc StopwordsInterface
* alias for languages
* use `lowercase` flag for stop word normalizing
* grpc: enum -> string
* consistent naming
* Remove UnspecifiedLanguage and use From for conversions
* fix conversions
* fmt
* use stopwords inside different tokenizer
* use ahash
* fix tests
* fix no-rocks-db feature
* simplify code and fix test
* update test
* fix clippy
* fmt
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Improve telemetry logic and test
* Parametrize telemetry test
* Consistency hash peeer ID across telemetry
* clean test
* Use Option in segments telemetry
* updat openapi spec
* Avoid test failure on change in order of params
