* don't optimize when loading a mmap-backed immutable index
* clear cache after laoding text index from mmap
* Revert "clear cache after laoding text index from mmap"
This reverts commit 110199e8eb.
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Migrate RocksDB based payload storage into Gridstore
* Add migrate_rocksdb_payload_storage feature flag
* Minor tweaks
* Use iter to migrate all payloads
* Fix clippy warning
* Update OpenAPI spec
* Track if mmap payload storage is populated, set memory/disk storage type
* Add test
* Log amount of bytes we migrate instead of point count
* Remove TODO
* Render storage size migrated in human human readable format, not bytes
* On migration failure, drop full payload storage directory
* Fix missing import
* Add in-ram-mmap payload storage type
* When RocksDB is not compiled, select in-ram mmap payload storage
* Enable payload storage on-disk check again in config mismatch optimizer
* Add feature flag to build in-ram-mmap payload storage
* Tweak is_on_disk condition
* Bump version estimates in runtime feature flags
* check if we try to deactivate last initializing replica
* consider more cases
* Update lib/collection/src/shards/replica_set/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add new tokenizer for multilingual
* Make codespell happy
* improve handling of when to apply stemming
* Fix stemming implementation
* Fix Japanese language detection. Add tests for MultilingualV2
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Make init() private
* move model into segment, filter non-alphanumeric tokens
* fmt
* exclude punctuation from tokenizer result
* fm
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* add `"match": { "phrase": ... }` condition
* gen grpc and openapi
* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)
* expose setting in rest and grpc
* phrase matching openapi test
* regen openapi
* [phrase matching] Text index fixes (#6730)
* allow rocksdb-based immutable text index
* fix repeated-token phrases
* fmt
* Update OpenAPI spec
---------
Co-authored-by: timvisee <tim@visee.me>
* add repeated word case in openapi test
* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)
* allow phrase filter when index is present
* prettier error message
* clippppppy
---------
Co-authored-by: timvisee <tim@visee.me>
* Add function for validating legacy collection names
* Apply legacy collection name validation everywhere
* Add more validation to gRPC endpoints
* Adjust tests
* Fix not changing storage states, migrate vector storages in full segment
* Destroy old vector storage data after updating segment config
* Disable existing migrations, only use segment level ones
* Remove segment config update warnings
* Correctly bump sparse vector storage type
* Fix clippy warnings without RocksDB feature flag
* check if the quantization config changed in order to evaluate if we can re-use HNSW index
* Touch up
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* refactor: add GraphLinks::to_edges
* refactor: copy-on-write is no more
* Healing: insert backlinks
* Replace SearchContext with FixedLengthPriorityQueue
* bq encodings
* are you happy clippy
* are you happy clippy
* are you happy clippy
* are you happy clippy
* gpu tests
* update models
* are you happy fmt
* move additional bits to the end
* fix tests
* Welford's Algorithm
* review remarks
* are you happy clippy
* remove debug println in test
* coderabit nitpicks
* remove unnecessary clone and partialeq
* Use f64 for Welford's Algorithm
* try fix ci
* revert cargo-nextest
* add debug assertions
* Make EncodedVectors::EncodedQuery associated type
* Make EncodedStorageBuilder::Storage associated type
* Make QueryScorer::TVector associated type
* Remove TVector from RawScorerImpl generic parameters
