* Add new tokenizer for multilingual
* Make codespell happy
* improve handling of when to apply stemming
* Fix stemming implementation
* Fix Japanese language detection. Add tests for MultilingualV2
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Make init() private
* move model into segment, filter non-alphanumeric tokens
* fmt
* exclude punctuation from tokenizer result
* fm
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* add `"match": { "phrase": ... }` condition
* gen grpc and openapi
* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)
* expose setting in rest and grpc
* phrase matching openapi test
* regen openapi
* [phrase matching] Text index fixes (#6730)
* allow rocksdb-based immutable text index
* fix repeated-token phrases
* fmt
* Update OpenAPI spec
---------
Co-authored-by: timvisee <tim@visee.me>
* add repeated word case in openapi test
* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)
* allow phrase filter when index is present
* prettier error message
* clippppppy
---------
Co-authored-by: timvisee <tim@visee.me>
* Add function for validating legacy collection names
* Apply legacy collection name validation everywhere
* Add more validation to gRPC endpoints
* Adjust tests
* Fix not changing storage states, migrate vector storages in full segment
* Destroy old vector storage data after updating segment config
* Disable existing migrations, only use segment level ones
* Remove segment config update warnings
* Correctly bump sparse vector storage type
* Fix clippy warnings without RocksDB feature flag
* check if the quantization config changed in order to evaluate if we can re-use HNSW index
* Touch up
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* refactor: add GraphLinks::to_edges
* refactor: copy-on-write is no more
* Healing: insert backlinks
* Replace SearchContext with FixedLengthPriorityQueue
* bq encodings
* are you happy clippy
* are you happy clippy
* are you happy clippy
* are you happy clippy
* gpu tests
* update models
* are you happy fmt
* move additional bits to the end
* fix tests
* Welford's Algorithm
* review remarks
* are you happy clippy
* remove debug println in test
* coderabit nitpicks
* remove unnecessary clone and partialeq
* Use f64 for Welford's Algorithm
* try fix ci
* revert cargo-nextest
* add debug assertions
* Make EncodedVectors::EncodedQuery associated type
* Make EncodedStorageBuilder::Storage associated type
* Make QueryScorer::TVector associated type
* Remove TVector from RawScorerImpl generic parameters
* check filesystem on start and do some simple mmap test to make sure it is persisted
* fmt
* fix fs type read on non-linux
* feature-flag for fs-check supported platforms + fs check with sub-folder
* fmt
* try larger magic file
* minor fixes
* fix compile on windows
* Add FAT, exFAT and APFS file systems
* Update lib/common/memory/src/checkfs.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add stopwords support
* Update codespell configuration and improve stopword assertion message
* Add stopwords schema to OpenAPI definition
* Rename StopwordsOption to StopwordsInterface
* Add stopwords support for multiple languages
* Update codespell configuration to skip all Rust files in stop_words directory
* Normalize stopwords to lowercase during insertion
* Add language aliases for stopwords
* + grpc
* Unspecified Language
* clippy fmt
* openapi
* Refactor to use BTreeSet
* update tokenizer test to include custom stopwords with contractions
* simplified grpc StopwordsInterface
* alias for languages
* use `lowercase` flag for stop word normalizing
* grpc: enum -> string
* consistent naming
* Remove UnspecifiedLanguage and use From for conversions
* fix conversions
* fmt
* use stopwords inside different tokenizer
* use ahash
* fix tests
* fix no-rocks-db feature
* simplify code and fix test
* update test
* fix clippy
* fmt
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move all inverted indexes to one module
Also, remove `old_mmap_postings` and `compressed_postings`
* move congruence test into its own file
* remove old de/serialize_tokenset functions
* Use sparse vector storage fixture without RocksDB
* Fix some tests
* Replace RocksDB sparse storage with Gristore in mutable text index tests
* Use in-memory vector storage in multivector HNSW test
* Flag many more RocksDB specifics in tests
* Use database placeholder type without RocksDB flag
* Flag more tests
* Flag even more tests
* Fix imports, fix typo, and repair base test build
* Initialize dummy database if RocksDB flag is disabled
* Assert correct storage types
* Don't use old vector storage type when RocksDB is disabled
* Expos default for vector storage type only in tests
* Only expose simple segment constructor in tests
* Don't derive Default
* Fix inverted appendable flag
* Flag index selector
* Flag take_database_snapshot
* Flag field index builder
* Flag bool index
* Flag full text index
* Flag geo index
* Flag map index
* Feature flag numeric index
* Flag skip_rocksdb
* Flag payload storage
* Flag segment
* Flag builder
* Flag backup
* Make RocksDB crate optional
* Tweak feature flag gate
* Disable default segment crate features from all dependents
* Handle flag in collection crate correctly
* Flag payload storage types in tests
* Temporarily disable on-disk check for sparse vectors
* Flag sparse vector RocksDB in GPU code
* Fix compilation error since recent merge
* Flag payload storage types in new function
* remove todo
* Explicitly implement default to more clearly state difference
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Reformat
* Add function to find appendable sparse storage files without opening it
* Actively migrate away from RocksDB based sparse vector storage
* Add test for simple sparse vector storage migration
* Fix test comments
* Also migrate vector deletes for sparse storage
* Use walkdir to list files
