* Fix not changing storage states, migrate vector storages in full segment
* Destroy old vector storage data after updating segment config
* Disable existing migrations, only use segment level ones
* Remove segment config update warnings
* Correctly bump sparse vector storage type
* Fix clippy warnings without RocksDB feature flag
* check if the quantization config changed in order to evaluate if we can re-use HNSW index
* Touch up
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* refactor: add GraphLinks::to_edges
* refactor: copy-on-write is no more
* Healing: insert backlinks
* Replace SearchContext with FixedLengthPriorityQueue
* bq encodings
* are you happy clippy
* are you happy clippy
* are you happy clippy
* are you happy clippy
* gpu tests
* update models
* are you happy fmt
* move additional bits to the end
* fix tests
* Welford's Algorithm
* review remarks
* are you happy clippy
* remove debug println in test
* coderabit nitpicks
* remove unnecessary clone and partialeq
* Use f64 for Welford's Algorithm
* try fix ci
* revert cargo-nextest
* add debug assertions
* Make EncodedVectors::EncodedQuery associated type
* Make EncodedStorageBuilder::Storage associated type
* Make QueryScorer::TVector associated type
* Remove TVector from RawScorerImpl generic parameters
* Add stopwords support
* Update codespell configuration and improve stopword assertion message
* Add stopwords schema to OpenAPI definition
* Rename StopwordsOption to StopwordsInterface
* Add stopwords support for multiple languages
* Update codespell configuration to skip all Rust files in stop_words directory
* Normalize stopwords to lowercase during insertion
* Add language aliases for stopwords
* + grpc
* Unspecified Language
* clippy fmt
* openapi
* Refactor to use BTreeSet
* update tokenizer test to include custom stopwords with contractions
* simplified grpc StopwordsInterface
* alias for languages
* use `lowercase` flag for stop word normalizing
* grpc: enum -> string
* consistent naming
* Remove UnspecifiedLanguage and use From for conversions
* fix conversions
* fmt
* use stopwords inside different tokenizer
* use ahash
* fix tests
* fix no-rocks-db feature
* simplify code and fix test
* update test
* fix clippy
* fmt
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move all inverted indexes to one module
Also, remove `old_mmap_postings` and `compressed_postings`
* move congruence test into its own file
* remove old de/serialize_tokenset functions
* Use sparse vector storage fixture without RocksDB
* Fix some tests
* Replace RocksDB sparse storage with Gristore in mutable text index tests
* Use in-memory vector storage in multivector HNSW test
* Flag many more RocksDB specifics in tests
* Use database placeholder type without RocksDB flag
* Flag more tests
* Flag even more tests
* Fix imports, fix typo, and repair base test build
* Initialize dummy database if RocksDB flag is disabled
* Assert correct storage types
* Don't use old vector storage type when RocksDB is disabled
* Expos default for vector storage type only in tests
* Only expose simple segment constructor in tests
* Don't derive Default
* Fix inverted appendable flag
* Flag index selector
* Flag take_database_snapshot
* Flag field index builder
* Flag bool index
* Flag full text index
* Flag geo index
* Flag map index
* Feature flag numeric index
* Flag skip_rocksdb
* Flag payload storage
* Flag segment
* Flag builder
* Flag backup
* Make RocksDB crate optional
* Tweak feature flag gate
* Disable default segment crate features from all dependents
* Handle flag in collection crate correctly
* Flag payload storage types in tests
* Temporarily disable on-disk check for sparse vectors
* Flag sparse vector RocksDB in GPU code
* Fix compilation error since recent merge
* Flag payload storage types in new function
* remove todo
* Explicitly implement default to more clearly state difference
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Reformat
* Add function to find appendable sparse storage files without opening it
* Actively migrate away from RocksDB based sparse vector storage
* Add test for simple sparse vector storage migration
* Fix test comments
* Also migrate vector deletes for sparse storage
* Use walkdir to list files
* Add Gridstore backend for mutable geo payload index
* Add mutable geo payload index builder for Gridstore
* Add tests
* Define type for raw geo point
* Rework geo point loading in Gridstore, load per point ID
* Bump Gridstore options
* Fix panic when storing empty list of geo points in Gridstore
* Review remarks
* Add function to open all multi dense vector storage types
* Move list files function into common crate
* Add function to find appendable multi dense storage files without opening it
* Actively migrate away from RocksDB based multi dense vector storage
* Add test for simple multi dense vector storage migration
* Also migrate vector deletes for multi dense storage
* Fix vector_io_read for multivector recommendation
* fix
* cleanup
* single disposable
* do not use disposable hardware counter for quantized multivectors
* Implement clear cache for immutable text index
* Add Gridstore backend for mutable full text payload index
* Add mutable full text payload index builder for Gridstore
* Fix compilation error
* Add tests
* Extract mutable inverted index builder to standalone type
* Return generic error in Gridstore iterator
* Remove intermediate buffer from Gridstore load, load index directly
* Minor fixes
* Remove TODO, no good solution
* Fix review remarks
* Serialize vector of ecow strings using cbor
* Adjust trait bounds for Gridstore support in map index
* Add Gridstore backend to mutable map index
* Add Gridstore builder
* Enable new map index in index selector
* Add tests
* Use different Gridstore block size depending on map data type
* Fix review remarks
* Extend Gridstore blob trait, support basic primitives and vector of them
* Implement mutable numeric index on top of Gridstore
* Wrap Gridstore in rwlock, flush asynchronously
* Set custom Gridstore options tuned to numeric index
* Fix compilation errors
* Destruct RocksDB selector everywhere
* Add Gridstore payload index selector, build numeric, fallback others
* Add feature flag to Gridstore mutable payload index
* Don't store empty values in gridstore, it's not supported
* Reformat
* Initialize Gridstore during init of index builder
* Use owned and referenced paths correctly around Gridstore
* Change Gridstore page size in numeric index as suggested
* Make Gridstore block size dependent on size of numeric type
* Add tests
* Add separate fixed size blob trait for vec, prevent footgun
* Improve page size calculation
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Simplify Blob requirement for numeric payload index
* Add missing test cases
* Replace ref mut with &mut
* Use correct hardware counter
* Implement clear cache for numeric payload index on Gridstore
* The block size is dynamic
* Error if trying to load index from storage that doesn't match backend
* fix counter
* Add comment on why we change page size
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add function to open all appendable dense vector storage types
* Add function to find appendable dense storage files without opening it
* Add function to migrate RocksDB dense vector storage into mmap
* Add feature flag and actively migrate on load
* Log number of migrated points
* Add function to open all simple dense vector storage types
* Add test for simple dense vector storage migration
* Use correct column families
* Use more generic feature flag
* Also migrate vector deletes
* Fix review remarks
* rename `Document`->`TokenSet`
* add point_to_doc to mutable text index
introduce point_to_doc
add TODO for using the new field
handle removals
* remove ser/de from `TokenSet`
* clippy
* review fixes
* Gridstore config can be Copy
* Add test asserting wipe behavior
* Add clear function to Gridstore
* Make wipe take ownership, use new clear function everywhere
* Add comment to emphasize wipe behavior
* Propagate errors
