* sorting
* migrate tests and move logic into SegmentBuilder
* add test and improve implementation
* improve code
* review
* code review improvements
* add index building to test
* Do not clone ranges
* Resolve clippy warnings due to recent PR on dev
* review suggestions
* Defragmentation in api (#4684)
* add tenant config to api
* deduplicate used defragmentation keys
* rename is_tenant to is_primary
* use all values to defrag key
* rename is_primary -> is_tenant
* update schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
* drop some code
* Drop JsonPathString
* Fix test_remove_key
Drop failing tests:
- Deleting array indices is not idempotent, so we don't support it.
- Empty JSONPath is not supported.
* Make json_path::path() non-generic
* Remove references to JsonPathV2
* Drop JsonPathInterface
* Move json_path::v2 code into json_path
* Drop validate_not_empty
* Drop JsonPath::head() as being unused
* Replace path() with JsonPath::new()
* Restore comments
* Move tests to json_path
* Use json() consistently in tests
* Replace many into calls with Into trait
---------
Co-authored-by: timvisee <tim@visee.me>
* Move build_index out of VectorIndex
* Build index in HNSWIndex::open()
* Introduce HnswIndexOpenArgs
* Proper deletion
* Improve tests
* HNSW::open(): add warn, comment and assert
* Revert to making up the config if it does not exist
* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
* Rename InvertedIndex* -> InvertedIndexCompressed*
* Extract method VectorIndexEnum::fill_idf_statistics
* Extend VectorIndexEnum with new variants
* Introduce sparse::InvertedIndex::Version
* Replace SparseVectorIndexVersion -> InvertedIndex::Version
* Introduce sparse_vector_index::OpenArgs
* SparseVectorIndex::open: do not build index if directory is empty
Otherwise it would build the index twice since `SegmentBuilder::build()`
calls `::open()`, then `::build_index()`. This restores the old (<=v1.9)
behavior.
* Renames
* fix hnsw full scan threshold
* add test
* are you happy clippy
* separate open_vector_storage
* remove public fields from builder
* wip: do not create segment in builder before build
* avoid arc in storage test and low-level loading functions
* WIP: remove internal segment from SegmentBuilder
* fmt
* finalize segment builder fixes
* Revert "are you happy clippy"
This reverts commit c04afa6989.
* Revert "add test"
This reverts commit 8e7ad6207e.
* Revert "fix hnsw full scan threshold"
This reverts commit 8904443fcb.
* remove _daatabse from builder
* fix optimizer test
* fix id tracker versions persistence
* do flush for segment components on build
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* use correct vector storage size
* remove dim from segment entry
* are you happy fmt
* codespell and proportions
* remove obsolete comment
* remove `try_vector_dim`
* are you happy fmt
* remove todo
* revert code of conduct
* check div 0
* Simplify a bit with max iterator
* Update lib/segment/src/index/hnsw_index/hnsw.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* are you happy fmt
* Update lib/segment/src/index/plain_payload_index.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* review fix
* set full scan threshold 0 for test
* use u128 also for multivector storages
* fix sparse vector size calculation
* Move size calculation into if-branch
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* add `query` to shard trait
* add missing conversions for query
* update grpc docs
* Query response has intermediate results
* add ShardQueryResponse description
* move pub use to the top, keep only one way of reaching reexports
* Allow multivec search on legacy REST API
* show that it works for gRPC as well
* better error message
* update error assertion
* show validation on REST as well
* remove unecessary test
* fix conversion - dim is not vec count
* fmt
* Use TypedMultiDenseVectorRef everywhere (#4224)
* Use TypedMultiDenseVectorRef everywhere
* remove obsolete test
* fix codespell
* fix build
* test single dense vector expansion on upsert
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* byte storage with quantization
raw scorer integration
config and test
are you happy fmt
fn renamings
cow refactor
use quantization branch
quantization update
* are you happy clippy
* don't use distance in quantized scorers
* fix build
* add fn quantization_preprocess
* apply preprocessing for only cosine float metric
* fix sparse vectors tests
* update openapi
* more complicated integration test
* update openapi comment
* mmap byte storages support
* fix async test
* move .unwrap closer to the actual check of the vector presence
* fmt
* remove distance similarity function
* avoid copying data while working with cow
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move check for unindexed segment size inside the segment to allow proxy shard decide where to search better
* fmt
* move indexed_only check inside the plain index
* Integrate Multivector at segment level
* more tests - green with lower accuracy
* decrease expected accuracy to fix test
* cleanup test
* multivector can not be empty
* vary number of vector per multivec point
* Multivec knob for SegmentConfig
* regen openapi
* add TODO for next step
* introduce multivecconfig to support more similarity aggregation
* update openapi
* colbert query scorers (#3713)
* colbert query scorers
fix build after rebase
* review remarks
multivector hnsw test
* fixes after rebase
* fix test
* fix build after rebase
* review remark
* Support set by key in low level.
* Rename key field.
* Format.
* Pass key.
* Format.
* Test.
* Clippy.
* Fix ci lint.
* Check grpc consistency.
* Update openapi.
* Fix empty key test case.
* Support array index.
* Format.
* Add test for non exists key.
* Clippy fix.
* Add idempotence test.
* Update index by updated payload.
* Add ut for utils.
* Add ut for 1 level key.
* Fix ut.
* Support no exits key.
* Fix test result.
* Fix after rebase
* handle wildcart insertion into non-existing array
* avoid double read of payload during update
* fix missing removing data from index in case if set_payload removes indexed field
---------
Co-authored-by: Shylock Hg <shylock@DESKTOP-40I855A>
Co-authored-by: Albert Safin <xzfcpw@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* docs, cardinality estimation, grpc not implemented yet
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* Impl min_should clause in REST API
* perform cardinality estimation by estimating cardinalities of intersection and combining as union
* add openapi spec with docs update
* add integration test
* Impl min_should clause in gRPC
* Cargo fmt & clippy
* Fix minor comments
* add equivalence test between min_should and must
* shortcut at min_count matches
* use `Filter::new_*` whenever possible
* Add missing min_should field
* Fix gRPC field ordering & remove deny_unknown_fields
* Empty commit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
