* include document & image objects into grpc API
* introduce image and object to rest api
* minor refactoring
* rename Vector -> VectorInternal
* decompose vector data structures
* add schema
* fmt
* grpc docs
* fix conversion
* fix clippy
* fix another conversion
* rename VectorInput -> VectorInputInternal
* replace grpc TryFrom with async functions
* fmt
* replace rest TryFrom with async functions
* add image and object into query rest
* separate inference related conversions
* move json-related conversions into a separate file
* move vector-related transformations into a separate file
* move more vector related-conversions into dedicated module
* tar_ext::BuilderExt: generalize and add drop workaround
- Generalize: instead of taking `std::fs::File`, take
`impl Write + Seek`.
- Add a workaround for drop: `tar::Builder<W>::drop` might panic if the
underlying writer panics when using in async context.
* Drop unnecessary write_fn usage
* tar_ext::BuilderExt: support both seekable and streaming variants
* tar_ext::BuilderExt: support both owned/borrowed variants
* map index: fix reachable code marked as unreachable
* plumber work to get mmap text index to interfaces
* test: add fixture for mmap text index, always create mmap segment
* various fixes
- ensure dir is created for mmap
- implement is_on_disk() for text index
- invert deleted condition for filter in mmap inverted index
* update grpc docs and openapi
* implement return of files
* review nit
* fix after rebase
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move FacetIndex into facet_index.rs
* add support for integer facets
* add support for uuid facets
* use separate internal structure
* rename FacetValue::Keyword into FacetValue::String in REST
* fix after rebase
* rename keyword -> string for internal data structures
* implement MatchAny and expect for filter on UUID mmap index
* implement MatchAny and expect for cardinality estimation on UUID mmap index
* refactor PayloadFieldIndex filter for handling incorrect empty query parsing case
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
* refactor: let SegmentBuilder::update take unlocked segments
* style: split long lines
* refactor: introduce TestSegments
* test: add tests for mmap indices
* faceting in segment
* Add segment integration test
* nits
* count from filtered stream, not value->points map directly
* drop AtomicRef from fn signature
* count only unique values per point
* use entry in hashmap
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* sorting
* migrate tests and move logic into SegmentBuilder
* add test and improve implementation
* improve code
* review
* code review improvements
* add index building to test
* Do not clone ranges
* Resolve clippy warnings due to recent PR on dev
* review suggestions
* Defragmentation in api (#4684)
* add tenant config to api
* deduplicate used defragmentation keys
* rename is_tenant to is_primary
* use all values to defrag key
* rename is_primary -> is_tenant
* update schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
* drop some code
* Drop JsonPathString
* Fix test_remove_key
Drop failing tests:
- Deleting array indices is not idempotent, so we don't support it.
- Empty JSONPath is not supported.
* Make json_path::path() non-generic
* Remove references to JsonPathV2
* Drop JsonPathInterface
* Move json_path::v2 code into json_path
* Drop validate_not_empty
* Drop JsonPath::head() as being unused
* Replace path() with JsonPath::new()
* Restore comments
* Move tests to json_path
* Use json() consistently in tests
* Replace many into calls with Into trait
---------
Co-authored-by: timvisee <tim@visee.me>
* Move build_index out of VectorIndex
* Build index in HNSWIndex::open()
* Introduce HnswIndexOpenArgs
* Proper deletion
* Improve tests
* HNSW::open(): add warn, comment and assert
* Revert to making up the config if it does not exist
* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
* Rename InvertedIndex* -> InvertedIndexCompressed*
* Extract method VectorIndexEnum::fill_idf_statistics
* Extend VectorIndexEnum with new variants
* Introduce sparse::InvertedIndex::Version
* Replace SparseVectorIndexVersion -> InvertedIndex::Version
* Introduce sparse_vector_index::OpenArgs
* SparseVectorIndex::open: do not build index if directory is empty
Otherwise it would build the index twice since `SegmentBuilder::build()`
calls `::open()`, then `::build_index()`. This restores the old (<=v1.9)
behavior.
* Renames
* fix hnsw full scan threshold
* add test
* are you happy clippy
* separate open_vector_storage
* remove public fields from builder
* wip: do not create segment in builder before build
* avoid arc in storage test and low-level loading functions
* WIP: remove internal segment from SegmentBuilder
* fmt
* finalize segment builder fixes
* Revert "are you happy clippy"
This reverts commit c04afa6989.
* Revert "add test"
This reverts commit 8e7ad6207e.
* Revert "fix hnsw full scan threshold"
This reverts commit 8904443fcb.
* remove _daatabse from builder
* fix optimizer test
* fix id tracker versions persistence
* do flush for segment components on build
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* use correct vector storage size
* remove dim from segment entry
* are you happy fmt
* codespell and proportions
* remove obsolete comment
* remove `try_vector_dim`
* are you happy fmt
* remove todo
* revert code of conduct
* check div 0
* Simplify a bit with max iterator
* Update lib/segment/src/index/hnsw_index/hnsw.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* are you happy fmt
* Update lib/segment/src/index/plain_payload_index.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* review fix
* set full scan threshold 0 for test
* use u128 also for multivector storages
* fix sparse vector size calculation
* Move size calculation into if-branch
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* add `query` to shard trait
* add missing conversions for query
* update grpc docs
* Query response has intermediate results
* add ShardQueryResponse description
* move pub use to the top, keep only one way of reaching reexports
* Allow multivec search on legacy REST API
* show that it works for gRPC as well
* better error message
* update error assertion
* show validation on REST as well
* remove unecessary test
* fix conversion - dim is not vec count
* fmt
* Use TypedMultiDenseVectorRef everywhere (#4224)
* Use TypedMultiDenseVectorRef everywhere
* remove obsolete test
* fix codespell
* fix build
* test single dense vector expansion on upsert
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
