* Revert "Merge pull request #4560"
This reverts commit 5b0536df39.
* point id examples
* payload example
* VectorStruct example
* fmt
* ShardKey example
* OrderValue example
* fix
* example for SeqNumberType, ScoreType
* Fix division by zero in SSE2, AVX and NEON implementations of u8 cosine
* Add zero division tests
* Fix incorrect test name
* Also test inverse, and add naive implementation test
* Use correct function in NEON test
* Fix NEON feature check
* Store denominator
* field index values as iterator instead of slice
* Use consistent assertions
* fix wrong values_count update
---------
Co-authored-by: timvisee <tim@visee.me>
* Use DatabaseColumnScheduledDeleteWrapper to make sure we only delete payload and indexed payload when flushed at the right time
* Remove double clone
* In on disk payload storage, don't return payload if delete is pending
* hide deleted_pending_persistence inside the abstraction
* Add RocksDB column operator filtering pending deletes
---------
Co-authored-by: timvisee <tim@visee.me>
* drop some code
* Drop JsonPathString
* Fix test_remove_key
Drop failing tests:
- Deleting array indices is not idempotent, so we don't support it.
- Empty JSONPath is not supported.
* Make json_path::path() non-generic
* Remove references to JsonPathV2
* Drop JsonPathInterface
* Move json_path::v2 code into json_path
* Drop validate_not_empty
* Drop JsonPath::head() as being unused
* Replace path() with JsonPath::new()
* Restore comments
* Move tests to json_path
* Use json() consistently in tests
* Replace many into calls with Into trait
---------
Co-authored-by: timvisee <tim@visee.me>
* universal-query: Grouping REST API
* add API docs
* bump API count
* add jwt validation test
* fix API
* better test
* stay on CoreSearchRequest where possible
* use existing scoring helper
* push broken test to illustrate issue
* track best score per point_id to enable sort on payload
* add example test for query discover with groups
* track ScoredPoint instead of PointId
* add default values for smoother ux
* schemars example for vectors in scored point
* schemars example for payload in scored point
* Using json! in schemars example for payload in scored point
* fix
* schemars example for shard_key in scored point
* schemars example for order_by in scored point
* schemars example for id, version, score in scored point
* schemars example for id, payload, vector, shard_key, order_value in …
* point id examples
* remove id_example in schema.rs
* schemars example for vector, named_sparse_vector, named_vector
* schemars example for order_by
* schemars example for OrderByInterface
* schemars example for Fusion
* schemars example for MultiDenseVector,DenseVector in VectorInput
* schemars example for QueryRequest
* schemars example for QueryRequestBatch
* schemars example for QueryResponse
* schemars example for FusionQuery, OrderByQuery, ContextQuery, Discove…
* example for WithLookup, ContextPair, ContextInput, DiscoverInput, Rec…
* fix
* Fix DynamicMmapFlag counting by not using get proxy
* Just count ones in the bitslice of our configured size
---------
Co-authored-by: timvisee <tim@visee.me>
* Add test for OOD during indexing
* Start indexing right after 1st OOD message
* Only send search request after insert loop
* Fail early when encountering out-of-storage during optimization (#4578)
* fs4@0.8.4
* fail early on low storage
* move `dir_size` to `common`
* move the ood bailout to `SegmentOptimizer::optimized_segment_builder`
* drop dead code
* move dir_size to common subcrate
---------
Co-authored-by: generall <andrey@vasnetsov.com>
---------
Co-authored-by: xhjkl <xhjkl@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* When opening a snapshot file, mask its contents on error
Before, it simply dumped part of the file contents back to API users.
This allowed reading arbitrary files.
* Remove intermediate step for reading entry type
* Move build_index out of VectorIndex
* Build index in HNSWIndex::open()
* Introduce HnswIndexOpenArgs
* Proper deletion
* Improve tests
* HNSW::open(): add warn, comment and assert
* Revert to making up the config if it does not exist
* Propagate payload index schema down to shard replica set + update handler
* Configure payload indices when creating new appendable segment
* When loading segments, make sure applied payload indices match config
* Add test to assert creating new segments with payload index
* Fix unit test because the collection payload schema wasn't updated
* Add test for updating payload index configuration on segment load
* Update test documentation
* Also create payload indices in temporary snapshot segment
* do not delete extra payload index from segments
* do not delete extra payload index from segments
* fix test
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
* Max size of multivector to fix chunk in storage
* Update lib/common/common/src/validation.rs
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Rename InvertedIndex* -> InvertedIndexCompressed*
* Extract method VectorIndexEnum::fill_idf_statistics
* Extend VectorIndexEnum with new variants
* Introduce sparse::InvertedIndex::Version
* Replace SparseVectorIndexVersion -> InvertedIndex::Version
* Introduce sparse_vector_index::OpenArgs
* SparseVectorIndex::open: do not build index if directory is empty
Otherwise it would build the index twice since `SegmentBuilder::build()`
calls `::open()`, then `::build_index()`. This restores the old (<=v1.9)
behavior.
* Renames
