* definitions
* create file
* ranges
* get values
* are you happy clippy
* remove unnecessary function
* are you happy codespell
* review fixes
* are you happy clippy
* fmt
* add padding
* update map public functions as in the dev
* are you happy clippy
* review remarks
* generic structure
* review remarks
* review fixes
* add in_memory_id_tracker
* add ImmutableIdTracker to IdTrackerEnum
* improve in in_memory_id_tracker
* Add benchmark and revert changes
* fix not compiling
* rebase to dev
* Minor ImmutableIdTracker improvements (#4698)
* remove unnecessary allocation + add minor improvements
* fix import
* refactor points_mapping to share function between immutable & in_memory id trackers
* test and fix for version persistence;
* fmt
* fix tests
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Implement random reads from segments up to local shard
* fix some review comments
- unimplemented for fixture implementation
- map after chaining
- kmerge instead of choose_multiple
* add custom filtered_read_by_index_shuffled
* cargo clippy
* grab from each segment depending on availability
* sample in O(limit) (#4680)
* edit comment
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* sorting
* migrate tests and move logic into SegmentBuilder
* add test and improve implementation
* improve code
* review
* code review improvements
* add index building to test
* Do not clone ranges
* Resolve clippy warnings due to recent PR on dev
* review suggestions
* Defragmentation in api (#4684)
* add tenant config to api
* deduplicate used defragmentation keys
* rename is_tenant to is_primary
* use all values to defrag key
* rename is_primary -> is_tenant
* update schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
* add immutable_id_tracker
* add dirty flag in test
* don't use immutable_id_tracker for now
* improve and integrate new immutable_id_tracker
* split external_to_internal into two BTreeMaps
* apply rquested changes
* delay mmap writes until flush
* remove unnecessary clone
* single source of truth for file path
* use custom de/serialization for more performance
* disable id tracker and fix codespell
* improve code & test
* Other minor nitpicks
* Apply suggestions from code review
* fix rebase issues
* basic custom mappings storage implementation
* add tests & fix bugs
* add more tests and fix bugs
* undo .codespellrc
* disable immutable_id_tracker completely for now
* fix clippy
* Remove unnecessary pub
* minor renaming
* Revert "Merge pull request #4560"
This reverts commit 5b0536df39.
* point id examples
* payload example
* VectorStruct example
* fmt
* ShardKey example
* OrderValue example
* fix
* example for SeqNumberType, ScoreType
* Fix division by zero in SSE2, AVX and NEON implementations of u8 cosine
* Add zero division tests
* Fix incorrect test name
* Also test inverse, and add naive implementation test
* Use correct function in NEON test
* Fix NEON feature check
* Store denominator
* field index values as iterator instead of slice
* Use consistent assertions
* fix wrong values_count update
---------
Co-authored-by: timvisee <tim@visee.me>
* Use DatabaseColumnScheduledDeleteWrapper to make sure we only delete payload and indexed payload when flushed at the right time
* Remove double clone
* In on disk payload storage, don't return payload if delete is pending
* hide deleted_pending_persistence inside the abstraction
* Add RocksDB column operator filtering pending deletes
---------
Co-authored-by: timvisee <tim@visee.me>
* drop some code
* Drop JsonPathString
* Fix test_remove_key
Drop failing tests:
- Deleting array indices is not idempotent, so we don't support it.
- Empty JSONPath is not supported.
* Make json_path::path() non-generic
* Remove references to JsonPathV2
* Drop JsonPathInterface
* Move json_path::v2 code into json_path
* Drop validate_not_empty
* Drop JsonPath::head() as being unused
* Replace path() with JsonPath::new()
* Restore comments
* Move tests to json_path
* Use json() consistently in tests
* Replace many into calls with Into trait
---------
Co-authored-by: timvisee <tim@visee.me>
* universal-query: Grouping REST API
* add API docs
* bump API count
* add jwt validation test
* fix API
* better test
* stay on CoreSearchRequest where possible
* use existing scoring helper
* push broken test to illustrate issue
* track best score per point_id to enable sort on payload
* add example test for query discover with groups
* track ScoredPoint instead of PointId
* add default values for smoother ux
* schemars example for vectors in scored point
* schemars example for payload in scored point
* Using json! in schemars example for payload in scored point
* fix
* schemars example for shard_key in scored point
* schemars example for order_by in scored point
* schemars example for id, version, score in scored point
* schemars example for id, payload, vector, shard_key, order_value in …
* point id examples
* remove id_example in schema.rs
* schemars example for vector, named_sparse_vector, named_vector
* schemars example for order_by
* schemars example for OrderByInterface
* schemars example for Fusion
* schemars example for MultiDenseVector,DenseVector in VectorInput
* schemars example for QueryRequest
* schemars example for QueryRequestBatch
* schemars example for QueryResponse
* schemars example for FusionQuery, OrderByQuery, ContextQuery, Discove…
* example for WithLookup, ContextPair, ContextInput, DiscoverInput, Rec…
* fix
* Fix DynamicMmapFlag counting by not using get proxy
* Just count ones in the bitslice of our configured size
---------
Co-authored-by: timvisee <tim@visee.me>
* Add test for OOD during indexing
* Start indexing right after 1st OOD message
* Only send search request after insert loop
* Fail early when encountering out-of-storage during optimization (#4578)
* fs4@0.8.4
* fail early on low storage
* move `dir_size` to `common`
* move the ood bailout to `SegmentOptimizer::optimized_segment_builder`
* drop dead code
* move dir_size to common subcrate
---------
Co-authored-by: generall <andrey@vasnetsov.com>
---------
Co-authored-by: xhjkl <xhjkl@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
