* Preallocate list of entires to prevent some unnecessary reallocations
* Implement Copy for OptimizerThresholds
* Add shard holder function get smallest segment
* Take the smallest segment in the segments updater
* Add test to assert inserting into smallest segment
* Fix compilation warnings
I realized we were sometimes ignoring `offset` when crafting the `PlannedQuery`, specifically in the case of root-level scroll-like requests. This PR fixes this and propagates the root level offset into all the limits in prefetches
Defining `prefetch`es without defining a rescoring `query` does not make much sense, and makes handling ambiguous. So we rather reject this combination.
in grpc:
- restructure `ContextInput`, so that it becomes easy to handle both pairs and positive/negative lists in the future
in rest:
- restructure context queries so that they don't repeat the label in order to be used . E.g. turn `{ query: { context: { context: [ ... ] } } }` into `{ query: { context: [ ... ] } }`
- fixes fusion so that `{query: { fusion: "rrf" } }` actually works
both:
- renames `RecommendInput` positives and negatives to singular (same as in reco api)
- adds an `order` field in `planned_query::ResultsMerge` to indicate the desired direction of merging, and merges accordingly
- converting from `ShardQueryRequest` to `PlannedQuery` now needs the collection params to determine the required order depending on the vector config
- handles previously unhandled `filter` and `score_threshold` in `LocalShard::merge_prefetches()`, either by passing it to the core_search/scroll rescoring or during regular merging
* Extract logic for creating thresholds config
* Put collection params and threshold config in update handler
* Add function to add a new appendable segment if all are over capacity
* Make new method static, call it before each optimization loop
* Update error message formatting
* Use exact point count in replication consensus test
* Add a test to assert segment creation when all are over capacity
* Suffix optimizer thresholds with _kb to clarify unit
* Move segment capacity check logic, run if optimizers are disabled
* fix: add -> mul
---------
Co-authored-by: generall <andrey@vasnetsov.com>
- make `VectorQuery` generic
- retrieve ids from `CollectionQueryRequest` and convert into vectors
- implement `RetrieveRequest` trait for it
- calculate avg vector with logic from `recommendations.rs`
- convert `CollectionQueryRequest` into `ShardQueryRequest`
- begin implementing `query()` fn in collection
* Rename segment addition functions, clarify this generates a new ID
* Don't randomize segment IDs, auto increment to prevent duplicates
* Rename swap to swap_new
* On snapshot unproxy, put segments back with their original segment ID
* Add sanity check to optimizer unproxy, must swap same number of segments
* Clean up
* Extend snapshot test, assert we end up with the same segment IDs
* fix hnsw full scan threshold
* add test
* are you happy clippy
* separate open_vector_storage
* remove public fields from builder
* wip: do not create segment in builder before build
* avoid arc in storage test and low-level loading functions
* WIP: remove internal segment from SegmentBuilder
* fmt
* finalize segment builder fixes
* Revert "are you happy clippy"
This reverts commit c04afa6989.
* Revert "add test"
This reverts commit 8e7ad6207e.
* Revert "fix hnsw full scan threshold"
This reverts commit 8904443fcb.
* remove _daatabse from builder
* fix optimizer test
* fix id tracker versions persistence
* do flush for segment components on build
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Fix incorrect error message
* Simplify shard transfer retrieval function
* Add consensus dispatcher function add current peer ID
* Add simple resharding driver state
* Add consensus dispatcher function to start shard transfer
* Add basic implementation for driving resharding migration transfers
* Add migrate points documentation
* Use consistent naming
* Add logic to wait for a shard transfer to end, either finish or abort
* Add logic for waiting on shard transfer end, either finish or abort
* Start awaiting transfer end before we propose one
* Add channel to broadcast shard transfer state updates
* Do not keep shard holder read lock while waiting for shard transfer end
* On shard transfer abort through consensus snapshot, notify on channel
* If point migration transfer timed out, abort it to clean up
* Fix typo, update comments
* Mark TODOs for resharding
* Initial stubs for resharding driver
* Add basic steps in resharding driver
* Link Notion document
* Clean up start function
* Describe difference between reshard key and task
* Report resharding abort/finish to consensus
* Add driver methods for remaining stages
* Fix typo
* Only start driver on the peer that is responsible for driving it
* Remove ReshardTask, repurpose ReshardKey for that
