* reorder arguments of scroll_by
* add slow request log for scroll request
* add slow request log for facet request
* add slow request log for count request
* add slow request log for retrieve request
* Remove obsolete clone
* generic implementation of Loggable for Arc
* Refactor scroll by arguments (#7296)
* separate internal scroll operation from the one used by user
* fmt
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Allow replicating points between shards
* Add integration test for replicating points between shards
* Drop filter from ForwardProxyShard to for now
* Clean up test
* fix assert failures
* Fix integration test
* Drop reliance on FilteredStreamRecords
* minor improvements
* Currently running optimizer count in metrics
* Clearly state the prerequisites of count_optimizers_running()
* Minor improvements
* improve metric naming
* Re-sort segments on flush, don't use proxy but inner segment state
* Also sort segments when proxying all segments for snapshot
The snapshot operation explicitly flushes segments. We therefore must
satisfy flush ordering there too.
* Add some helpful comments
* Add explicit flush ordering enum for segments
* Minor tweaks
* Fix spelling in comment
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* rename `NamedQuery::new_from_vector` -> `NamedQuery::new
* add feedback query in CollectionQuery. Extract pairs from feedback
* add tests
* fmt
* parameterize num pairs
* rename Linear to Simple
* follow refactor from base branch
* add feedback variant to query enum and internal service
remove internal service Feedback structs for QueryEnum
* rename Linear -> Simple
* follow refactor from base branch
* remove outdated validations
* remove reintroduced code
* use explicit From
* Explicitly use C memory representation for Gridstore value pointers
* Inline pure function
* Grow tracker file to be next multiple of two
* In debug builds, panic if getting out of bound point from Gridstore
* Also represent region gaps as C struct
* Remove debug assertion
* Adjust file size test
* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* no manual impl for Serialize and Hash
* new feedback query
* construct query scorer
* simplify formula
* better comment
* make it easier to add other feedback formulas
* rename query -> target
* rename in async raw scorer
* rename formula -> strategy
* better comments
* rename to Simple strategy
* clippyyyyy
* oops, reenable conditional compilation
* Fundamental refactor to precompute pair score
* rename FeedbackScorer -> FeedbackQuery
This type implements `Query` trait, not `Scorer`.`
* flush all segments in one thread sequentially
* bonus: move flush-related functions into dedicated file
* Minor comment tweaks
* await for flush on segment holder level
* fmt
* Minor improvement, preallocate vector for payload index flushers
* Remove invalid comment
---------
Co-authored-by: timvisee <tim@visee.me>
* add test for shard recovery after partial flush
* force flush with explicit fucntion
* Stop flush worker before explicitly flushing again
---------
Co-authored-by: timvisee <tim@visee.me>
* fix segment repair on load
* workaround for rocksdb
* Minor comment tweaks
* In in memory ID tracker, unset version first before dropping mapping
* rename cleanup_versions into fix inconsistencies
---------
Co-authored-by: timvisee <tim@visee.me>
* Allow specifying initial state while creating shard key
* Dont wait for Active state if initial_state is set
* Generate gRPC docs and OpenAPI spec
* Apply suggestions
* Skip from scheamrs not serde
* Add integration test
* minor suggestions
* Disallow peer to join with URI that is already used
* Add test for rejecting peer join with duplicate URI
* Improve peer rejection logic
* Try to rejoin twice, we expect a consistent result
