* Explicitly use C memory representation for Gridstore value pointers
* Inline pure function
* Grow tracker file to be next multiple of two
* In debug builds, panic if getting out of bound point from Gridstore
* Also represent region gaps as C struct
* Remove debug assertion
* Adjust file size test
* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* no manual impl for Serialize and Hash
* new feedback query
* construct query scorer
* simplify formula
* better comment
* make it easier to add other feedback formulas
* rename query -> target
* rename in async raw scorer
* rename formula -> strategy
* better comments
* rename to Simple strategy
* clippyyyyy
* oops, reenable conditional compilation
* Fundamental refactor to precompute pair score
* rename FeedbackScorer -> FeedbackQuery
This type implements `Query` trait, not `Scorer`.`
* flush all segments in one thread sequentially
* bonus: move flush-related functions into dedicated file
* Minor comment tweaks
* await for flush on segment holder level
* fmt
* Minor improvement, preallocate vector for payload index flushers
* Remove invalid comment
---------
Co-authored-by: timvisee <tim@visee.me>
* add test for shard recovery after partial flush
* force flush with explicit fucntion
* Stop flush worker before explicitly flushing again
---------
Co-authored-by: timvisee <tim@visee.me>
* fix segment repair on load
* workaround for rocksdb
* Minor comment tweaks
* In in memory ID tracker, unset version first before dropping mapping
* rename cleanup_versions into fix inconsistencies
---------
Co-authored-by: timvisee <tim@visee.me>
* Allow specifying initial state while creating shard key
* Dont wait for Active state if initial_state is set
* Generate gRPC docs and OpenAPI spec
* Apply suggestions
* Skip from scheamrs not serde
* Add integration test
* minor suggestions
* WIP: make proxy segment non-appendable
* make deleted points and index changes private to Proxy Segment
* consistently remove empty unused segment from holder on unproxifying and the end of optimization
* remove/fix obsolete test
* Fix comment, some general adjustments
* Also don't remove non-original segments
* Use ahash for map of proxy index changes
* Inline format arguments
* Simplify has point branching
* Assert that proxy flushes to latest wrapped segment version
* make vector deletetion a CoW operation
* relax debug assertions
* Update lib/shard/src/proxy_segment/mod.rs
* fix test_partial_snapshot_empty
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* make update batch smaller to prevent long blocks
* limit vector update operations as well
* limit update chunk for point upsertion
* Fix typo
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* separate snapshot-related functions into a new file
* move out snapshot-related function in local shard
* move snapshot test to local shard level
* move snapshot_all_segments and proxy_all_segments_and_apply to local shard
* use former scroll update lock to prevent snapshot operations overlapping with updates
* review fixes: rename all scroll_lock, use write lock for unproxy instead of read
Shutdown `shard` explicitly at the end of benchmark to avoid error
```
A Tokio 1.x context was found, but it is being shutdown.
```
Co-authored-by: Ivan Boldyrev <ivan.boldyrev@qdrant.com>
