* Timeout on point retrieve when acquiring segment holder
* review: less radical and cleaner
* highly embarassing I hope no one sees this!
* coderabbit
* Fix delete_page not dropping sequential mmap
* In Gridstore Page, support using a single mmap
* In Gridstore Page, only open multiple mmaps if supported at runtime
* Support opening single mmap in UniversalMmapChunk
* Rename environment var, log warning when QDRANT_NO_MULTI_MMAP is set
* Support opening single mmap in MmapDenseVectors
* Remove unused result response
* Import LazyLock
* Disable rocksdb compile time feature by default
* Also disable RocksDB feature in segment crate
* Enable RocksDB feature in all CI builds
* Remove extra job for testing non-RocksDB build, it's the default now
* Keep RocksDB structures in generated OpenAPI schema
* Fix obsolete --workspace flag breaking builds with explicit features
* Also build including RocksDB in e2e tests on CI
* Wrap the tracker and bitmask in Arc
* Implement deferred flusher for gridstore, defer tracker writes
* Remove clone
* Dynamically adjust list of pending updates to drain what is flushed
* Return proper flusher, defer premature Gridstore flushing errors
* Use deferred Gridstore flusher across storages
* Remove all Arc<RwLock<_>> wrappers around Gridstore in storages
* Flush pages inside closure, also wrap them in a lock
* Remove direct flush function from Gridstore
* Add test to assert behavior of deferred flushing in Gridstore
* Add much more extensive test, including concurrent flushes and deletes
* Add test for draining value pointer, make sure to drop when flushed
* Feature gate RocksDB components
* add stopflag check into query_points of payload index
* propagate stopflag more
* fmt
* bench: compare better against no atomic
* use stoppable iter with IteratorExt
* clippy
* Add cancelled error helper
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* Add vector count per vector-name to metrics API
* Add to metrics API
* Improve TinyMap::get_or_insert_default and add tests
* Minor improvements
* Update openapi
* Review remarks
* Remove `collection_vectors` since it can be calculated manually
* reorder arguments of scroll_by
* add slow request log for scroll request
* add slow request log for facet request
* add slow request log for count request
* add slow request log for retrieve request
* Remove obsolete clone
* generic implementation of Loggable for Arc
* Refactor scroll by arguments (#7296)
* separate internal scroll operation from the one used by user
* fmt
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Re-sort segments on flush, don't use proxy but inner segment state
* Also sort segments when proxying all segments for snapshot
The snapshot operation explicitly flushes segments. We therefore must
satisfy flush ordering there too.
* Add some helpful comments
* Add explicit flush ordering enum for segments
* Minor tweaks
* Fix spelling in comment
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* rename `NamedQuery::new_from_vector` -> `NamedQuery::new
* add feedback query in CollectionQuery. Extract pairs from feedback
* add tests
* fmt
* parameterize num pairs
* rename Linear to Simple
* follow refactor from base branch
* add feedback variant to query enum and internal service
remove internal service Feedback structs for QueryEnum
* rename Linear -> Simple
* follow refactor from base branch
* remove outdated validations
* remove reintroduced code
* use explicit From
* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* no manual impl for Serialize and Hash
* new feedback query
* construct query scorer
* simplify formula
* better comment
* make it easier to add other feedback formulas
* rename query -> target
* rename in async raw scorer
* rename formula -> strategy
* better comments
* rename to Simple strategy
* clippyyyyy
* oops, reenable conditional compilation
* Fundamental refactor to precompute pair score
* rename FeedbackScorer -> FeedbackQuery
This type implements `Query` trait, not `Scorer`.`
* flush all segments in one thread sequentially
* bonus: move flush-related functions into dedicated file
* Minor comment tweaks
* await for flush on segment holder level
* fmt
* Minor improvement, preallocate vector for payload index flushers
* Remove invalid comment
---------
Co-authored-by: timvisee <tim@visee.me>
* fix segment repair on load
* workaround for rocksdb
* Minor comment tweaks
* In in memory ID tracker, unset version first before dropping mapping
* rename cleanup_versions into fix inconsistencies
---------
Co-authored-by: timvisee <tim@visee.me>
* WIP: make proxy segment non-appendable
* make deleted points and index changes private to Proxy Segment
* consistently remove empty unused segment from holder on unproxifying and the end of optimization
* remove/fix obsolete test
* Fix comment, some general adjustments
* Also don't remove non-original segments
* Use ahash for map of proxy index changes
* Inline format arguments
* Simplify has point branching
* Assert that proxy flushes to latest wrapped segment version
* make vector deletetion a CoW operation
* relax debug assertions
* Update lib/shard/src/proxy_segment/mod.rs
* fix test_partial_snapshot_empty
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
