* Fix delete_page not dropping sequential mmap
* In Gridstore Page, support using a single mmap
* In Gridstore Page, only open multiple mmaps if supported at runtime
* Support opening single mmap in UniversalMmapChunk
* Rename environment var, log warning when QDRANT_NO_MULTI_MMAP is set
* Support opening single mmap in MmapDenseVectors
* Remove unused result response
* Import LazyLock
* Wrap the tracker and bitmask in Arc
* Implement deferred flusher for gridstore, defer tracker writes
* Remove clone
* Dynamically adjust list of pending updates to drain what is flushed
* Return proper flusher, defer premature Gridstore flushing errors
* Use deferred Gridstore flusher across storages
* Remove all Arc<RwLock<_>> wrappers around Gridstore in storages
* Flush pages inside closure, also wrap them in a lock
* Remove direct flush function from Gridstore
* Add test to assert behavior of deferred flushing in Gridstore
* Add much more extensive test, including concurrent flushes and deletes
* Add test for draining value pointer, make sure to drop when flushed
* Feature gate RocksDB components
* add stopflag check into query_points of payload index
* propagate stopflag more
* fmt
* bench: compare better against no atomic
* use stoppable iter with IteratorExt
* clippy
* Add cancelled error helper
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* Add vector count per vector-name to metrics API
* Add to metrics API
* Improve TinyMap::get_or_insert_default and add tests
* Minor improvements
* Update openapi
* Review remarks
* Remove `collection_vectors` since it can be calculated manually
* reorder arguments of scroll_by
* add slow request log for scroll request
* add slow request log for facet request
* add slow request log for count request
* add slow request log for retrieve request
* Remove obsolete clone
* generic implementation of Loggable for Arc
* Refactor scroll by arguments (#7296)
* separate internal scroll operation from the one used by user
* fmt
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Re-sort segments on flush, don't use proxy but inner segment state
* Also sort segments when proxying all segments for snapshot
The snapshot operation explicitly flushes segments. We therefore must
satisfy flush ordering there too.
* Add some helpful comments
* Add explicit flush ordering enum for segments
* Minor tweaks
* Fix spelling in comment
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* rename `NamedQuery::new_from_vector` -> `NamedQuery::new
* add feedback query in CollectionQuery. Extract pairs from feedback
* add tests
* fmt
* parameterize num pairs
* rename Linear to Simple
* follow refactor from base branch
* add feedback variant to query enum and internal service
remove internal service Feedback structs for QueryEnum
* rename Linear -> Simple
* follow refactor from base branch
* remove outdated validations
* remove reintroduced code
* use explicit From
* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* no manual impl for Serialize and Hash
* new feedback query
* construct query scorer
* simplify formula
* better comment
* make it easier to add other feedback formulas
* rename query -> target
* rename in async raw scorer
* rename formula -> strategy
* better comments
* rename to Simple strategy
* clippyyyyy
* oops, reenable conditional compilation
* Fundamental refactor to precompute pair score
* rename FeedbackScorer -> FeedbackQuery
This type implements `Query` trait, not `Scorer`.`
* flush all segments in one thread sequentially
* bonus: move flush-related functions into dedicated file
* Minor comment tweaks
* await for flush on segment holder level
* fmt
* Minor improvement, preallocate vector for payload index flushers
* Remove invalid comment
---------
Co-authored-by: timvisee <tim@visee.me>
* fix segment repair on load
* workaround for rocksdb
* Minor comment tweaks
* In in memory ID tracker, unset version first before dropping mapping
* rename cleanup_versions into fix inconsistencies
---------
Co-authored-by: timvisee <tim@visee.me>
* WIP: make proxy segment non-appendable
* make deleted points and index changes private to Proxy Segment
* consistently remove empty unused segment from holder on unproxifying and the end of optimization
* remove/fix obsolete test
* Fix comment, some general adjustments
* Also don't remove non-original segments
* Use ahash for map of proxy index changes
* Inline format arguments
* Simplify has point branching
* Assert that proxy flushes to latest wrapped segment version
* make vector deletetion a CoW operation
* relax debug assertions
* Update lib/shard/src/proxy_segment/mod.rs
* fix test_partial_snapshot_empty
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
