* Improve telemetry logic and test
* Parametrize telemetry test
* Consistency hash peeer ID across telemetry
* clean test
* Use Option in segments telemetry
* updat openapi spec
* Avoid test failure on change in order of params
* Test behaviour of Qdrant with shard initializing flag
* Corrupt shard directory and let Qdrant panic like prod
* Wait for shard transfer
* Restore dirty shards using other replicas
* remove unused code
* Request transfer only if replica is dead or dirty
* fmt
* remove comment
* fix clippy
* Delete shard initializing flag after initializing empty local shard
* Expect test to recover shard in existing test
* Review suggestions
* Run tests for longer
* Simplify tests
* Use 2k points
* condition for point_count
* Add comment
* fix flaky tests
* fix flaky tests
* handle edge case
* Include Active in expected states list
* Introduce is_recovery
* simplify tests
* get rid of is_dirty bool in DummyShard
* add missing negation in condition
* fix condition
* final fix for transfer condition
* Don't auto recover if in recovery mode, simplify state checking
* minor comment improvements
* tests scenario where node is killed after deleting shard initializing flag
* Fix failing CI
* Only automatically recover dead replicas
* Mark replica as dead to recover dummy shard
* fix failing test
* Sleep one second after killing peer, give time to release WAL lock
* Prevent waiting for peer to come online indefinitely
* update comment
* minor typo
---------
Co-authored-by: timvisee <tim@visee.me>
* Abort resharding if any Resharding replica is to be marked dead
* Abort resharding before marking shard as Dead
* add comments
* Abort resharding after marking as dead (#6394)
* Force delete points when aborting resharding, even if replica is dead
* Abort resharding after marking replica as dead
* Only abort resharding if we mark related replica as dead
* Just rely on shard replica state
* Propagate shard transfer error right away
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add bench for mmap_bitslice_buffered_update_wrapper
* Use ahash in buffered mmap/RocksDB types and in payload storage
* Function is used in testing only
* fix flush routine for persistent wrappers
* fmt
* rollback changes for write-only wrappers
* Use retain to drop flushed items
* Rename function
---------
Co-authored-by: timvisee <tim@visee.me>
* Parallelize large segment search batches
* Shortcut empty search and scroll requests
* Link to related GitHub issues
* Simplify query batch chunking, use fixed size chunks
* Don't measure read costs if we have a disposable counter
* Batch searches deeper down instead, keep original rate limiting
* don't propagate offset into prefetches
* I want to see CI with that change
* Move import into platform specific scope
* edit test to make sure offset is not propagated
* fix planned query test
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
* Pass FeatureFlags into VectorIndexBuildArgs
* Incremental HNSW index building: append-only case
* Use debug_assert
* first_few_ids
* Check deleted_point_count
* Drop unused method
* wip: implement explicit populate and clear_cache functions for all components
* fmt
* implement clear and populate for vector storages
* fmt
* implement clear and populate for payload storage
* wip: implement explicit populate and clear_cache functions payload indexes
* implement explicit populate and clear_cache functions payload indexes
* fix clippy on CI
* only compile posix_fadvise on linux
* only compile posix_fadvise on linux
* implement explicit populate and clear_cache functions for quantized vectors
* fmt
* remove post-load prefault
* fix typo
* implement is-on-disk for payload indexes, implement clear on drop for segment, implement clear after segment build
* fmt
* also evict quantized vectors after optimization
* re-use and replace advise_dontneed
* move problems.rs to collection crate
* change type on datetime_key expression
* extract from expression
* check for unindexed fields in formula
* clippy
* dedup code
* Only implement verification for `CollectionQuery` types (#6319)
* verify CollectionQuery's Prefetch
* remove implementation for REST query
* remove implementation for REST query groups
* add basic openapi test
* Add in rest and grpc
* add to QueryEnum
* implement Query trait
* connect to scorer creation
* upd tests
* additional changes
* fmt
* gen openapi and grpc docs
* coderabbit fix
* add changes in async scorer
* test sum_scores in more places, refactor to remove repetition
* use f64 as internal expression score
* allow datetime as expression
* add test cases
* fix schema, generate openapi and grpc docs
* fmt
* clippy
* fix test precision
* graceful handling of too large numbers, more tests
* rename to DatetimeExpression for consistency
* use datetime's timestamp as seconds
* capture variable name when parsing
* homogenize DateTime into Datetime
* use interface distinction between datetime strings and payload keys
* fix rebase
* fix after rebase
* Scroll lock: make sure no segments are modified between scroll and retrieve
* Take write lock in block_in_place to prevent blocking tokio runtime
* Use blocking write
* Recover collection snapshot on general runtime
* Make function const
---------
Co-authored-by: timvisee <tim@visee.me>
