* Improve telemetry logic and test
* Parametrize telemetry test
* Consistency hash peeer ID across telemetry
* clean test
* Use Option in segments telemetry
* updat openapi spec
* Avoid test failure on change in order of params
* Test behaviour of Qdrant with shard initializing flag
* Corrupt shard directory and let Qdrant panic like prod
* Wait for shard transfer
* Restore dirty shards using other replicas
* remove unused code
* Request transfer only if replica is dead or dirty
* fmt
* remove comment
* fix clippy
* Delete shard initializing flag after initializing empty local shard
* Expect test to recover shard in existing test
* Review suggestions
* Run tests for longer
* Simplify tests
* Use 2k points
* condition for point_count
* Add comment
* fix flaky tests
* fix flaky tests
* handle edge case
* Include Active in expected states list
* Introduce is_recovery
* simplify tests
* get rid of is_dirty bool in DummyShard
* add missing negation in condition
* fix condition
* final fix for transfer condition
* Don't auto recover if in recovery mode, simplify state checking
* minor comment improvements
* tests scenario where node is killed after deleting shard initializing flag
* Fix failing CI
* Only automatically recover dead replicas
* Mark replica as dead to recover dummy shard
* fix failing test
* Sleep one second after killing peer, give time to release WAL lock
* Prevent waiting for peer to come online indefinitely
* update comment
* minor typo
---------
Co-authored-by: timvisee <tim@visee.me>
* don't propagate offset into prefetches
* I want to see CI with that change
* Move import into platform specific scope
* edit test to make sure offset is not propagated
* fix planned query test
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
* move problems.rs to collection crate
* change type on datetime_key expression
* extract from expression
* check for unindexed fields in formula
* clippy
* dedup code
* Only implement verification for `CollectionQuery` types (#6319)
* verify CollectionQuery's Prefetch
* remove implementation for REST query
* remove implementation for REST query groups
* add basic openapi test
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* create initial strucutres
* clippy
* start field-query refactoring
* start field-query refactoring (2/N)
* start field-query refactoring (3/N): duplicate is_empty/null condiftions as field condition
* start field-query refactoring (4/N): re-instate is_empty fallback in case new index is not built yet
* filter for is_empty/is_null
* implement add/remove point
* upd schema
* open and create of null-index
* create null-index
* fix test
* Update lib/segment/src/index/query_optimization/condition_converter.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* unit test for null-index
* more unit tests
* add openapi tests
* fmt
* fix for integartion tests
* rabbit review fix
* make [null] non-empty
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add retry-after header for gRPC
* propagate retry-after within inter node communication
* snipe
* tweak and test
* log all parsing errors
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Do not deactivate partial or recovery replica on non-transient error
* Handle missing point edge case in handle_failed_replicas instead
* We already have the replica state
* Add test, move replica during constant set payload, assert shard move
* Add link to pull request in comment
* Retry-After on rate limiter hit for REST API
* fix tests
* Ceil retry after seconds number so clients don't retry too early
* Relax retry after time requirement a bit in case CI machines are slow
* Add rate limit test, test retry after if we replanish within a second
* Report time to wait in rate limit error
---------
Co-authored-by: timvisee <tim@visee.me>
* Restructure process spawning in consensus tests, build list of args
* Extend consensus test, also test if providing URIs through env vars work
* Set URI and bootstrap variables unconditionally
* Move shard key mapping type definition into separate module
* Create save on disk wrapper for shard mapping, persist in robust format
* Clean up shard key mapping wrapper
* Transform conversion functions into from implementations
* Add test, ensure shard key numbers are correctly persisted on restart
* Add limits for filter and conditions
* clippy
* Review remarks + nested condition test
* Fix opnapi specs
* Improve error message by giving info about limits and usage
