* Add collections_vector_total metric for backwards compatibility
Only this metric was removed since Qdrant version 1.15.5, and so this
adds it back for backwards compatibility. Though it has been marked as
deprecated now.
* Prefix replica metrics with collection, part of collection namespace
* Count running optimizations per collection
* Remove deprecation flag
* Minor optimization, preallocate vectors
* Correctly rebuild hash rings, don't use default with custom sharding
* Correctly initialize hash ring map with custom sharding
Don't load the non key ring by default
* Validate shard key for resharding more aggressively
When using auto sharding, require no shard key. When using custom
sharding, require a shard key.
* Only pass sharding method into shard holder construction
* Remove now obsolete debug assertion, we use same source config
* Correctly reconstruct hash rings, also add shard IDs in auto mode
* Add test, do not break consensus on invalid resharding down operation
* Add tests for resharding down shard count precondition
* Minor test tweaks
* Improve debug assertion
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* In metrics, aggregate page faults
* Include page fault of active children (recursively)
* type alias Pid: i32
* Code improvements
* Ignore already terminated processes
* Clippy
* Also count faults for joined descendant threads in children
* Import std::cmp::min
* Rework child fault recursion, remove second hash set
* Add simple test for iterating child processes
* Fix typo
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use limit values directly
* We don't include the parent PID
* Don't allocate hashmap when listing process children
* Extend process child PIDs test, fork recursively
---------
Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* simple integration test for tenant promotion
* add cleanup after promotion
* When switching to ReadActive state, set it for the correct peer ID
* fix test
---------
Co-authored-by: timvisee <tim@visee.me>
* Add ReadActive replica state
* Add consensus function to switch from Partial to ReadActive and confirm
* On stream records to a different shard ID, switch to ReadActive first
* Update OpenAPI spec
* In stream records peer sync, require all peers to be up-to-date
* Review fixes
* Rename ReadActive to ActiveRead
* update openapi spec
* explicit state checking functions
* typo
* update fallback routing with usage of a new state
---------
Co-authored-by: KShivendu <kshivendu1@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Change page fault metrics from gauges to counters
* Change collection_vectors_total to collection_vectors
* Rename metric for page faults in joined child processes
* Rename some procfs metrics
* Show single limit for number of open files
* For number of file descriptors and mmaps follow convential names
* Rename dead_shards_total to dead_replicas
* Fix metric prefix for gRPC entries, add missing underscore
A underscore was missing if no global prefix was set
* make jwt with payload filter fail
* remove `whole` access requirement
* remove unnecessary mut
* fmt
* adjust test
* fix test
* remove more tests
* fix test
* fix test again
* fix test again
* Replace deprecated PayloadConstraint with JSON Value placeholder
* Fix expect message
---------
Co-authored-by: timvisee <tim@visee.me>
* Add vector count per vector-name to metrics API
* Add to metrics API
* Improve TinyMap::get_or_insert_default and add tests
* Minor improvements
* Update openapi
* Review remarks
* Remove `collection_vectors` since it can be calculated manually
* Dead shards in /metrics
* Update src/common/metrics.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add replica metrics
* Clippy
* Properly ignore shards with resharding status
* Include 0 as value in metrics
* Remove total_active_replicas and total_expected_replicas
* Improve minimum active replica count in case of resharding
* Reorder metrics calls min < max (to trigger CI)
* Fix metrics help
* Fix typo
---------
Co-authored-by: timvisee <tim@visee.me>
* Respect payload and hashring filters while replicating point
* Merge points instead of replacing for filtered stream records transfer
* cleanup
* Update utils
* Allow filters in transfer
* drop hashring filter since its not required when destination has single shard
* test comment
* Pass actual filter
* use exact=True
* Forward updates that satisfy the filter
* make test long enough for extra points to be upserted
* minor improvements
* fix test
* try with only inserting new points
* Temporarily forward points that match before or after
* Fix failing test
* Only do inserts
* Dont merge points and update old points in test
* Fix the bug in integration test
* fix read_batch_with_hashring fn name and docs
* changes after rebase
* add validation check
* apply suggestions
* Always trim clock tag when forwarding to a different shard ID
* review fixes
---------
Co-authored-by: generall <andrey@vasnetsov.com>
