* gRPC API for Distance Matrix
* generate API docs
* apply correct limit to sample
* add JWT tests
* pluralitiy
* fix COO naming
* drops COO as it trips codespell and is redundant
* clarify docs
* Add on_disk:True for uuid index schema check
* Add tests for uuid payload index
* Fix test
* Add test comparing filtered queries between keyword index and uuid index
* Remove order by case
* Clean up
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
* Refactor: Convert app and cluster info metrics from counters to gauges
- Change app_info, app_status_recovery_mode, and cluster_enabled metrics from COUNTER to GAUGE
- Update metric values to use boolean as f64 for consistency and clarity
- Improve metric type accuracy for better representation of state information
* fix: Update docs regarding metrics type
* openapi: Update app_info metric type
* metrics: Update test to check for the right metric type
* expose random sampling query
* fix random query rescoring
* only ignore offset when there are no prefetches
* fix offset and with_payload/with_vector
* gen grpc docs
* make tests less flaky
* gen openapi schema
* Remove suffixed spaces
---------
Co-authored-by: timvisee <tim@visee.me>
* Resume resharding driver on restart if resharding is active
* Update resharding state comment started text
Using since versus started. 'Since' better clarifies that the current
state has been active since that time. While 'started at' could lead to
confusion on whether that time is for starting the whole resharding
operation or just that state.
* Add resharding resumption test during migrating points
* Recover resharding hash ring after loading shards
* Update resharding resume test, interrupt at multiple stages, less points
* Remove unused parameter
* Update collection shard count at resharding shard creation/destruction
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Use debug_assert_eq
* Remove trailing comma
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Apply resharding filter only to existing shards, include shard selection
* Add test for stable exact point count during resharding
* Improve filtering, only two separate requests, filtered and non-filtered
* Make scrolling stable while resharding
* Add test for stable scroll during resharding
* Make search stable while resharding
* Add test for stable search during resharding
* Remove resharding post filter in retrieve
* Also assert cardinality point count in resharding test
* Fix typos and some tweaks
* Only clone filter if resharding is active
* Also minimize cloning with resharding filter on count request
* Add test for stable exact point count during resharding with indexing
* Update lib/collection/src/shards/shard_holder/mod.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/collection/src/collection/point_ops.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Also apply resharding filter to retrieve
* Restructure resharding filter usage in `Collection::retrieve`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* universal-query: Grouping REST API
* add API docs
* bump API count
* add jwt validation test
* fix API
* better test
* stay on CoreSearchRequest where possible
* use existing scoring helper
* push broken test to illustrate issue
* track best score per point_id to enable sort on payload
* add example test for query discover with groups
* track ScoredPoint instead of PointId
* add default values for smoother ux
* Add basic resharding test, asserting shard and point counts
* Add resharding replica balancing test
* Always propagate deletes with wait, fix first resharding test flakyness
* Don't count points in balance test for now
* Fix resharding balance test, initial shard may not be on first node
* Count points in resharding balance test
* Test resharding with concurrent updates
* Add point data consistency check to resharding tests
* Fix not using given collection name, list point counts in debug output
* Use poetry for openapi tests
* Use poetry for consensus tests
* Use poetry for gen_storage_compat_data/populate_db.py
* Make consensus_tests scripts executable
* Use poetry for test-consensus-compose
* Cleanup
* Don't call poetry run in scripts
* Add test for OOD during indexing
* Start indexing right after 1st OOD message
* Only send search request after insert loop
* Fail early when encountering out-of-storage during optimization (#4578)
* fs4@0.8.4
* fail early on low storage
* move `dir_size` to `common`
* move the ood bailout to `SegmentOptimizer::optimized_segment_builder`
* drop dead code
* move dir_size to common subcrate
---------
Co-authored-by: generall <andrey@vasnetsov.com>
---------
Co-authored-by: xhjkl <xhjkl@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* universal-query: lookup_from implementation
* pushin test demonstrating broken equivalence
* more tests and fixes
* validate fusion not use with 'using' field
* cleanup collection info propagation
* do not clone the query in CollectionQueryResolveRequest
* cleanup
* address last code review
* universal-query: Add validations to query input
* update openapi spec
* use raw request to by-pass local validations
* add validation for order_by and factorize
* Propagate payload index schema down to shard replica set + update handler
* Configure payload indices when creating new appendable segment
* When loading segments, make sure applied payload indices match config
* Add test to assert creating new segments with payload index
* Fix unit test because the collection payload schema wasn't updated
* Add test for updating payload index configuration on segment load
* Update test documentation
* Also create payload indices in temporary snapshot segment
* do not delete extra payload index from segments
* do not delete extra payload index from segments
* fix test
---------
Co-authored-by: generall <andrey@vasnetsov.com>
