* cluster working state and last update in metrics
* Rename metric
* Remove error string
* Use timestamp instead
* Fix prometheus help text
* Change metric type to counter
* Add vector count per vector-name to metrics API
* Add to metrics API
* Improve TinyMap::get_or_insert_default and add tests
* Minor improvements
* Update openapi
* Review remarks
* Remove `collection_vectors` since it can be calculated manually
* Dead shards in /metrics
* Update src/common/metrics.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add replica metrics
* Clippy
* Properly ignore shards with resharding status
* Include 0 as value in metrics
* Remove total_active_replicas and total_expected_replicas
* Improve minimum active replica count in case of resharding
* Reorder metrics calls min < max (to trigger CI)
* Fix metrics help
* Fix typo
---------
Co-authored-by: timvisee <tim@visee.me>
* Currently running optimizer count in metrics
* Clearly state the prerequisites of count_optimizers_running()
* Minor improvements
* improve metric naming
* Bump segment versions on *-by-filter operations to acknowledge WAL
* Assert exactly one segment with the new operation version
* Review remarks
* Separate segment version number to not affect partial snapshots
* Adjust unit test
* Remove old bump_segment_version implementations
* Clarify max_persisted_segment_version_overwrite in flush_all
* Move trait implementations - Clippy
* Apply segment bumping to all *-by-filter functions
* Remove AtomicOptionU64
* Update function name and comments
* Remove max condition, start with overwrite value
* Add assertion, can never have zero segments
* Make max_persisted_segment_version_overwrite monotonic with fetch_max
---------
Co-authored-by: timvisee <tim@visee.me>
* Migrate shard-key format
* Reduce visibility of was_old_format attribute
* Add debug message when we migrate persisted shard key format
---------
Co-authored-by: timvisee <tim@visee.me>
* Store payload index type in config
* Fix CI
* Use new function
* Review remarks
* Fix Ci
* Update lib/segment/src/index/struct_payload_index.rs
* Add is_appendable function to storage type
* Fix storage: store index type in payload index config (#6852)
* Restructure payload index config for backwards compatibility
See: <https://github.com/qdrant/qdrant/pull/6812#issuecomment-3057928785>
* Use mapping of combined schema and type, use storage type for serde
* Replace common functions with defer
* Minor adjustments
* Don't deserialize skip_rocksdb field in payload config if not set
* Rename all payload index types to snake_case format when serializing
* Fix clippy warnings when building without RocksDB
* Change type of FullPayloadIndexType from enum to struct
* Remove both from mutability, assume mutable there
* Also rename the payload index type to snake case
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add new tokenizer for multilingual
* Make codespell happy
* improve handling of when to apply stemming
* Fix stemming implementation
* Fix Japanese language detection. Add tests for MultilingualV2
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Make init() private
* move model into segment, filter non-alphanumeric tokens
* fmt
* exclude punctuation from tokenizer result
* fm
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Make gridstore Advise::Random and add function for sequential read
* Reduce duplicate code in gridstore
* use sequential implementation in iterator
* actually use the sequential read
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* IteratorExt helper functions for Hw Measurements + Some impls
* More index impls; Performance improvements; Use new iterator
* Add missing measurement in hnsw search. Use disposable flag in HwCounterCell too
* use precomputed counts more
* Review nits
* Fix measurement of map_index iter_values_map
* Review remarks
* fix OnFinalCount
* remove comment
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Cardinality estimation measurements
* Apply hw measurements to latest changes from dev
* Clippy
* Also measure cardinality estimation for geo index
* Make measured units 'bytes'
* Use PointOffsetType instead of u32 for size calculation
* fix memory cost for check_values_any in mmap index
* fix double counting for value reading in mmap, remove hw_counter from mmap hashmap
* fmt
* fix hw measurement for text index
* Remove non necessary lifetime annotations
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add payload index filtering IO measurements for some indices
* Add payload index metric to api and telemetry
* Also account for index access overhead
* Review remarks
* Anonymize new HardwareUsage field
* Fix tests
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Finish io measurement for payload storage
* Remove done TODOs
* review remarks
* make signature of `wipe()` consistent
* Remove hardware_counter from tracker.rs and make interfaces consistent
* Add hw_counter to payloads update_storage function from dev
* Measure read io for payload storage
* Add Hardware Counter to update functions
* Fix tests and benches
* Rename (some) *_measured functions back to original
* Add limits for filter and conditions
* clippy
* Review remarks + nested condition test
* Fix opnapi specs
* Improve error message by giving info about limits and usage
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Strict mode config: Max collection size
* api specs
* Add tests + set/update payload check
* Improve function names and add comments
* rename config to separate vectors and payload
* fix tests
* Adjust configs docs
* add benchmark
* improve performance by caching shard info
* add bench for size_info() and fix tests
* Also limit the batch-size for vector updates (#5508)
* Also limit the batch-size for vector updates
* clippy
* add lost commit
* Load cache on collection initialization
* add unit type to parameter name
* fix renaming in test
* clearer error message
* fix test
* review remarks
* remove unused function for now
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* add HwMeasurementAcc and API helpers+types
* add config to API to toggle hardware reporting in API responses
* add Default impl for HwMeasurementAcc
* comments
* review remarks
* include usage into API schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add CPU measurement for multivectors
* add further test sparse vector search
* make updating vector_query_context consistent
* review fix + take element type into consideration
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add HardwareCounterCell and counting for sparse plain search
* Add measurement for indexed sparse vector
* add tests for calculations
* move SegmentQueryContent higher in the call stack
* move hardware counters inside the query context
* fix clippy
* Fix applying hardware measurements and add test for this
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* add tests for strict mode
* add good example for strict mode test
* Add succes case to test_request_exact
---------
Co-authored-by: timvisee <tim@visee.me>
* first grpc implementations
* make usage of 'scroll' more clear
* introduce 'CheckedTocProvider' trait
* implement more grpc types
* more grpc implementations
* rename StrictModeCheckedProvider to StrictModeCheckedTocProvider
* don't use deprecated feature for UncheckedTocProvider
* Trigger optimizers when uploading snapshots
* Trigger optimizers through new method when optimizer config is updated
* review remarks
* Add comment on why ignoring channel send errors is fine
* Remove obsolete return value omitting
---------
Co-authored-by: timvisee <tim@visee.me>
* implement verification for more types and improve trait
* implement StrictModeVerification for update requests
* more impls
* rebase
* add strict mode for more requests
* improve checking
* exhaustive match
* review remarks
* Migrate uuid index to mmap
* add uuid numeric index
* select correct index + tenants
* Only serialize if necessary
* select correct index
* reset Cargo.toml
* review: use only map index for uuid for now
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Jojii