* Partial function impls for TurboVectorStorage
* create/load functionality + tests
# Conflicts:
# lib/segment/src/vector_storage/turbo/mod.rs
# lib/segment/src/vector_storage/turbo/turbo_encoded_vectors.rs
* Fix vectors are not rotated back
* update_from impl
* Rebase fixes
* Proper update_from for ChunkedMmap + better tests
* Improve test coverage
* Move update_from to new DenseTQVectorStorage trait
* Review remark quantization::DistanceType
* Add populate and clear_cache to TurboVectorStorage and TurboEncodedVectorStorage
* [ai] TQDT in the API
* [ai] unify new sparse error for TQDT
* Rename to `turbo4`
* Add `Turbo4` to comments and doc strings.
* Also validate named sparse vector creation
* Don't insert deferred points into sparse index
# Conflicts:
# lib/segment/src/segment/tests.rs
# lib/segment/src/segment_constructor/segment_constructor_base.rs
# Conflicts:
# lib/segment/src/segment_constructor/segment_constructor_base.rs
* Clippy
* Assert consistency of deferred_internal_id var
* Return before SparseVector conversion in case of deferred point
* Use debug_assert instead
* Add deferred point count to UpdateQueueInfo
* Hide if prevent_unoptimized is false
* Clippy
* Openapi
* Iterate over appendable segments only
* Only calculate deferred point count if prevent_unoptimized is true
* Rebase fixes
* Codespell
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add ignore_deferred flag for internal scroll API
* Ignore deferred points in ProxyShard::update
* Use enum instead of bool as type in function parameters
* Use consistent parameter name
* fmt
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* More explicit naming
* Add DeferredBehavior to `count()` and disable filtering in stream_records
* Don't filter `retrieve()` everywhere
* Use consistent filter behavior in read operations
* Add TODO for ignored parameter in remote_shard
* Rebase fixes
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Implement filtering: Search
* Add tests for search and read_filtered
* Implement filtering: read_ordered_filtered + Tests
* Codespell + Clippy
* Reduce code duplication in tests
* Use helper methods of Id tracker
* Implement filtering: Random Scroll
* fixes after rebase
* [Unittest] Only set deferred ID if deferred points exist
* [Unit test] Reduce combinations in segment creation test
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* Use spawn_blocking for segment locking
* Wrap blocking task in AbortOnDropHandle
* Use single blocking thread for fetching all segment size information
---------
Co-authored-by: timvisee <tim@visee.me>
* cluster working state and last update in metrics
* Rename metric
* Remove error string
* Use timestamp instead
* Fix prometheus help text
* Change metric type to counter
* Add vector count per vector-name to metrics API
* Add to metrics API
* Improve TinyMap::get_or_insert_default and add tests
* Minor improvements
* Update openapi
* Review remarks
* Remove `collection_vectors` since it can be calculated manually
* Dead shards in /metrics
* Update src/common/metrics.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add replica metrics
* Clippy
* Properly ignore shards with resharding status
* Include 0 as value in metrics
* Remove total_active_replicas and total_expected_replicas
* Improve minimum active replica count in case of resharding
* Reorder metrics calls min < max (to trigger CI)
* Fix metrics help
* Fix typo
---------
Co-authored-by: timvisee <tim@visee.me>
* Currently running optimizer count in metrics
* Clearly state the prerequisites of count_optimizers_running()
* Minor improvements
* improve metric naming
* Bump segment versions on *-by-filter operations to acknowledge WAL
* Assert exactly one segment with the new operation version
* Review remarks
* Separate segment version number to not affect partial snapshots
* Adjust unit test
* Remove old bump_segment_version implementations
* Clarify max_persisted_segment_version_overwrite in flush_all
* Move trait implementations - Clippy
* Apply segment bumping to all *-by-filter functions
* Remove AtomicOptionU64
* Update function name and comments
* Remove max condition, start with overwrite value
* Add assertion, can never have zero segments
* Make max_persisted_segment_version_overwrite monotonic with fetch_max
---------
Co-authored-by: timvisee <tim@visee.me>
* Migrate shard-key format
* Reduce visibility of was_old_format attribute
* Add debug message when we migrate persisted shard key format
---------
Co-authored-by: timvisee <tim@visee.me>
* Store payload index type in config
* Fix CI
* Use new function
* Review remarks
* Fix Ci
* Update lib/segment/src/index/struct_payload_index.rs
* Add is_appendable function to storage type
* Fix storage: store index type in payload index config (#6852)
* Restructure payload index config for backwards compatibility
See: <https://github.com/qdrant/qdrant/pull/6812#issuecomment-3057928785>
* Use mapping of combined schema and type, use storage type for serde
* Replace common functions with defer
* Minor adjustments
* Don't deserialize skip_rocksdb field in payload config if not set
* Rename all payload index types to snake_case format when serializing
* Fix clippy warnings when building without RocksDB
* Change type of FullPayloadIndexType from enum to struct
* Remove both from mutability, assume mutable there
* Also rename the payload index type to snake case
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add new tokenizer for multilingual
* Make codespell happy
* improve handling of when to apply stemming
* Fix stemming implementation
* Fix Japanese language detection. Add tests for MultilingualV2
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Make init() private
* move model into segment, filter non-alphanumeric tokens
* fmt
* exclude punctuation from tokenizer result
* fm
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Make gridstore Advise::Random and add function for sequential read
* Reduce duplicate code in gridstore
* use sequential implementation in iterator
* actually use the sequential read
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* IteratorExt helper functions for Hw Measurements + Some impls
* More index impls; Performance improvements; Use new iterator
* Add missing measurement in hnsw search. Use disposable flag in HwCounterCell too
* use precomputed counts more
* Review nits
* Fix measurement of map_index iter_values_map
* Review remarks
* fix OnFinalCount
* remove comment
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Cardinality estimation measurements
* Apply hw measurements to latest changes from dev
* Clippy
* Also measure cardinality estimation for geo index
* Make measured units 'bytes'
* Use PointOffsetType instead of u32 for size calculation
* fix memory cost for check_values_any in mmap index
* fix double counting for value reading in mmap, remove hw_counter from mmap hashmap
* fmt
* fix hw measurement for text index
* Remove non necessary lifetime annotations
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add payload index filtering IO measurements for some indices
* Add payload index metric to api and telemetry
* Also account for index access overhead
* Review remarks
* Anonymize new HardwareUsage field
* Fix tests
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Finish io measurement for payload storage
* Remove done TODOs
* review remarks
* make signature of `wipe()` consistent
* Remove hardware_counter from tracker.rs and make interfaces consistent
* Add hw_counter to payloads update_storage function from dev
Jojii