* Add `SegmentManifest::segment_id` field
* Tweak `SegmentEntry::take_partial_snapshot` to support `ProxySegment`
* Track segments added into partial snapshot archive
* Debug assert that segment IDs are a valid UUID
---------
Co-authored-by: timvisee <tim@visee.me>
* Handle missing CF on the payload index flush level
* move error handling on the payload field level, so other fields are not skipped if one fails to flush
* Merge nested match statement
---------
Co-authored-by: timvisee <tim@visee.me>
* Add details to snapshot logging for more details
* Format code and improve log msg
* Remove unused import and improve logs
* Improve terminology for logging
* Minimize noise using logging levels
* Log when moving local shard
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* fmt fix
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Rename mutable ID tracker mappings file
* Make new ID tracker the default, add new variant for RocksDB tracker
* In mutable ID tracker, only list files if they exist on disk
* Don't use the new mutable ID tracker yet
* Feature flag usage of new mutable ID tracker
* Simplify RocksDB check a bit
* Rename both the mutable ID tracker files
* Keep track of how many bytes are read, truncate mappings on last partial
* Rely on reader position to detect incomplete entry at the end
* Fix typo
* Handle edge case properly, don't truncate file if fully read
* minor fix
* Add test to assert operation size equals bytes read from stream
* Functions can be constant
* Fix some nitpicks
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* IteratorExt helper functions for Hw Measurements + Some impls
* More index impls; Performance improvements; Use new iterator
* Add missing measurement in hnsw search. Use disposable flag in HwCounterCell too
* use precomputed counts more
* Review nits
* Fix measurement of map_index iter_values_map
* Review remarks
* fix OnFinalCount
* remove comment
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add decay expressions
* remove code duplication in conversion
* test and fix lambda roundtrip
* Remove debug print in decay midpoint validation
* Improve error messages for decay midpoint and scale validation
* use similar lambda for gauss and exp decays
* avoid abs before squaring
* add comment about decay range
* add debug assertion
* Persist version changes as internal to version mapping, not external
* Read and write versions into file directly
* Warn if versions file has unexpected size
* Optimize writing batch of versions, only seek if needed
* Don't explicitly resize file, it'll be done automatically
* Minor refactoring
* review nits
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Extract common points mappings check into function
* Fully rework load/store for point mapping changes, use byte format
* Add extra serialize/deserialize test for new format
* Fuse iterator
* Add another test case
* Take mutable reader when reading point mapping entry
* Flush versions in the same way we flush mappings now
* Don't explicitly fuse reader, it's not necessary
* Cardinality estimation measurements
* Apply hw measurements to latest changes from dev
* Clippy
* Also measure cardinality estimation for geo index
* Make measured units 'bytes'
* Use PointOffsetType instead of u32 for size calculation
* fix memory cost for check_values_any in mmap index
* fix double counting for value reading in mmap, remove hw_counter from mmap hashmap
* fmt
* fix hw measurement for text index
* Remove non necessary lifetime annotations
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add mutable ID tracker tests, ported from immutable ID tracker
* Also test raw mappings, reset inverse mapping when dropping point
* Always flush mappings before version, also in test
* Add initial mutable ID tracker
* Correctly handle duplicate point mappings and deleted flags
* Improve error handling in flush
* Preallocate capacity for pending mappings/versions more intelligently
* Warn or error about missing ID tracker files
* Reformat
* Don't crash if just the last mapping/version entry is corrupt
* Move mapping and point parsing into separate functions
* Extract loading logic into separate functions
* Do not allow partially corrupted ID tracker files for now
* Remove TODOs
* Minor improvements
* Fsync mappings and versions file after writing to it
* Return error when fsync fails
* Add payload index filtering IO measurements for some indices
* Add payload index metric to api and telemetry
* Also account for index access overhead
* Review remarks
* Anonymize new HardwareUsage field
* Fix tests
* Anonymize ShardKey in ReplicaSetTelemetry
* Anonymize vector names in CollectionConfigInternal
* no-op: impl HardwareUsage for HardwareUsage (used in HardwareTelemetry)
* no-op: impl Anonymize for OptimizersStatus (used in OptimizerTelemetry and CollectionsAggregatedTelemetry)
* no-op: impl Anonymize for TrackerStatus in (used in TrackerTelemetry)
* no-op: Anonymize VectorParams in VectorsConfig
* no-op: Copy fields explicitly in OperationDurationStatistics
* no-op: `.clone().map(|x| x.anonymize())` -> `.anonymize()`
* no-op: impl Anonymize for ConsensusThreadStatus (used in PeerInfo and ClusterStatusTelemetry)
* create initial strucutres
* clippy
* start field-query refactoring
* start field-query refactoring (2/N)
* start field-query refactoring (3/N): duplicate is_empty/null condiftions as field condition
* start field-query refactoring (4/N): re-instate is_empty fallback in case new index is not built yet
* filter for is_empty/is_null
* implement add/remove point
* upd schema
* open and create of null-index
* create null-index
* fix test
* Update lib/segment/src/index/query_optimization/condition_converter.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* unit test for null-index
* more unit tests
* add openapi tests
* fmt
* fix for integartion tests
* rabbit review fix
* make [null] non-empty
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
