* Remove from_iter_instead_of_collect from workspace lints
The lint was removed from clippy (beta) and now triggers
renamed_and_removed_lints warnings in every crate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::chunks_exact_to_as_chunks
Replace chunks_exact with a constant chunk size by as_chunks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::needless_late_init
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::useless_borrows_in_formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::uninlined_format_args
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::for_kv_map
Iterate map values directly instead of discarding keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_large_err on QueueProxyShard::new_from_version
The Err variant intentionally hands the LocalShard back to the caller.
Same pattern as the existing allow on ForwardProxyShard::new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_unit_err on wait_for_consensus_commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add mincore-based memory stats to MmapFile
Add `resident_bytes()`, `disk_bytes()`, and `probe_memory_stats()` methods
to `MmapFile` for measuring page cache residency via `mincore(2)`. This is
the foundation for per-collection memory usage reporting.
Also extract `page_size()` as a public function in `mmap::advice`, replacing
the internal `PAGE_SIZE_MASK` with a direct page size cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] introduce trait for reporting memory usage per component
* [AI] memory reporter implementation for vector storage
* [AI] implement MemoryReporter for QuantizedVectors
* [AI] implement MemoryReporter for VectorIndexEnum
* Implement MemoryReporter for IdTrackerEnum with RAM estimation
Add ram_usage_bytes() to all ID tracker types and their data structures:
- PointMappings, CompressedPointMappings, CompressedVersions,
CompressedInternalToExternal, CompressedExternalToInternal
- MutableIdTracker, ImmutableIdTracker, InMemoryIdTracker
All ID trackers load their data into RAM (none use mmap for working data).
Files are reported as OnDisk (persistence only), actual RAM footprint
is reported via extra_ram_bytes. Uses struct destructuring to ensure
new fields trigger compile errors if not accounted for.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] segment-level memory usage report
* [AI] Block 3: Aggregation Layer and Data Model + internal api for remote shard
* [AI] REST API handler
* fmt
* [AI] clippy fixes
* [AI] macos fix + proxy segment fix
* [AI] make text index estimation a bit more correct
* fix is_on_disk reporting for dense_vector_storage
* fix after rebase
* [AI] deep account for quantized vectors RAM usage + unify chunk size + shring volatile storage after load
* remove debug log
* cache in test
* make manual test easier to run
* rollback chunk size diff, but keep it for test only
* review fixes
* Use exhaustive match
* Use div_ceil on bits everywhere
It does not seem to be strictly necessary because the number of bits
should already be a multiple of the used container size bytes. Still
it's good practice to be careful with this calculation.
* Improve heap size bytes for encoded product quantization vectors
* Include vector stats for binary quantized vectors
* In volatile chunked vectors, include heap allocated vector
* Include rest of heap allocated structures for mutable map index
* In mutable geo index, the hash map is also heap allocated
* Update tests/manual/test_memory_reporting.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use array_windows when statically sized windows were used
* Bump MSRV to 1.94
* fix in gpu code
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* integrate generic posting list in full-text index
* improve intersection fn
improve iterator intersection
* make old module just for test
* recover old implementation of MmapPostings for tests
* add compatibility test
* fix compression
* clippy
* nits
* check if first id is already greater
protects against the case where the iterator's current element hasn't
been extracted yet
* update current element when there is no greater or equal
* reuse visitor in test
* optimize alignment
* bound checks and refactor is_in_range
* review
* edit comment
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* use chunks_exact in compressed sparse posting list builder
* generalized impls
make both impls similar
new design
make impls similar
generalize impls
* separate into files in new crate
* additional builder fns
new fn for builder
* retrieve current iterator position
* simplify traits
* restructure
* add iterator
* clippy
add contains fn
* fix find_chunk
* move builders to builder
* revamp generics and traits
* add model test vs var-sized posting list
* fmt
* clippy
* improve traits and generics
* fix get_by_offset
* generalize tests
* improve test
* restructure view into a new file
* to and from components
* value handler takes closure to prevent perf penalty
* clippy
* remove unused deps
* revert changes in sparse index
* self review nits
* reword
* clippy
* edit comment
* lock CHUNK_LEN, but assert it is synced with BitPacker4x
* remove Sized constraint in iterator
* rename VarSized* to Unsized*
And add more from_components helpers
* avoid intermediate allocation when writing UnsizedValue
* checked u32 conversions
* fix test
* clippy
* Add debug assertion, assert that point ID is in range
* Bound check in get_by_offset
* improve tests to check against various lengths and offsets
* clippy 😤
---------
Co-authored-by: timvisee <tim@visee.me>
