* load with any universal io
* Don't populate with sequential advice
* use appropriate fs for `exists`
* use `read_whole_via` more
* TODO
* clear ram cache after read_whole
* Oversample with approx facet
* return early for limit=0
* Document distributed limit flow on Collection::facet
Add an ASCII diagram showing how the facet limit is oversampled once on
the entry node and how peer nodes enter via facet_internal without
re-oversampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [AI + manual] initial impl
[AI] read_batch which actually batches
manual nits
[AI] better handling of local and remote paths
manual refactor, respect open options
don't delete local file
dumbify read_batch
we want to refactor it anyway
simplify
rename to `DiskCache` in `simple_disk_cache` module
* refactor to use always use ReadPipeline
pass meta to remote pipeline
* nits
* run tests for more Remotes
* fix no more <T> in UniversalRead
* fmt
* chore(deps): unify roaring as workspace dep, move duplicate to dev-deps
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* use generic in QuantizedMmapStorage
* rename to QuantizedStorage
* be explicit about S
* rename builder to `QuantizedStorageBuilder`
* rename file to `quantized_storage.rs`
* bytemuck::Pod already implies 'static
* no supertraits
* regions gaps with TypedStorage
* StoredBitSlice with TypedStorage
* bytemuck for PostingsHeader
* bytemuck for TrackerHeader
* bytemuck for MmapRange
* bytemuck for stored_point_to_values::Header
* derive bytemuck::Pod on `OptionalPointer`, get rid of legacy transmute
* remove one layer of Option in `get_raw`
* keep `UniversalRead<u8>` bound, do bytemuck conversion at call site (#8953)
Drop the `+ UniversalRead<OptionalPointer>` / `+ UniversalWrite<OptionalPointer>`
trait bounds added in the previous two commits. Reads and writes of
`OptionalPointer` and `TrackerHeader` now go through `bytemuck::from_bytes` /
`bytemuck::bytes_of` on the existing `u8` byte slices instead.
This keeps the unsafe-transmute removal but avoids the turbofish noise
(`UniversalWrite::<u8>::flusher(&self.storage)`, `<S as UniversalRead<u8>>::open`,
…) that two `T`s on `S` forced everywhere.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* extra refactoring after #8955
* explicit zeroed instantiation
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* genericize ChunkedVectors.status, remove `Sized` bound
* check exists with `UniversalReadFileOps`
* Inline UioChunkedVectors bound, drop the alias (#8952)
The empty trait + blanket impl was a stable-Rust trait-alias workaround
that hid a fairly short bound (UniversalWrite<T> + UniversalWrite<Status>
+ Send + 'static) at the cost of an indirection readers had to mentally
unwind. Spelling it out at the three sites that need it is shorter overall
and immediately tells the reader what is required.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* propagate to BufferedDynamicFlags
* use duplicate for tests
* propagate to Bitvec/Roaring flags
* propagate to RoaringFlags
* fixup! propagate to BufferedDynamicFlags
* propagate to BitvecFlags
* derive Pod for Point<T>
* remove clones on Copy type
* fix annoying clippy lint
* serde skip padding
* extract point into a new file
* assert aligment
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support`
Add BitSliceStorage<S> generic over
UniversalRead<u64>/UniversalWrite<u64>,
providing bit-level read and write operations over u64-element storage.
Read operations (S: UniversalRead<u64>):
- read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others)
- get_bit: single bit read via u64 element fetch
- read_bit_range: arbitrary bit range read
Write operations (S: UniversalWrite<u64>):
- set_bit / replace_bit: single bit write (skips write if unchanged)
- write_bit_range: arbitrary bit range write from BitSlice source
- fill_bit_range: fill range with a value
- set_bits_batch: batch individual bit updates coalesced by element
- flusher: flush underlying storage
* refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper
Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice
(Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>>
(storage-agnostic universal IO backend).
Updated consumers:
- MmapMapIndex
- MmapNumericIndex
- MmapGeoMapIndex
- MmapInvertedIndex (fulltext)
- ImmutableIdTracker
- Benchmark
Additionally:
- Add BitSliceStorage::create(path, num_bits) to encapsulate
file creation + sizing + open (eliminates duplicated size math
across 5 call sites)
- Add MmapBitSliceStorage type alias for
BitSliceStorage<MmapUniversal<u64>>
- Add count_ones() convenience method
- Use set_bits_batch() in wrapper flusher and all build paths
instead of per-bit set_bit() loops (coalesces u64 read-modify-writes)
- Fix silent error swallowing in wrapper get(): log + debug_assert
on I/O errors instead of .ok().flatten()
- Remove dead bitmap_mmap_size function from immutable_id_tracker
* use bitvec's approach for offset
* less api
* misc improvements
* refactor write method
* move to segment crate
* handle creation of file outside of StoredBitSlice logic
* fix codespell
* document bitwise calculations
* coalesce more updates into a single write, batch all writes
* use native `extend_from_bitslice` instead of iterating.
* clarity refactor
* clippyyy
* use `u64` as BitStore everywhere
* assume iterator is sorted
* only use chunk_by for generating runs
* fix update wrapper flusher
* review fixes
* larger set bits batch test
* remove reintroduced file
* oops, fix new test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* refactor PointToValues to use UniversalRead
It includes a significant change to `MmapValue` trait to be able to
handle Cow reads, instead of just references.
* fix str parsing
* fix incorrect path
* use fallible casting
* clippy
* No eager allocation
this also makes it so that borrowed strs can keep happening :heart-eyes:
* Lifetime cleanup
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* feat: add "any" to "match" statements
Adds support for using "any" to match on a list of different possible values
* enable proper counting in cardinality estimation
* fmt
* upd OpenAPI
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Luis Cossío