Commit Graph
17 Commits
Author SHA1 Message Date
xzfc 66d0c36009 Merge io and memory into common (#8155)
* Unify parking_lot/arc_lock feature

* Move lib/common/{io,memory}/* -> lib/common/common/*

- Mmap-related items are grouped into `common::mmap` sub-module:
  - `memory/src/chunked_utils.rs`      -> `common/src/mmap/chunked.rs`
  - `memory/src/madvise.rs`            -> `common/src/mmap/advice.rs`
  - `memory/src/mmap_ops.rs`           -> `common/src/mmap/ops.rs`
  - `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
  - `memory/src/mmap_type.rs`          -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
  - `common/src/fs.rs`          -> `common/src/fs/sync.rs`
  - `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
  - `io/src/move_files.rs`      -> `common/src/fs/move.rs`
  - `io/src/safe_delete.rs`     -> `common/src/fs/safe_delete.rs`
  - `memory/src/checkfs.rs`     -> `common/src/fs/check.rs`
  - `memory/src/fadvise.rs`     -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
  - `io/src/storage_version.rs` -> `common/src/storage_version.rs`

The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.

* Replace uses of `io` and `memory` with new paths in `common`

Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.

* Remove `io` and `memory` crates
2026-02-17 11:01:14 +01:00
xzfc 6a2830db1c Use fs-err (#7319) 2025-11-14 12:26:51 +01:00
Ivan PleshkovandTim Visée de21202aa7 Quantization storage builder for chunked mmap (#7174)
* Quantization storage builder for chunked mmap

* fix after rebase

* Update lib/segment/src/vector_storage/quantized/quantized_chunked_mmap_storage.rs

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2025-09-29 11:25:31 +02:00
Ivan Pleshkov f12b17545e Remove quantization load from trait (#7113)
* remove quantization load

* refactor load

* fix fmt error

* fix clippy
2025-08-26 13:18:36 +02:00
Ivan Pleshkov 9fca19f2dc Retrieve files list from quantization storage (#7095)
* Get files list from quantization

* fix ci
2025-08-26 13:17:46 +02:00
Ivan Pleshkov 6bc3cc813c EncodedStorage upsert vector (#7048)
* EncodedStorage upsert vector
2025-08-26 13:16:40 +02:00
Ivan Pleshkov 18ba202a76 Remove quantization save functions (#7043)
* remove quantization save

* revert convert_binary_encoding fn

* review remarks

* review remark

* review remarks
2025-08-26 13:16:39 +02:00
Ivan Pleshkov 52918b1dd5 Use PointOffsetType in quantization (#7058)
* Use PointOffsetType in quantization

* review remarks

* review remarks
2025-08-26 13:15:37 +02:00
Ivan Pleshkov e4bb8d4d5f Quantization flusher (#7046)
* quantization flusher

* review remarks
2025-08-14 14:30:26 +02:00
Ivan Pleshkov cdf05b30c3 Wrap quantization Vec<u8> in a separate structure (#7013)
* separate struct for vec u8 quantization storage

* cgf testing for test storage
2025-08-14 14:29:25 +02:00
Ivan Pleshkovandxzfc f4ebdd9c9d Appendable quantization storage (#6935)
* appendable qunatization storage

fmt

deprecate count in quantization config

fix arm tests build

qunatized storage vectors count

fix ci

are you happy clippy

fix compat tests

undo rename to `deprecated_count`

remove flusher

remove obsolete functions, don't use hardware counter in ram storage

are you happy clippy

fix gpu build

check ci when revert option

revert last commit

* debug ci

* log offsets

* log offsets

* fix compatibility tests

* deprecate count

* fix arm tests

* fix benches

* Update lib/quantization/src/encoded_vectors_binary.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2025-08-11 13:17:02 +02:00
xzfc aa2e46a949 Replace some generic parameters with associated types (#6711)
* Make EncodedVectors::EncodedQuery associated type

* Make EncodedStorageBuilder::Storage associated type

* Make QueryScorer::TVector associated type

* Remove TVector from RawScorerImpl generic parameters
2025-07-17 13:14:32 +02:00
Arnaud Gourlay 749b2a3a0f Clippy 1.87 autofix (#6516)
* Clippy 1.87 autofix

* fmt
2025-05-22 23:09:57 +02:00
xzfc 1f805b618e Clear disk cache after read (#6396)
* Move clear_disk_cache to memory::fadvise

* Add memory::fadvise::OneshotFile

* Use OneshotFile
2025-04-21 00:08:52 +02:00
Andrey Vasnetsov ad4743a85f vector-io-read measurement on query (#6197)
* remove mut getters from HardwareCounterCell, as mutability is not useful

* introduce vector-io multiplier

* remove RealCpuMeasurement structure

* set vector-io reads multipliers

* account vector reads in dense scorers

* fmt

* fix tests

* propagate hw_counter into posting list iterator

* fmt

* fix test

* wip: measure of sparse iterator

* fmt

* optimize skip_to

* minor refactoring

* keep current PointOffset in iterator to prevent unnecessary reads from memory

* adjust sparse search cpu cost - account for datatype

* fix test

* refactor search_context tests

* move tests into a module

* introduce more tests

* grammar

* review fixes

* fix clippy

* fix clippy again

* change disposable -> new
2025-03-31 16:19:02 +02:00
Andrey Vasnetsov 4abd77158a use fsync instead of flush (#5629) 2025-01-08 14:01:25 +01:00
Ivan Pleshkov 21a3c108bb Move quantization repo (#5096)
* move quantization repo

* are you happy fmt

* are you happy clippy

* remove dumping pq to image

* workspace deps

* are you happy clippy
2024-09-23 16:03:29 +02:00