Commit Graph

17 Commits

Author SHA1 Message Date
xzfc
4cabb7fd8e Merge io and memory into common (#8155)
* Unify parking_lot/arc_lock feature

* Move lib/common/{io,memory}/* -> lib/common/common/*

- Mmap-related items are grouped into `common::mmap` sub-module:
  - `memory/src/chunked_utils.rs`      -> `common/src/mmap/chunked.rs`
  - `memory/src/madvise.rs`            -> `common/src/mmap/advice.rs`
  - `memory/src/mmap_ops.rs`           -> `common/src/mmap/ops.rs`
  - `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
  - `memory/src/mmap_type.rs`          -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
  - `common/src/fs.rs`          -> `common/src/fs/sync.rs`
  - `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
  - `io/src/move_files.rs`      -> `common/src/fs/move.rs`
  - `io/src/safe_delete.rs`     -> `common/src/fs/safe_delete.rs`
  - `memory/src/checkfs.rs`     -> `common/src/fs/check.rs`
  - `memory/src/fadvise.rs`     -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
  - `io/src/storage_version.rs` -> `common/src/storage_version.rs`

The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.

* Replace uses of `io` and `memory` with new paths in `common`

Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.

* Remove `io` and `memory` crates
2026-02-17 10:58:59 +01:00
xzfc
6bced54ca1 Chore: promote dependencies to workspace level (#8061)
* Promote `env_logger` to workspace dependency

* Promote `anyhow` to workspace dependency

* Promote `rmp-serde` to workspace dependency

* Promote `tinyvec` to workspace dependency

* Promote `async-trait` to workspace dependency

* Promote `url` to workspace dependency

* Promote `self_cell` to workspace dependency

* Promote `cc` to workspace dependency

* Promote `bitpacking` to workspace dependency
2026-02-05 16:02:28 +01:00
Luis Cossío
d79cd9b99b activate fs-err features for nicer error messages (#7665)
* activate fs-err features for nicer error messages

* use in entire workspace
2025-12-02 10:40:36 -03:00
Ivan Pleshkov
8df3dda006 Scalar quantization encoding parameter (#7602)
* scalar quantization encoding parameter

fix ci

remove method

review remarks

fix ci

* uint8 - int8
2025-12-01 12:03:36 +01:00
Ivan Pleshkov
c737d9d087 P-Square one pass quantile estimation method (#7520)
* p square one pass method

* are you happy fmt

* fix n=9 case

* marker struct

* refactor

* proper tests

* add bench

* better namings

* are you happy clippy

* are you happy clippy

* fix additional markers order

* use ArrayVec instead of SmallVec

* use orderer float to sort f64

* Update lib/quantization/src/p_square.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* Update lib/quantization/src/p_square.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* mispelling

* review remarks

* fix typo

* change float checks order

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2025-11-21 20:21:32 +03:00
xzfc
a0d62330c7 Use fs-err (#7319) 2025-09-29 12:47:10 +00:00
Ivan Pleshkov
b03853c33e Quantization storage builder for chunked mmap (#7174)
* Quantization storage builder for chunked mmap

* fix after rebase

* Update lib/segment/src/vector_storage/quantized/quantized_chunked_mmap_storage.rs

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2025-08-29 16:10:30 +02:00
Ivan Pleshkov
a3e457d387 Wrap quantization Vec<u8> in a separate structure (#7013)
* separate struct for vec u8 quantization storage

* cgf testing for test storage
2025-08-12 13:20:48 +02:00
Ivan Pleshkov
b414a402bb Merge pull request #6728
* # This is a combination of 7 commits.

* SameAsStorage default value

* fix coderabbit warnings

* neon for u8 bq

* sse for u8 bq

* fix windows build

* rename function

* add comments

* fmt

* fix arm build

* review remarks
2025-07-03 14:09:12 +02:00
Arnaud Gourlay
423a39f301 Remove duplicated mmap utils (#6129) 2025-03-10 12:20:22 +01:00
Tim Visée
3e536347e1 Bump Rust edition to 2024 (#6042)
* Bump Rust edition to 2024

* gen is a reserved keyword now

* Remove ref mut on references

* Mark extern C as unsafe

* Wrap unsafe function bodies in unsafe block

* Geo hash implements Copy, don't reference but pass by value instead

* Replace secluded self import with parent

* Update execute_cluster_read_operation with new match semantics

* Fix lifetime issue

* Replace map_or with is_none_or

* set_var is unsafe now

* Reformat
2025-02-25 11:21:25 +01:00
Arnaud Gourlay
51b574c9bd Fix missing common testing dep on blobstore (#5875)
* Fix missing common testing dep on blobstore

* also for quantization
2025-01-27 10:33:36 +01:00
Ivan Pleshkov
1447d1efa8 atomic save quantization metadata (#5627) 2024-12-09 21:24:24 +01:00
Arnaud Gourlay
5bce09a1ed Merge quantization integration tests (#5473) 2024-11-19 13:10:01 +01:00
Jojii
7b677fb7fa Hardware counting for quantization (#5369)
* add cpu measurement for quantization

* clippy

* fix tests

* discard hardware counters in benchmarks

* Forwarding hardware counter in distributed setup (#5371)

* make hardware counter available in distributed setup

* clippy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-06 16:23:02 +01:00
Arnaud Gourlay
6f1ef42907 Promote criterion to workspace (#5291) 2024-10-22 16:29:23 +02:00
Ivan Pleshkov
f27b6909bd Move quantization repo (#5096)
* move quantization repo

* are you happy fmt

* are you happy clippy

* remove dumping pq to image

* workspace deps

* are you happy clippy
2024-09-17 16:30:26 +02:00