Commit Graph

16 Commits

Author SHA1 Message Date
Arnaud Gourlay
1fb02a693c Fix vector_io_read for multivector recommendation (#6622)
* Fix vector_io_read for multivector recommendation

* fix

* cleanup

* single disposable

* do not use disposable hardware counter for quantized multivectors
2025-06-05 19:32:57 +02:00
Arnaud Gourlay
1e3017b304 Clippy 1.87 autofix (#6516)
* Clippy 1.87 autofix

* fmt
2025-05-12 15:51:34 +02:00
xzfc
9fedc65bc4 Clear disk cache after read (#6396)
* Move clear_disk_cache to memory::fadvise

* Add memory::fadvise::OneshotFile

* Use OneshotFile
2025-04-17 10:57:47 +02:00
Andrey Vasnetsov
0266508736 disk cache hygiene (#6323)
* wip: implement explicit populate and clear_cache functions for all components

* fmt

* implement clear and populate for vector storages

* fmt

* implement clear and populate for payload storage

* wip: implement explicit populate and clear_cache functions payload indexes

* implement explicit populate and clear_cache functions payload indexes

* fix clippy on CI

* only compile posix_fadvise on linux

* only compile posix_fadvise on linux

* implement explicit populate and clear_cache functions for quantized vectors

* fmt

* remove post-load prefault

* fix typo

* implement is-on-disk for payload indexes, implement clear on drop for segment, implement clear after segment build

* fmt

* also evict quantized vectors after optimization

* re-use and replace advise_dontneed
2025-04-09 10:54:30 +02:00
Andrey Vasnetsov
82af7aa481 vector-io-read measurement on query (#6197)
* remove mut getters from HardwareCounterCell, as mutability is not useful

* introduce vector-io multiplier

* remove RealCpuMeasurement structure

* set vector-io reads multipliers

* account vector reads in dense scorers

* fmt

* fix tests

* propagate hw_counter into posting list iterator

* fmt

* fix test

* wip: measure of sparse iterator

* fmt

* optimize skip_to

* minor refactoring

* keep current PointOffset in iterator to prevent unnecessary reads from memory

* adjust sparse search cpu cost - account for datatype

* fix test

* refactor search_context tests

* move tests into a module

* introduce more tests

* grammar

* review fixes

* fix clippy

* fix clippy again

* change disposable -> new
2025-03-24 13:15:04 +01:00
Jojii
e957d6f44a Cardinality estimation IO measurements (#6117)
* Cardinality estimation measurements

* Apply hw measurements to latest changes from dev

* Clippy

* Also measure cardinality estimation for geo index

* Make measured units 'bytes'

* Use PointOffsetType instead of u32 for size calculation

* fix memory cost for check_values_any in mmap index

* fix double counting for value reading in mmap, remove hw_counter from mmap hashmap

* fmt

* fix hw measurement for text index

* Remove non necessary lifetime annotations

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-03-14 11:05:38 +01:00
Arnaud Gourlay
423a39f301 Remove duplicated mmap utils (#6129) 2025-03-10 12:20:22 +01:00
Tim Visée
3e536347e1 Bump Rust edition to 2024 (#6042)
* Bump Rust edition to 2024

* gen is a reserved keyword now

* Remove ref mut on references

* Mark extern C as unsafe

* Wrap unsafe function bodies in unsafe block

* Geo hash implements Copy, don't reference but pass by value instead

* Replace secluded self import with parent

* Update execute_cluster_read_operation with new match semantics

* Fix lifetime issue

* Replace map_or with is_none_or

* set_var is unsafe now

* Reformat
2025-02-25 11:21:25 +01:00
Luis Cossío
af74d1b96a bump and migrate to rand 0.9.0 (#5892)
* bump and migrate to rand 0.9.0

also bump rand_distr to 0.5.0 to match it

* Migrate AVX2 and SSE implementations

* Remove unused thread_rng placeholders

* More random migrations

* Migrate GPU tests

* bump seed

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-01-28 16:19:11 +01:00
Andrey Vasnetsov
2a5b46e780 use fsync instead of flush (#5629) 2024-12-10 00:13:03 +01:00
Ivan Pleshkov
1447d1efa8 atomic save quantization metadata (#5627) 2024-12-09 21:24:24 +01:00
xzfc
4548ebe4bf Reduce debug size (#5556)
* debug size: RawScorerImpl::peek_top_iter

* debug size: TypedMultiDenseVector::multi_vectors() uses

* debug size: stop_condition -> stopped
2024-12-02 14:33:35 +00:00
Ivan Pleshkov
a770a7f3a0 Gpu vectors (#5339)
* parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012991 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012977 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012901 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012766 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012730 +0100

vulkan api wrapper crate

fmt

gpu is not a workspace member

exclude gpu from workspace

disable tests for gpu crate

ash version up

change collection for pipeline builder

gpu test

gpu is not optional

check device features

remove feature

make all fields private

more comments

are you happy fmt

qdrant team as authors

gpu vectors

move changes from magic branch

move storage updates

are you happy codespell

are you happy clippy

don't make storage_impl public

review remarks

are you happy clippy

are you happy fmt

* Update lib/segment/src/index/hnsw_index/gpu/gpu_vector_storage/tests.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* review remarks

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2024-11-27 09:50:10 +01:00
Arnaud Gourlay
2a0f9e9212 Use references for less cloning when possible (#5409) 2024-11-11 10:26:17 +01:00
Jojii
7b677fb7fa Hardware counting for quantization (#5369)
* add cpu measurement for quantization

* clippy

* fix tests

* discard hardware counters in benchmarks

* Forwarding hardware counter in distributed setup (#5371)

* make hardware counter available in distributed setup

* clippy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-06 16:23:02 +01:00
Ivan Pleshkov
f27b6909bd Move quantization repo (#5096)
* move quantization repo

* are you happy fmt

* are you happy clippy

* remove dumping pq to image

* workspace deps

* are you happy clippy
2024-09-17 16:30:26 +02:00