Commit Graph

17 Commits

Author SHA1 Message Date
Andrey Vasnetsov
82af7aa481 vector-io-read measurement on query (#6197)
* remove mut getters from HardwareCounterCell, as mutability is not useful

* introduce vector-io multiplier

* remove RealCpuMeasurement structure

* set vector-io reads multipliers

* account vector reads in dense scorers

* fmt

* fix tests

* propagate hw_counter into posting list iterator

* fmt

* fix test

* wip: measure of sparse iterator

* fmt

* optimize skip_to

* minor refactoring

* keep current PointOffset in iterator to prevent unnecessary reads from memory

* adjust sparse search cpu cost - account for datatype

* fix test

* refactor search_context tests

* move tests into a module

* introduce more tests

* grammar

* review fixes

* fix clippy

* fix clippy again

* change disposable -> new
2025-03-24 13:15:04 +01:00
Jojii
e957d6f44a Cardinality estimation IO measurements (#6117)
* Cardinality estimation measurements

* Apply hw measurements to latest changes from dev

* Clippy

* Also measure cardinality estimation for geo index

* Make measured units 'bytes'

* Use PointOffsetType instead of u32 for size calculation

* fix memory cost for check_values_any in mmap index

* fix double counting for value reading in mmap, remove hw_counter from mmap hashmap

* fmt

* fix hw measurement for text index

* Remove non necessary lifetime annotations

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-03-14 11:05:38 +01:00
Arnaud Gourlay
423a39f301 Remove duplicated mmap utils (#6129) 2025-03-10 12:20:22 +01:00
Tim Visée
3e536347e1 Bump Rust edition to 2024 (#6042)
* Bump Rust edition to 2024

* gen is a reserved keyword now

* Remove ref mut on references

* Mark extern C as unsafe

* Wrap unsafe function bodies in unsafe block

* Geo hash implements Copy, don't reference but pass by value instead

* Replace secluded self import with parent

* Update execute_cluster_read_operation with new match semantics

* Fix lifetime issue

* Replace map_or with is_none_or

* set_var is unsafe now

* Reformat
2025-02-25 11:21:25 +01:00
Kumar Shivendu
38e1f4720b Quantization test improvements (#5976)
* minor change in test

* apply to other tests as well

* imporve var name

* add test for loading empty EncodedVectorsBin from disk

* fmt
2025-02-12 09:45:18 +01:00
Luis Cossío
af74d1b96a bump and migrate to rand 0.9.0 (#5892)
* bump and migrate to rand 0.9.0

also bump rand_distr to 0.5.0 to match it

* Migrate AVX2 and SSE implementations

* Remove unused thread_rng placeholders

* More random migrations

* Migrate GPU tests

* bump seed

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-01-28 16:19:11 +01:00
Arnaud Gourlay
51b574c9bd Fix missing common testing dep on blobstore (#5875)
* Fix missing common testing dep on blobstore

* also for quantization
2025-01-27 10:33:36 +01:00
Jojii
e0d0e233d8 Timeout aware hardware counter (#5555)
* Make hardware counting timeout aware

* improve test

* rebuild everything

* fmt

* post-rebase fixes

* upd tests

* fix tests

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-12-10 12:12:36 +01:00
Andrey Vasnetsov
2a5b46e780 use fsync instead of flush (#5629) 2024-12-10 00:13:03 +01:00
Ivan Pleshkov
1447d1efa8 atomic save quantization metadata (#5627) 2024-12-09 21:24:24 +01:00
xzfc
4548ebe4bf Reduce debug size (#5556)
* debug size: RawScorerImpl::peek_top_iter

* debug size: TypedMultiDenseVector::multi_vectors() uses

* debug size: stop_condition -> stopped
2024-12-02 14:33:35 +00:00
Ivan Pleshkov
a770a7f3a0 Gpu vectors (#5339)
* parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012991 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012977 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012901 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012766 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012730 +0100

vulkan api wrapper crate

fmt

gpu is not a workspace member

exclude gpu from workspace

disable tests for gpu crate

ash version up

change collection for pipeline builder

gpu test

gpu is not optional

check device features

remove feature

make all fields private

more comments

are you happy fmt

qdrant team as authors

gpu vectors

move changes from magic branch

move storage updates

are you happy codespell

are you happy clippy

don't make storage_impl public

review remarks

are you happy clippy

are you happy fmt

* Update lib/segment/src/index/hnsw_index/gpu/gpu_vector_storage/tests.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* review remarks

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2024-11-27 09:50:10 +01:00
Arnaud Gourlay
5bce09a1ed Merge quantization integration tests (#5473) 2024-11-19 13:10:01 +01:00
Arnaud Gourlay
2a0f9e9212 Use references for less cloning when possible (#5409) 2024-11-11 10:26:17 +01:00
Jojii
7b677fb7fa Hardware counting for quantization (#5369)
* add cpu measurement for quantization

* clippy

* fix tests

* discard hardware counters in benchmarks

* Forwarding hardware counter in distributed setup (#5371)

* make hardware counter available in distributed setup

* clippy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-06 16:23:02 +01:00
Arnaud Gourlay
6f1ef42907 Promote criterion to workspace (#5291) 2024-10-22 16:29:23 +02:00
Ivan Pleshkov
f27b6909bd Move quantization repo (#5096)
* move quantization repo

* are you happy fmt

* are you happy clippy

* remove dumping pq to image

* workspace deps

* are you happy clippy
2024-09-17 16:30:26 +02:00