Commit Graph

20 Commits

Author SHA1 Message Date
xzfc
1f805b618e Clear disk cache after read (#6396)
* Move clear_disk_cache to memory::fadvise

* Add memory::fadvise::OneshotFile

* Use OneshotFile
2025-04-21 00:08:52 +02:00
Andrey Vasnetsov
6e0ddbafa9 disk cache hygiene (#6323)
* wip: implement explicit populate and clear_cache functions for all components

* fmt

* implement clear and populate for vector storages

* fmt

* implement clear and populate for payload storage

* wip: implement explicit populate and clear_cache functions payload indexes

* implement explicit populate and clear_cache functions payload indexes

* fix clippy on CI

* only compile posix_fadvise on linux

* only compile posix_fadvise on linux

* implement explicit populate and clear_cache functions for quantized vectors

* fmt

* remove post-load prefault

* fix typo

* implement is-on-disk for payload indexes, implement clear on drop for segment, implement clear after segment build

* fmt

* also evict quantized vectors after optimization

* re-use and replace advise_dontneed
2025-04-20 23:54:40 +02:00
Arnaud Gourlay
89667860dd Minor cleanups (#6291) 2025-04-20 23:42:17 +02:00
Andrey Vasnetsov
ad4743a85f vector-io-read measurement on query (#6197)
* remove mut getters from HardwareCounterCell, as mutability is not useful

* introduce vector-io multiplier

* remove RealCpuMeasurement structure

* set vector-io reads multipliers

* account vector reads in dense scorers

* fmt

* fix tests

* propagate hw_counter into posting list iterator

* fmt

* fix test

* wip: measure of sparse iterator

* fmt

* optimize skip_to

* minor refactoring

* keep current PointOffset in iterator to prevent unnecessary reads from memory

* adjust sparse search cpu cost - account for datatype

* fix test

* refactor search_context tests

* move tests into a module

* introduce more tests

* grammar

* review fixes

* fix clippy

* fix clippy again

* change disposable -> new
2025-03-31 16:19:02 +02:00
Jojii
56a7cfdb20 Cardinality estimation IO measurements (#6117)
* Cardinality estimation measurements

* Apply hw measurements to latest changes from dev

* Clippy

* Also measure cardinality estimation for geo index

* Make measured units 'bytes'

* Use PointOffsetType instead of u32 for size calculation

* fix memory cost for check_values_any in mmap index

* fix double counting for value reading in mmap, remove hw_counter from mmap hashmap

* fmt

* fix hw measurement for text index

* Remove non necessary lifetime annotations

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-03-21 11:44:55 +01:00
Arnaud Gourlay
ba9955ea92 Remove duplicated mmap utils (#6129) 2025-03-21 11:42:23 +01:00
Tim Visée
8ad2b34265 Bump Rust edition to 2024 (#6042)
* Bump Rust edition to 2024

* gen is a reserved keyword now

* Remove ref mut on references

* Mark extern C as unsafe

* Wrap unsafe function bodies in unsafe block

* Geo hash implements Copy, don't reference but pass by value instead

* Replace secluded self import with parent

* Update execute_cluster_read_operation with new match semantics

* Fix lifetime issue

* Replace map_or with is_none_or

* set_var is unsafe now

* Reformat
2025-03-21 11:38:56 +01:00
Kumar Shivendu
f18994c1f0 Quantization test improvements (#5976)
* minor change in test

* apply to other tests as well

* imporve var name

* add test for loading empty EncodedVectorsBin from disk

* fmt
2025-02-17 10:20:44 +01:00
Luis Cossío
f110328296 bump and migrate to rand 0.9.0 (#5892)
* bump and migrate to rand 0.9.0

also bump rand_distr to 0.5.0 to match it

* Migrate AVX2 and SSE implementations

* Remove unused thread_rng placeholders

* More random migrations

* Migrate GPU tests

* bump seed

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-02-11 14:36:20 +01:00
Arnaud Gourlay
8559ba5c1a Fix missing common testing dep on blobstore (#5875)
* Fix missing common testing dep on blobstore

* also for quantization
2025-01-27 18:25:56 +01:00
Jojii
5aee24cc08 Timeout aware hardware counter (#5555)
* Make hardware counting timeout aware

* improve test

* rebuild everything

* fmt

* post-rebase fixes

* upd tests

* fix tests

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-01-08 14:03:52 +01:00
Andrey Vasnetsov
4abd77158a use fsync instead of flush (#5629) 2025-01-08 14:01:25 +01:00
Ivan Pleshkov
ac52fc5994 atomic save quantization metadata (#5627) 2025-01-08 14:01:02 +01:00
xzfc
8bbe66c0fb Reduce debug size (#5556)
* debug size: RawScorerImpl::peek_top_iter

* debug size: TypedMultiDenseVector::multi_vectors() uses

* debug size: stop_condition -> stopped
2024-12-09 11:16:38 +01:00
Ivan Pleshkov
5327a322a6 Gpu vectors (#5339)
* parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012991 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012977 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012901 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012766 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012730 +0100

vulkan api wrapper crate

fmt

gpu is not a workspace member

exclude gpu from workspace

disable tests for gpu crate

ash version up

change collection for pipeline builder

gpu test

gpu is not optional

check device features

remove feature

make all fields private

more comments

are you happy fmt

qdrant team as authors

gpu vectors

move changes from magic branch

move storage updates

are you happy codespell

are you happy clippy

don't make storage_impl public

review remarks

are you happy clippy

are you happy fmt

* Update lib/segment/src/index/hnsw_index/gpu/gpu_vector_storage/tests.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* review remarks

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2024-12-09 11:06:20 +01:00
Arnaud Gourlay
61f38ec072 Merge quantization integration tests (#5473) 2024-12-09 10:47:02 +01:00
Arnaud Gourlay
98633cbd3f Use references for less cloning when possible (#5409) 2024-11-12 14:39:13 +01:00
Jojii
faf645ec2e Hardware counting for quantization (#5369)
* add cpu measurement for quantization

* clippy

* fix tests

* discard hardware counters in benchmarks

* Forwarding hardware counter in distributed setup (#5371)

* make hardware counter available in distributed setup

* clippy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-08 11:26:39 +01:00
Arnaud Gourlay
50a36448cb Promote criterion to workspace (#5291) 2024-11-08 11:11:49 +01:00
Ivan Pleshkov
21a3c108bb Move quantization repo (#5096)
* move quantization repo

* are you happy fmt

* are you happy clippy

* remove dumping pq to image

* workspace deps

* are you happy clippy
2024-09-23 16:03:29 +02:00