Commit Graph

1113 Commits

Author SHA1 Message Date
Arnaud Gourlay
eaaa4e4ee1 Validate IntegerIndexParams (#6494)
* Validate IntegerIndexParams

* focus
2025-05-22 23:00:52 +02:00
Ivan Pleshkov
32fbc2eae2 Don't reallocate gpu resources between payload blocks (#6462)
* dont reallocate gpu resources between payload blocks

* are you happy codespell

* fix review remarks

* fix gpu tests

* are you happy clippy
2025-05-22 22:57:48 +02:00
Arnaud Gourlay
1da478c897 Fix strict mode unindexed group_by path (#6363)
* Fix strict mode unindexed group_by path

* check index schema for matching support

* Handle disabled lookup on integer index
2025-05-22 22:56:24 +02:00
Tim Visée
7869c1ee90 Use byte strings for geohash (#6437)
* Use byte strings for geohash

* Don't take references
2025-05-22 22:56:02 +02:00
Roman Titov
1bcf466b17 Make segment RocksDB optional, only initialize if required (#6441) 2025-05-22 22:53:41 +02:00
Tim Visée
a83db583bc Fully replace SmolStr with ecow string (#6427)
* Replace SmolStr with EcoString

* Directly format GeoHash, remove intermediate string
2025-05-22 22:50:43 +02:00
Arnaud Gourlay
63cb49f055 Debug assert a missing column family or not loaded payload index (#6193)
* assertion on non built index

* Remove old custome storage for gridstore
2025-05-22 22:50:36 +02:00
Luis Cossío
e19f39ba9d [#6266] Remove incorrect assertion (#6428) 2025-05-22 22:49:39 +02:00
Tim Visée
43a48b867a Use ecow string for shard keys (#6426) 2025-05-22 22:49:33 +02:00
Roman Titov
5483581064 Add partial snapshot API endpoints (#6304) 2025-05-22 22:49:26 +02:00
xzfc
ce1adf4026 Move filtering logic from RawScorer to FilteredScorer (#6245) 2025-05-22 22:49:19 +02:00
Tim Visée
452aea3f1f Use from rather than into (#6425) 2025-05-22 22:49:12 +02:00
Tim Visée
3b11a8f505 Use smol string as shard key (#6420)
* Use SmolStr as shard key type

* Simplify conversion
2025-05-22 22:49:03 +02:00
Andrey Vasnetsov
492bf0387b careful index creation (#6403)
* make sure to delete existing index, if it is not compatible with a new one

* fmt

* fix tests

* review fixes
2025-05-22 22:48:39 +02:00
Luis Cossío
031882ce5c [mutable text index] No need to wrap vec in option (#6391) 2025-05-22 22:46:47 +02:00
Luis Cossío
a304a51f84 [rate limits] consider length of multivectors for query cost (#6356)
* consider length of multivectors for rate limiter

* add openapi test

* organize imports

* better cost estimation for sparse vectors

* restore groups openapi test

* address @timvisee's review

* Fix consensus test
2025-05-22 22:46:39 +02:00
Luis Cossío
ef5dbdeae6 [full-text index] return earlier in text query parsing (#6264) 2025-05-22 22:46:31 +02:00
Andrey Vasnetsov
ebdb26841e fill ready-list after initial CPU build (#6410) 2025-04-22 10:18:46 +02:00
Tim Visée
a6d57e910a Qdrant 1.14: enable mutable ID tracker by default (#6268)
* Enable mutable ID tracker by default

* Remove now obsolete feature flag for new mutable ID tracker
2025-04-21 00:51:00 +02:00
Kumar Shivendu
83af9cf3ba Improve telemetry logic and test (#6399)
* Improve telemetry logic and test

* Parametrize telemetry test

* Consistency hash peeer ID across telemetry

* clean test

* Use Option in segments telemetry

* updat openapi spec

* Avoid test failure on change in order of params
2025-04-21 00:12:26 +02:00
Andrey Vasnetsov
bbea80e7d4 Telemetry improvements (#6390)
* allow shard states in anonymize telemetry (with hashed peer ids)

* introduce level3 and level4 for telemetry, where level4 = everything, incl. segments info

* upd openapi

* fix tests

* expose vector count & size stats on shard level to avoid reading of segments

* fix spelling

* upd schema

* fix tests

* Use unwrap_or_default

* [#6390] skip serializing shard details in Level2 (#6398)

* skip serializing shard details in Level2

* upd openapi

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
2025-04-21 00:10:28 +02:00
Tim Visée
e59d395d80 Use ahash for maps/sets holding point IDs, offsets or similar (#6388) 2025-04-21 00:10:22 +02:00
xzfc
1f805b618e Clear disk cache after read (#6396)
* Move clear_disk_cache to memory::fadvise

* Add memory::fadvise::OneshotFile

* Use OneshotFile
2025-04-21 00:08:52 +02:00
Andrey Vasnetsov
a9795f7ce4 faster mmap numeric index (#6381)
* wip: attempt to investigate slow mmap numeric index

* fmt

* Improve performance of ConditionedCounter (#6384)

* Improve Performance of ConditionedCounter

* Remove unneeded hw_counter from signatures

* Clippy

* Fix Clippy II

* Fix Clippy III

* Fix Clippy IV

* Use static string, don't allocate string

* Remove allow dead code attribute

---------

Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
2025-04-21 00:07:08 +02:00
Tim Visée
4f6d3b928a Speed up pending changes lookups using ahash (#6385)
* Add bench for mmap_bitslice_buffered_update_wrapper

* Use ahash in buffered mmap/RocksDB types and in payload storage

* Function is used in testing only
2025-04-21 00:06:46 +02:00
Andrey Vasnetsov
2b86a4b020 Fix update wrapper (#6383)
* fix flush routine for persistent wrappers

* fmt

* rollback changes for write-only wrappers

* Use retain to drop flushed items

* Rename function

---------

Co-authored-by: timvisee <tim@visee.me>
2025-04-21 00:05:56 +02:00
Jojii
121b2d543e Fix some wrong vector-read IO measurements (#6365)
* Fix some wrong vector_storage io measurements

* Review remarks
2025-04-21 00:02:45 +02:00
Luis Cossío
076bcdaa16 refactor QueryEnum::Nearest for using NamedQuery (#6349)
* refactor QueryEnum::Nearest for using NamedQuery

* use explicit `NamedQuery::from`
2025-04-20 23:58:39 +02:00
xzfc
7a1a96e22b Incremental HNSW index building: append-only case (#6325)
* Pass FeatureFlags into VectorIndexBuildArgs

* Incremental HNSW index building: append-only case

* Use debug_assert

* first_few_ids

* Check deleted_point_count

* Drop unused method
2025-04-20 23:55:57 +02:00
xzfc
3d988e66c4 Remove is_stopped from RawScorer (#6305)
* Introduce CancelledError

* Remove is_stopped from RawScorer
2025-04-20 23:55:39 +02:00
Andrey Vasnetsov
6e0ddbafa9 disk cache hygiene (#6323)
* wip: implement explicit populate and clear_cache functions for all components

* fmt

* implement clear and populate for vector storages

* fmt

* implement clear and populate for payload storage

* wip: implement explicit populate and clear_cache functions payload indexes

* implement explicit populate and clear_cache functions payload indexes

* fix clippy on CI

* only compile posix_fadvise on linux

* only compile posix_fadvise on linux

* implement explicit populate and clear_cache functions for quantized vectors

* fmt

* remove post-load prefault

* fix typo

* implement is-on-disk for payload indexes, implement clear on drop for segment, implement clear after segment build

* fmt

* also evict quantized vectors after optimization

* re-use and replace advise_dontneed
2025-04-20 23:54:40 +02:00
xzfc
ab40beb65d Update hnsw_build_asymptotic benchmark (#6299)
* Update hnsw_build_asymptotic benchmark

* Comments and logs
2025-04-20 23:53:09 +02:00
Luis Cossío
317f1e0dbe [strict mode | score boosting] Strict mode for formula queries (#6317)
* move problems.rs to collection crate

* change type on datetime_key expression

* extract from expression

* check for unindexed fields in formula

* clippy

* dedup code

* Only implement verification for `CollectionQuery` types (#6319)

* verify CollectionQuery's Prefetch

* remove implementation for REST query

* remove implementation for REST query groups

* add basic openapi test
2025-04-20 23:52:36 +02:00
Arnaud Gourlay
6e2191fcae Document Multivector Euclidean scoring discrepancy (#6320) 2025-04-20 23:52:28 +02:00
Jojii
4585c3571c Fix measuring in memory inverted index (#6316)
* Fix measuring in memory inverted index

* Reduce overhead of get_token_id

* Remove not used function
2025-04-20 23:51:43 +02:00
Jojii
231d579f70 Don't measure in memory geo, map and numeric indices (#6311) 2025-04-20 23:51:37 +02:00
Jojii
dcd5674ebb Writeback counter cell (#6190)
* Add WritebackCounterCell

* Replace custom counting with WritebackCounterCell

* Update lib/common/common/src/counter/counter_cell.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2025-04-20 23:51:26 +02:00
Jojii
07fbe5465b Don't measure in-memory hw for full text indices (#6309)
* Don't measure in-memory full text index

* Clippy

* Fix mmap_postings conditioned counting
2025-04-20 23:51:17 +02:00
Jojii
5117e24758 Only measure HW if on disk for bool index (#6296)
* Only measure HW if on disk for bool index

* Remove missed measurement
2025-04-20 23:51:08 +02:00
xzfc
ee7b764429 Use MaybeUninit in vector storages and query scorers (#6303) 2025-04-20 23:50:11 +02:00
Luis Cossío
e114801ebf sum_scores recommendation strategy (#6256)
* Add in rest and grpc

* add to QueryEnum

* implement Query trait

* connect to scorer creation

* upd tests

* additional changes

* fmt

* gen openapi and grpc docs

* coderabbit fix

* add changes in async scorer

* test sum_scores in more places, refactor to remove repetition
2025-04-20 23:47:58 +02:00
Luis Cossío
8084f4cac1 [score boosting] support datetime expressions (#6196)
* use f64 as internal expression score

* allow datetime as expression

* add test cases

* fix schema, generate openapi and grpc docs

* fmt

* clippy

* fix test precision

* graceful handling of too large numbers, more tests

* rename to DatetimeExpression for consistency

* use datetime's timestamp as seconds

* capture variable name when parsing

* homogenize DateTime into Datetime

* use interface distinction between datetime strings and payload keys

* fix rebase

* fix after rebase
2025-04-20 23:47:20 +02:00
Roman Titov
90340a098c Implement partial snapshot recovery (#6206)
- Fix creating partial snapshot without RocksDB backup
- Fix `segment_manifest.json` file placement
- Implement partial snapshot recovery
- Allow restoring partial snapshot for all shard types
- Validate segment manifests when recovering partial snapshot
- Log error, if streaming shard/partial snapshot failed
- Contextualize errors in `snapshot_files` function
- Tweak log messages in `Segment::take_snapshot` method
- Remove unnecessary lifetimes from `SegmentHolder` iter methods
2025-04-20 23:46:47 +02:00
Arnaud Gourlay
89667860dd Minor cleanups (#6291) 2025-04-20 23:42:17 +02:00
Arnaud Gourlay
33e7aa0b98 Fixes for Clippy 1.86 (#6290) 2025-04-20 23:42:07 +02:00
dependabot[bot]
58813a33e3 build(deps): bump geo from 0.29.3 to 0.30.0 (#6286) 2025-04-20 23:42:00 +02:00
Luis Cossío
17bcc79334 [query] Optimize fetching payloads/vectors at local shard (#6279)
* introduce `RootPlan`

* handle changes at local shard
2025-03-31 20:51:34 +02:00
Andrey Vasnetsov
2baa8cda53 implement missing validation for sparse and milti-vectors in grpc (#6274) 2025-03-31 16:21:25 +02:00
Tim Visée
c232595afb Mutable ID tracker: improve version flushing, grow file in one shot (#6267)
* In mutable ID tracker, grow versions file in one shot

* Fix typos
2025-03-31 16:21:21 +02:00
Andrey Vasnetsov
b5d5dd1579 add DimWeight mult (#6263) 2025-03-31 16:21:00 +02:00