upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-06 18:10:58 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,465 Commits 346 Branches 118 Tags
94de0e2d2dcf997bf26bec34dad407d8de9a9abb
Commit Graph

1780 Commits

Author SHA1 Message Date
Luis Cossío
3749fb6db6 [full-text index] return earlier in text query parsing (#6264) 2025-04-22 13:16:43 -04:00
Andrey Vasnetsov
13bf907261 fill ready-list after initial CPU build (#6410) 2025-04-22 10:18:09 +02:00
Tim Visée
9bf8340001 Qdrant 1.14: enable mutable ID tracker by default (#6268) 
* Enable mutable ID tracker by default

* Remove now obsolete feature flag for new mutable ID tracker
 2025-04-21 00:13:40 +02:00
Kumar Shivendu
d3d639ea9d Improve telemetry logic and test (#6399) 
* Improve telemetry logic and test

* Parametrize telemetry test

* Consistency hash peeer ID across telemetry

* clean test

* Use Option in segments telemetry

* updat openapi spec

* Avoid test failure on change in order of params
 2025-04-18 15:31:37 +02:00
Andrey Vasnetsov
c57d748dd4 Telemetry improvements (#6390) 
* allow shard states in anonymize telemetry (with hashed peer ids)

* introduce level3 and level4 for telemetry, where level4 = everything, incl. segments info

* upd openapi

* fix tests

* expose vector count & size stats on shard level to avoid reading of segments

* fix spelling

* upd schema

* fix tests

* Use unwrap_or_default

* [#6390] skip serializing shard details in Level2 (#6398)

* skip serializing shard details in Level2

* upd openapi

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
 2025-04-17 23:32:10 +02:00
Tim Visée
1370e01d58 Use ahash for maps/sets holding point IDs, offsets or similar (#6388) 2025-04-17 23:11:35 +02:00
xzfc
9fedc65bc4 Clear disk cache after read (#6396) 
* Move clear_disk_cache to memory::fadvise

* Add memory::fadvise::OneshotFile

* Use OneshotFile
 2025-04-17 10:57:47 +02:00
Andrey Vasnetsov
6d0da2e552 faster mmap numeric index (#6381) 
* wip: attempt to investigate slow mmap numeric index

* fmt

* Improve performance of ConditionedCounter (#6384)

* Improve Performance of ConditionedCounter

* Remove unneeded hw_counter from signatures

* Clippy

* Fix Clippy II

* Fix Clippy III

* Fix Clippy IV

* Use static string, don't allocate string

* Remove allow dead code attribute

---------

Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
 2025-04-15 16:58:38 +02:00
Tim Visée
10ef9eef72 Speed up pending changes lookups using ahash (#6385) 
* Add bench for mmap_bitslice_buffered_update_wrapper

* Use ahash in buffered mmap/RocksDB types and in payload storage

* Function is used in testing only
 2025-04-15 14:06:23 +02:00
Andrey Vasnetsov
8c9687b63c Fix update wrapper (#6383) 
* fix flush routine for persistent wrappers

* fmt

* rollback changes for write-only wrappers

* Use retain to drop flushed items

* Rename function

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-04-15 10:48:41 +02:00
dependabot[bot]
007213e76a build(deps): bump self_cell from 1.1.0 to 1.2.0 (#6380) 2025-04-14 21:07:04 +00:00
dependabot[bot]
8cd8c84b88 build(deps): bump anyhow from 1.0.97 to 1.0.98 (#6374) 2025-04-14 20:06:16 +00:00
Jojii
4275d336dc Fix some wrong vector-read IO measurements (#6365) 
* Fix some wrong vector_storage io measurements

* Review remarks
 2025-04-14 10:17:48 +02:00
Luis Cossío
ca22191322 refactor QueryEnum::Nearest for using NamedQuery (#6349) 
* refactor QueryEnum::Nearest for using NamedQuery

* use explicit `NamedQuery::from`
 2025-04-10 09:53:31 -04:00
xzfc
89d399ef99 Incremental HNSW index building: append-only case (#6325) 
* Pass FeatureFlags into VectorIndexBuildArgs

* Incremental HNSW index building: append-only case

* Use debug_assert

* first_few_ids

* Check deleted_point_count

* Drop unused method
 2025-04-09 18:56:53 +00:00
xzfc
a534e954ae Remove is_stopped from RawScorer (#6305) 
* Introduce CancelledError

* Remove is_stopped from RawScorer
 2025-04-09 10:54:42 +02:00
Andrey Vasnetsov
0266508736 disk cache hygiene (#6323) 
* wip: implement explicit populate and clear_cache functions for all components

* fmt

* implement clear and populate for vector storages

* fmt

* implement clear and populate for payload storage

* wip: implement explicit populate and clear_cache functions payload indexes

* implement explicit populate and clear_cache functions payload indexes

* fix clippy on CI

* only compile posix_fadvise on linux

* only compile posix_fadvise on linux

* implement explicit populate and clear_cache functions for quantized vectors

* fmt

* remove post-load prefault

* fix typo

* implement is-on-disk for payload indexes, implement clear on drop for segment, implement clear after segment build

* fmt

* also evict quantized vectors after optimization

* re-use and replace advise_dontneed
 2025-04-09 10:54:30 +02:00
dependabot[bot]
d2961a866c build(deps): bump smallvec from 1.14.0 to 1.15.0 (#6340) 
Bumps [smallvec](https://github.com/servo/rust-smallvec) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.14.0...v1.15.0)

---
updated-dependencies:
- dependency-name: smallvec
  dependency-version: 1.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-04-08 08:01:04 +02:00
xzfc
613ef07622 Update hnsw_build_asymptotic benchmark (#6299) 
* Update hnsw_build_asymptotic benchmark

* Comments and logs
 2025-04-07 13:23:22 +00:00
Arnaud Gourlay
d5aa0d78e8 Remove unused Issues dependency on Segment (#6331) 2025-04-07 08:56:39 -04:00
Luis Cossío
f9a537174c [strict mode | score boosting] Strict mode for formula queries (#6317) 
* move problems.rs to collection crate

* change type on datetime_key expression

* extract from expression

* check for unindexed fields in formula

* clippy

* dedup code

* Only implement verification for `CollectionQuery` types (#6319)

* verify CollectionQuery's Prefetch

* remove implementation for REST query

* remove implementation for REST query groups

* add basic openapi test
 2025-04-04 13:52:01 -03:00
Arnaud Gourlay
7723d91427 Document Multivector Euclidean scoring discrepancy (#6320) 2025-04-04 15:54:28 +02:00
Jojii
90d89ca203 Fix measuring in memory inverted index (#6316) 
* Fix measuring in memory inverted index

* Reduce overhead of get_token_id

* Remove not used function
 2025-04-03 21:43:58 +02:00
Jojii
bd9a657f07 Don't measure in memory geo, map and numeric indices (#6311) 2025-04-03 20:57:03 +02:00
Jojii
2ef9612432 Writeback counter cell (#6190) 
* Add WritebackCounterCell

* Replace custom counting with WritebackCounterCell

* Update lib/common/common/src/counter/counter_cell.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
 2025-04-03 20:09:29 +02:00
Jojii
bf706c21bc Don't measure in-memory hw for full text indices (#6309) 
* Don't measure in-memory full text index

* Clippy

* Fix mmap_postings conditioned counting
 2025-04-03 19:43:32 +02:00
Jojii
821aa0a33d Only measure HW if on disk for bool index (#6296) 
* Only measure HW if on disk for bool index

* Remove missed measurement
 2025-04-03 19:04:33 +02:00
xzfc
43dc5505fb Use MaybeUninit in vector storages and query scorers (#6303) 2025-04-03 09:44:57 +00:00
Luis Cossío
5670a7c917 sum_scores recommendation strategy (#6256) 
* Add in rest and grpc

* add to QueryEnum

* implement Query trait

* connect to scorer creation

* upd tests

* additional changes

* fmt

* gen openapi and grpc docs

* coderabbit fix

* add changes in async scorer

* test sum_scores in more places, refactor to remove repetition
 2025-04-03 09:01:39 +02:00
Luis Cossío
ec544ff627 [score boosting] support datetime expressions (#6196) 
* use f64 as internal expression score

* allow datetime as expression

* add test cases

* fix schema, generate openapi and grpc docs

* fmt

* clippy

* fix test precision

* graceful handling of too large numbers, more tests

* rename to DatetimeExpression for consistency

* use datetime's timestamp as seconds

* capture variable name when parsing

* homogenize DateTime into Datetime

* use interface distinction between datetime strings and payload keys

* fix rebase

* fix after rebase
 2025-04-02 12:40:00 -03:00
Roman Titov
dce4a94da0 Implement partial snapshot recovery (#6206) 
- Fix creating partial snapshot without RocksDB backup
- Fix `segment_manifest.json` file placement
- Implement partial snapshot recovery
- Allow restoring partial snapshot for all shard types
- Validate segment manifests when recovering partial snapshot
- Log error, if streaming shard/partial snapshot failed
- Contextualize errors in `snapshot_files` function
- Tweak log messages in `Segment::take_snapshot` method
- Remove unnecessary lifetimes from `SegmentHolder` iter methods
 2025-04-02 15:11:13 +02:00
Arnaud Gourlay
33d062a3a7 Minor cleanups (#6291) 2025-04-01 13:01:10 +02:00
Arnaud Gourlay
e77af48078 Fixes for Clippy 1.86 (#6290) 2025-04-01 12:26:37 +02:00
dependabot[bot]
c0aac9643d build(deps): bump geo from 0.29.3 to 0.30.0 (#6286) 2025-04-01 09:03:32 +00:00
Luis Cossío
162ab1ffc5 [query] Optimize fetching payloads/vectors at local shard (#6279) 
* introduce `RootPlan`

* handle changes at local shard
 2025-03-31 15:50:32 -03:00
Andrey Vasnetsov
fbf1ce4d17 implement missing validation for sparse and milti-vectors in grpc (#6274) 2025-03-31 11:22:53 +02:00
Tim Visée
f23a9926a9 Mutable ID tracker: improve version flushing, grow file in one shot (#6267) 
* In mutable ID tracker, grow versions file in one shot

* Fix typos
 2025-03-28 16:21:18 +01:00
Andrey Vasnetsov
08b0e1cd05 add DimWeight mult (#6263) 2025-03-27 14:38:40 +01:00
Luis Cossío
3ad35301a0 [order_by] Deprecate using order_value from payload, dedup by (order_value, id) tuple (#6233) 
* deprecate inserting order_value in payload, use the one in the struct instead

* fix for test

* dedup by (order_value, id) tuple

* Scroll dedup test must also consider order value

* Update assert message, include order value

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-03-27 10:07:43 -03:00
Roman Titov
553bce482c Merge take_snapshot and take_partial_snapshot methods (#6232) 2025-03-26 16:46:12 +01:00
Jojii
abbb56f651 Measure Payload IO reads on index creation (#6252) 2025-03-25 17:38:36 +01:00
dependabot[bot]
4264829239 build(deps): bump charabia from 0.9.2 to 0.9.3 (#6237) 
Bumps [charabia](https://github.com/meilisearch/charabia) from 0.9.2 to 0.9.3.
- [Release notes](https://github.com/meilisearch/charabia/releases)
- [Commits](https://github.com/meilisearch/charabia/compare/v0.9.2...v0.9.3)

---
updated-dependencies:
- dependency-name: charabia
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-03-25 10:09:25 +01:00
dependabot[bot]
7f3a2c2981 build(deps): bump log from 0.4.26 to 0.4.27 (#6247) 
* build(deps): bump log from 0.4.26 to 0.4.27

Bumps [log](https://github.com/rust-lang/log) from 0.4.26 to 0.4.27.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.26...0.4.27)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* put variables inside the strings for log macros

* also for pyroscope

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
 2025-03-25 10:08:21 +01:00
Jojii
ae64690c7a Measure Payload Index IO Writes (#6137) 
* Prepare measurement of index creation + Remove vector deletion
measurement

* add hw_counter to add_point functions

* Adjust add_point(..) function signatures

* Add new measurement type: payload index IO write

* Measure payload index IO writes

* Some Hw measurement performance improvements

* Review remarks

* Fix measurements in distributed setups

* review fixes

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2025-03-24 19:39:17 +01:00
Andrey Vasnetsov
82af7aa481 vector-io-read measurement on query (#6197) 
* remove mut getters from HardwareCounterCell, as mutability is not useful

* introduce vector-io multiplier

* remove RealCpuMeasurement structure

* set vector-io reads multipliers

* account vector reads in dense scorers

* fmt

* fix tests

* propagate hw_counter into posting list iterator

* fmt

* fix test

* wip: measure of sparse iterator

* fmt

* optimize skip_to

* minor refactoring

* keep current PointOffset in iterator to prevent unnecessary reads from memory

* adjust sparse search cpu cost - account for datatype

* fix test

* refactor search_context tests

* move tests into a module

* introduce more tests

* grammar

* review fixes

* fix clippy

* fix clippy again

* change disposable -> new
 2025-03-24 13:15:04 +01:00
Tim Visée
4f8735462d Add feature flag to skip RocksDB in immutable payload indices (#6226) 2025-03-22 11:01:24 +01:00
Andrey Vasnetsov
f5b4e382ba (Potentially) Fix missing CF on flush on payload index change (#6214) 
* Handle missing CF on the payload index flush level

* move error handling on the payload field level, so other fields are not skipped if one fails to flush

* Merge nested match statement

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-03-21 11:30:14 +01:00
Kumar Shivendu
903c9d5e0b Snapshot logging improve (#6192) 
* Add details to snapshot logging for more details

* Format code and improve log msg

* Remove unused import and improve logs

* Improve terminology for logging

* Minimize noise using logging levels

* Log when moving local shard

* Apply suggestions from code review

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

* fmt fix

---------

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
 2025-03-21 13:51:36 +05:30
Roman Titov
8b44f08187 Check use_mutable_id_tracker_without_rocksdb feature flag when building new segment (#6211) 2025-03-20 17:45:49 +01:00
Andrey Vasnetsov
3268c92435 dont anonymize shard transfers (#6208) 
* dont anonymize shard transfers

* rabbit review fix

* Remove unused Anonymize derive

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
 2025-03-20 15:12:00 +01:00