upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-07 02:20:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,465 Commits 346 Branches 118 Tags
94de0e2d2dcf997bf26bec34dad407d8de9a9abb
Commit Graph

3618 Commits

Author SHA1 Message Date
Luis Cossío
6ca50680f5 Buffered mutable bool index (#6977) 
* use buffered dynamic flags in bool index

* put base_dir out of Storage

* Adjust populate comment

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-08-06 09:10:27 -05:00
Ivan Pleshkov
766b527ac8 Appendable quantization storage (#6935) 
* appendable qunatization storage

fmt

deprecate count in quantization config

fix arm tests build

qunatized storage vectors count

fix ci

are you happy clippy

fix compat tests

undo rename to `deprecated_count`

remove flusher

remove obsolete functions, don't use hardware counter in ram storage

are you happy clippy

fix gpu build

check ci when revert option

revert last commit

* debug ci

* log offsets

* log offsets

* fix compatibility tests

* deprecate count

* fix arm tests

* fix benches

* Update lib/quantization/src/encoded_vectors_binary.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
 2025-08-06 13:51:01 +02:00
xzfc
542f03b033 TestRawScorerProducer to use VectorStorageEnum (#6965) 
* TestRawScorerProducer to use VectorStorageEnum

* TestRawScorerProducer:🆕 accept distance as an argument

* Drop TMetric from TestRawScorerProducer

* Remove `impl VectorStorage for TestRawScorerProducer`

* Add TestRawScorerProducer::internal_scorer

* Remove TestRawScorerProducer::get_vector

* Rename TestRawScorerProducer::{get_scorer => scorer}
 2025-08-06 09:41:13 +00:00
Kumar Shivendu
a4220fdd84 Partial snapshots update recovery ts (#6980) 
* Update recovery ts despite empty partial snapshots

* fmt

* apply suggestions with some improvements
 2025-08-06 10:48:03 +02:00
Arnaud Gourlay
255dd51e5a Fix Clippy 1.89 (#6981) 2025-08-05 18:06:04 +02:00
Arnaud Gourlay
a2b9a50381 Remove hardware measurement from fulltext index (#6884) 2025-08-05 17:28:49 +02:00
Tim Visée
6a116a54a9 Fix flushing of dynamic mmap flags (#6966) 
* Directly flush flags, don't use single-use flusher

* Explicitly flush before resizing flags file, prevent losing dirty pages

* Don't create now obsolete flusher when opening mmap

* clippy

* Move flush call

* flush before resizing only

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
 2025-08-05 17:05:29 +02:00
Luis Cossío
9fb92ca55d New mutable null index (#6954) 
* Introduce BufferedDynamicFlags

* clippy and extra changes

* fix flushing

* add mutable index impl

* use MutableNullIndex instead of MmapNullIndex

* use BufferedDynamicFlags

* remove mmap_null_index

* take updates before returning the flusher

* bump total_point_count on add and remove point

* improve test case

* Only clean up mutable null index files if index exists

* Remove mention of cached in context of length

* refactor flags into a single `Option<Storage>`

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-08-05 17:04:40 +02:00
Luis Cossío
4d92a33f74 cleanup versions on segment repair (#6975) 2025-08-05 14:38:15 +02:00
Jojii
c2a9183f5f BM25 in the Core (#6891) 
* Uncouple tokenizer and TextIndexParams

* Refactor token processing in `TokenizerConfig` (#6912)

* Refactor token processing in TokenizerConfig

* fix max length checking

* [Bm25] Execution implementation (#6939)

* Add Bm25

* Execute BM25 if config available

* cargo format

* Add mocked tests for inference and bm25

* Properly apply InferenceType

* Fix tests

* Review remarks

* Review remarks

* Add overwritten model name fix again

* use enum instead of multiple constants

* ensure handling all fields of InferenceInput in infer_local

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>

* review fixes

* fmt

* spell-check

* deduplicate code

---------

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2025-08-03 20:43:20 -06:00
Luis Cossío
a7701587f4 [decay formula] allow [0,1] range for midpoint in lin_decay (#6959) 
* allow [0,1] range for midpoint in lindecay

* clippyyyyy

* Better error message
 2025-08-01 13:49:54 -06:00
Roman Titov
bd34d52578 Cleanup partial snapshot stuff (#6925) 2025-08-01 18:33:32 +02:00
xzfc
e7038130e1 Add Distance::preprocess_vector (#6964) 2025-07-31 13:11:12 +00:00
xzfc
83005cd2db QueryScorerBytes (#6945) 
* Add EncodedVectorsBytes::score_point_vs_bytes

* Add QueryScorerBytes::score_bytes

* QuantizedVectors::query_scorer_bytes

* Split FilteredScorer into FilteredScorer and FilteredQuantizedScorer
 2025-07-31 03:05:10 +00:00
Kumar Shivendu
76cc63359d Bump dev version to 1.15.2-dev (#6955) 
* Bump dev version to 1.15.2-dev

* Updated cargo.lock

* Trigger CI
 2025-07-29 22:26:24 +05:30
Roman Titov
e8625ba903 Avoid creating partial snapshot if snapshot manifest contains no changes (#6924) 2025-07-29 14:18:57 +02:00
Luis Cossío
ba059b3e71 use roaring bitmap in map index (#6926) 2025-07-29 12:51:26 +02:00
Roman Titov
430b7925e2 Fix DateTimeWrapper parsing (#6952) 2025-07-29 12:41:32 +02:00
Roman Titov
22e15b9f40 Update collection-level payload index schema after recovering partial snapshot (#6886) 2025-07-29 12:25:24 +02:00
xzfc
9bd506d4f7 Introduce StableHash (#6940) 
* Introduce StableHash

* Use SipHasher24 explicitly
 2025-07-26 21:25:56 +00:00
Luis Cossío
03bbceaabb fix flaky welford calculation test (#6927) 
* use most significant digits for comparison

* use absolute or relative tolerance
 2025-07-25 08:47:45 -04:00
Andrey Vasnetsov
2f44084d7e Fix PointId Hash (#6932) 
* fmt

* clippy
 2025-07-24 08:31:29 +02:00
Andrey Vasnetsov
6fdf445570 UUID index bug (#6916) 
* reproduce bug as unit test

* reproduce in even faster unit test

* fix alignment in mmap_hashmap

* clippy

* review fixes
 2025-07-22 20:26:35 +02:00
xzfc
a959e8a671 MADV_POPULATE_READ on sequential mmaps (#6923) 2025-07-22 14:05:07 +02:00
dependabot[bot]
bd750eaf88 build(deps): bump roaring from 0.11.0 to 0.11.1 (#6921) 2025-07-22 06:46:43 +00:00
dependabot[bot]
7c3ac6f714 build(deps): bump object_store from 0.12.2 to 0.12.3 (#6918) 
---
updated-dependencies:
- dependency-name: object_store
  dependency-version: 0.12.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-07-22 08:22:04 +02:00
Tim Visée
b81e81c1f8 Fix checkfs in MUSL builds (#6903) 2025-07-20 16:15:50 +02:00
Luis Cossío
c02490a1a6 [MMR] make lazy matrix non-symmetric (#6900) 
* make lazy matrix non-symmetric

* remove SymmetricMatrix

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2025-07-18 00:40:14 +02:00
Luis Cossío
92c3bc810b also handle special case (#6898) 2025-07-17 14:42:14 -04:00
Luis Cossío
5c51c19434 MMR with postprocessed score (#6897) 
* handle score_threshold only during search

* use original score for mmr output
 2025-07-17 19:49:44 +02:00
Tim Visée
6f4069e711 Preprocess query vectors, sort sparse vectors (#6895) 
* Add test to assert behavior of unsorted sparse query

* Preprocess sparse vectors

* Fix typo
 2025-07-17 16:11:44 +02:00
Tim Visée
a5869beb99 Deprecations for Qdrant 1.15.0 (#6892) 
* Deprecate init_from

* Mark memmap_threshold as deprecated

* Mark locks API as deprecated

* Mark RBAC collection access payload filter as deprecated

* Allow deprecations in tests and benchmarks
 2025-07-17 16:10:45 +02:00
Jojii
16688a6a1b Replace Box with Either (#6887) 2025-07-17 11:04:43 +02:00
Luis Cossío
4f3d64ac59 [mmr] compute similarities lazily (#6877) 
* compute similarities lazily

* Update lib/collection/src/collection/mmr/lazy_matrix.rs

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-07-17 10:41:24 +02:00
Tim Visée
5ddda7e409 Don't delete payload index RocksDB if we still use RocksDB storage type (#6888) 2025-07-17 10:14:18 +02:00
Andrey Vasnetsov
168674973a lowercase stemmer language for consistency (#6889) 2025-07-17 09:53:34 +02:00
Arnaud Gourlay
1362563c2c Bump thread-priority 2.1.0 (#6885) 2025-07-16 17:14:06 +02:00
Tim Visée
cd1a31448d Fix broken is_empty filter on new points (#6882) 
* Add test to catch bug

Bug: <https://github.com/qdrant/qdrant/issues/6880>

* Fix loading null index from wrong path

* Simplify test

* Also bump total point count when removing a point

* Add debug assertion, ensure we don't create null index in selector

* Update bug link

* Remove unused import

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-07-16 14:25:09 +02:00
Kumar Shivendu
e14a003404 Fix bug that causes all replicas to die if node is restarted during resharding (#6800) 
* Fix bug that causes all replicas to die if node is restarted during resharding

* fix recursive async problem

* Avoid write lock unless required

* avoid using &mut when & is sufficient

* Remove is_in_progress since check_abort_resharding exists (#6806)

* On resharding abort, only abort transfers related to current operation

* Fix resharding dead replicas improvements (#6881)

* Apply suggestions to fix resharding dead replicas

* fmt

---------

Co-authored-by: timvisee <tim@visee.me>
 2025-07-16 17:51:30 +05:30
Tim Visée
edb460a5a6 After payload index migration, delete RocksDB files (#6873) 
* After payload index migration, delete RocksDB files

* Add comment on why it is safe to not clean up RocksDB files in some cases

* Update lib/segment/src/index/struct_payload_index.rs

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

* Reformat

---------

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
 2025-07-15 16:33:53 +02:00
Tim Visée
8b38f3a831 Adjust feature flag descriptions (#6875) 2025-07-15 15:31:37 +02:00
Tim Visée
70477eb5a9 Enable migration from RocksDB to mutable ID tracker by default (#6872) 
* Enable migration from RocksDB to mutable ID tracker by default

We planned to enable this by default in the Qdrant 1.15.0 release.

* Synchronize enable at version TODOs for other flags with planning

* Update OpenAPI spec
 2025-07-15 13:34:28 +02:00
Arnaud Gourlay
f95190c82e Do not track HW measurement when matching over a text index (#6833) 2025-07-15 11:30:48 +02:00
Arnaud Gourlay
a7ee4ceb51 Fix hardware counter vector_io for segment vector retrieval (#6863) 2025-07-15 11:30:22 +02:00
Tim Visée
70d231011b Fix payload index migration config and edge case handling (#6864) 
* Minor comment and log tweaks

* Persist payload index config when changed based on dirty flag

* Persist new index type when indices are rebuilt, they might have changed

* Still load RocksDB if some index still depends on it

* Fix typo
 2025-07-15 09:45:29 +02:00
dependabot[bot]
1c8e87d238 build(deps): bump ringbuffer from 0.15.0 to 0.16.0 (#6871) 2025-07-15 07:25:36 +00:00
dependabot[bot]
9d4b0929ad build(deps): bump sysinfo from 0.35.2 to 0.36.0 (#6869) 
---
updated-dependencies:
- dependency-name: sysinfo
  dependency-version: 0.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-07-15 09:08:12 +02:00
Ivan Pleshkov
6521b0df7d bq scalar query benches (#6831) 
* bq scalar query benches

* review remarks
 2025-07-14 19:28:07 +02:00
Tim Visée
09bdbaf58d In mmap payload indices, replace boxing with Either (#6861) 2025-07-14 15:48:34 +02:00
Roman Titov
c6e38d16a7 Avoid blocking async runtime when loading local shard (#6862) 2025-07-14 15:03:16 +02:00