Commit Graph
3055 Commits
Author SHA1 Message Date
Luis Cossíoandtimvisee f2d45b558d Use BitvecFlags in vector storages (#6994)
* put all flags impls in a single module

* use in appendable_dense_vector_storage

* use in appendable_mmap_multi_dense_storage

* use in mmap_sparse_vector_storage

* clippy

* Add comment we cannot rely on length of deleted vectors bitvec

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-11 13:17:39 +02:00
Luis Cossíoandtimvisee 75b09284eb BitvecFlags: bitvec container with dynamic mmap flags storage (#6990)
* separate flags buffering and in-memory container

* new bitvec flags

* fmt

* clippy

* make BufferedDynamicFlags public to crate only

* clear cache after loading

* clippyyy

* improve comment

* Use resize with a constant to eliminate a hot closure

* Add comment on why we take subslice of bitslice

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-11 13:17:33 +02:00
Roman Titov 990eee11b4 Refactor Collection::restore_shard_snapshot (#6988) 2025-08-11 13:17:22 +02:00
Roman Titov 31738d3b85 Track payload index schema version (#6819) 2025-08-11 13:17:19 +02:00
Roman Titov 2938660e92 Fix test_create_shard_key_read_availability flakiness (#6902) 2025-08-11 13:17:11 +02:00
Luis Cossíoandtimvisee 356cdc833b Buffered mutable bool index (#6977)
* use buffered dynamic flags in bool index

* put base_dir out of Storage

* Adjust populate comment

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-11 13:17:05 +02:00
Ivan Pleshkovandxzfc f4ebdd9c9d Appendable quantization storage (#6935)
* appendable qunatization storage

fmt

deprecate count in quantization config

fix arm tests build

qunatized storage vectors count

fix ci

are you happy clippy

fix compat tests

undo rename to `deprecated_count`

remove flusher

remove obsolete functions, don't use hardware counter in ram storage

are you happy clippy

fix gpu build

check ci when revert option

revert last commit

* debug ci

* log offsets

* log offsets

* fix compatibility tests

* deprecate count

* fix arm tests

* fix benches

* Update lib/quantization/src/encoded_vectors_binary.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2025-08-11 13:17:02 +02:00
xzfc cfa5f9e804 TestRawScorerProducer to use VectorStorageEnum (#6965)
* TestRawScorerProducer to use VectorStorageEnum

* TestRawScorerProducer:🆕 accept distance as an argument

* Drop TMetric from TestRawScorerProducer

* Remove `impl VectorStorage for TestRawScorerProducer`

* Add TestRawScorerProducer::internal_scorer

* Remove TestRawScorerProducer::get_vector

* Rename TestRawScorerProducer::{get_scorer => scorer}
2025-08-11 13:16:55 +02:00
Kumar Shivendu 8be66adfc3 Partial snapshots update recovery ts (#6980)
* Update recovery ts despite empty partial snapshots

* fmt

* apply suggestions with some improvements
2025-08-11 13:16:49 +02:00
Arnaud Gourlay 118ff6dda3 Fix Clippy 1.89 (#6981) 2025-08-11 13:16:42 +02:00
Arnaud Gourlay 5ac506751f Remove hardware measurement from fulltext index (#6884) 2025-08-11 13:16:36 +02:00
Tim ViséeandLuis Cossío 9e914acd12 Fix flushing of dynamic mmap flags (#6966)
* Directly flush flags, don't use single-use flusher

* Explicitly flush before resizing flags file, prevent losing dirty pages

* Don't create now obsolete flusher when opening mmap

* clippy

* Move flush call

* flush before resizing only

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-08-11 13:16:33 +02:00
Luis Cossíoandtimvisee f45ee98293 New mutable null index (#6954)
* Introduce BufferedDynamicFlags

* clippy and extra changes

* fix flushing

* add mutable index impl

* use MutableNullIndex instead of MmapNullIndex

* use BufferedDynamicFlags

* remove mmap_null_index

* take updates before returning the flusher

* bump total_point_count on add and remove point

* improve test case

* Only clean up mutable null index files if index exists

* Remove mention of cached in context of length

* refactor flags into a single `Option<Storage>`

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-11 13:16:27 +02:00
Luis Cossío 1fb00f7975 cleanup versions on segment repair (#6975) 2025-08-11 13:16:24 +02:00
26a496b40e BM25 in the Core (#6891)
* Uncouple tokenizer and TextIndexParams

* Refactor token processing in `TokenizerConfig` (#6912)

* Refactor token processing in TokenizerConfig

* fix max length checking

* [Bm25] Execution implementation (#6939)

* Add Bm25

* Execute BM25 if config available

* cargo format

* Add mocked tests for inference and bm25

* Properly apply InferenceType

* Fix tests

* Review remarks

* Review remarks

* Add overwritten model name fix again

* use enum instead of multiple constants

* ensure handling all fields of InferenceInput in infer_local

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>

* review fixes

* fmt

* spell-check

* deduplicate code

---------

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2025-08-11 13:16:15 +02:00
Luis Cossío 5113c7c0f2 [decay formula] allow [0,1] range for midpoint in lin_decay (#6959)
* allow [0,1] range for midpoint in lindecay

* clippyyyyy

* Better error message
2025-08-11 13:16:10 +02:00
Roman Titov 343628b87c Cleanup partial snapshot stuff (#6925) 2025-08-11 13:16:07 +02:00
xzfc e329b0a43e Add Distance::preprocess_vector (#6964) 2025-08-11 13:15:57 +02:00
xzfc 0a53f72e31 QueryScorerBytes (#6945)
* Add EncodedVectorsBytes::score_point_vs_bytes

* Add QueryScorerBytes::score_bytes

* QuantizedVectors::query_scorer_bytes

* Split FilteredScorer into FilteredScorer and FilteredQuantizedScorer
2025-08-11 13:15:49 +02:00
Roman Titov bf319a0b40 Avoid creating partial snapshot if snapshot manifest contains no changes (#6924) 2025-08-11 13:15:39 +02:00
Luis Cossío 7589b28c01 use roaring bitmap in map index (#6926) 2025-08-11 13:15:31 +02:00
Roman Titov fdc5366884 Fix DateTimeWrapper parsing (#6952) 2025-08-11 13:15:27 +02:00
Roman Titov e6bdcb2520 Update collection-level payload index schema after recovering partial snapshot (#6886) 2025-08-11 13:15:23 +02:00
xzfc b79757d072 Introduce StableHash (#6940)
* Introduce StableHash

* Use SipHasher24 explicitly
2025-08-11 13:15:14 +02:00
Luis Cossío 2e6358746a fix flaky welford calculation test (#6927)
* use most significant digits for comparison

* use absolute or relative tolerance
2025-08-11 13:15:10 +02:00
Andrey Vasnetsov af7ab5b192 bump version to v1.15.1 (#6934) 2025-07-24 09:34:36 +02:00
Andrey Vasnetsov 82508fd34a Fix PointId Hash (#6932)
* fmt

* clippy
2025-07-24 08:46:40 +02:00
Andrey Vasnetsov 6c9e12f948 UUID index bug (#6916)
* reproduce bug as unit test

* reproduce in even faster unit test

* fix alignment in mmap_hashmap

* clippy

* review fixes
2025-07-24 08:46:26 +02:00
xzfc 916d84a2aa MADV_POPULATE_READ on sequential mmaps (#6923) 2025-07-24 08:46:18 +02:00
dependabot[bot] 2e69ab0c13 build(deps): bump roaring from 0.11.0 to 0.11.1 (#6921) 2025-07-24 08:45:46 +02:00
dependabot[bot] 17efe03f64 build(deps): bump object_store from 0.12.2 to 0.12.3 (#6918)
---
updated-dependencies:
- dependency-name: object_store
  dependency-version: 0.12.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-07-24 08:45:31 +02:00
Tim Visée f53f8cbfc3 Fix checkfs in MUSL builds (#6903) 2025-07-24 08:44:51 +02:00
Tim Visée 137b6c1e4a Bump version to 1.15.0 (#6896)
* Bump version to 1.15.0

* Update missed cherry picks

* Update redoc to 1.15.0
2025-07-18 11:06:11 +02:00
Luis Cossíoandgenerall 5d47e6fd1b [MMR] make lazy matrix non-symmetric (#6900)
* make lazy matrix non-symmetric

* remove SymmetricMatrix

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-07-18 00:49:21 +02:00
Luis Cossío 49154ded32 also handle special case (#6898) 2025-07-18 00:49:13 +02:00
Luis Cossío 0462edcc1b MMR with postprocessed score (#6897)
* handle score_threshold only during search

* use original score for mmr output
2025-07-18 00:49:05 +02:00
Tim Visée 60c216be92 Preprocess query vectors, sort sparse vectors (#6895)
* Add test to assert behavior of unsorted sparse query

* Preprocess sparse vectors

* Fix typo
2025-07-17 16:13:01 +02:00
Tim Visée d6844b7f65 Deprecations for Qdrant 1.15.0 (#6892)
* Deprecate init_from

* Mark memmap_threshold as deprecated

* Mark locks API as deprecated

* Mark RBAC collection access payload filter as deprecated

* Allow deprecations in tests and benchmarks
2025-07-17 16:12:57 +02:00
Jojii e3738b38f6 Replace Box with Either (#6887) 2025-07-17 14:33:15 +02:00
Luis Cossíoandcoderabbitai[bot] efaa12f4f6 [mmr] compute similarities lazily (#6877)
* compute similarities lazily

* Update lib/collection/src/collection/mmr/lazy_matrix.rs

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-07-17 14:02:44 +02:00
Tim Visée d4c6d947da Don't delete payload index RocksDB if we still use RocksDB storage type (#6888) 2025-07-17 14:02:37 +02:00
Andrey Vasnetsov d22debcaf3 lowercase stemmer language for consistency (#6889) 2025-07-17 14:02:14 +02:00
Arnaud Gourlay 0dd3d8fccb Bump thread-priority 2.1.0 (#6885) 2025-07-17 14:02:06 +02:00
Tim Viséeandcoderabbitai[bot] 877adc6748 Fix broken is_empty filter on new points (#6882)
* Add test to catch bug

Bug: <https://github.com/qdrant/qdrant/issues/6880>

* Fix loading null index from wrong path

* Simplify test

* Also bump total point count when removing a point

* Add debug assertion, ensure we don't create null index in selector

* Update bug link

* Remove unused import

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-07-17 14:01:38 +02:00
Kumar Shivenduandtimvisee 105bdb4dc2 Fix bug that causes all replicas to die if node is restarted during resharding (#6800)
* Fix bug that causes all replicas to die if node is restarted during resharding

* fix recursive async problem

* Avoid write lock unless required

* avoid using &mut when & is sufficient

* Remove is_in_progress since check_abort_resharding exists (#6806)

* On resharding abort, only abort transfers related to current operation

* Fix resharding dead replicas improvements (#6881)

* Apply suggestions to fix resharding dead replicas

* fmt

---------

Co-authored-by: timvisee <tim@visee.me>
2025-07-17 14:00:50 +02:00
Tim ViséeandRoman Titov a2188f928f After payload index migration, delete RocksDB files (#6873)
* After payload index migration, delete RocksDB files

* Add comment on why it is safe to not clean up RocksDB files in some cases

* Update lib/segment/src/index/struct_payload_index.rs

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

* Reformat

---------

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
2025-07-17 14:00:42 +02:00
Tim Visée b110600233 Adjust feature flag descriptions (#6875) 2025-07-17 14:00:36 +02:00
Tim Visée 3b7616808c Enable migration from RocksDB to mutable ID tracker by default (#6872)
* Enable migration from RocksDB to mutable ID tracker by default

We planned to enable this by default in the Qdrant 1.15.0 release.

* Synchronize enable at version TODOs for other flags with planning

* Update OpenAPI spec
2025-07-17 14:00:30 +02:00
Arnaud Gourlay f76c6ad3e4 Do not track HW measurement when matching over a text index (#6833) 2025-07-17 14:00:23 +02:00
Arnaud Gourlay b8863aa0e5 Fix hardware counter vector_io for segment vector retrieval (#6863) 2025-07-17 14:00:16 +02:00