upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-09-25 07:27:41 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
edge-docs-diff
qdrant/lib/segment/Cargo.toml
T
Andrey VasnetsovClaude Opus 5qdrant-cloud-bottimvisee
 6cd816acac Add match: { substring } filter condition (#10711) 
* Add `match: { substring }` filter condition

Unindexed `text` and `text_any` matching became token-aware in #10341 and
#10593. Users who relied on the old raw substring behaviour get it back as
an explicit condition: `match: { "substring": "..." }` selects points with a
string value containing the given string, byte-wise and case-sensitive,
consistent with exact keyword and prefix matching.

Execution: a keyword index (with or without the `prefix` option) serves the
condition by scanning its value dictionary and uniting the postings of the
matching keys; cardinality reuses the prefix estimator, generalised into
`keys_union_cardinality`. The per-point checker goes through the forward
index. Without a keyword index the condition falls back to reading the
payload. Text, bool, integer and uuid indexes decline it.

Strict mode: the condition requires the `KeywordMatch` capability, so with
`unindexed_filtering_retrieve: false` it is rejected on unindexed and on
text-indexed fields and allowed on any keyword index.

API: `MatchSubstring` in the REST `Match` union with regenerated OpenAPI,
gRPC `Match.substring = 12`, edge python `MatchSubstring`, edge ffi
`Match::Substring`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Test substring fallback on a text-indexed field, document estimator params

A text index cannot serve `substring`, so on a field that has only a text
index the condition runs through the payload fallback; only strict mode may
reject it. Pin that in the OpenAPI suite and reword the strict-mode unit
test comment, which read as if the text index itself blocked the query.

Also spell out what `keys` and `postings` mean in `keys_union_cardinality`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Serve `match: { substring }` from the keyword key dictionary

The condition used to enumerate keys through `MapIndexRead::for_each_value`,
which on the on-disk variant drags the whole `value_to_points` file through
`for_each_entry`, plus one random read per matching key for its postings
count. Query planning paid that scan in full, before deciding whether to use
the clause at all.

Route it through the `prefix_index.bin` key dictionary instead: front-coded
keys with their postings counts inline, no postings. Estimation now reads
keys only and never touches `value_to_points`; filtering takes the matched
key list and resolves postings in one batched read, as prefix matching
already does.

This makes the `prefix` option a requirement: a keyword index without it has
no key dictionary, so it declines the condition and falls back to the payload
scan, the same as a text index. Strict mode follows — substring now infers
`KeywordPrefix`, so `unindexed_filtering_retrieve: false` names
`keyword (with prefix: true)` as the index to create.

`PrefixIndex::for_each_key` reads blocks in ~1 MiB chunks rather than the
whole key section at once: a substring cannot be pruned by the block index,
so the one-shot read would grow with the dictionary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Sync MatchSubstring OpenAPI description with Rust docs

After routing substring matching through the prefix key dictionary,
the schema docstring required regenerating so OpenAPI stays consistent.

* Scan the map index keys for substring match without a dictionary

Without the keyword dictionary a substring condition was declined by the
field index and left to the per-point condition checker, which reads the
forward index for every candidate point. Enumerate the distinct keys of
`values_to_points` instead: the same one-pass-over-distinct-values shape as
the dictionary scan, only over a structure that interleaves keys with their
postings. Filtering and cardinality estimation are then always served, so
the condition can act as a primary clause on a plain keyword index.

Prefix matching keeps its per-point fallback: an ordered dictionary is what
makes a prefix a bounded range, and enumerating every key to answer one is
not a trade worth making implicitly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Reject substring matching in strict mode

A substring condition is answered by looking at every distinct value of the
field: no index gives it a bounded access path, so there is no index a user
could create to make it affordable. Reject it under strict mode instead,
wherever a filter reaches verification — read and write filters, nested
sub-filters, and prefetch filters.

Filter limits are now checked before the unindexed-field check, so the
rejection is not reported as "create an index for this key", advice that
would lead to the same rejection afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Update the strict mode substring test to the new rejection

The test asserted that substring filtering under strict mode asks for a
keyword index with the `prefix` option. It is now rejected whatever index
the field carries, so every case in the test gets the same answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Estimate a substring condition without scanning

Counting the keys a substring matches costs the same scan as answering the
condition, and `filter` then repeats it to collect those keys. Report the
uninformed estimate instead — the one an unindexed condition has always
reported — and keep the primary clause, so the scan happens once, in
`filter`, and only when the planner picks the condition to drive iteration.

With no counts to collect, `substring_scan` collapses into `substring_keys`:
the in-RAM variants no longer look up a posting count per matched key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Don't to parse everything as UTF-8

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
2026-09-22 12:47:34 +02:00

235 lines
5.2 KiB
TOML
Raw Permalink Blame History

[package]
name = "segment"
version = "0.6.0"
authors = [
"Andrey Vasnetsov <vasnetsov93@gmail.com>",
"Qdrant Team <info@qdrant.tech>",
]
license = "Apache-2.0"
edition = "2024"
[lints]
workspace = true
[features]
default = []
testing = ["common/testing", "sparse/testing", "gpu/testing", "quantization/testing", "dep:exhaustive"]
gpu = ["gpu/gpu"]
[build-dependencies]
cc = { workspace = true }
[dev-dependencies]
clap = { workspace = true }
criterion = { workspace = true }
dataset = { path = "../common/dataset" }
env_logger = { workspace = true }
fs-err = { workspace = true, features = ["debug"] }
humantime = { workspace = true }
indicatif = { workspace = true }
ndarray = "0.17.2"
ndarray-npy = { version = "0.10.0", default-features = false }
rand_distr = { workspace = true }
walkdir = { workspace = true }
rstest = { workspace = true }
segment = { path = ".", default-features = false, features = ["testing"] }
proptest = { workspace = true }
anyhow = { workspace = true }
object_store = { workspace = true }
bytes = { workspace = true }
tokio = { workspace = true }
tap = { workspace = true }
[target.'cfg(not(target_os = "windows"))'.dev-dependencies]
pprof = { workspace = true }
[dependencies]
io_bridge_object_store = { path = "../common/io_bridge_object_store" }
blink-alloc = { workspace = true }
futures = { workspace = true }
bytemuck = { workspace = true }
data-encoding = { workspace = true }
fs-err = { workspace = true }
integer-encoding = { workspace = true }
tempfile = { workspace = true }
parking_lot = { workspace = true }
rayon = { workspace = true }
itertools = { workspace = true }
uuid = { workspace = true }
bincode = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
serde_cbor = { workspace = true }
serde-value = "0.7"
serde_variant = { workspace = true }
serde-untagged = "0.1.9"
ordered-float = { workspace = true }
thiserror = { workspace = true }
atomic_refcell = { workspace = true }
atomicwrites = { workspace = true }
memchr = { workspace = true }
memmap2 = { workspace = true }
schemars = { workspace = true }
log = { workspace = true }
geo = { version = "0.33.1", default-features = false }
geohash = "0.13.2"
num-traits = { workspace = true }
num-derive = "0.5.1"
num-cmp = "0.1.0"
once_cell = "1.21"
rand = { workspace = true }
bitvec = { workspace = true }
fs_extra = { workspace = true }
tinyvec = { workspace = true }
validator = { workspace = true }
chrono = { workspace = true }
ecow = { workspace = true }
fnv = { workspace = true }
indexmap = { workspace = true }
ahash = { workspace = true }
self_cell.workspace = true
sha2 = { workspace = true }
siphasher = "1.0.3"
smallvec = { workspace = true }
strum = { workspace = true }
tokio = { workspace = true }
byteorder = { workspace = true }
stumpalo = { workspace = true }
zerocopy = { workspace = true }
vaporetto = { version = "0.6.5" }
rust-stemmers = { package = "qdrant-rust-stemmers", version = "1.2.2" }
# Memory only; disable `user` so macOS does not pull objc2 via open-directory
# (objc2::Retained's IntoIterator blanket impl overflows trait resolution).
sysinfo = { workspace = true }
charabia = { version = "0.10.0", default-features = false, features = [
"greek",
"hebrew",
"thai",
"chinese-segmentation",
"chinese-normalization",
] }
blobstore = { path = "../blobstore" }
common = { path = "../common/common" }
macros = { path = "../macros" }
posting_list = { path = "../posting_list" }
quantization = { path = "../quantization" }
sparse = { path = "../sparse" }
gpu = { path = "../gpu" }
tracing = { workspace = true, optional = true }
macro_rules_attribute = "0.2.2"
nom = "8.0.0"
half = { workspace = true }
roaring = { workspace = true }
duplicate = { workspace = true }
exhaustive = { workspace = true, optional = true }
[target.'cfg(target_os = "linux")'.dependencies]
procfs = { version = "0.18", default-features = false }
[[bench]]
name = "vector_search"
harness = false
[[bench]]
name = "turbo_vector_search"
harness = false
[[bench]]
name = "turbo_full_scan"
harness = false
[[bench]]
name = "read_vectors"
harness = false
[[bench]]
name = "read_payloads"
harness = false
[[bench]]
name = "hnsw_build_graph"
harness = false
[[bench]]
name = "hnsw_search_graph"
harness = false
[[bench]]
name = "conditional_search"
harness = false
[[bench]]
name = "dynamic_mmap_flags"
harness = false
[[bench]]
name = "hnsw_build_asymptotic"
harness = false
[[bench]]
name = "serde_formats"
harness = false
[[bench]]
name = "id_type_benchmark"
harness = false
[[bench]]
name = "map_benchmark"
harness = false
[[bench]]
name = "boolean_filtering"
harness = false
[[bench]]
name = "numeric_index_check_values"
harness = false
[[bench]]
name = "sparse_index_search"
harness = false
[[bench]]
name = "sparse_index_build"
harness = false
[[bench]]
name = "sparse_vector_storage"
harness = false
[[bench]]
name = "multi_vector_search"
harness = false
[[bench]]
name = "metrics"
harness = false
[[bench]]
name = "range_filtering"
harness = false
[[bench]]
name = "in_memory_id_tracker"
harness = false
[[bench]]
name = "buffered_update_bitslice"
harness = false
[[bench]]
name = "segment_info"
harness = false
[[bench]]
name = "hnsw_incremental_build"
harness = false
[[bench]]
name = "facets"
harness = false
Reference in New Issue View Git Blame Copy Permalink