upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-07 02:20:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
e99d433fbab1beb91ccc19156afdb4dd18e5c562
qdrant/lib/api/build.rs
Andrey Vasnetsov 8c8a72d120 Remove read_multi_iter to fix macOS linker symbol overflow (#9643) 
* remove unused iter_offsets

* Replace MultivectorOffsetsStorage::iter_offsets with callback-based for_each_offset

First step of removing the iterator-returning read API (whose deep,
composable generic types blow up mangled symbol size). Convert the
offsets read from an iterator to a callback the caller pushes into:

- trait method iter_offsets -> for_each_offset(ids, FnMut(usize, MultivectorOffset))
  returning common::universal_io::Result<()>
- Mmap impl now uses the callback read_batch (drops one read_iter use)
- Ram / Chunked impls push into the callback; Chunked still goes through
  iter_vectors for now (converted in a later step)
- the single caller (for_each_in_multi_batch) passes a closure

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Implement read_batch directly on ReadPipeline, not via read_iter

read_batch now drives the pipeline itself (refill-then-wait loop, like
read_multi_iter) and invokes the callback per result, instead of
consuming the iterator returned by read_iter. A step toward removing the
iterator-returning read API.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Remove the ReadMulti RPC from the StorageRead gRPC service

ReadMulti was the only real consumer of UniversalRead::read_multi (which
itself relies on read_multi_iter). Removing the RPC end-to-end clears the
path to dropping that read API. StorageReadService keeps all its other
RPCs (ListFiles, FileExists, FileLength, ReadBytes, ReadBytesStream,
ReadWhole, ReadBatch).

- proto: drop `rpc ReadMulti` + ReadMulti{Entry,Request,Response}
- regenerated lib/api + uio-client generated code; drop ReadMulti
  validation rules in lib/api/build.rs
- tonic: delete the read_multi handler + its 2 tests
- uio-client: delete Client::read_multi, the mock-server impl, and 2 tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Remove UniversalRead::read_multi

Its only real consumer was the StorageRead ReadMulti gRPC handler (removed
in the previous commit); the two wrapper forwarders had no callers. Drop
the trait method and both forwarders (typed/read_only), and remove the
io_uring test that only existed to compare read_multi vs read_multi_iter
(read_multi_iter stays covered by the other tests). Another step toward
removing the iterator-returning read API.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Drive ReadPipeline directly in gridstore read_from_pages

Replace the read_multi_iter call in Pages::read_from_pages with a direct
pipeline loop (refill-then-drain), scheduling each multi-page read on its
own page file. Behavior unchanged; another step toward removing the
iterator-returning read_multi_iter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Drive ReadPipeline directly in gridstore read_batch_from_pages

Replace the second read_multi_iter call (in Pages::read_batch_from_pages)
with a direct pipeline loop, scheduling each (ReadMeta, page, range) on its
own page file and propagating errors via GridstoreError. Single/multi-page
buffering and out-of-order reassembly are unchanged. No more read_multi_iter
in gridstore.

Measured overhead on warm mmap (both paths are zero-copy borrows): ~0.3 ns
per read of fixed control cost, flat across read sizes — well under 0.1% of
a real payload read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Drive ReadPipeline directly in on-disk postings with_posting_views

Replace read_iter in OnDiskPostings::with_posting_views with a direct
pipeline loop. wait_bytemuck yields a file-borrowed Cow, so postings are
still stored zero-copy in raw_postings (a read_batch swap would have forced
an owned copy of every posting list per query on the mmap backend).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Read on-disk posting headers via read_batch, drop the HeadersBatch iterator

headers_iter now reads headers with the callback read_batch API: each header
is parsed (copied) out of the read bytes, so nothing borrows the file past the
read — no pipeline needed. Since the read is now eager, HeadersBatch holds the
collected Vec<HeaderResult> directly instead of a Box<dyn Iterator>, dropping
the boxing, the dynamic dispatch, and the struct's lifetime parameter.
with_posting_views takes the Vec and still pipelines the posting reads.

Removes the last read_iter use in this file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Use read_batch in simple_disk_cache populate_from

populate_from reads one byte per block only to fault blocks into the local
cache, discarding the bytes — a no-op-callback read_batch fits exactly. Drives
the same DiskCachePipeline as before; one less read_iter caller.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Implement read_iter directly on ReadPipeline, not via read_multi_iter

read_iter now drives the pipeline itself (refill-then-wait loop, mirroring
read_bytes_iter) instead of mapping its ranges onto self and calling
read_multi_iter. Same signature and iterator contract, so all callers are
unchanged. Leaves iter_vectors as read_multi_iter's only remaining caller.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Revert "Drive ReadPipeline directly in on-disk postings with_posting_views"

This reverts commit c02c41f17b.

* Add callback-based for_each_vector next to iter_vectors

for_each_vector drives the ReadPipeline directly across chunk files and
invokes a fallible callback per flattened multi-vector, returning
OperationResult, instead of returning an iterator built on read_multi_iter.
Callers will migrate onto it so iter_vectors (read_multi_iter's last caller)
can be removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Make dense for_each_in_batch / for_each_in_dense_batch fallible

Thread OperationResult up the dense batch-read path so io_uring read
errors propagate instead of being .expect()ed deep inside the storage.
The for_each_in_dense_batch scorer path (custom/metric query scorers)
now carries the Result to the infallible score() boundary where it is
.expect()ed; read_vectors keeps its () signature and .expect()s locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Convert dense read_vectors to for_each_vector

The read-only and appendable dense storage read_vectors impls drove
ChunkedVectors::iter_vectors directly; switch them to the callback-based
for_each_vector and .expect() the result at the (infallible) read_vectors
boundary. Removes the last dense-path iter_vectors callers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Convert quantized for_each_offset to for_each_vector + OperationResult

The two chunked MultivectorOffsetsStorage impls drove iter_vectors to read
the offset table; switch them to the callback-based for_each_vector.
for_each_vector returns OperationResult, so upgrade the for_each_offset
trait (and all four impls) from universal_io::Result to OperationResult
(the universal_io -> Operation direction, via ?). No error downgrade.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Convert EncodedStorage/EncodedVectors iter_batch to callback for_each_batch

The two chunked-mmap EncodedStorage impls drove ChunkedVectors::iter_vectors
to back iter_batch. Replace the iterator-returning iter_batch on both the
EncodedStorage and EncodedVectors traits (quantization crate) with a callback
for_each_batch(FnMut(usize, &[u8])), and switch the chunked impls to
for_each_vector. The callback is infallible: the chunked impls .expect() the
read internally, matching iter_vectors' prior panic-on-read-error behavior,
so no OperationError is downgraded. Scorers and the multivector readers adopt
the callback; the accumulating multivector path owns (to_vec) only when it
must buffer across components.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Convert multivector read paths to for_each_vector

The read_only multivector free fn chained two iter_vectors (offsets feeding
vectors) - the recursive iterator nesting behind the worst symbol bloat.
Replace it with a callback for_each_vector that resolves the per-point
offsets into a Vec first, then drives ChunkedVectors::for_each_vector over
the flattened vectors. Migrate both multivector storages' read_vectors and
for_each_in_batch_multi accordingly, .expect()ing at their infallible
boundaries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Remove read_multi_iter and ChunkedVectors::iter_vectors

With every caller migrated to callback-based for_each_vector/for_each_batch,
delete the last iterator-returning multi-read APIs: ChunkedVectors::iter_vectors
(segment) and the read_multi_iter trait method plus its mmap override, the
TypedStorage/ReadOnly wrapper forwarders, and the two io_uring unit tests.

These deeply-nested monomorphized iterator types (read_multi_iter feeding
read_multi_iter) produced >1 MiB mangled drop_in_place symbols that overflowed
the macOS ld symbol-name limit; the callback rewrite eliminates them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Pass owned Cow through for_each_vector/for_each_batch to avoid a copy

The callback-based readers handed the callback a borrowed `&[u8]`/`&[T]`,
forcing the quantized multivector batch read to `to_vec()` each sub-vector
into its per-point buffer. But io_uring-like backends already return a freshly
owned buffer per read (`ACow::Owned`), so that was a redundant second copy.

Change `ChunkedVectorsRead::for_each_vector` and the `EncodedStorage` /
`EncodedVectors` `for_each_batch` callbacks to receive `Cow<[..]>` by value.
The buffering path now `into_owned()`s it — a move when the backend returned
owned (the case this path targets), a copy only for a borrowed Cow (mmap),
which never reaches this path. Immediate-use callers (scorers,
score_point_max_similarity, the dense/multivector readers) just deref the Cow;
the dense readers drop their now-redundant `Cow::Borrowed` wraps.

Also clarifies the multivector reader: `SubVectorOwner`/`owners`/
`sub_vector_offsets` naming, docs, and a corrected comment noting the per-point
buffer is what makes regrouping order-independent under out-of-order completion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Drop the removed ReadMulti JWT access test; fix clippy unwrap_or_default

The ReadMulti StorageRead RPC was removed earlier in this branch, so the
consensus JWT-access test (and its registry entry) for it must go too. Also
switch the multivector buffer's `or_insert_with(SmallVec::new)` to
`or_default()` per clippy::unwrap_or_default.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Add MmapFile::read_batch

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
2026-07-01 14:03:32 +02:00

496 lines
30 KiB
Rust
Raw Blame History

use std::path::PathBuf;
use std::process::Command;
use std::{env, fs, str};
use common::defaults;
use tonic_prost_build::Builder;
fn main() -> std::io::Result<()> {
// Ensure Qdrant version is configured correctly
assert_eq!(
defaults::QDRANT_VERSION.to_string(),
env!("CARGO_PKG_VERSION"),
"crate version does not match with defaults.rs",
);
// Since `tonic` 0.14 (when `tonic-build` was refactored into `tonic-prost-build`),
// `tonic_prost_build` *does not* emit `cargo:rerun-if-changed=` directives, which forces Cargo
// to recompile `api` (and any other crate that *uses* `api`; which is most crates in Qdrant)
// on every `cargo check`/`cargo build`/`cargo run`.
//
// As a workaround, we emit `cargo:rerun-if-changed=` explicitly. 🤷‍♀️
//
// See:
// - https://github.com/grpc/grpc-rust/issues/2415
// - https://github.com/grpc/grpc-rust/issues/2511
#[expect(
clippy::disallowed_methods,
reason = "std::fs is allowed in build-script"
)]
for result in fs::read_dir("src/grpc/proto").unwrap() {
let proto = result.unwrap();
if proto.path().extension().and_then(|str| str.to_str()) == Some("proto") {
println!("cargo:rerun-if-changed={}", proto.path().display());
}
}
let build_out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());
// Build gRPC bits from proto file
tonic_prost_build::configure()
// Because we want to attach all validation rules to the generated gRPC types, we must do
// so by extending the builder. This is ugly, but better than manually implementing
// `Validation` for all these types and seems to be the best approach. The line below
// configures all attributes.
.configure_validation()
.file_descriptor_set_path(build_out_dir.join("qdrant_descriptor.bin"))
.out_dir("src/grpc/") // saves generated structures at this location
.compile_protos(
&["src/grpc/proto/qdrant.proto"], // proto entry point
&["src/grpc/proto"], // specify the root location to search proto dependencies
)?;
// Append trait extension imports to generated gRPC output
append_to_file(
"src/grpc/qdrant.rs",
"use super::validate::ValidateExt;\nuse validator::Validate;",
);
// Fetch git commit ID and pass it to the compiler
let git_commit_id = option_env!("GIT_COMMIT_ID").map(String::from).or_else(|| {
match Command::new("git").args(["rev-parse", "HEAD"]).output() {
Ok(output) if output.status.success() => {
Some(str::from_utf8(&output.stdout).unwrap().trim().to_string())
}
_ => {
println!("cargo:warning=current git commit hash could not be determined");
None
}
}
});
if let Some(commit_id) = git_commit_id {
println!("cargo:rustc-env=GIT_COMMIT_ID={commit_id}");
}
Ok(())
}
/// Extension to [`Builder`] to configure validation attributes.
trait BuilderExt {
fn configure_validation(self) -> Self;
fn validates(self, fields: &[(&str, &str)], extra_derives: &[&str]) -> Self;
fn derive_validate(self, path: &str) -> Self;
fn derive_validates(self, paths: &[&str]) -> Self;
fn field_validate(self, path: &str, constraint: &str) -> Self;
fn field_validates(self, paths: &[(&str, &str)]) -> Self;
}
impl BuilderExt for Builder {
fn configure_validation(self) -> Self {
configure_validation(self)
}
fn validates(self, fields: &[(&str, &str)], extra_derives: &[&str]) -> Self {
// Build list of structs to derive validation on, guess these from list of fields
let mut derives = fields
.iter()
.map(|(field, _)| field.split_once('.').unwrap().0)
.collect::<Vec<&str>>();
derives.extend(extra_derives);
derives.sort_unstable();
derives.dedup();
self.derive_validates(&derives).field_validates(fields)
}
fn derive_validate(self, path: &str) -> Self {
self.type_attribute(path, "#[derive(validator::Validate)]")
}
fn derive_validates(self, paths: &[&str]) -> Self {
paths.iter().fold(self, |c, path| c.derive_validate(path))
}
fn field_validate(self, path: &str, constraint: &str) -> Self {
if constraint.is_empty() {
self.field_attribute(path, "#[validate(nested)]")
} else {
self.field_attribute(path, format!("#[validate({constraint})]"))
}
}
fn field_validates(self, fields: &[(&str, &str)]) -> Self {
fields.iter().fold(self, |c, (path, constraint)| {
c.field_validate(path, constraint)
})
}
}
/// Configure additional attributes required for validation on generated gRPC types.
///
/// These are grouped by service file.
#[rustfmt::skip]
fn configure_validation(builder: Builder) -> Builder {
builder
// prost_wkt_types needed for serde support
.extern_path(".google.protobuf.Timestamp", "::prost_wkt_types::Timestamp")
// Service: collections.proto
.validates(&[
("GetCollectionInfoRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("CollectionExistsRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("CreateCollection.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name\")"),
("CreateCollection.hnsw_config", ""),
("CreateCollection.wal_config", ""),
("CreateCollection.optimizers_config", ""),
("CreateCollection.vectors_config", ""),
("CreateCollection.quantization_config", ""),
("CreateCollection.shard_number", "range(min = 1)"),
("CreateCollection.replication_factor", "range(min = 1)"),
("CreateCollection.write_consistency_factor", "range(min = 1)"),
("CreateCollection.strict_mode_config", ""),
("UpdateCollection.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("UpdateCollection.optimizers_config", ""),
("UpdateCollection.params", ""),
("UpdateCollection.timeout", "range(min = 1)"),
("UpdateCollection.hnsw_config", ""),
("UpdateCollection.vectors_config", ""),
("UpdateCollection.quantization_config", ""),
("UpdateCollection.strict_mode_config", ""),
("CollectionParamsDiff.replication_factor", "range(min = 1)"),
("CollectionParamsDiff.write_consistency_factor", "range(min = 1)"),
("DeleteCollection.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DeleteCollection.timeout", "range(min = 1)"),
("CollectionParams.vectors_config", ""),
("ChangeAliases.timeout", "range(min = 1)"),
("ListCollectionAliasesRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("HnswConfigDiff.ef_construct", "range(min = 4)"),
("WalConfigDiff.wal_capacity_mb", "range(min = 1)"),
("WalConfigDiff.wal_retain_closed", "range(min = 1)"),
("OptimizersConfigDiff.deleted_threshold", "range(min = 0.0, max = 1.0)"),
("OptimizersConfigDiff.vacuum_min_vector_number", "range(min = 100)"),
("OptimizersConfigDiff.max_segment_size", "range(min = 1)"),
("VectorsConfig.config", ""),
("VectorsConfigDiff.config", ""),
("VectorParams.size", "range(min = 1, max = 65536)"),
("VectorParams.hnsw_config", ""),
("VectorParams.quantization_config", ""),
("VectorParamsMap.map", ""),
("VectorParamsDiff.hnsw_config", ""),
("VectorParamsDiff.quantization_config", ""),
("VectorParamsDiffMap.map", ""),
("QuantizationConfig.quantization", ""),
("QuantizationConfigDiff.quantization", ""),
("ScalarQuantization.quantile", "range(min = 0.5, max = 1.0)"),
("SparseIndexConfig.datatype", "custom(function = \"crate::grpc::validate::validate_sparse_datatype\")"),
("UpdateCollectionClusterSetupRequest.timeout", "range(min = 1)"),
("UpdateCollectionClusterSetupRequest.operation", ""),
("StrictModeConfig.max_query_limit", "range(min = 1)"),
("StrictModeConfig.max_timeout", "range(min = 1)"),
("StrictModeConfig.max_points_count", "range(min = 1)"),
("StrictModeConfig.read_rate_limit", "range(min = 1)"),
("StrictModeConfig.write_rate_limit", "range(min = 1)"),
("StrictModeConfig.max_resident_memory_percent", "range(min = 1, max = 100)"),
("StrictModeConfig.max_disk_usage_percent", "range(min = 1, max = 100)"),
("StrictModeConfig.multivector_config", ""),
("StrictModeConfig.sparse_config", ""),
("StrictModeSparseConfig.sparse_config", ""),
("StrictModeSparse.max_length", "range(min = 1)"),
("StrictModeMultivectorConfig.multivector_config", ""),
("StrictModeMultivector.max_vectors", "range(min = 1)"),
], &[
"ListCollectionsRequest",
"ListAliasesRequest",
"CollectionClusterInfoRequest",
"UpdateCollectionClusterSetupRequest",
"ProductQuantization",
"BinaryQuantization",
"TurboQuantization",
"Disabled",
"QuantizationConfigDiff",
"quantization_config_diff::Quantization",
"Replica",
"ListShardKeysRequest",
])
// Service: collections_internal.proto
.validates(&[
("GetCollectionInfoRequestInternal.get_collection_info_request", ""),
("InitiateShardTransferRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("WaitForShardStateRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("WaitForShardStateRequest.timeout", "range(min = 1)"),
("GetShardRecoveryPointRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("UpdateShardCutoffPointRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("GetShardOptimizationsRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("GetShardMemoryReportRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
], &[])
// Service: points.proto
.validates(&[
("AcornSearchParams.max_selectivity", "range(min = 0.0, max = 1.0)"),
("PointsSelector.points_selector_one_of", ""),
("UpsertPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("UpsertPoints.points", ""),
("UpsertPoints.update_filter", ""),
("DeletePoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("UpdatePointVectors.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("UpdatePointVectors.points", ""),
("UpdatePointVectors.update_filter", ""),
("DeletePointVectors.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DeletePointVectors.vector_names", "length(min = 1, message = \"must specify vector names to delete\")"),
("DeletePointVectors.points_selector", ""),
("PointVectors.vectors", ""),
("GetPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("SetPayloadPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("SetPayloadPoints.points_selector", ""),
("DeletePayloadPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DeletePayloadPoints.points_selector", ""),
("ClearPayloadPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ClearPayloadPoints.points", ""),
("UpdateBatchPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("UpdateBatchPoints.operations", "length(min = 1)"),
("CreateFieldIndexCollection.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("CreateFieldIndexCollection.field_name", "length(min = 1)"),
("CreateFieldIndexCollection.field_index_params", ""),
("PayloadIndexParams.index_params", ""),
("DeleteFieldIndexCollection.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DeleteFieldIndexCollection.field_name", "length(min = 1)"),
("CreateVectorNameRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("CreateVectorNameRequest.vector_config", ""),
("DenseVectorCreationConfig.size", "range(min = 1, max = 65536)"),
("SparseVectorCreationConfig.datatype", "custom(function = \"crate::grpc::validate::validate_sparse_datatype\")"),
("DeleteVectorNameRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("SearchPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("SearchPoints.filter", ""),
("SearchPoints.limit", "range(min = 1)"),
("SearchPoints.params", ""),
("SearchPoints.timeout", "range(min = 1)"),
("SearchBatchPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("SearchBatchPoints.search_points", ""),
("SearchBatchPoints.timeout", "range(min = 1)"),
("SearchPointGroups.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("SearchPointGroups.group_by", "length(min = 1)"),
("SearchPointGroups.filter", ""),
("SearchPointGroups.params", ""),
("SearchPointGroups.group_size", "range(min = 1)"),
("SearchPointGroups.limit", "range(min = 1)"),
("SearchPointGroups.timeout", "range(min = 1)"),
("SearchParams.hnsw_ef", "range(min = 1)"),
("SearchParams.quantization", ""),
("SearchParams.acorn", ""),
("QuantizationSearchParams.oversampling", "range(min = 1.0)"),
("ScrollPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ScrollPoints.filter", ""),
("ScrollPoints.limit", "range(min = 1)"),
("RecommendPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("RecommendPoints.filter", ""),
("RecommendPoints.params", ""),
("RecommendPoints.timeout", "range(min = 1)"),
("RecommendPoints.positive_vectors", ""),
("RecommendPoints.negative_vectors", ""),
("RecommendBatchPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("RecommendBatchPoints.recommend_points", ""),
("RecommendBatchPoints.timeout", "range(min = 1)"),
("RecommendPointGroups.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("RecommendPointGroups.filter", ""),
("RecommendPointGroups.group_by", "length(min = 1)"),
("RecommendPointGroups.group_size", "range(min = 1)"),
("RecommendPointGroups.limit", "range(min = 1)"),
("RecommendPointGroups.params", ""),
("RecommendPointGroups.timeout", "range(min = 1)"),
("RecommendPointGroups.positive_vectors", ""),
("RecommendPointGroups.negative_vectors", ""),
("DiscoverPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DiscoverPoints.filter", ""),
("DiscoverPoints.params", ""),
("DiscoverPoints.limit", "range(min = 1)"),
("DiscoverPoints.timeout", "range(min = 1)"),
("DiscoverBatchPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DiscoverBatchPoints.discover_points", ""),
("DiscoverBatchPoints.timeout", "range(min = 1)"),
("CountPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("CountPoints.filter", ""),
("GeoBoundingBox.bottom_right", ""),
("GeoBoundingBox.top_left", ""),
("GeoLineString.points", ""),
("GeoPolygon.exterior", "custom(function = \"crate::grpc::validate::validate_geo_polygon_exterior\"), nested"),
("GeoPolygon.interiors", "custom(function = \"crate::grpc::validate::validate_geo_polygon_interiors\"), nested"),
("GeoRadius.center", ""),
("Filter.should", ""),
("Filter.must", ""),
("Filter.must_not", ""),
("NestedCondition.filter", ""),
("Condition.condition_one_of", ""),
("Filter.min_should", ""),
("MinShould.conditions", ""),
("MinShould.min_count", "range(min = 1, message = \"min_count must be greater than 0\")"),
("PointStruct.vectors", ""),
("Vectors.vectors_options", ""),
("NamedVectors.vectors", ""),
("DatetimeRange.lt", "custom(function = \"crate::grpc::validate::validate_timestamp\")"),
("DatetimeRange.gt", "custom(function = \"crate::grpc::validate::validate_timestamp\")"),
("DatetimeRange.lte", "custom(function = \"crate::grpc::validate::validate_timestamp\")"),
("DatetimeRange.gte", "custom(function = \"crate::grpc::validate::validate_timestamp\")"),
("VectorInput.variant", ""),
("RecommendInput.positive", ""),
("RecommendInput.negative", ""),
("DiscoverInput.target", ""),
("DiscoverInput.context", ""),
("ContextInputPair.positive", ""),
("ContextInputPair.negative", ""),
("ContextInput.pairs", ""),
("RelevanceFeedbackInput.target", ""),
("RelevanceFeedbackInput.feedback", ""),
("RelevanceFeedbackInput.strategy", ""),
("FeedbackStrategy.variant", ""),
("FeedbackItem.example", ""),
("NaiveFeedbackStrategy.b", "range(min = 0.0)"),
("Formula.expression", ""),
("Expression.variant", ""),
("MultExpression.mult", ""),
("SumExpression.sum", ""),
("DivExpression.left", ""),
("DivExpression.right", ""),
("PowExpression.base", ""),
("PowExpression.exponent", ""),
("DecayParamsExpression.x", ""),
("DecayParamsExpression.target", ""),
("NearestInputWithMmr.nearest", ""),
("NearestInputWithMmr.mmr", ""),
("Mmr.diversity", "range(min = 0.0, max = 1.0)"),
("Mmr.candidates_limit", "range(max = 16_384)"),
("Rrf.k", "range(min = 1)"),
("Query.variant", ""),
("PrefetchQuery.prefetch", ""),
("PrefetchQuery.query", ""),
("PrefetchQuery.filter", ""),
("PrefetchQuery.params", ""),
("PrefetchQuery.limit", "range(min = 1)"),
("QueryPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("QueryPoints.limit", "range(min = 1)"),
("QueryPoints.prefetch", ""),
("QueryPoints.query", ""),
("QueryPoints.filter", ""),
("QueryPoints.params", ""),
("QueryPoints.timeout", "range(min = 1)"),
("QueryBatchPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("QueryBatchPoints.query_points", ""),
("QueryBatchPoints.timeout", "range(min = 1)"),
("QueryPointGroups.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("QueryPointGroups.prefetch", ""),
("QueryPointGroups.query", ""),
("QueryPointGroups.group_by", "length(min = 1)"),
("QueryPointGroups.filter", ""),
("QueryPointGroups.params", ""),
("QueryPointGroups.group_size", "range(min = 1)"),
("QueryPointGroups.limit", "range(min = 1)"),
("QueryPointGroups.timeout", "range(min = 1)"),
("FacetCounts.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("FacetCounts.key", "length(min = 1)"),
("FacetCounts.filter", ""),
("FacetCounts.timeout", "range(min = 1)"),
("SearchMatrixPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("SearchMatrixPoints.filter", ""),
("SearchMatrixPoints.sample", "range(min = 2)"),
("SearchMatrixPoints.limit", "range(min = 1)"),
("SearchMatrixPoints.timeout", "range(min = 1)")
], &[
"SparseVectorCreationConfig",
])
.type_attribute(".", "#[derive(serde::Serialize)]")
// Service: points_internal_service.proto
.validates(&[
("UpsertPointsInternal.upsert_points", ""),
("DeletePointsInternal.delete_points", ""),
("UpdateVectorsInternal.update_vectors", ""),
("DeleteVectorsInternal.delete_vectors", ""),
("SetPayloadPointsInternal.set_payload_points", ""),
("DeletePayloadPointsInternal.delete_payload_points", ""),
("ClearPayloadPointsInternal.clear_payload_points", ""),
("CreateFieldIndexCollectionInternal.create_field_index_collection", ""),
("DeleteFieldIndexCollectionInternal.delete_field_index_collection", ""),
("CreateVectorNameInternal.create_vector_name", ""),
("DeleteVectorNameInternal.delete_vector_name", ""),
("UpdateOperation.update", ""),
("UpdateBatchInternal.operations", ""),
("SearchPointsInternal.search_points", ""),
("SearchBatchPointsInternal.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("SearchBatchPointsInternal.search_points", ""),
("CoreSearchPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("CoreSearchPoints.filter", ""),
("CoreSearchPoints.limit", "range(min = 1)"),
("CoreSearchPoints.params", ""),
("CoreSearchBatchPointsInternal.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("CoreSearchBatchPointsInternal.search_points", ""),
("RecoQuery.positives", ""),
("RecoQuery.negatives", ""),
("ContextPair.positive", ""),
("ContextPair.negative", ""),
("DiscoveryQuery.target", ""),
("DiscoveryQuery.context", ""),
("ContextQuery.context", ""),
("RecommendPointsInternal.recommend_points", ""),
("ScrollPointsInternal.scroll_points", ""),
("GetPointsInternal.get_points", ""),
("CountPointsInternal.count_points", ""),
("SyncPointsInternal.sync_points", ""),
("SyncPoints.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("QueryBatchPointsInternal.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("QueryBatchPointsInternal.timeout", "range(min = 1)"),
("FacetCountsInternal.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("FacetCountsInternal.timeout", "range(min = 1)"),
], &[])
// Service: raft_service.proto
.validates(&[
("AddPeerToKnownMessage.uri", "custom(function = \"common::validation::validate_not_empty\")"),
("AddPeerToKnownMessage.port", "range(min = 1)"),
], &[])
// Service: snapshot_service.proto
.validates(&[
("CreateSnapshotRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ListSnapshotsRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DeleteSnapshotRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DeleteSnapshotRequest.snapshot_name", "length(min = 1)"),
("DeleteFullSnapshotRequest.snapshot_name", "length(min = 1)"),
("CreateShardSnapshotRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ListShardSnapshotsRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DeleteShardSnapshotRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("DeleteShardSnapshotRequest.snapshot_name", "length(min = 1)"),
("RecoverShardSnapshotRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("RecoverShardSnapshotRequest.snapshot_name", "length(min = 1)"),
("RecoverShardSnapshotRequest.checksum", "custom(function = \"common::validation::validate_sha256_hash\")"),
("SnapshotDescription.creation_time", "custom(function = \"crate::grpc::validate::validate_timestamp\")"),
], &[
"CreateFullSnapshotRequest",
"ListFullSnapshotsRequest",
])
// Service: storage_read_service.proto
.validates(&[
("FileExistsRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("FileExistsRequest.path", "length(min = 1)"),
("ListFilesRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ListFilesRequest.prefix_path", "length(min = 1)"),
("FileLengthRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("FileLengthRequest.path", "length(min = 1)"),
("ReadBytesRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ReadBytesRequest.path", "length(min = 1)"),
("ReadBytesStreamRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ReadBytesStreamRequest.path", "length(min = 1)"),
("ReadWholeRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ReadWholeRequest.path", "length(min = 1)"),
("ReadBatchRequest.collection_name", "length(min = 1, max = 255), custom(function = \"common::validation::validate_collection_name_legacy\")"),
("ReadBatchRequest.path", "length(min = 1)"),
("ReadBatchRequest.ranges", "length(min = 1)"),
], &[])
}
fn append_to_file(path: &str, line: &str) {
use std::io::prelude::*;
#[expect(clippy::disallowed_types, reason = "build script, ok to use std::fs")]
writeln!(
std::fs::OpenOptions::new().append(true).open(path).unwrap(),
"{line}",
)
.unwrap()
}
Reference in New Issue View Git Blame Copy Permalink