mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-28 08:58:05 -05:00
Andrey VasnetsovandClaude Sonnet 5
ca20151659
* Add edge-tool: CLI for creating, seeding, optimizing, and uploading local edge collections Mirrors the style of lib/edge/tools/shard_update and shard_query: `create` builds a minimal EdgeShard on disk (dense/sparse vectors, quantization presets including turbo4, payload indexes, target segment count), `upsert` seeds it with random points matching its live schema, `optimize` runs the shard optimizers, and `upload` pushes the resulting directory to S3/GCS. Useful for quickly spinning up test collections without a running Qdrant server, then promoting them to object storage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: initialize feature flags, enable serverless_compatible, fix --sparse ambiguity Initialize the global feature-flag OnceLock at startup (with serverless_compatible set, cascading write_segment_manifest/append_only_mutations/compact_bitmask/ append_only_storages) so runs no longer spam "Feature flags not initialized!" and collections are created in the serverless-compatible format. Also splits --sparse into a plain boolean flag plus a repeatable --sparse-name: clap's optional-value parsing for the old `--sparse [NAME]` form silently swallowed a following positional PATH as the sparse vector's name whenever --sparse was the last flag before it (e.g. `create --dense 1024 --sparse ./col`). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: fix --sparse=NAME to require_equals instead of a separate flag The --sparse/--sparse-name split from the previous commit lost the ability to name a sparse vector with --sparse itself. Restore a single --sparse[=NAME] flag, but with require_equals(true): clap then only binds a value via --sparse=NAME, never via a following bare token, so it stays safe next to the trailing PATH positional in every position (bare --sparse, --sparse=NAME, or multiple --sparse=NAME occurrences) without reintroducing the ambiguity that made --sparse swallow PATH as its value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: remove --segments from create, it has no effect there EdgeOptimizersConfig::default_segment_number only feeds MergeOptimizer as a merge-down ceiling (reduce segment count when it exceeds the target); unlike the main collection's LocalShard::build_local, EdgeShard::new never loops to pre-create N appendable segments. A freshly created collection always starts at exactly 1 segment, so passing --segments to `create` was silently a no-op. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: add --indexing-threshold-kb to create Unlike --segments (removed previously), the indexing threshold is a parameter IndexingOptimizer actually consults on every optimize() run: segments larger than it get an HNSW index built. Verified end-to-end (create with a 1KB threshold, upsert 2000 points, optimize) that it produces an hnsw-indexed segment where it would otherwise stay plain. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: add --clean to upload, wiping the destination prefix first Lists every object under DESTINATION and deletes it via ObjectStore::delete_stream before uploading, so re-uploading a collection recreated with a different shape (different segment UUIDs) doesn't leave the old segment's files behind. Verified against the local S3 proxy: uploaded one collection, then a second, differently-shaped one to the same prefix without --clean (29 objects, stale leftovers from the first); re-uploading the second with --clean correctly dropped it back to exactly its own 19 files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
87 lines
2.7 KiB
Rust
87 lines
2.7 KiB
Rust
//! `--quantization` presets, mapped onto [`QuantizationConfig`].
|
|
#![allow(
|
|
deprecated,
|
|
reason = "always_ram is deprecated but still constructible"
|
|
)]
|
|
|
|
use clap::ValueEnum;
|
|
use edge::{
|
|
BinaryQuantizationConfig, CompressionRatio, ProductQuantizationConfig, QuantizationConfig,
|
|
ScalarQuantizationConfig, ScalarType,
|
|
};
|
|
use segment::types::{TurboQuantBitSize, TurboQuantQuantizationConfig, TurboQuantization};
|
|
|
|
#[derive(Copy, Clone, Debug, ValueEnum)]
|
|
pub enum QuantizationPreset {
|
|
#[value(name = "scalar")]
|
|
Scalar,
|
|
#[value(name = "binary")]
|
|
Binary,
|
|
#[value(name = "product-x4")]
|
|
ProductX4,
|
|
#[value(name = "product-x8")]
|
|
ProductX8,
|
|
#[value(name = "product-x16")]
|
|
ProductX16,
|
|
#[value(name = "product-x32")]
|
|
ProductX32,
|
|
#[value(name = "product-x64")]
|
|
ProductX64,
|
|
#[value(name = "turbo1")]
|
|
Turbo1,
|
|
#[value(name = "turbo1.5")]
|
|
Turbo1_5,
|
|
#[value(name = "turbo2")]
|
|
Turbo2,
|
|
#[value(name = "turbo4")]
|
|
Turbo4,
|
|
}
|
|
|
|
impl QuantizationPreset {
|
|
pub fn to_config(self) -> QuantizationConfig {
|
|
let product = |compression: CompressionRatio| -> QuantizationConfig {
|
|
ProductQuantizationConfig {
|
|
compression,
|
|
always_ram: None,
|
|
memory: None,
|
|
}
|
|
.into()
|
|
};
|
|
let turbo = |bits: TurboQuantBitSize| -> QuantizationConfig {
|
|
QuantizationConfig::Turbo(TurboQuantization {
|
|
turbo: TurboQuantQuantizationConfig {
|
|
always_ram: None,
|
|
memory: None,
|
|
bits: Some(bits),
|
|
},
|
|
})
|
|
};
|
|
|
|
match self {
|
|
Self::Scalar => ScalarQuantizationConfig {
|
|
r#type: ScalarType::Int8,
|
|
quantile: Some(0.99),
|
|
always_ram: None,
|
|
memory: None,
|
|
}
|
|
.into(),
|
|
Self::Binary => BinaryQuantizationConfig {
|
|
always_ram: None,
|
|
memory: None,
|
|
encoding: None,
|
|
query_encoding: None,
|
|
}
|
|
.into(),
|
|
Self::ProductX4 => product(CompressionRatio::X4),
|
|
Self::ProductX8 => product(CompressionRatio::X8),
|
|
Self::ProductX16 => product(CompressionRatio::X16),
|
|
Self::ProductX32 => product(CompressionRatio::X32),
|
|
Self::ProductX64 => product(CompressionRatio::X64),
|
|
Self::Turbo1 => turbo(TurboQuantBitSize::Bits1),
|
|
Self::Turbo1_5 => turbo(TurboQuantBitSize::Bits1_5),
|
|
Self::Turbo2 => turbo(TurboQuantBitSize::Bits2),
|
|
Self::Turbo4 => turbo(TurboQuantBitSize::Bits4),
|
|
}
|
|
}
|
|
}
|