mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-29 01:17:56 -05:00
Andrey VasnetsovandClaude Sonnet 5
ca20151659
* Add edge-tool: CLI for creating, seeding, optimizing, and uploading local edge collections Mirrors the style of lib/edge/tools/shard_update and shard_query: `create` builds a minimal EdgeShard on disk (dense/sparse vectors, quantization presets including turbo4, payload indexes, target segment count), `upsert` seeds it with random points matching its live schema, `optimize` runs the shard optimizers, and `upload` pushes the resulting directory to S3/GCS. Useful for quickly spinning up test collections without a running Qdrant server, then promoting them to object storage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: initialize feature flags, enable serverless_compatible, fix --sparse ambiguity Initialize the global feature-flag OnceLock at startup (with serverless_compatible set, cascading write_segment_manifest/append_only_mutations/compact_bitmask/ append_only_storages) so runs no longer spam "Feature flags not initialized!" and collections are created in the serverless-compatible format. Also splits --sparse into a plain boolean flag plus a repeatable --sparse-name: clap's optional-value parsing for the old `--sparse [NAME]` form silently swallowed a following positional PATH as the sparse vector's name whenever --sparse was the last flag before it (e.g. `create --dense 1024 --sparse ./col`). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: fix --sparse=NAME to require_equals instead of a separate flag The --sparse/--sparse-name split from the previous commit lost the ability to name a sparse vector with --sparse itself. Restore a single --sparse[=NAME] flag, but with require_equals(true): clap then only binds a value via --sparse=NAME, never via a following bare token, so it stays safe next to the trailing PATH positional in every position (bare --sparse, --sparse=NAME, or multiple --sparse=NAME occurrences) without reintroducing the ambiguity that made --sparse swallow PATH as its value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: remove --segments from create, it has no effect there EdgeOptimizersConfig::default_segment_number only feeds MergeOptimizer as a merge-down ceiling (reduce segment count when it exceeds the target); unlike the main collection's LocalShard::build_local, EdgeShard::new never loops to pre-create N appendable segments. A freshly created collection always starts at exactly 1 segment, so passing --segments to `create` was silently a no-op. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: add --indexing-threshold-kb to create Unlike --segments (removed previously), the indexing threshold is a parameter IndexingOptimizer actually consults on every optimize() run: segments larger than it get an HNSW index built. Verified end-to-end (create with a 1KB threshold, upsert 2000 points, optimize) that it produces an hnsw-indexed segment where it would otherwise stay plain. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: add --clean to upload, wiping the destination prefix first Lists every object under DESTINATION and deletes it via ObjectStore::delete_stream before uploading, so re-uploading a collection recreated with a different shape (different segment UUIDs) doesn't leave the old segment's files behind. Verified against the local S3 proxy: uploaded one collection, then a second, differently-shaped one to the same prefix without --clean (29 objects, stale leftovers from the first); re-uploading the second with --clean correctly dropped it back to exactly its own 19 files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
30 lines
695 B
TOML
30 lines
695 B
TOML
[package]
|
|
name = "edge-tool"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
publish = false
|
|
|
|
[lints]
|
|
workspace = true
|
|
|
|
[[bin]]
|
|
name = "edge-tool"
|
|
path = "src/main.rs"
|
|
|
|
[dependencies]
|
|
edge = { path = "../.." }
|
|
common = { path = "../../../common/common" }
|
|
segment = { path = "../../../segment", default-features = false }
|
|
io_bridge_object_store = { path = "../../../common/io_bridge_object_store" }
|
|
|
|
anyhow = { workspace = true }
|
|
clap = { workspace = true }
|
|
env_logger = { workspace = true }
|
|
log = { workspace = true }
|
|
rand = { workspace = true }
|
|
serde_json = { workspace = true }
|
|
fs-err = { workspace = true }
|
|
object_store = { workspace = true }
|
|
futures = { workspace = true }
|
|
walkdir = { workspace = true }
|