* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* Fixups of amalgamator
Fix issues that break `qdrant-edge` build process:
- `use … as segment;` - this causes `ast-grep` rules to replace wrong
paths. So, rename to avoid collisions.
- `#[macro_use]` and `extern crate` required be in the top-level
`lib.rs`.
- `format!("…", crate::something::…)` - `ast-grep` can't fix paths
inside macros. Fixed by moving `crate::something::…` out of the macro.
* Add lib/edge/publish workspace and amalgamation script
* Move `lib/edge/examples` into `lib/edge/publish/` workspace
And fix them to use the generated `qdrant-edge` crate.
* Add github workflow
* Cleanup `qdrant-edge` public API
Removes empty modules. Checked by `cargo doc`.
* Flag index selector
* Flag take_database_snapshot
* Flag field index builder
* Flag bool index
* Flag full text index
* Flag geo index
* Flag map index
* Feature flag numeric index
* Flag skip_rocksdb
* Flag payload storage
* Flag segment
* Flag builder
* Flag backup
* Make RocksDB crate optional
* Tweak feature flag gate
* Disable default segment crate features from all dependents
* Handle flag in collection crate correctly
* Flag payload storage types in tests
* Temporarily disable on-disk check for sparse vectors
* Flag sparse vector RocksDB in GPU code
* Fix compilation error since recent merge
* Flag payload storage types in new function
* remove todo
* Explicitly implement default to more clearly state difference
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move problems.rs to collection crate
* change type on datetime_key expression
* extract from expression
* check for unindexed fields in formula
* clippy
* dedup code
* Only implement verification for `CollectionQuery` types (#6319)
* verify CollectionQuery's Prefetch
* remove implementation for REST query
* remove implementation for REST query groups
* add basic openapi test
* submit an issue for unindexed field
solve unindexed field issue when an index is created for the field
also log a warning when issue is submitted
fmt
fix rebase problems
get collection name without panicking
better solving of issue
remove outdated TODO
add tests, fix GET /issues
review fixes
use `/` instead of `.` as separator
fmt
prepare extractor for time-based submission
smol rename
add check in search
hook unindexed field creation in search and search_batch of rest
update after rebase on `dev`
remove submit from `struct_payload_index`
solve issues inside of handlers, not in match router
post-process at Collection level
don't clone filters
remove from openapi spec
- Add hidden env variable to adjust slow search threshold
- Remove Solution::None
- Fix UnindexedField issue extractor
- Add endpoint to openapi, but without response body spec
- Move integration test to consensus_tests to set lower threshold
* event-based integration
* use typed Code
* remove payload_index_schema hack, get through collection info
* rename `notify` to `publish`
* update after rebase
* remove breakpoint
* update after rebase
* attach current payload schema to SlowQueryEvent
* collect filters refs lazily
* review fixes
* review fixes
* use regular config for setting the threshold, use 1.2 secs as default
* fix checking against current schema for all conditions
* Clone current segment config to deprecated type
* Remove segment level quantization config from segment config
* Also deprecate current VectorDataConfig
* Update old segment migration to work with new refactoring
* Move index into vector data config
* Move vector data config migration logic into segment level
* Remove hnsw_config from vector data config
* Rename collection params to vector data conversions function
* Move storage type into vector data config
* Set appendable flag correctly
* Clean up and reformat
* Make segment on disk flag not optional
* Add appendable flag to segment config to replace storage type
* Remove storage type from segment config
* Deprecate storage type enum
* Use consistent variable naming
* Cleanup
* Add segment config migration for v0.5.0 to current
* Bump segment to 0.6.0
* Remove serde defaults for new storage and vector data config types
These default value configurations are not needed anymore, because these
structs are not used to deserialize old data. All current fields should
always be available in these structs. When new fields are added in new
functions, the serde default annotation must be set again.
* Cleanup
* Update OpenAPI specification
This updates the returned data structure on telemetry endpoints, as a
result of segment configuration refactoring.
* Fix quantization configuration not falling back to collection config
* Fix compiler warning when building in release mode
* Move deprecated type structs into compat module
* Update allow deprecated attributes
* Assign quantization config only in segment optimizer
* Remove unsued parameter
* Add vector storage type enum to vector data config
* Remove appendable and on_disk flags from segment and vector config
* Update OpenAPI specification
* add tests
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* remove deleted flags from vector storage
* remove deleted flags from mmap
* new simple vector storage format
* are you happy clippy
* remove id_tracker from raw_scorer
* revert vector storage format changes
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* WIP: Fix `Segment::take_snapshot`
TODO:
- This commit, probably, breaks snapshotting of segments with memmapped vector storage
- `ProxySegment::take_snapshot` seems to potentially similar bug
* WIP: Fix `Segment::take_snapshot`
- Fix snapshotting of `StructPayloadIndex`
- Fix snapshotting of segments with memmapped vector storage
- Temporarily break `ProxySegment::take_snapshot`
* Fix `ProxySegment::take_snapshot`
* Remove `copy_segment_directory` test
* nitpicking
* clippy fixes
* use OperationError::service_error
* Cleanup `TinyMap` trait bounds and derive `Debug`
* Fix `test_snapshot` test
- Derive `Debug` for `NamedVectors`
* Move utility functions from `segment.rs` to `utils` module
* Contextualize `segment::utils::fs::move_all` a bit more carefully
* Fix a typo
* add backward compatibility with old snapshot formats
* fmt
* add snapshot for compatibility test
* git lfs is a piece of shit
* Nitpicking
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
- Add a simple abstraction/wrapper over `memmap2::Advice` and a global switch
to select "memory advice" used for memmapped storage to the `segment` crate
- Add `mmap_advice` configuration option to the `storage::types::StorageConfig`
- Implement `search-points` benchmark
* many named vectors per point (segment-level)
* operation result for dim function
* beautifulized vector name
* fix naming bug
* segment version migration
* fmt
* add segment tests
* are you happy clippy
* fix build
* [WIP] many named vectors per point (collection-level) (#975)
* config and search
* fix placeholders for proxy segment move
* remove VectorType from collection
* are you happy fmt
* vectors in grps messages
* create collections with vectors
* segment holder fixes
* are you happy fmt
* remove default vector name placeholders
* are you happy fmt
* are you happy clippy
* fix build
* fix web api
* are you happy clippy
* are you happy fmt
* record vector&vectors
* openapi update
* fix openapi integration tests
* segment builder fix todo
* vector names for update from segment
* remove unwrap
* backward compatibility
* upd openapi
* backward compatible PointStruct
* upd openapi
* fix record back-comp
* fmt
* vector configuration backward compatibility
* fix vetor storage size estimation
* fmt
* multi-vec segment test + index test
* fmt
* api integration tests
* [WIP] Named vectors struct (#1002)
* move to separate file
* named vectors as struct
* use cow
* fix build
* keys iterator
* avoid copy in PointStruct -> get_vectors
* avoid another copy
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* allow additional params for payload field index
* fmt
* wip: full text index building
* fmt
* text search request
* text search request
* full text index persitance and loading
* fmt
* enable fts index in mapping
* clippy
* fix tests + add integration test
* review fixes: extend payload index test
* revert incedental change
* segment telemetry
* anonymize trait
* fix build
* are you happy fmt
* anonimyze implementations
* sliding window avg (#826)
* Actix web telemetry (#828)
* actix web telemetry
* small as move
* use tokio mutex instead of std
* add comments
* are you happy fmt
* use u16 as http status code
* telemetry structs rename
* fix build
* using parking lot mutex
* telemetry web api (#842)
* telemetry web api
* telemetry openapi (#843)
* use async mutex for telemetry collector
* use tokio mutex for telemetry collector
* are you happy fmt
* improve alias tests
* introduce new alias persistence and remove Sled
* introduce FileStorageError
* use new error in AliasMapping
* aliases are kept in memory and saved to disk when modified
* better naming
* make alias renaming atomic
* fmt
* use correct alias and docs
* code review: cleaner error conversion
* wip: geo helpers
* draft impl
* code review
* clippy
* please clippy
* code review: use haversine distance from geo lib
* filter properly geohashes for the circle
* code review: better approximate match for circles by being less strict
* make sure to return at least the precision where the area is fully covered
* add test for distance smaller than highest precision
* random query test
* fmt
* improve area computation and document south pole issue
* wip: sphere_neighbor
* fix max latitude
* fmt
* protect from longitude overflow & increase reliability
* more testing
* refactor
* fix clippy
* cleanup tests
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* update segment version after completed update only
* more stable updates: check pre-existing points on update, fail recovery, WAL proper ack. check_unprocessed_points WIP
* switch to async channel
* perform update operations in a separate thread (#111)
* perform update operations in a separate thread
* ordered sending update signal
* locate a segment merging versioning bug
* rename id_mapper -> id_tracker
* per-record versioning
* clippy fixes
* cargo fmt
* rm limit of open files
* fail recovery test
* cargo fmt
* wait for worker stops befor dropping the runtime
* raw points scorer
* raw point scorer for memmap storage
* search interface prepare
* graph binary saving + store PointOffsetId as u32
* WIP: entry points
* connect new link method
* update libs + search layer method + visited list + search context + update rust
* implement Euclid metric + always use MinHeap for priority queue
* small refactor
* search for 0 level entry
* update visited pool to be lock free and thread safe
* use ef_construct from graph layer struct + limit visited links to M
* add metric pre-processing before on vector upsert
* old hnsw heuristic
* save hnsw graph for export
* search method + tests
* small fixes
* add benchmark and profiler
* build time optimizations
* use SeaHash
* remove unsed benchmark
* merge hnsw graph function
* WIP:HNSW index build function
* HNSW build_index with additional indexing
* refactor fixtures
* graph save and load test
* test and fixes for filterable HNSW
* enable hnsw index for query planning
* fix cardinality estimation tests + remove query planner as class
* small refactor
* store full copy of collection settings with collection + allow partial override on creation #16
* API for updating collection parameters #16
* refactor: move collection error -> types
* report collection status in info API #17
* update OpenAPI Schema
