* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* Remove UniversalRead::is_empty
Reason: Less methods to override in wrappers. And I don't think this
method makes much sense.
* Move common::universal_io::{ => wrappers}::read_only
* Update ReadOnly wrapper
Implement missing methods, and follow the code style of the upcoming
TypedStrorage wrapper.
* Add TypedStorage
* Use TypedStorage
* Add reminder comments
* Clarify TypedStorage use-case
* UIO traits: require Sized
I don't think we ever going to add unsized implementation. So, lets,
require it on the whole trait rather than on individual methods.
* Move `AccessPattern` to the `common` package
Was: segment::vector_storage::vector_storage_base::AccessPattern
Now: common::generic_consts::AccessPattern
* UIO traits: replace `bool` with `AccessPattern`
* refactor(edge): let edge re-export segment/shard/sparse
* chore(ast-grep-rules): handle path like `::shard::blah::Blah`
This commit was required during the refactoring. It turns out we don't
need it right now, but it might be useful in the future. Or not.
* refactor(edge-rs): Rehaul public API
* chore: gitignore .idea and .vscode globally
Reason: `lib/edge/publish` is a separate Cargo workspace and I'd like to
have a separate rust-analyzier configuration for it, which is stored in
`.vscode/settings.json`. Not sure about `.idea`, but I guess the same
logic applies.
* chore: Use Iterator::is_sorted (stabilized in Rust 1.82)
* chore: remove unrelated files
These were added in #5501 (42fd2e27), perhaps by accident?
* lib/edge/publish/cargo: improve target directory discovery
Don't hardcode workspace root, ask cargo instead.
* Fix warning in qdrant-edge amalgamation
`cargo check` in `lib/edge/publish` produces the following warning:
warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending`
* lib/edge/publish/examples: fix compiler warnings
* refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs
To match python examples that have the same constant in `common.py`.
* Use `lib/edge/data` dir for both Python and Rust examples
Before this commit, `lib/edge/publish/data` and `lib/edge/python/data`
were used. I'd like Python and Rust to use the same prepared resources
in the future.
* ❗Add lib/edge/Justfile with recipes to build/run examples
I constantly keep forgetting how to compile and run edge stuff.
Also, this would be used in CI in subsequent commits.
* ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes
Also now we can rename `creates-io-readme.md` to `README.md` like
reasonable people.
* ❗doc: Add qdrant-edge-py README.md
Since the previous commit removed qdrant-edge-py README.md, we need to
add something in its place instead.
The new README is adapted from Rust edge README.
And the previous README wasn't good anyway, see [1], it mentions maturin
which is irrelevant for users.
[1]: https://pypi.org/project/qdrant-edge-py/0.5.0/
* GHA: Add "Setup Qdrant" composite action
We need running Qdrant instance to test edge examples on CI. We can't
use docker for this because we want to run tests on Windows and macOS.
OTOH, this action downloads binaries for all platforms provided in
https://github.com/qdrant/qdrant/releases/tag/v1.17.0
* ❗GHA: Use single worflow for python and rust edge examples
Merge workflows to reuse build cache. Also, modernize it by using `uv`,
and recently introduced `setup-qdrant` and Justfile. Also, make it run
on three platforms (only on merges).
* ❗GHA: edge-{py,rust}-release: also run examples before publishing
Use similar approach as in the previous commit. But note that this time
we build/test also on ARM Linux and Windows.
* amalgamate.py: replace `env!("CARGO_PKG_VERSION")`
Otherwise segment thinks it's v0.0.0 which breaks examples.
* Fix lib/edge/python/examples/repr.py
It it broken by d9393acac6 (#7933), which renamed `SegmentConfig` →
`EdgeConfig` and removed `index` parameter from `VectorDataConfig` by
hardcoding it to `Plain`.
Before:
$ python lib/edge/python/examples/repr.py
NameError: name 'SegmentConfig' is not defined
After:
$ python lib/edge/python/examples/repr.py
EdgeConfig(vector_data={"": VectorDataConfig(size=128, ...)}, ...)
* gitignore output of Rust edge examples
* Restore `lib/edge/python/examples/.gitignore`
I'm not sure why it's removed in 65312a1e82e6ae4d/#7522.
* Add `prepare_facet_snapshot.sh` for `facet_test.rs`
The script is based on steps mentioned in #8045.
* Remove `edge-cli.rs` example
It was added in 895318913afbad4a/#7140 and seems incomplete.
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* amalgamate.py: specify encoding="utf-8"
For legacy platforms that still default to cp1252.
* amalgamate.py: use `shutil.which`
In case if `ast-grep` is a cmd wrapper.
* amalgamate.py: write rules to temporary file
Windows runner don't like inline-rules.
* amalgamate.py: Add decription to package
Required for publishing to crates.io.
* amalgamate.py: package build scripts for `common` and `segment`
Also, add more comments.
* amalgamate.py: keep generated module order deterministic
Not an issue, but nice to have. Suggested by CodeRabbit.
* edge-rust-release.yml: also check on arm machines
* edge-rust-release.yml: Use cargo build instead of cargo check
To catch linker errors.
* edge-rust-release.yml: disable fail-fast (for easier debugging)
* edge-rust-release.yml: add --no-verify to cargo publish
The previous job `edge-rust-check` already builds it, no need to build
it again in the `publish` job.
* Fixups of amalgamator
Fix issues that break `qdrant-edge` build process:
- `use … as segment;` - this causes `ast-grep` rules to replace wrong
paths. So, rename to avoid collisions.
- `#[macro_use]` and `extern crate` required be in the top-level
`lib.rs`.
- `format!("…", crate::something::…)` - `ast-grep` can't fix paths
inside macros. Fixed by moving `crate::something::…` out of the macro.
* Add lib/edge/publish workspace and amalgamation script
* Move `lib/edge/examples` into `lib/edge/publish/` workspace
And fix them to use the generated `qdrant-edge` crate.
* Add github workflow
* Cleanup `qdrant-edge` public API
Removes empty modules. Checked by `cargo doc`.
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Remove call to Archive::set_sync
Also, this method was the last remaining part of our `tar-rs` fork,
so we can switch to the upstream version now.
* Do syncfs
* Replace `Option<Segment>` with `enum LoadSegmentOutcome`
* Replace some Path/PathBuf with str/String
* Rename field Segment::{current_path -> segment_path}
* safe_delete
* Add progress_tracker.rs
* Pass progress tracker around
* Populate progress tracker with actual data
* Expose progress on `/collections/{name}/optimizations` endpoint
* Add VectorStorageEnum::{get_vector_bytes_opt, get_vector_layout}
* Update CompressedWithVectors format to include base vectors
* StorageGraphLinksVectors
* Clarify code
* PrimitiveVectorElement: use zerocopy
* PrimitiveVectorElement: also use zerocopy::FromBytes
* serialize_graph_links: back_index is Vec<PointOffsetType>
* for_each_link_with_vector: loosen lifetimes
* Add common::typelevel module
* Use common::typelevel for score_bytes
* Revert "Split FilteredScorer into FilteredScorer and FilteredQuantizedScorer"
This reverts commit ff3a93ee95.
* FilteredQuantizedScorer
* Extract `struct Filters` from `FilteredScorer` and `FilteredQuantizedScorer`
* Revert "QuantizedVectors::query_scorer_bytes"
This reverts commit 9dee824afd.
* Partial revert "Add QueryScorerBytes::score_bytes"
This reverts commit d80af4f698, but keeps
the QueryScorerBytes trait while removing implementations.
* Partial revert "Add EncodedVectorsBytes::score_point_vs_bytes"
This partially reverts commit c741e42f9a.
- trait `EncodedVectorsBytes` and its implementations: reverted/removed
- calls to `EncodedVectorsBytes::score_point_vs_bytes`: replaced with
calls to `EncodedVectors::score_bytes`
* Remove unused method FilteredScorer::new_from_raw
* TestRawScorerProducer to use VectorStorageEnum
* TestRawScorerProducer:🆕 accept distance as an argument
* Drop TMetric from TestRawScorerProducer
* Remove `impl VectorStorage for TestRawScorerProducer`
* Add TestRawScorerProducer::internal_scorer
* Remove TestRawScorerProducer::get_vector
* Rename TestRawScorerProducer::{get_scorer => scorer}
* refactor: add GraphLinks::to_edges
* refactor: copy-on-write is no more
* Healing: insert backlinks
* Replace SearchContext with FixedLengthPriorityQueue
* Make EncodedVectors::EncodedQuery associated type
* Make EncodedStorageBuilder::Storage associated type
* Make QueryScorer::TVector associated type
* Remove TVector from RawScorerImpl generic parameters
* Refactor out LinksContainer
This commit extracts Vec<PointOffsetType> and heuristic related
functions from graph_layers_builder.rs into a new file.
No significant changes in logic are made.
* Optimize LinksContainer
* Add illustration
* Fixups
* Add more comments and rename `score` to `cached_score`
xzfc