* ConditionChecker::check_batched: trait + ConditionCheckerEnum
* check_batched for OptimizedFilter
* OnDiskPointToValues::values_iter_batch: improve performance
* OnDiskPointToValues::values_iter_batch: update interface
Accept bitvec, call on every point, pass UserData.
* check_batched for geo index
* geo_index tests: add same_geo_index_between_points_with_dups_test
Catches broken load_from_on_disk/for_all_points_values.
* check_batched for map index
* check_batched for numeric index
* check_batched for full-text index
* tests for check_batched
* Review fixups
- default_check_batched: avoid running pred twice at boundary
- condition_checker: use explicit match over matches!/if-else
- Partitioner: use assert! over debug_assert!
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* ConditionChecker::check_batched: &mut self -> &self
* ConditionChecker::check_batched: make mandatory
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Non-generic TransformInto::transform
I think dyn is fine here because it is executed once per query, not in a
hot loop.
* Query::score_by impls: manual loops
Iterator chains (`.iter().map(..).sum()`) produce a lot work for the
compiler to do.
* QuantizedCustomQueryScorer: move generics from struct to `new` method
* Refactor geo_index tests: shorer geopoints
GeoPoint is serializable, no need to write json fields manually.
* Refactor geo_index tests: extract filtered_points() helper
* Refactor geo_index tests: extract assert_cardinality helpers
* Refactor geo_index tests: hoist duplicated check_cardinality_match to module level
* Normalize point_to_values test
Instead of two separate copypaste geo/string tests, make the test
generic and instantiate it 4 times.
* Refactor geo_index tests: reload_index
* Add NumericIndexValue
* MapConditionChecker: get rid of `F: Fn() -> bool` bound
To put this type into the upcoming `ConditionCheckerEnum`, it should be
nameable.
* filter_context: `Box<dyn ConditionChecker>` -> `OptimizedFilter`
Removes one level of dyn indirection, so faster checks.
* NullConditionChecker: merge IsEmpty/IsNull checkers into one
So, less variants in the upcoming `ConditionCheckerEnum`.
* Add DynConditionChecker type alias
* Add DynConditionChecker type alias (point_scorer.rs)
* Move trait ConditionChecker to `common`
Reason: will be used both in `segment` and `sparse` crates.
* Migrate InvertedIndexCompressedMmap to UniversalRead
* Rehaul search_scratch.rs (was scores_memory_pool.rs)
- Use `typed_arena::Arena` instead of `bumpalo::Bump`.
Reason: bump don't drop.
Drawback: `Arena::new()` is not free, and `Arena::clear()` doesn't
exist; but this commit has workarounds.
- Use names that make more sense.
* Misc fixes
* InvertedIndexCompressedMmap: explicit S type parameter
* dataset: add features to fs-err
Reason: fix `cargo check --all-targets -p dataset`. Feature unification
pulls `fs-err/debug` + `fs-err/tokio` without `fs-err/debug_tokio`.
* dataset: run `cargo metadata` to get target dir
Reason: avoid re-downloading in separate worktrees. The old
`cargo locate-project` ignores custom target dirs.
* sparse/benches/search: cache vectors/indices
* BorrowedReadPipeline/OwnedReadPipeline
* `fn new(file: &Self::File)` -> `fn new(file: Self::File)`
Also, partially revert some related changes from the previous commit.
* Callback-based InvertedIndex::for_each_vocab_with_postings_len
* Integrate UniversalHashMap into full-text index
* Integrate UniversalHashMap into immutable_map_index
* UniversalReadPipeline
* Performance: cache pointers in MmapFile
The previous commit removed `MmapFile::read_batch` method override. So,
`MmapFile` now re-uses the default `UniversalRead::read_batch` impl,
which is implemented using `UniversalReadPipeline` interface.
Unsurprisingly, it caused a slowdown in the benchmarks, particularly
this one:
cargo bench -p common --bench universal_io -- mmap/8bytes/read_batch_full
This commit reclaims the performance back.
* remove unfulfilled lint
* Review suggestions
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Segment::approximate_facet: don't use iter
* Callback-based FacetIndex
* take deferred internal id out of the loop
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* fix(amalgamate): strip `mod gpu`
Fixes `cargo +nightly fmt` in `lib/edge/publish`. Previously we had to
run `cargo +nightly fmt -p examples` to avoid the error.
* fix(Justfile): unset VIRTUAL_ENV
Otherwise it causes problems when running that Justfile from other
(unrelated) venvs. Particularly, `uv pip install` and `maturin develop`
will mess with cached venvs created by `uv run --script` which sounds
insane.
Problems:
- The pinned `uv` version sometimes pulls broken python interpreter.
- The pinned `just` is too old to read `lib/edge/Justfile`.
Solution: update everything.
nix-shell --run 'cd tools/nix && npins update' # for default.nix
nix-shell --run 'cd tools/nix && npins upgrade' # for sources.json
Also, in this nixpkgs version, renames `nixfmt-rfc-style` into `nixfmt`.
* IoUringState: generic `RequestId`
* UniversalRead: generic `RequestId`
* Simplify `gridstore::Pages::get_page_value_ranges`
Now we don't need two separate `SmallVec`s as we can put `buffer_offset`
into `RequestId`.
* Better doc comment
* Rename `RequestId` -> `Meta`
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* Remove UniversalRead::is_empty
Reason: Less methods to override in wrappers. And I don't think this
method makes much sense.
* Move common::universal_io::{ => wrappers}::read_only
* Update ReadOnly wrapper
Implement missing methods, and follow the code style of the upcoming
TypedStrorage wrapper.
* Add TypedStorage
* Use TypedStorage
* Add reminder comments
* Clarify TypedStorage use-case
* UIO traits: require Sized
I don't think we ever going to add unsized implementation. So, lets,
require it on the whole trait rather than on individual methods.
* Move `AccessPattern` to the `common` package
Was: segment::vector_storage::vector_storage_base::AccessPattern
Now: common::generic_consts::AccessPattern
* UIO traits: replace `bool` with `AccessPattern`
* refactor(edge): let edge re-export segment/shard/sparse
* chore(ast-grep-rules): handle path like `::shard::blah::Blah`
This commit was required during the refactoring. It turns out we don't
need it right now, but it might be useful in the future. Or not.
* refactor(edge-rs): Rehaul public API
* chore: gitignore .idea and .vscode globally
Reason: `lib/edge/publish` is a separate Cargo workspace and I'd like to
have a separate rust-analyzier configuration for it, which is stored in
`.vscode/settings.json`. Not sure about `.idea`, but I guess the same
logic applies.
* chore: Use Iterator::is_sorted (stabilized in Rust 1.82)
* chore: remove unrelated files
These were added in #5501 (42fd2e27), perhaps by accident?
* lib/edge/publish/cargo: improve target directory discovery
Don't hardcode workspace root, ask cargo instead.
* Fix warning in qdrant-edge amalgamation
`cargo check` in `lib/edge/publish` produces the following warning:
warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending`
* lib/edge/publish/examples: fix compiler warnings
* refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs
To match python examples that have the same constant in `common.py`.
* Use `lib/edge/data` dir for both Python and Rust examples
Before this commit, `lib/edge/publish/data` and `lib/edge/python/data`
were used. I'd like Python and Rust to use the same prepared resources
in the future.
* ❗Add lib/edge/Justfile with recipes to build/run examples
I constantly keep forgetting how to compile and run edge stuff.
Also, this would be used in CI in subsequent commits.
* ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes
Also now we can rename `creates-io-readme.md` to `README.md` like
reasonable people.
* ❗doc: Add qdrant-edge-py README.md
Since the previous commit removed qdrant-edge-py README.md, we need to
add something in its place instead.
The new README is adapted from Rust edge README.
And the previous README wasn't good anyway, see [1], it mentions maturin
which is irrelevant for users.
[1]: https://pypi.org/project/qdrant-edge-py/0.5.0/
* GHA: Add "Setup Qdrant" composite action
We need running Qdrant instance to test edge examples on CI. We can't
use docker for this because we want to run tests on Windows and macOS.
OTOH, this action downloads binaries for all platforms provided in
https://github.com/qdrant/qdrant/releases/tag/v1.17.0
* ❗GHA: Use single worflow for python and rust edge examples
Merge workflows to reuse build cache. Also, modernize it by using `uv`,
and recently introduced `setup-qdrant` and Justfile. Also, make it run
on three platforms (only on merges).
* ❗GHA: edge-{py,rust}-release: also run examples before publishing
Use similar approach as in the previous commit. But note that this time
we build/test also on ARM Linux and Windows.
* amalgamate.py: replace `env!("CARGO_PKG_VERSION")`
Otherwise segment thinks it's v0.0.0 which breaks examples.
* Fix lib/edge/python/examples/repr.py
It it broken by d9393acac6 (#7933), which renamed `SegmentConfig` →
`EdgeConfig` and removed `index` parameter from `VectorDataConfig` by
hardcoding it to `Plain`.
Before:
$ python lib/edge/python/examples/repr.py
NameError: name 'SegmentConfig' is not defined
After:
$ python lib/edge/python/examples/repr.py
EdgeConfig(vector_data={"": VectorDataConfig(size=128, ...)}, ...)
* gitignore output of Rust edge examples
* Restore `lib/edge/python/examples/.gitignore`
I'm not sure why it's removed in 65312a1e82e6ae4d/#7522.
* Add `prepare_facet_snapshot.sh` for `facet_test.rs`
The script is based on steps mentioned in #8045.
* Remove `edge-cli.rs` example
It was added in 895318913afbad4a/#7140 and seems incomplete.
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* amalgamate.py: specify encoding="utf-8"
For legacy platforms that still default to cp1252.
* amalgamate.py: use `shutil.which`
In case if `ast-grep` is a cmd wrapper.
* amalgamate.py: write rules to temporary file
Windows runner don't like inline-rules.
* amalgamate.py: Add decription to package
Required for publishing to crates.io.
* amalgamate.py: package build scripts for `common` and `segment`
Also, add more comments.
* amalgamate.py: keep generated module order deterministic
Not an issue, but nice to have. Suggested by CodeRabbit.
* edge-rust-release.yml: also check on arm machines
* edge-rust-release.yml: Use cargo build instead of cargo check
To catch linker errors.
* edge-rust-release.yml: disable fail-fast (for easier debugging)
* edge-rust-release.yml: add --no-verify to cargo publish
The previous job `edge-rust-check` already builds it, no need to build
it again in the `publish` job.
* Fixups of amalgamator
Fix issues that break `qdrant-edge` build process:
- `use … as segment;` - this causes `ast-grep` rules to replace wrong
paths. So, rename to avoid collisions.
- `#[macro_use]` and `extern crate` required be in the top-level
`lib.rs`.
- `format!("…", crate::something::…)` - `ast-grep` can't fix paths
inside macros. Fixed by moving `crate::something::…` out of the macro.
* Add lib/edge/publish workspace and amalgamation script
* Move `lib/edge/examples` into `lib/edge/publish/` workspace
And fix them to use the generated `qdrant-edge` crate.
* Add github workflow
* Cleanup `qdrant-edge` public API
Removes empty modules. Checked by `cargo doc`.
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Remove call to Archive::set_sync
Also, this method was the last remaining part of our `tar-rs` fork,
so we can switch to the upstream version now.
* Do syncfs
xzfc