Problems:
- The pinned `uv` version sometimes pulls broken python interpreter.
- The pinned `just` is too old to read `lib/edge/Justfile`.
Solution: update everything.
nix-shell --run 'cd tools/nix && npins update' # for default.nix
nix-shell --run 'cd tools/nix && npins upgrade' # for sources.json
Also, in this nixpkgs version, renames `nixfmt-rfc-style` into `nixfmt`.
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support`
Add BitSliceStorage<S> generic over
UniversalRead<u64>/UniversalWrite<u64>,
providing bit-level read and write operations over u64-element storage.
Read operations (S: UniversalRead<u64>):
- read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others)
- get_bit: single bit read via u64 element fetch
- read_bit_range: arbitrary bit range read
Write operations (S: UniversalWrite<u64>):
- set_bit / replace_bit: single bit write (skips write if unchanged)
- write_bit_range: arbitrary bit range write from BitSlice source
- fill_bit_range: fill range with a value
- set_bits_batch: batch individual bit updates coalesced by element
- flusher: flush underlying storage
* refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper
Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice
(Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>>
(storage-agnostic universal IO backend).
Updated consumers:
- MmapMapIndex
- MmapNumericIndex
- MmapGeoMapIndex
- MmapInvertedIndex (fulltext)
- ImmutableIdTracker
- Benchmark
Additionally:
- Add BitSliceStorage::create(path, num_bits) to encapsulate
file creation + sizing + open (eliminates duplicated size math
across 5 call sites)
- Add MmapBitSliceStorage type alias for
BitSliceStorage<MmapUniversal<u64>>
- Add count_ones() convenience method
- Use set_bits_batch() in wrapper flusher and all build paths
instead of per-bit set_bit() loops (coalesces u64 read-modify-writes)
- Fix silent error swallowing in wrapper get(): log + debug_assert
on I/O errors instead of .ok().flatten()
- Remove dead bitmap_mmap_size function from immutable_id_tracker
* use bitvec's approach for offset
* less api
* misc improvements
* refactor write method
* move to segment crate
* handle creation of file outside of StoredBitSlice logic
* fix codespell
* document bitwise calculations
* coalesce more updates into a single write, batch all writes
* use native `extend_from_bitslice` instead of iterating.
* clarity refactor
* clippyyy
* use `u64` as BitStore everywhere
* assume iterator is sorted
* only use chunk_by for generating runs
* fix update wrapper flusher
* review fixes
* larger set bits batch test
* remove reintroduced file
* oops, fix new test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Move pyproject.toml to root
* Migrate pyproject.toml from Poetry to uv
* Update GH workflows
* Update test script, doc and nix to use uv
* Use latest uv
* Fix uv.lock
* Cleanup shell.nix
* Cleanup
- Explicit `uv sync` is not required, `uv run` will install deps
automatically.
- We don't provide a python package, so the `[build-system]` section
is not needed.
* Fix UV_VERSION inconsistency
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Disable rocksdb compile time feature by default
* Also disable RocksDB feature in segment crate
* Enable RocksDB feature in all CI builds
* Remove extra job for testing non-RocksDB build, it's the default now
* Keep RocksDB structures in generated OpenAPI schema
* Fix obsolete --workspace flag breaking builds with explicit features
* Also build including RocksDB in e2e tests on CI
* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Introduce coverage reports for integration tests
* Install cargo-llvm-cov
* Use multiline script
* Explicityl setup COVERAGE env var
* fix integration tests and log generated data
* fix ls path
* upload lcov file to GH artifacts
* integration profraw dynamic filename
* Fix llvm profile filename template
* Use interrupt instead of kill and merge consensus test results into same file
* Drop upload artifact stage
* install llvm-cov
* upload as artifact and export coverage files
* try simplifying workflow
* Migrate coverage generation to existing dedicated gh workflow
* trigger on coverage related branches
* Build only if qdrant binary with cov doesnt exist
* Use valid yaml
* include mode in profraw filename
* split coverage workflow into parallel jobs
* add poetry version to env
* log poetry version to install
* clean up integration test workflow
* Simplify comments
* Fix coverage test OOD on CI
* Temporarily run coverage CI on PR
* Clean artifacts per package and store in tmp dir
* fix typo in gh workflow
* Add monitoring script again for debugging
* trigger script
* Fix monitor script path
* dont checkout dev branch
* Merge .info file only if test passed
* Try merging .info file only if it exists
* trigger ci
* fix workflow
* trigger ci
* Customize branch to check out
* Remove monitor.sh script
* remove unused env var
* Setup code coverage for Rust tests
* Add API key env var for codecov
* Save code coverage report for upload
* Trigger CI
* Try without explicit build step
* Run coverage job on self hosted runner
* Fix mistake because of which is was running fewer tests
* Run coverage on ubuntu latest
* Trigger CI
* fix name
* run coverage on all os
* Adjust after rebase to dev
* Avoid running coverage in other jobs
* fix test job
* reset test workflow
* Trigger CI
* run coverage after tests
* Add script to monitor resources
* fix indentation
* remove coverage dependency on test job
* fix needs field
* add monitoring script
* Run tail -f to monitor resources concurrently with tests
* Split commands
* Use --jobs=1 to possibly minimize RAM
* Try generating coverage one by one
* Dont clean coverage artifacts
* Merge .lcov files with lcov command
* Upload merged lcov.info file
* Run coverage for fewer packages
* Improve coverage script
* Move coverage.sh to tools dir
* Prepare for review
* Improve CI output and failure handling
* Generate HTML report if running locally
* Cleanup
* Run coverage on schedule and only for dev branch
