* ci(windows): skip IO-heavy tests that aren't OS-specific
On the Windows CI runner, several tests are 3-25x slower than on Ubuntu
purely due to slow filesystem IO. These tests exercise platform-agnostic
logic (optimizer, snapshot, WAL recovery, dedup, deferred points) and
are fully covered by the Linux and macOS jobs.
Mark them with `#[cfg_attr(target_os = "windows", ignore = "...")]` so:
- Windows CI skips them and finishes faster.
- They're still listed and runnable via `cargo test -- --ignored`
on Windows for local debugging.
Based on JUnit timings from CI run 26462785436 (PR #8827), this should
save ~5 minutes wall-clock on the Windows job, taking it closer to the
~13min Ubuntu and ~9min macOS jobs (currently 20m24s).
Tests affected:
- lib/wal: check_wal, check_last_index, check_clear, check_reopen,
check_truncate, check_prefix_truncate, test_prefix_truncate_parametric
- lib/edge/optimize: full tests module
- lib/segment deferred-point tests: read_operations,
dense_segment_combinations, sparse, facets
- lib/collection: snapshot_test, points_dedup, wal_recovery,
collection_test::test_ordered_read_api, snapshot_recovery_test
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep WAL and WAL-recovery tests on Windows
Reviewer correctly pointed out that WAL is mmap-backed and has
substantial Windows-specific code paths:
- Different segment allocation (fs4 vs rustix::ftruncate)
- Windows-specific delete_windows() with mmap-drop + retry loop
- Windows-specific sync_all() because directory fsync is unavailable
- Windows-specific lock proxy file (directories aren't lockable)
So those tests genuinely need Windows coverage. Reverted skips for:
- lib/wal/src/lib.rs: all check_* tests and test_prefix_truncate_parametric
- lib/collection/src/tests/wal_recovery_test.rs: all three tests
Still skipped on Windows (no OS-specific code in their production paths):
- lib/edge/optimize.rs (no cfg(windows) in source)
- lib/segment deferred-point tests (segment/ has no cfg(windows))
- lib/collection snapshot/dedup tests (collection/ has no cfg(windows))
- lib/collection integration snapshot_recovery + ordered_read_api
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep collection integration and snapshot_test
Per reviewer request, keep running these on Windows:
- lib/collection/tests/integration/* (snapshot_recovery_test,
collection_test::test_ordered_read_api)
- lib/collection/src/tests/snapshot_test.rs
These exercise higher-level collection/snapshot behavior that benefits
from cross-platform validation.
Remaining Windows skips (production code has no cfg(windows) branches):
- lib/edge/src/optimize.rs: 14 optimizer tests
- lib/segment/src/segment/tests/mod.rs: 4 deferred-point tests
- lib/collection/src/tests/points_dedup.rs: 2 dedup tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(windows): also skip HNSW/quantization integration tests
Per reviewer, also skip these segment integration test modules on Windows:
- hnsw_quantized_search_test::* (25 tests)
- multivector_filtrable_hnsw_test::* (rstest cases)
- multivector_quantization_test::* (rstest cases)
- byte_storage_quantization_test::* (rstest cases)
- payload_index_test::test_struct_payload_index_nested_fields
These exercise pure HNSW/quantization correctness on top of standard
segment IO that is already covered by tests we keep running on Windows.
Adds ~930s of sequential time to the Windows skip list, bringing the
expected wall-clock saving from ~3 min to ~8-10 min.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* Add progress_tracker.rs
* Pass progress tracker around
* Populate progress tracker with actual data
* Expose progress on `/collections/{name}/optimizations` endpoint
* Create and load for an appendable quantization
remove feature flag
more todo
review remarks
review remarks
* fix after rebase
* Rename is_appendable to supports_appendable
---------
Co-authored-by: timvisee <tim@visee.me>
* bq encodings
* are you happy clippy
* are you happy clippy
* are you happy clippy
* are you happy clippy
* gpu tests
* update models
* are you happy fmt
* move additional bits to the end
* fix tests
* Welford's Algorithm
* review remarks
* are you happy clippy
* remove debug println in test
* coderabit nitpicks
* remove unnecessary clone and partialeq
* Use f64 for Welford's Algorithm
* try fix ci
* revert cargo-nextest
* add debug assertions
* Use sparse vector storage fixture without RocksDB
* Fix some tests
* Replace RocksDB sparse storage with Gristore in mutable text index tests
* Use in-memory vector storage in multivector HNSW test
* Flag many more RocksDB specifics in tests
* Use database placeholder type without RocksDB flag
* Flag more tests
* Flag even more tests
* Fix imports, fix typo, and repair base test build
* Initialize dummy database if RocksDB flag is disabled
* Assert correct storage types
* Don't use old vector storage type when RocksDB is disabled
* Expos default for vector storage type only in tests
* Only expose simple segment constructor in tests
* Don't derive Default
* Fix inverted appendable flag
* Pass FeatureFlags into VectorIndexBuildArgs
* Incremental HNSW index building: append-only case
* Use debug_assert
* first_few_ids
* Check deleted_point_count
* Drop unused method
* Add in rest and grpc
* add to QueryEnum
* implement Query trait
* connect to scorer creation
* upd tests
* additional changes
* fmt
* gen openapi and grpc docs
* coderabbit fix
* add changes in async scorer
* test sum_scores in more places, refactor to remove repetition
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* rename cpu_budget -> resource_budget
* clippy
* add io budget to resources
* fmt
* move budget structures into a separate file
* add extend permit function
* dont extend existing permit
* switch from IO to CPU permit
* do not release resource before aquiring an extension
* fmt
* Review remarks
* Improve resource permit number assertion
* Make resource permit replace_with only acquire extra needed permits
* Remove obsolete drop implementation
* allocate IO budget same as CPU
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Measure read io for payload storage
* Add Hardware Counter to update functions
* Fix tests and benches
* Rename (some) *_measured functions back to original
* drop some code
* Drop JsonPathString
* Fix test_remove_key
Drop failing tests:
- Deleting array indices is not idempotent, so we don't support it.
- Empty JSONPath is not supported.
* Make json_path::path() non-generic
* Remove references to JsonPathV2
* Drop JsonPathInterface
* Move json_path::v2 code into json_path
* Drop validate_not_empty
* Drop JsonPath::head() as being unused
* Replace path() with JsonPath::new()
* Restore comments
* Move tests to json_path
* Use json() consistently in tests
* Replace many into calls with Into trait
---------
Co-authored-by: timvisee <tim@visee.me>
* Move build_index out of VectorIndex
* Build index in HNSWIndex::open()
* Introduce HnswIndexOpenArgs
* Proper deletion
* Improve tests
* HNSW::open(): add warn, comment and assert
* Revert to making up the config if it does not exist
* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
