Commit Graph

3618 Commits

Author SHA1 Message Date
Arnaud Gourlay
f4b4c6c503 Bump dev deps (#7104)
* bump criterion 0.7.0

* bump rstest 0.26.1

* bump pprof 0.15.0

* remove unused bitvec

* bump prettyplease 0.2.37
2025-08-20 12:43:58 +02:00
dependabot[bot]
599a6e42ad build(deps): bump anyhow from 1.0.98 to 1.0.99 (#7099) 2025-08-20 10:07:30 +00:00
Arnaud Gourlay
bd429e9a96 bump test_get_storage_size value to accommodate MacOS change (#7103) 2025-08-20 11:40:41 +02:00
Tim Visée
2b4f19f26d Fix new shard crate always pulling in RocksDB dependencies (#7097) 2025-08-19 17:25:18 +02:00
Roman Titov
60e0c07b0a Move ProxySegment from collection into shard crate (#7093) 2025-08-19 14:01:43 +02:00
dependabot[bot]
5eb81ab332 build(deps): bump serde-untagged from 0.1.7 to 0.1.8 (#7084)
Bumps [serde-untagged](https://github.com/dtolnay/serde-untagged) from 0.1.7 to 0.1.8.
- [Release notes](https://github.com/dtolnay/serde-untagged/releases)
- [Commits](https://github.com/dtolnay/serde-untagged/compare/0.1.7...0.1.8)

---
updated-dependencies:
- dependency-name: serde-untagged
  dependency-version: 0.1.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-19 09:45:16 +02:00
dependabot[bot]
05e6488f23 build(deps): bump async-trait from 0.1.88 to 0.1.89 (#7085)
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.88 to 0.1.89.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.88...0.1.89)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-version: 0.1.89
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-19 09:44:39 +02:00
Ivan Pleshkov
3f75fae516 EncodedStorage upsert vector (#7048)
* EncodedStorage upsert vector
2025-08-18 17:39:23 +02:00
xzfc
acfdf02da8 Optional scorer methods via TBool (#6989)
* Add common::typelevel module

* Use common::typelevel for score_bytes

* Revert "Split FilteredScorer into FilteredScorer and FilteredQuantizedScorer"

This reverts commit ff3a93ee95.

* FilteredQuantizedScorer

* Extract `struct Filters` from `FilteredScorer` and `FilteredQuantizedScorer`

* Revert "QuantizedVectors::query_scorer_bytes"

This reverts commit 9dee824afd.

* Partial revert "Add QueryScorerBytes::score_bytes"

This reverts commit d80af4f698, but keeps
the QueryScorerBytes trait while removing implementations.

* Partial revert "Add EncodedVectorsBytes::score_point_vs_bytes"

This partially reverts commit c741e42f9a.
- trait `EncodedVectorsBytes` and its implementations: reverted/removed
- calls to `EncodedVectorsBytes::score_point_vs_bytes`: replaced with
  calls to `EncodedVectors::score_bytes`

* Remove unused method FilteredScorer::new_from_raw
2025-08-18 14:40:12 +00:00
Roman Titov
4bcb21b4b7 Cleanup collection::operations and shard::operations modules (#7064) 2025-08-18 16:26:06 +02:00
Jai
57ae0dd044 Fix criterion batch_query_bench (#6825) (#7077) 2025-08-18 15:16:30 +02:00
Ivan Pleshkov
8961eeaf0a Remove quantization save functions (#7043)
* remove quantization save

* revert convert_binary_encoding fn

* review remarks

* review remark

* review remarks
2025-08-18 13:55:07 +02:00
Roman Titov
13b05d18a3 Extract CollectionUpdateOperations into separate shard crate (#7054) 2025-08-18 11:48:02 +02:00
Andrey Vasnetsov
4eea3512fc do not skip empty histogram values + remove too small buckets (#7073)
* do not skip empty histogram values + remove too small buckets

* fix tests and remove some more values from the histogram

* remove redundant test
2025-08-18 11:19:41 +02:00
Roman Titov
2c9c39b4c1 Move SegmentHolder::scroll_read_lock to LocalShard::scroll_read_lock (#7053) 2025-08-18 09:56:34 +02:00
Ivan Pleshkov
995db9462f Bq encoding conversions as a separate function (#7070)
* bq encoding conversions as a separate function

* rename

* rename
2025-08-15 16:49:40 +02:00
Andrey Vasnetsov
c188f5b7c4 Update if (#7006)
* implement condition parameter for upsert operation

* fmt

* update api schema

* implement conditional update for update-vectors op

* rename to update_filter for consistency

* add tests

* explicilty ignore strict mode for conditional updates

* rabbit review fix

* Also assert vector element before, confirm we don't normalize

* Simplify filter creation using new helper

* Rename merge_with_ids to with_point_ids

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
2025-08-15 15:47:43 +02:00
Ivan Pleshkov
0d1b0ead53 Use PointOffsetType in quantization (#7058)
* Use PointOffsetType in quantization

* review remarks

* review remarks
2025-08-14 16:16:08 +02:00
Tim Visée
60cf81fa64 Bump dev version to v1.15.4-dev (#7062) 2025-08-14 15:57:43 +02:00
Tim Visée
6bc846cfec Fix clippy warnings when compiling without RocksDB feature flag (#7059) 2025-08-14 14:25:57 +02:00
Andrey Vasnetsov
4a3008e77a Fix bm25 params (#7056)
* make bm25 parameters compatible with default fastembed params

* bm25 params in openapi schema

* fmt

* improve bm25 config deserialization

* more explicit docstring for disabling english
2025-08-14 14:23:29 +02:00
Luis Cossío
bd448eb580 make Gridstore::iter iterate by owned values (#7045) 2025-08-14 10:54:39 +02:00
Tim Visée
0deb1a40de In payload indices, make storage non optional (#7041)
* Make numeric index storage non optional

* Make map index storage non optional

* Make full text index storage non optional

* Remove unnecessary option

* Make geo index storage non optional

* Make bool index storage non optional

* Remove unnecessary result

* Make null index storage non optional
2025-08-14 09:52:46 +02:00
Tim Visée
e15fb6c90d Remove load function in payload indices (#7040)
* Remove load functions from numeric index

* Remove load functions from map index

* Remove unnecessary result and option

* Remove load functions from full text index

* Remove load functions from geo index

* Remove unnecessary result and option

* Remove load functions from bool index

* Remove load functions from null index

* Don't invoke load when loading payload indices, remove load from trait
2025-08-14 09:18:11 +02:00
Tim Visée
3e09dbb83a Open and load bool payload indices in single stage (#7039)
* Open and load simple bool index in single stage

* Open and load mutable bool index in single stage

* Open and load mutable null index in single stage

* Refactor index selector, don't return early
2025-08-13 17:37:54 +02:00
Tim Visée
c3b5fd0e87 Open and load geo payload indices in single stage (#7038)
* Open and load mutable geo index in single stage

* Open and load immutable geo index in single stage

* Open and load mmap geo index in single stage
2025-08-13 16:59:53 +02:00
Ivan Pleshkov
7eb562e891 Quantization flusher (#7046)
* quantization flusher

* review remarks
2025-08-13 12:57:16 +02:00
Jojii
152bc5ef10 Improve h-sum in dot_avx calculation (#7002)
* Reduce amount of h-sum calculations in dot_avx

* Improve HSUM calculation and apply 4way-hsum to other places
2025-08-13 12:21:27 +02:00
Guglielmo
c22a58264a fix criterion sparse_index_build #6826 (#7044)
sparse_index_build: change config type from ImmutableRam to MutableRam (#6826)
2025-08-13 12:09:36 +02:00
Kacper Łukawski
c5452933a2 Improve description of full_scan_threshold (#7033)
* Improve description of full_scan_threshold in configuration

* Update OpenAPI spec

* edit in grpc

* coderabbit nit

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-08-13 12:02:40 +02:00
Tim Visée
190f6f1ae6 Open and load full text payload indices in single stage (#7004)
* Open and load mutable full text index in single stage

* Open and load immutable full text index in single stage

* Open and load mmap full text index in single stage
2025-08-13 10:53:34 +02:00
Tim Visée
33d8d1135c Open and load map payload indices in single stage (#7003)
* Open and load mutable map index in single stage

* Open and load immutable map index in single stage

* Open and load mmap map index in single stage
2025-08-13 10:32:05 +02:00
Tim Visée
e0fc16db82 Open and load numeric payload indices in single stage (#6999)
* In struct payload index open, use option to specify loaded, set rebuild

* Open and load mutable numeric index in single stage

* Open and load immutable numeric index in single stage

* Open and load mmap numeric index in single stage

* new_null_index always returns null index

Missing refactor for MutableNullIndex. Since we have the
`create_if_missing` flag, we no longer have `open_if_exists` function,
ergo, signature for `new_null_index` should not include an `Option`.

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-08-13 10:08:59 +02:00
Ivan Pleshkov
1616806b8f BQ features without vector stats (#7009)
* bq all features without vector stats

* fix tests

* review suggestions

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-08-12 14:58:43 +02:00
Tim Visée
8c1a01724f Bump dev version to v1.15.3-dev (#7014) 2025-08-12 13:45:34 +02:00
Arnaud Gourlay
b4b2624e2c Update actix-web-validator 0.7.0 (#7017) 2025-08-12 13:45:20 +02:00
Ivan Pleshkov
d31cd696bb wrap quantization chunked vectors (#7011) 2025-08-12 13:30:54 +02:00
Ivan Pleshkov
a3e457d387 Wrap quantization Vec<u8> in a separate structure (#7013)
* separate struct for vec u8 quantization storage

* cgf testing for test storage
2025-08-12 13:20:48 +02:00
dependabot[bot]
39849018b7 build(deps): bump io-uring from 0.7.8 to 0.7.9 (#7027)
Bumps [io-uring](https://github.com/tokio-rs/io-uring) from 0.7.8 to 0.7.9.
- [Commits](https://github.com/tokio-rs/io-uring/commits)

---
updated-dependencies:
- dependency-name: io-uring
  dependency-version: 0.7.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-12 08:07:07 +02:00
dependabot[bot]
b264f62961 build(deps): bump sysinfo from 0.36.1 to 0.37.0 (#7029)
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.36.1 to 0.37.0.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.36.1...v0.37.0)

---
updated-dependencies:
- dependency-name: sysinfo
  dependency-version: 0.37.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-12 08:06:08 +02:00
dependabot[bot]
de64a6beff build(deps): bump roaring from 0.11.1 to 0.11.2 (#7030)
Bumps [roaring](https://github.com/RoaringBitmap/roaring-rs) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/RoaringBitmap/roaring-rs/releases)
- [Commits](https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.1...v0.11.2)

---
updated-dependencies:
- dependency-name: roaring
  dependency-version: 0.11.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-12 08:02:43 +02:00
xzfc
079ef6fb13 Implement GraphLinksFormat::CompressedWithVectors (#6982)
* Simplify graph_links tests

* Rstest-ify hnsw_index::tests::test_save_load

* Rename EncodedVectorsU8::{get_quantized_vector => get_quantized_vector_offset_and_code}

Reason: it was inconsistent with other implementations of
get_quantized_vector().

* Add QuantizedVectors::{get_quantized_vectors, get_quantized_vector_layout}

* Rename `links_map` to `for_each_link`

* pack_links: expose the updated `raw_links` order

* TestRawScorerProducer: support quantization

* Make GraphLinksSerializer::new return Result

* Rename links/compressed_links into `neighbors`

* Implement GraphLinksFormat::CompressedWithVectors

* GraphLayersWithVectors

* bitpacking: add packed_links_size

* Reorder neighbors: N__VVVVVVVcccccccccc -> Ncccccccccc__VVVVVVV
2025-08-11 17:28:43 +00:00
Tim Visée
d748f4cf57 Show unique index types in error message (#6998) 2025-08-07 16:48:39 +02:00
Tim Visée
1bba3891ec Fix clippy warnings for Rust 1.89 (#6997) 2025-08-07 14:34:52 +02:00
Kumar Shivendu
34e18eb22c Allow retaining more closed segments (#6976)
* Retain more closed segments

* Add back rocksdb

* Update WalOptions/WalConfig across the code

* Use NonZeroUsize

* Expose via APIs

* Update gRPC docs

* Fix stoarge compat test

* recompile openapi.json with rocksdb

* default wal retain closed fn

* update openapi.json

* Use qdrant/wal latest commit and remove from config.yaml
2025-08-07 13:44:26 +02:00
Luis Cossío
0a1a3d620b Use BitvecFlags in vector storages (#6994)
* put all flags impls in a single module

* use in appendable_dense_vector_storage

* use in appendable_mmap_multi_dense_storage

* use in mmap_sparse_vector_storage

* clippy

* Add comment we cannot rely on length of deleted vectors bitvec

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-07 11:54:14 +02:00
Luis Cossío
6344ab82ac BitvecFlags: bitvec container with dynamic mmap flags storage (#6990)
* separate flags buffering and in-memory container

* new bitvec flags

* fmt

* clippy

* make BufferedDynamicFlags public to crate only

* clear cache after loading

* clippyyy

* improve comment

* Use resize with a constant to eliminate a hot closure

* Add comment on why we take subslice of bitslice

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-07 11:30:42 +02:00
Roman Titov
8be5e1b07f Refactor Collection::restore_shard_snapshot (#6988) 2025-08-06 17:48:04 +02:00
Roman Titov
a08c89082c Track payload index schema version (#6819) 2025-08-06 16:35:43 +02:00
Roman Titov
563194febf Fix test_create_shard_key_read_availability flakiness (#6902) 2025-08-06 16:14:13 +02:00