Commit Graph

5465 Commits

Author SHA1 Message Date
Roman Titov
6fa24e1ee3 Move SerdeWal from collection into shard crate (#7146) 2025-08-26 16:27:35 +02:00
Arnaud Gourlay
91036211b1 Bump Tokio 1.47.1 (#7148) 2025-08-26 16:01:20 +02:00
dependabot[bot]
9b1bbac534 build(deps): bump url from 2.5.4 to 2.5.7 (#7144)
Bumps [url](https://github.com/servo/rust-url) from 2.5.4 to 2.5.7.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/commits)

---
updated-dependencies:
- dependency-name: url
  dependency-version: 2.5.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 13:20:57 +02:00
Andrey Vasnetsov
e744bd6014 implement custom collection metadata (#7123)
* implement custom collection metadata

* persist metadata change

* Also unset a key in the test

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-26 11:49:17 +02:00
dependabot[bot]
738e31aa9b build(deps): bump memmap2 from 0.9.7 to 0.9.8 (#7143)
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.7 to 0.9.8.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.7...v0.9.8)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-version: 0.9.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 11:23:43 +02:00
dependabot[bot]
d5f1c726a4 build(deps): bump cc from 1.2.33 to 1.2.34 (#7137)
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.2.33 to 1.2.34.
- [Release notes](https://github.com/rust-lang/cc-rs/releases)
- [Changelog](https://github.com/rust-lang/cc-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/cc-rs/compare/cc-v1.2.33...cc-v1.2.34)

---
updated-dependencies:
- dependency-name: cc
  dependency-version: 1.2.34
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 11:14:18 +02:00
dependabot[bot]
5e7f97ed0f build(deps): bump serde_json from 1.0.142 to 1.0.143 (#7138)
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.142 to 1.0.143.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.142...v1.0.143)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-version: 1.0.143
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 10:43:17 +02:00
dependabot[bot]
bef3e88340 build(deps): bump tinyvec from 1.9.0 to 1.10.0 (#7139)
Bumps [tinyvec](https://github.com/Lokathor/tinyvec) from 1.9.0 to 1.10.0.
- [Changelog](https://github.com/Lokathor/tinyvec/blob/main/CHANGELOG.md)
- [Commits](https://github.com/Lokathor/tinyvec/compare/v1.9.0...v1.10.0)

---
updated-dependencies:
- dependency-name: tinyvec
  dependency-version: 1.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 10:43:09 +02:00
dependabot[bot]
3ba8cd70b1 build(deps): bump tempfile from 3.20.0 to 3.21.0 (#7141)
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.20.0 to 3.21.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/commits)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-version: 3.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 10:42:56 +02:00
dependabot[bot]
0c937a3801 build(deps): bump io-uring from 0.7.9 to 0.7.10 (#7142)
Bumps [io-uring](https://github.com/tokio-rs/io-uring) from 0.7.9 to 0.7.10.
- [Commits](https://github.com/tokio-rs/io-uring/commits)

---
updated-dependencies:
- dependency-name: io-uring
  dependency-version: 0.7.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 10:34:39 +02:00
dependabot[bot]
8434547690 build(deps): bump charabia from 0.9.6 to 0.9.7 (#7140)
Bumps [charabia](https://github.com/meilisearch/charabia) from 0.9.6 to 0.9.7.
- [Release notes](https://github.com/meilisearch/charabia/releases)
- [Commits](https://github.com/meilisearch/charabia/compare/v0.9.6...v0.9.7)

---
updated-dependencies:
- dependency-name: charabia
  dependency-version: 0.9.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 10:34:29 +02:00
dependabot[bot]
98f77ace86 build(deps): bump thiserror from 2.0.15 to 2.0.16 (#7136)
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 2.0.15 to 2.0.16.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/2.0.15...2.0.16)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-version: 2.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 10:11:47 +02:00
Andrey Vasnetsov
55b0f86122 restore missed index laoding (#7133) 2025-08-26 09:48:52 +02:00
xzfc
937cd477ac Misc fixes (#7134)
* PrimitiveVectorElement: use zerocopy

* PrimitiveVectorElement: also use zerocopy::FromBytes

* serialize_graph_links: back_index is Vec<PointOffsetType>

* for_each_link_with_vector: loosen lifetimes
2025-08-26 09:43:37 +02:00
dependabot[bot]
579fb4ef85 build(deps): bump indexmap from 2.10.0 to 2.11.0 (#7135)
Bumps [indexmap](https://github.com/indexmap-rs/indexmap) from 2.10.0 to 2.11.0.
- [Changelog](https://github.com/indexmap-rs/indexmap/blob/main/RELEASES.md)
- [Commits](https://github.com/indexmap-rs/indexmap/compare/2.10.0...2.11.0)

---
updated-dependencies:
- dependency-name: indexmap
  dependency-version: 2.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 09:36:46 +02:00
xzfc
0bcdcffb69 Clean up GraphLayers::search_entry_on_level a bit (#6946)
review remarks

are you happy fmt

don't do unnecessary scoring
2025-08-25 19:07:35 +02:00
Arnaud Gourlay
0781ea8893 Segment holder has deterministic segment iteration (#7131) 2025-08-25 15:40:21 +02:00
Arnaud Gourlay
1e81d4f3b3 Make points upsert ordering deterministic (#7130)
* Make points upsert ordering deterministic

* use Iterator magic (thx Tim)
2025-08-25 14:49:03 +02:00
Arnaud Gourlay
db2489ba9f Test for segment double proxies (#7127) 2025-08-25 14:20:33 +02:00
Arnaud Gourlay
59257681c9 Log optimizer name running (#7129) 2025-08-25 14:20:14 +02:00
Kumar Shivendu
3b1ef64ebd fix dockerhub username in builds (#7128)
* fix dockerhub username in builds

* avoid explicitly setting dockerhub url

* Use --password-stdin instead of --password
2025-08-25 16:35:03 +05:30
Ivan Pleshkov
a2089f6a71 chunked mmap as a quantization storage (#7116) 2025-08-25 12:27:29 +02:00
Bastian Hofmann
4e8e9daf91 Sign container images with cosign (#7119)
* Sign container images with cosign

This allows users to securely verify that a Qdrant container image was created by us

* Add contents read permission
2025-08-25 11:45:23 +02:00
xzfc
f2ce816fbe Add memory::madvise::will_need_multiple_pages() (#7124) 2025-08-25 09:22:11 +00:00
xzfc
902916f524 Remove unused GraphLinks::on_disk method (#7125) 2025-08-25 11:19:18 +02:00
Andrey Vasnetsov
5235d86435 measure disk size of directories if available (#7120)
* measure disk size of directories if available

* review suggestions

* Update comment

---------

Co-authored-by: xzfc <xzfcpw@gmail.com>
2025-08-25 00:17:27 +02:00
Andrey Vasnetsov
bf60e96619 filter text any (#7100)
* internal implementation for filter any

* implement api for match `text_any`

* allow naive implementation of text match any

* dedup after kmerge + renames

* improve `expected_should_estimation`

* congruence test

* openapi test

* setup index in each test

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-08-24 12:09:05 +02:00
xzfc
95d7936687 Graph links serializer: avoid large intermediate allocations (#7118)
* Introduce io::file_operations::atomic_save

* Replace GraphLinksSerializer with serialize_graph_links
2025-08-22 17:08:35 +00:00
Ivan Pleshkov
879da486af Remove quantization load from trait (#7113)
* remove quantization load

* refactor load

* fix fmt error

* fix clippy
2025-08-22 14:37:32 +02:00
Kumar Shivendu
ad08d64372 Push dev branch images to dockerhub (#7107)
* Push dev branch images to dockerhub

* temporarily update dockerhub push condition

* Temporarily disable dockerhub push

* Suggestions

* Add back code to push to dockerhub

* Use github.ref_name instead of github.ref.name
2025-08-22 17:51:40 +05:30
Roman Titov
bfdc779203 Move SegmentHolder into shard crate (#7115) 2025-08-22 12:39:06 +02:00
Jojii
4012ea9e51 Mirgate from old shard key format on load (#7110)
* Migrate shard-key format

* Reduce visibility of was_old_format attribute

* Add debug message when we migrate persisted shard key format

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-22 10:47:28 +02:00
Roman Titov
5eb40111a5 Move UpdateTracker from SegmentHolder into UpdateHandler/CollectionUpdater (#7112) 2025-08-21 15:56:14 +02:00
Tim Visée
e02d3dcb38 In proxy segment, don't return deleted point versions (#7109)
* In proxy segment, don't return deleted point versions

More specifically, don't return a point version from the wrapped segment
if that point is already marked for deletion within the proxy.

* Do get wrapped segment point version if it's newer than soft delete

* Flatten function
2025-08-21 15:48:18 +02:00
Bastian Hofmann
49c25a9cc5 Generate SBOM during Docker build (#7111) 2025-08-21 13:25:23 +02:00
Luis Cossío
a2d3270eb8 Custom RRF k parameter (#7065)
* allow custom K parameter for RRF

* generate grpc docs and openapi

* use tagged type approach for parametrized fusions

* use params approach in grpc

* simplify api structure

* upd schema

* nits

* rest: parameterized rrf as query variant

* consistency in doc comments

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2025-08-20 15:05:27 -04:00
Roman Titov
5d826c583d Move PayloadIndexSchema into shard crate (#7108) 2025-08-20 18:58:03 +02:00
Roman Titov
0bfa8f61d5 Move SaveOnDisk into common crate (#7105) 2025-08-20 17:06:47 +02:00
Tim Visée
1d97e9a415 Keep segment ID mapping when proxying all segments (#7106)
* When swapping existing segment, remove first to allow segment ID reuse

* Share original segment ID with proxies, keeps original mapping

* Add replace function, that directly replaces existing segment
2025-08-20 15:18:06 +02:00
Ivan Pleshkov
f3b61bbf4a Retrieve files list from quantization storage (#7095)
* Get files list from quantization

* fix ci
2025-08-20 14:15:09 +02:00
Roman Titov
933c7c8ac1 Refactor ShardHolder to use SegmentConfig instead of CollectionParams (#7098) 2025-08-20 14:01:04 +02:00
Roman Titov
8a84c1abb0 Remove async from SegmentHolder::deduplicate_points (#7094) 2025-08-20 13:26:16 +02:00
Ivan Pleshkov
cb6958f56c using quantization in plain segment (#7060) 2025-08-20 13:08:58 +02:00
Arnaud Gourlay
f4b4c6c503 Bump dev deps (#7104)
* bump criterion 0.7.0

* bump rstest 0.26.1

* bump pprof 0.15.0

* remove unused bitvec

* bump prettyplease 0.2.37
2025-08-20 12:43:58 +02:00
dependabot[bot]
599a6e42ad build(deps): bump anyhow from 1.0.98 to 1.0.99 (#7099) 2025-08-20 10:07:30 +00:00
Arnaud Gourlay
bd429e9a96 bump test_get_storage_size value to accommodate MacOS change (#7103) 2025-08-20 11:40:41 +02:00
Tim Visée
2b4f19f26d Fix new shard crate always pulling in RocksDB dependencies (#7097) 2025-08-19 17:25:18 +02:00
Roman Titov
60e0c07b0a Move ProxySegment from collection into shard crate (#7093) 2025-08-19 14:01:43 +02:00
dependabot[bot]
f2c991a835 build(deps): bump syn from 2.0.104 to 2.0.106 (#7083)
Bumps [syn](https://github.com/dtolnay/syn) from 2.0.104 to 2.0.106.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/2.0.104...2.0.106)

---
updated-dependencies:
- dependency-name: syn
  dependency-version: 2.0.106
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-19 10:22:09 +02:00
dependabot[bot]
3d9da28423 build(deps): bump debian from 12-slim to 13-slim (#7081)
Bumps debian from 12-slim to 13-slim.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: 13-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-19 09:46:06 +02:00