upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-06 10:00:58 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,465 Commits 344 Branches 118 Tags
94de0e2d2dcf997bf26bec34dad407d8de9a9abb
Commit Graph

55 Commits

Author SHA1 Message Date
xzfc
4cabb7fd8e Merge io and memory into common (#8155) 
* Unify parking_lot/arc_lock feature

* Move lib/common/{io,memory}/* -> lib/common/common/*

- Mmap-related items are grouped into `common::mmap` sub-module:
  - `memory/src/chunked_utils.rs`      -> `common/src/mmap/chunked.rs`
  - `memory/src/madvise.rs`            -> `common/src/mmap/advice.rs`
  - `memory/src/mmap_ops.rs`           -> `common/src/mmap/ops.rs`
  - `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
  - `memory/src/mmap_type.rs`          -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
  - `common/src/fs.rs`          -> `common/src/fs/sync.rs`
  - `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
  - `io/src/move_files.rs`      -> `common/src/fs/move.rs`
  - `io/src/safe_delete.rs`     -> `common/src/fs/safe_delete.rs`
  - `memory/src/checkfs.rs`     -> `common/src/fs/check.rs`
  - `memory/src/fadvise.rs`     -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
  - `io/src/storage_version.rs` -> `common/src/storage_version.rs`

The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.

* Replace uses of `io` and `memory` with new paths in `common`

Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.

* Remove `io` and `memory` crates
 2026-02-17 10:58:59 +01:00
xzfc
6bced54ca1 Chore: promote dependencies to workspace level (#8061) 
* Promote `env_logger` to workspace dependency

* Promote `anyhow` to workspace dependency

* Promote `rmp-serde` to workspace dependency

* Promote `tinyvec` to workspace dependency

* Promote `async-trait` to workspace dependency

* Promote `url` to workspace dependency

* Promote `self_cell` to workspace dependency

* Promote `cc` to workspace dependency

* Promote `bitpacking` to workspace dependency
 2026-02-05 16:02:28 +01:00
dependabot[bot]
4370fbee22 build(deps): bump bitpacking from 0.9.2 to 0.9.3 (#7935) 
Bumps [bitpacking](https://github.com/quickwit-oss/bitpacking) from 0.9.2 to 0.9.3.
- [Release notes](https://github.com/quickwit-oss/bitpacking/releases)
- [Changelog](https://github.com/quickwit-oss/bitpacking/blob/master/CHANGELOG.md)
- [Commits](https://github.com/quickwit-oss/bitpacking/commits)

---
updated-dependencies:
- dependency-name: bitpacking
  dependency-version: 0.9.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-01-20 11:29:03 +01:00
Luis Cossío
d79cd9b99b activate fs-err features for nicer error messages (#7665) 
* activate fs-err features for nicer error messages

* use in entire workspace
 2025-12-02 10:40:36 -03:00
xzfc
a0d62330c7 Use fs-err (#7319) 2025-09-29 12:47:10 +00:00
Arnaud Gourlay
4b74671377 Move Bincode to workspace and comment version lock (#6151) 2025-03-11 12:16:35 +01:00
Tim Visée
3e536347e1 Bump Rust edition to 2024 (#6042) 
* Bump Rust edition to 2024

* gen is a reserved keyword now

* Remove ref mut on references

* Mark extern C as unsafe

* Wrap unsafe function bodies in unsafe block

* Geo hash implements Copy, don't reference but pass by value instead

* Replace secluded self import with parent

* Update execute_cluster_read_operation with new match semantics

* Fix lifetime issue

* Replace map_or with is_none_or

* set_var is unsafe now

* Reformat
 2025-02-25 11:21:25 +01:00
Luis Cossío
5d9d5bd3f9 Rename blob_store as gridstore (#5918) 
* rename blob_store into gridstore

Article is incoming and I feel like gridstore is a better fit to the
architecture

* fmt

* remove outdated TODO
 2025-01-30 14:56:34 -03:00
Jojii
e0d0e233d8 Timeout aware hardware counter (#5555) 
* Make hardware counting timeout aware

* improve test

* rebuild everything

* fmt

* post-rebase fixes

* upd tests

* fix tests

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-12-10 12:12:36 +01:00
Luis Cossío
2de122adea Compatibility for mmap sparse vectors (#5454) 
* implement mmap sparse vector storage

* add to VectorStorageEnum

* clippy

* add tests, fix both simple and mmap storages

* smol correction on total_vector_count

* add sparse storage type to config

* fix reading config without storage type

* generate openapi

* use blob_store by path

* hidden setting to enable new storage

* validate existing path in `BlobStore::open()`

* use new dir for each sparse vector name

* fix and rename `max_point_offset`

Plus some extra refactors

* add storage compat test, to always check both storages work

* fix opening of storage + other misc fixes

* FIX!!!

`Unset` operations in the Tracker weren't updating the
`next_pointer_id`. So, when reopening the storage, those points wouldn't
get marked as deleted in the bitslice, thus creating the illusion that
they should exist, when they did not.

* refactor naming from `iter_*` to `for_each_*`

* fix checking for BlobStore existance

* fix typo

* fix error message

* better docs for open_or_create

* fix after rebase
 2024-11-27 11:45:16 -06:00
Arnaud Gourlay
6f1ef42907 Promote criterion to workspace (#5291) 2024-10-22 16:29:23 +02:00
Arnaud Gourlay
90cd35a196 Promote ordered-float to workspace (#5285) 2024-10-22 10:45:38 +02:00
dependabot[bot]
1558e7dfcd build(deps): bump ordered-float from 4.3.0 to 4.4.0 (#5277) 
Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 4.3.0 to 4.4.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v4.3.0...v4.4.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-10-22 08:52:40 +02:00
Arnaud Gourlay
b6a58c9bef Downgrade asserts into logs for sparse vector inconsistencies (#5246) 2024-10-16 10:56:31 +02:00
dependabot[bot]
52db80a292 build(deps): bump ordered-float from 4.2.2 to 4.3.0 (#5154) 
Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 4.2.2 to 4.3.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v4.2.2...v4.3.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-10-01 09:19:46 +02:00
xzfc
a0ea3caccf Enable some of the pedantic clippy lints (#4715) 
* Use workspace lints

* Enable lint: manual_let_else

* Enable lint: enum_glob_use

* Enable lint: filter_map_next

* Enable lint: ref_as_ptr

* Enable lint: ref_option_ref

* Enable lint: manual_is_variant_and

* Enable lint: flat_map_option

* Enable lint: inefficient_to_string

* Enable lint: implicit_clone

* Enable lint: inconsistent_struct_constructor

* Enable lint: unnecessary_wraps

* Enable lint: needless_continue

* Enable lint: unused_self

* Enable lint: from_iter_instead_of_collect

* Enable lint: uninlined_format_args

* Enable lint: doc_link_with_quotes

* Enable lint: needless_raw_string_hashes

* Enable lint: used_underscore_binding

* Enable lint: ptr_as_ptr

* Enable lint: explicit_into_iter_loop

* Enable lint: cast_lossless
 2024-07-22 08:19:19 +00:00
Arnaud Gourlay
d5fc02149e remove unused dependencies (#4645) 2024-07-10 09:34:21 +02:00
xzfc
90ce6cbd01 Add MmapHashMap (#4612) 
* Promote some deps to workspace deps

* Add MmapHashMap

* Review fixups

* Use tempfile for atomicity
 2024-07-08 17:44:52 +00:00
Arnaud Gourlay
658a0beced Promote Rand dependency to workspace (#4606) 2024-07-03 13:11:11 +02:00
Arnaud Gourlay
e1b2af1a86 Promote log dep to workspace (#4605) 2024-07-03 12:35:04 +02:00
xzfc
d2ef556237 Make sparse vectors generic (#4364) 
* Make sparse vector generic

* Generic search_context tests
 2024-06-10 15:04:32 +00:00
dependabot[bot]
6ebd22d487 Bump parking_lot from 0.12.2 to 0.12.3 (#4337) 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.12.2 to 0.12.3.
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.12.2...0.12.3)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-05-28 09:02:20 +02:00
Arnaud Gourlay
cbcdefb310 Cleanup unused dependencies (#4306) 2024-05-23 10:06:44 +02:00
xzfc
93a42b4e2d Compressed posting lists (#4253) 2024-05-22 18:18:20 +00:00
xzfc
ccf7f1d24f InvertedIndexImmutableRam and index migrations (#4220) 
* Move StorageVersion from segment crate to common/io

* Refine StorageVersion API

* Move methods from SparseVectorDataConfig to enum SparseIndexType

* Introduce InvertedIndexImmutableRam

* Add migrate

* Don't migrate
 2024-05-16 06:47:47 +00:00
Luis Cossío
bfa72bb6d8 universal-query: Add query() to ShardOperation trait (#4210) 
* add `query` to shard trait

* add missing conversions for query

* update grpc docs

* Query response has intermediate results

* add ShardQueryResponse description

* move pub use to the top, keep only one way of reaching reexports
 2024-05-15 09:36:46 -04:00
xzfc
6f738ca95f Update sparse vector benchmarks (#4163) 2024-05-07 20:34:37 +00:00
dependabot[bot]
699d83036d Bump parking_lot from 0.12.1 to 0.12.2 (#4136) 2024-04-29 22:45:18 +02:00
Arnaud Gourlay
4ff2fb855b promote Validator to workspace (#4013) 2024-04-11 12:01:27 +02:00
dependabot[bot]
1c1b74ec99 Bump tempfile from 3.10.0 to 3.10.1 (#3763) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.10.0 to 3.10.1.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.10.0...v3.10.1)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-05 10:39:48 +01:00
Arnaud Gourlay
ab90b03aa0 Remove unused proptest on Sparse crate (#3667) 2024-02-22 11:35:22 +01:00
dependabot[bot]
b7165b5a81 Bump tempfile from 3.9.0 to 3.10.0 (#3596) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.9.0 to 3.10.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.9.0...v3.10.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-02-13 08:22:15 +01:00
Arnaud Gourlay
d7d5c9bb43 Pool larger memory batches for sparse vector search (#3560) 
* Pool larger memory batches for sparse vector search

* rework naming

* refine naming

* address great comments

* bump ADVANCE_BATCH_SIZE to use power of 2

* 16k performs bad - trying 8k

* 10k offers the best tradeoff
 2024-02-12 14:58:03 +01:00
Roman Titov
b8627c5e8e Add clock_tag field to update operations (#3408) 2024-02-01 11:44:13 +01:00
Tim Visée
06712e8f4d Move common dependencies to workspace (#3490) 2024-01-31 09:01:38 +01:00
dependabot[bot]
1fc3a22f75 Bump itertools from 0.12.0 to 0.12.1 (#3486) 
Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.12.0 to 0.12.1.
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.12.0...v0.12.1)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-30 10:30:07 +01:00
dependabot[bot]
8e5587c390 Bump memmap2 from 0.9.3 to 0.9.4 (#3484) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.3 to 0.9.4.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.3...v0.9.4)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-30 09:42:42 +01:00
Arnaud Gourlay
a20bb6ce5a Remove unused dependencies (#3435) 2024-01-22 15:36:38 +01:00
dependabot[bot]
4b08e7310e Bump tempfile from 3.8.1 to 3.9.0 (#3353) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.8.1 to 3.9.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.8.1...v3.9.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-09 12:17:26 +01:00
dependabot[bot]
c1210186b0 Bump memmap2 from 0.9.2 to 0.9.3 (#3283) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.2 to 0.9.3.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.2...v0.9.3)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-02 11:36:14 +01:00
dependabot[bot]
71fb667c93 Bump ordered-float from 4.1.1 to 4.2.0 (#3242) 
Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 4.1.1 to 4.2.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v4.1.1...v4.2.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-19 20:08:01 +01:00
dependabot[bot]
f1de3b9586 Bump memmap2 from 0.9.0 to 0.9.2 (#3239) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.0 to 0.9.2.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.0...v0.9.2)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-19 16:00:45 +00:00
Arnaud Gourlay
8cad65b218 Fix validation of unicity of sparse index (#3072) 2023-11-21 12:08:51 +01:00
dependabot[bot]
c9b9646432 Bump schemars from 0.8.15 to 0.8.16 (#3058) 
Bumps [schemars](https://github.com/GREsau/schemars) from 0.8.15 to 0.8.16.
- [Release notes](https://github.com/GREsau/schemars/releases)
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.15...v0.8.16)

---
updated-dependencies:
- dependency-name: schemars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-21 08:21:30 +01:00
Ivan Pleshkov
ee23eda42c Sparse vectors api preparations (#3038) 
* Sparse vectors api preparations

generic transpose

remove features

unnecessary local variable

add snake case

* distance in config function
 2023-11-17 16:22:10 +01:00
dependabot[bot]
629e71c08d Bump memmap2 from 0.7.1 to 0.9.0 (#2999) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.7.1 to 0.9.0.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.7.1...v0.9.0)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-14 08:59:46 +01:00
dependabot[bot]
7331952054 Bump tempfile from 3.8.0 to 3.8.1 (#2996) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.8.0 to 3.8.1.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/commits)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-14 07:19:29 +01:00
Arnaud Gourlay
bc5d0976a1 Sparse vector validation (#2965) 
* Sparse vector validation

* cheaper indices uniqueness validation and tests
 2023-11-10 16:42:10 +01:00
Arnaud Gourlay
ec2dc36719 SparseVectorIndex implements VectorIndex (#2900) 
* SparseVector implements VectorIndex

* dedicated telemetry

* conflict

* easy code reviews

* simplify tracking indexed points count

* move telemetry conversion to sparse file

* move max_result_count out of inverted index trait

* unify sparse vector fixtures

* simpler conversion

* add todo regarding OOM potential

* reuse check deleted from raw scorer with TODO

* change new to open to handle mmap index
 2023-11-03 16:23:05 +01:00
Arnaud Gourlay
0fa6688764 Document and test underfetching with sparse vectors (#2908) 
* Document and test underfetching with sparse vectors

* deduplicate test point ids
 2023-11-02 15:30:43 +01:00