upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-10-01 18:37:42 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,198 Commits 451 Branches 119 Tags
2aff9fc8a3a93c1ead02bdda7bb6bca4e09efe48
Commit Graph
364 Commits
Author SHA1 Message Date
Ivan Pleshkov 2aff9fc8a3 vector index as enum 2023-03-08 21:02:59 +00:00
Ivan Pleshkov 41d5ecd63b separate payload and vector index bases 2023-03-08 20:00:08 +00:00
Ivan PleshkovandAndrey Vasnetsov 5174388ab4 integrate quantized data to storages (#1311) 
* integrate quantized data to storages

* revert gitignore

* are you happy clippy

* quantize in optimizer

* provide flag

* fix segfault

* skip quantization flag, update scores

* use quantization flag

* are you happy fmt

* use quantization flag

* quantized search test

* are you happy fmt

* refactor test, refactor scorer choosing

* are you happy fmt

* run quantization on segment builder

* decrease testing parameters

* simplify segment

* update version

* remove use_quantization flag

* provide quantization config

* quantization version up

* euclid dist

* add euclid test

* saveload

* fix initialization bugs

* quantization lib version up

* fix arm build

* refactor scorer selecting

* quant lib version up

* are you happy fmt

* are you happy fmt

* are you happy clippy

* add save/load test for simple storage

* add comments

* quantiles

* quantization mmap

* remove f32

* mmap test

* fix mmap slice

* fix mmap test

* use chunks for quantization storage

* fix build

* are you happy fmt

* update quantization library

* update quantization lib

* update quantization lib

* integrate api changes

* are you happy fmt

* change quantization api

* additional checks in tests

* update quantization version

* fix unit tests

* add quantization to storage config

* use quantization for all cardinality search cases

* Integrate quantization suggestions 2 (#1520)

* review api

* wip: refactor quantization integrations

* wip: refactor quantization integrations

* wip: fmt

* include quantization into snapshot

* fmt

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-03 17:46:17 +01:00
Ivan Pleshkov 83efed1fcf fix windows warning unused parallel_graph_build (#1513) 2023-03-01 11:10:22 +01:00
dependabot[bot] b5ded897a2 Bump atomicwrites from 0.3.1 to 0.4.0 (#1511) 
Bumps [atomicwrites](https://github.com/untitaker/rust-atomicwrites) from 0.3.1 to 0.4.0.
- [Release notes](https://github.com/untitaker/rust-atomicwrites/releases)
- [Commits](https://github.com/untitaker/rust-atomicwrites/compare/0.3.1...0.4.0)

---
updated-dependencies:
- dependency-name: atomicwrites
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-03-01 08:35:21 +01:00
dependabot[bot] d6e6d75ff1 Bump schemars from 0.8.11 to 0.8.12 (#1505) 
Bumps [schemars](https://github.com/GREsau/schemars) from 0.8.11 to 0.8.12.
- [Release notes](https://github.com/GREsau/schemars/releases)
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.11...v0.8.12)

---
updated-dependencies:
- dependency-name: schemars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-28 09:51:03 +01:00
dependabot[bot] b419a5929b Bump tempfile from 3.3.0 to 3.4.0 (#1506) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases)
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS)
- [Commits](https://github.com/Stebalien/tempfile/commits)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-28 09:50:24 +01:00
Arnaud Gourlay ea485a9029 Fix condition check for values_count (#1502) 2023-02-28 09:47:28 +01:00
Arnaud GourlayandAndrey Vasnetsov 2770a24766 Nested payload filters (#1487) 
* Nested payload filters

* close ToDo + add parsing of multuiple array values

* fmt

* improve testing nested arrays

* fix NumericIndex to accumulate points_to_values mapping

* revert numberic index + strict array field access

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-02-23 15:57:12 +01:00
dependabot[bot] a2ff17dc7f Bump memmap2 from 0.5.9 to 0.5.10 (#1491) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.5.9 to 0.5.10.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases)
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.5.9...v0.5.10)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-23 10:56:20 +01:00
dependabot[bot] 349bdfe40f Bump memmap2 from 0.5.8 to 0.5.9 (#1479) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases)
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.5.8...v0.5.9)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-20 09:48:15 +01:00
Arnaud Gourlay 3803af502d Access payload key through nested Array (#1465) 
* Access payload key through nested Array

* support removal through array index

* propagate type changes

* fmt

* avoid allocating Vec of one element
 2023-02-17 17:41:28 +01:00
Luis CossíoandAndrey Vasnetsov c15fec2a2a feat: add "any" to "match" statements (#1466) 
* feat: add "any" to "match" statements

Adds support for using "any" to match on a list of different possible values

* enable proper counting in cardinality estimation

* fmt

* upd OpenAPI

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-02-15 11:11:20 +01:00
dependabot[bot] 97aaeb100f Bump rocksdb from 0.20.0 to 0.20.1 (#1464) 
Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb) from 0.20.0 to 0.20.1.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases)
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.20.0...v0.20.1)

---
updated-dependencies:
- dependency-name: rocksdb
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-10 20:30:27 +01:00
dependabot[bot]andIvan Pleshkov 1204682000 Bump rocksdb from 0.19.0 to 0.20.0 (#1461) 
* Bump rocksdb from 0.19.0 to 0.20.0

Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb) from 0.19.0 to 0.20.0.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases)
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.19.0...v0.20.0)

---
updated-dependencies:
- dependency-name: rocksdb
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix api changes

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
 2023-02-10 10:22:34 +01:00
Andrey Vasnetsov ac251358b9 Fix flacky windows test (#1448) 
* use async sleep

* fmt

* disablee test_parallel_graph_build on CI on windows untill there is a way to reproduce
 2023-02-09 20:48:00 +01:00
Roman Titov 1be2f67a2e Reimplement ExtendedPointId deserialization (#1453) 
Reimplement `ExtendedPointId` deserialization, to produce better error message when deserialization fails
 2023-02-10 02:36:36 +10:00
Andrey Vasnetsov 26551c64ef rollback custom deserialization for point id 2023-02-08 14:13:57 +01:00
Andrey Vasnetsov 39cc731611 Windows support (#1432) 
* windows-support

* fmt

* install protoc with action

* upd WAL revision
 2023-02-06 14:00:23 +01:00
dependabot[bot] 9f6fccece0 Bump fs_extra from 1.2.0 to 1.3.0 (#1428) 
Bumps [fs_extra](https://github.com/webdesus/fs_extra) from 1.2.0 to 1.3.0.
- [Release notes](https://github.com/webdesus/fs_extra/releases)
- [Commits](https://github.com/webdesus/fs_extra/commits)

---
updated-dependencies:
- dependency-name: fs_extra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-03 19:51:10 +01:00
Andrey Vasnetsov e3066b96fc allow to overwrite max number of cpus with env variable (#1425) 2023-02-02 14:44:03 +01:00
dependabot[bot] 7abb9754b3 Bump uuid from 1.2.2 to 1.3.0 (#1420) 
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.2.2 to 1.3.0.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/1.2.2...1.3.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-01 19:59:41 +01:00
Arnaud Gourlay 68b34b1550 Implement Deserialize for ExtendedPointId to improve error reporting (#1416) 
* Implement Deserialize for ExtendedPointId to improve error reporting

* better error message
 2023-01-31 19:57:39 +01:00
Arnaud Gourlay 4c0f849749 Clippy rust 1.67 (#1406) 
* inline format! args

* inline format! args

* explicit lifetime could be elided

* fmt
 2023-01-26 17:48:52 +01:00
Roman TitovandAndrey Vasnetsov 6b66d33398 Implemented response resolver (#1370, #1381) 
- Implement `Resolve` trait that "merges" multiple `retrieve`/`scroll_by`/`search` responses
  into a single response, to ensure it's consistent across multiple nodes in the cluster
- Implement tests for the `Resolve` trait implementation
- Add a few additional derives required for the `Resolve` trait implementation

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-01-25 05:24:55 +10:00
Arnaud Gourlay 064a1389b9 Name rayon hnsw threads for clarity (#1399) 2023-01-24 19:36:16 +01:00
Andrey Vasnetsov ca9545366d Fix flacky hnsw (#1383) 
* use fixed seed in random generator

* fmt
 2023-01-23 10:10:46 +01:00
Andrey Vasnetsov 5877039380 Faster snapshot creation (#1353) 
* deduplication after proxy snapshot recovery

* revert changes in apply_points_to_appendable

* clippy fixes

* spawn blocking in snapshot creation
 2023-01-20 23:52:45 +01:00
Andrey Vasnetsov a8dad69dba enable payload field anonymization (#1368) 2023-01-19 10:23:17 +01:00
Arnaud Gourlay 8c2c4f08eb Remove some blocking IO calls on Tokio threads (#1366) 2023-01-19 07:58:45 +01:00
dependabot[bot] 0f03526a78 Bump geohash from 0.12.0 to 0.13.0 (#1357) 
Bumps [geohash](https://github.com/georust/geohash.rs) from 0.12.0 to 0.13.0.
- [Release notes](https://github.com/georust/geohash.rs/releases)
- [Commits](https://github.com/georust/geohash.rs/compare/v0.12.0...v0.13.0)

---
updated-dependencies:
- dependency-name: geohash
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-17 08:47:18 +01:00
Andrey Vasnetsov 0c1833db7f Disable extra segments checks (#1352) 
* optional segment check

* alternative implementation
 2023-01-16 12:05:32 +01:00
Roman TitovandAndrey Vasnetsov ced0abb4c7 Fix segment snapshotting (#1321) (#1334) 
* WIP: Fix `Segment::take_snapshot`

TODO:
- This commit, probably, breaks snapshotting of segments with memmapped vector storage
- `ProxySegment::take_snapshot` seems to potentially similar bug

* WIP: Fix `Segment::take_snapshot`

- Fix snapshotting of `StructPayloadIndex`
- Fix snapshotting of segments with memmapped vector storage
- Temporarily break `ProxySegment::take_snapshot`

* Fix `ProxySegment::take_snapshot`

* Remove `copy_segment_directory` test

* nitpicking

* clippy fixes

* use OperationError::service_error

* Cleanup `TinyMap` trait bounds and derive `Debug`

* Fix `test_snapshot` test

- Derive `Debug` for `NamedVectors`

* Move utility functions from `segment.rs` to `utils` module

* Contextualize `segment::utils::fs::move_all` a bit more carefully

* Fix a typo

* add backward compatibility with old snapshot formats

* fmt

* add snapshot for compatibility test

* git lfs is a piece of shit

* Nitpicking

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-01-13 11:44:42 +01:00
Arnaud GourlayandAndrey Vasnetsov 72dadee806 Repair segment data while checking consistency (#1314) 
* Repair segment data while checking consistency

* code review

* code review

* code review

* code review drop_by_internal_id

* refactoring

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-01-10 20:02:43 +01:00
Arnaud Gourlay 1435ea2069 Remove unused json-patch dependency (#1331) 2023-01-09 19:55:01 +01:00
Roman Titov 2ed11d544b Add configuration option to select "memory advice" used for memmapped storage 
- Add a simple abstraction/wrapper over `memmap2::Advice` and a global switch
  to select "memory advice" used for memmapped storage to the `segment` crate
- Add `mmap_advice` configuration option to the `storage::types::StorageConfig`
- Implement `search-points` benchmark
 2023-01-06 21:02:57 +10:00
dependabot[bot] 49bc997e52 Bump atomic_refcell from 0.1.8 to 0.1.9 (#1313) 
Bumps [atomic_refcell](https://github.com/bholley/atomic_refcell) from 0.1.8 to 0.1.9.
- [Release notes](https://github.com/bholley/atomic_refcell/releases)
- [Commits](https://github.com/bholley/atomic_refcell/commits)

---
updated-dependencies:
- dependency-name: atomic_refcell
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-04 20:56:39 +01:00
Andrey Vasnetsov bd05a69a5f Id mapper inconsistency (#1302) 
* always flush wal

* always flush wal fix

* always flush wal fmt

* flush wal during background flush

* async wal flush

* use id-tracker internal id for next-id instead of vector storage

* add flush order and recovery comment

fix merge bug

* longer timeout in test
 2022-12-25 22:36:31 +01:00
Andrey Vasnetsov 431fcbb4ca better validation for transfer initiation 2022-12-23 21:16:13 +01:00
dependabot[bot] 16a4d54885 Bump num_cpus from 1.14.0 to 1.15.0 (#1296) 
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.14.0...v1.15.0)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-20 19:35:10 +01:00
dependabot[bot] 1f90e12eec Bump semver from 1.0.14 to 1.0.16 (#1292) 
Bumps [semver](https://github.com/dtolnay/semver) from 1.0.14 to 1.0.16.
- [Release notes](https://github.com/dtolnay/semver/releases)
- [Commits](https://github.com/dtolnay/semver/compare/1.0.14...1.0.16)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 14:52:56 +01:00
Arnaud Gourlay d77debf122 Clippy for Rust 1.66 (#1284) 2022-12-16 10:53:51 +01:00
Ivan Pleshkov 02071314bd remove obsolete internal scorer (#1271) 2022-12-14 14:47:22 +04:00
Andrey Vasnetsov aeba32ba3e Shard build refactoring (#1280) 
* refactor some code of shard creation and optimizers

* Shard build refactoring debug (#1279)

* add debig code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* refactor stop-checking during the optimization

* remove debug logs

* improve delete-renaming schema

* fmt

* move collection file removing to the async task

* rename check_optimization_stopped into more general check_process_stopped
 2022-12-14 10:11:46 +01:00
dependabot[bot] fdf531955d Bump rayon from 1.6.0 to 1.6.1 (#1277) 
Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.6.0 to 1.6.1.
- [Release notes](https://github.com/rayon-rs/rayon/releases)
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.6.0...rayon-core-v1.6.1)

---
updated-dependencies:
- dependency-name: rayon
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-14 09:04:07 +01:00
dependabot[bot] 760b9e8710 Bump geo from 0.23.0 to 0.23.1 (#1270) 
Bumps [geo](https://github.com/georust/geo) from 0.23.0 to 0.23.1.
- [Release notes](https://github.com/georust/geo/releases)
- [Changelog](https://github.com/georust/geo/blob/main/CHANGES.md)
- [Commits](https://github.com/georust/geo/compare/geo-0.23.0...geo-0.23.1)

---
updated-dependencies:
- dependency-name: geo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-14 08:42:18 +01:00
dependabot[bot] c9c9cbd300 Bump json-patch from 0.2.7 to 0.3.0 (#1276) 
Bumps [json-patch](https://github.com/idubrov/json-patch) from 0.2.7 to 0.3.0.
- [Release notes](https://github.com/idubrov/json-patch/releases)
- [Changelog](https://github.com/idubrov/json-patch/blob/main/CHANGELOG.md)
- [Commits](https://github.com/idubrov/json-patch/compare/v0.2.7...v0.3.0)

---
updated-dependencies:
- dependency-name: json-patch
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-14 08:41:54 +01:00
Andrey Vasnetsov d05b2def10 allow set different M param for default HNSW and payload-related links (#1257) 
* allow set different M param for default HNSW and payload-related links

* review fixes
 2022-12-06 13:53:42 +01:00
dependabot[bot] 614aa69c43 Bump json-patch from 0.2.6 to 0.2.7 (#1268) 
Bumps [json-patch](https://github.com/idubrov/json-patch) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/idubrov/json-patch/releases)
- [Commits](https://github.com/idubrov/json-patch/compare/v0.2.6...v0.2.7)

---
updated-dependencies:
- dependency-name: json-patch
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-06 13:52:33 +01:00
Andrey Vasnetsov 0925d02667 include backtrace for service errors (#1256) 2022-12-06 12:31:38 +01:00