upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-10-01 18:37:42 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,219 Commits 451 Branches 119 Tags
699ab0d60f00a35b8bca097e1dec69a3c58c064f
Commit Graph
295 Commits
Author SHA1 Message Date
Ivan Pleshkov 699ab0d60f Don't update payload if point was not upserted (#1564) 
* fix values count loading in geoindex

* don't update payload if point was not upserted

* remove debug comments

* post-check of internal id

* revert changes from geo indexing branch
 2023-03-15 16:30:45 +01:00
Ivan PleshkovandAndrey Vasnetsov 97fafce25f Fix values count loading in geoindex (#1563) 
* fix values count loading in geoindex

* WIP: fix loading statistics in geo-index

* WIP: fmt

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:30:35 +01:00
Andrey Vasnetsov 36c46a8dbf Fix histogram for large integer values (#1562) 
* make histogram generic

* fmt
 2023-03-15 16:30:25 +01:00
Andrey Vasnetsov 12c8d89d1f ensure segment flushing order + extend logging (#1551) 
* ensure segment flushing order + extend logging

* fix review
 2023-03-15 16:30:12 +01:00
Ivan Pleshkov c60ef44836 Enum vector index (#1539) 
* separate payload and vector index bases

* vector index as enum
 2023-03-15 16:29:54 +01:00
Ivan PleshkovandAndrey Vasnetsov 9639df1998 Remove vector scorer trait (#1546) 
* remove vector scorer

* nits

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:29:02 +01:00
Tim Visée 82b123e054 Fix clippy warnings (#1542) 2023-03-15 16:27:37 +01:00
Andrey Vasnetsov 8b7e508d27 Quantization re scoring param (#1538) 
* parameter to disable rescoring with quantization

* fix

* fix default

* review fixes

* upd api autogen
 2023-03-15 16:27:32 +01:00
Ivan Pleshkov 9bb29c26a6 Vector storages enum (#1533) 
* separate scoring and data containing

* vector storage enum

* fix test build

* are you happy clippy

* review fixes
 2023-03-15 16:27:25 +01:00
Ivan PleshkovandAndrey Vasnetsov 128e49fcc3 integrate quantized data to storages (#1311) 
* integrate quantized data to storages

* revert gitignore

* are you happy clippy

* quantize in optimizer

* provide flag

* fix segfault

* skip quantization flag, update scores

* use quantization flag

* are you happy fmt

* use quantization flag

* quantized search test

* are you happy fmt

* refactor test, refactor scorer choosing

* are you happy fmt

* run quantization on segment builder

* decrease testing parameters

* simplify segment

* update version

* remove use_quantization flag

* provide quantization config

* quantization version up

* euclid dist

* add euclid test

* saveload

* fix initialization bugs

* quantization lib version up

* fix arm build

* refactor scorer selecting

* quant lib version up

* are you happy fmt

* are you happy fmt

* are you happy clippy

* add save/load test for simple storage

* add comments

* quantiles

* quantization mmap

* remove f32

* mmap test

* fix mmap slice

* fix mmap test

* use chunks for quantization storage

* fix build

* are you happy fmt

* update quantization library

* update quantization lib

* update quantization lib

* integrate api changes

* are you happy fmt

* change quantization api

* additional checks in tests

* update quantization version

* fix unit tests

* add quantization to storage config

* use quantization for all cardinality search cases

* Integrate quantization suggestions 2 (#1520)

* review api

* wip: refactor quantization integrations

* wip: refactor quantization integrations

* wip: fmt

* include quantization into snapshot

* fmt

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:26:00 +01:00
Ivan Pleshkov c2316db156 fix windows warning unused parallel_graph_build (#1513) 2023-03-15 16:25:39 +01:00
Arnaud Gourlay a93a55e900 Fix condition check for values_count (#1502) 2023-03-15 16:24:25 +01:00
Arnaud GourlayandAndrey Vasnetsov 3d8b5131bd Nested payload filters (#1487) 
* Nested payload filters

* close ToDo + add parsing of multuiple array values

* fmt

* improve testing nested arrays

* fix NumericIndex to accumulate points_to_values mapping

* revert numberic index + strict array field access

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:24:04 +01:00
Arnaud Gourlay 3ad2e86e5e Access payload key through nested Array (#1465) 
* Access payload key through nested Array

* support removal through array index

* propagate type changes

* fmt

* avoid allocating Vec of one element
 2023-03-15 16:21:05 +01:00
Luis CossíoandAndrey Vasnetsov c414a77105 feat: add "any" to "match" statements (#1466) 
* feat: add "any" to "match" statements

Adds support for using "any" to match on a list of different possible values

* enable proper counting in cardinality estimation

* fmt

* upd OpenAPI

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-03-15 16:20:11 +01:00
dependabot[bot]andIvan Pleshkov 56d7e67b2f Bump rocksdb from 0.19.0 to 0.20.0 (#1461) 
* Bump rocksdb from 0.19.0 to 0.20.0

Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb) from 0.19.0 to 0.20.0.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases)
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.19.0...v0.20.0)

---
updated-dependencies:
- dependency-name: rocksdb
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix api changes

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
 2023-03-15 16:19:18 +01:00
Andrey Vasnetsov d146457794 Fix flacky windows test (#1448) 
* use async sleep

* fmt

* disablee test_parallel_graph_build on CI on windows untill there is a way to reproduce
 2023-03-05 13:04:24 +01:00
Roman Titov 9e685483da Reimplement ExtendedPointId deserialization (#1453) 
Reimplement `ExtendedPointId` deserialization, to produce better error message when deserialization fails
 2023-03-05 13:00:12 +01:00
Andrey Vasnetsov 1a39990ee9 rollback custom deserialization for point id 2023-02-08 13:58:02 +01:00
Andrey Vasnetsov 2bcba1de7a Windows support (#1432) 
* windows-support

* fmt

* install protoc with action

* upd WAL revision
 2023-02-06 15:22:24 +01:00
Andrey Vasnetsov f3f9c49b1d allow to overwrite max number of cpus with env variable (#1425) 2023-02-06 15:21:23 +01:00
Arnaud Gourlay 93c3f26d50 Implement Deserialize for ExtendedPointId to improve error reporting (#1416) 
* Implement Deserialize for ExtendedPointId to improve error reporting

* better error message
 2023-02-06 15:21:07 +01:00
Arnaud Gourlay 66aa2c99ce Clippy rust 1.67 (#1406) 
* inline format! args

* inline format! args

* explicit lifetime could be elided

* fmt
 2023-02-06 15:20:13 +01:00
Roman TitovandAndrey Vasnetsov b07bd81af4 Implemented response resolver (#1370, #1381) 
- Implement `Resolve` trait that "merges" multiple `retrieve`/`scroll_by`/`search` responses
  into a single response, to ensure it's consistent across multiple nodes in the cluster
- Implement tests for the `Resolve` trait implementation
- Add a few additional derives required for the `Resolve` trait implementation

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-02-06 15:18:33 +01:00
Arnaud Gourlay cbd385c977 Name rayon hnsw threads for clarity (#1399) 2023-02-06 15:18:17 +01:00
Andrey Vasnetsov 0e329957b6 Faster snapshot creation (#1353) 
* deduplication after proxy snapshot recovery

* revert changes in apply_points_to_appendable

* clippy fixes

* spawn blocking in snapshot creation
 2023-02-06 15:16:52 +01:00
Andrey Vasnetsov fc94d57c71 enable payload field anonymization (#1368) 2023-02-06 15:16:04 +01:00
Arnaud Gourlay 26fc1ed583 Remove some blocking IO calls on Tokio threads (#1366) 2023-02-06 15:15:47 +01:00
Andrey Vasnetsov 35f575ac9e Disable extra segments checks (#1352) 
* optional segment check

* alternative implementation
 2023-02-06 15:14:30 +01:00
Roman TitovandAndrey Vasnetsov 6eca194f71 Fix segment snapshotting (#1321) (#1334) 
* WIP: Fix `Segment::take_snapshot`

TODO:
- This commit, probably, breaks snapshotting of segments with memmapped vector storage
- `ProxySegment::take_snapshot` seems to potentially similar bug

* WIP: Fix `Segment::take_snapshot`

- Fix snapshotting of `StructPayloadIndex`
- Fix snapshotting of segments with memmapped vector storage
- Temporarily break `ProxySegment::take_snapshot`

* Fix `ProxySegment::take_snapshot`

* Remove `copy_segment_directory` test

* nitpicking

* clippy fixes

* use OperationError::service_error

* Cleanup `TinyMap` trait bounds and derive `Debug`

* Fix `test_snapshot` test

- Derive `Debug` for `NamedVectors`

* Move utility functions from `segment.rs` to `utils` module

* Contextualize `segment::utils::fs::move_all` a bit more carefully

* Fix a typo

* add backward compatibility with old snapshot formats

* fmt

* add snapshot for compatibility test

* git lfs is a piece of shit

* Nitpicking

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-01-13 12:14:58 +01:00
Arnaud GourlayandAndrey Vasnetsov 81b3313d31 Repair segment data while checking consistency (#1314) 
* Repair segment data while checking consistency

* code review

* code review

* code review

* code review drop_by_internal_id

* refactoring

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-01-13 12:14:25 +01:00
Roman Titov 3ef66f1e96 Add configuration option to select "memory advice" used for memmapped storage 
- Add a simple abstraction/wrapper over `memmap2::Advice` and a global switch
  to select "memory advice" used for memmapped storage to the `segment` crate
- Add `mmap_advice` configuration option to the `storage::types::StorageConfig`
- Implement `search-points` benchmark
 2023-01-13 12:13:34 +01:00
Andrey Vasnetsov bcb52f9aee Id mapper inconsistency (#1302) 
* always flush wal

* always flush wal fix

* always flush wal fmt

* flush wal during background flush

* async wal flush

* use id-tracker internal id for next-id instead of vector storage

* add flush order and recovery comment

fix merge bug

* longer timeout in test
 2022-12-25 23:02:46 +01:00
Andrey Vasnetsov 68942b202f better validation for transfer initiation 2022-12-25 22:51:54 +01:00
Arnaud Gourlay 9702054127 Clippy for Rust 1.66 (#1284) 2022-12-25 22:46:53 +01:00
Ivan Pleshkov 8b35d90e25 remove obsolete internal scorer (#1271) 2022-12-25 22:46:46 +01:00
Andrey Vasnetsov 8e8ed800f4 Shard build refactoring (#1280) 
* refactor some code of shard creation and optimizers

* Shard build refactoring debug (#1279)

* add debig code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* refactor stop-checking during the optimization

* remove debug logs

* improve delete-renaming schema

* fmt

* move collection file removing to the async task

* rename check_optimization_stopped into more general check_process_stopped
 2022-12-25 22:46:41 +01:00
Andrey Vasnetsov 4e084a6852 allow set different M param for default HNSW and payload-related links (#1257) 
* allow set different M param for default HNSW and payload-related links

* review fixes
 2022-12-06 15:04:19 +01:00
Andrey Vasnetsov 6a0d907755 include backtrace for service errors (#1256) 2022-12-06 15:03:25 +01:00
Andrey Vasnetsov 2d02f209ae Missed points on update fix (#1255) 
* add context to point-not-found error

* fmt

* add backtrace

* use upgradable lock

* unique points in insert operation

* remove debug code
 2022-12-06 15:03:20 +01:00
Ivan PleshkovandAndrey Vasnetsov 5e3416e2ce Hnsw links memmap (#1211) 
* add on disk key

* remove obsolete graph initialization

* remove obsolete max level

* update openapi

* graph links trait

* use mmap option

* same format for ram and mmap

* fix segment unit tests

* are you happy fmt

* are you happy clippy

* fix ci and add mmap test

* review fixes

* remove unused try-from

* fix version compatibility

* avoid loading from disk during conversion

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-11-25 12:17:57 +01:00
Arnaud GourlayandAndrey Vasnetsov 361c7ff716 Probabilistic search sampling for better limit (#1199) 
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-11-25 12:14:55 +01:00
Ivan Pleshkov a465679fb8 Hnsw links storage (#1171) 
* hnsw compact links storage

* dump links memsize

* are you happy fmt

* fix build

* calculate capacity and add shrink

* fmt

* remove garbage

* fix unit tests

* default values for graph layers

* remove obsolete fields

* add comments

* more comments

* use one flattened links structure

* refactor and add unit tests

* move merge into graph builder

* fix messages in test_hnsw_graph_properties
 2022-11-25 12:11:10 +01:00
Andrey Vasnetsovdependabot[bot]erare-humanumIvan PleshkovArnaud Gourlay
 		e027db4d27 V0.11.2 (#1209) 
* Update: unmaintained crate memmap -> memmap2 (#559) (#1160)

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* Consensus q n a (#1169)

* Expand comments and minor refactoring for raft state

* fmt

* add comments to consensus.rs

* fix "Consensus q n a" compatibility

* Use less ram for id tracker (#1176)

* use less ram for id tracker

* are you happy clippy

* use vec for internals

* use versions for internal ids

* keys test

* Use less ram for id tracker fixes (#1182)

* WIP: internal_to_version

* fmt

* fix unit tests

* add comment

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* remove suggesting changes in replications on replication factor change (#1177)

* Bump actix-cors from 0.6.3 to 0.6.4 (#1185)

Bumps [actix-cors](https://github.com/actix/actix-extras) from 0.6.3 to 0.6.4.
- [Release notes](https://github.com/actix/actix-extras/releases)
- [Commits](https://github.com/actix/actix-extras/compare/cors-v0.6.3...cors-v0.6.4)

---
updated-dependencies:
- dependency-name: actix-cors
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* enable HTTP compression middleware (#1184)

* Use systematically assert_http_ok for better error reporting (#1183)

* Use systematically assert_http_ok for better error reporting

* extraction assertion to use it outside of pytest

* Bump pprof from 0.10.1 to 0.11.0 (#1188)

Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.10.1 to 0.11.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases)
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Cosine dist zero vectors (#1198)

* skip pre-processing of zero-length vector for cosine distance

* fmt

* Bump env_logger from 0.9.1 to 0.9.3 (#1201)

Bumps [env_logger](https://github.com/env-logger-rs/env_logger) from 0.9.1 to 0.9.3.
- [Release notes](https://github.com/env-logger-rs/env_logger/releases)
- [Changelog](https://github.com/env-logger-rs/env_logger/blob/main/CHANGELOG.md)
- [Commits](https://github.com/env-logger-rs/env_logger/compare/v0.9.1...v0.9.3)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump indicatif from 0.17.1 to 0.17.2 (#1202)

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.17.1 to 0.17.2.
- [Release notes](https://github.com/console-rs/indicatif/releases)
- [Commits](https://github.com/console-rs/indicatif/compare/0.17.1...0.17.2)

---
updated-dependencies:
- dependency-name: indicatif
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump ordered-float from 3.3.0 to 3.4.0 (#1204)

Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.3.0...v3.4.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump clap from 4.0.18 to 4.0.22 (#1205)

Bumps [clap](https://github.com/clap-rs/clap) from 4.0.18 to 4.0.22.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.18...v4.0.22)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump num_cpus from 1.13.1 to 1.14.0 (#1203)

* wait for state deactivation on replica update failure (#1200)

* wait for state deactivation on replica update failure

* review fixes

* upd version to 0.11.2

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: erare-humanum <116254494+erare-humanum@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-11-09 14:03:49 +01:00
Andrey Vasnetsov 38bc2eb0f2 update rust version and fix clippy complains (#1190) 2022-11-04 11:53:28 +01:00
Andrey Vasnetsov 65a76c6b61 update and improve openapi definitions 2022-10-24 16:03:28 +02:00
Andrey Vasnetsov ab19576c5d Force sync vector storage offset (#1110) 
* simplify vector storage `update` semantic, always assume inplace vector replacement

* fmt
 2022-10-24 12:34:26 +02:00
Andrey Vasnetsov 54e0857708 ignore points with missing expernal ids instead of panic (#1109) 2022-10-24 12:33:43 +02:00
Arnaud Gourlay 1ff09516a0 limit set to zero returns an empty result (#1083) 2022-10-24 12:32:46 +02:00
Andrey Vasnetsov 21754ef2ef report indexed payload points in info api + other fixes (#1164) 
* report indexed payload points in info api + other fixes

* rollback debug changes

* clippy

* clippy
 2022-10-24 09:43:02 +02:00