upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-01 07:30:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
2,826 Commits 345 Branches 116 Tags
83b93b951002a75845bc00ec00b50dafae86bfc8
Commit Graph

926 Commits

Author SHA1 Message Date
Arnaud Gourlay
3782792a5d Shortcut HNSW build on empty storages (#4441) 2024-06-11 14:23:02 +02:00
Ivan Pleshkov
6b35507770 Hnsw update entry point after linking (#4442) 
* hnsw update entry point after linking

* are you happy fmt
 2024-06-11 12:48:15 +02:00
Luis Cossío
fbb4c2740d universal-query: Fix local shard ordering (#4404) 
- adds an `order` field in `planned_query::ResultsMerge` to indicate the desired direction of merging, and merges accordingly
  - converting from `ShardQueryRequest` to `PlannedQuery` now needs the collection params to determine the required order depending on the vector config
- handles previously unhandled `filter` and `score_threshold` in `LocalShard::merge_prefetches()`, either by passing it to the core_search/scroll rescoring or during regular merging
 2024-06-10 18:07:13 -04:00
Arnaud Gourlay
c53a5470a9 Fix multivector simple storage metadata (#4424) 
* Fix multivector simple storage metadata

* code review naming
 2024-06-10 18:13:13 +02:00
xzfc
d2ef556237 Make sparse vectors generic (#4364) 
* Make sparse vector generic

* Generic search_context tests
 2024-06-10 15:04:32 +00:00
Luis Cossío
ff77ed14b2 universal-query: finish implementation of query() fn in collection (#4362) 
- reuse merging from shards in both `query()` and `query_internal()`
- check filters for issues if it was a slow request
 2024-06-07 15:49:13 -04:00
Luis Cossío
9ec1266dca universal-query: convert CollectionQueryRequest into ShardQueryRequest (resolve ids into vectors) (#4360) 
- make `VectorQuery` generic
- retrieve ids from `CollectionQueryRequest` and convert into vectors
- implement `RetrieveRequest` trait for it
- calculate avg vector with logic from `recommendations.rs`
- convert `CollectionQueryRequest` into `ShardQueryRequest`
- begin implementing `query()` fn in collection
 2024-06-07 12:58:47 -04:00
Arnaud Gourlay
bd3a107ae0 Enable RocksDB WAL for data consistency on crash (#4411) 2024-06-07 15:22:53 +02:00
Arnaud Gourlay
4db2b7a0de Add get optional vector for all storages (#4368) 
* Add get optional vector for dense simple storage

* more bound checks

* remove storage get_opt blanket implementation

* remove duplication

* remove duplication

* Update lib/segment/src/vector_storage/multi_dense/simple_multi_dense_vector_storage.rs

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
 2024-06-06 18:07:06 +02:00
Ivan Pleshkov
61cfb8bcb5 Test fix segment builder for sparse (#4397) 
* test fix segment builder for sparse

* are you happy fmt
 2024-06-05 13:52:08 +02:00
Andrey Vasnetsov
308f611fe0 initialize sparse vectors in the segment builder (#4396) 2024-06-05 12:01:20 +02:00
dependabot[bot]
95edc18c45 Bump quantization from 14f42f9 to 0caf67d (#4383) 
* Bump quantization from `14f42f9` to `0caf67d`

Bumps [quantization](https://github.com/qdrant/quantization) from `14f42f9` to `0caf67d`.
- [Commits](14f42f9443...0caf67d96f)

---
updated-dependencies:
- dependency-name: quantization
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix build

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
 2024-06-05 11:39:40 +02:00
Ivan Pleshkov
a629fc4d4c Fix hnsw full scan threshold (#4369) 
* fix hnsw full scan threshold

* add test

* are you happy clippy

* separate open_vector_storage

* remove public fields from builder

* wip: do not create segment in builder before build

* avoid arc in storage test and low-level loading functions

* WIP: remove internal segment from SegmentBuilder

* fmt

* finalize segment builder fixes

* Revert "are you happy clippy"

This reverts commit c04afa6989.

* Revert "add test"

This reverts commit 8e7ad6207e.

* Revert "fix hnsw full scan threshold"

This reverts commit 8904443fcb.

* remove _daatabse from builder

* fix optimizer test

* fix id tracker versions persistence

* do flush for segment components on build

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-06-04 11:16:11 +02:00
dependabot[bot]
39f41183d8 Bump rstest from 0.19.0 to 0.21.0 (#4385) 
Bumps [rstest](https://github.com/la10736/rstest) from 0.19.0 to 0.21.0.
- [Release notes](https://github.com/la10736/rstest/releases)
- [Changelog](https://github.com/la10736/rstest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/la10736/rstest/compare/v0.19.0...v0.21.0)

---
updated-dependencies:
- dependency-name: rstest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-03 18:52:12 -04:00
Arnaud Gourlay
d766d2dddc Fix Versions persistence atomicity (#4376) 2024-06-01 20:29:18 +02:00
Arnaud Gourlay
603efeda05 Rocksdb remove should use the same options as put (#4370) 2024-05-31 15:45:55 +02:00
Arnaud Gourlay
cfb2d87d93 Log failed segment flush on drop (#4371) 2024-05-31 15:45:28 +02:00
Luis Cossío
57aee5addf universal-query: Impl of query_internal in collection (#4331) 
* move ScoredPointTies to segment, make inner by reference

* `query_internal` implementation

* remove empty utils mod

* use `then_with`

* Improve readability, remove duplicated code

* refactoring suggestions

* don't collect eagerly

* remove unused import

* dont panic on empty transpose input

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-05-31 08:56:06 -04:00
Jojii
e41661a4e4 Add IdTrackerEnum (#4356) 
* add IdTrackerEnum

* rename enum variant
 2024-05-31 09:02:34 +02:00
xzfc
f4c7ba15af Sparse: misc fixes (#4361) 
* Borrow cows
* Hide sparse_vector_fixture under feature="testing"
 2024-05-30 20:58:00 +00:00
Arnaud Gourlay
bfc4b1930a Add optional post WAL data consistency check (#4359) 
* Add optional post WAL data consistency check

* fix multi features build
 2024-05-30 21:28:30 +02:00
Ivan Pleshkov
912e081bff Mmap for quantized multivector offsets (#4354) 
* mmap for quantized multivector offsets

* resolve save vector

* fmt

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-05-30 14:56:58 +02:00
Jojii
f6fa1216c5 Add config option shard_nr_per_node (#4309) 2024-05-30 14:43:01 +02:00
Luis Cossío
285ade0fda universal-query: Clone less in local shard (#4347) 
* use Cow instead of mutation

* rebase fixes

* review fixes
 2024-05-29 13:47:05 -04:00
xzfc
f026fc8a42 Recreate sparse vector index (#4313) 
* Recreate sparse vector index

* Rename index file; add simple migration
 2024-05-29 17:04:17 +00:00
Ivan Pleshkov
ef1a8b8f55 Use correct vector storage size (#4312) 
* use correct vector storage size

* remove dim from segment entry

* are you happy fmt

* codespell and proportions

* remove obsolete comment

* remove `try_vector_dim`

* are you happy fmt

* remove todo

* revert code of conduct

* check div 0

* Simplify a bit with max iterator

* Update lib/segment/src/index/hnsw_index/hnsw.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* are you happy fmt

* Update lib/segment/src/index/plain_payload_index.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* review fix

* set full scan threshold 0 for test

* use u128 also for multivector storages

* fix sparse vector size calculation

* Move size calculation into if-branch

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
 2024-05-28 16:38:56 +02:00
Ivan Pleshkov
93ed4ab5c8 Multidense vectors quantization (#4202) 
* quantized multivector definition

new multiquantization scorers

encode query

maxsim for quantized vectors

remove obsolete todo

reuse existing scorers

create multivector quantized storage

save load offsets

add test

fix vectors count

tempopery disable test while debugging

fix tests

fix build

less static lifetimes

less static lifetimes

fix build

* fix build after rebase

* add persistence test

* fix codespell

* increase accuracy in tests

* review remarks

* add comment references

* are you happy codespell

* don't use bincode
 2024-05-28 12:36:09 +02:00
Tim Visée
1af6a9b234 Fix missing segments, use correct path for new segment created during snapshot (#4332) 
* Put temporary segment in correct path

* Use shard directory rather than collection directory in test

* Fix collection path getter, it actually returns segments path

* Use segments path for temporary segment

* The build segment function actually wants the segments path

* Refactor parameter name
 2024-05-27 19:03:02 +02:00
Arnaud Gourlay
30f2968c28 Add vector name in storage consistency error (#4330) 2024-05-27 15:46:10 +02:00
xhjkl
f8c586e962 Huge limit now does not crash (#4328) 
* commentary

* FixedLengthPriorityQueue: clamp preallocated capacity
 2024-05-27 09:34:48 +02:00
Andrey Vasnetsov
d4dc0f3f42 allow reading of the non-existent sparse vector same way we allow dense (#4322) 2024-05-24 21:42:38 +02:00
Luis Cossío
b14d0b2463 universal-query: user-facing gRPC and REST request types (#4303) 
* add grpc `QueryPoints`

* add rest types

* move to REST types to `api::rest::schema`, decouple `OrderByInterface` and `RecommendStrategy`

* openapi changes

* add score_threshold param

* decouple `ShardKeySelector`, add shard key param

* remove commented-out code

* new codespell
 2024-05-24 08:23:12 -04:00
Jojii
6249623f35 Add vectors and quantization options and defaults (#4299) 
* add vectors and quantization options and defaults

* remove default quantization settings and add config options for consensus

* change config structure

* change parameter hierarchy

* apply new config values

* review changes

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-05-23 14:16:00 +02:00
Arnaud Gourlay
cbcdefb310 Cleanup unused dependencies (#4306) 2024-05-23 10:06:44 +02:00
xzfc
93a42b4e2d Compressed posting lists (#4253) 2024-05-22 18:18:20 +00:00
Luis Cossío
2a82ef41a0 universal-query: Introduce order_value field in ScoredPoint (#4291) 
* - Fix ordering in group-by for custom scoring
- Rename `OrderingValue` -> `OrderedValue`
- Introduce `order_value` field in `ScoredPoint`

* modify `Ord` implementation

* rename to `OrderValue`

* `has_custom_scoring` -> `is_distance_scored`

* regen apis

* flip bools in `is_distance_scored`
 2024-05-22 09:56:31 -04:00
Arnaud Gourlay
0071f8690c Rescore universal query (#4252) 
* Rescore universal query

* fix tests

* top level RRF fusion should not merge results at shard level
 2024-05-22 08:06:29 +02:00
Ivan Pleshkov
b1ac559aad quantization scorer with multivector support (#4289) 2024-05-21 17:40:41 +02:00
Ivan Pleshkov
67ec5b0476 multivector storage iterate over all inner vectors (#4288) 2024-05-21 15:45:15 +02:00
Ivan Pleshkov
a162284916 add static lifetime to primirive type (#4286) 2024-05-21 14:02:33 +02:00
Ivan Pleshkov
a68965f05a Fix wrong multivectors count in storage (#4283) 
* fix wrong multivectors count in storage

* add test
 2024-05-21 12:27:07 +02:00
dependabot[bot]
32532d75fa --- (#4268) 
updated-dependencies:
- dependency-name: serde-untagged
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-05-21 10:23:48 +02:00
dependabot[bot]
dcfd0b3e05 --- (#4269) 
updated-dependencies:
- dependency-name: smol_str
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-05-21 09:38:57 +02:00
Luis Cossío
fef3e142fd proofread issue messages (#4266) 2024-05-20 11:38:46 -04:00
Luis Cossío
b92c44c548 allow missing safety doc on half float similarities (#4259) 2024-05-17 19:59:13 +02:00
Arnaud Gourlay
b7396202b1 Minor cleanup from Rover's lints (#4249) 2024-05-16 21:47:43 +02:00
Arnaud Gourlay
9d7b322802 Introduce RRF scoring (#4247) 
* Introduce RFF scoring

* fix acronymes juggling

* proper name

* fix naming again

* code review

* cleanup

* unstable sort for extra perf

* score -> point
 2024-05-16 17:53:26 +02:00
Luis Cossío
7dcdae04c7 universal-query: Test and fix PlannedQuery conversion (#4242) 
* test and fix PlannedQuery conversion

* add `ScoringQuery::RRF`

* add more test cases, fix more
 2024-05-16 10:33:00 -04:00
xzfc
ccf7f1d24f InvertedIndexImmutableRam and index migrations (#4220) 
* Move StorageVersion from segment crate to common/io

* Refine StorageVersion API

* Move methods from SparseVectorDataConfig to enum SparseIndexType

* Introduce InvertedIndexImmutableRam

* Add migrate

* Don't migrate
 2024-05-16 06:47:47 +00:00
Ivan Pleshkov
5d503b5dde fix arm linking error on docker (#4244) 2024-05-15 20:51:09 +02:00