upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-06 18:10:58 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,465 Commits 346 Branches 118 Tags
94de0e2d2dcf997bf26bec34dad407d8de9a9abb
Commit Graph

1780 Commits

Author SHA1 Message Date
Arnaud Gourlay
32e1bbb601 Fix sparse mmap snapshot (#3166) 2023-12-06 10:58:11 +01:00
Arnaud Gourlay
c3ac45313f Fix bench for mmap sparse search (#3171) 2023-12-06 10:10:20 +01:00
Arnaud Gourlay
4954d6d35c Clean two sparse vectors todos (#3163) 2023-12-05 21:29:19 +01:00
Ivan Pleshkov
7d57fb8859 Sparse vectors advanced search api support (#3128) 
* sparse vectors query scorer

plain search test without internals

fix sparse_vector_index_plain_search

fix async scorer build

fix unit tests

are you happy fmt

sparse index with full query support

fix benches, use exact flag to force plain search

add alias and comment to exact flag

fix tests

recommendations TODOs fix

add simple test

add todo

refactor index

test vs comparison dense and sparse discovery

are you happy fmt

propogate error from avg negatives

reuse filtered points list

rollback api changes

better discovery test

are you happy fmt

rollback openapi

fix build

fix tests

review remark recommendations

review remarks

review remarks

codespell

integrate plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix comment of plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix tests

* simplify search query function
 2023-12-05 20:32:23 +01:00
dependabot[bot]
8d4e615fbe Bump uuid from 1.5.0 to 1.6.1 (#3151) 
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.5.0 to 1.6.1.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/1.5.0...1.6.1)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-05 14:31:01 +01:00
Arnaud Gourlay
032025c447 Fix optimizer change sparse index on disk (#3143) 
* Estimate sparse vector dimension for optimizer (#3142)

* Estimate sparse vector dimension for optimizer

* add hack explanation + todo

Fix optimizer change sparse index on disk

fix index to disk

is appendable fix

add index files if necessary

fix test

* fix tests

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
 2023-12-05 00:24:30 +01:00
Arnaud Gourlay
f5b62d1500 Estimate sparse vector dimension for optimizer (#3142) 
* Estimate sparse vector dimension for optimizer

* add hack explanation + todo
 2023-12-04 23:01:37 +01:00
dependabot[bot]
bed4c3059e Bump atomicwrites from 0.4.1 to 0.4.2 (#3147) 
Bumps [atomicwrites](https://github.com/untitaker/rust-atomicwrites) from 0.4.1 to 0.4.2.
- [Commits](https://github.com/untitaker/rust-atomicwrites/compare/0.4.1...0.4.2)

---
updated-dependencies:
- dependency-name: atomicwrites
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-04 22:29:58 +01:00
Arnaud Gourlay
f39c95c34b Mmap InvertedIndex uses Sequential madvice (#3092) 
* Mmap InvertedIndex uses Sequential madvice

* use madvice Normal

* fix new bench compile after rebase
 2023-12-04 14:55:35 +01:00
Ivan Pleshkov
d272bf4b90 Sparse index persistence (#3131) 
* sparse index persistence

* fix tests

* remove defaults
 2023-12-04 12:02:53 +01:00
Andrey Vasnetsov
355f62f9cf build HNSW with quantized data (#3133) 
* build HNSW with quantized data

* review fix
 2023-12-04 10:38:58 +01:00
Kaan C. Fidan
c7402a45d7 Manhattan distance (#3079) 
* implemented Manhattan distance

* updated quantization dependency

* fixed negative distances and doc consistency

* fixed neon implementation

* updated quantization dependency

* updated quantization dependency

* removed redundant copy operation

* updated quantization dependency

* Change back to upstream quantization dependency

---------

Co-authored-by: timvisee <tim@visee.me>
 2023-12-02 21:28:38 +01:00
Ivan Pleshkov
e7364616c3 remove unnecessary validations for queryenum (#3118) 2023-12-01 16:51:19 +01:00
Ivan Pleshkov
336efea59e Sparse index segment and collection config (#2802) 
* quantization storage as separate entity

sparse index try to extend segment types

fix build

fix async scorer

codespell

update openapi

update vector index

remove code duplications

more fixes

more fixes

fix build

fix deserialization test

remove transform_into

are you happy clippy

update openapi

update openapi

are you happy clippy

fix build

optional serialize

more defaults

update openapi

fix comments

generic transpose_map_into_named_vector

rename fields in tests

remove obsolete parts

only named sparse config

VectorStruct without unnamed sparse

NamedVectorStruct without unnamed sparse

remove obsolete test

update openapi

mmap index

revert preprocess function

are you happy fmt

update openapi

fix build

fix tests

are you happy fmt

fix for client generation

fix sparse segment creation

fix basic sparse test

fix conflicts

remove obsolete convertion

fix build

config diffs

update openapi

review remarks

update openapi

fix batch upsert

add failing test showing bad ids matching

fix sparse vector insertion

remove on_disk flag

update openapi

revert debug assert

simplify conversions

update openapi

remove on disk storage flag

update openapi

default for vector config

update openapi comment

remove diffs

update openapi

* enable consensus test

* add comment

* update openapi
 2023-12-01 13:10:58 +01:00
Ivan Pleshkov
8dcf63c509 Optional sparse full scan threshold (#3116) 
* optional sparse full scan threshold

* update openapi

* skip serializing if none

* review remarks
 2023-11-30 21:47:30 +01:00
Ivan Pleshkov
db330370d4 Sparse vector storage rocksdb only (#3112) 
* sparse vector storage rocksdb only

* fix build

* spelling fix

* simplify get_cf

* restore deleted flags loading
 2023-11-30 17:48:19 +01:00
Ivan Pleshkov
4277f8bf42 Sparse vector propagate REST validation (#3111) 
* sparse vector propagate validation

* more validations

* add tests

* are you happy clippy

* fix python tests

* fix error name

* error name

* add error message

* review remarks
 2023-11-30 09:55:53 +01:00
Arnaud Gourlay
df543d0b0e Fix sparse vector search with_vector (#3110) 2023-11-29 15:16:59 +01:00
Ivan Pleshkov
e154041153 sparse vector basic python client tests fixes (#3108) 2023-11-29 14:01:11 +01:00
Arnaud Gourlay
4ca56d44b1 Compute actual indexed_vector_count for more consistence in mix scenario (#3107) 2023-11-29 13:46:35 +01:00
Andrey Vasnetsov
7bb220e9ef Demote column family not found during flush (#3105) 
* Demote payload index column family not found error

* clippy + fmt
 2023-11-28 15:37:07 +01:00
Andrey Vasnetsov
2e2ee3b7e3 Tolerate rocksdb tail wal errors (#3093) 
* make rocksdb ignore its internal WAL consistency for tail records, as we have our own WAL

* fmt
 2023-11-28 10:10:03 +01:00
dependabot[bot]
20e8d3356e Bump charabia from 0.8.3 to 0.8.5 (#3098) 
Bumps [charabia](https://github.com/meilisearch/charabia) from 0.8.3 to 0.8.5.
- [Release notes](https://github.com/meilisearch/charabia/releases)
- [Commits](https://github.com/meilisearch/charabia/compare/v0.8.3...v0.8.5)

---
updated-dependencies:
- dependency-name: charabia
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-28 08:50:50 +01:00
dependabot[bot]
fa5c64bea0 Bump proptest from 1.3.1 to 1.4.0 (#3102) 
Bumps [proptest](https://github.com/proptest-rs/proptest) from 1.3.1 to 1.4.0.
- [Release notes](https://github.com/proptest-rs/proptest/releases)
- [Changelog](https://github.com/proptest-rs/proptest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/proptest-rs/proptest/compare/v1.3.1...v1.4.0)

---
updated-dependencies:
- dependency-name: proptest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-28 07:48:06 +01:00
dependabot[bot]
25097090de Bump num-derive from 0.4.0 to 0.4.1 (#3101) 
Bumps [num-derive](https://github.com/rust-num/num-derive) from 0.4.0 to 0.4.1.
- [Changelog](https://github.com/rust-num/num-derive/blob/master/RELEASES.md)
- [Commits](https://github.com/rust-num/num-derive/compare/num-derive-0.4.0...num-derive-0.4.1)

---
updated-dependencies:
- dependency-name: num-derive
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-28 07:42:17 +01:00
Tim Visée
510e498d1f Default max_values_per_point to zero, add tests (#3039) 
* Prevent another division by zero in geo index

* Set default `max_values_per_point` to zero

* Extend empty cardinality test

* Add unit tests for empty index types with cardinality

* Shortcut cardinality estimation, zero if no values per point

* Use const functions for cardinality type

* Remove redundant checked division, we do a rhs zero check above
 2023-11-24 16:18:34 +01:00
Tim Visée
970fcc3836 Remove redundant copy from sum iterator (#3085) 
* Remove redundant copy from sum iterator

* Remove another redundant copied call
 2023-11-24 13:02:42 +01:00
Tim Visée
3f1a34fabd Simplify RocksDB column operator, restructure to remove just_seeked (#3083) 2023-11-23 14:29:07 +01:00
Arnaud Gourlay
bbc7b37123 Accept empty sparse vectors (#3057) 
* Accept empty sparse vectors

* sparse vector index is responsbile for filtering empty vectors
 2023-11-21 11:27:12 +01:00
Ivan Pleshkov
0607be64c9 Sparse vectors rest search and upsert (#3051) 
* sparse vector sparse search and upsert

are you happy fmt

fix build

update openapi

batch changes

update openapi

named sparse vector

* review remarks

* cowvalue to cowvector
 2023-11-21 09:18:15 +01:00
dependabot[bot]
e966352920 Bump procfs from 0.15.1 to 0.16.0 (#3059) 
Bumps [procfs](https://github.com/eminence/procfs) from 0.15.1 to 0.16.0.
- [Release notes](https://github.com/eminence/procfs/releases)
- [Commits](https://github.com/eminence/procfs/compare/v0.15.1...v0.16.0)

---
updated-dependencies:
- dependency-name: procfs
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-21 09:06:34 +01:00
dependabot[bot]
56ac57ab4c Bump atomic_refcell from 0.1.11 to 0.1.13 (#3066) 
Bumps [atomic_refcell](https://github.com/bholley/atomic_refcell) from 0.1.11 to 0.1.13.
- [Commits](https://github.com/bholley/atomic_refcell/commits)

---
updated-dependencies:
- dependency-name: atomic_refcell
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-21 09:02:32 +01:00
dependabot[bot]
f59230fbd1 Bump itertools from 0.11.0 to 0.12.0 (#3061) 
Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.11.0 to 0.12.0.
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-21 09:01:29 +01:00
dependabot[bot]
62af777db8 Bump rayon from 1.7.0 to 1.8.0 (#3060) 
Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.7.0 to 1.8.0.
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](https://github.com/rayon-rs/rayon/compare/rayon-core-v1.7.0...rayon-core-v1.8.0)

---
updated-dependencies:
- dependency-name: rayon
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-21 08:26:17 +01:00
dependabot[bot]
c9b9646432 Bump schemars from 0.8.15 to 0.8.16 (#3058) 
Bumps [schemars](https://github.com/GREsau/schemars) from 0.8.15 to 0.8.16.
- [Release notes](https://github.com/GREsau/schemars/releases)
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.15...v0.8.16)

---
updated-dependencies:
- dependency-name: schemars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-11-21 08:21:30 +01:00
Kerem Celik
8d303c563c Increased maximum visited-list pool size to 128 (#2931) 
* Increased visited-list pool size to 128

* Set visited-list pool limit to CPU count, clamp between 16 and 128

* Preallocate visited list pool vector with full capacity

---------

Co-authored-by: timvisee <tim@visee.me>
 2023-11-20 16:10:59 +01:00
Arnaud Gourlay
d69b38a05d Better heuristic for sparse vector pruning (#3048) 2023-11-20 15:58:10 +01:00
Arnaud Gourlay
e7fa2e0810 Merge scoring and overlap check for sparse vector (#3047) 2023-11-20 15:43:48 +01:00
Arnaud Gourlay
3547b441c5 Remove distance from sparse storage (#3040) 2023-11-17 22:12:18 +01:00
Ivan Pleshkov
ee23eda42c Sparse vectors api preparations (#3038) 
* Sparse vectors api preparations

generic transpose

remove features

unnecessary local variable

add snake case

* distance in config function
 2023-11-17 16:22:10 +01:00
Arnaud Gourlay
bc06546d61 Delete RocksDB files more aggressively (#3036) 2023-11-17 15:01:16 +01:00
Arnaud Gourlay
e386f58695 Remove sorted constraint on sparse vectors (#3030) 
* Remove sorted constraint on sparse vectors

* make sure to sort the query as well
 2023-11-17 14:01:08 +01:00
Tim Visée
e65eee44a8 Use stabilized div_ceil from from std (#2770) 2023-11-17 12:55:25 +01:00
Arnaud Gourlay
3a0353ede0 Deactivate sparse index pruning for negative query vectors (#3024) 2023-11-16 10:51:02 +01:00
Ivan Pleshkov
3093476201 integrate sparse vectors into queries (#3028) 2023-11-16 10:43:09 +01:00
Arnaud Gourlay
c32efd574f Fix sparse index build benchmark compilation (#3016) 2023-11-14 21:02:52 +01:00
Arnaud Gourlay
7a140b69da Benchmark sparse index build (#2984) 
* Benchmark sparse index build

* add vector count assertion
 2023-11-14 20:01:50 +01:00
Arnaud Gourlay
6266a438f0 Add fallback to plain search for sparse vectors (#2963) 2023-11-14 19:58:52 +01:00
Arnaud Gourlay
46c5b2a30c Fix vector count in sparse index (#2992) 
* Fix vector count in sparse index

* simplify naming

* rely on max id for count
 2023-11-14 17:31:18 +01:00
Andrey Vasnetsov
9973991958 Huge refactoring to make read requests aware of shard key selector (#3004) 
* huge refactoring to make read requests avare of shard key selector

* fix integration test

* review fixes

* allow lookup_from specific shards
 2023-11-14 16:47:05 +01:00