upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-07-31 15:11:35 -05:00
Code Issues Packages Projects Releases Wiki Activity
3,332 Commits 343 Branches 116 Tags
b94d02ad30d1d2c2065b0d64c94de9d7d7560879
Commit Graph

54 Commits

Author SHA1 Message Date
xzfc
10ce515ecf shell.nix: bump to rust 1.81, misc changes (#5185) 
* shell.nix: add a comment

* tools/nix/update.py: let it run outside of the nix-shell

* shell.nix: bump to rust 1.81
 2024-10-07 10:30:20 +00:00
xzfc
35dad67771 Add shell.nix (#4821) 2024-08-08 16:57:14 +00:00
Luis Merino
36d7e11624 Fix web UI index path http security headers (#4517) 
* Draft: web-ui root endpoint x-frame-options: deny header

* Switch to async

* Simplify setting frame options header by using DefaultHeaders

---------

Co-authored-by: timvisee <tim@visee.me>
 2024-06-25 13:44:25 +02:00
xzfc
b87e6d2bdd Improve tools/*.sh (#4365) 
* Speedup tools/generate_openapi_models.sh

* Improve tools/generate_grpc_docs.sh
 2024-05-31 09:51:37 +00:00
xzfc
3ad1528ebe Use /usr/bin/env bash in shebangs (#3570) 2024-02-09 11:42:29 +01:00
Kaan C. Fidan
c7402a45d7 Manhattan distance (#3079) 
* implemented Manhattan distance

* updated quantization dependency

* fixed negative distances and doc consistency

* fixed neon implementation

* updated quantization dependency

* updated quantization dependency

* removed redundant copy operation

* updated quantization dependency

* Change back to upstream quantization dependency

---------

Co-authored-by: timvisee <tim@visee.me>
 2023-12-02 21:28:38 +01:00
Tim Visée
d321b64b97 Add tool script to clean old RocksDB log files from disk (#3071) 2023-11-22 09:58:53 +01:00
Andrey Vasnetsov
74de1483be Shard key create confirmation (#3027) 
* create dedicated API for creating shards with explicit avait on the consensus

* fmt

* update api definitions
 2023-11-16 15:58:32 +01:00
Roman Titov
cab234c391 Add shard snapshot gRPC API (#2825) 
* Fix paste-bugs in `snapshot_service.proto`

* Add shard snapshot gRCP API definition

* Add validation to shard snapshot gRPC API definition

* Implement conversions between gRPC and `collection` types

* Extract shard snapshot API implementation into common sub-module

* Implement shard snapshot gRPC API

* Generate gRPC docs

* Refactor `ShardSnapshots` gRPC service to be internal API only

* fixup! Refactor `ShardSnapshots` gRPC service to be internal API only

Move `ShardSnapshotRecoverResponse` to `shard_snapshots_service.proto`

* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only

Update `api/src/grpc/qdrant.rs`

* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only

Update gRPC docs

* Switch `ShardSnapshots` gRPC service to use `validate_and_log` instead of `validate`
 2023-10-18 14:34:58 +02:00
Andrey Vasnetsov
326ef54664 openapi definitions for shard shashots API (#2571) 
* openapi definitions for shard shashots API

* review fixes
 2023-09-04 14:54:08 +02:00
Luis Cossío
0caeb3a5e3 remove dist zip after unzipping it (#2487) 2023-08-22 18:02:02 +02:00
Andrey Vasnetsov
0ccca76949 allow to configure qdrant init file with env variable (#2316) 
* allow to configure qdrant init file with env variable

* Return path type from get_init_file_path

---------

Co-authored-by: timvisee <tim@visee.me>
 2023-07-24 16:37:36 +02:00
Tim Visée
e1a69099fc Add script to list commits that have yet to be picked into master (#2108) 
* Add script to list commits that have yet to be picked into master

This allows us to easily detect cherry-picks we missed from dev to
master on our release branch.

* Update missed cherry picks ignore hash
 2023-06-28 14:57:49 +02:00
Andrey Vasnetsov
aef9e0833d use bash in the entrypoint.sh for good traps 2023-06-19 14:34:16 +02:00
Andrey Vasnetsov
967d387a37 Web UI integration (#2009) 
* wip: integrate UI web interface

* fmt

* Api key exclusion rule + fixes

* upd welcome url

* rollback api key in config
 2023-06-06 21:42:24 +02:00
Roman Titov
abffa27ff0 Refactor Dockerfile (#1594) 
* Refactor Dockerfile

- fix cross-compilation
- improve caching

TODO:
- check if `lld` is used/works for linkage (and enable if not used, or remove if doesn't work)

* Remove `aarch64` linker config from `.cargo/config.toml` (seems to be unnecessary)

* Expose `LINKER` argument and enable `lld` linker

* Add `mold` linker support

* Document Dockerfile

* Add closing ` in the Dockerfile comments

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
 2023-05-26 16:01:43 +02:00
Roman Titov
7fa637c0f0 Add script to generate docker-compose.yaml for Qdrant cluster (#1489) 
* Add a simple script to generate `docker-compose.yaml` for a cluster of specified number of nodes

* Improve service error from `RequestError` conversion

* fixup! Improve service error from `RequestError` conversion [skip ci]

* fixup! Add a simple script to generate `docker-compose.yaml` for a cluster of specified number of nodes [skip ci]

* Add CPU limit to the `docker-compose-gen.sh` [skip ci]

* fixup! Add CPU limit to the `docker-compose-gen.sh` [skip ci]

* Move docker-compose generator script into tools directory, add docs

* Apply suggestions from code review

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

* Simplify error message constructing

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
 2023-05-25 16:49:12 +02:00
Andrey Vasnetsov
e8edbe02ed it's a trap! (#1975) 2023-05-25 15:57:36 +02:00
Andrey Vasnetsov
72c3b9070d temporary disable mold to let the docker build (#1954) 2023-05-23 20:44:37 +02:00
Luis Cossío
e90a03e00b Group by key (#1768) 
* test: test must_not is_null

* vcs: ignore vscode files

* feat: group-by initial implementation

* cargo fmt

* refactor: same request behavior on reco and search

* refactor: get rid of RefCell

* refactor-fix: correct hashmap keys, and early stops

* chore: small improvements

* feat: groups aggregator

* fix: pull changes from other files

* cargo fix

* cargo fmt

* docs: edit docstrings

* allow dead code (while the complete feature is beint built)

* chore: restructure

* feat: introduce GroupKey, minor other improvements

* cargo fmt

* chore: specify aggregator visibility

* fix: oops, leaking "private" type

* refactor-fix: restructure and refactor group_by

* cargo fix

* fix: don't panic when there is no group-by field

* remove print statements

* amend: `>=`  -> `==`

* perf: remove double clone

* chore: sync aggregator from other branch

* chore: cleanup print statemets

* test: ignore big tests

* cargo fmt

* refactor: add early stop when the groups have been filled, improve code

* chore: sync aggregator, remove print from test

* refactor: consider shard_selection, improve collection_by_name handling

* feat: add bucketing to table of content

* refactor: better errors, improve tests

* test: add integration tests

* feat: add endpoints

* refactor: introduce ScoredPoint wrapper, restructure types

* sync aggregator

* edit internal grouping visibility

* feat: group_by internals

* cargo fmt

* cargo fmt

* refactor: turn inner fn into closure

* test: fix test to support new vector output representation

* feat: wire up grouping with actix

* expose grouped_by field

* fix: change output group format

* feat: wire up openapi

* fix: finish wiring up grouping in actix

* tests: fix test_group.py

* cargo fmt

* refactor: extract constants

* remove Hash from ScoredPoint

* `Option<collection_by_name>` -> `collection_by_name`

* fix: handle better cases on `match_on`

* fix: consider that subsequent calls can bring better results

* cargo fmt

* fix clippy warnings

* cargo fmt

* refactor: move `Group` to `types`, localize `hydrate_from`, remove `Deref` impls

* refactor `add_points`

* refactor: turn `GroupKey` into enum

* refactor: make `HashablePoint` inner struct private

* feat: add grpc layer, make new `PointGroup` type to use as output

* fix: update openapi models

* docs: update grpc docs

* fix merge errors

* refactor: add BaseGroupRequest to make code DRYer, improve doc comments

* cargo fmt

* perf: increase precision; choose best groups by score

* misc: add more integration tests, fix review comments

* cargo fmt

* fix: reimplement interface to flatten search and recommend requests, excluding offset

* cargo fmt

* refactor: move `r#do` impl to `GroupRequest`

* fix: update grpc docs

* perf: sort in reverse order

* fix: use fist value of a Value::Array

* fix: validate group_by to not support bracket notation, fix int. tests

* fix: update grpc validation

* tests: update collection_tests

* refactor: move validation to the api layers

* Oops: reupdate tests

* refactor: let the derives derive (thanks @ffuugoo)

* refactor: use a new GroupId on the output

also increases performance by copying less

* remove hashable set, take ordering into an account, fix mutliple groups values support

* fmt

* refactor group_id + rename per_group -> group_size, fix clippy

* remove GroupKey wrapper

* @agourlay review fixes

* refactor: `group_min_scores` and `group_max_scores` ->  `group_best_scores`

* refactor: use set difference on `keys_of_unfilled_best_groups`

* refactor: use set intersection on `len_of_filled_best_groups`

* refactor: turn best_group_keys into iterator

* fix: remove [] syntax limitation

* fix: update openapi.json

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-05-15 23:05:20 +02:00
Jesse
34055f0ddf Custom build profile for CI docker builds (#1859) 
* Use custom build profile for docker builds

* trigger ci

* change source code to trigger ci with rebuild one more time

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-05-08 20:39:59 +02:00
Jesse
b67c655ddb Add caching of docker layers in CI (#1856) 
* Add caching of docker layers in CI

Build required docker images for CI in a workflow step using buildkit's
gha cache type. This will populate the local layer cache from github
actions' cache. Builds in subsequent CI steps will be nearly instant,
because all layers can be reused.

* add minor change to see if build time is any faster

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-05-08 17:02:29 +02:00
Roman Titov
16b7fb7688 Improve handling out-of-RAM errors during Qdrant startup (#1777) 
* WIP: Start working on out-of-RAM errors handling [skip ci]

* Implement basic handling of out-of-RAM errors during Qdrant startup

* Try to fix CI fail by allowing both V1 and V2 cgroups

* Try to fix CI fail by improving cgroups handling

* Fix cgroups path detection/handling (+ some minor stylistic changes)

* fixup! Fix cgroups path detection/handling (+ some minor stylistic changes)

* Add test

* Enable low RAM test

* fixup! Add test

* free memory checks

* rm unused function

* Oom fallback script (#1809)

* add recover mode in qdrant + script for handelling OOM

* fix clippy

* reformat entrypoint.sh

* fix test

* add logging to test

* fix test

* fix test

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2023-05-02 20:54:13 +02:00
Roman Titov
cf04b1a92b Add /cluster/recover API endpoint (#1451) 
- Add required internal `raft::raw_node::RawNode::report_snapshot` calls
- Add methods to propagate `raft::raw_node::RawNode::request_snapshot` call to the consensus thread
- Add `/cluster/recover` API endpooint
- Add `test_collection_recovery.py` integration test (disabled on CI; documents current, incorrect behavior)
 2023-02-15 04:19:11 +10:00
Ivan Pleshkov
f50f1f4c71 #1119 disable updating feature (#1126) 
* disable updating feature

* customize error message

* update namings

* Update src/actix/api/service_api.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* fix build

* update api

* update api and allow deletion

* update openapi

* replace Bad Request with Forbidden

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-18 12:58:38 +02:00
Andrey Vasnetsov
01807d5cf3 Multiplatform build fix 2 (#1000) 
* upd dockerfile to multi-arch build

* debug actions

* fail faster on install

* add buildplatform

* enable CI

* dedicated push for github registry

* default platform

* fix default target arch

* revert debug ci changes
 2022-09-12 13:26:40 +02:00
Ivan Pleshkov
42e930ab8f Segment telemetry (#814) 
* segment telemetry

* anonymize trait

* fix build

* are you happy fmt

* anonimyze implementations

* sliding window avg (#826)

* Actix web telemetry (#828)

* actix web telemetry

* small as move

* use tokio mutex instead of std

* add comments

* are you happy fmt

* use u16 as http status code

* telemetry structs rename

* fix build

* using parking lot mutex

* telemetry web api (#842)

* telemetry web api

* telemetry openapi (#843)

* use async mutex for telemetry collector

* use tokio mutex for telemetry collector

* are you happy fmt
 2022-07-22 19:27:07 +04:00
Andrey Vasnetsov
fac87018c4 Snapshoting API (#764) 
* wip: rest api for snapshots

* fmt

* fix tests

* fmt

* wip: collection snapshoting and recovery

* fmt

* remote shard snapshots

* remote shard test

* fmt

* extend proxy snapshot test + fix double read lock

* fmt + clippy

* openapi schema

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* reduce writes on snapshots location

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-07-01 13:21:57 +02:00
Snyk bot
00408e4c84 fix: upgrade @openapi-contrib/json-schema-to-openapi-schema from 1.2.0 to 1.3.0 (#665) 
Snyk has created this PR to upgrade @openapi-contrib/json-schema-to-openapi-schema from 1.2.0 to 1.3.0.

See this package in npm:
https://www.npmjs.com/package/@openapi-contrib/json-schema-to-openapi-schema

See this project in Snyk:
https://app.snyk.io/org/generall/project/c2542b2d-d01b-437b-a5c7-acdb8524555a?utm_source=github&utm_medium=referral&page=upgrade-pr
 2022-06-07 13:25:58 +02:00
Andrey Vasnetsov
7c254a0de0 preserve fields order in OpenAPI schema (#655) 2022-06-02 19:12:30 +02:00
Andrey Vasnetsov
c286bde562 Cluster api (#631) 
* cluster status API

* add cluster api to OpenAPI specification
 2022-05-27 14:52:51 +02:00
Andrey Vasnetsov
5800319edb refactor distances + add score_threshold + fix negative euclid distance (#569) 
* refactor distances + add score_threshold + fix negative euclid distance

* generate docs

* fix clippy

* Update lib/segment/src/spaces/tools.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-05-09 17:32:48 +02:00
dependabot[bot]
2a70d5c70b Bump urijs from 1.19.10 to 1.19.11 in /tools/schema2openapi (#468) 
Bumps [urijs](https://github.com/medialize/URI.js) from 1.19.10 to 1.19.11.
- [Release notes](https://github.com/medialize/URI.js/releases)
- [Changelog](https://github.com/medialize/URI.js/blob/gh-pages/CHANGELOG.md)
- [Commits](https://github.com/medialize/URI.js/compare/v1.19.10...v1.19.11)

---
updated-dependencies:
- dependency-name: urijs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-13 09:27:02 +02:00
Arnaud Gourlay
77a3b64593 add document generation step to the grpc guide (#422) 2022-03-30 16:57:50 +02:00
Andrey Vasnetsov
f311fc873f Update generate_grpc_docs.sh (#420) 2022-03-30 11:25:17 +02:00
dependabot[bot]
5ee3cf7fc0 Bump urijs from 1.19.9 to 1.19.10 in /tools/schema2openapi (#370) 
Bumps [urijs](https://github.com/medialize/URI.js) from 1.19.9 to 1.19.10.
- [Release notes](https://github.com/medialize/URI.js/releases)
- [Changelog](https://github.com/medialize/URI.js/blob/gh-pages/CHANGELOG.md)
- [Commits](https://github.com/medialize/URI.js/compare/v1.19.9...v1.19.10)

---
updated-dependencies:
- dependency-name: urijs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-09 09:02:49 +01:00
dependabot[bot]
fc9e88f42c Bump urijs from 1.19.8 to 1.19.9 in /tools/schema2openapi (#365) 
Bumps [urijs](https://github.com/medialize/URI.js) from 1.19.8 to 1.19.9.
- [Release notes](https://github.com/medialize/URI.js/releases)
- [Changelog](https://github.com/medialize/URI.js/blob/gh-pages/CHANGELOG.md)
- [Commits](https://github.com/medialize/URI.js/compare/v1.19.8...v1.19.9)

---
updated-dependencies:
- dependency-name: urijs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-04 08:11:02 +01:00
Andrey Vasnetsov
c6441f06c1 [gPRC] docs + refactoring (#363) 
* wip: doc generator script

* fix enum default value + CollectionInfo upd + collections comments

* refactor grpc + generate docs

* disable clippy for auto-generated files

* Update src/tonic/proto/points.proto

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-03-03 15:30:09 +01:00
Andrey Vasnetsov
bdc3194b13 allow to return vectors and payload with recommendation request #353 (#357) 
* allow to return vectors and payload with recommendation request #353

* fmt

* unify `with_vector` comment line

* upd openapi
 2022-03-02 10:42:58 +01:00
dependabot[bot]
606b96290b Bump urijs from 1.19.5 to 1.19.8 in /tools/schema2openapi (#344) 
Bumps [urijs](https://github.com/medialize/URI.js) from 1.19.5 to 1.19.8.
- [Release notes](https://github.com/medialize/URI.js/releases)
- [Changelog](https://github.com/medialize/URI.js/blob/gh-pages/CHANGELOG.md)
- [Commits](https://github.com/medialize/URI.js/compare/v1.19.5...v1.19.8)

---
updated-dependencies:
- dependency-name: urijs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-27 23:54:44 +01:00
dependabot[bot]
738b3ee6cc Bump node-fetch from 2.6.1 to 2.6.7 in /tools/schema2openapi (#285) 
Bumps [node-fetch](https://github.com/node-fetch/node-fetch) from 2.6.1 to 2.6.7.
- [Release notes](https://github.com/node-fetch/node-fetch/releases)
- [Commits](https://github.com/node-fetch/node-fetch/compare/v2.6.1...v2.6.7)

---
updated-dependencies:
- dependency-name: node-fetch
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-03 22:21:43 +01:00
Andrey Vasnetsov
d51a70fa93 add openapi validation during generation #208 (#248) 
* add openapi validation during generation #208

* fix: POST -> PUT in point update api implementation and docs #208

* fix: openapi structure exposure

* fix: api usage in stress test
 2022-01-24 17:33:57 +01:00
Arnaud Gourlay
97cb5091bc Split Points API #208 (#221) 
Split Points API #208
 2022-01-24 07:54:47 +01:00
Andrey Vasnetsov
d7148126ce Split collection update API into several endpoints (#126) 
* split storage operation structures #32

* cargo fmt #32

* split collection update api into several endpoints #32

* cargo fmt #32

* fix tonic-related code with new structures

* upd alias structures

* use ytt teplate engine for OpenAPI Endpoint schema generation
 2021-12-13 09:38:33 +01:00
Andrey Vasnetsov
97b2270485 Rustdoc and README for internal entities and processes (#123) 
* extend comments for strorage crate

* update comments and readme for collection crate

* apply cargo fmt

* fix tests

* apply fmt
 2021-11-29 09:39:22 +01:00
trean
d30e6fa8ee Implementation of points scroll API #38 (#40) 
* WIP: filtered points iterator #38

* add paginated filtered point request function #38

* add scroll api + openapi definitions #38

* fix openapi #38
 2021-06-20 15:30:12 +02:00
Andrey Vasnetsov
5c2afdc5b9 geo-radius + update wiki 2021-01-24 17:16:06 +01:00
dependabot[bot]
9c65409a01 Bump urijs from 1.19.2 to 1.19.5 in /tools/schema2openapi 
Bumps [urijs](https://github.com/medialize/URI.js) from 1.19.2 to 1.19.5.
- [Release notes](https://github.com/medialize/URI.js/releases)
- [Changelog](https://github.com/medialize/URI.js/blob/gh-pages/CHANGELOG.md)
- [Commits](https://github.com/medialize/URI.js/compare/v1.19.2...v1.19.5)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-12-31 07:35:32 +00:00
Andrey Vasnetsov
09fea5a087 remove incorrect usage of "allOf" from API specification 2020-11-29 00:15:15 +01:00
Andrey Vasnetsov
07f9fa4113 redoc documentation 2020-10-27 13:52:00 +01:00