Files
qdrant/docs/grpc/docs.md
Egor Ivkov 2bb52abc71 Multiple shards in collection (#360)
* Multiple shards in collection

* Collection and Shard async drop

* Generate APIs for shard_number param
2022-03-07 16:22:47 +03:00

34 KiB
Raw Blame History

Protocol Documentation

Table of Contents

Top

collections.proto

CollectionConfig

Field Type Label Description
params CollectionParams Collection parameters
hnsw_config HnswConfigDiff Configuration of vector index
optimizer_config OptimizersConfigDiff Configuration of the optimizers
wal_config WalConfigDiff Configuration of the Write-Ahead-Log

CollectionDescription

Field Type Label Description
name string Name of the collection

CollectionInfo

Field Type Label Description
status CollectionStatus operating condition of the collection
optimizer_status OptimizerStatus status of collection optimizers
vectors_count uint64 number of vectors in the collection
segments_count uint64 Number of independent segments
disk_data_size uint64 Used disk space
ram_data_size uint64 Used RAM (not implemented)
config CollectionConfig Configuration
payload_schema CollectionInfo.PayloadSchemaEntry repeated Collection data types

CollectionInfo.PayloadSchemaEntry

Field Type Label Description
key string
value PayloadSchemaInfo

CollectionOperationResponse

Field Type Label Description
result bool if operation made changes
time double Time spent to process

CollectionParams

Field Type Label Description
vector_size uint64 Size of the vectors
distance Distance Distance function used for comparing vectors
shard_number uint32 Number of shards in collection

CreateCollection

Field Type Label Description
collection_name string Name of the collection
vector_size uint64 Size of the vectors
distance Distance Distance function used for comparing vectors
hnsw_config HnswConfigDiff optional Configuration of vector index
wal_config WalConfigDiff optional Configuration of the Write-Ahead-Log
optimizers_config OptimizersConfigDiff optional Configuration of the optimizers

DeleteCollection

Field Type Label Description
collection_name string Name of the collection

GetCollectionInfoRequest

Field Type Label Description
collection_name string Name of the collection

GetCollectionInfoResponse

Field Type Label Description
result CollectionInfo
time double Time spent to process

HnswConfigDiff

Field Type Label Description
m uint64 optional Number of edges per node in the index graph. Larger the value - more accurate the search, more space required.
ef_construct uint64 optional Number of neighbours to consider during the index building. Larger the value - more accurate the search, more time required to build index.
full_scan_threshold uint64 optional Minimal amount of points for additional payload-based indexing. If payload chunk is smaller than full_scan_threshold additional indexing won't be used - in this case full-scan search should be preferred by query planner and additional indexing is not required.

ListCollectionsRequest

ListCollectionsResponse

Field Type Label Description
collections CollectionDescription repeated
time double Time spent to process

OptimizerStatus

Field Type Label Description
ok bool
error string

OptimizersConfigDiff

Field Type Label Description
deleted_threshold double optional The minimal fraction of deleted vectors in a segment, required to perform segment optimization
vacuum_min_vector_number uint64 optional The minimal number of vectors in a segment, required to perform segment optimization
default_segment_number uint64 optional Target amount of segments optimizer will try to keep. Real amount of segments may vary depending on multiple parameters:
  • Amount of stored points. - Current write RPS.

It is recommended to select default number of segments as a factor of the number of search threads, so that each segment would be handled evenly by one of the threads. | | max_segment_size | uint64 | optional | Do not create segments larger this number of points. Large segments might require disproportionately long indexation times, therefore it makes sense to limit the size of segments.

If indexation speed have more priority for your - make this parameter lower. If search speed is more important - make this parameter higher. | | memmap_threshold | uint64 | optional | Maximum number of vectors to store in-memory per segment. Segments larger than this threshold will be stored as read-only memmaped file. | | indexing_threshold | uint64 | optional | Maximum number of vectors allowed for plain index. Default value based on https://github.com/google-research/google-research/blob/master/scann/docs/algorithms.md | | payload_indexing_threshold | uint64 | optional | Starting from this amount of vectors per-segment the engine will start building index for payload. | | flush_interval_sec | uint64 | optional | Interval between forced flushes. | | max_optimization_threads | uint64 | optional | Max number of threads, which can be used for optimization. If 0 - NUM_CPU - 1 will be used |

PayloadSchemaInfo

Field Type Label Description
data_type PayloadSchemaType Field data type
indexed bool If this field is indexed

UpdateCollection

Field Type Label Description
collection_name string Name of the collection
optimizers_config OptimizersConfigDiff optional New configuration parameters for the collection

WalConfigDiff

Field Type Label Description
wal_capacity_mb uint64 optional Size of a single WAL block file
wal_segments_ahead uint64 optional Number of segments to create in advance

CollectionStatus

Name Number Description
UnknownCollectionStatus 0
Green 1 All segments are ready
Yellow 2 Optimization in process
Red 3 Something went wrong

Distance

Name Number Description
UnknownDistance 0
Cosine 1
Euclid 2
Dot 3

PayloadSchemaType

Name Number Description
UnknownType 0
Keyword 1
Integer 2
Float 3
Geo 4

Collections

Method Name Request Type Response Type Description
Get GetCollectionInfoRequest GetCollectionInfoResponse Get detailed information about specified existing collection
List ListCollectionsRequest ListCollectionsResponse Get list name of all existing collections
Create CreateCollection CollectionOperationResponse Create new collection with given parameters
Update UpdateCollection CollectionOperationResponse Update parameters of the existing collection
Delete DeleteCollection CollectionOperationResponse Drop collection and all associated data

Top

points.proto

ClearPayloadPoints

Field Type Label Description
collection_name string name of the collection
wait bool optional Wait until the changes have been applied?
points PointsSelector Affected points

Condition

Field Type Label Description
field FieldCondition
hasId HasIdCondition
filter Filter

CreateFieldIndexCollection

Field Type Label Description
collection_name string name of the collection
wait bool optional Wait until the changes have been applied?
field_name string Field name to index

DeleteFieldIndexCollection

Field Type Label Description
collection_name string name of the collection
wait bool optional Wait until the changes have been applied?
field_name string Field name to delete

DeletePayloadPoints

Field Type Label Description
collection_name string name of the collection
wait bool optional Wait until the changes have been applied?
keys string repeated List of keys to delete
points PointId repeated Affected points

DeletePoints

Field Type Label Description
collection_name string name of the collection
wait bool optional Wait until the changes have been applied?
points PointsSelector Affected points

FieldCondition

Field Type Label Description
key string
match Match Check if point has field with a given value
range Range Check if points value lies in a given range
geo_bounding_box GeoBoundingBox Check if points geo location lies in a given area
geo_radius GeoRadius Check if geo point is within a given radius

Filter

Field Type Label Description
should Condition repeated At least one of those conditions should match
must Condition repeated All conditions must match
must_not Condition repeated All conditions must NOT match

FloatPayload

Field Type Label Description
values double repeated

GeoBoundingBox

Field Type Label Description
top_left GeoPoint north-west corner
bottom_right GeoPoint south-east corner

GeoPayload

Field Type Label Description
values GeoPoint repeated

GeoPoint

Field Type Label Description
lon double
lat double

GeoRadius

Field Type Label Description
center GeoPoint Center of the circle
radius float In meters

GetPoints

Field Type Label Description
collection_name string name of the collection
ids PointId repeated List of points to retrieve
with_vector bool optional Return point vector with the result.
with_payload WithPayloadSelector Options for specifying which payload to include or not

GetResponse

Field Type Label Description
result RetrievedPoint repeated
time double Time spent to process

HasIdCondition

Field Type Label Description
has_id PointId repeated

IntegerPayload

Field Type Label Description
values int64 repeated

KeywordPayload

Field Type Label Description
values string repeated

Match

Field Type Label Description
keyword string Match string keyword
integer int64 Match integer

Payload

Field Type Label Description
keyword KeywordPayload
integer IntegerPayload
float FloatPayload
geo GeoPayload

PayloadExcludeSelector

Field Type Label Description
exclude string repeated List of payload keys to exclude from the result

PayloadIncludeSelector

Field Type Label Description
include string repeated List of payload keys to include into result

PointId

Field Type Label Description
num uint64 Numerical ID of the point
uuid string UUID

PointStruct

Field Type Label Description
id PointId
vector float repeated
payload PointStruct.PayloadEntry repeated

PointStruct.PayloadEntry

Field Type Label Description
key string
value Payload

PointsIdsList

Field Type Label Description
ids PointId repeated

PointsOperationResponse

Field Type Label Description
result UpdateResult
time double Time spent to process

PointsSelector

Field Type Label Description
points PointsIdsList
filter Filter

Range

Field Type Label Description
lt double optional
gt double optional
gte double optional
lte double optional

RecommendPoints

Field Type Label Description
collection_name string name of the collection
positive PointId repeated Look for vectors closest to those
negative PointId repeated Try to avoid vectors like this
filter Filter Filter conditions - return only those points that satisfy the specified conditions
top uint64 Max number of result
with_vector bool optional Return point vector with the result.
with_payload WithPayloadSelector Options for specifying which payload to include or not
params SearchParams Search config

RecommendResponse

Field Type Label Description
result ScoredPoint repeated
time double Time spent to process

RetrievedPoint

Field Type Label Description
id PointId
payload RetrievedPoint.PayloadEntry repeated
vector float repeated

RetrievedPoint.PayloadEntry

Field Type Label Description
key string
value Payload

ScoredPoint

Field Type Label Description
id PointId Point id
payload ScoredPoint.PayloadEntry repeated Payload
score float Similarity score
vector float repeated Vector
version uint64 Last update operation applied to this point

ScoredPoint.PayloadEntry

Field Type Label Description
key string
value Payload

ScrollPoints

Field Type Label Description
collection_name string
filter Filter Filter conditions - return only those points that satisfy the specified conditions
offset PointId optional Start with this ID
limit uint32 optional Max number of result
with_vector bool optional Return point vector with the result.
with_payload WithPayloadSelector Options for specifying which payload to include or not

ScrollResponse

Field Type Label Description
next_page_offset PointId optional Use this offset for the next query
result RetrievedPoint repeated
time double Time spent to process

SearchParams

Field Type Label Description
hnsw_ef uint64 optional Params relevant to HNSW index. Size of the beam in a beam-search. Larger the value - more accurate the result, more time required for search.

SearchPoints

Field Type Label Description
collection_name string name of the collection
vector float repeated vector
filter Filter Filter conditions - return only those points that satisfy the specified conditions
top uint64 Max number of result
with_vector bool optional Return point vector with the result.
with_payload WithPayloadSelector Options for specifying which payload to include or not
params SearchParams Search config

SearchResponse

Field Type Label Description
result ScoredPoint repeated
time double Time spent to process

SetPayloadPoints

Field Type Label Description
collection_name string name of the collection
wait bool optional Wait until the changes have been applied?
payload SetPayloadPoints.PayloadEntry repeated New payload values
points PointId repeated List of point to modify

SetPayloadPoints.PayloadEntry

Field Type Label Description
key string
value Payload

UpdateResult

Field Type Label Description
operation_id uint64 Number of operation
status UpdateStatus Operation status

UpsertPoints

Field Type Label Description
collection_name string name of the collection
wait bool optional Wait until the changes have been applied?
points PointStruct repeated

WithPayloadSelector

Field Type Label Description
enable bool If true - return all payload, if false - none
include PayloadIncludeSelector
exclude PayloadExcludeSelector

UpdateStatus

Name Number Description
UnknownUpdateStatus 0
Acknowledged 1 Update is received, but not processed yet
Completed 2 Update is applied and ready for search

Points

Method Name Request Type Response Type Description
Upsert UpsertPoints PointsOperationResponse Perform insert + updates on points. If point with given ID already exists - it will be overwritten.
Delete DeletePoints PointsOperationResponse Delete points
Get GetPoints GetResponse Retrieve points
SetPayload SetPayloadPoints PointsOperationResponse Set payload for points
DeletePayload DeletePayloadPoints PointsOperationResponse Delete specified key payload for points
ClearPayload ClearPayloadPoints PointsOperationResponse Remove all payload for specified points
CreateFieldIndex CreateFieldIndexCollection PointsOperationResponse Create index for field in collection
DeleteFieldIndex DeleteFieldIndexCollection PointsOperationResponse Delete field index for collection
Search SearchPoints SearchResponse Retrieve closest points based on vector similarity and given filtering conditions
Scroll ScrollPoints ScrollResponse Iterate over all or filtered points points
Recommend RecommendPoints RecommendResponse Look for the points which are closer to stored positive examples and at the same time further to negative examples.

Top

qdrant.proto

HealthCheckReply

Field Type Label Description
title string
version string

HealthCheckRequest

Qdrant

Method Name Request Type Response Type Description
HealthCheck HealthCheckRequest HealthCheckReply

Scalar Value Types

.proto Type Notes C++ Java Python Go C# PHP Ruby
double double double float float64 double float Float
float float float float float32 float float Float
int32 Uses variable-length encoding. Inefficient for encoding negative numbers if your field is likely to have negative values, use sint32 instead. int32 int int int32 int integer Bignum or Fixnum (as required)
int64 Uses variable-length encoding. Inefficient for encoding negative numbers if your field is likely to have negative values, use sint64 instead. int64 long int/long int64 long integer/string Bignum
uint32 Uses variable-length encoding. uint32 int int/long uint32 uint integer Bignum or Fixnum (as required)
uint64 Uses variable-length encoding. uint64 long int/long uint64 ulong integer/string Bignum or Fixnum (as required)
sint32 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int int32 int integer Bignum or Fixnum (as required)
sint64 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long int64 long integer/string Bignum
fixed32 Always four bytes. More efficient than uint32 if values are often greater than 2^28. uint32 int int uint32 uint integer Bignum or Fixnum (as required)
fixed64 Always eight bytes. More efficient than uint64 if values are often greater than 2^56. uint64 long int/long uint64 ulong integer/string Bignum
sfixed32 Always four bytes. int32 int int int32 int integer Bignum or Fixnum (as required)
sfixed64 Always eight bytes. int64 long int/long int64 long integer/string Bignum
bool bool boolean boolean bool bool boolean TrueClass/FalseClass
string A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode string string string String (UTF-8)
bytes May contain any arbitrary sequence of bytes. string ByteString str []byte ByteString string String (ASCII-8BIT)