mirror of https://github.com/qdrant/qdrant.git synced 2026-08-04 09:01:00 -05:00

Files

Gabriel Velo c15981092a [WIP] [real-time index] Implement payloadstorage for structpayloadindex (#642 )

* [real-time index] Extend FieldIndex enum and StructPayloadIndex with method from PayloadStorage

* [real-time index] add missing remove_point methods

* [real-time index] add new index to FieldIndex enum

* fix compile

* are you happy fmt

* merge load and remove

* fix test generics

* decrement points count

* remove from histogram

* simplify histogram usage

* [real-time index] remove old tests and fix clippy warnings

* histogram: method to derive range by size (#657)

* [real-time index] add histogram based payload_blocks implementation.

* payload blocks

* fmt

* clippy

* [real-time index] refactor Segment to use PayloadIndex instead of PayloadStorage.

* fix tests

* fmt

* clippy

* rename indexes

* remove redundent params

* add struct payload deletion test + fix delete payload in map index

* remove payload threshold

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

2022-06-06 17:14:20 +02:00

40 KiB

Raw Blame History

Protocol Documentation

collections.proto
collections_service.proto
- Collections
json_with_int.proto
points.proto
points_service.proto
- Points
qdrant.proto
Scalar Value Types

Top

collections.proto

AliasOperations

Field	Type	Label	Description
create_alias	CreateAlias
rename_alias	RenameAlias
delete_alias	DeleteAlias

ChangeAliases

Field	Type	Label	Description
actions	AliasOperations	repeated	List of actions
timeout	uint64	optional	Wait timeout for operation commit in seconds, if not specified - default value will be supplied

CollectionConfig

Field	Type	Description
params	CollectionParams	Collection parameters
hnsw_config	HnswConfigDiff	Configuration of vector index
optimizer_config	OptimizersConfigDiff	Configuration of the optimizers
wal_config	WalConfigDiff	Configuration of the Write-Ahead-Log

CollectionDescription

Field	Type	Label	Description
name	string		Name of the collection

CollectionInfo

Field	Type	Label	Description
status	CollectionStatus		operating condition of the collection
optimizer_status	OptimizerStatus		status of collection optimizers
vectors_count	uint64		number of vectors in the collection
segments_count	uint64		Number of independent segments
disk_data_size	uint64		Used disk space
ram_data_size	uint64		Used RAM (not implemented)
config	CollectionConfig		Configuration
payload_schema	CollectionInfo.PayloadSchemaEntry	repeated	Collection data types

CollectionInfo.PayloadSchemaEntry

Field	Type	Label	Description
key	string
value	PayloadSchemaInfo

CollectionOperationResponse

Field	Type	Label	Description
result	bool		if operation made changes
time	double		Time spent to process

CollectionParams

Field	Type	Description
vector_size	uint64	Size of the vectors
distance	Distance	Distance function used for comparing vectors
shard_number	uint32	Number of shards in collection
on_disk_payload	bool	If true - point's payload will not be stored in memory

CreateAlias

Field	Type	Label	Description
collection_name	string		Name of the collection
alias_name	string		New name of the alias

CreateCollection

Field	Type	Label	Description
collection_name	string		Name of the collection
vector_size	uint64		Size of the vectors
distance	Distance		Distance function used for comparing vectors
hnsw_config	HnswConfigDiff	optional	Configuration of vector index
wal_config	WalConfigDiff	optional	Configuration of the Write-Ahead-Log
optimizers_config	OptimizersConfigDiff	optional	Configuration of the optimizers
shard_number	uint32	optional	Number of shards in the collection, default = 1
on_disk_payload	bool	optional	If true - point's payload will not be stored in memory
timeout	uint64	optional	Wait timeout for operation commit in seconds, if not specified - default value will be supplied

DeleteAlias

Field	Type	Label	Description
alias_name	string		Name of the alias

DeleteCollection

Field	Type	Label	Description
collection_name	string		Name of the collection
timeout	uint64	optional	Wait timeout for operation commit in seconds, if not specified - default value will be supplied

GetCollectionInfoRequest

Field	Type	Label	Description
collection_name	string		Name of the collection

GetCollectionInfoResponse

Field	Type	Label	Description
result	CollectionInfo
time	double		Time spent to process

HnswConfigDiff

Field	Type	Label	Description
m	uint64	optional	Number of edges per node in the index graph. Larger the value - more accurate the search, more space required.
ef_construct	uint64	optional	Number of neighbours to consider during the index building. Larger the value - more accurate the search, more time required to build index.
full_scan_threshold	uint64	optional	Minimal size (in KiloBytes) of vectors for additional payload-based indexing. If payload chunk is smaller than `full_scan_threshold` additional indexing won't be used - in this case full-scan search should be preferred by query planner and additional indexing is not required. Note: 1Kb = 1 vector of size 256

ListCollectionsRequest

ListCollectionsResponse

Field	Type	Label	Description
collections	CollectionDescription	repeated
time	double		Time spent to process

OptimizerStatus

Field	Type	Label	Description
ok	bool
error	string

OptimizersConfigDiff

Field	Type	Label	Description
deleted_threshold	double	optional	The minimal fraction of deleted vectors in a segment, required to perform segment optimization
vacuum_min_vector_number	uint64	optional	The minimal number of vectors in a segment, required to perform segment optimization
default_segment_number	uint64	optional	Target amount of segments optimizer will try to keep. Real amount of segments may vary depending on multiple parameters:

Amount of stored points. - Current write RPS.

It is recommended to select default number of segments as a factor of the number of search threads, so that each segment would be handled evenly by one of the threads. | | max_segment_size | uint64 | optional | Do not create segments larger this size (in KiloBytes). Large segments might require disproportionately long indexation times, therefore it makes sense to limit the size of segments.

If indexation speed have more priority for your - make this parameter lower. If search speed is more important - make this parameter higher. Note: 1Kb = 1 vector of size 256 | | memmap_threshold | uint64 | optional | Maximum size (in KiloBytes) of vectors to store in-memory per segment. Segments larger than this threshold will be stored as read-only memmaped file. To enable memmap storage, lower the threshold Note: 1Kb = 1 vector of size 256 | | indexing_threshold | uint64 | optional | Maximum size (in KiloBytes) of vectors allowed for plain index. Default value based on https://github.com/google-research/google-research/blob/master/scann/docs/algorithms.md Note: 1Kb = 1 vector of size 256 | | flush_interval_sec | uint64 | optional | Interval between forced flushes. | | max_optimization_threads | uint64 | optional | Max number of threads, which can be used for optimization. If 0 - NUM_CPU - 1 will be used |

PayloadSchemaInfo

Field	Type	Label	Description
data_type	PayloadSchemaType		Field data type

RenameAlias

Field	Type	Label	Description
old_alias_name	string		Name of the alias to rename
new_alias_name	string		Name of the alias

UpdateCollection

Field	Type	Label	Description
collection_name	string		Name of the collection
optimizers_config	OptimizersConfigDiff	optional	New configuration parameters for the collection
timeout	uint64	optional	Wait timeout for operation commit in seconds, if not specified - default value will be supplied

WalConfigDiff

Field	Type	Label	Description
wal_capacity_mb	uint64	optional	Size of a single WAL block file
wal_segments_ahead	uint64	optional	Number of segments to create in advance

CollectionStatus

Name	Number	Description
UnknownCollectionStatus	0
Green	1	All segments are ready
Yellow	2	Optimization in process
Red	3	Something went wrong

Distance

Name	Number	Description
UnknownDistance	0
Cosine	1
Euclid	2
Dot	3

PayloadSchemaType

Name	Number	Description
UnknownType	0
Keyword	1
Integer	2
Float	3
Geo	4

Top

collections_service.proto

Collections

Method Name	Request Type	Response Type	Description
Get	GetCollectionInfoRequest	GetCollectionInfoResponse	Get detailed information about specified existing collection
List	ListCollectionsRequest	ListCollectionsResponse	Get list name of all existing collections
Create	CreateCollection	CollectionOperationResponse	Create new collection with given parameters
Update	UpdateCollection	CollectionOperationResponse	Update parameters of the existing collection
Delete	DeleteCollection	CollectionOperationResponse	Drop collection and all associated data
UpdateAliases	ChangeAliases	CollectionOperationResponse	Update Aliases of the existing collection

Top

json_with_int.proto

ListValue

ListValue is a wrapper around a repeated field of values.

The JSON representation for ListValue is JSON array.

Field	Type	Label	Description
values	Value	repeated	Repeated field of dynamically typed values.

Struct

Struct represents a structured data value, consisting of fields which map to dynamically typed values. In some languages, Struct might be supported by a native representation. For example, in scripting languages like JS a struct is represented as an object. The details of that representation are described together with the proto support for the language.

The JSON representation for Struct is JSON object.

Field	Type	Label	Description
fields	Struct.FieldsEntry	repeated	Unordered map of dynamically typed values.

Struct.FieldsEntry

Field	Type	Label	Description
key	string
value	Value

Value

Value represents a dynamically typed value which can be either null, a number, a string, a boolean, a recursive struct value, or a list of values. A producer of value is expected to set one of that variants, absence of any variant indicates an error.

The JSON representation for Value is JSON value.

Field	Type	Description
null_value	NullValue	Represents a null value.
double_value	double	Represents a double value.
integer_value	int64	Represents an integer value
string_value	string	Represents a string value.
bool_value	bool	Represents a boolean value.
struct_value	Struct	Represents a structured value.
list_value	ListValue	Represents a repeated `Value`.

NullValue

NullValue is a singleton enumeration to represent the null value for the Value type union.

The JSON representation for NullValue is JSON null.

Name	Number	Description
NULL_VALUE	0	Null value.

Top

points.proto

ClearPayloadPoints

Field	Type	Label	Description
collection_name	string		name of the collection
wait	bool	optional	Wait until the changes have been applied?
points	PointsSelector		Affected points

Condition

Field	Type	Label	Description
field	FieldCondition
isEmpty	IsEmptyCondition
hasId	HasIdCondition
filter	Filter

CreateFieldIndexCollection

Field	Type	Label	Description
collection_name	string		name of the collection
wait	bool	optional	Wait until the changes have been applied?
field_name	string		Field name to index
field_type	FieldType	optional	Field type.

DeleteFieldIndexCollection

Field	Type	Label	Description
collection_name	string		name of the collection
wait	bool	optional	Wait until the changes have been applied?
field_name	string		Field name to delete

DeletePayloadPoints

Field	Type	Label	Description
collection_name	string		name of the collection
wait	bool	optional	Wait until the changes have been applied?
keys	string	repeated	List of keys to delete
points	PointId	repeated	Affected points

DeletePoints

Field	Type	Label	Description
collection_name	string		name of the collection
wait	bool	optional	Wait until the changes have been applied?
points	PointsSelector		Affected points

FieldCondition

Field	Type	Description
key	string
match	Match	Check if point has field with a given value
range	Range	Check if points value lies in a given range
geo_bounding_box	GeoBoundingBox	Check if points geo location lies in a given area
geo_radius	GeoRadius	Check if geo point is within a given radius
values_count	ValuesCount	Check number of values for a specific field

Filter

Field	Type	Label	Description
should	Condition	repeated	At least one of those conditions should match
must	Condition	repeated	All conditions must match
must_not	Condition	repeated	All conditions must NOT match

GeoBoundingBox

Field	Type	Label	Description
top_left	GeoPoint		north-west corner
bottom_right	GeoPoint		south-east corner

GeoPoint

Field	Type	Label	Description
lon	double
lat	double

GeoRadius

Field	Type	Label	Description
center	GeoPoint		Center of the circle
radius	float		In meters

GetPoints

Field	Type	Label	Description
collection_name	string		name of the collection
ids	PointId	repeated	List of points to retrieve
with_vector	bool	optional	Return point vector with the result.
with_payload	WithPayloadSelector		Options for specifying which payload to include or not

GetResponse

Field	Type	Label	Description
result	RetrievedPoint	repeated
time	double		Time spent to process

HasIdCondition

Field	Type	Label	Description
has_id	PointId	repeated

IsEmptyCondition

Field	Type	Label	Description
key	string

Match

Field	Type	Description
keyword	string	Match string keyword
integer	int64	Match integer
boolean	bool	Match boolean

PayloadExcludeSelector

Field	Type	Label	Description
fields	string	repeated	List of payload keys to exclude from the result

PayloadIncludeSelector

Field	Type	Label	Description
fields	string	repeated	List of payload keys to include into result

PointId

Field	Type	Label	Description
num	uint64		Numerical ID of the point
uuid	string		UUID

PointStruct

Field	Type	Label
id	PointId
vector	float	repeated
payload	PointStruct.PayloadEntry	repeated

PointStruct.PayloadEntry

Field	Type	Label	Description
key	string
value	Value

PointsIdsList

Field	Type	Label	Description
ids	PointId	repeated

PointsOperationResponse

Field	Type	Label	Description
result	UpdateResult
time	double		Time spent to process

PointsSelector

Field	Type	Label	Description
points	PointsIdsList
filter	Filter

Range

Field	Type	Label
lt	double	optional
gt	double	optional
gte	double	optional
lte	double	optional

RecommendPoints

Field	Type	Label	Description
collection_name	string		name of the collection
positive	PointId	repeated	Look for vectors closest to those
negative	PointId	repeated	Try to avoid vectors like this
filter	Filter		Filter conditions - return only those points that satisfy the specified conditions
top	uint64		Max number of result
with_vector	bool	optional	Return point vector with the result.
with_payload	WithPayloadSelector		Options for specifying which payload to include or not
params	SearchParams		Search config
score_threshold	float	optional	If provided - cut off results with worse scores

RecommendResponse

Field	Type	Label	Description
result	ScoredPoint	repeated
time	double		Time spent to process

RetrievedPoint

Field	Type	Label
id	PointId
payload	RetrievedPoint.PayloadEntry	repeated
vector	float	repeated

RetrievedPoint.PayloadEntry

Field	Type	Label	Description
key	string
value	Value

ScoredPoint

Field	Type	Label	Description
id	PointId		Point id
payload	ScoredPoint.PayloadEntry	repeated	Payload
score	float		Similarity score
vector	float	repeated	Vector
version	uint64		Last update operation applied to this point

ScoredPoint.PayloadEntry

Field	Type	Label	Description
key	string
value	Value

ScrollPoints

Field	Type	Label	Description
collection_name	string
filter	Filter		Filter conditions - return only those points that satisfy the specified conditions
offset	PointId	optional	Start with this ID
limit	uint32	optional	Max number of result
with_vector	bool	optional	Return point vector with the result.
with_payload	WithPayloadSelector		Options for specifying which payload to include or not

ScrollResponse

Field	Type	Label	Description
next_page_offset	PointId	optional	Use this offset for the next query
result	RetrievedPoint	repeated
time	double		Time spent to process

SearchParams

Field	Type	Label	Description
hnsw_ef	uint64	optional	Params relevant to HNSW index. Size of the beam in a beam-search. Larger the value - more accurate the result, more time required for search.

SearchPoints

Field	Type	Label	Description
collection_name	string		name of the collection
vector	float	repeated	vector
filter	Filter		Filter conditions - return only those points that satisfy the specified conditions
top	uint64		Max number of result
with_vector	bool	optional	Return point vector with the result.
with_payload	WithPayloadSelector		Options for specifying which payload to include or not
params	SearchParams		Search config
score_threshold	float	optional	If provided - cut off results with worse scores

SearchResponse

Field	Type	Label	Description
result	ScoredPoint	repeated
time	double		Time spent to process

SetPayloadPoints

Field	Type	Label	Description
collection_name	string		name of the collection
wait	bool	optional	Wait until the changes have been applied?
payload	SetPayloadPoints.PayloadEntry	repeated	New payload values
points	PointId	repeated	List of point to modify

SetPayloadPoints.PayloadEntry

Field	Type	Label	Description
key	string
value	Value

UpdateResult

Field	Type	Label	Description
operation_id	uint64		Number of operation
status	UpdateStatus		Operation status

UpsertPoints

Field	Type	Label	Description
collection_name	string		name of the collection
wait	bool	optional	Wait until the changes have been applied?
points	PointStruct	repeated

ValuesCount

Field	Type	Label
lt	uint64	optional
gt	uint64	optional
gte	uint64	optional
lte	uint64	optional

WithPayloadSelector

Field	Type	Description
enable	bool	If `true` - return all payload, if `false` - none
include	PayloadIncludeSelector
exclude	PayloadExcludeSelector

FieldType

Name	Number	Description
FieldTypeKeyword	0
FieldTypeInteger	1
FieldTypeFloat	2
FieldTypeGeo	3

UpdateStatus

Name	Number	Description
UnknownUpdateStatus	0
Acknowledged	1	Update is received, but not processed yet
Completed	2	Update is applied and ready for search

Top

points_service.proto

Points

Method Name	Request Type	Response Type	Description
Upsert	UpsertPoints	PointsOperationResponse	Perform insert + updates on points. If point with given ID already exists - it will be overwritten.
Delete	DeletePoints	PointsOperationResponse	Delete points
Get	GetPoints	GetResponse	Retrieve points
SetPayload	SetPayloadPoints	PointsOperationResponse	Set payload for points
DeletePayload	DeletePayloadPoints	PointsOperationResponse	Delete specified key payload for points
ClearPayload	ClearPayloadPoints	PointsOperationResponse	Remove all payload for specified points
CreateFieldIndex	CreateFieldIndexCollection	PointsOperationResponse	Create index for field in collection
DeleteFieldIndex	DeleteFieldIndexCollection	PointsOperationResponse	Delete field index for collection
Search	SearchPoints	SearchResponse	Retrieve closest points based on vector similarity and given filtering conditions
Scroll	ScrollPoints	ScrollResponse	Iterate over all or filtered points points
Recommend	RecommendPoints	RecommendResponse	Look for the points which are closer to stored positive examples and at the same time further to negative examples.

Top

qdrant.proto

HealthCheckReply

Field	Type	Label	Description
title	string
version	string

HealthCheckRequest

Qdrant

Method Name	Request Type	Response Type	Description
HealthCheck	HealthCheckRequest	HealthCheckReply

Scalar Value Types

.proto Type	Notes	C++	Java	Python	Go	C#	PHP	Ruby
double		double	double	float	float64	double	float	Float
float		float	float	float	float32	float	float	Float
int32	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
int64	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.	int64	long	int/long	int64	long	integer/string	Bignum
uint32	Uses variable-length encoding.	uint32	int	int/long	uint32	uint	integer	Bignum or Fixnum (as required)
uint64	Uses variable-length encoding.	uint64	long	int/long	uint64	ulong	integer/string	Bignum or Fixnum (as required)
sint32	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sint64	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.	int64	long	int/long	int64	long	integer/string	Bignum
fixed32	Always four bytes. More efficient than uint32 if values are often greater than 2^28.	uint32	int	int	uint32	uint	integer	Bignum or Fixnum (as required)
fixed64	Always eight bytes. More efficient than uint64 if values are often greater than 2^56.	uint64	long	int/long	uint64	ulong	integer/string	Bignum
sfixed32	Always four bytes.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sfixed64	Always eight bytes.	int64	long	int/long	int64	long	integer/string	Bignum
bool		bool	boolean	boolean	bool	bool	boolean	TrueClass/FalseClass
string	A string must always contain UTF-8 encoded or 7-bit ASCII text.	string	String	str/unicode	string	string	string	String (UTF-8)
bytes	May contain any arbitrary sequence of bytes.	string	ByteString	str	[]byte	ByteString	string	String (ASCII-8BIT)

40 KiB Raw Blame History Unescape Escape