Commit Graph

35 Commits

Author SHA1 Message Date
George
9f9c500bf4 Merge branch 'dev' into drop-python3.9 2025-12-04 11:58:23 +07:00
George
458646cb95 tests: speed up tests (#1130)
* fix: do not try to check compatibility in test_client_init

* tests: remove sparse-code vectors (10_000 dim), remove euclid recommend methods as non-applicable for sparse

* tests: add payload indexes to query group test

* debug: add durations=0 to pytest

* debug: test only test query group

* debug: try adding payload indexes to query group test

* rollback test launch
2025-12-04 11:22:46 +07:00
George Panchuk
698db1d33f new: drop python3.9, replace union and optional with | where possible 2025-11-14 03:20:36 +07:00
George
c1df3a7c28 tests: relax test comparison for distance matrix test (#1042) 2025-07-15 15:39:37 +02:00
George
fbf72f4a31 new: relax test comparison (#1039)
* new: relax test comparison

* rollback: rollback is context search addition

* rollback: return redundant parentheses

* Sparse no overlap distance computation (#1040)

* fix: fix sparse distance computation for recommend, discovery and context queries

* fix: use empty_is_zero

* add zero case to score cmp

* add snapshot saving on failure

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-07-15 13:30:23 +02:00
Luis Cossío
e704e4fe02 [score boosting] Support datetime expressions (#930)
* gen grpc

* generate rest client

* use float for formula evaluation, match multivalue behavior

* support datetime expressions

* populate inspection cache

* review remarks

* support datetime_key

* fix: address AI comments

* fix: regen inspection cache

* regen inspection cache with pydantic 2.10

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
2025-04-03 14:51:14 +03:00
Luis Cossío
983fce8a1b [score boosting] support decay functions (#929)
* update grpc client

* generate rest client

* add decay expressions evaluation and fixtures

* add conversions

* validate in formula and fix test

* add fixtures

* new: update inspection cache

* Review remarks

* review nit

* fix rebase

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
2025-03-24 15:38:22 -03:00
George
baa3e264c8 Drop python 3.8 (#848)
* new: drop python 3.8 support, update type hints

* fix: remove 3.8 from ci

* fix: update type hints

* fix: make netlify use python3.10

* fix: try python3.9 for sphinx

* debug: try updating sphinx

* new: bump ffastembed to 0.4.2

* fix: install numpy<2 for mypy

* fix: install numpy via poetry
2024-11-15 15:54:32 +01:00
George
c152e3c7bf Agnostic local inference (#799)
* new: update previous qdrant version

* new: rollback rest client as types moved to the core

* new: add model_fields_set pydantic compat

* new: add Document to models

* new: add local inference inspectors

* new: add inference functionality to qdrant client

* fix: fix type hints

* fix: fix type hints

* fix: fix type hints

* new: remove _cloud_inference arg from methods

* refactor

* tests: update tests, add path check

* fix: fix type hint

* fix: fix type hint

* fix: resolve plain query

* fix: remove redundant checks

* fix: fix resolve query doc

* tests: add test case for several docs in prefetch

* fix: fix tests

* new: forbid cloud inference with local mode

* fix: fix is_query assignment, fix embed dict

* tests: fix sparse vector values comparison

* fix: regen async

* fix: embed batch

* tests: extend local inference tests

* tests: add update operations and update vectors tests

* tests: remove redundant tests, add plain query points test

* fix: fix type hint

* fix: add missing import

* fix: yet another type hint fix

* tests: add query points groups and query batch points tests

* fix: do not try to embed a vector, comment missed disabled test stmt

* new: add late interaction embeddings without add and query methods

* fix: fix LITE import

* fix: fix problem with retrieving not available path, enable tests

* fix: fix exception message

* refactor: update type hint

* fix: update type hints

* chore: add docstrings to embed methods

* refactor: simplify inspector init

* fix: fix type hint

* new: reuse parser, fix type import

* refactor: rename parser, add docstrings

* fix: fix type hint

* fix: fix type hint

* fix: fix type hint

* refactor: smol refactor, docstrings

* fix: fix inspect embed

* deprecation warning for grpc.PointStruct

* new: populate inspections cache (#823)

* new: populate inspections cache

* fix: fix type hints

* fix: regen async

* fix: fix type hint

* fix: fix typo in filename

* fix: fix points check

* fix: fix type hint
2024-10-29 15:48:23 +01:00
Luis Cossío
a6e2324884 chore: Update clients (#758)
* update rest client

* update grpc client with grpcio 1.65.4

* update grpc client with grpcio==1.49.1

* generate grpc client with grpcio==1.48.2

* generate grpc client with grpcio==1.66.1 and mypy-protobuf extension

* add mypy-protobuf to pyproject

* relax version for grpcio, but keep grpcio-tools at 1.48.2

* improve generation script

* fix bash script comment

Co-authored-by: George <george.panchuk@qdrant.tech>

* use old grpcio to generate client

* rollbakc mypy-protobuf

* regenerate with distance matrix

* new: update clients (#774)

* new: update clients

* rollback grpcio version

* feature: Facets (#759)

* facet in local_collection.py

* replace usages of calculate_payload_mask

* - qdrant remote
- qdrant base
- qdrant client
- conversions

* congruence tests + local mode fixes

* generate async client

* add type stubs and misc fixes

* fix mypy in Python 3.8

* generate async client

* review remarks

* gen async client

* update for bool facets

* Feature: Distance Matrix API (#769)

* add remote impls

* add client impl

* regen async client

* start local

* regen async

* local mode

* start congruence

* Fix local mode

* regen async of course

* test filtering

* fix min samples count

* simplify comparaison loop

* simplify samples loop

* add rest/gRPC conversion tests

* fix conversions + tests

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2024-10-05 23:03:13 +02:00
George
36995d41e5 fix: fix update_vectors in local mode, fix record comparison in tests (#787) 2024-09-24 16:15:33 +02:00
Hossam Hagag
c92a0432f3 Refactor recreate collection (#719)
* deprecated recreate_collection from tests

* checked if collection exists before deletion

* reverted deletion of recreate_collection

* added missing collection_exsits before deletion

* nit if condition

* fix type hint in local_collection

* removed run mypy on async client generator

* added checks for create_collection if exists

* nit change collection name

* add else to create_collection

* nit

* nit

* refactoring: remove redundant checks

* fix: do not test collection_exists through itself

* refactoring: remove redundant collection exists checks

* fix: remove redundant collection exists checks

* refactoring: remove redundant collection checks

* refactoring: remove redundant collection existence check in test sparse search

* refactoring: remove redundant checks in test_updates

* refactoring: remove redundant checks

* fix: remove redundant checks

* refactoring: remove redundant checks

* refactoring: remove redundant imports

* new: add recreate collection test

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
2024-08-09 23:33:31 +02:00
George
541d130b1a fix: do not compare indexed count in tests (#702) 2024-07-22 20:39:29 +03:00
Arnaud Gourlay
02e85a8094 unpack query responses when comparing 2024-06-28 17:57:30 +02:00
Luis Cossío
c3e8ec00e5 Local mode for Query 2024-06-27 19:46:25 +02:00
Andrey Vasnetsov
815a784469 Update openapi v1 10 x (#622)
* update api with conversions

* introduce IDF modifier for sparse vectors in local mode

* generate stuff

* regen async

* Update tests/fixtures/points.py

* fix: address review comments

* upd models

* fix: update grpc overwrite payload conversion, fix order by conversion (#653)

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
2024-06-20 17:55:35 +02:00
Andrey Vasnetsov
67103fca36 do not use vectors_count in test and in local mode (#600)
* do not use vectors_count in test and in local mode

* fix: remove indexed vectors count comparison in sparse vector tests

* tests: return tests for full snapshot, increase timeout

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
2024-04-18 17:26:33 +02:00
George
af2f2e92b1 new: check for nan vectors in local mode (#560)
* new: check for nan vectors in local mode

* tests: add tests for search with query containing nans

* refactoring: move sparse nan test to test_sparse_search

* tests: add tests for queries with nans in recommendation api

* tests: add tests for query with nan in discovery

* refactoring: update imports in discovery tests
2024-04-02 17:05:40 +02:00
Luis Cossío
332777db2a Local mode of order_by parameter in scroll + datetime support (#491)
* generate rest client (only points_api)

* generate grpc client (points part only)

* add local mode implementation

* update collections_api.py

* add grpc conversions

* fix problematic StartFrom Union order

* add basic congruence test

* route order_by in qdrant clients

* fix local mode `start_from` logic

* add library stubs for dateutil

* generate async client

* generate the rest of the rest client 👻

* test datetime values too

* add int Range and int StartFrom

* update points.proto and points_pb2.py

* fix conversion of range interface

* add conversion fixtures and use better conversion of datetime to timestamp

* remove integer range

* restore `value_by_key()`

* OrderBy grpc to rest conversions

* generate async client

* use `OrderByInterface` instead of only `OrderBy`

* - use equivalent qdrant core datetime parsing,
- use type alias for OrderingValue, instead of custom class,
- better random date generation

* Drop custom datetime parser implementation

* nit fixes

* fix flakiness of the congruence test by subsorting

* better datetime to timestamp conversion

* uncomment conversion fixtures, add Direction

* restore poetry.lock and remove dateutil dep

* poetry lock --no-update

* rename datetime.py to datetime_utils.py

* add StrictInt to StartFrom union

* use more date formats in fixtures

* move conversion test to `test_validate_conversions.py`

* drop `%:z` formats

* add more complex timezones to payload fixtures, fix deserialization

* update datetime parsing test
2024-03-01 09:51:39 -03:00
George
a2adc2d2ad new: deprecate upload records, update tests, prohibit migration of co… (#447)
* new: deprecate upload records, update tests, prohibit migration of collections with custom shards

* Update qdrant_client/qdrant_client.py

* Update qdrant_client/async_qdrant_client.py

* new: replace autogenerated int ids with uuids (#448)

* fix: remove redundant kwargs

* fix: regen async

* fix: remove redundant import

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2024-01-19 20:11:31 +01:00
Andrey Vasnetsov
dd4b91c23e Manhattan and shard key (#391)
* support for manhattan distance

* conversion for shard_key_selector

* use shard_key_selector in read api

* allow shard_keys

* regen async

* fix tests

* type checker

* type checker

* tests for custom sharding

* async

* review fixes

* review fixes

* review fixes
2023-12-04 17:05:27 +01:00
Arnaud Gourlay
016032c28f Sparse vectors API and local mode (#378)
* draft

* thread sparse_vectors_config through

* test ingest sparse data

* make async_client consistent for create collection

* regen async

* align client hierarchy

* do not give up

* disable async consistency check for now

* regen

* workaround async client breakage

* generate gRPC

* new healtcheck proto files

* fix gen branch

* fix grpc generation - had to reinstall to downgrade grpcio

* bump test docker image

* wait for update when migrating

* regen REST + update config

* add sparse search congruence

* congruence scroll sparse vectors

* congruence test retrieve sparse

* more sparse tests and waiting on remote

* remove healthcheck service

* misc

* reinstate len check

* fix scoring if no overlap

* sparse distance as numpy float without rounding

* fix payload selector

* fix payload exclude

* dedicated collection name to not collide with openapi engine tests

* fix with_vector, scoring no overlap, closeness assetion

* convert float only once

* regen gRPC to exclude health_check

* code review cleanup

* validate and sort sparse vector before persistence

* regen REST

* test deletes and update with fix

* fix async client

* fix batch size

* fix async client

* don't give up

* remove validated todo

* fix sparse vectors loading test

* fix sparse vectors loading test

* rm empty

* mention sparse vectors in the fake local info

* fix upload collection wait propagation

* fix async. init_from tests

* resync async

* rework test sparse vector generation

* regen clients

* update to dev

* update REST models

* async client

* do not change it yet

* fix migration test

* do not create a sparse configuration by default

* fix backward compatibility by disabling serialization of none fields

* fix type imports

* regen gRPC

* test BatchVectorStruct REST to gRPC conversion

* test gRPC->REST conversion

* sparse grpc tests and conversions fix

* generate async client

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2023-12-02 20:45:49 +01:00
Luis Cossío
d3332eae2a Add Discovery API and local mode (#368)
* add score calculations

* fix type errors, route scorers accordingly

* generate grpc client (only discovery stuff)

* generate rest client

* discover preprocessing + local mode handling

* implement discover for remote client

* add failing tests, fix some stuff

* update rest client and local mode

* fixes and more tests

* fix mypy errors

* generate async client

* tiny edit docstring

* simplify custom distance calculation,
fix euclidean case for recommend best score

* fix python version for generating grpc client

* update grpc (select only discovery changes)

* generate grpc: add also ShardKey changes,
fix conversion changes for deprecated PointsUpdateOperation operations

* fix coverage on conversions

* generate async client

* generate async client with python 3.11

* change dict -> typings.Dict

* add deprecated field handling for grpc.PointsUpdateOperation

* update grpc client

* fix grpc conversion issues and review comments

* increase coverage

* also test for prefer_grpc = True

* fix has_field

* unflake test_context_many_pairs

* don't convert if None in model conversions

* move docstring to qdrant_client.py

* handle request timeout on all other interfaces

* oopsie: define timeout param in search_batch

* generate async client
2023-11-22 16:02:41 -03:00
George
bf1c593e93 fix: add missing parameters in recommend batch (#338) 2023-10-15 16:04:21 +02:00
Andrey Vasnetsov
bb8c442169 Recommendation api update + local mode (#314)
* WIP: recommendation api update

* implement local new reco + add new tests

* refactor calculate_best_scores, normalize score assertion

* fix raw vectors in new recommend

* fix mypy errors, add new reco on group recommend

* fix more lints from ci

* fix pyright lint

* formattng

* Add descriptions to enums

* no mutable default argument

* edit score comparison precision based on magnitude

* Address review comments

* remove custom message for type ignore

* add coverage test

* improve coverage test a little

* add more fixtures for conversion test

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2023-10-05 03:38:19 +02:00
Andrey Vasnetsov
578db8107a add tests for vector operations (#275)
* add tests for vector operations

* rename vectors->points in vector update API

* fix tests
2023-09-04 18:14:19 +02:00
George
897a7ff6c9 Migration tool (#243)
* new: migration tool v1

* new: add docstrings

* fix: fix collection comparison, add tests

* fix: fix mypy complaints

* tests: test multiple collections scenario

* new: add migrate method to client's interface

* new: update migration tool

* new: simplify collisions

* new: update migration logic and tests

* fix: fix mypy complaints

* tests: replace tokenizer type in favour of backward compatibility tests
2023-08-21 11:44:26 +02:00
Kacper Łukawski
c7f221e2a4 Fix cosine normalization (#213)
* Enable vectors comparison in tests (round to 1e-3)

* Normalize vectors if cosine distance is used

* fix: compare scroll results

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
2023-07-23 00:06:18 +04:00
generall
6616e1838d fix types 2023-07-22 11:58:37 +02:00
generall
aacd291ff3 more timeouts 2023-07-21 23:15:25 +02:00
Kacper Łukawski
b9a7a15cee [WIP] Enable testing on macos (#199)
* Enable testing on windows and macos

* Add platform to the name

* Add Docker on MacOS

* Temporarily disable tests on Windows

* Explicitly set ports to be opened

* Add problematic case for MacOS

* Increase the timeout in tests
2023-07-07 13:31:32 +02:00
George
a3778e4230 new: add grpc endpoints (#176)
* new: add grpc endpoints

* fix: fix payload index conversions

* fix: fix payload schema fixtures, vector param conversion

* fix: fix collection info conversion

* refactoring: remove ipdb
2023-06-09 18:38:00 +02:00
Andrey Vasnetsov
329b4150cf v1.2.0 (#173)
* wip: interface for new APIs

* upd docker version

* remove duplicated comments + extent interface

* start implementing remote client + remove duplicated comments there

* implement remote methods

* quantization converter

* extend coverege

* wip: saving and loading of optional vectors

* update and delete vectors test

* rm unused imports

* nested filters

* fix mypy

* fix pyright

* wip: add search groups and recommend groups (#174)

* wip: add search groups and recommend groups

* fix: fix signature

* fix: fix mypy

* simplify group-by and condition checks

* fix tests

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* add match except condition + improve group-by tests

* fix: lock typing exchanges due to broken release

* tests: add maybe and maybe_null checks to group tests

* new: add new methods to type stubs

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
2023-05-24 11:06:10 +02:00
Andrey Vasnetsov
c6db1e092d upd to qdrant v1.1.1 + fix isEmpty in local mode (#152)
* upd to qdrant v1.1.1 + fix isEmpty in local mode

* fix mypy
2023-04-11 20:33:00 +02:00
Andrey Vasnetsov
392e3b6d1f Local qdrant (#137)
* base class for qdrant

* WIP: implement local qdrant client

* fix mypy

* fix mypy

* search tests

* search tests

* fix mypy

* scroll test

* filters: fixtures, tests, and fixes

* fix types

* fix types

* fix types

* fix: fix local CollectionInfo, add __test__ to avoid pytest complaints, fix typo

* fix: fix typo in import

* tests: add local upload tests (#139)

* new: make local collection info more similar to remote one

* new: add local upsert tests

* tests: moved compare collections

* scroll tests

* recommendations test

* tests: fix vector comparison in utils, add retrieve tests

* fix: fix types

* persistence test

* fix: fix types

* fix: fix db path creation

* tests: add delete points tests, refactoring (#140)

* skip local tests on old version

* test aliases

* test count

* tests: add delete payload tests, move set and overwrite payload tests

* tests: fix pytest warning

* cover some more stuff with tests

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
2023-03-28 14:57:05 +02:00