Commit Graph

39 Commits

Author SHA1 Message Date
George Panchuk
693d1e48a0 new: replace union with | in client classes 2025-12-02 12:37:40 +07:00
George Panchuk
698db1d33f new: drop python3.9, replace union and optional with | where possible 2025-11-14 03:20:36 +07:00
George
ff7f584d33 new: update models, remove init_from and locks (#1100)
* new: update models, remove init_from and locks

* deprecate: remove init from tests

* deprecate: remove lock tests

* new: convert ascii_folding

* fix: fix type stub

* new: convert acorn

* new: convert shard key with fallback

* new: update grpcio and grpcio tools in generator (#1106)

* new: update grpcio and grpcio tools in generator

* fix: bind grpcio and tools versions to 1.62.0 in generator

* Remove deprecated methods (#1103)

* deprecate: remove old api methods

* deprecate: remove type stub for removed methods

* deprecate: remove old api methods from test_qdrant_client

* deprecate: replace search with query points in test_in_memory

* deprecate: replace search methods in fastembed mixin with query points

* deprecate: replace old api methods in test async qdrant client

* deprecate: replace search with query points in test delete points

* deprecate: replace discover and context with query points in test_discovery

* deprecate: replace recommend_groups with query_points_groups in test_group_recommend

* deprecate: replace search_groups in test_group_search

* deprecate: replace recommend with query points in test_recommendation

* deprecate: replace search with query points in test search

* deprecate: replace context and discover with query points in test sparse discovery

* deprecate: replace search with query points in test sparse idf search

* deprecate: replace recommend with query points in test sparse recommend

* deprecate: replace search with query points in test sparse search

* deprecate: replace missing search request with query request in qdrant_fastembed

* deprecate: replace search with query points in test multivector search queries

* deprecate: replace upload records with upload points in test_updates

* deprecate: remove redundant structs (#1104)

* deprecate: remove redundant structs

* fix: do not use removed conversions in local mode

* fix: remove redundant conversions, simplify types.QueryRequest

* deprecate: replace old style grpc vector conversion to a new one (#1105)

* deprecate: replace old style grpc vector conversion to a new one

* fix: ignore union attr in conversion

* review fixes

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: generall <andrey@vasnetsov.com>

* new: deprecate add, query, query_batch in fastembed mixin (#1102)

* new: deprecate add, query, query_batch in fastembed mixin

* 1.16 -> 1.17

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-11-11 21:17:08 +07:00
George
200020f19a wip: builtin bm25 support (#1060)
* wip: builtin bm25 support

* remove debug print

* new: add builtin embedder, add server version check for inference

* fix: regen async

* fix: fix fastembed import

* fix: regen async

* tests: add tests for builtin inference

* fix: fix fastembed checks, improve exceptions
2025-09-15 16:18:41 +08:00
George
42856424a6 new: move fastembed imports and constants into a class to support cus… (#994)
* new: move fastembed imports and constants into a class to support custom models and improve exceptions

* new: add is_supported model to FastEmbedMisc

* fix: fix types
2025-05-20 14:46:18 +04:00
George
ce4c51dd3c Default inference batch size (#907)
* fix: fix inference batch size propagation

* fix: propagate batch size to embed worker

* fix: fix type ignore comment

* fix: replace mocking class method with object method

* fix: add missing inference batch size

* fix: fix batch size propagation to worker
2025-02-21 12:47:57 +01:00
George
b4ae703d1b Local inference upload collection and upload records (#862)
* wip: add draft implementation of batch processing

* fix: embed dict and list of docs, remove redundant code

* new: regen async, small refactor

* refactor: add docstrings, rename methods

* Upload points local inference (#881)

* new: separate single and plural model embeddings

* fix: fix lazy embed models

* new: add inference object inspections to upload methods

* wip: local inference upload parallel

* new: add local inference to upload points and upload collection, refactor mixin

* fix: remove redundant code

* redundant import

* tests: check is query for query points batch

* refactor: refactor semi ordered map

* tests: add test for local inference with batches with docs and vectors

* tests: check the order of dict processing

* new: distinguish models by options

* fix: fix typing

* fix: fix types

* new: embed batches with different options

* tests: add tests for batch with different options

* fix: ignore ide incorrect type inspection

* tests: wait for points to be inserted

* fix: set threads to 1 in parallel inference

* new: adjust max internal batch size

* fix: fix type hints

* function to get embeddings size (#892)

* function to get embeddings size

* async client

* keep sync

* new: extend embedding size to support image and late interaction models

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>

* new: add local inference batch size (#894)

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2025-01-29 15:53:24 +01:00
Hossam Hagag
47ff758c56 refactor: Refactored the way of warnings (#864)
* refactor: Refactored the way of warnings

* remove unused imports

* fix: Fix stacklevel in warnings

* regenerated async

* nit

* Updated some warnings with show once

* fix: Fix stack levels

* Updated async

* Update qdrant_client/qdrant_remote.py

Co-authored-by: George <george.panchuk@qdrant.tech>

* Update async client

* Updated warnings

* Updated warnings

* Updated warnings

* fix: fix warning level

* fix: fix warning level in async

* fix: revert append payload condition

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
2025-01-06 16:44:06 +01:00
George
baa3e264c8 Drop python 3.8 (#848)
* new: drop python 3.8 support, update type hints

* fix: remove 3.8 from ci

* fix: update type hints

* fix: make netlify use python3.10

* fix: try python3.9 for sphinx

* debug: try updating sphinx

* new: bump ffastembed to 0.4.2

* fix: install numpy<2 for mypy

* fix: install numpy via poetry
2024-11-15 15:54:32 +01:00
George
8e4279d1d6 fix: fix resolve query type hint (#853) 2024-11-15 14:25:47 +01:00
George
8d6bee2434 new: make model name mandatory in inference structures (#846) 2024-11-12 19:09:44 +01:00
George
e15b21e5fb Alter image structure (#845)
* new: assign image.image Any type in grpc

* new: update image structure, rename embed utils Path to FieldPath
2024-11-11 18:03:26 +01:00
George
d2771afdfb Local inference image support (#836)
* new: add backbone for image support

* new: convert b64 to pil, embed images, add test

* tests: add test file

* refactor: replace 3 different inference object vars with a common one

* fix: fix type hints

* fix: fix type hints

* tests: add tests

* fix: remove redundant imports

* new: propagate image options

* Custom inference object (#837)

* new: add inference object support

* new: add inference object support

* fix: remove redundant import

* refactor: return newline

* fix: fix propagate options test
2024-11-04 08:37:53 +01:00
George
5e0a40e954 Propagate options (#835)
* new: propagate model params in document options

* new: update fastembed version

* fix: fix option propagation, add tests

* fix: update poetry lock

* fix: don't change the set model on embed raw types

* tests: extend propagate options test

* fix: erase cls model storage after each test
2024-10-31 09:56:58 +01:00
George
c152e3c7bf Agnostic local inference (#799)
* new: update previous qdrant version

* new: rollback rest client as types moved to the core

* new: add model_fields_set pydantic compat

* new: add Document to models

* new: add local inference inspectors

* new: add inference functionality to qdrant client

* fix: fix type hints

* fix: fix type hints

* fix: fix type hints

* new: remove _cloud_inference arg from methods

* refactor

* tests: update tests, add path check

* fix: fix type hint

* fix: fix type hint

* fix: resolve plain query

* fix: remove redundant checks

* fix: fix resolve query doc

* tests: add test case for several docs in prefetch

* fix: fix tests

* new: forbid cloud inference with local mode

* fix: fix is_query assignment, fix embed dict

* tests: fix sparse vector values comparison

* fix: regen async

* fix: embed batch

* tests: extend local inference tests

* tests: add update operations and update vectors tests

* tests: remove redundant tests, add plain query points test

* fix: fix type hint

* fix: add missing import

* fix: yet another type hint fix

* tests: add query points groups and query batch points tests

* fix: do not try to embed a vector, comment missed disabled test stmt

* new: add late interaction embeddings without add and query methods

* fix: fix LITE import

* fix: fix problem with retrieving not available path, enable tests

* fix: fix exception message

* refactor: update type hint

* fix: update type hints

* chore: add docstrings to embed methods

* refactor: simplify inspector init

* fix: fix type hint

* new: reuse parser, fix type import

* refactor: rename parser, add docstrings

* fix: fix type hint

* fix: fix type hint

* fix: fix type hint

* refactor: smol refactor, docstrings

* fix: fix inspect embed

* deprecation warning for grpc.PointStruct

* new: populate inspections cache (#823)

* new: populate inspections cache

* fix: fix type hints

* fix: regen async

* fix: fix type hint

* fix: fix typo in filename

* fix: fix points check

* fix: fix type hint
2024-10-29 15:48:23 +01:00
George
8abfd0e6cd fix: fix docstrings to make sphinx happy (#814)
* fix: fix docstrings to make sphinx happy

* debug: downgrade poetry version

* fix: update virtualenv

* debug: remove debug line
2024-10-18 10:55:50 +04:00
George
52ccd4181b fix: add local files only to set sparse model (#802) 2024-10-07 10:53:36 +02:00
Hossam Hagag
d0bda1b3bb chore: Change embed to query_embed in fastembed mixin for sparse models (#797) 2024-10-05 23:29:24 +02:00
George
04845eeba0 new: remove default hybrid search for query_points with fastembed (#735)
* new: remove default hybrid search for query_points with fastembed

* tmp: disable query extension for vector input, etc
2024-08-12 16:14:32 +02:00
George
d354293a4e new: set idf modifier in fastembed mixin for bm models (#686)
* new: set idf modifier in fastembed mixin for bm models

* fix: update poetry lock

* fix: tmp fix, restrict pyright version

* refactor: update bm model extraction, update fastembed

* fix: fix list of bm models

* fix: fix default bm models list value

* refactoring: remove redundant import
2024-07-30 18:01:47 +02:00
Luis Cossío
c3e8ec00e5 Local mode for Query 2024-06-27 19:46:25 +02:00
George Panchuk
c599e20268 fix: fix isinstance check, query conversion, query batch request 2024-06-27 18:31:41 +03:00
George Panchuk
40e0c62a4f new: update query signature, id type and return type 2024-06-26 17:42:39 +03:00
Andrey Vasnetsov
6ff600c94b rollback fast-embed query method (#658)
* rollback fastembed query method

* un-deprecate query

* fix: regen async, roll back generator tmp workaround

* fix: don't skip fastembed tests

* new: add query batch points interface, fix minor bugs (#660)

* new: add query batch points interface, fix minor bugs

* fix: do not exclude ids from lookup collection from search (#661)

* Update qdrant_client/qdrant_client.py

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>

* fix: regen async

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2024-06-26 17:13:40 +03:00
George
00da88e947 new: add query interface draft (#655)
* new: add query interface draft

* fix: temporarily update fastembed generator

* review fixes

* generate async client

* generate async client

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-06-25 09:21:28 +02:00
Andrey Vasnetsov
815a784469 Update openapi v1 10 x (#622)
* update api with conversions

* introduce IDF modifier for sparse vectors in local mode

* generate stuff

* regen async

* Update tests/fixtures/points.py

* fix: address review comments

* upd models

* fix: update grpc overwrite payload conversion, fix order by conversion (#653)

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
2024-06-20 17:55:35 +02:00
Andrey Vasnetsov
8556cc86d6 Drop black (#623)
* replace black with ruff

* use line length

* remove isort

* regen async

* regen async

* fix ruff version [no-ci]
2024-05-07 20:47:27 +02:00
George
3bdb6f6719 new: add gpu support for fastembed, add fastembed providers (#612)
* new: add gpu support for fastembed, add fastembed providers

* new: update fastembed, add fastembed-gpu, add type alias for onnx providers

* new: update readme

* fix: fix poetry.lock
2024-05-03 21:23:44 +02:00
Andrey Vasnetsov
822aefa1c1 WIP: hybrid search with fastembed (#553)
* WIP: hybrid search with fastembed

* hybrid queries with fastembed

* test for hybrid

* fix typo

* new: extend hybrid search tests, fix mypy, small refactoring (#554)

* refactor: align model name parameters in setters, update tests

* fix: fix async

* fix: add a good test, fix sparse vectors in query batch

* refactoring: reduce branching, refactor fastembed tests

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
2024-03-27 13:57:58 +01:00
Nirant
fc4b3cf9e2 Upgrade FastEmbed Version (#493)
* Update fastembed to v0.2.1

* chore(qdrant_fastembed.py): update DEFAULT_EMBEDDING_MODEL

* fix(fastembed integration): upgrade to latest version

* Prefer black over ruff

* Prefer black over ruff

* Remove hardcoded directory structure from Qdrant Client checks

* new: deprecate current default model, deprecate max token length, update fastembed

* fix: make embedding_model_name method sync

* fix: update poetry lock

* refactor: use list_supported_models() (#501)

* fix: fix fastembed check

* fix: fix fastembed class var assignment

* fix: remove fastembed deprecation from qdrant client (#524)

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
Co-authored-by: Anush <anushshetty90@gmail.com>
2024-03-05 21:09:31 +01:00
George
a2adc2d2ad new: deprecate upload records, update tests, prohibit migration of co… (#447)
* new: deprecate upload records, update tests, prohibit migration of collections with custom shards

* Update qdrant_client/qdrant_client.py

* Update qdrant_client/async_qdrant_client.py

* new: replace autogenerated int ids with uuids (#448)

* fix: remove redundant kwargs

* fix: regen async

* fix: remove redundant import

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2024-01-19 20:11:31 +01:00
Praveen Palanisamy
70754dbd91 Allow fastembed embedding model params configuration in parity with fastembed APIs (#416)
* Update set_model -> set_model_params;
Allow setting all fastembed.DefaultEmbedding init args for full pass-through

* Add fastembed test for setting custom embedding model params

* Update async client gen to include set_model_params

* Update Async client APIs

* Add explicit Optional since PEP 484 is still in effect

* Regenerate async client APIs

* Revert name change: set_model_params -> set_model

* Add **kwargs to set_model method sig

* Update name set_model_params -> set_model

* Regenerate async client APIs

* Use and pass **kwargs

* Regenerate async client APIs using Py 3.10.x
2024-01-07 18:48:58 +01:00
generall
45c947e34d fix _embed_documents performance 2023-10-16 23:02:57 +02:00
Andrey Vasnetsov
0f32e578ec update interface and version for fastembed (#340)
* update interface and version for fastembed

* fix types

* fix types

* regen async

* use python 3.11 to check compatibility

* fix docstring

* regen async

* propagate batch size
2023-10-16 18:13:52 +02:00
George
eafd92a9a3 fix: explicitly convert np vector to list for previous versions of pydantic (#289) 2023-09-11 19:47:46 +02:00
Andrey Vasnetsov
8452f2ae72 fix fastembed for pydantic 1.x (#285)
* fix fastembed for pydantic 1.x

* add extras setup in readme

* move fastembed from group to extras

* lock
2023-09-11 15:09:12 +02:00
Andrey Vasnetsov
aed7ecaefb sync API (#280)
* sync API

* fix tests

* backward compatibility version up
2023-09-07 11:29:10 +02:00
Nirant
a26a07769c Fixes for test, type errors (#266)
* * fix(qdrant_fastembed.py): change method name from encode to embed in QdrantFastembedMixin class
* fix(qdrant_fastembed.py): change method name from encode to embed

* * fix(qdrant_fastembed.py): change DEFAULT_EMBEDDING_MODEL from "sentence-transformers/all-MiniLM-L6-v2" to "BAAI/bge-small-en"

* * fix(qdrant_fastembed.py): add support for different embedding types in QdrantFastembedMixin
* feat(qdrant_fastembed.py): add embed_type parameter to embed method in QdrantFastembedMixin

* * fix(qdrant_fastembed.py): change vector field name from "text-{model_name}" to "fast-{model_name}"

* * fix(qdrant_fastembed.py): add comments to clarify code intent

* * refactor(qdrant_fastembed.py): remove unused import 'Sequence' from typing module

* * test(test_fast_embed.py): refactor test_add_without_query function to simplify parameter handling and remove unnecessary checks and assignments

* * fix(qdrant_fastembed.py): fix type error in NamedVector Query

* * fix(qdrant_fastembed.py): change method name from `embed` to `query_embed` in QdrantFastembedMixin class
* fix(qdrant_fastembed.py): fix query_vectors assignment in `query` method of QdrantFastembedMixin class
2023-08-23 14:50:09 +02:00
Nirant
536b0eeb22 Add fastembed integration (#210)
* Add skeleton for fastvector registration

* * fix(qdrant_client.py): fix import statement for QdrantClientMixin from fastembed instead of fastvector
* fix(qdrant_client.py): remove unnecessary print statement

* * chore(README.md): add bullet point for extensive test coverage
* feat(README.md): add section for Fast Embeddings + Simpler API
* fix(README.md): fix formatting of code block and update code example for Qdrant Client usage
*

* * fix(qdrant_client.py): remove unused imports and code
* feat(qdrant_client.py): add support for adding and querying documents with fastembed installed
* test(qdrant_client.py): add tests for adding and querying documents with and without fastembed installed

* * feat(qdrant_client.py): add support for fastembed.qdrant_mixin.QdrantAPIExtensions for upsert_docs and search_docs methods

* * fix(qdrant_client.py): remove unused import of SearchParams
* feat(qdrant_client.py): add return type hint to QdrantClient.search_docs method

* * fix(qdrant_client.py): fix import statement for SearchParams class in qdrant_client.py

* * feat(models.py): add QueryResponse model for handling query responses

* * feat(qdrant_client.py): add batch_iterable method to QdrantClient class

* * chore(qdrant_client.py): reformat code for better readability

* * chore(test_qdrant_client.py): add type hinting to fastembed_setup fixture
* test(test_qdrant_client.py): add test for client_close function

* * chore(.gitignore): add pattern for ignoring .tar.gz files

* * refactor(test_qdrant_client.py): remove unused imports and variables
* chore(test_qdrant_client.py): reformat import statements for better readability

* * chore(test_qdrant_client.py): remove unused fixtures and test cases

* * chore(qdrant_client.py): remove unused imports

* Add test skeleton

* * chore(.gitignore): add local_cache/*/* to ignore local cache files

* * fix(qdrant_client.py): change import statement for fastembed to use DefaultEmbedding instead of FlagEmbedding
* refactor(qdrant_client.py): refactor code to remove unnecessary loop

* * fix(qdrant_client.py): import missing models in qdrant_client.py
* feat(qdrant_client.py): add support for search parameters in search method
* refactor(qdrant_client.py): refactor indexing logic to handle embeddings correctly

* * test(test_fast_embed.py): add assertion to verify count of documents in collection
* test(test_fast_embed.py): remove unused code
* test(test_fast_embed.py): add TODO comment for future assertions

* * chore(pyproject.toml): add optional fastembed group to poetry dependencies
* feat(pyproject.toml): add fastembed dependency to fastembed group

* * chore(qdrant_client.py): add check for fastembed installation in QdrantClient constructor

* * fix(test_fast_embed.py): skip test if FastEmbed is not installed
* fix(test_fast_embed.py): add default values for test_no_install parameters
* fix(test_fast_embed.py): skip test if FastEmbed is installed

* Delete unused usage file

* * chore(models.py): remove unused QueryResponse class
* feat(qdrant_client.py): add QueryResponse class

* * refactor(qdrant_client.py): organize imports and fix type hinting for QueryResponse class

* review fixes

* make mypy happy

* review fixes

* fix mypy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2023-08-18 15:17:46 +02:00