* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* WIP: introduce new vector store type
* handling of InRamMmap
* fmt
* feature-flag
* fmt
* Use if else
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/common/common/src/flags.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* also choose madvise for single-file in-ram-mmap
* simplify generics
* gpu fix
* fix bug
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add segment ID in telemetry response
* Migrate optimization log segment IDS to use UUIDs
* fmt
* clippy
* Bring back proxy segment SegmentEntry impl
* update openapi spec
* Bring back segment ID
* update comment
* Precompute segment uuid from filesyste while creating segment
* Update return types
* Update OpenAPI spec
* remove segment_ prefix from uuid field and use inline block
* Update OpenAPI spec
* Trigger CI
* clippy
* Improve comments to differentiate between segment id and uuid
* Remove duplicate function that shadowed segment entry function
---------
Co-authored-by: timvisee <tim@visee.me>
* make some existing fields optional
particularly, RemoteShardTelemetry's `searches` and `updates`, and
LocalShardTelemetry's `optimizations`
* Include shards information in cluster telemetry
* upd openapi
* nits
* only include partial snapshot info if non-empty
* fixes
* feat: Add enable_hnsw option for payload field indexes
Add optional enable_hnsw parameter to all payload index types to control
whether additional HNSW graph links are built for each indexed field.
- Add enable_hnsw field to all 8 payload index param types
- Update gRPC proto definitions and conversions
- Update OpenAPI schema
- Modify HNSW graph builder to respect enable_hnsw flag
- Add enable_hnsw() helper methods to PayloadSchemaParams and PayloadFieldSchema
- Update all tests to include new field (default: None)
When enable_hnsw is true and payload_M > 0, additional HNSW links will
be built for the payload field. Default value is true for backward compatibility.
* Fix Some format problems
* fix: address comment problem
---------
Co-authored-by: EC2 Default User <ec2-user@ip-10-78-171-148.ec2.internal>
* wrap details in another struct, less Options
* include version in peers info
* update openapi
* include app telemetry in internal service
* move debug assertion after check
* Add a test for recovery after kill during Partial
* Address AI review
* simplify test
* Merge pull request #7829
* introduce a new state
* switch to ManualRecovery for user-initiated snapshot operations
* fmt
* fix test and regen api
* test fix: instead of force recovery on update failed snapshots now re…
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Move pyproject.toml to root
* Migrate pyproject.toml from Poetry to uv
* Update GH workflows
* Update test script, doc and nix to use uv
* Use latest uv
* Fix uv.lock
* Cleanup shell.nix
* Cleanup
- Explicit `uv sync` is not required, `uv run` will install deps
automatically.
- We don't provide a python package, so the `[build-system]` section
is not needed.
* Fix UV_VERSION inconsistency
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Add progress_tracker.rs
* Pass progress tracker around
* Populate progress tracker with actual data
* Expose progress on `/collections/{name}/optimizations` endpoint
* Add ReadActive replica state
* Add consensus function to switch from Partial to ReadActive and confirm
* On stream records to a different shard ID, switch to ReadActive first
* Update OpenAPI spec
* In stream records peer sync, require all peers to be up-to-date
* Review fixes
* Rename ReadActive to ActiveRead
* update openapi spec
* explicit state checking functions
* typo
* update fallback routing with usage of a new state
---------
Co-authored-by: KShivendu <kshivendu1@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add vector count per vector-name to metrics API
* Add to metrics API
* Improve TinyMap::get_or_insert_default and add tests
* Minor improvements
* Update openapi
* Review remarks
* Remove `collection_vectors` since it can be calculated manually
* add feedback variant to query enum and internal service
remove internal service Feedback structs for QueryEnum
* rename Linear -> Simple
* follow refactor from base branch
* remove outdated validations
* remove reintroduced code
* use explicit From
* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
