* Extract logic for creating thresholds config
* Put collection params and threshold config in update handler
* Add function to add a new appendable segment if all are over capacity
* Make new method static, call it before each optimization loop
* Update error message formatting
* Use exact point count in replication consensus test
* Add a test to assert segment creation when all are over capacity
* Suffix optimizer thresholds with _kb to clarify unit
* Move segment capacity check logic, run if optimizers are disabled
* fix: add -> mul
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* feat: Working pyroscope profiler integration
* fix: Remove UpdateDebugConfigResponse
* fix: Format
* fix: Simplify debug settings type
* fix: Remove /debug from API spec and clean pyrostate state
* fix: Update openapi.json
* refactor: Move models to suitable paths
* refactor: Improve API and pyroscope state structure
* fix: Format
* feat: Use null debug for pyroscope config
* fix: Pyroscope shouldnt be used and enabled in windows
* fix: Format
* fix: Remove pyroscope for macos
* fix: Add more compile conditions
* fix: Try to fix OS target issue
* feat: Move DebugConfig and Pyroscope to common::debug
* feat: Introduce and use DebugState instead of PyroscopeState
* feat: Simplify debug_api.rs and move logic to debug.rs
* fix: format
* fix: Use debug patch enum
* feat: Use parking lot Mutex
* fix: Propagate errors instead of panic
* fix: Format
* feat: Forward stop agent errors
* feat: Use enums
* fix: For non linux OS
* fix: Missing import for non linux os
* fix: Remove redundant logs
* feat: Take lock throughout the patch
* Don't unwrap pyroscope state, it may be None
* fix: Remove debug: true from tls config tests
* refactor: Rename debug setting to debugger
* refactor: Rename debug to debugger everywhere
* Improve logs
---------
Co-authored-by: timvisee <tim@visee.me>
* Allow multivec search on legacy REST API
* show that it works for gRPC as well
* better error message
* update error assertion
* show validation on REST as well
* remove unecessary test
* fix conversion - dim is not vec count
* fmt
* Use TypedMultiDenseVectorRef everywhere (#4224)
* Use TypedMultiDenseVectorRef everywhere
* remove obsolete test
* fix codespell
* fix build
* test single dense vector expansion on upsert
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* tests: Add test on low disk
* Remove redundant assertion
* keep container after failure and print latest logs in console
* Add fs2 crate as a dependency and ensure sufficient disk space in LocalShard operations
* small fix
* Update test
* small fix
* use available_space
* Use `fs2` -> `fs4` and offload sync IO with `tokio::task::spawn_blocking`
* create DiskUsageWathcer
* chore: Remove unnecessary println statement in update_handler.rs
* chore: Fix typo in DiskUsageWatcher struct name
* chore: Refactor DiskUsageWatcher to improve disk usage tracking and update logic
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add SnapshotsStorageConfig enum(Local or S3) and deserialize implementation
* [refactor] use snapshots_config instead of s3_config
* update config
* add AWS official`aws-sdk-s3`
* implement store_file() WITHOUT error handling
* implement list_snapshots
* implement delete_snapshot
* run `cargo +nightly fmt`
* delete println
* implement get_stored_file
* Add error handlings
* Refactor AWS S3 configuration and error handling
* fix bugs
* create an empty test file
* fix `alias_test.rs` for StorageConfig type
* tempolary delete some test and try s3 test
* Update integration-tests.yml to use snap instead of apt-get for installing yq
* Update integration-tests.yml to use sudo when installing yq
* add sudo
* make (full/non-full) snapshots downloadable
* debug
* small fix
* Add S3 endpoint URL configuration option
* fix
* fix
* debug
* fix endpoint
* update to http://127.0.0.1:9000/
* update
* fix
* fix `#[get("/collections/{collection}/shards/{shard}/snapshots/{snapshot}")]` for s3
* put original tests back
* refactor
* small fix (delete println & echo)
* use object_store and refactor
* create snapshot_storage_ops and implement
* Refactor get_appropriate_chunk_size function to adjust chunk size based on service limits and file size
* cargo +nightly fmt --all
* make it more abstract
* Refactor SnapshotsStorageConfig deserialization in SnapShotsConfig
* small update
* small fix
* Update dependencies in Cargo.lock
* Update minio image to satantime/minio-server
* Refactor snapshot storage paths in snapshots_manager.rs and snapshot_storage_ops.rs
* Fix issue with downloaded file size not matching expected size in download_snapshot function
* add flush
* Use Streaming instead of donloading once
* apply `cargo +nightly fmt --all`
* Fix issue with opening file in SnapshotStream::LocalFS variant
* Fix error handling in SnapshotStream::LocalFS variant
* Add integration test for Shard Snapshot API with S3 storage (#7)
* submit an issue for unindexed field
solve unindexed field issue when an index is created for the field
also log a warning when issue is submitted
fmt
fix rebase problems
get collection name without panicking
better solving of issue
remove outdated TODO
add tests, fix GET /issues
review fixes
use `/` instead of `.` as separator
fmt
prepare extractor for time-based submission
smol rename
add check in search
hook unindexed field creation in search and search_batch of rest
update after rebase on `dev`
remove submit from `struct_payload_index`
solve issues inside of handlers, not in match router
post-process at Collection level
don't clone filters
remove from openapi spec
- Add hidden env variable to adjust slow search threshold
- Remove Solution::None
- Fix UnindexedField issue extractor
- Add endpoint to openapi, but without response body spec
- Move integration test to consensus_tests to set lower threshold
* event-based integration
* use typed Code
* remove payload_index_schema hack, get through collection info
* rename `notify` to `publish`
* update after rebase
* remove breakpoint
* update after rebase
* attach current payload schema to SlowQueryEvent
* collect filters refs lazily
* review fixes
* review fixes
* use regular config for setting the threshold, use 1.2 secs as default
* fix checking against current schema for all conditions
* REST API multivector for write and retrieve
* add test showing search is not supported
* use helper
* debug asserts
* better naming
* iterator style
* better naming
* add validation test for checking all inner vectors have the same size
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* failing test
* - better deserialization of ordering value
- ensure ordering value crosses a remote shard call
- fetch `with_payload` after merging from shards
* WIP: Fix inter shard order by tests fix (#4056)
* tests: Adjust assertion to check point.payload.index value
* tests: Fix failing test
* Revert "tests: Fix failing test"
This reverts commit f21fa4a8f8.
* fix default `with_payload` value
* set with_payload_interface=false
* remove the extra `retrieve` call
---------
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
* separate access tests from validation tests
* bring changes from rbac-integration-tests review fixes
* bring changes from rbac-payload-access-tests
* update from base branch
* use const instead of magic number
* move COLL_NAME
* force usage of kill_all_processes()
* add payload constraint spec
* test that payload constraint filters queries
* fmt
* make rest check only for 403 status
* move file to own folder
* improve error message
* update spec for collection cluster info
* fix: use `.get()` instead of direct access
* use const instead of magic number
* test of stream shard transfer consistency
* ensure that remote shard object is created when shard transfer created
* fmt
* special error type for handelling creation of partial shard
* ensure we dont try to create remote shard to itself
* create exhaustive access test (missing body stubs)
* use json body in rest requests
* use better stubs
* green test for create_shard_key
* make it work for delete_shard_key
* make it work for list_collections
* make it work for get_collection
* make it work for create_collection
* make it work for update_collection_params
* make it work for delete_collection
* make it work for update_aliases
* make it work for create_index
* make it work for collection_exists
* make it work for delete_index
* make it work for get_collection_cluster_info
* don't build grpcio and use `uv` for faster deps installation
* partially prepare for update cluster ops
* make it work for list_collection_aliases
* make it work for list_aliases
* make it work for list_collection_snapshots
* make it work for create_collection_snapshot
* huge refactor + make it work for delete_collection_snapshot
* make it work for download_collection_snapshot
* test_upload_collection_snapshot
* test_recover_collection_snapshot
* test_recover_collection_snapshot
* test_upload_shard_snapshot
* test_recover_shard_snapshot
* test_list_shard_snapshots
* test_create_shard_snapshot
* test_delete_shard_snapshot
* test_download_shard_snapshot
* test_list_snapshots and test_create_snapshot
* test_delete_snapshot
* test_download_full_snapshot
* test_get_cluster
* test_recover_raft_state
* test_delete_peer
* prepare for splitting into another PR
* skip api exhaustiveness checks
* lil cleanup
* fix integration tests workflow
* test search api
* test recommend api
* test discover api
* test count and scroll
* test get and upsert points
* test update points batch
* test update and delete vectors
* test set, overwrite, delete, and clear payload
* test service level actions
* finish testing (with errors), introduce not needing success
* make tests much faster
* simplify delete tests
* complete collection cluster operations
* cleanup
* update after rebase
* change permissions for create and delete shard keys
* review fixes
* add newline
* Initial non-blocking snapshot implementation
* Minor refactoring
* Add some comments, improve log messages
* Propagate proxy segment changes into wrapped segment when unproxying
* Use upgradable read lock for propagating proxy segment changes
* Extract proxy/unproxy functions for segments, better error handling
* Don't stop early on error, always clean up proxied segments
* Propagate proxy changes in two batches to minimize write locking
* Use upgradable read lock when propagating proxy changes in two batches
* Do not fall back to non-appendable segment configurations
* Resolve remaining TODOs
* Use LockedSegmentHolder type alias everywhere
* Better state handling in method to proxy all segments
* When proxying all segments, lock only after creating temporary segment
* Pass actual proxied segments around to minimize segment holder locking
* Propagate proxy segment changes to wrapped on drop, not to writable
* Minor improvements
* Fix proxy logic returning non-proxied segments
* Share single segment holder lock and upgrade/downgrade it
* Minor improvements
* Make appendable segment check more efficient
* Do not explicitly drop segments lock, it's not necessary
* Add consensus test to assert data consistency while snapshotting
* Fix incorrect documentation
* Extract payload storage type decision logic to collection params function
* Resolve TODO, we always expect to get a shard here
* Only upgrade propagate to wrapped readers if lists are not empty
* Set correct operation versions
* review fixes
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add shard transfer consensus method to restart shard transfer
* Arrange shard transfer fallback through consensus
* Properly handle stopping existing transfer, don't finish
* Use user specified shard transfer method as default when falling back
* Report correct fallback transfer method being used
* Always make WAL delta transfer fall back to stream records
* Remove unnecessary clones
* When doing shard transfer fallback, confirm consensus accepted it
* Confirm shard transfer restarts through consensus, rather than state
* Update WAL delta fallback test, assert we switch to stream_records
* Fix incorrect snapshot security test path
* Explicitly check if snapshot files exist
* Change shard snapshot invalid name error type to not found
That makes it consistent with behavior of our other snapshot APIs.
* Extend snapshot API security tests, also test collection and shard API
* Improve function to get collection snapshot path
* Fix arbitrary path traversal in full snapshot path function
* Add integration test to assert arbitrary path traversal is blocked
* Add consensus test for WAL delta abort and retry
* Use new function to assert vector equality
* Do not rely on baseline time measurement to abort in the middle
* Update values after changing throttle interval
