* Introduce coverage reports for integration tests
* Install cargo-llvm-cov
* Use multiline script
* Explicityl setup COVERAGE env var
* fix integration tests and log generated data
* fix ls path
* upload lcov file to GH artifacts
* integration profraw dynamic filename
* Fix llvm profile filename template
* Use interrupt instead of kill and merge consensus test results into same file
* Drop upload artifact stage
* install llvm-cov
* upload as artifact and export coverage files
* try simplifying workflow
* Migrate coverage generation to existing dedicated gh workflow
* trigger on coverage related branches
* Build only if qdrant binary with cov doesnt exist
* Use valid yaml
* include mode in profraw filename
* split coverage workflow into parallel jobs
* add poetry version to env
* log poetry version to install
* clean up integration test workflow
* Simplify comments
* Fix coverage test OOD on CI
* Temporarily run coverage CI on PR
* Clean artifacts per package and store in tmp dir
* fix typo in gh workflow
* Add monitoring script again for debugging
* trigger script
* Fix monitor script path
* dont checkout dev branch
* Merge .info file only if test passed
* Try merging .info file only if it exists
* trigger ci
* fix workflow
* trigger ci
* Customize branch to check out
* Remove monitor.sh script
* remove unused env var
* Setup code coverage for Rust tests
* Add API key env var for codecov
* Save code coverage report for upload
* Trigger CI
* Try without explicit build step
* Run coverage job on self hosted runner
* Fix mistake because of which is was running fewer tests
* Run coverage on ubuntu latest
* Trigger CI
* fix name
* run coverage on all os
* Adjust after rebase to dev
* Avoid running coverage in other jobs
* fix test job
* reset test workflow
* Trigger CI
* run coverage after tests
* Add script to monitor resources
* fix indentation
* remove coverage dependency on test job
* fix needs field
* add monitoring script
* Run tail -f to monitor resources concurrently with tests
* Split commands
* Use --jobs=1 to possibly minimize RAM
* Try generating coverage one by one
* Dont clean coverage artifacts
* Merge .lcov files with lcov command
* Upload merged lcov.info file
* Run coverage for fewer packages
* Improve coverage script
* Move coverage.sh to tools dir
* Prepare for review
* Improve CI output and failure handling
* Generate HTML report if running locally
* Cleanup
* Run coverage on schedule and only for dev branch
* parent 88f68d95b5
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1736245999 +0100
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1736331073 +0100
gpu test and release workflows
no unprivileged, add sdk
fix ci
include llvm
try to use external sdk
use another vk sdk script
remove ninja
native sdk installation
debug vulkan version
use non-dev packages
debug vkinfo
remove llvm dep
ubuntu process-results
ghcr
* remove running gpu test while pr
* Fix shard snapshot operations for s3
* tests/shard-snapshot-api.sh: Improve error reporting
* Drop SnapshotStorageManager::get_shard_snapshot_path
* Fix s3 test on CI
* Move CODE_OF_CONDUCT.md to docs
* Move CONTRIBUTING.md to docs
* Drop QUICK_START_GRPC.md
* Drop docs/favicon.ico as being unused
* Move QUICK_START.md to docs
* Use poetry for openapi tests
* Use poetry for consensus tests
* Use poetry for gen_storage_compat_data/populate_db.py
* Make consensus_tests scripts executable
* Use poetry for test-consensus-compose
* Cleanup
* Don't call poetry run in scripts
* Add test for OOD during indexing
* Start indexing right after 1st OOD message
* Only send search request after insert loop
* Fail early when encountering out-of-storage during optimization (#4578)
* fs4@0.8.4
* fail early on low storage
* move `dir_size` to `common`
* move the ood bailout to `SegmentOptimizer::optimized_segment_builder`
* drop dead code
* move dir_size to common subcrate
---------
Co-authored-by: generall <andrey@vasnetsov.com>
---------
Co-authored-by: xhjkl <xhjkl@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* tests: Add test on low disk
* Remove redundant assertion
* keep container after failure and print latest logs in console
* Add fs2 crate as a dependency and ensure sufficient disk space in LocalShard operations
* small fix
* Update test
* small fix
* use available_space
* Use `fs2` -> `fs4` and offload sync IO with `tokio::task::spawn_blocking`
* create DiskUsageWathcer
* chore: Remove unnecessary println statement in update_handler.rs
* chore: Fix typo in DiskUsageWatcher struct name
* chore: Refactor DiskUsageWatcher to improve disk usage tracking and update logic
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add SnapshotsStorageConfig enum(Local or S3) and deserialize implementation
* [refactor] use snapshots_config instead of s3_config
* update config
* add AWS official`aws-sdk-s3`
* implement store_file() WITHOUT error handling
* implement list_snapshots
* implement delete_snapshot
* run `cargo +nightly fmt`
* delete println
* implement get_stored_file
* Add error handlings
* Refactor AWS S3 configuration and error handling
* fix bugs
* create an empty test file
* fix `alias_test.rs` for StorageConfig type
* tempolary delete some test and try s3 test
* Update integration-tests.yml to use snap instead of apt-get for installing yq
* Update integration-tests.yml to use sudo when installing yq
* add sudo
* make (full/non-full) snapshots downloadable
* debug
* small fix
* Add S3 endpoint URL configuration option
* fix
* fix
* debug
* fix endpoint
* update to http://127.0.0.1:9000/
* update
* fix
* fix `#[get("/collections/{collection}/shards/{shard}/snapshots/{snapshot}")]` for s3
* put original tests back
* refactor
* small fix (delete println & echo)
* use object_store and refactor
* create snapshot_storage_ops and implement
* Refactor get_appropriate_chunk_size function to adjust chunk size based on service limits and file size
* cargo +nightly fmt --all
* make it more abstract
* Refactor SnapshotsStorageConfig deserialization in SnapShotsConfig
* small update
* small fix
* Update dependencies in Cargo.lock
* Update minio image to satantime/minio-server
* Refactor snapshot storage paths in snapshots_manager.rs and snapshot_storage_ops.rs
* Fix issue with downloaded file size not matching expected size in download_snapshot function
* add flush
* Use Streaming instead of donloading once
* apply `cargo +nightly fmt --all`
* Fix issue with opening file in SnapshotStream::LocalFS variant
* Fix error handling in SnapshotStream::LocalFS variant
* Add integration test for Shard Snapshot API with S3 storage (#7)
* fix: Replace newline with spaces to find docker resources to remove
* fix: Use xargs instead of tr
* fix: Avoid quotes to pass multiple args
* fix: Use tr -d
* create exhaustive access test (missing body stubs)
* use json body in rest requests
* use better stubs
* green test for create_shard_key
* make it work for delete_shard_key
* make it work for list_collections
* make it work for get_collection
* make it work for create_collection
* make it work for update_collection_params
* make it work for delete_collection
* make it work for update_aliases
* make it work for create_index
* make it work for collection_exists
* make it work for delete_index
* make it work for get_collection_cluster_info
* don't build grpcio and use `uv` for faster deps installation
* partially prepare for update cluster ops
* make it work for list_collection_aliases
* make it work for list_aliases
* make it work for list_collection_snapshots
* make it work for create_collection_snapshot
* huge refactor + make it work for delete_collection_snapshot
* make it work for download_collection_snapshot
* test_upload_collection_snapshot
* test_recover_collection_snapshot
* test_recover_collection_snapshot
* test_upload_shard_snapshot
* test_recover_shard_snapshot
* test_list_shard_snapshots
* test_create_shard_snapshot
* test_delete_shard_snapshot
* test_download_shard_snapshot
* test_list_snapshots and test_create_snapshot
* test_delete_snapshot
* test_download_full_snapshot
* test_get_cluster
* test_recover_raft_state
* test_delete_peer
* prepare for splitting into another PR
* skip api exhaustiveness checks
* lil cleanup
* fix integration tests workflow
* test search api
* test recommend api
* test discover api
* test count and scroll
* test get and upsert points
* test update points batch
* test update and delete vectors
* test set, overwrite, delete, and clear payload
* test service level actions
* finish testing (with errors), introduce not needing success
* make tests much faster
* simplify delete tests
* complete collection cluster operations
* cleanup
* update after rebase
* change permissions for create and delete shard keys
* review fixes
* add newline
* feat: Print commit id in dev container build workflow
* fix: Keep .git while building Qdrant binary in docker
* fix: Remove redundant printing of git commit id
* fix: Copy only git files first
* fix: Docker build should have commit id if present
* refactor: Use fewer lines of code
* feat: Use git commit id from build arg
* refactor: Use cleaner code
* Fix string reference issue
* fix: Dont print info log if commit is not set
* fix: Include git commit id in release containers
---------
Co-authored-by: timvisee <tim@visee.me>
