* Add stopwords support
* Update codespell configuration and improve stopword assertion message
* Add stopwords schema to OpenAPI definition
* Rename StopwordsOption to StopwordsInterface
* Add stopwords support for multiple languages
* Update codespell configuration to skip all Rust files in stop_words directory
* Normalize stopwords to lowercase during insertion
* Add language aliases for stopwords
* + grpc
* Unspecified Language
* clippy fmt
* openapi
* Refactor to use BTreeSet
* update tokenizer test to include custom stopwords with contractions
* simplified grpc StopwordsInterface
* alias for languages
* use `lowercase` flag for stop word normalizing
* grpc: enum -> string
* consistent naming
* Remove UnspecifiedLanguage and use From for conversions
* fix conversions
* fmt
* use stopwords inside different tokenizer
* use ahash
* fix tests
* fix no-rocks-db feature
* simplify code and fix test
* update test
* fix clippy
* fmt
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add temporary job for unit tests on CI without RocksDB feature
* Remove trailing white spaces
* Fix artifact name collision
* Don't upload test report for tests without RocksDB
* Introduce coverage reports for integration tests
* Install cargo-llvm-cov
* Use multiline script
* Explicityl setup COVERAGE env var
* fix integration tests and log generated data
* fix ls path
* upload lcov file to GH artifacts
* integration profraw dynamic filename
* Fix llvm profile filename template
* Use interrupt instead of kill and merge consensus test results into same file
* Drop upload artifact stage
* install llvm-cov
* upload as artifact and export coverage files
* try simplifying workflow
* Migrate coverage generation to existing dedicated gh workflow
* trigger on coverage related branches
* Build only if qdrant binary with cov doesnt exist
* Use valid yaml
* include mode in profraw filename
* split coverage workflow into parallel jobs
* add poetry version to env
* log poetry version to install
* clean up integration test workflow
* Simplify comments
* Fix coverage test OOD on CI
* Temporarily run coverage CI on PR
* Clean artifacts per package and store in tmp dir
* fix typo in gh workflow
* Add monitoring script again for debugging
* trigger script
* Fix monitor script path
* dont checkout dev branch
* Merge .info file only if test passed
* Try merging .info file only if it exists
* trigger ci
* fix workflow
* trigger ci
* Customize branch to check out
* Remove monitor.sh script
* remove unused env var
* Setup code coverage for Rust tests
* Add API key env var for codecov
* Save code coverage report for upload
* Trigger CI
* Try without explicit build step
* Run coverage job on self hosted runner
* Fix mistake because of which is was running fewer tests
* Run coverage on ubuntu latest
* Trigger CI
* fix name
* run coverage on all os
* Adjust after rebase to dev
* Avoid running coverage in other jobs
* fix test job
* reset test workflow
* Trigger CI
* run coverage after tests
* Add script to monitor resources
* fix indentation
* remove coverage dependency on test job
* fix needs field
* add monitoring script
* Run tail -f to monitor resources concurrently with tests
* Split commands
* Use --jobs=1 to possibly minimize RAM
* Try generating coverage one by one
* Dont clean coverage artifacts
* Merge .lcov files with lcov command
* Upload merged lcov.info file
* Run coverage for fewer packages
* Improve coverage script
* Move coverage.sh to tools dir
* Prepare for review
* Improve CI output and failure handling
* Generate HTML report if running locally
* Cleanup
* Run coverage on schedule and only for dev branch
* parent 88f68d95b5
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1736245999 +0100
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1736331073 +0100
gpu test and release workflows
no unprivileged, add sdk
fix ci
include llvm
try to use external sdk
use another vk sdk script
remove ninja
native sdk installation
debug vulkan version
use non-dev packages
debug vkinfo
remove llvm dep
ubuntu process-results
ghcr
* remove running gpu test while pr
* Fix shard snapshot operations for s3
* tests/shard-snapshot-api.sh: Improve error reporting
* Drop SnapshotStorageManager::get_shard_snapshot_path
* Fix s3 test on CI
* Move CODE_OF_CONDUCT.md to docs
* Move CONTRIBUTING.md to docs
* Drop QUICK_START_GRPC.md
* Drop docs/favicon.ico as being unused
* Move QUICK_START.md to docs
* Use poetry for openapi tests
* Use poetry for consensus tests
* Use poetry for gen_storage_compat_data/populate_db.py
* Make consensus_tests scripts executable
* Use poetry for test-consensus-compose
* Cleanup
* Don't call poetry run in scripts
* Add test for OOD during indexing
* Start indexing right after 1st OOD message
* Only send search request after insert loop
* Fail early when encountering out-of-storage during optimization (#4578)
* fs4@0.8.4
* fail early on low storage
* move `dir_size` to `common`
* move the ood bailout to `SegmentOptimizer::optimized_segment_builder`
* drop dead code
* move dir_size to common subcrate
---------
Co-authored-by: generall <andrey@vasnetsov.com>
---------
Co-authored-by: xhjkl <xhjkl@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* tests: Add test on low disk
* Remove redundant assertion
* keep container after failure and print latest logs in console
* Add fs2 crate as a dependency and ensure sufficient disk space in LocalShard operations
* small fix
* Update test
* small fix
* use available_space
* Use `fs2` -> `fs4` and offload sync IO with `tokio::task::spawn_blocking`
* create DiskUsageWathcer
* chore: Remove unnecessary println statement in update_handler.rs
* chore: Fix typo in DiskUsageWatcher struct name
* chore: Refactor DiskUsageWatcher to improve disk usage tracking and update logic
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add SnapshotsStorageConfig enum(Local or S3) and deserialize implementation
* [refactor] use snapshots_config instead of s3_config
* update config
* add AWS official`aws-sdk-s3`
* implement store_file() WITHOUT error handling
* implement list_snapshots
* implement delete_snapshot
* run `cargo +nightly fmt`
* delete println
* implement get_stored_file
* Add error handlings
* Refactor AWS S3 configuration and error handling
* fix bugs
* create an empty test file
* fix `alias_test.rs` for StorageConfig type
* tempolary delete some test and try s3 test
* Update integration-tests.yml to use snap instead of apt-get for installing yq
* Update integration-tests.yml to use sudo when installing yq
* add sudo
* make (full/non-full) snapshots downloadable
* debug
* small fix
* Add S3 endpoint URL configuration option
* fix
* fix
* debug
* fix endpoint
* update to http://127.0.0.1:9000/
* update
* fix
* fix `#[get("/collections/{collection}/shards/{shard}/snapshots/{snapshot}")]` for s3
* put original tests back
* refactor
* small fix (delete println & echo)
* use object_store and refactor
* create snapshot_storage_ops and implement
* Refactor get_appropriate_chunk_size function to adjust chunk size based on service limits and file size
* cargo +nightly fmt --all
* make it more abstract
* Refactor SnapshotsStorageConfig deserialization in SnapShotsConfig
* small update
* small fix
* Update dependencies in Cargo.lock
* Update minio image to satantime/minio-server
* Refactor snapshot storage paths in snapshots_manager.rs and snapshot_storage_ops.rs
* Fix issue with downloaded file size not matching expected size in download_snapshot function
* add flush
* Use Streaming instead of donloading once
* apply `cargo +nightly fmt --all`
* Fix issue with opening file in SnapshotStream::LocalFS variant
* Fix error handling in SnapshotStream::LocalFS variant
* Add integration test for Shard Snapshot API with S3 storage (#7)
* fix: Replace newline with spaces to find docker resources to remove
* fix: Use xargs instead of tr
* fix: Avoid quotes to pass multiple args
* fix: Use tr -d
