* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Introduce coverage reports for integration tests
* Install cargo-llvm-cov
* Use multiline script
* Explicityl setup COVERAGE env var
* fix integration tests and log generated data
* fix ls path
* upload lcov file to GH artifacts
* integration profraw dynamic filename
* Fix llvm profile filename template
* Use interrupt instead of kill and merge consensus test results into same file
* Drop upload artifact stage
* install llvm-cov
* upload as artifact and export coverage files
* try simplifying workflow
* Migrate coverage generation to existing dedicated gh workflow
* trigger on coverage related branches
* Build only if qdrant binary with cov doesnt exist
* Use valid yaml
* include mode in profraw filename
* split coverage workflow into parallel jobs
* add poetry version to env
* log poetry version to install
* clean up integration test workflow
* Simplify comments
* Fix coverage test OOD on CI
* Temporarily run coverage CI on PR
* Clean artifacts per package and store in tmp dir
* fix typo in gh workflow
* Add monitoring script again for debugging
* trigger script
* Fix monitor script path
* dont checkout dev branch
* Merge .info file only if test passed
* Try merging .info file only if it exists
* trigger ci
* fix workflow
* trigger ci
* Customize branch to check out
* Remove monitor.sh script
* remove unused env var
* Setup code coverage for Rust tests
* Add API key env var for codecov
* Save code coverage report for upload
* Trigger CI
* Try without explicit build step
* Run coverage job on self hosted runner
* Fix mistake because of which is was running fewer tests
* Run coverage on ubuntu latest
* Trigger CI
* fix name
* run coverage on all os
* Adjust after rebase to dev
* Avoid running coverage in other jobs
* fix test job
* reset test workflow
* Trigger CI
* run coverage after tests
* Add script to monitor resources
* fix indentation
* remove coverage dependency on test job
* fix needs field
* add monitoring script
* Run tail -f to monitor resources concurrently with tests
* Split commands
* Use --jobs=1 to possibly minimize RAM
* Try generating coverage one by one
* Dont clean coverage artifacts
* Merge .lcov files with lcov command
* Upload merged lcov.info file
* Run coverage for fewer packages
* Improve coverage script
* Move coverage.sh to tools dir
* Prepare for review
* Improve CI output and failure handling
* Generate HTML report if running locally
* Cleanup
* Run coverage on schedule and only for dev branch
* Fix paste-bugs in `snapshot_service.proto`
* Add shard snapshot gRCP API definition
* Add validation to shard snapshot gRPC API definition
* Implement conversions between gRPC and `collection` types
* Extract shard snapshot API implementation into common sub-module
* Implement shard snapshot gRPC API
* Generate gRPC docs
* Refactor `ShardSnapshots` gRPC service to be internal API only
* fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Move `ShardSnapshotRecoverResponse` to `shard_snapshots_service.proto`
* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Update `api/src/grpc/qdrant.rs`
* fixup! fixup! Refactor `ShardSnapshots` gRPC service to be internal API only
Update gRPC docs
* Switch `ShardSnapshots` gRPC service to use `validate_and_log` instead of `validate`
* Add script to list commits that have yet to be picked into master
This allows us to easily detect cherry-picks we missed from dev to
master on our release branch.
* Update missed cherry picks ignore hash
* Refactor Dockerfile
- fix cross-compilation
- improve caching
TODO:
- check if `lld` is used/works for linkage (and enable if not used, or remove if doesn't work)
* Remove `aarch64` linker config from `.cargo/config.toml` (seems to be unnecessary)
* Expose `LINKER` argument and enable `lld` linker
* Add `mold` linker support
* Document Dockerfile
* Add closing ` in the Dockerfile comments
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add a simple script to generate `docker-compose.yaml` for a cluster of specified number of nodes
* Improve service error from `RequestError` conversion
* fixup! Improve service error from `RequestError` conversion [skip ci]
* fixup! Add a simple script to generate `docker-compose.yaml` for a cluster of specified number of nodes [skip ci]
* Add CPU limit to the `docker-compose-gen.sh` [skip ci]
* fixup! Add CPU limit to the `docker-compose-gen.sh` [skip ci]
* Move docker-compose generator script into tools directory, add docs
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Simplify error message constructing
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Use custom build profile for docker builds
* trigger ci
* change source code to trigger ci with rebuild one more time
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add caching of docker layers in CI
Build required docker images for CI in a workflow step using buildkit's
gha cache type. This will populate the local layer cache from github
actions' cache. Builds in subsequent CI steps will be nearly instant,
because all layers can be reused.
* add minor change to see if build time is any faster
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* WIP: Start working on out-of-RAM errors handling [skip ci]
* Implement basic handling of out-of-RAM errors during Qdrant startup
* Try to fix CI fail by allowing both V1 and V2 cgroups
* Try to fix CI fail by improving cgroups handling
* Fix cgroups path detection/handling (+ some minor stylistic changes)
* fixup! Fix cgroups path detection/handling (+ some minor stylistic changes)
* Add test
* Enable low RAM test
* fixup! Add test
* free memory checks
* rm unused function
* Oom fallback script (#1809)
* add recover mode in qdrant + script for handelling OOM
* fix clippy
* reformat entrypoint.sh
* fix test
* add logging to test
* fix test
* fix test
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* segment telemetry
* anonymize trait
* fix build
* are you happy fmt
* anonimyze implementations
* sliding window avg (#826)
* Actix web telemetry (#828)
* actix web telemetry
* small as move
* use tokio mutex instead of std
* add comments
* are you happy fmt
* use u16 as http status code
* telemetry structs rename
* fix build
* using parking lot mutex
* telemetry web api (#842)
* telemetry web api
* telemetry openapi (#843)
* use async mutex for telemetry collector
* use tokio mutex for telemetry collector
* are you happy fmt
