* immutable map index integration
* remove wipe
* fix unit tests
* get appendable flag from config
* minor refactoring
* fix chunked mmap appendable flag
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Centralize generic range validation
* Centralize validate not empty
* Centralize collection name validation
* Add geo polygon validation in REST API and centralize validation
* Add tests for common validation functions
* Don't use reference in generic range validation function
* Other improvements
* immutable map index
* use (super) instead of (crate)
* more comments
* decompose remove_point
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Delete temp snapshot files on error
* shorter resulting path to please max length on Windows
* Remove one layer of tmp folder for shards
Unicity of segment tmp folder provided by Uuid
* On segment flush, read-lock all segments to prevent CoW between flushes
For example, we have a point on an immutable segment. If we use a
set-payload operation, we do copy-on-write. The point from immutable
segment A is deleted, the updated point is stored on appendable segment
B.
Because of flush ordering segment B (appendable) is flushed before
segment A (not-appendable). If the copy-on-write operation happens in
between, the point is deleted from A but the new point in B is not
persisted. We cannot recover this by replaying the WAL in case of a
crash because the point in A does not exist anymore, making
copy-on-write impossible.
Locking all segments prevents copy-on-write operations from occurring in
between flushes.
* Return proper status on set payload operations in segment
* Disable propagating point deletions to its vectors, it looses vectors
* Update vector tests after disabling point delete propagation to vectors
* Implement retry with exponential backoff for read-locking all segments
* Use try_read_for rather than sleeping after lock attempts
* Simplify locking many, just lock one by one
Refs: <https://github.com/qdrant/qdrant/pull/2527>
Co-authored-by: generall <andrey@vasnetsov.com>
* Comment out now obsolete test
* Error handling in method getting segment locks by ID
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* add ignore_plain_index to speed up search
* remove unnecessary & for vectors_batch
* format
* add special handle for proxy segments where the wrapped segment is plain
indexed
* review refactoring
* rollback changes in google.protobuf.rs
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add optional `tracing` crate dependency to all `lib/*` sub-crates
* Add optional alternative "tracing-logger"...
...that works a bit better with logs produced by the `tracing` crate
* Make `tracing-logger` reuse `log_level` config parameter (and `QDRANT__LOG_LEVEL` env-var)...
...instead of default `RUST_LOG` env-var only
* Replace `env_logger` with `tracing` (#2381)
* Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
TODO:
- Implement slog drain that would *properly* convert slog records into tracing events
(`slog-stdlog` is fine for now, though)
* fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Fix `consensus` test
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Forget to `unwrap` the result 🙈
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
`tracing_subscriber::fmt::init` initializes `tracing_log::LogTracer`
automatically with default `tracing-subscriber` features enabled 💁♀️
* Update `DEVELOPMENT.md` documentation regarding `tracing`
* Update default log level for development to make it less noisy
---------
Co-authored-by: timvisee <tim@visee.me>
* Implement "intelligent" default log-level filtering
---------
Co-authored-by: timvisee <tim@visee.me>
* optimize strings ram usage for map index
* conversion tests
* are you happy fmt
* better unicode test
* better unicode test
* are you happy clippy
* better comment
* are you happy fmt
* Use SmolStr instead of custom string
* Reoptimize segments on quantization mismatch
* Add collection quantization params to collection update REST endpoint
* Do not require rebuild of quantization for on_disk change
* Add collection quantization params to collection update gRPC endpoint
* Add option to update vector specific quantization config to REST API
* Add option to update vector specific quantization config to gRPC API
* Update OpenAPI and gRPC specification
* Fix quantization mismatch detecting not working as expected
* Fix config mismatch optimizer not using vector specific quantization
* Add unit test for quantization in config mismatch optimizer
* Add some quantization params to collection update integration test
* Reformat
* Apply suggestions from code review
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Remove if-statement, simply return
* Move quantization params in named vector in integration test
* Test updating quantization params in multivec integration test
* Fix gRPC docs
* Fix quantization rebuild on changing enabled state on indexed segment
* allow disabling quantization config
* fmt
* clippy
* compare all params of the quantization for rebuild
* explicitly use params on update
* test for disabling quantization
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* pass atomic bool from local shard to raw scorer
* pass atomic bool from local shard to raw scorer
* is_stopped in async scorer
* fmt
* is_stopped in quantized scorer
* terminating scorer if stopped
* enable timeout in local_shard
* allow timeout configuration
* use tokio spawn to ensure timeout handling if request is dropped
* Revert "use tokio spawn to ensure timeout handling if request is dropped"
This reverts commit 1068cf48d4.
* use stopping guard instead of task
* report error if search request is stopped
* fmt
* refactor transient error handelling
* dedup without large hashmap
* are you happy fmt
* add comment with iterator ordering
* Update lib/collection/src/collection_manager/holders/segment_holder.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* remove unnecessary continue
* review remarks
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* charabia tokenizer for CJK, init
* add charabia to .proto
* generate grpc docs
* rename charabia to multilingual
* disable charabia default features
* ignore stopwords and separators in tokenizer
* fix rebase conflict
* fix test
* fix test
* Bump `charabia` crate version...
...and enable `charabia` features that does not require any additional dependencies by default
* Expose optional `charabia` features in the `segment` crate
* Expose optional `charabia` features in the `qdrant` crate
* fix codespell
* more regression tests
* more regression tests
* make language-specific tests feature-flagged
---------
Co-authored-by: Anatolii Smolianinov <zarkonesmall@gmail.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
