* add ignore_plain_index to speed up search
* remove unnecessary & for vectors_batch
* format
* add special handle for proxy segments where the wrapped segment is plain
indexed
* review refactoring
* rollback changes in google.protobuf.rs
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add optional `tracing` crate dependency to all `lib/*` sub-crates
* Add optional alternative "tracing-logger"...
...that works a bit better with logs produced by the `tracing` crate
* Make `tracing-logger` reuse `log_level` config parameter (and `QDRANT__LOG_LEVEL` env-var)...
...instead of default `RUST_LOG` env-var only
* Replace `env_logger` with `tracing` (#2381)
* Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
TODO:
- Implement slog drain that would *properly* convert slog records into tracing events
(`slog-stdlog` is fine for now, though)
* fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Fix `consensus` test
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Forget to `unwrap` the result 🙈
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
`tracing_subscriber::fmt::init` initializes `tracing_log::LogTracer`
automatically with default `tracing-subscriber` features enabled 💁♀️
* Update `DEVELOPMENT.md` documentation regarding `tracing`
* Update default log level for development to make it less noisy
---------
Co-authored-by: timvisee <tim@visee.me>
* Implement "intelligent" default log-level filtering
---------
Co-authored-by: timvisee <tim@visee.me>
* optimize strings ram usage for map index
* conversion tests
* are you happy fmt
* better unicode test
* better unicode test
* are you happy clippy
* better comment
* are you happy fmt
* Use SmolStr instead of custom string
* Reoptimize segments on quantization mismatch
* Add collection quantization params to collection update REST endpoint
* Do not require rebuild of quantization for on_disk change
* Add collection quantization params to collection update gRPC endpoint
* Add option to update vector specific quantization config to REST API
* Add option to update vector specific quantization config to gRPC API
* Update OpenAPI and gRPC specification
* Fix quantization mismatch detecting not working as expected
* Fix config mismatch optimizer not using vector specific quantization
* Add unit test for quantization in config mismatch optimizer
* Add some quantization params to collection update integration test
* Reformat
* Apply suggestions from code review
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Remove if-statement, simply return
* Move quantization params in named vector in integration test
* Test updating quantization params in multivec integration test
* Fix gRPC docs
* Fix quantization rebuild on changing enabled state on indexed segment
* allow disabling quantization config
* fmt
* clippy
* compare all params of the quantization for rebuild
* explicitly use params on update
* test for disabling quantization
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* pass atomic bool from local shard to raw scorer
* pass atomic bool from local shard to raw scorer
* is_stopped in async scorer
* fmt
* is_stopped in quantized scorer
* terminating scorer if stopped
* enable timeout in local_shard
* allow timeout configuration
* use tokio spawn to ensure timeout handling if request is dropped
* Revert "use tokio spawn to ensure timeout handling if request is dropped"
This reverts commit 1068cf48d4.
* use stopping guard instead of task
* report error if search request is stopped
* fmt
* refactor transient error handelling
* dedup without large hashmap
* are you happy fmt
* add comment with iterator ordering
* Update lib/collection/src/collection_manager/holders/segment_holder.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* remove unnecessary continue
* review remarks
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* charabia tokenizer for CJK, init
* add charabia to .proto
* generate grpc docs
* rename charabia to multilingual
* disable charabia default features
* ignore stopwords and separators in tokenizer
* fix rebase conflict
* fix test
* fix test
* Bump `charabia` crate version...
...and enable `charabia` features that does not require any additional dependencies by default
* Expose optional `charabia` features in the `segment` crate
* Expose optional `charabia` features in the `qdrant` crate
* fix codespell
* more regression tests
* more regression tests
* make language-specific tests feature-flagged
---------
Co-authored-by: Anatolii Smolianinov <zarkonesmall@gmail.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add github action to codespell master on push and PRs
* Add rudimentary codespell config
* some skips
* fix some ambigous typos
* [DATALAD RUNCMD] run codespell throughout
=== Do not change lines below ===
{
"chain": [],
"cmd": "codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Add dev branch as target for the workflow
* nitpick: use CardinalityEstimation::exact() instead of explicit struct
* wip: implement binary index
* persist binary index
* add tests, refactor binary memory
* use bitflags for BinaryItem, use early return from @timvisee 's review
* fix iterator's end +1
* fix indexed_count, keep track of falses and trues counts
* Fix true/false counts if flag for point was set already
* Remove match in filter structure
* - rename `*_count()`
- fix set_or_insert to actually set
- add more tests
* review suggestions for binary index (#2186)
* small taste refactors
* move memory into its own module
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
This is one of a series of commits for the new feature Geo Filter by Polygon(#795) that add function polygon_hashes that retrieves as-high-as-possible with maximum of max_regions number of geo-hash guaranteed to contain the whole polygon.
It also includes two helper functions: 1) check_polygon_intersection and 2) minimum_bounding_rectangle_for_polygon.
Test cases validate the functionality for different scenarios.
* Use unwrap_or_default rather than constructing it manually
* Validate oversampling parameter and surrounding structs
* Validate search limit to be 1 or higher
* Validate hnsw_ef parameter to be 4 or higher
* Also validate parameters in in gRPC
* Update OpenAPI specification
* Add unlikely to reach comment
* Remove hnsw_ef validation rules for now
* Update OpenAPI specification
