* Centralize generic range validation
* Centralize validate not empty
* Centralize collection name validation
* Add geo polygon validation in REST API and centralize validation
* Add tests for common validation functions
* Don't use reference in generic range validation function
* Other improvements
* Add gRPC endpoint to request remote HTTP port
* Add basic gRPC test for fetching remote HTTP port
* Extract gRPC endpoint for fetching HTTP port into internal service
* Add basic internal gRPC test
* Other minor improvements
* add ignore_plain_index to speed up search
* remove unnecessary & for vectors_batch
* format
* add special handle for proxy segments where the wrapped segment is plain
indexed
* review refactoring
* rollback changes in google.protobuf.rs
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add optional `tracing` crate dependency to all `lib/*` sub-crates
* Add optional alternative "tracing-logger"...
...that works a bit better with logs produced by the `tracing` crate
* Make `tracing-logger` reuse `log_level` config parameter (and `QDRANT__LOG_LEVEL` env-var)...
...instead of default `RUST_LOG` env-var only
* Replace `env_logger` with `tracing` (#2381)
* Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
TODO:
- Implement slog drain that would *properly* convert slog records into tracing events
(`slog-stdlog` is fine for now, though)
* fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Fix `consensus` test
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Forget to `unwrap` the result 🙈
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
`tracing_subscriber::fmt::init` initializes `tracing_log::LogTracer`
automatically with default `tracing-subscriber` features enabled 💁♀️
* Update `DEVELOPMENT.md` documentation regarding `tracing`
* Update default log level for development to make it less noisy
---------
Co-authored-by: timvisee <tim@visee.me>
* Implement "intelligent" default log-level filtering
---------
Co-authored-by: timvisee <tim@visee.me>
* Reoptimize segments on quantization mismatch
* Add collection quantization params to collection update REST endpoint
* Do not require rebuild of quantization for on_disk change
* Add collection quantization params to collection update gRPC endpoint
* Add option to update vector specific quantization config to REST API
* Add option to update vector specific quantization config to gRPC API
* Update OpenAPI and gRPC specification
* Fix quantization mismatch detecting not working as expected
* Fix config mismatch optimizer not using vector specific quantization
* Add unit test for quantization in config mismatch optimizer
* Add some quantization params to collection update integration test
* Reformat
* Apply suggestions from code review
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Remove if-statement, simply return
* Move quantization params in named vector in integration test
* Test updating quantization params in multivec integration test
* Fix gRPC docs
* Fix quantization rebuild on changing enabled state on indexed segment
* allow disabling quantization config
* fmt
* clippy
* compare all params of the quantization for rebuild
* explicitly use params on update
* test for disabling quantization
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add update vector config type
* Add mutable getter to update vector config
* Add update vector config to update collection REST operation, reorder
* Add logic to update vector HNSW in collection
* Update OpenAPI specification
* If HNSW diff is empty, consider it as unset
* Update descriptions to clarify an empty HNSW diff unsets
* Make new vector HNSW diff update existing diff
This means that the existing vector HNSW diff is kept intact. Specified
fields are updated in the existing diff. An empty HNSW diff object may
be provided to unset the diff.
* Add update vector config to update collection gRPC operation
* Update gRPC docs
* Use vector specific HNSW config in integration test
* Extract & improve gRPC type conversions, fix collection update with None
* Validate vector specific HNSW config in collection update gRPC endpoint
* Explicitly test we do not rebuild on on_disk change
* Use new method to recreate optimizers
* Don't test on_disk change anymore
* Reuse collection write lock to save, do not relock
* Fix invalid update collection request body in OpenAPI test
* Simplify update collection test
* Add update collection test for HNSW parameters
* Extract vector param update logic into local functions
* Reformat
* Transform UpdateVectorParams into VectorParamsDiff
* Transform UpdateVectorsConfig into VectorsConfigDiff
* Combine serde attributes
* Make HNSW diff empty check generic over DiffConfig types
* Move DiffConfig implementation
* Fix typo
* charabia tokenizer for CJK, init
* add charabia to .proto
* generate grpc docs
* rename charabia to multilingual
* disable charabia default features
* ignore stopwords and separators in tokenizer
* fix rebase conflict
* fix test
* fix test
* Bump `charabia` crate version...
...and enable `charabia` features that does not require any additional dependencies by default
* Expose optional `charabia` features in the `segment` crate
* Expose optional `charabia` features in the `qdrant` crate
* fix codespell
* more regression tests
* more regression tests
* make language-specific tests feature-flagged
---------
Co-authored-by: Anatolii Smolianinov <zarkonesmall@gmail.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add github action to codespell master on push and PRs
* Add rudimentary codespell config
* some skips
* fix some ambigous typos
* [DATALAD RUNCMD] run codespell throughout
=== Do not change lines below ===
{
"chain": [],
"cmd": "codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Add dev branch as target for the workflow
* Use unwrap_or_default rather than constructing it manually
* Validate oversampling parameter and surrounding structs
* Validate search limit to be 1 or higher
* Validate hnsw_ef parameter to be 4 or higher
* Also validate parameters in in gRPC
* Update OpenAPI specification
* Add unlikely to reach comment
* Remove hnsw_ef validation rules for now
* Update OpenAPI specification
* wip: add rest param
* WIP: Refactor `search_with_graph` to use oversampling
* fixup! WIP: Refactor `search_with_graph` to use oversampling
Fix `search_with_graph` quantization config handling
* fixup! WIP: Refactor `search_with_graph` to use oversampling
Use `truncate` instead of `shrink_to` 🤦♀️
* schrink -> truncate
* protection against wrong oversampling value
* Revert second "fixup! WIP: Refactor `search_with_graph` to use oversampling"
Accidentally commited the wrong file 🤦♀️
* fix max method notation
* rollbacK: max notation
* Add `oversampling` field to `QuantizationSearchParams` gRPC type
* Simplify `oversampled_top` calculation
* `./tools/generate_grpc_docs.sh`
* `./tools/generate_openapi_models.sh`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* rename `Lookup` -> `RetrievedLookup`
* feat: expose lookup in groups
- make `with_vectors` and `with_payload` optional
- include in grpc
- move GroupId to inside of PointGroup message
- include `WithLookup` inside of `GroupRequest`, doesn't make sense to keep them separated
* chore: remove allow(dead_code) from builder
* feat: introduce `WithLookupInterface` to allow specifying only the name of the lookup collection
* fix: WithLookupInterface with collection name only brings all payload and vectors
* fix: update grpc docs
* add tests and set better defaults
* remove `RetrievedLookup` from proto
* short-circuit `fill_search_result_with_payload` when it's not needed
* cargo fmt
* proto: revert putting `GroupId` inside `PointGroup`
* update grpc docs
* Remove WithLookupInterface sugar from grpc
- set with_vectors of lookup default to false
- set with_payload of lookup default to true
* update integration tests
* use singular lookup
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* fix: validate collection name while creating collection
* fix: validate collection name for Windows
* Use simpler method for checking if collection name contains illegal char
* Add strict collection name validation to gRPC create collection
* Add strict collection name validation to REST create collection
* Add validation error rendering for does_not_contain
* Remove collection name validation from storage internals
* Add some test cases for stricter collection name validation
* Revert ToC changes
---------
Co-authored-by: timvisee <tim@visee.me>
* Clone current segment config to deprecated type
* Remove segment level quantization config from segment config
* Also deprecate current VectorDataConfig
* Update old segment migration to work with new refactoring
* Move index into vector data config
* Move vector data config migration logic into segment level
* Remove hnsw_config from vector data config
* Rename collection params to vector data conversions function
* Move storage type into vector data config
* Set appendable flag correctly
* Clean up and reformat
* Make segment on disk flag not optional
* Add appendable flag to segment config to replace storage type
* Remove storage type from segment config
* Deprecate storage type enum
* Use consistent variable naming
* Cleanup
* Add segment config migration for v0.5.0 to current
* Bump segment to 0.6.0
* Remove serde defaults for new storage and vector data config types
These default value configurations are not needed anymore, because these
structs are not used to deserialize old data. All current fields should
always be available in these structs. When new fields are added in new
functions, the serde default annotation must be set again.
* Cleanup
* Update OpenAPI specification
This updates the returned data structure on telemetry endpoints, as a
result of segment configuration refactoring.
* Fix quantization configuration not falling back to collection config
* Fix compiler warning when building in release mode
* Move deprecated type structs into compat module
* Update allow deprecated attributes
* Assign quantization config only in segment optimizer
* Remove unsued parameter
* Add vector storage type enum to vector data config
* Remove appendable and on_disk flags from segment and vector config
* Update OpenAPI specification
* add tests
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* nested object filter
* code review
* add support for must_not in nested
* extract functions
* support and test must_not in SimpleConditionChecker
* add index matching unit test (to be continued)
* remove extra clone
* test with should
* WIP: Nested object filter suggestions (#1855)
* switch to bitvec
* fix clippy
* more tests
* fmt
* fix some tests
* add test with text
* support for nested should
* do not rely on indexes for nested queries & fix test
* use index to make index-aware checks in nested payload
* fix value-count tests
* re-fa-cto-ring
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* wip: chunked mmap
* Fix typo
* insert and get methods
* dynamic bitvec
* clippy
* wip: vector storage
* wip: fmt
* wip: mmap chunks
* wip: mmap problems
* Share transmuted mutable reference over mmap
* option to enable appendable mmap vectors
* fmt
* rename storage status file
* update tests
* fix get deleted value range
* add recovery to vector storage tests
* add flush to tests
* fix transmute from immutable to mutable
* make transmuted pointer private
* remove unused unsafe functions
* force WAL flush if wait=true
* move wal flush into updater thread
* remove flush from update api
* Minimize pub visibility for specialized/dangerous functions
* Allocate vector with predefined capacity
* Inline format parameters
* Assert we have multiple chunks while testing, test is useless otherwise
* Remove unnecessary scope
* Remove unnecessary dereference
* Random bool has 0.5 as standard distribution, use iter::repeat_with
* Replace RemovableMmap::new with Default derive
* Rename len to num_flags
* Use Option replace as it is convention alongside take
* Add FileId enum to replace error prone manual ID rotating
* Use debug_assert_eq where applicable
* Refactor drop and set to replace
* Change default chunk size for chunked mmap vectors to 32MB
This change is made as per GitHub review, because allocating a few
storages with 128MB would take a significant amount of time and storage.
See: https://github.com/qdrant/qdrant/pull/1838#discussion_r1187215475
* Replace for-loops with iterators
* Draft: add typed mmap to improve code safety (#1860)
* Add typed mmap
* Replace some crude mmap usages with typed mmap
* Use typed mmap for deleted flags
* Simplify dynamic mmap flags a lot with new typed mmap, remove flags option
* Reformat
* Remove old mmap functions that are now unused
* Reimplement mmap locking for mmap_vectors
* Add MmapBitSlice tests
* Replace MmapChunk with new typed mmap
* Update docs
* Clean-up
* Disable alignment assertions on Windows for now
* Rename mmap lock to mlock to prevent confusion with lockable types
* one more small test
* Some review fixes
* Add aliasing note
* Add basic error handling in typed mmap constructors
* Use typed mmap error handling throughout project
* Move mmap type module to common
* Fix transmute functions being unsound
See https://github.com/qdrant/qdrant/pull/1860#discussion_r1188593854
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
