* Add healthz, livez and readyz endpoints
* Describe healthz, livez and readyz endpoints in OpenAPI definition
* Add integration test for healthz, livez and readyz endpoints
* Fix typo
* Add name to optimizers
* Track optimizer status in update handler
* Remove unused optimizer telemetry implementation
* Report tracked optimizer status in local shard telemetry
* Keep just the last 16 optimizer trackers and non successful ones
* Also eventually truncate cancelled optimizer statuses
* Fix codespell
* Assert basic optimizer log state in unit test
* Remove repetitive suffix from optimizer names
* Loosen requirements for optimizer status test to prevent flakyness
* Add gRPC endpoint to request remote HTTP port
* Add basic gRPC test for fetching remote HTTP port
* Extract gRPC endpoint for fetching HTTP port into internal service
* Add basic internal gRPC test
* Other minor improvements
* add ignore_plain_index to speed up search
* remove unnecessary & for vectors_batch
* format
* add special handle for proxy segments where the wrapped segment is plain
indexed
* review refactoring
* rollback changes in google.protobuf.rs
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add optional `tracing` crate dependency to all `lib/*` sub-crates
* Add optional alternative "tracing-logger"...
...that works a bit better with logs produced by the `tracing` crate
* Make `tracing-logger` reuse `log_level` config parameter (and `QDRANT__LOG_LEVEL` env-var)...
...instead of default `RUST_LOG` env-var only
* Replace `env_logger` with `tracing` (#2381)
* Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
TODO:
- Implement slog drain that would *properly* convert slog records into tracing events
(`slog-stdlog` is fine for now, though)
* fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Fix `consensus` test
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Forget to `unwrap` the result 🙈
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
`tracing_subscriber::fmt::init` initializes `tracing_log::LogTracer`
automatically with default `tracing-subscriber` features enabled 💁♀️
* Update `DEVELOPMENT.md` documentation regarding `tracing`
* Update default log level for development to make it less noisy
---------
Co-authored-by: timvisee <tim@visee.me>
* Implement "intelligent" default log-level filtering
---------
Co-authored-by: timvisee <tim@visee.me>
* Reoptimize segments on quantization mismatch
* Add collection quantization params to collection update REST endpoint
* Do not require rebuild of quantization for on_disk change
* Add collection quantization params to collection update gRPC endpoint
* Add option to update vector specific quantization config to REST API
* Add option to update vector specific quantization config to gRPC API
* Update OpenAPI and gRPC specification
* Fix quantization mismatch detecting not working as expected
* Fix config mismatch optimizer not using vector specific quantization
* Add unit test for quantization in config mismatch optimizer
* Add some quantization params to collection update integration test
* Reformat
* Apply suggestions from code review
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Remove if-statement, simply return
* Move quantization params in named vector in integration test
* Test updating quantization params in multivec integration test
* Fix gRPC docs
* Fix quantization rebuild on changing enabled state on indexed segment
* allow disabling quantization config
* fmt
* clippy
* compare all params of the quantization for rebuild
* explicitly use params on update
* test for disabling quantization
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* unit-test for updating multivec config
* Remove single vector support from vector params update in REST API
With this change, a user is always required to specify vector names when
updating their parameters. Updating vector parameters in a collection
with a single vector is still possible by providing an empty name. This
is very reasonable as updating vector parameters on a single vector
collection is a bad pattern, since parameters can be set on the
collection itself.
* Update integration tests to reflect REST API change
* Update OpenAPI specification
* Remove obsolete VectorsConfigDiff functions, update docs
* Update OpenAPI specification
* Add name validation to update collection endpoint for vector params
---------
Co-authored-by: timvisee <tim@visee.me>
* Add update vector config type
* Add mutable getter to update vector config
* Add update vector config to update collection REST operation, reorder
* Add logic to update vector HNSW in collection
* Update OpenAPI specification
* If HNSW diff is empty, consider it as unset
* Update descriptions to clarify an empty HNSW diff unsets
* Make new vector HNSW diff update existing diff
This means that the existing vector HNSW diff is kept intact. Specified
fields are updated in the existing diff. An empty HNSW diff object may
be provided to unset the diff.
* Add update vector config to update collection gRPC operation
* Update gRPC docs
* Use vector specific HNSW config in integration test
* Extract & improve gRPC type conversions, fix collection update with None
* Validate vector specific HNSW config in collection update gRPC endpoint
* Explicitly test we do not rebuild on on_disk change
* Use new method to recreate optimizers
* Don't test on_disk change anymore
* Reuse collection write lock to save, do not relock
* Fix invalid update collection request body in OpenAPI test
* Simplify update collection test
* Add update collection test for HNSW parameters
* Extract vector param update logic into local functions
* Reformat
* Transform UpdateVectorParams into VectorParamsDiff
* Transform UpdateVectorsConfig into VectorsConfigDiff
* Combine serde attributes
* Make HNSW diff empty check generic over DiffConfig types
* Move DiffConfig implementation
* Fix typo
* charabia tokenizer for CJK, init
* add charabia to .proto
* generate grpc docs
* rename charabia to multilingual
* disable charabia default features
* ignore stopwords and separators in tokenizer
* fix rebase conflict
* fix test
* fix test
* Bump `charabia` crate version...
...and enable `charabia` features that does not require any additional dependencies by default
* Expose optional `charabia` features in the `segment` crate
* Expose optional `charabia` features in the `qdrant` crate
* fix codespell
* more regression tests
* more regression tests
* make language-specific tests feature-flagged
---------
Co-authored-by: Anatolii Smolianinov <zarkonesmall@gmail.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add github action to codespell master on push and PRs
* Add rudimentary codespell config
* some skips
* fix some ambigous typos
* [DATALAD RUNCMD] run codespell throughout
=== Do not change lines below ===
{
"chain": [],
"cmd": "codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Add dev branch as target for the workflow
* Use unwrap_or_default rather than constructing it manually
* Validate oversampling parameter and surrounding structs
* Validate search limit to be 1 or higher
* Validate hnsw_ef parameter to be 4 or higher
* Also validate parameters in in gRPC
* Update OpenAPI specification
* Add unlikely to reach comment
* Remove hnsw_ef validation rules for now
* Update OpenAPI specification
* wip: add rest param
* WIP: Refactor `search_with_graph` to use oversampling
* fixup! WIP: Refactor `search_with_graph` to use oversampling
Fix `search_with_graph` quantization config handling
* fixup! WIP: Refactor `search_with_graph` to use oversampling
Use `truncate` instead of `shrink_to` 🤦♀️
* schrink -> truncate
* protection against wrong oversampling value
* Revert second "fixup! WIP: Refactor `search_with_graph` to use oversampling"
Accidentally commited the wrong file 🤦♀️
* fix max method notation
* rollbacK: max notation
* Add `oversampling` field to `QuantizationSearchParams` gRPC type
* Simplify `oversampled_top` calculation
* `./tools/generate_grpc_docs.sh`
* `./tools/generate_openapi_models.sh`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* rename `Lookup` -> `RetrievedLookup`
* feat: expose lookup in groups
- make `with_vectors` and `with_payload` optional
- include in grpc
- move GroupId to inside of PointGroup message
- include `WithLookup` inside of `GroupRequest`, doesn't make sense to keep them separated
* chore: remove allow(dead_code) from builder
* feat: introduce `WithLookupInterface` to allow specifying only the name of the lookup collection
* fix: WithLookupInterface with collection name only brings all payload and vectors
* fix: update grpc docs
* add tests and set better defaults
* remove `RetrievedLookup` from proto
* short-circuit `fill_search_result_with_payload` when it's not needed
* cargo fmt
* proto: revert putting `GroupId` inside `PointGroup`
* update grpc docs
* Remove WithLookupInterface sugar from grpc
- set with_vectors of lookup default to false
- set with_payload of lookup default to true
* update integration tests
* use singular lookup
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* feat: add lookup_ids function
- rename GroupId -> PseudoId
- impl try_from PseudoId to PointIdType
- create lookup_ids function
- add tests
* fix: keep `GroupId` name in the openapi schema
* refactor: make new type for PseudoId
* refactor: make new `Lookup` output for lookup_ids()
* changes from review, thanks @agourlay
* Add optional `tracing` support
TODO:
- add documentation
* Add explicit compile-time check for `--cfg tokio_unstable` rust flags...
...if `console-subscriber` feature is enabled
* Add basic documentation
* fixup! Add basic documentation
Add links, reworded few phrases and document `tracing` feature
* fixup! fixup! Add basic documentation
- Fix `tokio-tracing` link
* fixup! Add basic documentation
Too much code-blocks...
* Move tracing setup into a separate function
* fixup! Move tracing setup into a separate function
- move tracing setup to a separate file
- fix error handling
* fixup! Move tracing setup into a separate function
Add missing file 🙈
* Clone current segment config to deprecated type
* Remove segment level quantization config from segment config
* Also deprecate current VectorDataConfig
* Update old segment migration to work with new refactoring
* Move index into vector data config
* Move vector data config migration logic into segment level
* Remove hnsw_config from vector data config
* Rename collection params to vector data conversions function
* Move storage type into vector data config
* Set appendable flag correctly
* Clean up and reformat
* Make segment on disk flag not optional
* Add appendable flag to segment config to replace storage type
* Remove storage type from segment config
* Deprecate storage type enum
* Use consistent variable naming
* Cleanup
* Add segment config migration for v0.5.0 to current
* Bump segment to 0.6.0
* Remove serde defaults for new storage and vector data config types
These default value configurations are not needed anymore, because these
structs are not used to deserialize old data. All current fields should
always be available in these structs. When new fields are added in new
functions, the serde default annotation must be set again.
* Cleanup
* Update OpenAPI specification
This updates the returned data structure on telemetry endpoints, as a
result of segment configuration refactoring.
* Fix quantization configuration not falling back to collection config
* Fix compiler warning when building in release mode
* Move deprecated type structs into compat module
* Update allow deprecated attributes
* Assign quantization config only in segment optimizer
* Remove unsued parameter
* Add vector storage type enum to vector data config
* Remove appendable and on_disk flags from segment and vector config
* Update OpenAPI specification
* add tests
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Flatten points selector in delete vectors REST endpoint
This makes the vector deletion API consistent with other endpoints using
the point selector.
* Update tests for flattened delete vectors point selector
* make api consistent with existing ones
* update openapi
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* nested object filter
* code review
* add support for must_not in nested
* extract functions
* support and test must_not in SimpleConditionChecker
* add index matching unit test (to be continued)
* remove extra clone
* test with should
* WIP: Nested object filter suggestions (#1855)
* switch to bitvec
* fix clippy
* more tests
* fmt
* fix some tests
* add test with text
* support for nested should
* do not rely on indexes for nested queries & fix test
* use index to make index-aware checks in nested payload
* fix value-count tests
* re-fa-cto-ring
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* wip: chunked mmap
* Fix typo
* insert and get methods
* dynamic bitvec
* clippy
* wip: vector storage
* wip: fmt
* wip: mmap chunks
* wip: mmap problems
* Share transmuted mutable reference over mmap
* option to enable appendable mmap vectors
* fmt
* rename storage status file
* update tests
* fix get deleted value range
* add recovery to vector storage tests
* add flush to tests
* fix transmute from immutable to mutable
* make transmuted pointer private
* remove unused unsafe functions
* force WAL flush if wait=true
* move wal flush into updater thread
* remove flush from update api
* Minimize pub visibility for specialized/dangerous functions
* Allocate vector with predefined capacity
* Inline format parameters
* Assert we have multiple chunks while testing, test is useless otherwise
* Remove unnecessary scope
* Remove unnecessary dereference
* Random bool has 0.5 as standard distribution, use iter::repeat_with
* Replace RemovableMmap::new with Default derive
* Rename len to num_flags
* Use Option replace as it is convention alongside take
* Add FileId enum to replace error prone manual ID rotating
* Use debug_assert_eq where applicable
* Refactor drop and set to replace
* Change default chunk size for chunked mmap vectors to 32MB
This change is made as per GitHub review, because allocating a few
storages with 128MB would take a significant amount of time and storage.
See: https://github.com/qdrant/qdrant/pull/1838#discussion_r1187215475
* Replace for-loops with iterators
* Draft: add typed mmap to improve code safety (#1860)
* Add typed mmap
* Replace some crude mmap usages with typed mmap
* Use typed mmap for deleted flags
* Simplify dynamic mmap flags a lot with new typed mmap, remove flags option
* Reformat
* Remove old mmap functions that are now unused
* Reimplement mmap locking for mmap_vectors
* Add MmapBitSlice tests
* Replace MmapChunk with new typed mmap
* Update docs
* Clean-up
* Disable alignment assertions on Windows for now
* Rename mmap lock to mlock to prevent confusion with lockable types
* one more small test
* Some review fixes
* Add aliasing note
* Add basic error handling in typed mmap constructors
* Use typed mmap error handling throughout project
* Move mmap type module to common
* Fix transmute functions being unsound
See https://github.com/qdrant/qdrant/pull/1860#discussion_r1188593854
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add update/delete REST endpoints
* Add update/delete vector REST endpoints to OpenAPI specification
* Fix update vector with no changes not triggering a validation error
* Update OpenAPI specification for updated request type
* Add segment entry function to update named vectors
* Use already available function to update existing vectors
We already had a segment function to update existing named vectors. This
change ensure we use that instead of separating it separately. As a
bonus, this adds support for setting multiple named vectors at once.
* Update set vectors ourselves, don't drop omitted vectors
* Refactor vector updating functions, separate update and replace
* Add basic vector ops, add update/delete functionality to segment updater
* Add internal and public gRPC types and actions for vectors
* Add gRPC API actions
* Reformat
* Add VectorOperations to vector ops, add basic validation
* Validate gRPC vector types
* Validate vector operation structs
* Construct PointIdsList through From trait
* Update gRPC docs
* Use VectorsSelector for vector deletions in gRPC
* Add support for updating multiple points/vectors in update vectors API
* Update gRPC docs
* Fix incorrect gRPC type numbering
* Return point ID error from vector update/delete functions if not found
* Fix disbalanced vectors test
* feat: use 0 as disable for indexing and memmap
* amend: edit test, update doc comments, remove validation
* fix: use constant instead of literal, reword doc comments
* test: update test for loop and timeout
* fix: remove validation on OptimizerConfigDiff and fix test_validatioin.py
* fix: remove validation from build.rs, remove internal hack from doccomments
* fix: update grpc docs
* revert max_optimization_threads validation in test
* update config comments and minor test change
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
