* immutable map index integration
* remove wipe
* fix unit tests
* get appendable flag from config
* minor refactoring
* fix chunked mmap appendable flag
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* deactivate failed shard o transfer if possible
* revert unused changes
* initialize empty shard in case of data transfer into dummy
* clippy fix
* account for an automatic recovery of the corrupted shard
* Centralize generic range validation
* Centralize validate not empty
* Centralize collection name validation
* Add geo polygon validation in REST API and centralize validation
* Add tests for common validation functions
* Don't use reference in generic range validation function
* Other improvements
* WIP: Handle errors during shard snapshot recovery
* fixup! WIP: Handle errors during shard snapshot recovery
Fix typo
* Set shard state to `Dead` on shard snapshot restore failure
* Log failures during error recovery 🙈
* Handle `LocalShard::clear` failure during during local replica restore
* Initialize `DummyShard` if local replica restore failed
* Mark peer as "locally disabled" if local replica restore failed
* Notify peer failure with `ShardReplicaSet::notify_peer_failure_cb`
* fixup! Notify peer failure with `ShardReplicaSet::notify_peer_failure_cb`
Fix typo
* minor refactoring
* fmt
* fmt
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* immutable map index
* use (super) instead of (crate)
* more comments
* decompose remove_point
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Delete temp snapshot files on error
* shorter resulting path to please max length on Windows
* Remove one layer of tmp folder for shards
Unicity of segment tmp folder provided by Uuid
* On segment flush, read-lock all segments to prevent CoW between flushes
For example, we have a point on an immutable segment. If we use a
set-payload operation, we do copy-on-write. The point from immutable
segment A is deleted, the updated point is stored on appendable segment
B.
Because of flush ordering segment B (appendable) is flushed before
segment A (not-appendable). If the copy-on-write operation happens in
between, the point is deleted from A but the new point in B is not
persisted. We cannot recover this by replaying the WAL in case of a
crash because the point in A does not exist anymore, making
copy-on-write impossible.
Locking all segments prevents copy-on-write operations from occurring in
between flushes.
* Return proper status on set payload operations in segment
* Disable propagating point deletions to its vectors, it looses vectors
* Update vector tests after disabling point delete propagation to vectors
* Implement retry with exponential backoff for read-locking all segments
* Use try_read_for rather than sleeping after lock attempts
* Simplify locking many, just lock one by one
Refs: <https://github.com/qdrant/qdrant/pull/2527>
Co-authored-by: generall <andrey@vasnetsov.com>
* Comment out now obsolete test
* Error handling in method getting segment locks by ID
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add queue proxy shard type
This is similar to the forward proxy shard, but instead of forwarding
operations to a remote it queues them. These queued operations can be
transferred to a remote at a later time.
* Add queue proxy methods for shard to replica set
* Allow to specify max ack WAL version to prevent truncating too early
* Prevent truncating WAL messages a queue proxy shard still relies on
* Fix off by one error, don't batch last item that is already sent
* Minor improvements
* Use forward_update rather than update
* Fix allowing forward proxy into queue proxy for same peer
* Add name to optimizers
* Track optimizer status in update handler
* Remove unused optimizer telemetry implementation
* Report tracked optimizer status in local shard telemetry
* Keep just the last 16 optimizer trackers and non successful ones
* Also eventually truncate cancelled optimizer statuses
* Fix codespell
* Assert basic optimizer log state in unit test
* Remove repetitive suffix from optimizer names
* Loosen requirements for optimizer status test to prevent flakyness
* Add gRPC endpoint to request remote HTTP port
* Add basic gRPC test for fetching remote HTTP port
* Extract gRPC endpoint for fetching HTTP port into internal service
* Add basic internal gRPC test
* Other minor improvements
* add ignore_plain_index to speed up search
* remove unnecessary & for vectors_batch
* format
* add special handle for proxy segments where the wrapped segment is plain
indexed
* review refactoring
* rollback changes in google.protobuf.rs
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add optional `tracing` crate dependency to all `lib/*` sub-crates
* Add optional alternative "tracing-logger"...
...that works a bit better with logs produced by the `tracing` crate
* Make `tracing-logger` reuse `log_level` config parameter (and `QDRANT__LOG_LEVEL` env-var)...
...instead of default `RUST_LOG` env-var only
* Replace `env_logger` with `tracing` (#2381)
* Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
TODO:
- Implement slog drain that would *properly* convert slog records into tracing events
(`slog-stdlog` is fine for now, though)
* fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Fix `consensus` test
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
Forget to `unwrap` the result 🙈
* fixup! fixup! Replace `env_logger` with `tracing`/`tracing-subscriber`/`tracing-log`
`tracing_subscriber::fmt::init` initializes `tracing_log::LogTracer`
automatically with default `tracing-subscriber` features enabled 💁♀️
* Update `DEVELOPMENT.md` documentation regarding `tracing`
* Update default log level for development to make it less noisy
---------
Co-authored-by: timvisee <tim@visee.me>
* Implement "intelligent" default log-level filtering
---------
Co-authored-by: timvisee <tim@visee.me>
* optimize strings ram usage for map index
* conversion tests
* are you happy fmt
* better unicode test
* better unicode test
* are you happy clippy
* better comment
* are you happy fmt
* Use SmolStr instead of custom string
