* Deprecate init_from
* Mark memmap_threshold as deprecated
* Mark locks API as deprecated
* Mark RBAC collection access payload filter as deprecated
* Allow deprecations in tests and benchmarks
* rename cpu_budget -> resource_budget
* clippy
* add io budget to resources
* fmt
* move budget structures into a separate file
* add extend permit function
* dont extend existing permit
* switch from IO to CPU permit
* do not release resource before aquiring an extension
* fmt
* Review remarks
* Improve resource permit number assertion
* Make resource permit replace_with only acquire extra needed permits
* Remove obsolete drop implementation
* allocate IO budget same as CPU
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
* Enable mmap storage using blob store for payloads by default
* adapt test for new mmap payload storage
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* implement mmap sparse vector storage
* add to VectorStorageEnum
* clippy
* add tests, fix both simple and mmap storages
* smol correction on total_vector_count
* add sparse storage type to config
* fix reading config without storage type
* generate openapi
* use blob_store by path
* hidden setting to enable new storage
* validate existing path in `BlobStore::open()`
* use new dir for each sparse vector name
* fix and rename `max_point_offset`
Plus some extra refactors
* add storage compat test, to always check both storages work
* fix opening of storage + other misc fixes
* FIX!!!
`Unset` operations in the Tracker weren't updating the
`next_pointer_id`. So, when reopening the storage, those points wouldn't
get marked as deleted in the bitslice, thus creating the illusion that
they should exist, when they did not.
* refactor naming from `iter_*` to `for_each_*`
* fix checking for BlobStore existance
* fix typo
* fix error message
* better docs for open_or_create
* fix after rebase
* Add UUID to collection...
...and recreate collection, when applying Raft snapshot, if UUID of collection is different
* fixup! Add UUID to collection...
Remove UUID field from gRPC and exclude it from OpenAPI spec 🤡
* fixup! fixup! Add UUID to collection...
Always generate collection UUID 🤦♀️
* Raft snapshot recreate collection no expose UUID (#5452)
* separate colleciton config structure from API
* fmt
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add SnapshotsStorageConfig enum(Local or S3) and deserialize implementation
* [refactor] use snapshots_config instead of s3_config
* update config
* add AWS official`aws-sdk-s3`
* implement store_file() WITHOUT error handling
* implement list_snapshots
* implement delete_snapshot
* run `cargo +nightly fmt`
* delete println
* implement get_stored_file
* Add error handlings
* Refactor AWS S3 configuration and error handling
* fix bugs
* create an empty test file
* fix `alias_test.rs` for StorageConfig type
* tempolary delete some test and try s3 test
* Update integration-tests.yml to use snap instead of apt-get for installing yq
* Update integration-tests.yml to use sudo when installing yq
* add sudo
* make (full/non-full) snapshots downloadable
* debug
* small fix
* Add S3 endpoint URL configuration option
* fix
* fix
* debug
* fix endpoint
* update to http://127.0.0.1:9000/
* update
* fix
* fix `#[get("/collections/{collection}/shards/{shard}/snapshots/{snapshot}")]` for s3
* put original tests back
* refactor
* small fix (delete println & echo)
* use object_store and refactor
* create snapshot_storage_ops and implement
* Refactor get_appropriate_chunk_size function to adjust chunk size based on service limits and file size
* cargo +nightly fmt --all
* make it more abstract
* Refactor SnapshotsStorageConfig deserialization in SnapShotsConfig
* small update
* small fix
* Update dependencies in Cargo.lock
* Update minio image to satantime/minio-server
* Refactor snapshot storage paths in snapshots_manager.rs and snapshot_storage_ops.rs
* Fix issue with downloaded file size not matching expected size in download_snapshot function
* add flush
* Use Streaming instead of donloading once
* apply `cargo +nightly fmt --all`
* Fix issue with opening file in SnapshotStream::LocalFS variant
* Fix error handling in SnapshotStream::LocalFS variant
* Add integration test for Shard Snapshot API with S3 storage (#7)
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* Introduce Access object and use it
* Make Access non-optional
* Allow creating sharding key if collection claim is not present
* Tear up rbac crate into the main crate and storage crate
* [snapshots]: Define trait for snapshots managemet
* Define a trait with methods for managing snapshots after their
creation. The goal of this trait is to allow implementation of
snapshot management logic for different storage backends without major
changes in the code.
Signed-off-by: Alexandru Cihodaru <alexandru.cihodaru@gmail.com>
* [snapshots]: Move snapshot deletion logic
* Move the snapshot deletion logic into the implementation of
SnapshotStorage for LocalFileSystemConfig.
* Replace snapshot deletion logic with calls to the SnapshotStorage
implementation.
Signed-off-by: Alexandru Cihodaru <alexandru.cihodaru@gmail.com>
* [snapshots]: Move snapshot list logic
* Move the snapshot listing logic into the implementation of
SnapshotStorage for LocalFileSystemConfig.
Signed-off-by: Alexandru Cihodaru <alexandru.cihodaru@gmail.com>
* [snapshots]: Move snapshot creation logic
* Move the snapshot creation logic into the implementation of
SnapshotStorage for LocalFileSystemConfig.
Signed-off-by: Alexandru Cihodaru <alexandru.cihodaru@gmail.com>
* review fixes
* fmt
* refactor full storage snapshot
* create directory on get_stored_file + make sure temp archive file is removed
---------
Signed-off-by: Alexandru Cihodaru <alexandru.cihodaru@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Move CPU count function to common, fix wrong CPU count in visited list
* Change default number of rayon threads to 8
* Use CPU budget and CPU permits for optimizer tasks to limit utilization
* Respect configured thread limits, use new sane defaults in config
* Fix spelling issues
* Fix test compilation error
* Improve breaking if there is no CPU budget
* Block optimizations until CPU budget, fix potentially getting stuck
Our optimization worker now blocks until CPU budget is available to
perform the task.
Fix potential issue where optimization worker could get stuck. This
would happen if no optimization task is started because there's no
available CPU budget. This ensures the worker is woken up again to
retry.
* Utilize n-1 CPUs with optimization tasks
* Better handle situations where CPU budget is drained
* Dynamically scale rayon CPU count based on CPU size
* Fix incorrect default for max_indexing_threads conversion
* Respect max_indexing_threads for collection
* Make max_indexing_threads optional, use none to set no limit
* Update property documentation and comments
* Property max_optimization_threads is per shard, not per collection
* If we reached shard optimization limit, skip further checks
* Add remaining TODOs
* Fix spelling mistake
* Align gRPC comment blocks
* Fix compilation errors since last rebase
* Make tests aware of CPU budget
* Use new CPU budget calculation function everywhere
* Make CPU budget configurable in settings, move static budget to common
* Do not use static CPU budget, instance it and pass it through
* Update CPU budget description
* Move heuristic into defaults
* Fix spelling issues
* Move cpu_budget property to a better place
* Move some things around
* Minor review improvements
* Use range match statement for CPU count heuristics
* Systems with 1 or 2 CPUs do not keep cores unallocated by default
* Fix compilation errors since last rebase
* Update lib/segment/src/types.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/storage/src/content_manager/toc/transfer.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename cpu_budget to optimizer_cpu_budget
* Update OpenAPI specification
* Require at least half of the desired CPUs for optimizers
This prevents running optimizations with just one CPU, which could be
very slow.
* Don't use wildcard in CPU heuristic match statements
* Rename cpu_budget setting to optimizer_cpu_budget
* Update CPU budget comments
* Spell acquire correctly
* Change if-else into match
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename max_rayon_threads to num_rayon_threads, add explanation
* Explain limit in update handler
* Remove numbers for automatic selection of indexing threads
* Inline max_workers variable
* Remove CPU budget from ShardTransferConsensus trait, it is in collection
* small allow(dead_code) => cfg(test)
* Remove now obsolete lazy_static
* Fix incorrect CPU calculation in CPU saturation test
* Make waiting for CPU budget async, don't block current thread
* Prevent deadlock on optimizer signal channel
Do not block the optimization worker task anymore to wait for CPU budget
to be available. That prevents our optimizer signal channel from being
drained, blocking incoming updates because the cannot send another
optimizer signal. Now, prevent blocking this task all together and
retrigger the optimizers separately when CPU budget is available again.
* Fix incorrect CPU calculation in optimization cancel test
* Rename CPU budget wait function to notify
* Detach API changes from CPU saturation internals
This allows us to merge into a patch version of Qdrant. We can
reintroduce the API changes in the upcoming minor release to make all of
it fully functional.
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Do not select ourselves when finding auto shard transfer source
* Limit automatic shard transfers with basic transfer count check
* Extract shard recovery transfer request logic to separate function
* Add global shard transfer tracker structure to allow rate limiting
* Count and limit incoming and outgoing shard transfers separately
* Make automatic shard transfer limit configurable
* Move shard transfer tracker from global to collection level
* Comment out new config parameters
* Fix incorrect comment
* Fix missing space in log message
* Fix negated condition
* Remove logic for requesting shard transfer on replica state change
* Check shard transfer limits in consensus sync, use consensus state
Instead of bothering with tracking proposed shard transfers, this now
purely relies on state that is already in consensus. Each time we sync
consensus, we request the maximum number of shard transfers up to the
limit at that time.
* Remove now obsolete shard transfer tracker
* Revert now obsolete changes
* Improve transfer IO counting comment, revert now obsolete code
* Fix typos
* Fix flipped variables, don't take self reference on Copyable type
* Update lib/collection/src/collection/shard_transfer.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Handle incoming/outgoing transfer counts separately, don't tuple it
* Improve loop for counting incoming/outgoing transfers
* Remove unused test function
* Add consensus tests for automatic shard transfer limits
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Remove debug lines from new test
* Reorder last test a bit to resolve flakyness
* We can have one more transfer for recovery on the other alive node
* Attempt to reduce test flakyness, more points and more frequent polling
* Update config/config.yaml
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Explicitly note default shard transfer limit in configuration
* Use default for shard transfer IO everywhere
* Rename transfer limit check function to be more explicit
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Add config option to set default automatic shard transfer method
* Also use configured shard transfer method if user doesn't specify
This is not for automatic shard transfers, but when a user initiates a
transfer manually.
* Improve comment for shard transfer method configuration
* Fix test compilation
* Clone inside blocks
* Add shard transfer method to distinguish between batching and snapshots
* Add stub method to drive snapshot transfer
* Store remote shard in forward proxy, merge unproxy methods
* On snapshot shard transfer, create a shard snapshot
* Unify logic for unproxifying forward and queue proxy
* Error snapshot transfer if shard is not a queue proxy
* Add remote shard function to request remote HTTP port
* Handle all specific shard types when proxifying
* Allow queue proxy for some shard holder snapshot methods
* Bring local and remote shard snapshot transfer URLs into transfer logic
* Expose optional shard transfer method parameter in REST and gRPC API
* Expose shard transfer method in list of active transfers
* Fix off-by-one error in queue proxy shard batch transfer logic
* Do not set max ack version for WAL twice, already set when finalizing
* Merge comment for two similar calls
* Use reqwest client to transfer and recover shard snapshot on remote
Using the reqwest client should be temporary. We better switch to a gRPC
call here eventually to use our existing channels. That way we don't
require an extra HTTP client (and dependency) just for this.
* Send queue proxy updates to remote when shard is transferred
* On shard queue transfer, set max WAL ack to last transferred
* Add safe queue proxy destructor, skip destructing in error
This adds a finalize method to safely destruct a queue proxy shard. It
ensures that all remaining updates are transferred to the remote, and
that the max acknowledged version for our WAL is released. Only then is
the queue proxy shard destructed unwrapping the inner local shard.
Our unproxify logic now ensures that the queue proxy shard remains if
transferring the updates fails.
* Clean up method driving shard snapshot transfer a bit
* Change default shard transfer method to stream records
This changes the default transfer method to stream records rather than
using a snaphsot transfer. We can switch this once snapshot transfer is
fully integrated.
* Improve error handling, don't panic but return proper error
* Do not unwrap in type conversions
* Update OpenAPI and gRPC specification
* Resolve and remove some TODOs
* During shard snapshot transfer, use REST port from config
* Always release max acknowledged WAL version on queue proxy finalize
* Rework queue unproxying, transform into forward proxy to handle errors
When a queue or forward proxy shard needs to be unproxified into a local
shard again we typically don't have room to handle errors. A queue proxy
shard may error if it fails to send updates to the remote shard, while a
forward proxy does not fail at all when transforming.
We now transfer queued updates before a shard is unproxified. This
allows for proper error handling. After everything is transferred the
shard is transformed into a forward proxy which can eventually be safely
unproxified later.
* Add trace logging for transferring queue proxy updates in batch
* Simplify snapshot method conversion from gRPC
* Remove remote shard parameter
* Add safe guard to queue proxy handler, panic in debug if not finalized
* Improve safety and architecture of queue proxy shard
Switch from an explicit finalized flag to an outer-inner architecture.
This improves the interface and robustness of the type.
* Do not panic on drop if already unwinding
* Make REST port interface in channel service for local node explicitly
* Recover shard on remote over gRPC, remove reqwest client
* Use shard transfer priority for shard snapshot recovery
* Remove obsolete comment
* Simplify qualified path with use
* Don't construct URLs ourselves as a string, use `parse` and `set_port`
* Use `set_path` when building shard download URL
* Fix error handling in queue to forward proxy transformation
Before, we didn't handle finalization errors properly. If this failed,
tie shard would be lost. With this change the queue proxy shard is put
back.
* Set default shard transfer method to stream records, eliminate panics
* Fix shard snapshot transfer not correctly aborting due to queue proxy
When a shard transfer fails (for any reason), the transfer is aborted.
If we still have a queue proxy shard it should also be reverted, and
collected updates should be forgotten. Before this change it would try
to send all collected updates to the remote, even if the transfer
failed.
* Review fixes
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Review fixes
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Initiate forward and queue proxy shard in specialized transfer methods
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add consensus interface to shard transfer, repurpose dispatcher (#2873)
* Add shard transfer consensus interface
* Integrate shard transfer consensus interface into toc and transfer logic
* Repurpose dispatcher for getting consensus into shard transfer
* Derive clone
* Mark consensus as unused for now
* Use custom dispatcher with weak ref to prevent Arc cycle for ToC
* Add comment on why a weak reference is used
* Do exhaustive match in shard unproxy logic
* Restructure match statement, use match if
* When queue proxifying shard, allow forward proxy state if same remote
* Before retrying a shard transfer after error, destruct queue proxy
* Synchronize consensus across all nodes for shard snapshot transfer (#2874)
* Move await consensus commit functions into channel service
* Add shard consensus method to synchronize consensus across all nodes
* Move transfer config, channels and local address into snapshot transfer
* Await other nodes to reach consensus before finalizing shard transfer
* Do not fail right away awaiting consensus if still on older term
Instead, give the node time to reach the same term.
* Fix `await_commit_on_all_peers` not catching peer errors properly
* Change return type of `wait_for_consensus_commit` to `Result`
This is of course more conventional, and automatically sets `must_use`.
* Explicitly note number of peers when awaiting consensus
* Before consensus sync, wait for local shard to reach partial state
* Fix timeout error handling when waiting for replica set state
* Wait for replica set to have remote in partial state instead
* Set `(Partial)Snapshot` states for shard snapshot transfer through consensus (#2881)
* When doing a shard snapshot transfer, set shard to `PartialSnapshot`
* Add shard transfer method to set shard state to partial
It currently uses a naive implementation. Using a custom consensus
operation to also confirm a transfer is still active will be implemented
later.
* Add consensus snapshot transfer operation to change shard to partial
The operation `ShardTransferOperations::SnapshotRecovered` is called
after the shard snapshot is recovered on the remote and it progresses
the transfer further.
The operation sets the shard state from `PartialSnapshot` to `Partial`
and ensures the transfer is still active.
* Confirm consensus put shard into partial state, retry 3 times
* Get replica set once
* Add extensive shard snapshot transfer process docs, clean up function
* Fix typo
* Review suggestion
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Add delay between consensus confirmation retries
* Rename retry timeout to retry delay
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* On replicate shard, remember specified method
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* add update_concurrency to control the parallelism of updating shards
* reformat the files
* reformat
* add update_concurrency control to shard updates
* use unwrap_or
* test using buffer_ordered
* use buffered (testing)
* Pre-allocate space for update futures
* use NonZeroUsize
* linter
* use NonZeroUsize in the test
* consensus test for update_concurrency
* update config commit
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* pass atomic bool from local shard to raw scorer
* pass atomic bool from local shard to raw scorer
* is_stopped in async scorer
* fmt
* is_stopped in quantized scorer
* terminating scorer if stopped
* enable timeout in local_shard
* allow timeout configuration
* use tokio spawn to ensure timeout handling if request is dropped
* Revert "use tokio spawn to ensure timeout handling if request is dropped"
This reverts commit 1068cf48d4.
* use stopping guard instead of task
* report error if search request is stopped
* fmt
* refactor transient error handelling
* TableOfContent now accepts Handle, not Runtime
Runtime is left within main() and dropped there to shutdown the system
* Correct termination of consensus loop
Now consensus loop is joined by main thread and
correctly terminated between iterations of the loop
* Add clippy #[allow]
* Move complicated type into definition
* Remove duplicate on_consensus_stopped call
* wait_unwrap() is now time bounded
Drop is happening only if Arc<TableOfContent> is droped by all threads
in 30 seconds to limit shutdown time
---------
Co-authored-by: timvisee <tim@visee.me>
* Issue 1905: Configurable location for the tmp snapshot files
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* fix code review suggestions
* clippy fix
* Propagate temp path, use configured dir for snapshot creation
* Use real temp dir in snapshot tests
* Mention default temporary snapshot file path in configuration
* Use temp everywhere rather than a mix of temp and tmp
* Use consistent naming for temporary snapshot directories
* Extract logic for temporary storage path into toc method
* Resolve clippy warnings
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* async raw scorer
* fmt
* Disable `async_raw_scorer` on non-Linux platforms
* Refactor `async_raw_scorer.rs`
* Conditionally enable `async_raw_scorer` in `segment` crate
* Add `async_scorer` config parameter to the config...
...and enable `async_raw_scorer`, if config parameter is set to `true`
* fixup! Add `async_scorer` config parameter to the config...
Fix tests
* Add basic `async_raw_scorer` test
* Extend `async_raw_scorer` tests to be more extensive
* Async uring vector storage io uring (#2041)
* replace tokio-uring with just low level io-uring
* fnt
* minor fixes
* add sync
* wip: try to use less submissions
* fmt
* fix uring size
* larger buffer
* check for overflow
* submit with re-try
* mmap owns uring context
* large disk parallelism
* rollbacK: large disk parallelism
* fix windows build
* explicitly panic on uring fail
* fix windows build again
* use async scorer in the quantization re-scoring
* refactor
* rename UringReader
* refactor buffers
* fix for windows
* error checking
* fix handing
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Clone current segment config to deprecated type
* Remove segment level quantization config from segment config
* Also deprecate current VectorDataConfig
* Update old segment migration to work with new refactoring
* Move index into vector data config
* Move vector data config migration logic into segment level
* Remove hnsw_config from vector data config
* Rename collection params to vector data conversions function
* Move storage type into vector data config
* Set appendable flag correctly
* Clean up and reformat
* Make segment on disk flag not optional
* Add appendable flag to segment config to replace storage type
* Remove storage type from segment config
* Deprecate storage type enum
* Use consistent variable naming
* Cleanup
* Add segment config migration for v0.5.0 to current
* Bump segment to 0.6.0
* Remove serde defaults for new storage and vector data config types
These default value configurations are not needed anymore, because these
structs are not used to deserialize old data. All current fields should
always be available in these structs. When new fields are added in new
functions, the serde default annotation must be set again.
* Cleanup
* Update OpenAPI specification
This updates the returned data structure on telemetry endpoints, as a
result of segment configuration refactoring.
* Fix quantization configuration not falling back to collection config
* Fix compiler warning when building in release mode
* Move deprecated type structs into compat module
* Update allow deprecated attributes
* Assign quantization config only in segment optimizer
* Remove unsued parameter
* Add vector storage type enum to vector data config
* Remove appendable and on_disk flags from segment and vector config
* Update OpenAPI specification
* add tests
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* wip: chunked mmap
* Fix typo
* insert and get methods
* dynamic bitvec
* clippy
* wip: vector storage
* wip: fmt
* wip: mmap chunks
* wip: mmap problems
* Share transmuted mutable reference over mmap
* option to enable appendable mmap vectors
* fmt
* rename storage status file
* update tests
* fix get deleted value range
* add recovery to vector storage tests
* add flush to tests
* fix transmute from immutable to mutable
* make transmuted pointer private
* remove unused unsafe functions
* force WAL flush if wait=true
* move wal flush into updater thread
* remove flush from update api
* Minimize pub visibility for specialized/dangerous functions
* Allocate vector with predefined capacity
* Inline format parameters
* Assert we have multiple chunks while testing, test is useless otherwise
* Remove unnecessary scope
* Remove unnecessary dereference
* Random bool has 0.5 as standard distribution, use iter::repeat_with
* Replace RemovableMmap::new with Default derive
* Rename len to num_flags
* Use Option replace as it is convention alongside take
* Add FileId enum to replace error prone manual ID rotating
* Use debug_assert_eq where applicable
* Refactor drop and set to replace
* Change default chunk size for chunked mmap vectors to 32MB
This change is made as per GitHub review, because allocating a few
storages with 128MB would take a significant amount of time and storage.
See: https://github.com/qdrant/qdrant/pull/1838#discussion_r1187215475
* Replace for-loops with iterators
* Draft: add typed mmap to improve code safety (#1860)
* Add typed mmap
* Replace some crude mmap usages with typed mmap
* Use typed mmap for deleted flags
* Simplify dynamic mmap flags a lot with new typed mmap, remove flags option
* Reformat
* Remove old mmap functions that are now unused
* Reimplement mmap locking for mmap_vectors
* Add MmapBitSlice tests
* Replace MmapChunk with new typed mmap
* Update docs
* Clean-up
* Disable alignment assertions on Windows for now
* Rename mmap lock to mlock to prevent confusion with lockable types
* one more small test
* Some review fixes
* Add aliasing note
* Add basic error handling in typed mmap constructors
* Use typed mmap error handling throughout project
* Move mmap type module to common
* Fix transmute functions being unsound
See https://github.com/qdrant/qdrant/pull/1860#discussion_r1188593854
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* WIP: Start working on out-of-RAM errors handling [skip ci]
* Implement basic handling of out-of-RAM errors during Qdrant startup
* Try to fix CI fail by allowing both V1 and V2 cgroups
* Try to fix CI fail by improving cgroups handling
* Fix cgroups path detection/handling (+ some minor stylistic changes)
* fixup! Fix cgroups path detection/handling (+ some minor stylistic changes)
* Add test
* Enable low RAM test
* fixup! Add test
* free memory checks
* rm unused function
* Oom fallback script (#1809)
* add recover mode in qdrant + script for handelling OOM
* fix clippy
* reformat entrypoint.sh
* fix test
* add logging to test
* fix test
* fix test
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add QuantizationConfigDiff type
* Add quantization config diff to vector parameters
* Prefer vector config over collection config for quantization
* Update OpenAPI specification
* Validate quantization configuration quantile in 0.5-1.0 range
As per https://github.com/qdrant/qdrant/pull/1681
* Add test if check if vector specific quantization config is persisted
* Alias quantization to quantization_config in vector parameters
* Remove quantization config diff, use full vector specific config instead
* Regenerate OpenAPI specification and gRPC docs
* Fix compilation error
* Add error handling to quantization config conversions
* Fix quantization integration test, make HNSW test stricter
* Validate VectorConfig/VectorParams, remove obsolete validation
* Add HNSW config diff to vector parameters
* Validate params in collection config
* Add HNSW config to segment vector data config
* Add VectorsConfig params iterator for more elegant conversions
* Prefer vector HNSW config over collection config for building HNSW index
* Base segment vector param HNSW config on collection config
* General improvements
* Rewrite HNSW ef_construct extract function to also consider vector configs
* Update OpenAPI specification
* Add test to check if vector specific HNSW config is persisted
* review changes
* review changes
* Regenerate gRPC docs
* Fix test on Windows
* Regenerate OpenAPI specification
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* use common channel pool for healthcheck requests
* add dynamic channel pool
* fmt
* lower parallel connections num
* fmt
* fix clippy
* fix clippy
* better status error message
* smarter channel dropping conditions
* per-channel stats
* refactor pool
* rollback dockerfile
* health-check timeout
* prevent reporting last active peer as dead
* introduce update rate limit
* only rate-limit client requests
* ability to save snapshot without locking wal (#1685)
* ability to save snapshot without locking wal
* fix empty wal saving
* skip waiting on local shard in listener mode even if it is a direct request to listener shard
* snapshot recovery test + better handling of segment versions
* use latest wal
* review changes
