mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-03 16:40:54 -05:00
5bdf5e4bfe
* Clone inside blocks * Add shard transfer method to distinguish between batching and snapshots * Add stub method to drive snapshot transfer * Store remote shard in forward proxy, merge unproxy methods * On snapshot shard transfer, create a shard snapshot * Unify logic for unproxifying forward and queue proxy * Error snapshot transfer if shard is not a queue proxy * Add remote shard function to request remote HTTP port * Handle all specific shard types when proxifying * Allow queue proxy for some shard holder snapshot methods * Bring local and remote shard snapshot transfer URLs into transfer logic * Expose optional shard transfer method parameter in REST and gRPC API * Expose shard transfer method in list of active transfers * Fix off-by-one error in queue proxy shard batch transfer logic * Do not set max ack version for WAL twice, already set when finalizing * Merge comment for two similar calls * Use reqwest client to transfer and recover shard snapshot on remote Using the reqwest client should be temporary. We better switch to a gRPC call here eventually to use our existing channels. That way we don't require an extra HTTP client (and dependency) just for this. * Send queue proxy updates to remote when shard is transferred * On shard queue transfer, set max WAL ack to last transferred * Add safe queue proxy destructor, skip destructing in error This adds a finalize method to safely destruct a queue proxy shard. It ensures that all remaining updates are transferred to the remote, and that the max acknowledged version for our WAL is released. Only then is the queue proxy shard destructed unwrapping the inner local shard. Our unproxify logic now ensures that the queue proxy shard remains if transferring the updates fails. * Clean up method driving shard snapshot transfer a bit * Change default shard transfer method to stream records This changes the default transfer method to stream records rather than using a snaphsot transfer. We can switch this once snapshot transfer is fully integrated. * Improve error handling, don't panic but return proper error * Do not unwrap in type conversions * Update OpenAPI and gRPC specification * Resolve and remove some TODOs * During shard snapshot transfer, use REST port from config * Always release max acknowledged WAL version on queue proxy finalize * Rework queue unproxying, transform into forward proxy to handle errors When a queue or forward proxy shard needs to be unproxified into a local shard again we typically don't have room to handle errors. A queue proxy shard may error if it fails to send updates to the remote shard, while a forward proxy does not fail at all when transforming. We now transfer queued updates before a shard is unproxified. This allows for proper error handling. After everything is transferred the shard is transformed into a forward proxy which can eventually be safely unproxified later. * Add trace logging for transferring queue proxy updates in batch * Simplify snapshot method conversion from gRPC * Remove remote shard parameter * Add safe guard to queue proxy handler, panic in debug if not finalized * Improve safety and architecture of queue proxy shard Switch from an explicit finalized flag to an outer-inner architecture. This improves the interface and robustness of the type. * Do not panic on drop if already unwinding * Make REST port interface in channel service for local node explicitly * Recover shard on remote over gRPC, remove reqwest client * Use shard transfer priority for shard snapshot recovery * Remove obsolete comment * Simplify qualified path with use * Don't construct URLs ourselves as a string, use `parse` and `set_port` * Use `set_path` when building shard download URL * Fix error handling in queue to forward proxy transformation Before, we didn't handle finalization errors properly. If this failed, tie shard would be lost. With this change the queue proxy shard is put back. * Set default shard transfer method to stream records, eliminate panics * Fix shard snapshot transfer not correctly aborting due to queue proxy When a shard transfer fails (for any reason), the transfer is aborted. If we still have a queue proxy shard it should also be reverted, and collected updates should be forgotten. Before this change it would try to send all collected updates to the remote, even if the transfer failed. * Review fixes Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Review fixes Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Initiate forward and queue proxy shard in specialized transfer methods Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Add consensus interface to shard transfer, repurpose dispatcher (#2873) * Add shard transfer consensus interface * Integrate shard transfer consensus interface into toc and transfer logic * Repurpose dispatcher for getting consensus into shard transfer * Derive clone * Mark consensus as unused for now * Use custom dispatcher with weak ref to prevent Arc cycle for ToC * Add comment on why a weak reference is used * Do exhaustive match in shard unproxy logic * Restructure match statement, use match if * When queue proxifying shard, allow forward proxy state if same remote * Before retrying a shard transfer after error, destruct queue proxy * Synchronize consensus across all nodes for shard snapshot transfer (#2874) * Move await consensus commit functions into channel service * Add shard consensus method to synchronize consensus across all nodes * Move transfer config, channels and local address into snapshot transfer * Await other nodes to reach consensus before finalizing shard transfer * Do not fail right away awaiting consensus if still on older term Instead, give the node time to reach the same term. * Fix `await_commit_on_all_peers` not catching peer errors properly * Change return type of `wait_for_consensus_commit` to `Result` This is of course more conventional, and automatically sets `must_use`. * Explicitly note number of peers when awaiting consensus * Before consensus sync, wait for local shard to reach partial state * Fix timeout error handling when waiting for replica set state * Wait for replica set to have remote in partial state instead * Set `(Partial)Snapshot` states for shard snapshot transfer through consensus (#2881) * When doing a shard snapshot transfer, set shard to `PartialSnapshot` * Add shard transfer method to set shard state to partial It currently uses a naive implementation. Using a custom consensus operation to also confirm a transfer is still active will be implemented later. * Add consensus snapshot transfer operation to change shard to partial The operation `ShardTransferOperations::SnapshotRecovered` is called after the shard snapshot is recovered on the remote and it progresses the transfer further. The operation sets the shard state from `PartialSnapshot` to `Partial` and ensures the transfer is still active. * Confirm consensus put shard into partial state, retry 3 times * Get replica set once * Add extensive shard snapshot transfer process docs, clean up function * Fix typo * Review suggestion Co-authored-by: Luis Cossío <luis.cossio@qdrant.com> * Add delay between consensus confirmation retries * Rename retry timeout to retry delay --------- Co-authored-by: Luis Cossío <luis.cossio@qdrant.com> * On replicate shard, remember specified method --------- Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
