mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-27 16:37:42 -05:00
Daniel Borosandtimvisee
b2b509cf73
* shard: persist proxy pending changes on flush, stop holding back WAL ack Hook the pending changes component into the proxy segment's flush: the proxy flusher first persists the buffered operations into the pending changes log, then passes the flush along to the wrapped segment. The proxy's `persistent_version` now covers what the log durably holds on top of what the wrapped segment persisted itself. That is what lifts the WAL cap proxies imposed so far. `flush_all` compares each segment's version against its persistent version; a proxy used to report only the wrapped segment's persisted version while its own version climbed with every buffered operation, so the WAL could never be acknowledged past the point the proxy was created at, and a restart replayed all of it — potentially very expensive operations, such as an update by filter, all over again. With the buffered state durable on disk the generic rule acknowledges the full version, and a restart recovers it from the log instead. Dropping a proxy's data drops the component first, which waits for any in-flight pending changes flusher so it cannot append to the segment directory while that is being deleted. Update the proxy flush test to the new semantics, add a segment holder test asserting the acknowledged version advances past a proxied delete, and update the ack pin rationale in `finish_optimization`: the pin is still needed after the proxies leave the holder, it just snapshots a persistent version that now includes the log. * Persist wrapped segment before pending changes Prevents raising version of proxy segment too early * collection: test crash recovery through persisted proxy changes End-to-end test of the persisted pending changes: wrap every segment of a local shard in a proxy, delete points so the deletes are only buffered, flush, and assert the acknowledgeable version covers them. Then acknowledge the WAL up to that version, drop the shard without ever propagating the proxies, and load it again: the deletes are gone from the WAL and must come back through the pending changes logs. The delete under test is deliberately not the last WAL entry, as the acknowledge never passes the last entry and that one is always replayed. * test: cover untested pending proxy changes paths * test: pin persisted proxy changes findings * Fix bad merge * Remove corrupt length test, we cannot detect if last entry was corrupt Remove a test that asserts an entry that isn't the last cannot have a corrupt length. We cannot reliably detect whether the invalid length was the last entry or not, because nothing else tells us how many entries we expect in the file. At the same time we don't expect random bit flips. So I removed the test. * Simulate segments flush to clear pending changes log file * Update test, also assert proxy segment version * Fix bad merge * Fix blocked test * Enable necessary feature flags in tests (2/2) --------- Co-authored-by: timvisee <tim@visee.me>