mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-28 00:47:32 -05:00
Daniel Borosandtimvisee
76a8647728
* test: pin persisted proxy segments follow-up findings * fix: tests * Respect `up_to` in flushing pending proxy changes * Fix unproxy divergence, centralize logic in single shared function * Don't pass locked segments we don't use * Close the post-swap window losing acknowledged proxy changes `finish_optimization` left a window between `swap_new` and the end of the function where the propagated proxy changes were durable in no reachable place. The proxies had left the holder, so a flush pass no longer saw them as unsaved work and acknowledged the WAL past what their pending changes logs persisted, while their source files were still on disk contradicting the newer state. Their `ack_pin` was only registered at the very end, and the optimized segment, the only durable home of those changes, had no version file yet, so `normalize_segment_dir` deleted it on the next load. A failure or crash in that window lost every change past the proxies' logs. Close it from both sides, reordering only: - Save the optimized segment's version file right after the flush that made the propagated changes durable, before the swap. That flush is what `SegmentBuilder::build` postponed the version file for, so the segment is loadable from the swap onwards. - Register the proxies' deferred destruction (and with it their `ack_pin`) under the same write lock that evicted them, so no flush pass can observe the holder without either the proxies or their pin. Collecting the deferred point ids moves up with the registration, as it borrows the swapped-out proxies. The deferred destruction still cannot run before the manifest is synced: `locked_proxies` holds the segments alive until the end of the function, so `try_drop_data` retries until then. * Rename the optimization test hook after the window it guards The hook no longer sits before the version save, it marks a failure anywhere in the window after the optimized segment was swapped in. Rename it and the test accordingly, and restate the test doc as the invariant that window must uphold rather than the bug it used to describe. * Pin WAL ack while creating snapshot * Patch test that was stuck * Initialize necessary feature flags in tests * Correctly propagate changes in two stages, lock updates on second stage * Use existing WAL ack pinning infrastructure * test: pin unproxy phase 2 propagation failure losing acknowledged changes * test: pin WAL ack pin at zero suppressing clock persistence * Fix propagate and unproxy data consistency error on failure * Store clocks before checking WAL ack pin * fix: wait for the flush worker when stopping it in tests * fix: linter --------- Co-authored-by: timvisee <tim@visee.me>