Four-angle /simplify on the cumulative branch diff:
- The idle-reaper grace flag now lives entirely inside _get_generator's
lock: the prefetch claims it only when THAT call builds the model, and
every other getter call consumes it. This deletes the duplicated
call-site clears in embed/detect (detect no longer touches the
generator's grace at all — it was clearing a flag for a model it never
uses), and closes the lock-gap window where the prefetch's claim could
land on an already-used model, which the old comment claimed was
impossible.
- Shared _env_float(name, default) for main.py's three inline float-env
parsers (capture delay, watermark delay, MCP start timeout): one
NaN/negative-rejecting implementation instead of three drifting
copies; the older two lacked the isfinite guard entirely.
- Test cleanups: dead isinstance-Future assert half removed, the
fake-audioseal Event-wait simplified to sleep, the reaper-diversion
guard simplified to a plain no-op lambda, stale setdefault sentence
dropped from the conftest comment.
Skipped with reason: merging the double will_mark() gate (they guard
different invariants — pool creation vs model load, both tested) and
hoisting the reaper guard to conftest (an autouse module-attr patch
would break tests that verify release_idle_models directly).
CodeRabbit on 28c7bace:
1. (Major) get_watermark_pool's double-checked pattern re-read the
global after an unlocked null-check, so shutdown_watermark_pool's
reset could land in between and the caller received None. The
executor is now captured and returned under _watermark_pool_lock.
2. (Minor) the idle-grace test overwrote _prefetched_unused after the
embed call, making the embed's clearing unobservable — a failing
embed would have passed unnoticed. It now asserts the flag directly,
and a guard diverts any leaked idle reaper (idle_worker resolves
release_idle_models per call) to a no-op for the test's duration.
Second CI red on the same test, different assert: the conftest fix killed
the leaked PRELOAD task, but a test lifespan that exits without shutdown
also leaves idle_worker running, and idle_worker calls
release_idle_models on these same module globals from another thread —
re-stamping _last_used mid-test. Each phase of the test now re-
establishes its preconditions immediately before its release call and
pins now= to a far-future monotonic, so an interleaved reaper tick
cannot change the outcome. Verified against the full 5801-test suite
run in one process.
The shutdown drain killed the module singleton with no replacement, so
any process that keeps running after a lifespan shutdown — the CI suite
does exactly this — dead-submitted on the next watermark op: "cannot
schedule new futures after shutdown" (CI red; independently confirmed
by Greptile P1, CodeRabbit Major, and the plugin code review at 95/100
confidence). shutdown_watermark_pool() now resets the singleton under
its build lock before draining, so the next get_watermark_pool() hands
out a live replacement. Regression test covers
drained-pool-refuses + replacement-accepts.
Same round, minor findings: the drain's except now logs with exc_info
instead of a bare pass (GHAS CodeQL empty-except); the watermark delay
knob rejects negative/non-finite overrides (CodeRabbit); conftest sets
OMNIVOICE_PRELOAD_WATERMARK=0 unconditionally so a stray export from
the runner shell cannot re-enable background warm-ups mid-suite
(CodeRabbit).
Two CI failures, both understood:
1. test_shutdown_preload_race_1000 pins the production _cancel_and_await
_tasks call site by regex; the new fifth handle broke the pattern. The
guard now pins all FIVE handles (its property — every preload handle
awaited under one generous bound — is unchanged).
2. test_prefetched_model_gets_one_extra_idle_window flaked only in the
full suite: many tests boot the app lifespan, and any that exits
without a lifespan shutdown leaves the deferred watermark-preload
task pending — 35s later it fires mid-suite in another thread and
re-stamps _last_used under whatever test is running. conftest now
defaults OMNIVOICE_PRELOAD_WATERMARK=0 for the test session (a test
can still opt in), and the grace test neutralizes will_mark so a
leaked warm-up can't touch it.
Bot findings: Greptile P1 + CodeRabbit — cancelling the preload task
doesn't stop a watermark-pool thread already inside the ~42s cold
import, and nothing drained that pool at shutdown (only the GPU pool
was reset). Shutdown now drains the watermark pool's queue
(shutdown(wait=False, cancel_futures=True)) — bounded abandon, same
documented reality that Python can't kill a running thread. CodeRabbit
Major: the warm-up reads its own delay knob
(OMNIVOICE_PRELOAD_WATERMARK_DELAY, default 35s) instead of reusing the
capture-ASR delay, so a capture env override no longer retimes it.
CodeRabbit Minor: the _prefetched_unused claim/clear transitions now
happen under _generator_lock, so the retention grace can't be granted
to a model that has actually been used; the test fixture resets all
lifecycle globals.
Skipped with reason: gating prefetch on local-checkpoint presence — the
warm-up downloads only what the first embed would download anyway;
time-shifting that download is the feature, not a new network call.
The first mark_synthetic serialized the audioseal import plus the
generator load INSIDE the first synthesis — measured at ~42s inline on
a cold filesystem (macOS, 2026-08-17 report), pushing a cold first
synthesis to ~87s and 3s past a 90s client timeout. The generator now
warms on a background task ~35s after boot (+5s past the capture-ASR
warm so the two cold imports don't contend), on the watermark pool,
cancellable at shutdown (OMNIVOICE_PRELOAD_WATERMARK=0 opts out; the
pool is only created when will_mark() says watermarking is active, and
setup-half failures log immediately instead of surfacing at shutdown).
Because the prefetch thread races the first embed, the lazy builds now
hold per-model locks — one build per model, no cross-blocking: a
detector load no longer queues behind a ~42s generator build, and
release_idle_models takes both locks in a fixed order. A
prefetch-warmed, never-used generator survives ONE extra idle-reaper
window so a first synthesis shortly after boot still finds it warm;
real embed/detect use clears the grace.
Also: embed/detect failures now log the full traceback (exc_info). The
catch-all printed only the message, which today left a
ModuleNotFoundError('getopt') inside AudioSeal's forward undiagnosable
from the log — audio silently ships unmarked when this fires.
debpalash