On a Tesla T4 the backend exited during the first /generate with no
traceback and no HTTP response, leaving the client with
RemoteDisconnected and every later call with ConnectionRefused. Three
separate defects combined, which is why none of the reporter's
workarounds helped.
1. torch.compile(mode="reduce-overhead") captures CUDA graphs. T4
(sm_75) passed the existing arch gate, so capture was attempted and
aborted the process from inside the native CUDA library — below the
interpreter, where neither the #278 eager-fallback wrapper nor any
except clause can see it. The compile mode is now resolved per GPU:
Ampere (sm_80) and newer keep the cudagraph mode, older cards drop to
the non-cudagraph "default" mode and keep their compiled Inductor
kernels. Fails open on any probe error, so no GPU that works today
loses the optimization. OMNIVOICE_FORCE_CUDAGRAPH=1 restores it.
2. should_torch_compile() never read TORCH_COMPILE_DISABLE. main.py sets
it on win32, build_engine_env injected it into subprocesses, and
docs/install/windows.md tells users to export it — but the in-process
gate ignored it, so the reporter exported the documented variable and
still got "torch.compile applied". The gate now honours
TORCH_COMPILE_DISABLE / TORCHDYNAMO_DISABLE / TORCHINDUCTOR_DISABLE on
every platform, and an env opt-out on the parent propagates to engine
subprocesses. The settings DB path is logged alongside the toggle:
the reporter had three omnivoice.db files and edited one the backend
never opened.
3. Settings -> Performance -> "Disable torch.compile" was rendered
disabled outside Windows in both the Tauri and Electron UIs, so the
one control that would have stopped this was unreachable for the
affected Linux user. The toggle is now live on every platform, and
build_engine_env honours it everywhere rather than only on win32.
Also arms faulthandler before torch is imported, so a fatal native
signal writes the faulting thread's Python stack to backend_err.log
instead of the process vanishing silently. This does not prevent a
crash; it makes one diagnosable. OMNIVOICE_DISABLE_FAULTHANDLER=1 skips
it.
Tests fail before / pass after, verified by stashing the source and
running the new tests against unfixed code. The crash test kills a real
child interpreter with a real SIGSEGV and requires a named Python frame
in the output. test_torch_compile_path_gate's fixture now clears the
compile-disable env vars: main.py setdefaults them on win32, so on a
Windows runner they leaked into os.environ and decided those tests.
Not verified on real hardware — no Turing GPU available. The sm_80 floor
is inferred from the crash report and from docs/hardware-notes-tesla-t4.md,
which already flagged cudagraphs on T4 as attempted by default and never
evaluated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-up. The budget matched model names by substring, so a custom
repo whose name contains "turbo", "small" or "base" (or a word such as
"database") got a reduced budget and could be admitted to CUDA without
enough memory. Reduced budgets now apply only to the exact OpenAI
checkpoint ids, .en variants included. Any other repository, fine-tunes
included, keeps the conservative 5.0 GB, as the engine doc says.
Review follow-up. A generation first waits in the GPU pool's queue, on its
own clock (GPU_QUEUE_TIMEOUT_S, 1800 s by default), before its execution
budget starts, so a 630 s wait could still give up before the backend
returned its queue error. The generate wait now adds the queue budget.
Transcription is unchanged: run_transcribe_guarded starts its 300 s clock
at submission, so queue time already counts against it.
Parity tests pin each tool's wait above the backend's own worst case,
from ASR_TRANSCRIBE_TIMEOUT_S, GPU_QUEUE_TIMEOUT_S and generate_timeout_s
on cpu, cuda and mps. A later backend change that outgrows the MCP wait
now fails a test. A new case covers a CPU budget larger than the GPU one.
The CUDA preflight demanded 5.0 GB of free VRAM for every model, sized for
full large-v3. The default model is large-v3-turbo, which the pipeline
loads in fp16: about 1.6 GB of weights, not the 3.2 GiB fp32 figure in the
old comment. So a 6 GB card with nothing else resident reported 5.0 GB
free and was sent to CPU every time, although CUDA ran the same audio in
37 s against minutes on CPU.
The budget is now fp16 weights + 1.5 GB workspace (batch 16) + 0.5 GB
headroom, per model and capped at the old 5.0 GB. That is 3.6 GB for turbo
and 5.0 GB for full large-v3 and any unrecognised model. Both CPU-fallback
warnings name the model and the OMNIVOICE_ASR_VRAM_PREFLIGHT=0 opt-out.
The engine doc lists the budgets.
Fixes#2041
The MCP tools gave up on a backend POST after OMNIVOICE_MCP_TIMEOUT_S,
default 120 s, while the backend's own budgets run longer: 300 s for ASR,
and 300 s or more for generation. A transcription the backend would have
finished came back as an empty client-side timeout, and the abandoned
job kept holding the worker. The docstring said the timeout followed
OMNIVOICE_GENERATE_TIMEOUT_S, but it never read it.
Unset, each tool now waits for the backend's budget plus 30 s, never less
than 120 s. transcribe follows OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S, and
generate_speech follows the larger of the GPU and CPU generation budgets,
scaled by text length like the backend. An explicit
OMNIVOICE_MCP_TIMEOUT_S still wins. docs/mcp.md says so.
Fixes#2040
XNU stops signalling a process as soon as it starts exiting but posts
NOTE_EXIT later in the same exit. A KILL that lands in that window gets
EPERM while the exit probe still reads "alive", so stopping a child that
TERM had just ended could fail with "Operation not permitted". This
flaked the macOS run of contained_exit_probe_preserves_a_live_child.
The EPERM branch now re-probes for up to 250 ms before treating the error
as a live, unsignalable root. Live roots, reaped roots and probe failures
are still errors.
Conflicts: RecoBanner (deleted here: the recommendation card is gone),
ModelStoreTab + EngineCompatibilityMatrix (this branch's rewrite kept),
supertonic3 backend (main's own-venv check kept, message without the
retired "→ Engines" step), CHANGELOG (base layout + the #2020 line).
Carried over from main and the review:
- The list row hides Install while only the license review is left
(main's #2017 rule, now in the row; Accept lives in the panel).
- Engine action aria labels go through i18n (engines.aria*, all 21
locales) instead of hardcoded English.
- Every "Model Catalogue → Engines/Models" path main added, plus the
frontend strings that still named the retired panes, now point at the
one-page catalogue.
- test_engine_unavailable_reason_1866 reads the license matcher from its
new home, engines/engineDisplay.js.
Palash Debnath