On a Tesla T4 the backend exited during the first /generate with no
traceback and no HTTP response, leaving the client with
RemoteDisconnected and every later call with ConnectionRefused. Three
separate defects combined, which is why none of the reporter's
workarounds helped.
1. torch.compile(mode="reduce-overhead") captures CUDA graphs. T4
(sm_75) passed the existing arch gate, so capture was attempted and
aborted the process from inside the native CUDA library — below the
interpreter, where neither the #278 eager-fallback wrapper nor any
except clause can see it. The compile mode is now resolved per GPU:
Ampere (sm_80) and newer keep the cudagraph mode, older cards drop to
the non-cudagraph "default" mode and keep their compiled Inductor
kernels. Fails open on any probe error, so no GPU that works today
loses the optimization. OMNIVOICE_FORCE_CUDAGRAPH=1 restores it.
2. should_torch_compile() never read TORCH_COMPILE_DISABLE. main.py sets
it on win32, build_engine_env injected it into subprocesses, and
docs/install/windows.md tells users to export it — but the in-process
gate ignored it, so the reporter exported the documented variable and
still got "torch.compile applied". The gate now honours
TORCH_COMPILE_DISABLE / TORCHDYNAMO_DISABLE / TORCHINDUCTOR_DISABLE on
every platform, and an env opt-out on the parent propagates to engine
subprocesses. The settings DB path is logged alongside the toggle:
the reporter had three omnivoice.db files and edited one the backend
never opened.
3. Settings -> Performance -> "Disable torch.compile" was rendered
disabled outside Windows in both the Tauri and Electron UIs, so the
one control that would have stopped this was unreachable for the
affected Linux user. The toggle is now live on every platform, and
build_engine_env honours it everywhere rather than only on win32.
Also arms faulthandler before torch is imported, so a fatal native
signal writes the faulting thread's Python stack to backend_err.log
instead of the process vanishing silently. This does not prevent a
crash; it makes one diagnosable. OMNIVOICE_DISABLE_FAULTHANDLER=1 skips
it.
Tests fail before / pass after, verified by stashing the source and
running the new tests against unfixed code. The crash test kills a real
child interpreter with a real SIGSEGV and requires a named Python frame
in the output. test_torch_compile_path_gate's fixture now clears the
compile-disable env vars: main.py setdefaults them on win32, so on a
Windows runner they leaked into os.environ and decided those tests.
Not verified on real hardware — no Turing GPU available. The sm_80 floor
is inferred from the crash report and from docs/hardware-notes-tesla-t4.md,
which already flagged cudagraphs on T4 as attempted by default and never
evaluated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.9.21+ Linux wheels are built with -march=native and crash first
synthesis on many CPUs. Keep the last known-good 0.9.20 in lock.
Co-authored-by: Cursor <cursoragent@cursor.com>
Review follow-up. The budget matched model names by substring, so a custom
repo whose name contains "turbo", "small" or "base" (or a word such as
"database") got a reduced budget and could be admitted to CUDA without
enough memory. Reduced budgets now apply only to the exact OpenAI
checkpoint ids, .en variants included. Any other repository, fine-tunes
included, keeps the conservative 5.0 GB, as the engine doc says.
Review follow-up. A generation first waits in the GPU pool's queue, on its
own clock (GPU_QUEUE_TIMEOUT_S, 1800 s by default), before its execution
budget starts, so a 630 s wait could still give up before the backend
returned its queue error. The generate wait now adds the queue budget.
Transcription is unchanged: run_transcribe_guarded starts its 300 s clock
at submission, so queue time already counts against it.
Parity tests pin each tool's wait above the backend's own worst case,
from ASR_TRANSCRIBE_TIMEOUT_S, GPU_QUEUE_TIMEOUT_S and generate_timeout_s
on cpu, cuda and mps. A later backend change that outgrows the MCP wait
now fails a test. A new case covers a CPU budget larger than the GPU one.
Review follow-up. The shared loader resamples soundfile-readable audio to
16 kHz with a linear interpolation, which can alias 44.1/48 kHz input into
Whisper's band. PyTorch Whisper now reads such audio at its native rate,
as before, so the pipeline's band-limited resampler converts it. Only
files soundfile can't open (MP4/M4A) take the ffmpeg path, which resamples
properly. New tests: a 48 kHz stereo WAV reaches the pipeline at 48 kHz
without ffmpeg, a POST /transcribe with an .m4a returns 200, and the MCP
tool names M4A bytes .m4a.
The CUDA preflight demanded 5.0 GB of free VRAM for every model, sized for
full large-v3. The default model is large-v3-turbo, which the pipeline
loads in fp16: about 1.6 GB of weights, not the 3.2 GiB fp32 figure in the
old comment. So a 6 GB card with nothing else resident reported 5.0 GB
free and was sent to CPU every time, although CUDA ran the same audio in
37 s against minutes on CPU.
The budget is now fp16 weights + 1.5 GB workspace (batch 16) + 0.5 GB
headroom, per model and capped at the old 5.0 GB. That is 3.6 GB for turbo
and 5.0 GB for full large-v3 and any unrecognised model. Both CPU-fallback
warnings name the model and the OMNIVOICE_ASR_VRAM_PREFLIGHT=0 opt-out.
The engine doc lists the budgets.
Fixes#2041
The MCP tools gave up on a backend POST after OMNIVOICE_MCP_TIMEOUT_S,
default 120 s, while the backend's own budgets run longer: 300 s for ASR,
and 300 s or more for generation. A transcription the backend would have
finished came back as an empty client-side timeout, and the abandoned
job kept holding the worker. The docstring said the timeout followed
OMNIVOICE_GENERATE_TIMEOUT_S, but it never read it.
Unset, each tool now waits for the backend's budget plus 30 s, never less
than 120 s. transcribe follows OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S, and
generate_speech follows the larger of the GPU and CPU generation budgets,
scaled by text length like the backend. An explicit
OMNIVOICE_MCP_TIMEOUT_S still wins. docs/mcp.md says so.
Fixes#2040
PyTorchWhisperBackend.transcribe read uploads with soundfile, and
libsndfile cannot open MP4/M4A (AAC). /transcribe and the MCP tool both
accept .m4a, so every such upload failed with "Format not recognised"
before ASR ran. It now uses _load_audio_16k_mono_f32, the loader the
sherpa backend already uses: soundfile first, then the validated ffmpeg
path, returning the 16 kHz mono float32 Whisper expects.
Fixes#2039
The drain thread took its buffer from spawn but read self._proc when it
started, so a drain that started late could read a replacement process's
stderr into the old buffer. Spawn now passes the process and its buffer
together. Tests cover a late drain and the error-frame branch of the ready
diagnostics.
test_blocked_reconnect_persistence_does_not_stall_another_worker failed on
a Windows runner (#2020's Smoke job) when its final wait for the Control
stream to end expired at 2s. That wait only proves the stream ends: after
both releases, the real disconnect path persists all 64 queued tasks, and
a slow runner took longer. The assertions that matter, another worker's
heartbeat and prewarm staying under 0.2s while reconciliation is blocked,
are untouched.
The three waits that only bound a Control stream's exit now share
_CONTROL_EXIT_TIMEOUT_S = 10s. A stream that never ends still fails.
Palash Debnath