Shivendra-CoherentandClaude Opus 5
64f74e0dd2
On a Tesla T4 the backend exited during the first /generate with no traceback and no HTTP response, leaving the client with RemoteDisconnected and every later call with ConnectionRefused. Three separate defects combined, which is why none of the reporter's workarounds helped. 1. torch.compile(mode="reduce-overhead") captures CUDA graphs. T4 (sm_75) passed the existing arch gate, so capture was attempted and aborted the process from inside the native CUDA library — below the interpreter, where neither the #278 eager-fallback wrapper nor any except clause can see it. The compile mode is now resolved per GPU: Ampere (sm_80) and newer keep the cudagraph mode, older cards drop to the non-cudagraph "default" mode and keep their compiled Inductor kernels. Fails open on any probe error, so no GPU that works today loses the optimization. OMNIVOICE_FORCE_CUDAGRAPH=1 restores it. 2. should_torch_compile() never read TORCH_COMPILE_DISABLE. main.py sets it on win32, build_engine_env injected it into subprocesses, and docs/install/windows.md tells users to export it — but the in-process gate ignored it, so the reporter exported the documented variable and still got "torch.compile applied". The gate now honours TORCH_COMPILE_DISABLE / TORCHDYNAMO_DISABLE / TORCHINDUCTOR_DISABLE on every platform, and an env opt-out on the parent propagates to engine subprocesses. The settings DB path is logged alongside the toggle: the reporter had three omnivoice.db files and edited one the backend never opened. 3. Settings -> Performance -> "Disable torch.compile" was rendered disabled outside Windows in both the Tauri and Electron UIs, so the one control that would have stopped this was unreachable for the affected Linux user. The toggle is now live on every platform, and build_engine_env honours it everywhere rather than only on win32. Also arms faulthandler before torch is imported, so a fatal native signal writes the faulting thread's Python stack to backend_err.log instead of the process vanishing silently. This does not prevent a crash; it makes one diagnosable. OMNIVOICE_DISABLE_FAULTHANDLER=1 skips it. Tests fail before / pass after, verified by stashing the source and running the new tests against unfixed code. The crash test kills a real child interpreter with a real SIGSEGV and requires a named Python frame in the output. test_torch_compile_path_gate's fixture now clears the compile-disable env vars: main.py setdefaults them on win32, so on a Windows runner they leaked into os.environ and decided those tests. Not verified on real hardware — no Turing GPU available. The sm_80 floor is inferred from the crash report and from docs/hardware-notes-tesla-t4.md, which already flagged cudagraphs on T4 as attempted by default and never evaluated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.4 KiB
Electron compute and performance settings
Settings > Compute device exposes the existing device override, the torch.compile workaround, generation time budgets, and hardware readouts.
Device choices come from the backend's detected families plus Auto. The chosen preference and currently active family are displayed separately. Environment-pinned choices are disabled, an ignored unavailable override is explained, and a changed preference shows its actual restart requirement. Failed saves keep the last confirmed state. Nothing automatically restarts the backend or changes the active model.
The torch.compile workaround matches Tauri: since #2135 it is selectable on every platform, because the compile failures it works around are not Windows-only. Generation budgets preserve separate GPU and CPU limits, validate the existing positive/21600-second range, and keep edits during refetches. An externally overridden budget reports that fact instead of implying the saved value will take effect after restart. Hardware RAM/VRAM readouts poll only while this view is mounted.
During synthesis, the fixed-width primary action polls the existing model-status contract and names the active runtime phase: starting the AI runtime, loading weights, warming speech recognition, optimizing the model, generating, or receiving audio. Model-load percentage and elapsed time share the reserved status line, and the progress track switches from model loading to streamed audio delivery without moving the controls.
The backend publishes explicit model lifecycle transitions to the renderer event stream. Engine Ready refreshes immediately from those events and keeps one-second polling only while work is active; idle model, worker, batch, performance-profile and diarisation checks back off to bounded 15–30 second recovery intervals.
electron/tests/performance-settings-smoke.mjs checks saved/active separation, failed saves, environment pinning, unavailable devices, the platform guard, budget validation and external overrides with mocked contracts. Live read-only checks verified the compute, compile and hardware response schemas. Tests did not change the user's device, optimization or timeout preferences. Native hardware behavior on macOS/Linux still requires platform verification.
Speed and quality presets
The engine sidebar and Performance settings expose Fast, Balanced, Quality and Max. A global choice resets family overrides; a family choice overrides the global preference. Changes are blocked while foreground or batch work is active. Choosing a preset never downloads weights or activates cloud providers. A ready network translator remains authoritative when explicitly selected; if that provider becomes unavailable, profile reconciliation recovers to the tier-appropriate installed local translator instead of leaving translation unusable.
Currently connected controls are OmniVoice sampling (8/16/32/64 steps), Faster-Whisper decoding search (1/3/5/8), Sherpa transducer dictation search (greedy through 8-path modified beam search), local NLLB beam search (1/3/5/8), and the installed diarisation runtimes. Clone, Dubbing, Batch, Voice Conversion, Stories and Audiobook all resolve untouched TTS controls through this shared contract; an explicit Production override still wins. Dictation keeps the selected language model and rebuilds its warm recognizer after a tier change. When both diarisation choices are installed, Fast/Balanced select native audio.cpp Sortformer and Quality/Max select pyannote; with only one runtime, its family control stays unavailable rather than accepting a no-op preference. Max selects the strongest already-installed compatible Faster-Whisper, dictation and NLLB choices without downloading anything. LLM remains unavailable until the selected runtime exposes a meaningful comparable effort control. The API reports only implemented targets.
Performance Settings presents each family as a discrete Fast/Balanced/Quality/Max control and identifies the effective local model, runtime, and decoding effort beneath it. A choice is persisted before any optional renderer-side synchronization and receives explicit applied or failed feedback, so a cold engine catalogue cannot make the control appear inert. Families without an installed compatible target stay disabled, name the required engine, and link to its Models view; an engine with no comparable effort contract explains that limitation. The compact sidebar keeps the slider form of the same setting.
Settings > Models turns those tiers into one-click, target-aware model packs. Each pack previews its exact compatible models, installed size, remaining download and aggregate progress before starting the existing resumable installer. Fast installs the smallest local ASR and dictation set; Balanced selects the faster Whisper Turbo and Parakeet set; Quality and Max add Whisper large-v3 and local NLLB. The saved tier is reconciled after every successful model download, so newly available engines become active without another selection or restart. Diarisation stays explicit because native audio.cpp setup and gated pyannote access require separate consent; LLM stays explicit because it has no common local performance target.
Backend startup was checked live after the lifecycle changes: OmniVoice loaded successfully and the performance-profile endpoint responded. This does not establish the cause of historical native crashes or verify recovery from every stall.
Crash-isolated Faster-Whisper receives the same ASR decoding preset with each transcription request. The parent snapshots the selected beam/best-of values; the child validates them before loading the model. Existing callers without decoding options retain their original defaults, and changing a preset does not require restarting the child.
Reference transcription
Uploaded voice references use the selected ASR engine through the shared transcribe endpoint's reference mode. This skips word alignment, checks locally installed models before loading, and never enables LLM refinement. Dictation selection remains independent. Missing models leave the optional transcript editable and retryable; asynchronous results do not overwrite manual edits or a subsequently selected saved voice.
Reference mode fails closed if local installation cannot be verified, including unknown model selections and preflight errors. The loader rechecks the actual selected engine and every fallback immediately before loading, bypassing stale positive cache entries.