On a Tesla T4 the backend exited during the first /generate with no
traceback and no HTTP response, leaving the client with
RemoteDisconnected and every later call with ConnectionRefused. Three
separate defects combined, which is why none of the reporter's
workarounds helped.
1. torch.compile(mode="reduce-overhead") captures CUDA graphs. T4
(sm_75) passed the existing arch gate, so capture was attempted and
aborted the process from inside the native CUDA library — below the
interpreter, where neither the #278 eager-fallback wrapper nor any
except clause can see it. The compile mode is now resolved per GPU:
Ampere (sm_80) and newer keep the cudagraph mode, older cards drop to
the non-cudagraph "default" mode and keep their compiled Inductor
kernels. Fails open on any probe error, so no GPU that works today
loses the optimization. OMNIVOICE_FORCE_CUDAGRAPH=1 restores it.
2. should_torch_compile() never read TORCH_COMPILE_DISABLE. main.py sets
it on win32, build_engine_env injected it into subprocesses, and
docs/install/windows.md tells users to export it — but the in-process
gate ignored it, so the reporter exported the documented variable and
still got "torch.compile applied". The gate now honours
TORCH_COMPILE_DISABLE / TORCHDYNAMO_DISABLE / TORCHINDUCTOR_DISABLE on
every platform, and an env opt-out on the parent propagates to engine
subprocesses. The settings DB path is logged alongside the toggle:
the reporter had three omnivoice.db files and edited one the backend
never opened.
3. Settings -> Performance -> "Disable torch.compile" was rendered
disabled outside Windows in both the Tauri and Electron UIs, so the
one control that would have stopped this was unreachable for the
affected Linux user. The toggle is now live on every platform, and
build_engine_env honours it everywhere rather than only on win32.
Also arms faulthandler before torch is imported, so a fatal native
signal writes the faulting thread's Python stack to backend_err.log
instead of the process vanishing silently. This does not prevent a
crash; it makes one diagnosable. OMNIVOICE_DISABLE_FAULTHANDLER=1 skips
it.
Tests fail before / pass after, verified by stashing the source and
running the new tests against unfixed code. The crash test kills a real
child interpreter with a real SIGSEGV and requires a named Python frame
in the output. test_torch_compile_path_gate's fixture now clears the
compile-disable env vars: main.py setdefaults them on win32, so on a
Windows runner they leaked into os.environ and decided those tests.
Not verified on real hardware — no Turing GPU available. The sm_80 floor
is inferred from the crash report and from docs/hardware-notes-tesla-t4.md,
which already flagged cudagraphs on T4 as attempted by default and never
evaluated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two review findings on the troubleshooting section.
The get_arch_list() check told users to run `import torch` in a section whose
documented symptom is `import torch` crashing natively, so it was useless for
exactly its intended readers. It now leads with that: if the import crashes,
that is the symptom, skip to the fix. The check keeps its remaining purpose of
ruling the missing-kernel theory out where torch does import.
"The cu128 wheels dropped Maxwell and Pascal" attributed the drop to the CUDA
variant when it tracks the torch release. CU128_ARCHS carries sm_61 from the
#1285 build, which predates 2.8; the pinned 2.8.0+cu128 build reports sm_70
first. Both are right for their own release, so the blanket claim was not.
Stated as the two measurements instead, and deliberately not extended to
2.9.x, which is unmeasured here.
Signed-off-by: Moep90 <volleyballlive@googlemail.com>
Signed-off-by: moep90 <volleyballlive@googlemail.com>
Audit leftover in the same file: the "Keeping the change" note explained that
the default pin cannot move because "a build that adds sm_120 can drop older
architectures". That still implies the pinned build lacks sm_120, sixty lines
below the corrected cause. The reason the default cannot move is simply that a
newer build can drop older GPUs, which cu128 already did for Maxwell and
Pascal.
Deliberately not restating the resulting floor. docs/competitive-analysis.md
says Turing sm_75, but the pinned 2.8.0+cu128 build reports sm_70 first, so the
claim is unverified and left out rather than copied.
Signed-off-by: Moep90 <volleyballlive@googlemail.com>
Signed-off-by: moep90 <volleyballlive@googlemail.com>
Review catch: the code comments were corrected while
docs/install/troubleshooting.md still gave users the false diagnosis in two
places, which is the docs-sync rule exactly. The user-facing wording was the
stronger of the two:
5b: "the wheel does not contain code for the GPU" and "no setting works
around it"
1052: "This is a property of the pinned build, not of your driver or your
install"
The second is actively misleading. Since torch 2.8.0+cu128 does list sm_120 in
get_arch_list(), the driver, the platform and the native init path are exactly
where the cause plausibly is, and the doc steered readers away from them.
Both sections now state that the cause is not established, that the pinned
build does contain Blackwell code, and that moving the trio to 2.9.x is the
known workaround for the users who hit it. 5b also gains a get_arch_list()
command so a reader can check their own build instead of trusting a blanket
claim. The fix steps are unchanged.
Also softens the CU128_ARCHS citation in the three comments. That list is
fixture data fed to a mocked torch, captured verbatim from the #1285 report,
so it corroborates the arch list but does not assert what the installed wheel
contains. Saying it "already asserts" overstated it.
Signed-off-by: Moep90 <volleyballlive@googlemail.com>
Signed-off-by: moep90 <volleyballlive@googlemail.com>
XNU stops signalling a process as soon as it starts exiting but posts
NOTE_EXIT later in the same exit. A KILL that lands in that window gets
EPERM while the exit probe still reads "alive", so stopping a child that
TERM had just ended could fail with "Operation not permitted". This
flaked the macOS run of contained_exit_probe_preserves_a_live_child.
The EPERM branch now re-probes for up to 250 ms before treating the error
as a live, unsignalable root. Live roots, reaped roots and probe failures
are still errors.
The engine matrix (five columns, three-line rows, every chip on every row)
becomes a shadcn table with three columns — Engine · Runs on · Status — and
one primary action per row (Use / Install). Everything else lives in a
detail panel for the selected row: GPU compatibility chips, isolation,
hints and reasons, health and self-test probes, one-click install
progress, setup snippet, disk usage, docs, license, the curated-model
picker, and now the engine's downloadable WEIGHTS.
Weights belong to their engine: every models.yaml entry names the backend
ids that load it (`engines:`), the detail panel lists and installs them
(EngineWeights, on the model store's install/cancel/remove flow via the
extracted useModelDownloads hook), and the sherpa-onnx engine shows its
dictation-model picker there. The page's "Downloaded weights" list and
recommendation card are gone; only weights no engine owns (speaker
diarisation) remain in a small "Other weights" list. A backend test pins
the mapping: every entry has an `engines` list and every id is a real
backend.
- useEngineInventory: the matrix's state machines extracted verbatim
(shared/local fetch, residency, health/self-test cooldowns, install
poller with overlap guard + epoch, disk-usage generations, license).
- Row status phrases: GPU active / CPU fallback / CPU / Available /
Needs setup / Installing… / failed; routing "unavailable" never reads
Ready. Group captions keep "Ready to use" / "Add more engines".
- Engine titles read "Engines" (each locale's own word); backend
"Model Catalogue → Engines/Models" messages and docs updated to the
new structure.
- Dead matrix CSS (phone-tier grid) removed; scopeReco and RecoBanner gone.
- SetupSummary: an installed engine whose routing is "unavailable" reads
Needs setup, not Ready (select is refused for it too); a failed /engines
or /dictation/models fetch renders as an error with Retry instead of
posing as "Off" / "Needs setup".
- Bulk installs (summary, model store, recommendation card) wait for every
request to settle before re-enabling, and report which repos failed —
one early rejection can no longer re-arm the button mid-flight.
- The weights list stays mounted across family switches (hidden under LLM)
so download progress and Retry/Dismiss state survive navigation.
- Settings search: Hugging Face mirror terms route to Network; the legacy
"models" tab id resolves to Storage. "Manage models" opens the TTS tab.
- Locales: uk "Рушії", zh-TW "引擎", vi "Engine" for the Engines heading.
- Docs name the family tab wherever the instruction depends on it.
The Model Catalogue put the same decision on two axes: an Engines pane with
TTS/ASR/LLM tabs and a Models pane with TTS/ASR/Dictation/Diarisation
sections, dictation shown in both, plus storage stats, the HF token and the
voice-preview toggle parked on the model list. Settings → Voice still carried
Engines and Models entries that only pointed back here.
Now the page reads top-down: a SetupSummary (speech, transcription,
dictation, language model — engine, device, one status word, Change), the
engine list for one family, and that family's downloadable weights under it
(TTS under TTS; offline ASR, streaming dictation and diarisation under ASR;
nothing for LLM, whose engines bring their own). One storage line points at
Settings → Storage.
- ModelStoreTab takes a `family` and scopes sections and the recommendation
preset to it (scopeReco); stats strip, HF-token toolbar and previews
panel removed from it.
- Settings: Engines/Models categories and CataloguePointer removed; models
directory → Storage, HF mirror → Network (both restart-flagged), voice
previews → Storage. "Manage models" in disk usage opens the catalogue.
- Store: openCatalogue takes a family (pane key tolerated, ignored);
pendingCatalogueTab gone.
- Engine matrix title is now the locale's plain "Engines".
- i18n: catalogue.* summary keys in all 21 locales; pane/pointer keys dropped.
- Docs: "Model Catalogue → Engines" is "Model Catalogue"; "→ Models" is
"→ Downloaded weights".
Lands #1998 by @yangfan-yf-yf. Correct finding: the PowerShell example I added
in #1993 generated the administrator key with `python -c`, and the whole point
of the Docker path is that the host does not need Python. On a Windows host
without it, the very first line of the setup fails.
One thing on top. The key is also accepted as an `?api_key=` query parameter
(core/auth.py), and raw Base64 carries `+`, `/` and `=`. A `+` in a query
string decodes to a space, so a user who pasted such a key into a URL would get
a silent mismatch with nothing to explain it. The Bash line next to it uses
`secrets.token_urlsafe` and never had this shape, so the two now agree:
trim the padding, map `+` to `-` and `/` to `_`.
Verified in Windows PowerShell 5.1 (5.1.26100): the block parses and runs, and
the key is 43 URL-safe characters — the same shape `secrets.token_urlsafe(32)`
produces. validate-install-docs.py and both docker/changelog test files pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ypcgSsh5j2PEonSJiAU1S
The code half of #1931 landed already: `torchaudio.set_audio_backend()` is
guarded, so the torch 2.9.x upgrade a Blackwell card needs no longer trades one
`ml_imports` crash for another. Two things were still missing.
The changelog said the upgrade was documented. It was not — nothing in docs/
mentions sm_120, Blackwell, or the 50-series at all, so a user hitting a native
access violation inside `import torch` had the issue thread and nothing else.
troubleshooting.md now carries it: why the pinned torch 2.8.0 cannot work
(no sm_120 kernels in the wheel — not a setting, not a workaround), the trio
that has to move together, the verification command that proves the kernels
arrived, and the fact that the change is to the repo's own pins so a later pull
will undo it.
The part most likely to be missed is that there are TWO pin lists.
`constraint-dependencies` governs `uv sync`/`uv lock`/`uv run`;
deploy/torch-constraints.txt governs the `uv pip install` paths, which ignore
project-level uv settings. Editing one leaves the other behind, which is what
`RuntimeError: operator torchvision::nms does not exist` looks like from the
outside. Both are named.
The second gap: nothing protected the guard. CI runs the pinned torch 2.8.0,
where `set_audio_backend` still exists, so deleting the `hasattr` as a
"simplify this no-op" cleanup would pass every test in the suite and restore a
hard startup crash for every RTX 50-series user. tests/ now walks the backend
AST and fails on any reach for a torchaudio API that 2.9 removed unless
something proves it is there — a `hasattr`/`getattr` check or a `try`. Fails
with the guard removed, passes with it.
Not addressed here, because it is already fixed: the reporter's third
observation, that launching through the desktop shell hung inside `import
torch`'s native init, is the OpenBLAS/stdin-pipe deadlock closed under #1952.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ypcgSsh5j2PEonSJiAU1S
@yangfan-yf-yf pushed two more commits to #1987 after the first pass landed.
Two things in them were worth taking:
- an explicit `compose pull` step, so the platform override is proven before
`up -d` rather than discovered when the pull inside it fails; and
- a PowerShell form. An ARM64 Windows host cannot use `export`, and the
surrounding page only ever shows Bash — so the guidance did not actually
reach the users most likely to need it.
Not taken: the same commits also moved `--platform linux/amd64` into the
default `docker pull` / `docker run` quick start. That is a no-op for the
amd64 majority and contradicts the Architecture section directly above, which
introduces the flag as the conditional ARM64 step. The canonical command stays
the one almost everyone should run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ypcgSsh5j2PEonSJiAU1S
Lands #1987 by @yangfan-yf-yf, which closes#1921.
The published images are linux/amd64 only, and the quick start reached image
resolution before saying so — an ARM64 user met "no matching manifest for
linux/arm64/v8" with no explanation. Verified against docker.yml, which says
so in its own comment: "only building linux/amd64".
One gap in the original: the platform override was documented for docker pull
and docker run, but Compose has no per-command --platform flag, so the
recommended Compose command still resolved the missing ARM64 manifest and
failed exactly as before. DOCKER_DEFAULT_PLATFORM covers it, with the same
caveat the rest of the section makes — emulation, not native support, and only
the CPU profile makes sense under it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ypcgSsh5j2PEonSJiAU1S
Palash Debnath