feat(diagnostics): say why a GPU host fell back to CPU (#1274) (#1425)

The About page reported 'Compute device: cpu / GPU active: no / VRAM 0.00 GB' on a machine with a working GPU. Every line was true and none was usable — it is also exactly what a machine with no GPU at all reports, so the report could not distinguish a driver that isn't loaded from a container that cannot open the device from a ROCm older than the card.

The probe already knew all of it; torch.cuda.is_available() returning False simply produced no note. Each cause now reads differently: a missing device node names the --device flags, a permissions failure names --group-add and how to find the host's real render/video GIDs (copied numbers are the most common way this ends up on CPU in Docker), a card newer than the shipped ROCm points at rocminfo, an HSA_OVERRIDE_GFX_VERSION that is doing more harm than good is named first because it is both likelier and cheaper to test, and an unreachable NVIDIA driver gets its own advice.

The probe never diagnoses from a measurement it did not complete: when torch.cuda.is_available() itself raises, the exception is reported and no device findings are asserted beside it. Metadata access that raises is contained too — this runs on the path whose whole job is to explain a failure, so it cannot become one.
This commit is contained in:
Palash Debnath
2026-08-08 15:23:38 +05:30
committed by GitHub
parent 30c05fa038
commit 1eb59c6f18
4 changed files with 379 additions and 3 deletions
+1
View File
@@ -41,6 +41,7 @@ The bundled TTS model package (`pyproject.toml`) is versioned independently.
### Fixed
- A generation abandoned while stuck on an internal lock now says so, instead of blaming your hardware and suggesting shorter text. Nothing had been computed, so none of that advice applied. (#1416, #1419)
- A machine with a GPU that ends up on CPU now says why — a missing device node, a permissions problem, a card newer than the installed ROCm, an `HSA_OVERRIDE_GFX_VERSION` that is doing more harm than good, or an NVIDIA driver the container can't reach each read differently. Before, all of them looked identical to having no GPU at all. (#1274, #1228)
- A slow machine is no longer told its IndexTTS-2 install isn't there. The check that confirms an engine's virtualenv gave up after 10 seconds and counted that as a broken install, so a cold first run 500'd; it now waits longer and treats slow as unproven, not broken. (#1414) — thanks @OracleNightmare!
- A broken Python environment now says so, instead of blaming the app's own install. A missing or mismatched torch/transformers surfaced as "omnivoice not importable" and sent people reinstalling the wrong thing. (#1415)
- A model that fails to load at startup no longer leaves the app looking healthy while producing nothing — the failure and its remedy now show up in the model status. (#1415)
+119
View File
@@ -174,6 +174,110 @@ def gfx_for_hsa_override(value: str) -> str | None:
return f"gfx{int(major)}{minor}{step}"
#: The ROCm kernel driver interface. Its absence, or its presence without
#: permission, are the two commonest reasons a ROCm host silently runs on CPU.
_KFD_DEVICE = "/dev/kfd"
def why_no_gpu(torch) -> tuple[str, ...]:
"""Why ``torch.cuda.is_available()`` said no, as user-facing advisories.
This branch used to produce **nothing** (#1274/#1228). A host with a GPU
the app could not use reported "Compute device: cpu / GPU active: no" and
stopped there true, useless, and indistinguishable from a machine that
has no GPU at all. Two rounds of back-and-forth per report followed, and
the reporter still ended up guessing (numeric ``--group-add`` values
copied from another host, an ``HSA_OVERRIDE_GFX_VERSION`` that may or may
not have been needed).
The distinctions worth making are cheap, and the probe already knows them:
* the wheel has no GPU support compiled in at all no amount of
device-passing or env vars will change that;
* it is a ROCm wheel and ``/dev/kfd`` is absent in a container that is a
missing ``--device`` flag, not a driver problem;
* ``/dev/kfd`` is there but this process cannot open it a group
membership problem, which is the one that bites hardest in Docker
because the ``render``/``video`` GIDs differ between hosts and the
numbers are usually copied from somewhere else;
* everything is present and the runtime still enumerated nothing the
GPU is likely newer than this build's ROCm.
Never raises, and returns ``()`` rather than guessing when it cannot tell.
"""
# Metadata access itself can raise: `torch.version` is a module attribute
# on a real torch, but a partially-initialised or shimmed torch-like object
# can expose it as a property that throws. This function's contract is that
# it never raises — it is called from the diagnostics path, where an
# exception would take out the very report meant to explain the problem
# (CodeRabbit, #1425).
try:
version = getattr(torch, "version", None)
hip = getattr(version, "hip", None)
cuda = getattr(version, "cuda", None)
except Exception: # noqa: BLE001 - never raise from a diagnostic
return ()
if not hip and not cuda:
# A build with no GPU support compiled in. Deliberately silent: this
# is also every macOS wheel (MPS is probed separately, below) and
# every CPU Docker image, so a note here would fire on hosts that are
# working exactly as intended. The situations worth explaining are the
# ones where the build clearly meant to use a GPU and could not.
return ()
if hip:
# /dev/kfd only exists on Linux; on any other platform its absence
# says nothing, so don't invent a reason.
if sys.platform.startswith("linux"):
if not os.path.exists(_KFD_DEVICE):
return (
f"ROCm {hip} is installed but {_KFD_DEVICE} is not "
"present — the amdgpu kernel driver isn't loaded, or (in "
"Docker) the container was started without "
"--device /dev/kfd --device /dev/dri",
)
if not os.access(_KFD_DEVICE, os.R_OK | os.W_OK):
return (
f"ROCm {hip} is installed and {_KFD_DEVICE} exists, but "
"this process cannot open it — add the groups that own "
"it (`ls -l /dev/kfd /dev/dri/render*`; in Docker pass "
"--group-add with THAT host's render/video GIDs, which "
"differ between machines)",
)
override = (os.environ.get("HSA_OVERRIDE_GFX_VERSION") or "").strip()
if override:
# Checked BEFORE blaming the ROCm version, because it is the more
# likely cause and the cheaper thing to test. An override remaps
# the GPU onto a different architecture, and pointing a natively
# supported card at one the runtime cannot match to the physical
# agent can leave HSA with no usable agents at all — which is not
# "a kernel failed" but "there is no device", exactly what the
# #1274 reporter saw. Their card (gfx1151) is natively supported
# by the ROCm this image ships, so the override they set is very
# likely what hid it.
return (
f"ROCm {hip} is installed and the device nodes are reachable, "
f"but no GPU was enumerated while HSA_OVERRIDE_GFX_VERSION="
f"{override} is set. Try removing that override first — this "
"ROCm supports most current cards natively, and remapping one "
"it already supports can leave the runtime with no usable "
"device. VoiceStudio sets the override itself when a card "
"genuinely needs it",
)
return (
f"ROCm {hip} is installed and the device nodes are reachable, but "
"no GPU was enumerated — most often a card newer than this "
"build's ROCm. Check `rocminfo` on the host",
)
return (
f"this is a CUDA {cuda} build but no CUDA device was found — the "
"NVIDIA driver is missing or too old, or (in Docker) the container "
"was started without --gpus all",
)
def arch_unsupported(torch) -> tuple[str, tuple[str, ...]] | None:
"""``(device_arch, build_archs)`` when device 0's architecture is absent
from this torch build's compiled arch list — i.e. kernels cannot launch
@@ -296,9 +400,11 @@ def _probe() -> HostCaps:
# ── CUDA / ROCm (both present through torch.cuda) ────────────────────
cuda_ok = False
cuda_probe_failed = False
try:
cuda_ok = bool(torch.cuda.is_available())
except Exception as exc: # broken CUDA init (forked process / driver crash)
cuda_probe_failed = True
notes.append(f"CUDA init raised: {type(exc).__name__}")
if cuda_ok:
@@ -335,6 +441,19 @@ def _probe() -> HostCaps:
f"build's archs ({', '.join(archs)}) — {KERNEL_RISK_MARKER}"
)
elif not cuda_probe_failed:
# A GPU-capable build that found nothing must say why (#1274/#1228).
# Silence here is what made "Compute device: cpu" indistinguishable
# from a machine with no GPU at all.
#
# Only when the probe actually completed, though. If
# `torch.cuda.is_available()` RAISED we know nothing about the host's
# devices, and `why_no_gpu()` would report its findings as fact —
# "no CUDA device was found" beside "CUDA init raised", which reads as
# a diagnosis when it is an unfinished probe. The exception note above
# is the whole truth in that case (CodeRabbit, #1425).
notes.extend(why_no_gpu(torch))
# ── Intel XPU via IPEX ───────────────────────────────────────────────
try:
import intel_extension_for_pytorch # noqa: F401
+15 -3
View File
@@ -119,6 +119,15 @@ using it: **Settings → System** shows the device VoiceStudio actually resolved
If it reads `cpu` while the command above prints `True`, the backend log line
starting `Falling back to CPU:` names the architecture mismatch it hit.
If the command prints `False`, **Settings → System** now says why, and the
three answers need different fixes:
| What it says | What to do |
|---|---|
| `/dev/kfd is not present` | The container was started without `--device /dev/kfd --device /dev/dri`, or the host's `amdgpu` driver isn't loaded. |
| `this process cannot open it` | A group problem. Run `ls -l /dev/kfd /dev/dri/render*` **on the host**, and pass those GIDs with `--group-add`. The numbers differ between machines — a `--group-add 39` copied from someone else's command grants nothing. |
| `no GPU was enumerated` | The device nodes are fine and the runtime still found nothing — usually a card newer than the image's ROCm. Check `rocminfo` on the host, and see the `HSA_OVERRIDE_GFX_VERSION` note above. |
## Docker Compose (recommended)
```bash
@@ -210,7 +219,10 @@ Two paths are worth persisting across container restarts:
- **GPU not detected (AMD):** make sure you pulled the `:rocm` tag (the default
image is CUDA-only) and passed `--device /dev/kfd --device /dev/dri`. Check
the container sees the card with
`docker exec omnivoice rocminfo | grep -i gfx`; on RDNA3 consumer cards try
`-e HSA_OVERRIDE_GFX_VERSION=11.0.0` — see
[Pull and run (AMD GPU / ROCm)](#pull-and-run-amd-gpu--rocm) above.
`docker exec omnivoice rocminfo | grep -i gfx`. On consumer cards, run
**without** any `HSA_OVERRIDE_GFX_VERSION` first — the backend sets it
itself when your card needs it, and overriding a natively-supported GPU
only forces it onto foreign kernels. See
[Pull and run (AMD GPU / ROCm)](#pull-and-run-amd-gpu--rocm) above for when
to set one by hand.
- More entries: [docs/install/troubleshooting.md](troubleshooting.md).
+244
View File
@@ -0,0 +1,244 @@
"""A GPU host that runs on CPU must say why (#1274, #1228).
The reporter's About page said, in full:
Compute device: cpu
GPU active: no
VRAM (allocated): 0.00 GB
on a Strix Halo box running the ROCm image with ``--device /dev/kfd``,
``--group-add 39 --group-add 105`` and ``HSA_OVERRIDE_GFX_VERSION=11.0.0``.
Every one of those lines is true and none of them is usable. It is also
exactly what a machine with no GPU at all reports, so the report could not
distinguish "the driver isn't loaded" from "the container can't open the
device" from "this ROCm is older than this GPU" — and the numeric group IDs
in that command are copied from some other host, which is the single most
common way this ends up on CPU in Docker.
The probe already knew all of it. ``torch.cuda.is_available()`` returning
False simply produced no note at all, so nothing reached the user.
"""
from __future__ import annotations
import sys
from types import SimpleNamespace
import pytest
def _torch(*, hip=None, cuda=None):
return SimpleNamespace(version=SimpleNamespace(hip=hip, cuda=cuda))
def why_no_gpu(torch):
"""Resolved per call, not bound at import.
A module-level `from core.device_caps import why_no_gpu` captures whatever
object existed at collection time. Tests here patch `sys.modules["torch"]`
and other suites purge `sys.modules`, so the imported name can end up
referring to a module object nothing else in the process is using the
assertions then pass against a stale copy (CodeRabbit, #1425).
"""
import importlib
return importlib.import_module("core.device_caps").why_no_gpu(torch)
def _joined(torch) -> str:
return " ".join(why_no_gpu(torch)).lower()
# ── the wheel itself has no GPU support ────────────────────────────────────
def test_a_cpu_only_wheel_says_nothing():
"""Silence is right here, not an oversight: a wheel with no GPU support is
also every macOS build (MPS is probed separately) and every CPU Docker
image. A note would fire on hosts working exactly as intended, and the
baseline `notes == ()` contract in test_device_caps.py pins that."""
assert why_no_gpu(_torch()) == ()
# ── ROCm ───────────────────────────────────────────────────────────────────
@pytest.fixture
def linux(monkeypatch):
monkeypatch.setattr(sys, "platform", "linux")
def test_rocm_without_the_kernel_interface_names_the_docker_flags(linux, monkeypatch):
monkeypatch.setattr("core.device_caps.os.path.exists", lambda p: False)
msg = _joined(_torch(hip="6.4.0"))
assert "/dev/kfd" in msg
assert "--device /dev/kfd" in msg
assert "amdgpu" in msg
def test_rocm_that_cannot_open_the_device_names_the_group_trap(linux, monkeypatch):
"""The reporter's most likely case, and the one a generic message cannot
help with: the render/video GIDs differ per host, so a copied
`--group-add 39 --group-add 105` silently grants nothing."""
monkeypatch.setattr("core.device_caps.os.path.exists", lambda p: True)
monkeypatch.setattr("core.device_caps.os.access", lambda p, m: False)
msg = _joined(_torch(hip="6.4.0"))
assert "cannot open it" in msg
assert "--group-add" in msg
assert "differ between machines" in msg
# It must tell them how to find the right numbers, not just that theirs
# might be wrong.
assert "ls -l /dev/kfd" in msg
def test_rocm_with_everything_reachable_points_at_the_rocm_version(linux, monkeypatch):
monkeypatch.delenv("HSA_OVERRIDE_GFX_VERSION", raising=False)
monkeypatch.setattr("core.device_caps.os.path.exists", lambda p: True)
monkeypatch.setattr("core.device_caps.os.access", lambda p, m: True)
msg = _joined(_torch(hip="6.4.0"))
assert "no gpu was enumerated" in msg
assert "newer than this build" in msg
assert "rocminfo" in msg
def test_an_hsa_override_is_the_first_suspect(linux, monkeypatch):
"""The #1274 reporter set HSA_OVERRIDE_GFX_VERSION=11.0.0 on a gfx1151
card that the shipped ROCm 7.2.4 supports natively. Remapping a card the
runtime already handles can leave it with no usable agent at all which
reads as "no GPU", not as a kernel failure. It is both the likelier cause
and the cheaper one to test, so it is named before the ROCm version."""
monkeypatch.setenv("HSA_OVERRIDE_GFX_VERSION", "11.0.0")
monkeypatch.setattr("core.device_caps.os.path.exists", lambda p: True)
monkeypatch.setattr("core.device_caps.os.access", lambda p, m: True)
msg = _joined(_torch(hip="7.2.4"))
assert "hsa_override_gfx_version=11.0.0" in msg
assert "removing that override first" in msg
# Must not send them chasing the ROCm version instead.
assert "newer than this build" not in msg
def test_no_override_still_points_at_the_rocm_version(linux, monkeypatch):
monkeypatch.delenv("HSA_OVERRIDE_GFX_VERSION", raising=False)
monkeypatch.setattr("core.device_caps.os.path.exists", lambda p: True)
monkeypatch.setattr("core.device_caps.os.access", lambda p, m: True)
msg = _joined(_torch(hip="7.2.4"))
assert "newer than this build" in msg
assert "hsa_override" not in msg
def test_rocm_off_linux_does_not_invent_a_device_node_reason(monkeypatch):
"""/dev/kfd is a Linux path; its absence anywhere else says nothing, and a
confident wrong reason is worse than a vague right one."""
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.delenv("HSA_OVERRIDE_GFX_VERSION", raising=False)
msg = _joined(_torch(hip="6.4.0"))
assert "/dev/kfd" not in msg
assert "no gpu was enumerated" in msg
# ── CUDA ───────────────────────────────────────────────────────────────────
def test_cuda_without_a_device_names_its_own_causes():
msg = _joined(_torch(cuda="12.8"))
assert "cuda 12.8" in msg
assert "driver" in msg
assert "--gpus all" in msg
def test_rocm_wins_over_cuda_when_both_are_set():
"""A ROCm wheel reports a `torch.version.cuda` too; the HIP branch is the
correct reading and its advice is completely different."""
msg = _joined(_torch(hip="6.4.0", cuda="12.8"))
assert "rocm 6.4.0" in msg
assert "--gpus all" not in msg
# ── contract ───────────────────────────────────────────────────────────────
def test_it_never_raises_on_odd_torch_objects():
"""This runs inside a probe whose whole contract is that it cannot raise."""
for weird in (SimpleNamespace(), object(), None):
assert isinstance(why_no_gpu(weird), tuple)
def test_metadata_access_that_raises_is_not_an_exception_either():
"""`torch.version` is a plain module attribute on a real torch, but a
partially-initialised or shimmed torch-like object can expose it as a
property that throws and `getattr(torch, "version", None)` does not
swallow that, the default only covers a MISSING attribute.
This runs on the diagnostics path, so raising here takes out the very
report meant to explain why the GPU is unavailable (CodeRabbit, #1425).
"""
class _Hostile:
@property
def version(self):
raise RuntimeError("torch is half-initialised")
assert why_no_gpu(_Hostile()) == ()
class _HostileInner:
"""Raises one level down — `torch.version` resolves, `.hip` does not."""
class _V:
@property
def hip(self):
raise RuntimeError("hip probe exploded")
version = _V()
assert why_no_gpu(_HostileInner()) == ()
def test_the_probe_attaches_the_reason(monkeypatch, request):
"""End to end: the note has to reach `HostCaps.notes`, which is what the
About page and the diagnostics bundle read. Fail-before, this branch
produced nothing at all."""
import core.device_caps as dc
fake = SimpleNamespace(
version=SimpleNamespace(hip="6.4.0", cuda=None),
cuda=SimpleNamespace(is_available=lambda: False),
)
request.addfinalizer(dc.refresh)
# Registered BEFORE the fake torch goes in, so it runs even if an
# assertion below fails. `detect_host_caps()` memoises, and a cached probe
# built from this fake would be handed to every later test in the process
# — the trailing `dc.refresh()` alone only cleans up on the happy path
# (CodeRabbit, #1425).
monkeypatch.setitem(sys.modules, "torch", fake)
monkeypatch.setattr(sys, "platform", "linux")
monkeypatch.setattr(dc.os.path, "exists", lambda p: False)
dc.refresh()
caps = dc.detect_host_caps()
assert caps.family == "cpu"
assert any("/dev/kfd" in n for n in caps.notes), caps.notes
def test_a_failed_cuda_probe_does_not_also_claim_no_device_was_found(monkeypatch, request):
"""`torch.cuda.is_available()` raising tells us nothing about the host.
The probe reports `CUDA init raised: ` for that, which is true. Running
`why_no_gpu()` afterwards adds a second note asserting what it found
"no CUDA device was found" as though the probe had completed. Two notes,
one of them a diagnosis drawn from an unfinished measurement, is worse
than the one true note alone (CodeRabbit, #1425).
"""
import core.device_caps as dc
def _boom():
raise RuntimeError("CUDA driver initialisation failed")
fake = SimpleNamespace(
version=SimpleNamespace(hip=None, cuda="12.8"),
cuda=SimpleNamespace(is_available=_boom),
)
request.addfinalizer(dc.refresh)
monkeypatch.setitem(sys.modules, "torch", fake)
monkeypatch.setattr(sys, "platform", "linux")
dc.refresh()
caps = dc.detect_host_caps()
assert any("cuda init raised" in n.lower() for n in caps.notes), caps.notes
assert not any("no cuda device" in n.lower() for n in caps.notes), (
"the probe reported what it found after failing to look"
)