From c5c57508b38c08fce8e7e3e16a7d8dd32b17ffaa Mon Sep 17 00:00:00 2001 From: Palash Debnath Date: Wed, 1 Jul 2026 17:59:49 +0530 Subject: [PATCH] fix(device): fall back to CPU when the GPU arch is unsupported, not 500 every generate (#756) (#757) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(settings): contain + tighten the whole Settings surface (measure cap, container-query stacking, wrap the shared rows) Two systemic issues drove 'too spread out' + 'elements go out of view' across many Settings pages: 1. Spread — .settings-content capped at 1280px, so on wide windows every label-left/control-right row left a huge void. Introduce a --settings-measure token (720px, macOS-like) + --settings-rail, and cap the content to it, left-aligned under the nav. One token now controls the reading width. 2. Overflow + bad responsiveness — the row stack break was a *viewport* media query (560px), but the 168px nav rail means a 760px-viewport window only has ~530px of content, so rows went side-by-side in a cramped box. Make .settings-content a container (container-type: inline-size) and stack on the CONTENT width via @container, keeping the viewport @media as a fallback for the .st-row instances used outside Settings (Splash/FirstRun/Dub/SetupWizard). 3. The shared .perfpanel__row (button/badge row reused by 6+ panels: RemoteBackend, HFMirror, LLMEndpoint, Pronunciation, MCPBindings, …) was an inline-flex with no wrap and no max-width, so it ran off the right edge — add flex-wrap + max-width:100% + min-width:0. Plus two rigid-width fixes that escaped the row cap: ApiKeys input min-width:220→0, Appearance scale floor. Frontend builds clean; tokens, @container query, and the wrap all verified in the emitted CSS bundle. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(settings): center the settings block + tighten measure (kill the lopsided right void) The capped content was left-aligned, so on a wide window everything jammed to the left with a dead empty third on the right (screenshot). Center the whole settings block (nav rail + content) as a unit via max-width + margin-inline:auto, and drop the measure 720→660 so label→control rows read denser. The cap is computed from the tokens (rail + gap + measure + page padding) so the content track lands exactly at --settings-measure. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(device): fall back to CPU when the GPU arch is unsupported, instead of 500-ing every generate (#756) get_best_device() called check_device_compatibility() and, on an unsupported compute capability, only LOGGED a warning then still returned 'cuda' — so the model loaded on a GPU whose kernels can't launch and every generate 500'd with 'CUDA error: no kernel image is available for execution'. Both a too-old card (Pascal sm_61, GTX 10-series) and a too-new one (Blackwell sm_120 on pre-cu128 wheels) hit this. Now an unsupported arch falls back to CPU (works, just slower) with a clear warning; OMNIVOICE_FORCE_CUDA=1 overrides. Belt-and-suspenders: _oom_friendly_reraise classifies a raw 'no kernel image is available' as an unsupported-GPU error (switch to CPU / install matching torch) rather than the OOM/Flush message. Tests: get_best_device → cpu on incompatible, stays cuda on compatible, honors the force override; reraise gives the actionable GPU message, not OOM. Co-Authored-By: Claude Opus 4.8 (1M context) * test(device): patch detect_host_caps via string path so the #756 fallback test is full-suite robust The first version aliased the import + inserted backend on sys.path, which patched a module copy get_best_device's local 'from core.device_caps import detect_host_caps' didn't resolve in the full suite (passed alone, failed in CI). Use the string-form monkeypatch target; verified passing alongside the other device/model tests. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(changelog): fold #757 device-fallback entry into [0.3.8]; drop the merge's stale [Unreleased] dupe --------- Co-authored-by: mergetest Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 10 +++++ backend/api/routers/generation.py | 12 ++++++ backend/services/model_manager.py | 13 +++++++ tests/test_device_unsupported_fallback.py | 46 +++++++++++++++++++++++ tests/test_generation_audio_guard.py | 16 ++++++++ 5 files changed, 97 insertions(+) create mode 100644 tests/test_device_unsupported_fallback.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 55a0fbf8..431f2d89 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -177,6 +177,16 @@ across dub, generate, and design (a corrupt-binary failure no longer poses as Closes the whole class of GPU-job-hang reports (#851 — #850, #802, #755, #723, #721, and the 0.3.7 cohort, all tracked in #730). +- **An unsupported GPU now falls back to CPU instead of 500-ing every generate.** + When the installed PyTorch build has no kernels for your GPU's compute + capability — a too-old card (Pascal / GTX 10-series) or a too-new one + (Blackwell RTX 50-series on pre-cu128 wheels) — CUDA failed at launch with the + cryptic `CUDA error: no kernel image is available for execution`. The backend + now detects that up front and runs on CPU (slower, but it works), and any raw + occurrence is reported as "your GPU isn't supported — switch to CPU or install a + matching PyTorch," not a Flush-the-memory dead end. Force the GPU anyway with + `OMNIVOICE_FORCE_CUDA=1`. (#756) + - **The "TRANSLATION FAILED" banner now dismisses and clears itself.** The Dub translation-error banner used to be sticky — it survived a successful re-try and never went away. It now has a close (×), auto-clears on the next corrective diff --git a/backend/api/routers/generation.py b/backend/api/routers/generation.py index 46b7f1ec..81a2becb 100644 --- a/backend/api/routers/generation.py +++ b/backend/api/routers/generation.py @@ -214,6 +214,18 @@ def _oom_friendly_reraise(e): # sys.stdout/stderr to swallow EPIPE, but a C-level write inside the native # engine/torch can still raise one past that guard. Flush won't help — # relaunching the app re-parents the backend to a live shell. + # #756: the GPU's compute capability isn't in this PyTorch build's arch list, + # so CUDA can't launch kernels ("no kernel image is available for execution"). + # NOT OOM. get_best_device() now falls back to CPU up front, but classify the + # raw error too in case CUDA was forced (OMNIVOICE_FORCE_CUDA) or a sub-path + # still ran on the GPU — point at the real fix, not the Flush button. + if "no kernel image is available" in _low: + raise RuntimeError( + f"Your GPU isn't supported by the installed PyTorch build (CUDA can't " + f"launch kernels for its compute capability). Switch the compute device " + f"to CPU in Settings, or install a matching PyTorch (e.g. a cu128 build " + f"for newer GPUs). The Flush button won't help. Underlying error: {e}" + ) from e if isinstance(e, BrokenPipeError) or "broken pipe" in _low or "errno 32" in _low: raise RuntimeError( f"The backend lost its output pipe mid-generation — the desktop app " diff --git a/backend/services/model_manager.py b/backend/services/model_manager.py index bcee15dc..00aa86a3 100644 --- a/backend/services/model_manager.py +++ b/backend/services/model_manager.py @@ -370,6 +370,19 @@ def get_best_device(): compatible, warning = check_device_compatibility() if not compatible: logger.warning(warning) + # #756: the GPU's compute capability isn't in this torch build's arch + # list, so CUDA kernels can't launch ("no kernel image is available + # for execution") — every generate would 500. Too-old (Pascal sm_61) + # and too-new (Blackwell sm_120 on pre-cu128 wheels) both land here. + # Fall back to CPU so the app WORKS (slowly) instead of dead-ending; + # OMNIVOICE_FORCE_CUDA=1 overrides for users who installed a matching + # torch and know the arch_list probe is wrong for their setup. + if not _env_flag("OMNIVOICE_FORCE_CUDA"): + logger.warning( + "Falling back to CPU: this GPU is unsupported by the installed " + "PyTorch build (set OMNIVOICE_FORCE_CUDA=1 to force CUDA anyway)." + ) + return "cpu" return "cuda" # ── Intel Arc / discrete GPU via IPEX ──────────────────────────── diff --git a/tests/test_device_unsupported_fallback.py b/tests/test_device_unsupported_fallback.py new file mode 100644 index 00000000..f6cdbdb1 --- /dev/null +++ b/tests/test_device_unsupported_fallback.py @@ -0,0 +1,46 @@ +"""#756: a GPU whose compute capability isn't in the installed PyTorch build's +arch list can't launch CUDA kernels ("no kernel image is available for +execution"), so every generate 500s. get_best_device() must fall back to CPU so +the app still works (slowly) instead of dead-ending — unless the user explicitly +forces CUDA. These tests pin that fallback (and the override) without a GPU. +""" +from types import SimpleNamespace + +import pytest + +import services.model_manager as mm + + +@pytest.fixture +def cuda_host(monkeypatch): + # Pretend a CUDA GPU is present. Patch detect_host_caps via its string path so + # the lookup resolves the same module object get_best_device imports locally + # (`from core.device_caps import detect_host_caps`) — patching an aliased + # import can miss that in a full-suite run. + monkeypatch.setattr( + "core.device_caps.detect_host_caps", lambda: SimpleNamespace(family="cuda") + ) + monkeypatch.setattr(mm, "_lazy_torch", lambda: SimpleNamespace()) + monkeypatch.setattr(mm, "_configure_rocm_if_needed", lambda _torch: None) + monkeypatch.delenv("OMNIVOICE_FORCE_CUDA", raising=False) + + +def test_unsupported_gpu_falls_back_to_cpu(cuda_host, monkeypatch): + monkeypatch.setattr( + mm, "check_device_compatibility", + lambda: (False, "GTX 1080 Ti (sm_61) is not supported by this PyTorch build"), + ) + assert mm.get_best_device() == "cpu" + + +def test_supported_gpu_stays_on_cuda(cuda_host, monkeypatch): + monkeypatch.setattr(mm, "check_device_compatibility", lambda: (True, None)) + assert mm.get_best_device() == "cuda" + + +def test_force_cuda_overrides_the_fallback(cuda_host, monkeypatch): + monkeypatch.setattr( + mm, "check_device_compatibility", lambda: (False, "unsupported arch"), + ) + monkeypatch.setenv("OMNIVOICE_FORCE_CUDA", "1") + assert mm.get_best_device() == "cuda" diff --git a/tests/test_generation_audio_guard.py b/tests/test_generation_audio_guard.py index c45ee845..830eceb6 100644 --- a/tests/test_generation_audio_guard.py +++ b/tests/test_generation_audio_guard.py @@ -102,6 +102,22 @@ def test_broken_pipe_is_a_lost_pipe_not_oom(): assert "ran out of memory" not in msg +def test_no_kernel_image_is_an_unsupported_gpu_not_oom(): + # #756: a GPU whose compute capability isn't in the torch build's arch list + # (Pascal sm_61 on new wheels, Blackwell sm_120 on old wheels) raises "CUDA + # error: no kernel image is available for execution". That's NOT OOM and Flush + # won't help — point at CPU / a matching torch. + err = RuntimeError( + "CUDA error: no kernel image is available for execution on the device" + ) + with pytest.raises(RuntimeError) as ei: + _oom_friendly_reraise(err) + msg = str(ei.value) + assert "GPU isn't supported" in msg or "isn't supported by the installed" in msg + assert "CPU" in msg + assert "ran out of memory" not in msg + + def test_winerror_193_is_a_corrupt_binary_not_oom(): # #705: a corrupt / wrong-architecture native component (torch, ffmpeg, an # engine binary) fails on Windows with "[WinError 193] %1 is not a valid