b8e219e7d1
The wave of "Can't reach the local backend" reports — all on ~8 GB NVIDIA cards, all during generate bursts — is the backend *process* dying, not a transport blip. Root cause: the GPU pool was sized at 2.5 GB/job, so an 8 GB card (~7 GB free) got 2 workers. The interactive clone path co-loads WhisperX large-v3 ASR (~3 GB) alongside TTS (~1.6 GB), so two concurrent clone jobs is ~10 GB on an 8 GB card → a sticky CUDA "illegal memory access" that aborts the whole interpreter (uncatchable by the per-request OOM guard, which only re-raises a clean torch.cuda.OutOfMemoryError as HTTP 500). Budget 5 GB/job (the real TTS+ASR concurrent footprint) instead of 2.5 GB: ≤10 GB cards now serialize to a single GPU worker — no concurrent-kernel contention, so the crash can't happen — while 16/24 GB cards still parallelize. Overridable via OMNIVOICE_GPU_WORKERS. This *prevents* the crash; the auto-restart supervisor (#572) *recovers* from any other cause — defense in depth. Extracted `_workers_for_free_vram()` (pure) with tests pinning 8 GB → 1 worker, the larger-card ladder, the floor/cap, and a guard on the budget constant so a regression toward 2.5 GB can't silently re-enable the crash. Co-authored-by: mergetest <test@local> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
44 lines
1.8 KiB
Python
44 lines
1.8 KiB
Python
"""GPU worker-sizing policy (#567 crash prevention).
|
|
|
|
The "Can't reach the local backend" crash wave on 8 GB GPUs was an OOM/CUDA
|
|
fault from running 2 concurrent clone jobs (TTS + co-loaded WhisperX ASR) when
|
|
the pool was sized to >1 worker on a small card. These pin the sizing policy so
|
|
an 8 GB card serializes to a single worker (no contention → no crash) while
|
|
larger cards still parallelize, all without needing a GPU.
|
|
"""
|
|
from services.model_manager import (
|
|
_workers_for_free_vram,
|
|
_GPU_WORKER_CAP,
|
|
_GPU_VRAM_PER_JOB_GB,
|
|
)
|
|
|
|
|
|
def test_eight_gb_card_serializes_to_one_worker():
|
|
# An 8 GB card reports ~7 GB free when the pool is sized — must be 1 worker
|
|
# so two concurrent clone jobs can't blow past VRAM (#567/#570/#571/#580+).
|
|
assert _workers_for_free_vram(7.0) == 1
|
|
assert _workers_for_free_vram(6.5) == 1
|
|
# ≤10 GB stays single-worker under the 5 GB/job budget.
|
|
assert _workers_for_free_vram(9.5) == 1
|
|
|
|
|
|
def test_larger_cards_still_parallelize():
|
|
assert _workers_for_free_vram(11.0) == 2 # 12 GB
|
|
assert _workers_for_free_vram(15.0) == 3 # 16 GB
|
|
assert _workers_for_free_vram(23.0) == _GPU_WORKER_CAP # 24 GB → capped
|
|
|
|
|
|
def test_floor_and_cap():
|
|
# Never zero (a tiny/!-reported free figure still gets one worker)...
|
|
assert _workers_for_free_vram(0.4) == 1
|
|
assert _workers_for_free_vram(0.0) == 1
|
|
# ...and never above the cap, however large the card.
|
|
assert _workers_for_free_vram(256.0) == _GPU_WORKER_CAP
|
|
|
|
|
|
def test_budget_is_conservative_enough_for_the_asr_coload():
|
|
# Guard the constant itself: the co-loaded WhisperX large-v3 (~3 GB) plus
|
|
# TTS (~1.6 GB) means a concurrent clone job needs ~5 GB; a regression back
|
|
# toward 2.5 GB would re-enable the 2-worker-on-8 GB crash.
|
|
assert _GPU_VRAM_PER_JOB_GB >= 5.0
|