* fix(gpu): record WHERE a wedged job was stuck when its budget expires
Three reporters on v0.4.2 hit "TTS generate ran for more than 300s of
actual compute time and was abandoned" (#1338, #1329, #1348) — two of them
on an RTX 3050 and an RTX 3060, rendering a single sentence. That is not a
machine too slow for the job. The message says "too heavy for the
available compute" because it is the only story the timeout path can tell.
And nothing in the log could contradict it. The timeout branch logged THAT
the budget was exceeded, reset the pool, and returned. The worker cannot
be cancelled, so it was still running on a real stack — and we threw that
away, which is why every report of this class arrives undiagnosable and
the only advice available is "reproduce it under a debugger".
sys._current_frames() reads the frame of every live thread including one
wedged inside a C call, which is exactly this case. Filtered to gpu-pool
workers so the log names the stuck job rather than the web server, capped
at 25 frames, and it can never raise — a diagnostic that throws would
replace a real GpuJobTimeoutError with an unrelated crash.
Ordering is load-bearing and asserted: the capture runs BEFORE reset(),
because reset() swaps in a fresh executor and the wedged thread then stops
being identifiable as a pool worker — the diagnostic would still run, still
log, and be empty, which looks like it worked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(gpu): redact home paths from captured stacks; label stale workers
Both review findings on this PR, both valid.
CodeRabbit (CWE-532): traceback frames carry absolute source paths, which
on a user machine start with their home directory — their account name.
This log lands in backend.log, which goes into diagnostic bundles and
prefilled bug reports, so it has to be sanitized like every other surfaced
text. Reuses core.failure.sanitize rather than inventing a second answer;
if sanitizing itself fails the stacks are dropped, not logged raw.
greptile P1: a wedged worker survives reset() — it cannot be cancelled and
keeps running under the same gpu-pool name the replacement pool uses. The
second timeout in a session would log both with nothing to tell them
apart, and the stale one is the more misleading, since it names an
operation that is not the job that just failed. The live pool is now
identified through its own thread set and the others are marked STALE.
That set comes from ThreadPoolExecutor._threads, which is private, so
unknown internals degrade to labelling nothing rather than to failing —
a diagnostic that vanishes because an attribute moved is worse than an
unlabelled one, and that degrade path has its own test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
