feat: pipeline error transparency — no more silent "unknown error" (plan-04, closes #131) (#136)

* docs(plan-04): spec + plan for pipeline error transparency (#131)

speckit spec/plan/research/data-model/contract/quickstart for plan-04.
Grounds the fix in the real code map: shared failure-event builder
(backend/core/failure.py) feeding tasks.py + dub_pipeline.py + dub_core.py,
non-empty reason guarantee, sanitized diagnostic block, frontend renderer
with docs deeplink. Closes-target: #131 (children #122, #63).

Design only — no code changes yet.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(pipeline): structured, non-empty failure events + logged tracebacks (#131)

plan-04 backend: no more silent "unknown error". A shared failure helper
guarantees a non-empty reason at every emit site and a sanitized,
copyable diagnostic block.

- backend/core/failure.py: build_failure()/build_failure_event() (reason
  falls back to the exception class name), sanitize() (reuses the
  logging_filter HF-token regex + redacts *TOKEN*/*KEY*/*SECRET* env values
  + home→~), diagnostic() (reuses the env capture), classify() reusing the
  error_docs_map 5-class taxonomy for the docs deeplink + hint.
- core/tasks.py worker: structured event instead of bare str(e); keeps the
  logged traceback.
- services/dub_pipeline.py: enrich download/extract error yields; ADD the
  missing outer `except Exception` (the #122 path — unhandled ingest errors
  were never surfaced with stage context); surface the previously-silent
  demucs/scene/thumbnail degradations as non-fatal `warning` events.
- api/routers/batch.py: guaranteed non-empty batch failure reason.

SSE payload is additive (legacy `error`/`stage`/`detail` keys preserved),
so existing frontends keep working and already show the specific reason.

Tests (TDD, fail-before/pass-after): 14 cases — non-empty-reason guarantee,
redaction, diagnostic sanitization, and the 3 Test-matrix triggers
(worker / extract / url). 483 passed, 0 regressions.

Closes #131. Refs #122, #63.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(dub-ui): show specific cause + docs deeplink + copyable diagnostic (#131)

plan-04 frontend. The backend now sends a structured, non-empty failure;
surface it to the user instead of "extract: unknown error".

- dubSlice: DubFailure type + dubFailure state/setter.
- useDubWorkflow: capture the structured failure on the SSE error event
  (reason/error_class/stage/hint/docs_topic/diagnostic); clear on new runs.
- DubTab: DubFailureNotice renders the actionable hint, an "Open docs"
  deeplink (via the existing errorDocsMap classifier), and a "Copy
  diagnostic" button — shown beneath the error badge in both failure banners.

typecheck + build clean; 66 frontend tests pass.

Refs #131, #122, #63.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(failure): annotate intentional best-effort excepts (CodeQL)

The new security workflow's CodeQL flagged 5 bare `except: pass` blocks.
All are deliberate best-effort guards (sanitize/diagnostic must never throw
on the failure path; the test cancels the worker to tear it down). Added
explanatory comments per CodeQL's py/empty-except rule. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Palash Debnath
2026-05-29 09:12:23 +05:30
committed by GitHub
co-authored by Claude Opus 4.8
parent 8162f52c08
commit b64f53b0af
18 changed files with 1225 additions and 13 deletions
+3 -1
View File
@@ -19,6 +19,7 @@ from fastapi import APIRouter, File, UploadFile, HTTPException, Form
from pydantic import BaseModel
from core.config import DATA_DIR
from core import failure
router = APIRouter()
logger = logging.getLogger("omnivoice.batch")
@@ -79,7 +80,8 @@ async def _worker():
job["finished_at"] = time.time()
except Exception as e:
job["status"] = "failed"
job["error"] = str(e)[:500]
# plan-04 (#131): guaranteed non-empty, structured reason.
job["error"] = failure.build_failure(e, stage="batch", include_diagnostic=False)["reason"]
job["finished_at"] = time.time()
logger.error("Batch job %s failed: %s", job_id, e, exc_info=True)
finally:
+188
View File
@@ -0,0 +1,188 @@
"""Shared pipeline-failure helper (plan-04 / #131).
Single source of truth for "what a failure looks like" so every emit site —
the TaskManager worker, the dub ingest pipeline, the dub/batch routers — produces
the same structured, **non-empty**, sanitized failure event instead of its own
ad-hoc ``str(e)`` (which is empty or cryptic for many exception types, and was
the root of the "extract: unknown error" reports in #122/#63).
Guarantees:
- ``reason`` is ALWAYS non-empty (falls back to the exception class name).
- ``detail`` / ``diagnostic`` are sanitized: HF tokens, ``*TOKEN*/*KEY*/*SECRET*``
env values, and absolute home paths never leak (Constitution I).
- the 5-class docs taxonomy is reused from ``core.error_docs_map`` (not
duplicated) so the deeplink contract stays single-sourced.
"""
from __future__ import annotations
import os
import platform
import re
import sys
from pathlib import Path
from typing import Any, Optional
from core import error_docs_map
from core.logging_filter import REDACTED, _HF_TOKEN_RE
# Env vars whose *name* implies a credential — their values are redacted.
_SECRET_NAME_RE = re.compile(r"(TOKEN|KEY|SECRET)", re.IGNORECASE)
_REDACTED_VALUE = "***REDACTED***"
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
# taxonomy; the docs URL itself stays owned by error_docs_map.
_HINTS: dict[str, str] = {
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
"PYANNOTE_LICENSE_REQUIRED": "Accept the pyannote model licenses on Hugging Face, then retry.",
}
def classify(reason: str) -> str:
"""Map a failure reason to a docs-taxonomy key, or "" when unknown.
Heuristic substring match — mirrors the frontend ``classifyError`` so the
backend log / diagnostic names the same class the UI deeplink will use.
"""
low = (reason or "").lower()
if "pkg_resources" in low:
return "PKG_RESOURCES_MISSING"
if "quarantine" in low or "is damaged" in low or "gatekeeper" in low:
return "GATEKEEPER_QUARANTINE"
if "webkit" in low or "white screen" in low or "dmabuf" in low or "appimage" in low:
return "APPIMAGE_WEBKIT_WHITESCREEN"
if "pyannote" in low or ("gated" in low and "model" in low) or "accept the" in low:
return "PYANNOTE_LICENSE_REQUIRED"
if ("huggingface" in low or "hf_token" in low or "401" in low or "unauthorized" in low) and (
"token" in low or "auth" in low or "401" in low or "unauthorized" in low
):
return "HF_AUTH_FAILED"
return ""
def sanitize(text: Optional[str]) -> str:
"""Redact secrets and strip the home path from a string.
- HF tokens (reuses the regex from ``core.logging_filter``)
- values of env vars whose name matches ``*TOKEN*/*KEY*/*SECRET*``
- the user's absolute home directory → ``~``
"""
if not text:
return text or ""
out = _HF_TOKEN_RE.sub(REDACTED, str(text))
for name, val in os.environ.items():
# Only redact substantial values so short/empty ones don't blank the text.
if val and len(val) >= 6 and _SECRET_NAME_RE.search(name):
out = out.replace(val, _REDACTED_VALUE)
try:
home = str(Path.home())
if home and home in out:
out = out.replace(home, "~")
except Exception:
# Best-effort: sanitize() must never raise (it runs on the failure path);
# if home-dir resolution fails, leave the text as-is rather than throw.
pass
return out
def _env_summary() -> str:
lines: list[str] = []
try:
lines.append(f"OS: {platform.platform()}")
except Exception:
# Best-effort env summary — omit the OS line rather than fail diagnostics.
pass
lines.append(f"Python: {sys.version.split()[0]}")
try:
import psutil # already a runtime dep
vm = psutil.virtual_memory()
lines.append(f"CPU: {os.cpu_count()} cores")
lines.append(f"RAM: {round(vm.total / 1024 ** 3, 1)} GB")
except Exception:
# Best-effort — omit CPU/RAM if psutil is unavailable or probing fails.
pass
# Only probe the GPU if torch is ALREADY imported — importing it here just
# to build a diagnostic would add seconds to every failure (and to tests).
torch = sys.modules.get("torch")
if torch is not None:
try:
if torch.cuda.is_available():
lines.append(f"GPU: CUDA {torch.cuda.get_device_name(0)}")
elif getattr(getattr(torch, "backends", None), "mps", None) and torch.backends.mps.is_available():
lines.append("GPU: MPS (Apple)")
else:
lines.append("GPU: CPU only")
except Exception:
# Best-effort — omit the GPU line if torch probing raises.
pass
return "\n".join(lines)
def diagnostic(*, reason: str, error_class: str, stage: str) -> str:
"""A sanitized, copy-paste-friendly diagnostic block for a failed job."""
block = (
"OmniVoice diagnostic\n"
"--------------------\n"
f"Stage: {stage}\n"
f"Error: {error_class}\n"
f"Reason: {reason}\n"
f"{_env_summary()}\n"
)
return sanitize(block)
def build_failure(
exc_or_msg: Any,
*,
stage: str,
context: Optional[dict] = None,
include_diagnostic: bool = True,
) -> dict:
"""Build the structured failure fields (no ``type`` — caller/prep_event adds it).
``reason`` is guaranteed non-empty: ``str(exc)`` → exception class name.
"""
if isinstance(exc_or_msg, BaseException):
error_class = type(exc_or_msg).__name__
raw = str(exc_or_msg).strip() or error_class
else:
error_class = "Error"
raw = str(exc_or_msg).strip() or "Unknown failure"
reason = sanitize(raw) or error_class
docs_topic = classify(raw)
fields: dict[str, Any] = {
"reason": reason,
"error": reason, # backward-compat mirror for older frontends
"error_class": error_class,
"stage": stage,
"hint": _HINTS.get(docs_topic, ""),
"docs_topic": docs_topic,
"docs_url": error_docs_map.ERROR_DOCS.get(docs_topic, ""),
"detail": sanitize(raw),
}
if context:
fields["context"] = {k: sanitize(str(v)) for k, v in context.items()}
if include_diagnostic:
fields["diagnostic"] = diagnostic(reason=reason, error_class=error_class, stage=stage)
return fields
def build_failure_event(
exc_or_msg: Any,
*,
stage: str,
event_type: str = "error",
context: Optional[dict] = None,
include_diagnostic: bool = True,
) -> dict:
"""``build_failure`` plus a ``type`` key, for SSE event sites (tasks.py)."""
return {
"type": event_type,
**build_failure(
exc_or_msg, stage=stage, context=context, include_diagnostic=include_diagnostic
),
}
+7 -3
View File
@@ -4,6 +4,7 @@ import json
import logging
from core import job_store
from core import failure
logger = logging.getLogger("omnivoice.tasks")
@@ -127,13 +128,16 @@ class TaskManager:
except Exception as e:
logger.exception("Task %s failed", task_id)
t["status"] = "failed"
t["error"] = str(e)
# plan-04 (#131): structured, non-empty failure event instead of
# a bare str(e) (which is empty/cryptic for many exception types).
evt = failure.build_failure_event(e, stage="task", context={"task_id": task_id})
t["error"] = evt["reason"]
try:
job_store.mark_failed(task_id, str(e))
job_store.mark_failed(task_id, evt["reason"])
except Exception:
logger.exception("job_store.mark_failed failed")
try:
await self._push_event(task_id, f"data: {json.dumps({'type': 'error', 'error': str(e)})}\n\n")
await self._push_event(task_id, f"data: {json.dumps(evt)}\n\n")
except Exception as push_err:
logger.warning("Failed to push error event for %s: %s", task_id, push_err)
finally:
+16 -2
View File
@@ -45,6 +45,7 @@ from services.ffmpeg_utils import find_ffmpeg, _get_semaphore, _spawn_with_retry
from services.model_manager import get_best_device
from core.db import db_conn, get_db
from core import event_bus
from core import failure
logger = logging.getLogger("omnivoice.dub_pipeline")
@@ -415,7 +416,8 @@ async def ingest_pipeline(
fetch_subs=fetch_subs, sub_langs=sub_langs,
)
except Exception as e:
yield prep_event("error", stage="download", error=str(e)[:300])
logger.exception("Download failed for job %s", job_id)
yield prep_event("error", **failure.build_failure(e, stage="download"))
shutil.rmtree(job_dir, ignore_errors=True)
return
filename = title or os.path.basename(video_path)
@@ -458,7 +460,8 @@ async def ingest_pipeline(
except asyncio.CancelledError:
raise
except Exception as e:
yield prep_event("error", stage="extract", error=str(e)[:300])
logger.exception("Extract failed for job %s", job_id)
yield prep_event("error", **failure.build_failure(e, stage="extract"))
return
try:
@@ -552,6 +555,8 @@ async def ingest_pipeline(
raise
except Exception as e:
logger.warning("Demucs failed for %s, falling back to mixed audio: %s", job_id, e)
# plan-04: surface the degradation (job continues with mixed audio).
yield prep_event("warning", **failure.build_failure(e, stage="demucs", include_diagnostic=False))
vocals_path = audio_path
no_vocals_path = None
yield prep_event("demucs_done",
@@ -569,6 +574,7 @@ async def ingest_pipeline(
raise
except Exception as e:
logger.warning("Scene detection failed for %s: %s", job_id, e)
yield prep_event("warning", **failure.build_failure(e, stage="scene", include_diagnostic=False))
yield prep_event("scene_done", count=len(scene_cuts))
thumb_path = os.path.join(job_dir, "thumb.jpg")
@@ -583,6 +589,7 @@ async def ingest_pipeline(
raise
except Exception as e:
logger.warning("Thumbnail extraction failed for %s: %s", job_id, e)
yield prep_event("warning", **failure.build_failure(e, stage="thumbnail", include_diagnostic=False))
_dub_jobs[job_id].update({
"vocals_path": vocals_path,
@@ -602,6 +609,13 @@ async def ingest_pipeline(
_dub_jobs.pop(job_id, None)
yield prep_event("cancelled")
raise
except Exception as e:
# plan-04 (#131): no unhandled ingest failure may be silent. Log the
# real traceback and surface a structured, non-empty reason with stage
# context instead of letting it bubble up as a bare task error.
logger.exception("Ingest pipeline failed for job %s", job_id)
yield prep_event("error", **failure.build_failure(e, stage="ingest"))
return
finally:
with _active_procs_lock:
_active_procs.pop(job_id, None)
+15 -6
View File
@@ -30,6 +30,7 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
const setDubFilename = useAppStore(s => s.setDubFilename);
const setDubDuration = useAppStore(s => s.setDubDuration);
const setDubError = useAppStore(s => s.setDubError);
const setDubFailure = useAppStore(s => s.setDubFailure);
const setDubTracks = useAppStore(s => s.setDubTracks);
const setDubTranscript = useAppStore(s => s.setDubTranscript);
const setDubProgress = useAppStore(s => s.setDubProgress);
@@ -141,7 +142,15 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
case 'scene_done': break;
case 'cached': setDubPrepStage('cached'); break;
case 'ready': close(); ctrl.signal.removeEventListener('abort', onAbort); resolve(m); return;
case 'error': close(); ctrl.signal.removeEventListener('abort', onAbort); reject(new Error(`${m.stage || 'prep'}: ${m.error || 'unknown error'}`)); return;
case 'error': {
close(); ctrl.signal.removeEventListener('abort', onAbort);
// plan-04 (#131): the backend now always sends a non-empty reason;
// capture the structured failure so the UI can show a hint + docs link
// + copyable diagnostic instead of a bare "unknown error".
const reason = m.reason || m.error || 'unknown error';
setDubFailure({ reason, errorClass: m.error_class, stage: m.stage, hint: m.hint, docsTopic: m.docs_topic, diagnostic: m.diagnostic });
reject(new Error(`${m.stage || 'prep'}: ${reason}`)); return;
}
case 'cancelled': close(); ctrl.signal.removeEventListener('abort', onAbort); reject(Object.assign(new Error('aborted'), { name: 'AbortError' })); return;
default: break;
}
@@ -153,12 +162,12 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
else reject(new Error('prep stream closed unexpectedly'));
}
};
}), [setDubPrepStage, setDubJobId, setDubDuration, setDubFilename]);
}), [setDubPrepStage, setDubJobId, setDubDuration, setDubFilename, setDubFailure]);
// ── Handlers ──
const handleDubUpload = useCallback(async (dubVideoFile) => {
if (!dubVideoFile) return;
setDubStep('uploading'); setDubError(''); setDubTracks([]); setDubPrepStage('download');
setDubStep('uploading'); setDubError(''); setDubFailure(null); setDubTracks([]); setDubPrepStage('download');
const ctrl = new AbortController();
dubAbortCtrlRef.current = ctrl;
const clientJobId = Math.random().toString(36).slice(2, 10);
@@ -184,12 +193,12 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
else { setDubError(err.message); setDubStep('idle'); toast.error('Upload failed: ' + err.message); useAppStore.getState().errorPill(err.message); }
setTranscribeStart(null);
} finally { dubAbortCtrlRef.current = null; }
}, [setDubStep, setDubError, setDubTracks, setDubPrepStage, setDubJobId, setDubFilename, setDubTaskId, setDubSegments, _waitForPrep, _waitForTranscribe, loadProjects, loadProfiles]);
}, [setDubStep, setDubError, setDubFailure, setDubTracks, setDubPrepStage, setDubJobId, setDubFilename, setDubTaskId, setDubSegments, _waitForPrep, _waitForTranscribe, loadProjects, loadProfiles]);
const handleDubIngestUrl = useCallback(async (url, opts = {}) => {
const clean = (url || '').trim();
if (!clean) return;
setDubStep('uploading'); setDubError(''); setDubTracks([]); setDubPrepStage('download');
setDubStep('uploading'); setDubError(''); setDubFailure(null); setDubTracks([]); setDubPrepStage('download');
const ctrl = new AbortController();
dubAbortCtrlRef.current = ctrl;
const clientJobId = Math.random().toString(36).slice(2, 10);
@@ -215,7 +224,7 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
else { setDubError(err.message); setDubStep('idle'); toast.error('URL ingest failed: ' + err.message); useAppStore.getState().errorPill(err.message); }
setTranscribeStart(null);
} finally { dubAbortCtrlRef.current = null; }
}, [setDubStep, setDubError, setDubTracks, setDubPrepStage, setDubJobId, setDubTaskId, setDubSegments, _waitForPrep, _waitForTranscribe, loadProjects, loadProfiles]);
}, [setDubStep, setDubError, setDubFailure, setDubTracks, setDubPrepStage, setDubJobId, setDubTaskId, setDubSegments, _waitForPrep, _waitForTranscribe, loadProjects, loadProfiles]);
const handleDubAbort = useCallback(async () => {
const jobId = dubClientJobIdRef.current || dubJobId;
+17
View File
@@ -723,3 +723,20 @@
border-color: color-mix(in srgb, #fe8019 45%, transparent) !important;
background: color-mix(in srgb, #fe8019 10%, transparent) !important;
}
/* plan-04 (#131): actionable failure detail beneath the error badge. */
.dub-failure-notice {
display: flex;
flex-direction: column;
gap: 4px;
margin-top: 4px;
}
.dub-failure-notice__hint {
font-size: 11px;
opacity: 0.85;
}
.dub-failure-notice__actions {
display: flex;
gap: 6px;
flex-wrap: wrap;
}
+38
View File
@@ -3,6 +3,7 @@ import {
PanelLeftOpen, PanelLeftClose, Film, Save, UploadCloud, Sparkles, Loader, Square,
FileText, Play, DownloadIcon, Volume2, Link2,
Languages, ChevronDown, ChevronUp, Wand2, Trash2, Check, Globe, UserSquare2, User, AlertCircle,
ExternalLink, Copy,
} from 'lucide-react';
// lucide-react exports DownloadIcon as "Download"; alias here to match App.jsx naming.
import { Download as Download } from 'lucide-react';
@@ -18,6 +19,7 @@ import { API } from '../api/client';
import { listTranslationEngines, installTranslationEngine } from '../api/engines';
import toast from 'react-hot-toast';
import { Button, Segmented, Badge, Progress } from '../ui';
import { openDocsFor, classifyError } from '../utils/errorDocsMap';
import GlossaryPanel from '../components/GlossaryPanel';
import ExportModal from '../components/ExportModal';
import MultiLangPicker from '../components/MultiLangPicker';
@@ -29,6 +31,39 @@ const LazyFallback = () => (
<div className="dub-lazy-fallback">Loading</div>
);
/** plan-04 (#131): actionable failure detail — hint + docs deeplink + a copyable
* diagnostic block — shown beneath the error badge when the backend sent a
* structured failure. */
function DubFailureNotice({ failure }) {
if (!failure) return null;
const topic = failure.docsTopic || classifyError(failure.reason);
const copyDiagnostic = async () => {
try {
await navigator.clipboard.writeText(failure.diagnostic || failure.reason);
toast.success('Diagnostic copied');
} catch {
toast.error('Copy failed');
}
};
return (
<div className="dub-failure-notice">
{failure.hint && <span className="dub-failure-notice__hint">{failure.hint}</span>}
<div className="dub-failure-notice__actions">
{topic && (
<Button variant="subtle" size="sm" onClick={() => openDocsFor(topic)}>
<ExternalLink size={11} /> Open docs
</Button>
)}
{failure.diagnostic && (
<Button variant="subtle" size="sm" onClick={copyDiagnostic}>
<Copy size={11} /> Copy diagnostic
</Button>
)}
</div>
</div>
);
}
export default function DubTab(props) {
const {
// Props that stay prop-threaded: non-serialisable state + handlers that
@@ -71,6 +106,7 @@ export default function DubTab(props) {
const setDubInstruct = useAppStore(s => s.setDubInstruct);
const dubTracks = useAppStore(s => s.dubTracks);
const dubError = useAppStore(s => s.dubError);
const dubFailure = useAppStore(s => s.dubFailure);
const dubProgress = useAppStore(s => s.dubProgress);
const isTranslating = useAppStore(s => s.isTranslating);
const preserveBg = useAppStore(s => s.preserveBg);
@@ -254,6 +290,7 @@ export default function DubTab(props) {
<Badge tone="danger">
<AlertCircle size={11} /> {dubError}
</Badge>
<DubFailureNotice failure={dubFailure} />
{handleDubRetryTranscribe && (
<Button
variant="subtle"
@@ -944,6 +981,7 @@ export default function DubTab(props) {
<Badge tone="danger">
<AlertCircle size={11} /> {dubError}
</Badge>
<DubFailureNotice failure={dubFailure} />
</div>
)}
<div className="dub-outputs-row">
+16 -1
View File
@@ -43,6 +43,17 @@ function resolve<T>(updater: Updater<T>, prev: T): T {
return typeof updater === 'function' ? (updater as (prev: T) => T)(prev) : updater;
}
/** Structured pipeline failure (plan-04 #131) — carries the specific cause,
* an actionable hint, an optional docs-topic key, and a copyable diagnostic. */
export interface DubFailure {
reason: string;
errorClass?: string;
stage?: string;
hint?: string;
docsTopic?: string;
diagnostic?: string;
}
export interface DubSlice {
// ── Pipeline state ────────────────────────────────────────────────────
dubJobId: string | null;
@@ -51,6 +62,7 @@ export interface DubSlice {
dubPrepStage: DubPrepStage;
dubProgress: DubProgress;
dubError: string;
dubFailure: DubFailure | null;
isTranslating: boolean;
// ── Content ───────────────────────────────────────────────────────────
@@ -98,6 +110,7 @@ export interface DubSlice {
setDubPrepStage: (v: Updater<DubPrepStage>) => void;
setDubProgress: (v: Updater<DubProgress>) => void;
setDubError: (v: Updater<string>) => void;
setDubFailure: (v: Updater<DubSlice['dubFailure']>) => void;
setIsTranslating: (v: Updater<boolean>) => void;
setDubSegments: (v: Updater<DubSegment[]>) => void;
setDubTranscript: (v: Updater<string>) => void;
@@ -119,7 +132,7 @@ export interface DubSlice {
const INITIAL: Omit<DubSlice,
| 'setDubJobId' | 'setDubStep' | 'setDubTaskId' | 'setDubPrepStage'
| 'setDubProgress' | 'setDubError' | 'setIsTranslating' | 'setDubSegments'
| 'setDubProgress' | 'setDubError' | 'setDubFailure' | 'setIsTranslating' | 'setDubSegments'
| 'setDubTranscript' | 'setDubFilename' | 'setDubDuration' | 'setDubTracks'
| 'setDubLang' | 'setDubLangCode' | 'setDubInstruct' | 'setPreserveBg'
| 'setDefaultTrack' | 'setExportTracks' | 'setPreviewSegIds' | 'setSpeakerClones'
@@ -131,6 +144,7 @@ const INITIAL: Omit<DubSlice,
dubPrepStage: null,
dubProgress: { current: 0, total: 0, text: '' },
dubError: '',
dubFailure: null,
isTranslating: false,
dubSegments: [],
dubTranscript: '',
@@ -158,6 +172,7 @@ export const createDubSlice: StateCreator<DubSlice, [], [], DubSlice> = (set, ge
setDubPrepStage: (v) => set((s) => ({ dubPrepStage: resolve(v, s.dubPrepStage) })),
setDubProgress: (v) => set((s) => ({ dubProgress: resolve(v, s.dubProgress) })),
setDubError: (v) => set((s) => ({ dubError: resolve(v, s.dubError) })),
setDubFailure: (v) => set((s) => ({ dubFailure: resolve(v, s.dubFailure) })),
setIsTranslating:(v) => set((s) => ({ isTranslating:resolve(v, s.isTranslating) })),
setDubSegments: (v) => set((s) => ({ dubSegments: resolve(v, s.dubSegments) })),
setDubTranscript:(v) => set((s) => ({ dubTranscript:resolve(v, s.dubTranscript) })),
@@ -0,0 +1,40 @@
# Specification Quality Checklist: Pipeline Error Transparency
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-05-29
**Feature**: [spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Spec seeded directly from issue #131 (defect + fix-sequence + test-matrix),
which removed the usual ambiguity — no [NEEDS CLARIFICATION] markers needed.
- Three reasonable defaults were taken (documented in Assumptions): reuse of the
existing error→docs deeplink, reuse of the existing failure-render surface in
the UI, and reuse of the bug-reporter redaction rules for the diagnostic block.
- Root-causing individual failures is explicitly out of scope (routes to
plan-01/02/03).
@@ -0,0 +1,55 @@
# Contract: SSE `error` / `warning` Event (additive)
**Channel**: `GET /tasks/stream/{task_id}` (Server-Sent Events) and the dub
generate/transcribe streams. Emitter: `backend/core/failure.build_failure_event`
via `backend/core/tasks.py` and `backend/services/dub_pipeline.py`.
## Backward compatibility
This is an **additive** change. The previous payload —
`{"type":"error","error":"<str>","stage?":"...","detail?":"..."}` — remains
valid: `error`, `stage`, and `detail` keys are still present. New keys are added.
An older frontend that reads only `error` continues to work.
## Event shape
```jsonc
// fatal failure
{
"type": "error",
"reason": "ffprobe could not open source: no such file", // non-empty, required
"error": "ffprobe could not open source: no such file", // compat mirror of reason
"error_class": "FileNotFoundError",
"stage": "extract",
"hint": "Pick a media file that exists and has an audio track.",
"docs_topic": "", // taxonomy key or "" — frontend resolves URL
"detail": "Traceback summary (sanitized) …",
"diagnostic": "OmniVoice diagnostic\n----…" // sanitized, copyable
}
// non-fatal degradation (job continues)
{
"type": "warning",
"reason": "Demucs unavailable — using mixed audio.",
"error_class": "RuntimeError",
"stage": "demucs",
"hint": "Install demucs for vocal isolation; dubbing proceeds without it."
}
```
## Guarantees (tested)
1. `type:"error"` events ALWAYS carry a non-empty `reason`.
2. The same failure produces a backend log line with a full traceback and the
`stage` (via `logger.exception`).
3. `detail` and `diagnostic` are sanitized: no `*TOKEN*/*KEY*/*SECRET*` values,
no `hf_…` tokens, home dir rendered as `~`.
4. `type:"warning"` does NOT set the terminal job error; the job continues.
## Consumer (frontend)
`frontend/src/api/dub.ts` parses the event; `store/dubSlice.ts` stores it
(`dubError` = `reason`, plus structured `dubFailure`). The renderer shows
`reason` + `hint`, resolves `docs_topic`/`reason` via
`errorDocsMap.classifyError` to a deeplink, and binds "Copy diagnostic" to
`diagnostic`.
@@ -0,0 +1,57 @@
# Phase 1 Data Model: Pipeline Error Transparency
No persisted entities — these are transient, in-flight shapes carried over SSE
and rendered in the frontend store. No DB, no migration.
## FailureEvent (SSE `error` payload)
Produced by `backend/core/failure.build_failure_event()`. Additive over the
current `{type:"error", error, stage?, detail?}` shape — old keys preserved.
| Field | Type | Required | Notes |
|-------|------|----------|-------|
| `type` | string | yes | Always `"error"` (or `"warning"` for non-fatal degradations). |
| `reason` | string | yes | **Non-empty.** Human-readable cause. Fallback chain: `str(exc)``repr(exc)``type(exc).__name__`. |
| `error_class` | string | yes | Exception type name (e.g. `FileNotFoundError`) or `"Error"`. |
| `stage` | string | yes | Where it failed: `download`/`extract`/`demucs`/`scene`/`thumbnail`/`task`/`preflight`/`upload`. |
| `hint` | string | no | One-line "what to do". Empty string when none. |
| `docs_topic` | string | no | Taxonomy key (e.g. `PKG_RESOURCES_MISSING`) or empty. Frontend resolves to a URL. |
| `detail` | string | no | Sanitized fuller message (formerly the truncated `str(e)[:300]`). |
| `diagnostic` | string | no | Sanitized, copyable block: failure summary + env summary. |
| `error` | string | yes (compat) | Mirror of `reason` so older frontends keep working. |
### Validation / invariants
- `reason` MUST be non-empty (enforced + unit-tested).
- `detail` and `diagnostic` MUST be sanitized: no values for env vars matching
`*TOKEN*|*KEY*|*SECRET*`, no `hf_…` tokens, home dir rendered as `~`.
- For `type:"warning"` (non-fatal degradation) the job continues; the event is
informational and does not set the terminal job error.
## DiagnosticBlock (string content of `diagnostic`)
Composed by `failure.diagnostic(event)`. Plain text, copy-paste friendly:
```
OmniVoice diagnostic
--------------------
Stage: extract
Error: FileNotFoundError
Reason: ffprobe could not open source: no such file
OS: <platform.platform()>
Python: <sys.version 1-line>
OmniVoice: <version>
CPU/RAM: <psutil summary>
GPU: <cuda/mps/none + VRAM>
Engine: <active TTS engine>
```
Reuses the opt-in bug-reporter environment capture. Excludes: audio content,
file paths under the home dir (shown relative to `~`), any secret-like env var.
## Frontend store delta (`dubSlice`)
`dubError: string` → extended so the renderer has the structured fields. Minimal
change: keep `dubError` (= `reason`) and add an optional
`dubFailure: { reason, errorClass, stage, hint, docsTopic, diagnostic } | null`.
`dubError` is never set to an empty string.
@@ -0,0 +1,148 @@
# Implementation Plan: Pipeline Error Transparency
**Branch**: `001-pipeline-error-transparency` | **Date**: 2026-05-29 | **Spec**: [spec.md](./spec.md)
**Input**: Feature specification from `specs/001-pipeline-error-transparency/spec.md` (plan-04 / #131; children #122, #63)
## Summary
Make every pipeline failure (dub/extract/ingest) self-describing: a non-empty,
specific reason in the UI (error class + actionable hint + docs deeplink when
known), a full traceback with stage/context in the backend log for every
failure path (including failures before the ingest stage), and a copyable,
sanitized diagnostic block. Achieved with one shared failure-event builder on
the backend, hardening of the three known emit/swallow sites, and a richer
frontend error renderer. Additive SSE payload — no data/schema change.
## Technical Context
**Language/Version**: Python 3.11 (backend), TypeScript/React (frontend, Vite + bun)
**Primary Dependencies**: FastAPI + SSE (`backend/core/tasks.py` TaskManager), Zustand store (frontend); existing `errorDocsMap.ts` classifier and `backend/core/logging_filter.py` redactor — both reused, not rebuilt.
**Storage**: N/A — no persisted state changes. SSE payload gains additive fields only.
**Testing**: pytest (`tests/`, `backend/tests/`), node:test (frontend). Regression tests required per Constitution V (fail before, pass after).
**Target Platform**: macOS (AS+Intel), Windows x64, Linux (AppImage + deb) — identical default behavior.
**Project Type**: Web application (FastAPI backend + React/Tauri desktop frontend).
**Performance Goals**: No hot-path impact — code runs only on the failure path.
**Constraints**: Local-first (diagnostic shown to user, never transmitted); no secrets / home paths in logs or diagnostic; backward-compatible additive payload.
**Scale/Scope**: ~3 backend change sites + 1 new helper module + 1 frontend renderer + regression tests. Closes #131, #122, #63.
## Constitution Check
*GATE: passes — this feature is the direct implementation of Principles II and V.*
| Principle | Status | Notes |
|-----------|--------|-------|
| I. Local-First Sovereignty | ✅ | Diagnostic block is rendered locally for the user to copy; nothing is transmitted. Redaction reuses `logging_filter` + adds `*TOKEN*/*KEY*/*SECRET*` env-value and home-path stripping. No new outbound calls. |
| II. First-Run That Actually Works | ✅ (implements) | Replaces "unknown error" with a real cause + docs deeplink; this principle defines the feature. |
| III. Cross-Platform Default Parity | ✅ | Failure handling is pure Python/JS with no OS branch; behavior identical on all three platforms. WAV-only path (no ffmpeg) covered by test matrix. |
| IV. Backward-Compatible Evolution | ✅ | SSE error payload gains fields (`error_class`, `reason`, `hint`, `stage`, `diagnostic`) but keeps the existing `error`/`detail` keys, so older frontends still work. No DB/schema/engine change. |
| V. Root-Cause Architecture & Regression Tests | ✅ (implements) | plan-04 cluster master; ships regression tests for the 3 Test-matrix triggers (fail before fix, pass after); enforces "visible errors" globally via the shared builder. |
No violations → Complexity Tracking omitted.
## Project Structure
### Documentation (this feature)
```text
specs/001-pipeline-error-transparency/
├── plan.md # This file
├── spec.md # Feature spec
├── research.md # Phase 0 — decisions
├── data-model.md # Phase 1 — FailureEvent + DiagnosticBlock shapes
├── contracts/
│ └── sse-error-event.md # SSE "error" event contract (additive)
├── quickstart.md # How to verify the 3 Test-matrix triggers
└── checklists/
└── requirements.md # Spec quality checklist (done)
```
### Source Code (repository root)
```text
backend/
├── core/
│ ├── failure.py # NEW — build_failure_event() + sanitize() + diagnostic()
│ ├── tasks.py # CHANGE — worker except (L127-138) uses build_failure_event
│ └── logging_filter.py # REUSE/EXTEND — redaction helpers
├── services/
│ └── dub_pipeline.py # CHANGE — enrich download/extract error yields; surface
│ # demucs/scene/thumbnail degradations as visible
│ # "warning" events instead of silent logger.warning
└── api/routers/
├── dub_core.py # CHANGE — guard upload/ingest-url handlers so pre-task
│ # failures emit a structured reason (not a bare 500)
└── batch.py # CHANGE — batch error uses the shared builder
frontend/
├── src/
│ ├── api/dub.ts # CHANGE — parse structured error fields from SSE
│ ├── store/dubSlice.ts # CHANGE — store structured failure (reason never empty)
│ ├── utils/errorDocsMap.ts # REUSE (+extend taxonomy only if a new class is needed)
│ └── components/ # CHANGE — render reason + hint + docs link + "Copy diagnostic"
└── ...
tests/
├── test_dub_error_transparency.py # NEW — 3 Test-matrix triggers + builder + sanitize
└── backend/test_dub_pipeline_wav.py # EXTEND — WAV-only failure transparency
```
**Structure Decision**: Existing web-app layout (backend/ + frontend/). The only
new file is `backend/core/failure.py` — a single, well-bounded helper so the
"non-empty reason + redaction + diagnostic" logic lives in one place and every
emit site calls it, rather than duplicating fallback logic across tasks.py,
dub_pipeline.py, dub_core.py, and batch.py.
## Approach (the three change points + the helper)
1. **`backend/core/failure.py` (new, the keystone).**
- `build_failure_event(exc_or_msg, *, stage, context=None) -> dict` returns
`{"type":"error","stage":stage,"error_class":<ExcType name or "Error">,
"reason":<non-empty>,"hint":<one-liner or "">,"docs_topic":<key or "">,
"detail":<sanitized full str>,"diagnostic":<sanitized block>}`.
- **Non-empty guarantee**: `reason = str(exc).strip() or repr(exc).strip() or
type(exc).__name__`. This is the core fix for empty/cryptic `str(e)`.
- `classify(reason) -> (docs_topic, hint)`: small backend mirror of the
frontend taxonomy keys so the server log + diagnostic name the class too;
frontend remains the source of truth for the actual docs URL.
- `sanitize(text)`: applies the existing HF-token regex from
`logging_filter`, plus env-value redaction for names matching
`*TOKEN*|*KEY*|*SECRET*`, plus home-dir → `~`.
- `diagnostic(event)`: composes the failure + a sanitized env summary
(reuse the bug-reporter capture: OS/CPU/GPU/versions; never audio/secrets).
2. **`backend/core/tasks.py` worker except (L127-138).** Keep
`logger.exception(...)` (full traceback). Replace the bare
`{'type':'error','error':str(e)}` push with
`build_failure_event(e, stage="task", context={"task_id":task_id})`. Keep the
legacy `error` key populated (= `reason`) for backward compat.
3. **`backend/services/dub_pipeline.py`.** Route the download/extract `except`
yields through `build_failure_event` (adds `error_class`/`hint`). Convert the
silent demucs/scene/thumbnail `logger.warning`-only fallbacks (L554/571/585)
to *also* yield a non-fatal `prep_event("warning", stage=..., reason=...)` so
the degradation is visible without failing the job. No bare `except: pass`.
4. **`backend/api/routers/dub_core.py` (and `batch.py`).** Wrap the pre-task
work (file write, URL/preflight, arg build) so a failure there emits the same
structured reason via SSE / job error instead of an opaque 500 or a truncated
`str(e)` — this is the #122 "exception before ingest_pipeline" path.
5. **Frontend.** `api/dub.ts` parses the new fields; `dubSlice` stores a
structured failure (reason guaranteed non-empty, falling back to a generic
"Something failed — see logs" only if the backend somehow sent nothing). The
error UI shows reason + hint, the `errorDocsMap.classifyError` deeplink, and a
"Copy diagnostic" button bound to the `diagnostic` field.
## Phase 0 / Phase 1 artifacts
research.md, data-model.md, contracts/sse-error-event.md, quickstart.md generated
alongside this plan (decisions are settled; few unknowns). See those files.
@@ -0,0 +1,36 @@
# Quickstart: Verifying Pipeline Error Transparency
How to confirm the three Test-matrix triggers from #131. Each must show a
specific cause in the UI **and** a full traceback in the backend log.
## Automated (the regression gate)
```bash
uv run pytest tests/test_dub_error_transparency.py -q
uv run pytest tests/backend/test_dub_pipeline_wav.py -q # WAV-only failure path
```
These assert, for each trigger: the emitted SSE event is `type:"error"` with a
non-empty `reason` + `error_class` + `stage`, the backend log (captured via
`caplog`) contains a traceback, and `detail`/`diagnostic` contain no secret-like
values or absolute home paths. They are written to FAIL on `main` (where the
event is a bare/empty `str(e)`) and PASS after the fix.
## Manual (in-app)
1. **Extract fails (bad input)** — start a dub on a corrupt or audio-less file.
- Expect: UI shows e.g. "extract — FileNotFoundError: … " + a hint (not
"unknown error"); backend log (LogsFooter → System) shows the traceback.
2. **Remote ingest fails** — dub-by-URL with an unreachable/invalid URL.
- Expect: specific cause (e.g. yt-dlp error) + hint; traceback in log.
3. **WAV-only input fails** — feed a malformed WAV (no ffmpeg/extract path).
- Expect: specific cause; traceback in log.
4. **Copy diagnostic** — on any failure, click "Copy diagnostic"; paste it.
- Expect: cause + stage + env summary; NO tokens/keys/secrets; home dir shown
as `~`.
## Cross-platform check (Constitution III)
Run trigger 3 (WAV-only, no ffmpeg) on macOS, Windows, and Linux; the UI message
and log behavior must be identical. The `smoke-matrix` CI job exercises the
in-process path on all three.
@@ -0,0 +1,79 @@
# Phase 0 Research: Pipeline Error Transparency
The spec was seeded from #131 with a complete defect + fix-sequence, and the
codebase was mapped before planning. Few open unknowns; the decisions below
resolve them.
## Decision 1 — One shared failure-event builder vs. fixing each site inline
**Decision**: Add `backend/core/failure.py` with `build_failure_event()` and route
every emit site (tasks.py, dub_pipeline.py, dub_core.py, batch.py) through it.
**Rationale**: The defect is duplicated fallback logic — each site does its own
`str(e)` (some `[:300]`, some `[:500]`, some empty). Centralizing the non-empty
guarantee + redaction + classification means a single tested code path governs
"what a failure looks like," satisfying Constitution V's "errors MUST be visible"
globally rather than per-site.
**Alternatives considered**: Patch each `except` independently — rejected: leaves
the next new emit site free to reintroduce a bare `str(e)`.
## Decision 2 — Where error classification / docs mapping lives
**Decision**: Frontend `errorDocsMap.ts` stays the source of truth for the docs
URL (it already has the 5-class taxonomy + `classifyError`). Backend emits
`error_class` + `reason` + a backend-side `docs_topic` *key* (not URL) for the
log/diagnostic only.
**Rationale**: Avoids duplicating the URL table on two sides; the frontend
already renders deeplinks. Backend only needs the class *name* for its log line
and diagnostic block.
**Alternatives considered**: Full backend URL table — rejected (duplication,
drift risk). Pure-frontend classification with backend sending only `str(e)`
rejected: the backend log/diagnostic should also name the class.
## Decision 3 — Non-fatal degradations (demucs/scene/thumbnail)
**Decision**: Keep them non-fatal (job continues with the fallback) but emit a
visible `prep_event("warning", stage, reason)` instead of a silent
`logger.warning`-only path.
**Rationale**: These are legitimate graceful degradations, not job failures, so
they must not fail the job (backward-compat with current behavior). But "silent"
violates the transparency principle — the user should see that demucs was
skipped. A non-fatal warning event threads that needle.
**Alternatives considered**: Promote them to fatal errors — rejected: changes
current successful-with-fallback behavior, would regress real jobs.
## Decision 4 — Diagnostic block delivery
**Decision**: Include a sanitized `diagnostic` string inside the SSE error event
payload; the frontend offers a "Copy diagnostic" button. No new endpoint.
**Rationale**: The failure event is already flowing to the client; attaching the
diagnostic avoids a second round-trip and a stateful "last failure" store. The
block is built and sanitized server-side where the env info lives.
**Alternatives considered**: A `/diag/last-failure` endpoint — rejected: adds
state + an extra call for no benefit; the SSE event is the natural carrier.
## Decision 5 — Redaction reuse
**Decision**: Reuse `backend/core/logging_filter.py`'s HF-token regex; add
`*TOKEN*|*KEY*|*SECRET*` env-value redaction and home-dir → `~` in
`failure.sanitize()`. Reuse the opt-in bug-reporter's environment capture for
the diagnostic summary.
**Rationale**: Single redaction definition; consistent with Constitution I and
the bug-reporter's existing privacy contract.
## Open risk — #122 "no logs at all"
The deepest symptom (#122: no backend log lines, instrumented code never
reached) may be a *logging-configuration* gap rather than a swallow. Mitigation
in-scope: (a) guard the pre-task handlers in `dub_core.py` so failures there are
caught + logged + surfaced; (b) verify `logger.exception` in tasks.py reaches the
in-app log buffer that `LogsFooter` reads. The specific root cause of #122's
underlying error, once visible, routes to plan-01/02/03 per spec scope.
@@ -0,0 +1,168 @@
# Feature Specification: Pipeline Error Transparency
**Feature Branch**: `001-pipeline-error-transparency`
**Created**: 2026-05-29
**Status**: Draft
**Input**: plan-04 (#131); children #122, #63. "No more silent 'unknown error'."
## User Scenarios & Testing *(mandatory)*
### User Story 1 - See why a job actually failed (Priority: P1)
A user runs a dubbing or extract job and it fails. Today they see only
"extract: unknown error" with no indication of cause. After this change, the
failure message names the real cause in plain language and tells them what to
do next (e.g., "Source media has no audio track — pick a file with audio", or a
link to the matching troubleshooting doc).
**Why this priority**: Until the failure reason is visible to the user, every
downstream bug is un-triageable and every job failure is a dead end. This is the
single highest-leverage slice — it delivers value on its own.
**Independent Test**: Trigger any pipeline failure (e.g., feed a corrupt or
audio-less file) and confirm the UI shows a specific, human-readable cause
instead of "unknown error".
**Acceptance Scenarios**:
1. **Given** a media file that fails to extract, **When** the user starts the
job, **Then** the UI shows a specific cause (error type + one-line "what to
do") and never a bare "unknown error".
2. **Given** a failure that maps to a known troubleshooting topic, **When** the
error is shown, **Then** the user is offered the matching docs deeplink.
3. **Given** any failure, **When** it surfaces, **Then** the reason text is
non-empty.
---
### User Story 2 - Full failure detail in the logs (Priority: P2)
A user (or the maintainer helping them) opens the backend logs after a failure
and finds the complete exception — type, message, and stack trace — with enough
context (which stage, which input) to diagnose it. Today the logs can be
completely silent: in #122 the instrumented code is never even reached.
**Why this priority**: Self-describing logs turn a low-information report into an
actionable one and make the maintainer's triage possible without a live repro.
**Independent Test**: Force a failure, then inspect the backend log and confirm
a full traceback with stage/context is present for that job.
**Acceptance Scenarios**:
1. **Given** a pipeline failure at any stage, **When** it occurs, **Then** the
backend log contains the real exception with a full stack trace and the
failing stage/context.
2. **Given** an exception thrown before the main ingest stage is reached,
**When** it occurs, **Then** it is still logged (no swallowed/short-circuited
failures).
---
### User Story 3 - Copyable diagnostic block (Priority: P3)
When a job fails, the user can copy a self-contained diagnostic block (cause,
stage, sanitized environment summary) to paste into a bug report, so even a
terse report is answerable.
**Why this priority**: A prevention net that shrinks low-information reports
(#63-style). Builds on US1/US2; valuable but not required for the core fix.
**Independent Test**: Trigger a failure, use the "copy diagnostic" affordance,
and confirm the copied text contains the cause, stage, and a sanitized
environment summary — and nothing sensitive.
**Acceptance Scenarios**:
1. **Given** a failed job, **When** the user copies the diagnostic block,
**Then** it contains the failure cause, stage, and a sanitized environment
summary.
2. **Given** the diagnostic block, **When** it is generated, **Then** it
contains no secrets (no `*TOKEN*`/`*KEY*`/`*SECRET*` values) and no absolute
home paths (home dir shown as `~`).
### Edge Cases
- **Failure before any stage runs** (setup/validation): still produces a
specific reason and a log entry — this is the #122 case.
- **Non-exception failures** (a stage exits non-zero, a subprocess like ffmpeg
fails, a None/empty result): surfaced as a specific reason, not "unknown".
- **WAV-only input** that fails without touching the ffmpeg/extract path: same
transparency guarantees apply.
- **Failure cause has no matching docs topic**: show the specific cause without
a deeplink rather than suppressing the message.
- **Very long or multi-line underlying error**: UI shows a concise summary;
full detail remains in logs and the diagnostic block.
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: The system MUST surface a specific, human-readable failure reason
for every pipeline failure (dub, extract, ingest). A bare "unknown error" MUST
never be the only thing shown.
- **FR-002**: Every failure reason presented to the user MUST be non-empty and
MUST include the underlying error type plus a one-line actionable hint.
- **FR-003**: The system MUST log the real exception — type, message, full stack
trace — together with the failing stage and relevant input context, for every
failure path, including failures that occur before the main ingest stage.
- **FR-004**: The system MUST NOT swallow or short-circuit pipeline exceptions
without logging them.
- **FR-005**: When a failure maps to a known troubleshooting topic, the system
MUST offer the corresponding docs deeplink (reusing the existing error→docs
mechanism).
- **FR-006**: The system MUST emit a structured failure event to the frontend so
the frontend always receives a non-empty machine-readable reason (not just a
display string).
- **FR-007**: The system MUST provide a copyable diagnostic block containing the
cause, failing stage, and a sanitized environment summary.
- **FR-008**: The diagnostic block and any logged context MUST redact secrets
(values of env vars matching `*TOKEN*`/`*KEY*`/`*SECRET*`) and MUST strip
absolute home paths (render the home directory as `~`).
- **FR-009**: Behavior MUST be identical by default on macOS, Windows, and Linux
(no platform-divergent default error handling).
- **FR-010**: The change MUST be backward-compatible — no change to project data,
no schema migration, and existing successful jobs behave exactly as before.
### Key Entities
- **Failure event**: a structured record of a pipeline failure — cause/error
type, human-readable reason, actionable hint, optional docs topic, failing
stage, and timestamp. Delivered to the frontend and used to render the message
and the diagnostic block.
- **Diagnostic block**: a user-copyable, sanitized text rendering of a failure
event plus environment summary, intended for pasting into a bug report.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: 100% of induced pipeline failures (bad input, failed remote
ingest, WAV-only failure) display a specific cause in the UI — 0% show a bare
"unknown error".
- **SC-002**: 100% of induced pipeline failures produce a backend log entry
containing a full stack trace and the failing stage.
- **SC-003**: Every failure delivered to the frontend carries a non-empty reason
(no empty/placeholder reason in any tested path).
- **SC-004**: A user can produce a copyable diagnostic block for a failed job in
one action, and that block contains no secrets or absolute home paths in any
tested case.
- **SC-005**: The three Test-matrix triggers from #131 (extract-fails,
remote-ingest-fails, WAV-only-fails) all satisfy SC-001 and SC-002 under
automated test.
## Assumptions
- An error→docs deeplink mechanism already exists (shipped on the onboarding /
bug-report work) and can be reused for FR-005; this feature wires causes to it
rather than building it anew.
- The frontend already has a place to render job failures (toast/status) that
can be extended to show a specific reason and a "copy diagnostic" affordance.
- "Sanitized environment summary" reuses the redaction rules already defined for
the opt-in bug reporter (OS/CPU/GPU/versions; no audio, no secrets, no home
paths).
- Identifying and fixing the *root cause* of any specific failure once it is
visible is out of scope here and routes to plan-01/02/03 (#128130).
@@ -0,0 +1,69 @@
# Tasks: Pipeline Error Transparency
**Feature**: plan-04 (#131) | **Branch**: `001-pipeline-error-transparency`
**Spec**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md)
**TDD**: strict red-green — every test task is written and confirmed FAILING before its implementation task.
## Phase 1: Setup
- [ ] T001 Confirm no new dependencies are required (psutil, torch, platform already pinned); add the new test module path `tests/test_dub_error_transparency.py` to the pytest collection by creating an empty stub so CI discovers it.
## Phase 2: Foundational — shared failure helper (BLOCKS all stories)
- [ ] T002 [P] Write failing unit tests for the failure helper in `tests/test_failure_helper.py`: (a) `build_failure_event` produces a non-empty `reason` when `str(exc)` is empty (e.g. a custom `Exception()` with no message) — falls back to `repr` then type name; (b) event always contains `type`, `reason`, `error_class`, `stage`, and a compat `error` key equal to `reason`; (c) `sanitize()` redacts `hf_…` tokens, values of env vars matching `*TOKEN*|*KEY*|*SECRET*`, and rewrites the home dir to `~`; (d) `diagnostic()` output contains stage + error_class + an env summary and NONE of the redacted material. Run and confirm RED (module does not exist yet).
- [ ] T003 Implement `backend/core/failure.py`: `build_failure_event(exc_or_msg, *, stage, context=None)`, `sanitize(text)` (reuse `_HF_TOKEN_RE` from `backend/core/logging_filter.py`), `classify(reason) -> (docs_topic, hint)` (mirror the 5 `errorDocsMap` keys), and `diagnostic(event)` (reuse the opt-in bug-reporter env capture). Make T002 GREEN.
## Phase 3: User Story 1 — See why a job actually failed (P1) 🎯 MVP
**Goal**: UI shows a specific, non-empty cause (never "unknown error") + actionable hint + docs deeplink.
**Independent test**: induce any pipeline failure → UI shows specific cause, not "unknown error".
- [ ] T004 [P] [US1] Write failing test in `tests/test_dub_error_transparency.py`: extract-fails-on-bad-input → the SSE/event payload emitted by the worker is `type:"error"` with non-empty `reason` + `error_class` + `stage="extract"`. Confirm RED on current `str(e)`-only behavior.
- [ ] T005 [P] [US1] Write failing test (same file): remote/url ingest failure → structured non-empty `reason` + `stage`. Confirm RED.
- [ ] T006 [US1] Harden `backend/core/tasks.py` worker `except` (L127-138): keep `logger.exception`, replace the bare `{'type':'error','error':str(e)}` push with `build_failure_event(e, stage="task", context={"task_id": task_id})` (preserve the `error` key).
- [ ] T007 [US1] Route the `download`/`extract` `except` yields in `backend/services/dub_pipeline.py` (L418, L461) through `build_failure_event` so they carry `error_class` + `hint` and a non-empty `reason`.
- [ ] T008 [US1] Guard pre-task work in `backend/api/routers/dub_core.py` (upload + ingest-url handlers) and `backend/api/routers/batch.py` (L82) so a failure before the task starts emits a structured reason (the #122 path), not a bare 500 / truncated `str(e)`. Make T004/T005 GREEN.
- [ ] T009 [P] [US1] Frontend: parse the new fields in `frontend/src/api/dub.ts`; store structured failure in `frontend/src/store/dubSlice.ts` (`dubError` = `reason`, never empty; add optional `dubFailure`).
- [ ] T010 [US1] Frontend: render `reason` + `hint` + the `errorDocsMap.classifyError` deeplink in the dub failure UI component (consumer of `dubError`). Extend `errorDocsMap` taxonomy ONLY if a needed class is missing.
## Phase 4: User Story 2 — Full failure detail in the logs (P2)
**Goal**: every failure path logs the real exception with full traceback + stage/context, including failures before the ingest stage and non-fatal degradations.
**Independent test**: force a failure → backend log has a traceback + stage for that job.
- [ ] T011 [P] [US2] Write failing test in `tests/test_dub_error_transparency.py` using `caplog`: each of the 3 triggers logs a full traceback (via `logger.exception`) including the `stage`. Confirm RED where logging is currently silent/absent.
- [ ] T012 [US2] Ensure every emit site calls `logger.exception(...)` (or `logger.error(..., exc_info=True)`) with stage + context before/at the structured event — audit tasks.py, dub_pipeline.py, dub_core.py, batch.py for any `except` that logs nothing or `except: pass`; fix each. Make T011 GREEN.
- [ ] T013 [US2] Convert the silent demucs/scene/thumbnail fallbacks in `backend/services/dub_pipeline.py` (L554/571/585) to ALSO yield a non-fatal `prep_event("warning", stage=…, reason=…)` (job still continues) so degradations are visible. Add a test asserting `type:"warning"` does not set the terminal job error.
## Phase 5: User Story 3 — Copyable diagnostic block (P3)
**Goal**: one-action copyable, sanitized diagnostic for a failed job.
**Independent test**: trigger a failure → copy diagnostic → contains cause+stage+env, no secrets/home paths.
- [ ] T014 [P] [US3] Write failing test (`tests/test_dub_error_transparency.py`): the `diagnostic` field on an error event contains stage + error_class + env summary and NONE of: `hf_…` tokens, `*TOKEN*/*KEY*/*SECRET*` values, absolute home path. Confirm RED.
- [ ] T015 [US3] Populate the `diagnostic` field in `build_failure_event` for fatal errors (via `failure.diagnostic`). Make T014 GREEN.
- [ ] T016 [US3] Frontend: add a "Copy diagnostic" button in the failure UI bound to `dubFailure.diagnostic` (clipboard copy). Add a node:test for the parse→store path producing a non-empty diagnostic.
## Phase 6: Polish & Cross-Cutting
- [ ] T017 [P] Extend `tests/backend/test_dub_pipeline_wav.py` with the WAV-only failure transparency case (no ffmpeg path) — the 3rd Test-matrix trigger. Confirm it asserts SC-001 + SC-002.
- [ ] T018 [P] Update `docs/` troubleshooting: note that failures now show a specific cause + copyable diagnostic (ties error→docs deeplink).
- [ ] T019 Run full suite (`uv run pytest tests/ backend/tests/ -q` + frontend `bun test`) and the `quickstart.md` manual triggers; confirm all green and 0 "unknown error" outputs across the 3 triggers.
- [ ] T020 Self-review against Constitution I/III/IV/V (no outbound calls, identical cross-platform default, no data/schema change, regression tests fail-before/pass-after) before opening the PR.
## Dependencies & order
- **Phase 2 (T002-T003) blocks everything** — the helper is the keystone.
- US1 (P1) is the MVP and can ship alone. US2 builds on US1's emit sites. US3 builds on the helper's `diagnostic()`.
- Within a story: test task(s) first (RED), then implementation (GREEN).
## Parallel opportunities
- T002 (helper tests) ∥ nothing (foundational, first).
- T004, T005 (US1 backend tests) ∥ T009 (frontend parse) once T003 lands.
- T011 (US2 log test) ∥ T014 (US3 diagnostic test) — different assertions, same file (coordinate edits).
- T017, T018 polish ∥ each other.
## MVP scope
**User Story 1 only** (T001-T010): replaces "unknown error" with a specific cause in the UI + logs the real exception at the worker boundary. Delivers the core of #131/#122 on its own.
+181
View File
@@ -0,0 +1,181 @@
"""plan-04 (#131) — pipeline error transparency regression tests.
Test-matrix from the issue (every failure → specific UI cause + logged
traceback). Written RED before the emit-site changes. Fixture-free: the worker
path is pure-Python; the dub-pipeline paths force failure via a missing file and
a monkeypatched downloader, so they don't need real media/network.
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import uuid
os.environ.setdefault("OMNIVOICE_DISABLE_FILE_LOG", "1")
import pytest
from core.db import init_db
from core.tasks import TaskManager
from services import dub_pipeline as dp
@pytest.fixture(autouse=True)
def _db():
init_db()
yield
def _error_events(history) -> list[dict]:
out = []
for e in history:
if e and isinstance(e, str) and e.startswith("data:") and '"type": "error"' in e:
out.append(json.loads(e[len("data: "):]))
return out
async def _empty_boom(*a, **k):
"""Async-generator task that raises with an EMPTY message (the cryptic case)."""
if False:
yield
raise ValueError("")
async def _runtime_boom(*a, **k):
if False:
yield
raise RuntimeError("ffprobe blew up")
async def _drain_failing_task(boom):
"""Run a failing task and return all SSE events the worker emits.
Race-free: the listener is registered BEFORE the worker is started, so no
event (including the terminal error + EOF) can be missed, and we drain to
the EOF sentinel instead of cancelling the worker mid-push.
"""
tm = TaskManager()
tid = f"t_{uuid.uuid4().hex[:8]}"
await tm.add_task(tid, "prep", boom)
q: asyncio.Queue = asyncio.Queue()
await tm.add_listener(tid, q)
worker = asyncio.create_task(tm.worker())
events: list = []
try:
while True:
ev = await asyncio.wait_for(q.get(), timeout=10)
if ev is None: # EOF sentinel pushed in the worker's finally
break
events.append(ev)
finally:
worker.cancel()
try:
await worker
except asyncio.CancelledError:
# Expected: we cancel the worker loop to tear it down after draining.
pass
return events
# ── US1 + US2: worker failure path (the #122 "unknown error" / silent log) ──
def test_worker_failure_emits_structured_nonempty_reason():
"""A task that raises with an EMPTY message must still surface a specific,
non-empty reason + error_class + stage — not a bare/empty string — and log
the real exception with a traceback (US2)."""
# Capture directly on the task logger — robust against the app's logging
# config (propagate flags) and asyncio task boundaries.
records: list[logging.LogRecord] = []
class _Capture(logging.Handler):
def emit(self, record): # noqa: D401
records.append(record)
handler = _Capture(level=logging.ERROR)
tlog = logging.getLogger("omnivoice.tasks")
# The app may have run dictConfig(disable_existing_loggers=True) on import,
# which leaves this logger disabled in the test process. Force it live so we
# can assert the worker actually logs the traceback.
tlog.disabled = False
tlog.setLevel(logging.DEBUG)
tlog.addHandler(handler)
try:
events = asyncio.run(_drain_failing_task(_empty_boom))
finally:
tlog.removeHandler(handler)
errs = _error_events(events)
assert errs, "worker must push a structured error event"
evt = errs[-1]
assert evt["reason"], "reason must be non-empty even for an empty-message exception"
assert evt["error_class"] == "ValueError"
assert evt["stage"] == "task"
# US2: the real exception was logged with a traceback
assert any(r.exc_info for r in records), "expected a logged traceback"
# ── US1: extract-fails-on-bad-input (Test-matrix #1) ────────────────────────
def test_extract_failure_yields_structured_error(tmp_path):
async def _run():
events = []
src = {"path": str(tmp_path / "does_not_exist.mp4")}
async for ev in dp.ingest_pipeline("j_ext", str(tmp_path), src):
events.append(ev)
return events
errs = _error_events(asyncio.run(_run()))
assert errs, "a failed extract must yield an error event"
evt = errs[-1]
assert evt["stage"] in ("extract", "ingest")
assert evt["reason"]
assert evt["error_class"] # structured, not a bare string
# ── US1: remote/url ingest failure (Test-matrix #2) ─────────────────────────
def test_url_ingest_failure_yields_structured_error(tmp_path, monkeypatch):
def _boom(*a, **k):
raise RuntimeError("yt-dlp: Video unavailable")
monkeypatch.setattr(dp, "yt_download_sync", _boom)
async def _run():
events = []
src = {"kind": "url", "url": "https://example.com/watch?v=x"}
async for ev in dp.ingest_pipeline("j_url", str(tmp_path), src):
events.append(ev)
return events
errs = _error_events(asyncio.run(_run()))
assert errs, "a failed url ingest must yield an error event"
evt = errs[-1]
assert evt["stage"] in ("download", "ingest")
assert evt["error_class"] == "RuntimeError"
assert "unavailable" in evt["reason"].lower()
# ── US3: fatal error event carries a sanitized diagnostic block ─────────────
def test_fatal_error_event_carries_sanitized_diagnostic():
leaked = "hf_" + "C" * 36
prev = os.environ.get("HF_TOKEN")
os.environ["HF_TOKEN"] = leaked
try:
events = asyncio.run(_drain_failing_task(_runtime_boom))
finally:
if prev is None:
os.environ.pop("HF_TOKEN", None)
else:
os.environ["HF_TOKEN"] = prev
errs = _error_events(events)
assert errs, "fatal error must emit a structured event"
evt = errs[-1]
assert evt.get("diagnostic"), "fatal error must carry a copyable diagnostic block"
assert "task" in evt["diagnostic"]
assert leaked not in evt["diagnostic"], "diagnostic must not leak the HF token"
+92
View File
@@ -0,0 +1,92 @@
"""plan-04 (#131) — unit tests for the shared failure helper.
These are the foundational (Phase 2) tests: the non-empty-reason guarantee,
redaction, classification, and the diagnostic block. Written RED before
`core/failure.py` exists.
"""
from pathlib import Path
from core import failure
# ── Non-empty reason guarantee (the core fix) ───────────────────────────────
def test_reason_non_empty_when_exception_message_empty():
evt = failure.build_failure(ValueError(""), stage="extract")
assert evt["reason"], "reason must never be empty"
assert evt["error_class"] == "ValueError"
assert evt["stage"] == "extract"
# Backward-compat mirror used by older frontends.
assert evt["error"] == evt["reason"]
def test_reason_uses_message_when_present():
evt = failure.build_failure(FileNotFoundError("no such file: clip.mp4"), stage="extract")
assert "no such file" in evt["reason"]
assert evt["error_class"] == "FileNotFoundError"
def test_accepts_plain_string_message():
evt = failure.build_failure("preflight: ffmpeg not found", stage="preflight")
assert evt["reason"] == "preflight: ffmpeg not found"
assert evt["stage"] == "preflight"
def test_build_failure_event_carries_type():
evt = failure.build_failure_event(RuntimeError("boom"), stage="task")
assert evt["type"] == "error"
assert evt["reason"] == "boom"
# warning variant for non-fatal degradations
warn = failure.build_failure_event(RuntimeError("demucs down"), stage="demucs", event_type="warning")
assert warn["type"] == "warning"
# ── Redaction (Constitution I) ──────────────────────────────────────────────
def test_sanitize_redacts_hf_token():
tok = "hf_" + "A" * 36
out = failure.sanitize(f"auth failed using {tok} on download")
assert tok not in out
def test_sanitize_redacts_secret_env_values(monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "sk-supersecretvalue123456")
out = failure.sanitize("request died: sk-supersecretvalue123456 rejected")
assert "sk-supersecretvalue123456" not in out
def test_sanitize_strips_home_path():
home = str(Path.home())
out = failure.sanitize(f"could not open {home}/Movies/clip.mp4")
assert home not in out
assert "~" in out
# ── Diagnostic block (US3) ──────────────────────────────────────────────────
def test_diagnostic_has_context_and_no_secrets(monkeypatch):
leaked = "hf_" + "B" * 36
monkeypatch.setenv("HF_TOKEN", leaked)
evt = failure.build_failure(RuntimeError("ffprobe exploded"), stage="extract")
diag = evt["diagnostic"]
assert "extract" in diag
assert "RuntimeError" in diag
assert leaked not in diag
# carries an environment summary
assert ("OS" in diag) or ("Python" in diag)
# ── Classification → docs topic + hint (US1, FR-005) ────────────────────────
def test_docs_topic_and_hint_for_known_class():
evt = failure.build_failure(
ModuleNotFoundError("No module named 'pkg_resources'"), stage="task"
)
assert evt["docs_topic"] == "PKG_RESOURCES_MISSING"
assert evt["hint"], "known classes must carry an actionable hint"
def test_unknown_cause_has_empty_topic_but_still_non_empty_reason():
evt = failure.build_failure(RuntimeError("totally novel failure xyz"), stage="task")
assert evt["docs_topic"] == ""
assert evt["reason"]