* docs(plan-04): spec + plan for pipeline error transparency (#131) speckit spec/plan/research/data-model/contract/quickstart for plan-04. Grounds the fix in the real code map: shared failure-event builder (backend/core/failure.py) feeding tasks.py + dub_pipeline.py + dub_core.py, non-empty reason guarantee, sanitized diagnostic block, frontend renderer with docs deeplink. Closes-target: #131 (children #122, #63). Design only — no code changes yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): structured, non-empty failure events + logged tracebacks (#131) plan-04 backend: no more silent "unknown error". A shared failure helper guarantees a non-empty reason at every emit site and a sanitized, copyable diagnostic block. - backend/core/failure.py: build_failure()/build_failure_event() (reason falls back to the exception class name), sanitize() (reuses the logging_filter HF-token regex + redacts *TOKEN*/*KEY*/*SECRET* env values + home→~), diagnostic() (reuses the env capture), classify() reusing the error_docs_map 5-class taxonomy for the docs deeplink + hint. - core/tasks.py worker: structured event instead of bare str(e); keeps the logged traceback. - services/dub_pipeline.py: enrich download/extract error yields; ADD the missing outer `except Exception` (the #122 path — unhandled ingest errors were never surfaced with stage context); surface the previously-silent demucs/scene/thumbnail degradations as non-fatal `warning` events. - api/routers/batch.py: guaranteed non-empty batch failure reason. SSE payload is additive (legacy `error`/`stage`/`detail` keys preserved), so existing frontends keep working and already show the specific reason. Tests (TDD, fail-before/pass-after): 14 cases — non-empty-reason guarantee, redaction, diagnostic sanitization, and the 3 Test-matrix triggers (worker / extract / url). 483 passed, 0 regressions. Closes #131. Refs #122, #63. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(dub-ui): show specific cause + docs deeplink + copyable diagnostic (#131) plan-04 frontend. The backend now sends a structured, non-empty failure; surface it to the user instead of "extract: unknown error". - dubSlice: DubFailure type + dubFailure state/setter. - useDubWorkflow: capture the structured failure on the SSE error event (reason/error_class/stage/hint/docs_topic/diagnostic); clear on new runs. - DubTab: DubFailureNotice renders the actionable hint, an "Open docs" deeplink (via the existing errorDocsMap classifier), and a "Copy diagnostic" button — shown beneath the error badge in both failure banners. typecheck + build clean; 66 frontend tests pass. Refs #131, #122, #63. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(failure): annotate intentional best-effort excepts (CodeQL) The new security workflow's CodeQL flagged 5 bare `except: pass` blocks. All are deliberate best-effort guards (sanitize/diagnostic must never throw on the failure path; the test cancels the worker to tear it down). Added explanatory comments per CodeQL's py/empty-except rule. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
8162f52c08
commit
b64f53b0af
@@ -19,6 +19,7 @@ from fastapi import APIRouter, File, UploadFile, HTTPException, Form
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core.config import DATA_DIR
|
||||
from core import failure
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.batch")
|
||||
@@ -79,7 +80,8 @@ async def _worker():
|
||||
job["finished_at"] = time.time()
|
||||
except Exception as e:
|
||||
job["status"] = "failed"
|
||||
job["error"] = str(e)[:500]
|
||||
# plan-04 (#131): guaranteed non-empty, structured reason.
|
||||
job["error"] = failure.build_failure(e, stage="batch", include_diagnostic=False)["reason"]
|
||||
job["finished_at"] = time.time()
|
||||
logger.error("Batch job %s failed: %s", job_id, e, exc_info=True)
|
||||
finally:
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
"""Shared pipeline-failure helper (plan-04 / #131).
|
||||
|
||||
Single source of truth for "what a failure looks like" so every emit site —
|
||||
the TaskManager worker, the dub ingest pipeline, the dub/batch routers — produces
|
||||
the same structured, **non-empty**, sanitized failure event instead of its own
|
||||
ad-hoc ``str(e)`` (which is empty or cryptic for many exception types, and was
|
||||
the root of the "extract: unknown error" reports in #122/#63).
|
||||
|
||||
Guarantees:
|
||||
- ``reason`` is ALWAYS non-empty (falls back to the exception class name).
|
||||
- ``detail`` / ``diagnostic`` are sanitized: HF tokens, ``*TOKEN*/*KEY*/*SECRET*``
|
||||
env values, and absolute home paths never leak (Constitution I).
|
||||
- the 5-class docs taxonomy is reused from ``core.error_docs_map`` (not
|
||||
duplicated) so the deeplink contract stays single-sourced.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from core import error_docs_map
|
||||
from core.logging_filter import REDACTED, _HF_TOKEN_RE
|
||||
|
||||
# Env vars whose *name* implies a credential — their values are redacted.
|
||||
_SECRET_NAME_RE = re.compile(r"(TOKEN|KEY|SECRET)", re.IGNORECASE)
|
||||
_REDACTED_VALUE = "***REDACTED***"
|
||||
|
||||
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
|
||||
# taxonomy; the docs URL itself stays owned by error_docs_map.
|
||||
_HINTS: dict[str, str] = {
|
||||
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
|
||||
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
|
||||
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
|
||||
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
|
||||
"PYANNOTE_LICENSE_REQUIRED": "Accept the pyannote model licenses on Hugging Face, then retry.",
|
||||
}
|
||||
|
||||
|
||||
def classify(reason: str) -> str:
|
||||
"""Map a failure reason to a docs-taxonomy key, or "" when unknown.
|
||||
|
||||
Heuristic substring match — mirrors the frontend ``classifyError`` so the
|
||||
backend log / diagnostic names the same class the UI deeplink will use.
|
||||
"""
|
||||
low = (reason or "").lower()
|
||||
if "pkg_resources" in low:
|
||||
return "PKG_RESOURCES_MISSING"
|
||||
if "quarantine" in low or "is damaged" in low or "gatekeeper" in low:
|
||||
return "GATEKEEPER_QUARANTINE"
|
||||
if "webkit" in low or "white screen" in low or "dmabuf" in low or "appimage" in low:
|
||||
return "APPIMAGE_WEBKIT_WHITESCREEN"
|
||||
if "pyannote" in low or ("gated" in low and "model" in low) or "accept the" in low:
|
||||
return "PYANNOTE_LICENSE_REQUIRED"
|
||||
if ("huggingface" in low or "hf_token" in low or "401" in low or "unauthorized" in low) and (
|
||||
"token" in low or "auth" in low or "401" in low or "unauthorized" in low
|
||||
):
|
||||
return "HF_AUTH_FAILED"
|
||||
return ""
|
||||
|
||||
|
||||
def sanitize(text: Optional[str]) -> str:
|
||||
"""Redact secrets and strip the home path from a string.
|
||||
|
||||
- HF tokens (reuses the regex from ``core.logging_filter``)
|
||||
- values of env vars whose name matches ``*TOKEN*/*KEY*/*SECRET*``
|
||||
- the user's absolute home directory → ``~``
|
||||
"""
|
||||
if not text:
|
||||
return text or ""
|
||||
out = _HF_TOKEN_RE.sub(REDACTED, str(text))
|
||||
for name, val in os.environ.items():
|
||||
# Only redact substantial values so short/empty ones don't blank the text.
|
||||
if val and len(val) >= 6 and _SECRET_NAME_RE.search(name):
|
||||
out = out.replace(val, _REDACTED_VALUE)
|
||||
try:
|
||||
home = str(Path.home())
|
||||
if home and home in out:
|
||||
out = out.replace(home, "~")
|
||||
except Exception:
|
||||
# Best-effort: sanitize() must never raise (it runs on the failure path);
|
||||
# if home-dir resolution fails, leave the text as-is rather than throw.
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def _env_summary() -> str:
|
||||
lines: list[str] = []
|
||||
try:
|
||||
lines.append(f"OS: {platform.platform()}")
|
||||
except Exception:
|
||||
# Best-effort env summary — omit the OS line rather than fail diagnostics.
|
||||
pass
|
||||
lines.append(f"Python: {sys.version.split()[0]}")
|
||||
try:
|
||||
import psutil # already a runtime dep
|
||||
|
||||
vm = psutil.virtual_memory()
|
||||
lines.append(f"CPU: {os.cpu_count()} cores")
|
||||
lines.append(f"RAM: {round(vm.total / 1024 ** 3, 1)} GB")
|
||||
except Exception:
|
||||
# Best-effort — omit CPU/RAM if psutil is unavailable or probing fails.
|
||||
pass
|
||||
# Only probe the GPU if torch is ALREADY imported — importing it here just
|
||||
# to build a diagnostic would add seconds to every failure (and to tests).
|
||||
torch = sys.modules.get("torch")
|
||||
if torch is not None:
|
||||
try:
|
||||
if torch.cuda.is_available():
|
||||
lines.append(f"GPU: CUDA {torch.cuda.get_device_name(0)}")
|
||||
elif getattr(getattr(torch, "backends", None), "mps", None) and torch.backends.mps.is_available():
|
||||
lines.append("GPU: MPS (Apple)")
|
||||
else:
|
||||
lines.append("GPU: CPU only")
|
||||
except Exception:
|
||||
# Best-effort — omit the GPU line if torch probing raises.
|
||||
pass
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def diagnostic(*, reason: str, error_class: str, stage: str) -> str:
|
||||
"""A sanitized, copy-paste-friendly diagnostic block for a failed job."""
|
||||
block = (
|
||||
"OmniVoice diagnostic\n"
|
||||
"--------------------\n"
|
||||
f"Stage: {stage}\n"
|
||||
f"Error: {error_class}\n"
|
||||
f"Reason: {reason}\n"
|
||||
f"{_env_summary()}\n"
|
||||
)
|
||||
return sanitize(block)
|
||||
|
||||
|
||||
def build_failure(
|
||||
exc_or_msg: Any,
|
||||
*,
|
||||
stage: str,
|
||||
context: Optional[dict] = None,
|
||||
include_diagnostic: bool = True,
|
||||
) -> dict:
|
||||
"""Build the structured failure fields (no ``type`` — caller/prep_event adds it).
|
||||
|
||||
``reason`` is guaranteed non-empty: ``str(exc)`` → exception class name.
|
||||
"""
|
||||
if isinstance(exc_or_msg, BaseException):
|
||||
error_class = type(exc_or_msg).__name__
|
||||
raw = str(exc_or_msg).strip() or error_class
|
||||
else:
|
||||
error_class = "Error"
|
||||
raw = str(exc_or_msg).strip() or "Unknown failure"
|
||||
|
||||
reason = sanitize(raw) or error_class
|
||||
docs_topic = classify(raw)
|
||||
fields: dict[str, Any] = {
|
||||
"reason": reason,
|
||||
"error": reason, # backward-compat mirror for older frontends
|
||||
"error_class": error_class,
|
||||
"stage": stage,
|
||||
"hint": _HINTS.get(docs_topic, ""),
|
||||
"docs_topic": docs_topic,
|
||||
"docs_url": error_docs_map.ERROR_DOCS.get(docs_topic, ""),
|
||||
"detail": sanitize(raw),
|
||||
}
|
||||
if context:
|
||||
fields["context"] = {k: sanitize(str(v)) for k, v in context.items()}
|
||||
if include_diagnostic:
|
||||
fields["diagnostic"] = diagnostic(reason=reason, error_class=error_class, stage=stage)
|
||||
return fields
|
||||
|
||||
|
||||
def build_failure_event(
|
||||
exc_or_msg: Any,
|
||||
*,
|
||||
stage: str,
|
||||
event_type: str = "error",
|
||||
context: Optional[dict] = None,
|
||||
include_diagnostic: bool = True,
|
||||
) -> dict:
|
||||
"""``build_failure`` plus a ``type`` key, for SSE event sites (tasks.py)."""
|
||||
return {
|
||||
"type": event_type,
|
||||
**build_failure(
|
||||
exc_or_msg, stage=stage, context=context, include_diagnostic=include_diagnostic
|
||||
),
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import json
|
||||
import logging
|
||||
|
||||
from core import job_store
|
||||
from core import failure
|
||||
|
||||
logger = logging.getLogger("omnivoice.tasks")
|
||||
|
||||
@@ -127,13 +128,16 @@ class TaskManager:
|
||||
except Exception as e:
|
||||
logger.exception("Task %s failed", task_id)
|
||||
t["status"] = "failed"
|
||||
t["error"] = str(e)
|
||||
# plan-04 (#131): structured, non-empty failure event instead of
|
||||
# a bare str(e) (which is empty/cryptic for many exception types).
|
||||
evt = failure.build_failure_event(e, stage="task", context={"task_id": task_id})
|
||||
t["error"] = evt["reason"]
|
||||
try:
|
||||
job_store.mark_failed(task_id, str(e))
|
||||
job_store.mark_failed(task_id, evt["reason"])
|
||||
except Exception:
|
||||
logger.exception("job_store.mark_failed failed")
|
||||
try:
|
||||
await self._push_event(task_id, f"data: {json.dumps({'type': 'error', 'error': str(e)})}\n\n")
|
||||
await self._push_event(task_id, f"data: {json.dumps(evt)}\n\n")
|
||||
except Exception as push_err:
|
||||
logger.warning("Failed to push error event for %s: %s", task_id, push_err)
|
||||
finally:
|
||||
|
||||
@@ -45,6 +45,7 @@ from services.ffmpeg_utils import find_ffmpeg, _get_semaphore, _spawn_with_retry
|
||||
from services.model_manager import get_best_device
|
||||
from core.db import db_conn, get_db
|
||||
from core import event_bus
|
||||
from core import failure
|
||||
|
||||
logger = logging.getLogger("omnivoice.dub_pipeline")
|
||||
|
||||
@@ -415,7 +416,8 @@ async def ingest_pipeline(
|
||||
fetch_subs=fetch_subs, sub_langs=sub_langs,
|
||||
)
|
||||
except Exception as e:
|
||||
yield prep_event("error", stage="download", error=str(e)[:300])
|
||||
logger.exception("Download failed for job %s", job_id)
|
||||
yield prep_event("error", **failure.build_failure(e, stage="download"))
|
||||
shutil.rmtree(job_dir, ignore_errors=True)
|
||||
return
|
||||
filename = title or os.path.basename(video_path)
|
||||
@@ -458,7 +460,8 @@ async def ingest_pipeline(
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
yield prep_event("error", stage="extract", error=str(e)[:300])
|
||||
logger.exception("Extract failed for job %s", job_id)
|
||||
yield prep_event("error", **failure.build_failure(e, stage="extract"))
|
||||
return
|
||||
|
||||
try:
|
||||
@@ -552,6 +555,8 @@ async def ingest_pipeline(
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.warning("Demucs failed for %s, falling back to mixed audio: %s", job_id, e)
|
||||
# plan-04: surface the degradation (job continues with mixed audio).
|
||||
yield prep_event("warning", **failure.build_failure(e, stage="demucs", include_diagnostic=False))
|
||||
vocals_path = audio_path
|
||||
no_vocals_path = None
|
||||
yield prep_event("demucs_done",
|
||||
@@ -569,6 +574,7 @@ async def ingest_pipeline(
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.warning("Scene detection failed for %s: %s", job_id, e)
|
||||
yield prep_event("warning", **failure.build_failure(e, stage="scene", include_diagnostic=False))
|
||||
yield prep_event("scene_done", count=len(scene_cuts))
|
||||
|
||||
thumb_path = os.path.join(job_dir, "thumb.jpg")
|
||||
@@ -583,6 +589,7 @@ async def ingest_pipeline(
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.warning("Thumbnail extraction failed for %s: %s", job_id, e)
|
||||
yield prep_event("warning", **failure.build_failure(e, stage="thumbnail", include_diagnostic=False))
|
||||
|
||||
_dub_jobs[job_id].update({
|
||||
"vocals_path": vocals_path,
|
||||
@@ -602,6 +609,13 @@ async def ingest_pipeline(
|
||||
_dub_jobs.pop(job_id, None)
|
||||
yield prep_event("cancelled")
|
||||
raise
|
||||
except Exception as e:
|
||||
# plan-04 (#131): no unhandled ingest failure may be silent. Log the
|
||||
# real traceback and surface a structured, non-empty reason with stage
|
||||
# context instead of letting it bubble up as a bare task error.
|
||||
logger.exception("Ingest pipeline failed for job %s", job_id)
|
||||
yield prep_event("error", **failure.build_failure(e, stage="ingest"))
|
||||
return
|
||||
finally:
|
||||
with _active_procs_lock:
|
||||
_active_procs.pop(job_id, None)
|
||||
|
||||
@@ -30,6 +30,7 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
|
||||
const setDubFilename = useAppStore(s => s.setDubFilename);
|
||||
const setDubDuration = useAppStore(s => s.setDubDuration);
|
||||
const setDubError = useAppStore(s => s.setDubError);
|
||||
const setDubFailure = useAppStore(s => s.setDubFailure);
|
||||
const setDubTracks = useAppStore(s => s.setDubTracks);
|
||||
const setDubTranscript = useAppStore(s => s.setDubTranscript);
|
||||
const setDubProgress = useAppStore(s => s.setDubProgress);
|
||||
@@ -141,7 +142,15 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
|
||||
case 'scene_done': break;
|
||||
case 'cached': setDubPrepStage('cached'); break;
|
||||
case 'ready': close(); ctrl.signal.removeEventListener('abort', onAbort); resolve(m); return;
|
||||
case 'error': close(); ctrl.signal.removeEventListener('abort', onAbort); reject(new Error(`${m.stage || 'prep'}: ${m.error || 'unknown error'}`)); return;
|
||||
case 'error': {
|
||||
close(); ctrl.signal.removeEventListener('abort', onAbort);
|
||||
// plan-04 (#131): the backend now always sends a non-empty reason;
|
||||
// capture the structured failure so the UI can show a hint + docs link
|
||||
// + copyable diagnostic instead of a bare "unknown error".
|
||||
const reason = m.reason || m.error || 'unknown error';
|
||||
setDubFailure({ reason, errorClass: m.error_class, stage: m.stage, hint: m.hint, docsTopic: m.docs_topic, diagnostic: m.diagnostic });
|
||||
reject(new Error(`${m.stage || 'prep'}: ${reason}`)); return;
|
||||
}
|
||||
case 'cancelled': close(); ctrl.signal.removeEventListener('abort', onAbort); reject(Object.assign(new Error('aborted'), { name: 'AbortError' })); return;
|
||||
default: break;
|
||||
}
|
||||
@@ -153,12 +162,12 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
|
||||
else reject(new Error('prep stream closed unexpectedly'));
|
||||
}
|
||||
};
|
||||
}), [setDubPrepStage, setDubJobId, setDubDuration, setDubFilename]);
|
||||
}), [setDubPrepStage, setDubJobId, setDubDuration, setDubFilename, setDubFailure]);
|
||||
|
||||
// ── Handlers ──
|
||||
const handleDubUpload = useCallback(async (dubVideoFile) => {
|
||||
if (!dubVideoFile) return;
|
||||
setDubStep('uploading'); setDubError(''); setDubTracks([]); setDubPrepStage('download');
|
||||
setDubStep('uploading'); setDubError(''); setDubFailure(null); setDubTracks([]); setDubPrepStage('download');
|
||||
const ctrl = new AbortController();
|
||||
dubAbortCtrlRef.current = ctrl;
|
||||
const clientJobId = Math.random().toString(36).slice(2, 10);
|
||||
@@ -184,12 +193,12 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
|
||||
else { setDubError(err.message); setDubStep('idle'); toast.error('Upload failed: ' + err.message); useAppStore.getState().errorPill(err.message); }
|
||||
setTranscribeStart(null);
|
||||
} finally { dubAbortCtrlRef.current = null; }
|
||||
}, [setDubStep, setDubError, setDubTracks, setDubPrepStage, setDubJobId, setDubFilename, setDubTaskId, setDubSegments, _waitForPrep, _waitForTranscribe, loadProjects, loadProfiles]);
|
||||
}, [setDubStep, setDubError, setDubFailure, setDubTracks, setDubPrepStage, setDubJobId, setDubFilename, setDubTaskId, setDubSegments, _waitForPrep, _waitForTranscribe, loadProjects, loadProfiles]);
|
||||
|
||||
const handleDubIngestUrl = useCallback(async (url, opts = {}) => {
|
||||
const clean = (url || '').trim();
|
||||
if (!clean) return;
|
||||
setDubStep('uploading'); setDubError(''); setDubTracks([]); setDubPrepStage('download');
|
||||
setDubStep('uploading'); setDubError(''); setDubFailure(null); setDubTracks([]); setDubPrepStage('download');
|
||||
const ctrl = new AbortController();
|
||||
dubAbortCtrlRef.current = ctrl;
|
||||
const clientJobId = Math.random().toString(36).slice(2, 10);
|
||||
@@ -215,7 +224,7 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
|
||||
else { setDubError(err.message); setDubStep('idle'); toast.error('URL ingest failed: ' + err.message); useAppStore.getState().errorPill(err.message); }
|
||||
setTranscribeStart(null);
|
||||
} finally { dubAbortCtrlRef.current = null; }
|
||||
}, [setDubStep, setDubError, setDubTracks, setDubPrepStage, setDubJobId, setDubTaskId, setDubSegments, _waitForPrep, _waitForTranscribe, loadProjects, loadProfiles]);
|
||||
}, [setDubStep, setDubError, setDubFailure, setDubTracks, setDubPrepStage, setDubJobId, setDubTaskId, setDubSegments, _waitForPrep, _waitForTranscribe, loadProjects, loadProfiles]);
|
||||
|
||||
const handleDubAbort = useCallback(async () => {
|
||||
const jobId = dubClientJobIdRef.current || dubJobId;
|
||||
|
||||
@@ -723,3 +723,20 @@
|
||||
border-color: color-mix(in srgb, #fe8019 45%, transparent) !important;
|
||||
background: color-mix(in srgb, #fe8019 10%, transparent) !important;
|
||||
}
|
||||
|
||||
/* plan-04 (#131): actionable failure detail beneath the error badge. */
|
||||
.dub-failure-notice {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
margin-top: 4px;
|
||||
}
|
||||
.dub-failure-notice__hint {
|
||||
font-size: 11px;
|
||||
opacity: 0.85;
|
||||
}
|
||||
.dub-failure-notice__actions {
|
||||
display: flex;
|
||||
gap: 6px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
PanelLeftOpen, PanelLeftClose, Film, Save, UploadCloud, Sparkles, Loader, Square,
|
||||
FileText, Play, DownloadIcon, Volume2, Link2,
|
||||
Languages, ChevronDown, ChevronUp, Wand2, Trash2, Check, Globe, UserSquare2, User, AlertCircle,
|
||||
ExternalLink, Copy,
|
||||
} from 'lucide-react';
|
||||
// lucide-react exports DownloadIcon as "Download"; alias here to match App.jsx naming.
|
||||
import { Download as Download } from 'lucide-react';
|
||||
@@ -18,6 +19,7 @@ import { API } from '../api/client';
|
||||
import { listTranslationEngines, installTranslationEngine } from '../api/engines';
|
||||
import toast from 'react-hot-toast';
|
||||
import { Button, Segmented, Badge, Progress } from '../ui';
|
||||
import { openDocsFor, classifyError } from '../utils/errorDocsMap';
|
||||
import GlossaryPanel from '../components/GlossaryPanel';
|
||||
import ExportModal from '../components/ExportModal';
|
||||
import MultiLangPicker from '../components/MultiLangPicker';
|
||||
@@ -29,6 +31,39 @@ const LazyFallback = () => (
|
||||
<div className="dub-lazy-fallback">Loading…</div>
|
||||
);
|
||||
|
||||
/** plan-04 (#131): actionable failure detail — hint + docs deeplink + a copyable
|
||||
* diagnostic block — shown beneath the error badge when the backend sent a
|
||||
* structured failure. */
|
||||
function DubFailureNotice({ failure }) {
|
||||
if (!failure) return null;
|
||||
const topic = failure.docsTopic || classifyError(failure.reason);
|
||||
const copyDiagnostic = async () => {
|
||||
try {
|
||||
await navigator.clipboard.writeText(failure.diagnostic || failure.reason);
|
||||
toast.success('Diagnostic copied');
|
||||
} catch {
|
||||
toast.error('Copy failed');
|
||||
}
|
||||
};
|
||||
return (
|
||||
<div className="dub-failure-notice">
|
||||
{failure.hint && <span className="dub-failure-notice__hint">{failure.hint}</span>}
|
||||
<div className="dub-failure-notice__actions">
|
||||
{topic && (
|
||||
<Button variant="subtle" size="sm" onClick={() => openDocsFor(topic)}>
|
||||
<ExternalLink size={11} /> Open docs
|
||||
</Button>
|
||||
)}
|
||||
{failure.diagnostic && (
|
||||
<Button variant="subtle" size="sm" onClick={copyDiagnostic}>
|
||||
<Copy size={11} /> Copy diagnostic
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function DubTab(props) {
|
||||
const {
|
||||
// Props that stay prop-threaded: non-serialisable state + handlers that
|
||||
@@ -71,6 +106,7 @@ export default function DubTab(props) {
|
||||
const setDubInstruct = useAppStore(s => s.setDubInstruct);
|
||||
const dubTracks = useAppStore(s => s.dubTracks);
|
||||
const dubError = useAppStore(s => s.dubError);
|
||||
const dubFailure = useAppStore(s => s.dubFailure);
|
||||
const dubProgress = useAppStore(s => s.dubProgress);
|
||||
const isTranslating = useAppStore(s => s.isTranslating);
|
||||
const preserveBg = useAppStore(s => s.preserveBg);
|
||||
@@ -254,6 +290,7 @@ export default function DubTab(props) {
|
||||
<Badge tone="danger">
|
||||
<AlertCircle size={11} /> {dubError}
|
||||
</Badge>
|
||||
<DubFailureNotice failure={dubFailure} />
|
||||
{handleDubRetryTranscribe && (
|
||||
<Button
|
||||
variant="subtle"
|
||||
@@ -944,6 +981,7 @@ export default function DubTab(props) {
|
||||
<Badge tone="danger">
|
||||
<AlertCircle size={11} /> {dubError}
|
||||
</Badge>
|
||||
<DubFailureNotice failure={dubFailure} />
|
||||
</div>
|
||||
)}
|
||||
<div className="dub-outputs-row">
|
||||
|
||||
@@ -43,6 +43,17 @@ function resolve<T>(updater: Updater<T>, prev: T): T {
|
||||
return typeof updater === 'function' ? (updater as (prev: T) => T)(prev) : updater;
|
||||
}
|
||||
|
||||
/** Structured pipeline failure (plan-04 #131) — carries the specific cause,
|
||||
* an actionable hint, an optional docs-topic key, and a copyable diagnostic. */
|
||||
export interface DubFailure {
|
||||
reason: string;
|
||||
errorClass?: string;
|
||||
stage?: string;
|
||||
hint?: string;
|
||||
docsTopic?: string;
|
||||
diagnostic?: string;
|
||||
}
|
||||
|
||||
export interface DubSlice {
|
||||
// ── Pipeline state ────────────────────────────────────────────────────
|
||||
dubJobId: string | null;
|
||||
@@ -51,6 +62,7 @@ export interface DubSlice {
|
||||
dubPrepStage: DubPrepStage;
|
||||
dubProgress: DubProgress;
|
||||
dubError: string;
|
||||
dubFailure: DubFailure | null;
|
||||
isTranslating: boolean;
|
||||
|
||||
// ── Content ───────────────────────────────────────────────────────────
|
||||
@@ -98,6 +110,7 @@ export interface DubSlice {
|
||||
setDubPrepStage: (v: Updater<DubPrepStage>) => void;
|
||||
setDubProgress: (v: Updater<DubProgress>) => void;
|
||||
setDubError: (v: Updater<string>) => void;
|
||||
setDubFailure: (v: Updater<DubSlice['dubFailure']>) => void;
|
||||
setIsTranslating: (v: Updater<boolean>) => void;
|
||||
setDubSegments: (v: Updater<DubSegment[]>) => void;
|
||||
setDubTranscript: (v: Updater<string>) => void;
|
||||
@@ -119,7 +132,7 @@ export interface DubSlice {
|
||||
|
||||
const INITIAL: Omit<DubSlice,
|
||||
| 'setDubJobId' | 'setDubStep' | 'setDubTaskId' | 'setDubPrepStage'
|
||||
| 'setDubProgress' | 'setDubError' | 'setIsTranslating' | 'setDubSegments'
|
||||
| 'setDubProgress' | 'setDubError' | 'setDubFailure' | 'setIsTranslating' | 'setDubSegments'
|
||||
| 'setDubTranscript' | 'setDubFilename' | 'setDubDuration' | 'setDubTracks'
|
||||
| 'setDubLang' | 'setDubLangCode' | 'setDubInstruct' | 'setPreserveBg'
|
||||
| 'setDefaultTrack' | 'setExportTracks' | 'setPreviewSegIds' | 'setSpeakerClones'
|
||||
@@ -131,6 +144,7 @@ const INITIAL: Omit<DubSlice,
|
||||
dubPrepStage: null,
|
||||
dubProgress: { current: 0, total: 0, text: '' },
|
||||
dubError: '',
|
||||
dubFailure: null,
|
||||
isTranslating: false,
|
||||
dubSegments: [],
|
||||
dubTranscript: '',
|
||||
@@ -158,6 +172,7 @@ export const createDubSlice: StateCreator<DubSlice, [], [], DubSlice> = (set, ge
|
||||
setDubPrepStage: (v) => set((s) => ({ dubPrepStage: resolve(v, s.dubPrepStage) })),
|
||||
setDubProgress: (v) => set((s) => ({ dubProgress: resolve(v, s.dubProgress) })),
|
||||
setDubError: (v) => set((s) => ({ dubError: resolve(v, s.dubError) })),
|
||||
setDubFailure: (v) => set((s) => ({ dubFailure: resolve(v, s.dubFailure) })),
|
||||
setIsTranslating:(v) => set((s) => ({ isTranslating:resolve(v, s.isTranslating) })),
|
||||
setDubSegments: (v) => set((s) => ({ dubSegments: resolve(v, s.dubSegments) })),
|
||||
setDubTranscript:(v) => set((s) => ({ dubTranscript:resolve(v, s.dubTranscript) })),
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Specification Quality Checklist: Pipeline Error Transparency
|
||||
|
||||
**Purpose**: Validate specification completeness and quality before proceeding to planning
|
||||
**Created**: 2026-05-29
|
||||
**Feature**: [spec.md](../spec.md)
|
||||
|
||||
## Content Quality
|
||||
|
||||
- [x] No implementation details (languages, frameworks, APIs)
|
||||
- [x] Focused on user value and business needs
|
||||
- [x] Written for non-technical stakeholders
|
||||
- [x] All mandatory sections completed
|
||||
|
||||
## Requirement Completeness
|
||||
|
||||
- [x] No [NEEDS CLARIFICATION] markers remain
|
||||
- [x] Requirements are testable and unambiguous
|
||||
- [x] Success criteria are measurable
|
||||
- [x] Success criteria are technology-agnostic (no implementation details)
|
||||
- [x] All acceptance scenarios are defined
|
||||
- [x] Edge cases are identified
|
||||
- [x] Scope is clearly bounded
|
||||
- [x] Dependencies and assumptions identified
|
||||
|
||||
## Feature Readiness
|
||||
|
||||
- [x] All functional requirements have clear acceptance criteria
|
||||
- [x] User scenarios cover primary flows
|
||||
- [x] Feature meets measurable outcomes defined in Success Criteria
|
||||
- [x] No implementation details leak into specification
|
||||
|
||||
## Notes
|
||||
|
||||
- Spec seeded directly from issue #131 (defect + fix-sequence + test-matrix),
|
||||
which removed the usual ambiguity — no [NEEDS CLARIFICATION] markers needed.
|
||||
- Three reasonable defaults were taken (documented in Assumptions): reuse of the
|
||||
existing error→docs deeplink, reuse of the existing failure-render surface in
|
||||
the UI, and reuse of the bug-reporter redaction rules for the diagnostic block.
|
||||
- Root-causing individual failures is explicitly out of scope (routes to
|
||||
plan-01/02/03).
|
||||
@@ -0,0 +1,55 @@
|
||||
# Contract: SSE `error` / `warning` Event (additive)
|
||||
|
||||
**Channel**: `GET /tasks/stream/{task_id}` (Server-Sent Events) and the dub
|
||||
generate/transcribe streams. Emitter: `backend/core/failure.build_failure_event`
|
||||
via `backend/core/tasks.py` and `backend/services/dub_pipeline.py`.
|
||||
|
||||
## Backward compatibility
|
||||
|
||||
This is an **additive** change. The previous payload —
|
||||
`{"type":"error","error":"<str>","stage?":"...","detail?":"..."}` — remains
|
||||
valid: `error`, `stage`, and `detail` keys are still present. New keys are added.
|
||||
An older frontend that reads only `error` continues to work.
|
||||
|
||||
## Event shape
|
||||
|
||||
```jsonc
|
||||
// fatal failure
|
||||
{
|
||||
"type": "error",
|
||||
"reason": "ffprobe could not open source: no such file", // non-empty, required
|
||||
"error": "ffprobe could not open source: no such file", // compat mirror of reason
|
||||
"error_class": "FileNotFoundError",
|
||||
"stage": "extract",
|
||||
"hint": "Pick a media file that exists and has an audio track.",
|
||||
"docs_topic": "", // taxonomy key or "" — frontend resolves URL
|
||||
"detail": "Traceback summary (sanitized) …",
|
||||
"diagnostic": "OmniVoice diagnostic\n----…" // sanitized, copyable
|
||||
}
|
||||
|
||||
// non-fatal degradation (job continues)
|
||||
{
|
||||
"type": "warning",
|
||||
"reason": "Demucs unavailable — using mixed audio.",
|
||||
"error_class": "RuntimeError",
|
||||
"stage": "demucs",
|
||||
"hint": "Install demucs for vocal isolation; dubbing proceeds without it."
|
||||
}
|
||||
```
|
||||
|
||||
## Guarantees (tested)
|
||||
|
||||
1. `type:"error"` events ALWAYS carry a non-empty `reason`.
|
||||
2. The same failure produces a backend log line with a full traceback and the
|
||||
`stage` (via `logger.exception`).
|
||||
3. `detail` and `diagnostic` are sanitized: no `*TOKEN*/*KEY*/*SECRET*` values,
|
||||
no `hf_…` tokens, home dir rendered as `~`.
|
||||
4. `type:"warning"` does NOT set the terminal job error; the job continues.
|
||||
|
||||
## Consumer (frontend)
|
||||
|
||||
`frontend/src/api/dub.ts` parses the event; `store/dubSlice.ts` stores it
|
||||
(`dubError` = `reason`, plus structured `dubFailure`). The renderer shows
|
||||
`reason` + `hint`, resolves `docs_topic`/`reason` via
|
||||
`errorDocsMap.classifyError` to a deeplink, and binds "Copy diagnostic" to
|
||||
`diagnostic`.
|
||||
@@ -0,0 +1,57 @@
|
||||
# Phase 1 Data Model: Pipeline Error Transparency
|
||||
|
||||
No persisted entities — these are transient, in-flight shapes carried over SSE
|
||||
and rendered in the frontend store. No DB, no migration.
|
||||
|
||||
## FailureEvent (SSE `error` payload)
|
||||
|
||||
Produced by `backend/core/failure.build_failure_event()`. Additive over the
|
||||
current `{type:"error", error, stage?, detail?}` shape — old keys preserved.
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|-------|------|----------|-------|
|
||||
| `type` | string | yes | Always `"error"` (or `"warning"` for non-fatal degradations). |
|
||||
| `reason` | string | yes | **Non-empty.** Human-readable cause. Fallback chain: `str(exc)` → `repr(exc)` → `type(exc).__name__`. |
|
||||
| `error_class` | string | yes | Exception type name (e.g. `FileNotFoundError`) or `"Error"`. |
|
||||
| `stage` | string | yes | Where it failed: `download`/`extract`/`demucs`/`scene`/`thumbnail`/`task`/`preflight`/`upload`. |
|
||||
| `hint` | string | no | One-line "what to do". Empty string when none. |
|
||||
| `docs_topic` | string | no | Taxonomy key (e.g. `PKG_RESOURCES_MISSING`) or empty. Frontend resolves to a URL. |
|
||||
| `detail` | string | no | Sanitized fuller message (formerly the truncated `str(e)[:300]`). |
|
||||
| `diagnostic` | string | no | Sanitized, copyable block: failure summary + env summary. |
|
||||
| `error` | string | yes (compat) | Mirror of `reason` so older frontends keep working. |
|
||||
|
||||
### Validation / invariants
|
||||
|
||||
- `reason` MUST be non-empty (enforced + unit-tested).
|
||||
- `detail` and `diagnostic` MUST be sanitized: no values for env vars matching
|
||||
`*TOKEN*|*KEY*|*SECRET*`, no `hf_…` tokens, home dir rendered as `~`.
|
||||
- For `type:"warning"` (non-fatal degradation) the job continues; the event is
|
||||
informational and does not set the terminal job error.
|
||||
|
||||
## DiagnosticBlock (string content of `diagnostic`)
|
||||
|
||||
Composed by `failure.diagnostic(event)`. Plain text, copy-paste friendly:
|
||||
|
||||
```
|
||||
OmniVoice diagnostic
|
||||
--------------------
|
||||
Stage: extract
|
||||
Error: FileNotFoundError
|
||||
Reason: ffprobe could not open source: no such file
|
||||
OS: <platform.platform()>
|
||||
Python: <sys.version 1-line>
|
||||
OmniVoice: <version>
|
||||
CPU/RAM: <psutil summary>
|
||||
GPU: <cuda/mps/none + VRAM>
|
||||
Engine: <active TTS engine>
|
||||
```
|
||||
|
||||
Reuses the opt-in bug-reporter environment capture. Excludes: audio content,
|
||||
file paths under the home dir (shown relative to `~`), any secret-like env var.
|
||||
|
||||
## Frontend store delta (`dubSlice`)
|
||||
|
||||
`dubError: string` → extended so the renderer has the structured fields. Minimal
|
||||
change: keep `dubError` (= `reason`) and add an optional
|
||||
`dubFailure: { reason, errorClass, stage, hint, docsTopic, diagnostic } | null`.
|
||||
`dubError` is never set to an empty string.
|
||||
@@ -0,0 +1,148 @@
|
||||
# Implementation Plan: Pipeline Error Transparency
|
||||
|
||||
**Branch**: `001-pipeline-error-transparency` | **Date**: 2026-05-29 | **Spec**: [spec.md](./spec.md)
|
||||
|
||||
**Input**: Feature specification from `specs/001-pipeline-error-transparency/spec.md` (plan-04 / #131; children #122, #63)
|
||||
|
||||
## Summary
|
||||
|
||||
Make every pipeline failure (dub/extract/ingest) self-describing: a non-empty,
|
||||
specific reason in the UI (error class + actionable hint + docs deeplink when
|
||||
known), a full traceback with stage/context in the backend log for every
|
||||
failure path (including failures before the ingest stage), and a copyable,
|
||||
sanitized diagnostic block. Achieved with one shared failure-event builder on
|
||||
the backend, hardening of the three known emit/swallow sites, and a richer
|
||||
frontend error renderer. Additive SSE payload — no data/schema change.
|
||||
|
||||
## Technical Context
|
||||
|
||||
**Language/Version**: Python 3.11 (backend), TypeScript/React (frontend, Vite + bun)
|
||||
|
||||
**Primary Dependencies**: FastAPI + SSE (`backend/core/tasks.py` TaskManager), Zustand store (frontend); existing `errorDocsMap.ts` classifier and `backend/core/logging_filter.py` redactor — both reused, not rebuilt.
|
||||
|
||||
**Storage**: N/A — no persisted state changes. SSE payload gains additive fields only.
|
||||
|
||||
**Testing**: pytest (`tests/`, `backend/tests/`), node:test (frontend). Regression tests required per Constitution V (fail before, pass after).
|
||||
|
||||
**Target Platform**: macOS (AS+Intel), Windows x64, Linux (AppImage + deb) — identical default behavior.
|
||||
|
||||
**Project Type**: Web application (FastAPI backend + React/Tauri desktop frontend).
|
||||
|
||||
**Performance Goals**: No hot-path impact — code runs only on the failure path.
|
||||
|
||||
**Constraints**: Local-first (diagnostic shown to user, never transmitted); no secrets / home paths in logs or diagnostic; backward-compatible additive payload.
|
||||
|
||||
**Scale/Scope**: ~3 backend change sites + 1 new helper module + 1 frontend renderer + regression tests. Closes #131, #122, #63.
|
||||
|
||||
## Constitution Check
|
||||
|
||||
*GATE: passes — this feature is the direct implementation of Principles II and V.*
|
||||
|
||||
| Principle | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| I. Local-First Sovereignty | ✅ | Diagnostic block is rendered locally for the user to copy; nothing is transmitted. Redaction reuses `logging_filter` + adds `*TOKEN*/*KEY*/*SECRET*` env-value and home-path stripping. No new outbound calls. |
|
||||
| II. First-Run That Actually Works | ✅ (implements) | Replaces "unknown error" with a real cause + docs deeplink; this principle defines the feature. |
|
||||
| III. Cross-Platform Default Parity | ✅ | Failure handling is pure Python/JS with no OS branch; behavior identical on all three platforms. WAV-only path (no ffmpeg) covered by test matrix. |
|
||||
| IV. Backward-Compatible Evolution | ✅ | SSE error payload gains fields (`error_class`, `reason`, `hint`, `stage`, `diagnostic`) but keeps the existing `error`/`detail` keys, so older frontends still work. No DB/schema/engine change. |
|
||||
| V. Root-Cause Architecture & Regression Tests | ✅ (implements) | plan-04 cluster master; ships regression tests for the 3 Test-matrix triggers (fail before fix, pass after); enforces "visible errors" globally via the shared builder. |
|
||||
|
||||
No violations → Complexity Tracking omitted.
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Documentation (this feature)
|
||||
|
||||
```text
|
||||
specs/001-pipeline-error-transparency/
|
||||
├── plan.md # This file
|
||||
├── spec.md # Feature spec
|
||||
├── research.md # Phase 0 — decisions
|
||||
├── data-model.md # Phase 1 — FailureEvent + DiagnosticBlock shapes
|
||||
├── contracts/
|
||||
│ └── sse-error-event.md # SSE "error" event contract (additive)
|
||||
├── quickstart.md # How to verify the 3 Test-matrix triggers
|
||||
└── checklists/
|
||||
└── requirements.md # Spec quality checklist (done)
|
||||
```
|
||||
|
||||
### Source Code (repository root)
|
||||
|
||||
```text
|
||||
backend/
|
||||
├── core/
|
||||
│ ├── failure.py # NEW — build_failure_event() + sanitize() + diagnostic()
|
||||
│ ├── tasks.py # CHANGE — worker except (L127-138) uses build_failure_event
|
||||
│ └── logging_filter.py # REUSE/EXTEND — redaction helpers
|
||||
├── services/
|
||||
│ └── dub_pipeline.py # CHANGE — enrich download/extract error yields; surface
|
||||
│ # demucs/scene/thumbnail degradations as visible
|
||||
│ # "warning" events instead of silent logger.warning
|
||||
└── api/routers/
|
||||
├── dub_core.py # CHANGE — guard upload/ingest-url handlers so pre-task
|
||||
│ # failures emit a structured reason (not a bare 500)
|
||||
└── batch.py # CHANGE — batch error uses the shared builder
|
||||
|
||||
frontend/
|
||||
├── src/
|
||||
│ ├── api/dub.ts # CHANGE — parse structured error fields from SSE
|
||||
│ ├── store/dubSlice.ts # CHANGE — store structured failure (reason never empty)
|
||||
│ ├── utils/errorDocsMap.ts # REUSE (+extend taxonomy only if a new class is needed)
|
||||
│ └── components/ # CHANGE — render reason + hint + docs link + "Copy diagnostic"
|
||||
└── ...
|
||||
|
||||
tests/
|
||||
├── test_dub_error_transparency.py # NEW — 3 Test-matrix triggers + builder + sanitize
|
||||
└── backend/test_dub_pipeline_wav.py # EXTEND — WAV-only failure transparency
|
||||
```
|
||||
|
||||
**Structure Decision**: Existing web-app layout (backend/ + frontend/). The only
|
||||
new file is `backend/core/failure.py` — a single, well-bounded helper so the
|
||||
"non-empty reason + redaction + diagnostic" logic lives in one place and every
|
||||
emit site calls it, rather than duplicating fallback logic across tasks.py,
|
||||
dub_pipeline.py, dub_core.py, and batch.py.
|
||||
|
||||
## Approach (the three change points + the helper)
|
||||
|
||||
1. **`backend/core/failure.py` (new, the keystone).**
|
||||
- `build_failure_event(exc_or_msg, *, stage, context=None) -> dict` returns
|
||||
`{"type":"error","stage":stage,"error_class":<ExcType name or "Error">,
|
||||
"reason":<non-empty>,"hint":<one-liner or "">,"docs_topic":<key or "">,
|
||||
"detail":<sanitized full str>,"diagnostic":<sanitized block>}`.
|
||||
- **Non-empty guarantee**: `reason = str(exc).strip() or repr(exc).strip() or
|
||||
type(exc).__name__`. This is the core fix for empty/cryptic `str(e)`.
|
||||
- `classify(reason) -> (docs_topic, hint)`: small backend mirror of the
|
||||
frontend taxonomy keys so the server log + diagnostic name the class too;
|
||||
frontend remains the source of truth for the actual docs URL.
|
||||
- `sanitize(text)`: applies the existing HF-token regex from
|
||||
`logging_filter`, plus env-value redaction for names matching
|
||||
`*TOKEN*|*KEY*|*SECRET*`, plus home-dir → `~`.
|
||||
- `diagnostic(event)`: composes the failure + a sanitized env summary
|
||||
(reuse the bug-reporter capture: OS/CPU/GPU/versions; never audio/secrets).
|
||||
|
||||
2. **`backend/core/tasks.py` worker except (L127-138).** Keep
|
||||
`logger.exception(...)` (full traceback). Replace the bare
|
||||
`{'type':'error','error':str(e)}` push with
|
||||
`build_failure_event(e, stage="task", context={"task_id":task_id})`. Keep the
|
||||
legacy `error` key populated (= `reason`) for backward compat.
|
||||
|
||||
3. **`backend/services/dub_pipeline.py`.** Route the download/extract `except`
|
||||
yields through `build_failure_event` (adds `error_class`/`hint`). Convert the
|
||||
silent demucs/scene/thumbnail `logger.warning`-only fallbacks (L554/571/585)
|
||||
to *also* yield a non-fatal `prep_event("warning", stage=..., reason=...)` so
|
||||
the degradation is visible without failing the job. No bare `except: pass`.
|
||||
|
||||
4. **`backend/api/routers/dub_core.py` (and `batch.py`).** Wrap the pre-task
|
||||
work (file write, URL/preflight, arg build) so a failure there emits the same
|
||||
structured reason via SSE / job error instead of an opaque 500 or a truncated
|
||||
`str(e)` — this is the #122 "exception before ingest_pipeline" path.
|
||||
|
||||
5. **Frontend.** `api/dub.ts` parses the new fields; `dubSlice` stores a
|
||||
structured failure (reason guaranteed non-empty, falling back to a generic
|
||||
"Something failed — see logs" only if the backend somehow sent nothing). The
|
||||
error UI shows reason + hint, the `errorDocsMap.classifyError` deeplink, and a
|
||||
"Copy diagnostic" button bound to the `diagnostic` field.
|
||||
|
||||
## Phase 0 / Phase 1 artifacts
|
||||
|
||||
research.md, data-model.md, contracts/sse-error-event.md, quickstart.md generated
|
||||
alongside this plan (decisions are settled; few unknowns). See those files.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Quickstart: Verifying Pipeline Error Transparency
|
||||
|
||||
How to confirm the three Test-matrix triggers from #131. Each must show a
|
||||
specific cause in the UI **and** a full traceback in the backend log.
|
||||
|
||||
## Automated (the regression gate)
|
||||
|
||||
```bash
|
||||
uv run pytest tests/test_dub_error_transparency.py -q
|
||||
uv run pytest tests/backend/test_dub_pipeline_wav.py -q # WAV-only failure path
|
||||
```
|
||||
|
||||
These assert, for each trigger: the emitted SSE event is `type:"error"` with a
|
||||
non-empty `reason` + `error_class` + `stage`, the backend log (captured via
|
||||
`caplog`) contains a traceback, and `detail`/`diagnostic` contain no secret-like
|
||||
values or absolute home paths. They are written to FAIL on `main` (where the
|
||||
event is a bare/empty `str(e)`) and PASS after the fix.
|
||||
|
||||
## Manual (in-app)
|
||||
|
||||
1. **Extract fails (bad input)** — start a dub on a corrupt or audio-less file.
|
||||
- Expect: UI shows e.g. "extract — FileNotFoundError: … " + a hint (not
|
||||
"unknown error"); backend log (LogsFooter → System) shows the traceback.
|
||||
2. **Remote ingest fails** — dub-by-URL with an unreachable/invalid URL.
|
||||
- Expect: specific cause (e.g. yt-dlp error) + hint; traceback in log.
|
||||
3. **WAV-only input fails** — feed a malformed WAV (no ffmpeg/extract path).
|
||||
- Expect: specific cause; traceback in log.
|
||||
4. **Copy diagnostic** — on any failure, click "Copy diagnostic"; paste it.
|
||||
- Expect: cause + stage + env summary; NO tokens/keys/secrets; home dir shown
|
||||
as `~`.
|
||||
|
||||
## Cross-platform check (Constitution III)
|
||||
|
||||
Run trigger 3 (WAV-only, no ffmpeg) on macOS, Windows, and Linux; the UI message
|
||||
and log behavior must be identical. The `smoke-matrix` CI job exercises the
|
||||
in-process path on all three.
|
||||
@@ -0,0 +1,79 @@
|
||||
# Phase 0 Research: Pipeline Error Transparency
|
||||
|
||||
The spec was seeded from #131 with a complete defect + fix-sequence, and the
|
||||
codebase was mapped before planning. Few open unknowns; the decisions below
|
||||
resolve them.
|
||||
|
||||
## Decision 1 — One shared failure-event builder vs. fixing each site inline
|
||||
|
||||
**Decision**: Add `backend/core/failure.py` with `build_failure_event()` and route
|
||||
every emit site (tasks.py, dub_pipeline.py, dub_core.py, batch.py) through it.
|
||||
|
||||
**Rationale**: The defect is duplicated fallback logic — each site does its own
|
||||
`str(e)` (some `[:300]`, some `[:500]`, some empty). Centralizing the non-empty
|
||||
guarantee + redaction + classification means a single tested code path governs
|
||||
"what a failure looks like," satisfying Constitution V's "errors MUST be visible"
|
||||
globally rather than per-site.
|
||||
|
||||
**Alternatives considered**: Patch each `except` independently — rejected: leaves
|
||||
the next new emit site free to reintroduce a bare `str(e)`.
|
||||
|
||||
## Decision 2 — Where error classification / docs mapping lives
|
||||
|
||||
**Decision**: Frontend `errorDocsMap.ts` stays the source of truth for the docs
|
||||
URL (it already has the 5-class taxonomy + `classifyError`). Backend emits
|
||||
`error_class` + `reason` + a backend-side `docs_topic` *key* (not URL) for the
|
||||
log/diagnostic only.
|
||||
|
||||
**Rationale**: Avoids duplicating the URL table on two sides; the frontend
|
||||
already renders deeplinks. Backend only needs the class *name* for its log line
|
||||
and diagnostic block.
|
||||
|
||||
**Alternatives considered**: Full backend URL table — rejected (duplication,
|
||||
drift risk). Pure-frontend classification with backend sending only `str(e)` —
|
||||
rejected: the backend log/diagnostic should also name the class.
|
||||
|
||||
## Decision 3 — Non-fatal degradations (demucs/scene/thumbnail)
|
||||
|
||||
**Decision**: Keep them non-fatal (job continues with the fallback) but emit a
|
||||
visible `prep_event("warning", stage, reason)` instead of a silent
|
||||
`logger.warning`-only path.
|
||||
|
||||
**Rationale**: These are legitimate graceful degradations, not job failures, so
|
||||
they must not fail the job (backward-compat with current behavior). But "silent"
|
||||
violates the transparency principle — the user should see that demucs was
|
||||
skipped. A non-fatal warning event threads that needle.
|
||||
|
||||
**Alternatives considered**: Promote them to fatal errors — rejected: changes
|
||||
current successful-with-fallback behavior, would regress real jobs.
|
||||
|
||||
## Decision 4 — Diagnostic block delivery
|
||||
|
||||
**Decision**: Include a sanitized `diagnostic` string inside the SSE error event
|
||||
payload; the frontend offers a "Copy diagnostic" button. No new endpoint.
|
||||
|
||||
**Rationale**: The failure event is already flowing to the client; attaching the
|
||||
diagnostic avoids a second round-trip and a stateful "last failure" store. The
|
||||
block is built and sanitized server-side where the env info lives.
|
||||
|
||||
**Alternatives considered**: A `/diag/last-failure` endpoint — rejected: adds
|
||||
state + an extra call for no benefit; the SSE event is the natural carrier.
|
||||
|
||||
## Decision 5 — Redaction reuse
|
||||
|
||||
**Decision**: Reuse `backend/core/logging_filter.py`'s HF-token regex; add
|
||||
`*TOKEN*|*KEY*|*SECRET*` env-value redaction and home-dir → `~` in
|
||||
`failure.sanitize()`. Reuse the opt-in bug-reporter's environment capture for
|
||||
the diagnostic summary.
|
||||
|
||||
**Rationale**: Single redaction definition; consistent with Constitution I and
|
||||
the bug-reporter's existing privacy contract.
|
||||
|
||||
## Open risk — #122 "no logs at all"
|
||||
|
||||
The deepest symptom (#122: no backend log lines, instrumented code never
|
||||
reached) may be a *logging-configuration* gap rather than a swallow. Mitigation
|
||||
in-scope: (a) guard the pre-task handlers in `dub_core.py` so failures there are
|
||||
caught + logged + surfaced; (b) verify `logger.exception` in tasks.py reaches the
|
||||
in-app log buffer that `LogsFooter` reads. The specific root cause of #122's
|
||||
underlying error, once visible, routes to plan-01/02/03 per spec scope.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Feature Specification: Pipeline Error Transparency
|
||||
|
||||
**Feature Branch**: `001-pipeline-error-transparency`
|
||||
|
||||
**Created**: 2026-05-29
|
||||
|
||||
**Status**: Draft
|
||||
|
||||
**Input**: plan-04 (#131); children #122, #63. "No more silent 'unknown error'."
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
### User Story 1 - See why a job actually failed (Priority: P1)
|
||||
|
||||
A user runs a dubbing or extract job and it fails. Today they see only
|
||||
"extract: unknown error" with no indication of cause. After this change, the
|
||||
failure message names the real cause in plain language and tells them what to
|
||||
do next (e.g., "Source media has no audio track — pick a file with audio", or a
|
||||
link to the matching troubleshooting doc).
|
||||
|
||||
**Why this priority**: Until the failure reason is visible to the user, every
|
||||
downstream bug is un-triageable and every job failure is a dead end. This is the
|
||||
single highest-leverage slice — it delivers value on its own.
|
||||
|
||||
**Independent Test**: Trigger any pipeline failure (e.g., feed a corrupt or
|
||||
audio-less file) and confirm the UI shows a specific, human-readable cause
|
||||
instead of "unknown error".
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** a media file that fails to extract, **When** the user starts the
|
||||
job, **Then** the UI shows a specific cause (error type + one-line "what to
|
||||
do") and never a bare "unknown error".
|
||||
2. **Given** a failure that maps to a known troubleshooting topic, **When** the
|
||||
error is shown, **Then** the user is offered the matching docs deeplink.
|
||||
3. **Given** any failure, **When** it surfaces, **Then** the reason text is
|
||||
non-empty.
|
||||
|
||||
---
|
||||
|
||||
### User Story 2 - Full failure detail in the logs (Priority: P2)
|
||||
|
||||
A user (or the maintainer helping them) opens the backend logs after a failure
|
||||
and finds the complete exception — type, message, and stack trace — with enough
|
||||
context (which stage, which input) to diagnose it. Today the logs can be
|
||||
completely silent: in #122 the instrumented code is never even reached.
|
||||
|
||||
**Why this priority**: Self-describing logs turn a low-information report into an
|
||||
actionable one and make the maintainer's triage possible without a live repro.
|
||||
|
||||
**Independent Test**: Force a failure, then inspect the backend log and confirm
|
||||
a full traceback with stage/context is present for that job.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** a pipeline failure at any stage, **When** it occurs, **Then** the
|
||||
backend log contains the real exception with a full stack trace and the
|
||||
failing stage/context.
|
||||
2. **Given** an exception thrown before the main ingest stage is reached,
|
||||
**When** it occurs, **Then** it is still logged (no swallowed/short-circuited
|
||||
failures).
|
||||
|
||||
---
|
||||
|
||||
### User Story 3 - Copyable diagnostic block (Priority: P3)
|
||||
|
||||
When a job fails, the user can copy a self-contained diagnostic block (cause,
|
||||
stage, sanitized environment summary) to paste into a bug report, so even a
|
||||
terse report is answerable.
|
||||
|
||||
**Why this priority**: A prevention net that shrinks low-information reports
|
||||
(#63-style). Builds on US1/US2; valuable but not required for the core fix.
|
||||
|
||||
**Independent Test**: Trigger a failure, use the "copy diagnostic" affordance,
|
||||
and confirm the copied text contains the cause, stage, and a sanitized
|
||||
environment summary — and nothing sensitive.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** a failed job, **When** the user copies the diagnostic block,
|
||||
**Then** it contains the failure cause, stage, and a sanitized environment
|
||||
summary.
|
||||
2. **Given** the diagnostic block, **When** it is generated, **Then** it
|
||||
contains no secrets (no `*TOKEN*`/`*KEY*`/`*SECRET*` values) and no absolute
|
||||
home paths (home dir shown as `~`).
|
||||
|
||||
### Edge Cases
|
||||
|
||||
- **Failure before any stage runs** (setup/validation): still produces a
|
||||
specific reason and a log entry — this is the #122 case.
|
||||
- **Non-exception failures** (a stage exits non-zero, a subprocess like ffmpeg
|
||||
fails, a None/empty result): surfaced as a specific reason, not "unknown".
|
||||
- **WAV-only input** that fails without touching the ffmpeg/extract path: same
|
||||
transparency guarantees apply.
|
||||
- **Failure cause has no matching docs topic**: show the specific cause without
|
||||
a deeplink rather than suppressing the message.
|
||||
- **Very long or multi-line underlying error**: UI shows a concise summary;
|
||||
full detail remains in logs and the diagnostic block.
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
- **FR-001**: The system MUST surface a specific, human-readable failure reason
|
||||
for every pipeline failure (dub, extract, ingest). A bare "unknown error" MUST
|
||||
never be the only thing shown.
|
||||
- **FR-002**: Every failure reason presented to the user MUST be non-empty and
|
||||
MUST include the underlying error type plus a one-line actionable hint.
|
||||
- **FR-003**: The system MUST log the real exception — type, message, full stack
|
||||
trace — together with the failing stage and relevant input context, for every
|
||||
failure path, including failures that occur before the main ingest stage.
|
||||
- **FR-004**: The system MUST NOT swallow or short-circuit pipeline exceptions
|
||||
without logging them.
|
||||
- **FR-005**: When a failure maps to a known troubleshooting topic, the system
|
||||
MUST offer the corresponding docs deeplink (reusing the existing error→docs
|
||||
mechanism).
|
||||
- **FR-006**: The system MUST emit a structured failure event to the frontend so
|
||||
the frontend always receives a non-empty machine-readable reason (not just a
|
||||
display string).
|
||||
- **FR-007**: The system MUST provide a copyable diagnostic block containing the
|
||||
cause, failing stage, and a sanitized environment summary.
|
||||
- **FR-008**: The diagnostic block and any logged context MUST redact secrets
|
||||
(values of env vars matching `*TOKEN*`/`*KEY*`/`*SECRET*`) and MUST strip
|
||||
absolute home paths (render the home directory as `~`).
|
||||
- **FR-009**: Behavior MUST be identical by default on macOS, Windows, and Linux
|
||||
(no platform-divergent default error handling).
|
||||
- **FR-010**: The change MUST be backward-compatible — no change to project data,
|
||||
no schema migration, and existing successful jobs behave exactly as before.
|
||||
|
||||
### Key Entities
|
||||
|
||||
- **Failure event**: a structured record of a pipeline failure — cause/error
|
||||
type, human-readable reason, actionable hint, optional docs topic, failing
|
||||
stage, and timestamp. Delivered to the frontend and used to render the message
|
||||
and the diagnostic block.
|
||||
- **Diagnostic block**: a user-copyable, sanitized text rendering of a failure
|
||||
event plus environment summary, intended for pasting into a bug report.
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
|
||||
### Measurable Outcomes
|
||||
|
||||
- **SC-001**: 100% of induced pipeline failures (bad input, failed remote
|
||||
ingest, WAV-only failure) display a specific cause in the UI — 0% show a bare
|
||||
"unknown error".
|
||||
- **SC-002**: 100% of induced pipeline failures produce a backend log entry
|
||||
containing a full stack trace and the failing stage.
|
||||
- **SC-003**: Every failure delivered to the frontend carries a non-empty reason
|
||||
(no empty/placeholder reason in any tested path).
|
||||
- **SC-004**: A user can produce a copyable diagnostic block for a failed job in
|
||||
one action, and that block contains no secrets or absolute home paths in any
|
||||
tested case.
|
||||
- **SC-005**: The three Test-matrix triggers from #131 (extract-fails,
|
||||
remote-ingest-fails, WAV-only-fails) all satisfy SC-001 and SC-002 under
|
||||
automated test.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- An error→docs deeplink mechanism already exists (shipped on the onboarding /
|
||||
bug-report work) and can be reused for FR-005; this feature wires causes to it
|
||||
rather than building it anew.
|
||||
- The frontend already has a place to render job failures (toast/status) that
|
||||
can be extended to show a specific reason and a "copy diagnostic" affordance.
|
||||
- "Sanitized environment summary" reuses the redaction rules already defined for
|
||||
the opt-in bug reporter (OS/CPU/GPU/versions; no audio, no secrets, no home
|
||||
paths).
|
||||
- Identifying and fixing the *root cause* of any specific failure once it is
|
||||
visible is out of scope here and routes to plan-01/02/03 (#128–130).
|
||||
@@ -0,0 +1,69 @@
|
||||
# Tasks: Pipeline Error Transparency
|
||||
|
||||
**Feature**: plan-04 (#131) | **Branch**: `001-pipeline-error-transparency`
|
||||
**Spec**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md)
|
||||
**TDD**: strict red-green — every test task is written and confirmed FAILING before its implementation task.
|
||||
|
||||
## Phase 1: Setup
|
||||
|
||||
- [ ] T001 Confirm no new dependencies are required (psutil, torch, platform already pinned); add the new test module path `tests/test_dub_error_transparency.py` to the pytest collection by creating an empty stub so CI discovers it.
|
||||
|
||||
## Phase 2: Foundational — shared failure helper (BLOCKS all stories)
|
||||
|
||||
- [ ] T002 [P] Write failing unit tests for the failure helper in `tests/test_failure_helper.py`: (a) `build_failure_event` produces a non-empty `reason` when `str(exc)` is empty (e.g. a custom `Exception()` with no message) — falls back to `repr` then type name; (b) event always contains `type`, `reason`, `error_class`, `stage`, and a compat `error` key equal to `reason`; (c) `sanitize()` redacts `hf_…` tokens, values of env vars matching `*TOKEN*|*KEY*|*SECRET*`, and rewrites the home dir to `~`; (d) `diagnostic()` output contains stage + error_class + an env summary and NONE of the redacted material. Run and confirm RED (module does not exist yet).
|
||||
- [ ] T003 Implement `backend/core/failure.py`: `build_failure_event(exc_or_msg, *, stage, context=None)`, `sanitize(text)` (reuse `_HF_TOKEN_RE` from `backend/core/logging_filter.py`), `classify(reason) -> (docs_topic, hint)` (mirror the 5 `errorDocsMap` keys), and `diagnostic(event)` (reuse the opt-in bug-reporter env capture). Make T002 GREEN.
|
||||
|
||||
## Phase 3: User Story 1 — See why a job actually failed (P1) 🎯 MVP
|
||||
|
||||
**Goal**: UI shows a specific, non-empty cause (never "unknown error") + actionable hint + docs deeplink.
|
||||
**Independent test**: induce any pipeline failure → UI shows specific cause, not "unknown error".
|
||||
|
||||
- [ ] T004 [P] [US1] Write failing test in `tests/test_dub_error_transparency.py`: extract-fails-on-bad-input → the SSE/event payload emitted by the worker is `type:"error"` with non-empty `reason` + `error_class` + `stage="extract"`. Confirm RED on current `str(e)`-only behavior.
|
||||
- [ ] T005 [P] [US1] Write failing test (same file): remote/url ingest failure → structured non-empty `reason` + `stage`. Confirm RED.
|
||||
- [ ] T006 [US1] Harden `backend/core/tasks.py` worker `except` (L127-138): keep `logger.exception`, replace the bare `{'type':'error','error':str(e)}` push with `build_failure_event(e, stage="task", context={"task_id": task_id})` (preserve the `error` key).
|
||||
- [ ] T007 [US1] Route the `download`/`extract` `except` yields in `backend/services/dub_pipeline.py` (L418, L461) through `build_failure_event` so they carry `error_class` + `hint` and a non-empty `reason`.
|
||||
- [ ] T008 [US1] Guard pre-task work in `backend/api/routers/dub_core.py` (upload + ingest-url handlers) and `backend/api/routers/batch.py` (L82) so a failure before the task starts emits a structured reason (the #122 path), not a bare 500 / truncated `str(e)`. Make T004/T005 GREEN.
|
||||
- [ ] T009 [P] [US1] Frontend: parse the new fields in `frontend/src/api/dub.ts`; store structured failure in `frontend/src/store/dubSlice.ts` (`dubError` = `reason`, never empty; add optional `dubFailure`).
|
||||
- [ ] T010 [US1] Frontend: render `reason` + `hint` + the `errorDocsMap.classifyError` deeplink in the dub failure UI component (consumer of `dubError`). Extend `errorDocsMap` taxonomy ONLY if a needed class is missing.
|
||||
|
||||
## Phase 4: User Story 2 — Full failure detail in the logs (P2)
|
||||
|
||||
**Goal**: every failure path logs the real exception with full traceback + stage/context, including failures before the ingest stage and non-fatal degradations.
|
||||
**Independent test**: force a failure → backend log has a traceback + stage for that job.
|
||||
|
||||
- [ ] T011 [P] [US2] Write failing test in `tests/test_dub_error_transparency.py` using `caplog`: each of the 3 triggers logs a full traceback (via `logger.exception`) including the `stage`. Confirm RED where logging is currently silent/absent.
|
||||
- [ ] T012 [US2] Ensure every emit site calls `logger.exception(...)` (or `logger.error(..., exc_info=True)`) with stage + context before/at the structured event — audit tasks.py, dub_pipeline.py, dub_core.py, batch.py for any `except` that logs nothing or `except: pass`; fix each. Make T011 GREEN.
|
||||
- [ ] T013 [US2] Convert the silent demucs/scene/thumbnail fallbacks in `backend/services/dub_pipeline.py` (L554/571/585) to ALSO yield a non-fatal `prep_event("warning", stage=…, reason=…)` (job still continues) so degradations are visible. Add a test asserting `type:"warning"` does not set the terminal job error.
|
||||
|
||||
## Phase 5: User Story 3 — Copyable diagnostic block (P3)
|
||||
|
||||
**Goal**: one-action copyable, sanitized diagnostic for a failed job.
|
||||
**Independent test**: trigger a failure → copy diagnostic → contains cause+stage+env, no secrets/home paths.
|
||||
|
||||
- [ ] T014 [P] [US3] Write failing test (`tests/test_dub_error_transparency.py`): the `diagnostic` field on an error event contains stage + error_class + env summary and NONE of: `hf_…` tokens, `*TOKEN*/*KEY*/*SECRET*` values, absolute home path. Confirm RED.
|
||||
- [ ] T015 [US3] Populate the `diagnostic` field in `build_failure_event` for fatal errors (via `failure.diagnostic`). Make T014 GREEN.
|
||||
- [ ] T016 [US3] Frontend: add a "Copy diagnostic" button in the failure UI bound to `dubFailure.diagnostic` (clipboard copy). Add a node:test for the parse→store path producing a non-empty diagnostic.
|
||||
|
||||
## Phase 6: Polish & Cross-Cutting
|
||||
|
||||
- [ ] T017 [P] Extend `tests/backend/test_dub_pipeline_wav.py` with the WAV-only failure transparency case (no ffmpeg path) — the 3rd Test-matrix trigger. Confirm it asserts SC-001 + SC-002.
|
||||
- [ ] T018 [P] Update `docs/` troubleshooting: note that failures now show a specific cause + copyable diagnostic (ties error→docs deeplink).
|
||||
- [ ] T019 Run full suite (`uv run pytest tests/ backend/tests/ -q` + frontend `bun test`) and the `quickstart.md` manual triggers; confirm all green and 0 "unknown error" outputs across the 3 triggers.
|
||||
- [ ] T020 Self-review against Constitution I/III/IV/V (no outbound calls, identical cross-platform default, no data/schema change, regression tests fail-before/pass-after) before opening the PR.
|
||||
|
||||
## Dependencies & order
|
||||
|
||||
- **Phase 2 (T002-T003) blocks everything** — the helper is the keystone.
|
||||
- US1 (P1) is the MVP and can ship alone. US2 builds on US1's emit sites. US3 builds on the helper's `diagnostic()`.
|
||||
- Within a story: test task(s) first (RED), then implementation (GREEN).
|
||||
|
||||
## Parallel opportunities
|
||||
|
||||
- T002 (helper tests) ∥ nothing (foundational, first).
|
||||
- T004, T005 (US1 backend tests) ∥ T009 (frontend parse) once T003 lands.
|
||||
- T011 (US2 log test) ∥ T014 (US3 diagnostic test) — different assertions, same file (coordinate edits).
|
||||
- T017, T018 polish ∥ each other.
|
||||
|
||||
## MVP scope
|
||||
|
||||
**User Story 1 only** (T001-T010): replaces "unknown error" with a specific cause in the UI + logs the real exception at the worker boundary. Delivers the core of #131/#122 on its own.
|
||||
@@ -0,0 +1,181 @@
|
||||
"""plan-04 (#131) — pipeline error transparency regression tests.
|
||||
|
||||
Test-matrix from the issue (every failure → specific UI cause + logged
|
||||
traceback). Written RED before the emit-site changes. Fixture-free: the worker
|
||||
path is pure-Python; the dub-pipeline paths force failure via a missing file and
|
||||
a monkeypatched downloader, so they don't need real media/network.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
|
||||
os.environ.setdefault("OMNIVOICE_DISABLE_FILE_LOG", "1")
|
||||
|
||||
import pytest
|
||||
|
||||
from core.db import init_db
|
||||
from core.tasks import TaskManager
|
||||
from services import dub_pipeline as dp
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _db():
|
||||
init_db()
|
||||
yield
|
||||
|
||||
|
||||
def _error_events(history) -> list[dict]:
|
||||
out = []
|
||||
for e in history:
|
||||
if e and isinstance(e, str) and e.startswith("data:") and '"type": "error"' in e:
|
||||
out.append(json.loads(e[len("data: "):]))
|
||||
return out
|
||||
|
||||
|
||||
async def _empty_boom(*a, **k):
|
||||
"""Async-generator task that raises with an EMPTY message (the cryptic case)."""
|
||||
if False:
|
||||
yield
|
||||
raise ValueError("")
|
||||
|
||||
|
||||
async def _runtime_boom(*a, **k):
|
||||
if False:
|
||||
yield
|
||||
raise RuntimeError("ffprobe blew up")
|
||||
|
||||
|
||||
async def _drain_failing_task(boom):
|
||||
"""Run a failing task and return all SSE events the worker emits.
|
||||
|
||||
Race-free: the listener is registered BEFORE the worker is started, so no
|
||||
event (including the terminal error + EOF) can be missed, and we drain to
|
||||
the EOF sentinel instead of cancelling the worker mid-push.
|
||||
"""
|
||||
tm = TaskManager()
|
||||
tid = f"t_{uuid.uuid4().hex[:8]}"
|
||||
await tm.add_task(tid, "prep", boom)
|
||||
q: asyncio.Queue = asyncio.Queue()
|
||||
await tm.add_listener(tid, q)
|
||||
worker = asyncio.create_task(tm.worker())
|
||||
events: list = []
|
||||
try:
|
||||
while True:
|
||||
ev = await asyncio.wait_for(q.get(), timeout=10)
|
||||
if ev is None: # EOF sentinel pushed in the worker's finally
|
||||
break
|
||||
events.append(ev)
|
||||
finally:
|
||||
worker.cancel()
|
||||
try:
|
||||
await worker
|
||||
except asyncio.CancelledError:
|
||||
# Expected: we cancel the worker loop to tear it down after draining.
|
||||
pass
|
||||
return events
|
||||
|
||||
|
||||
# ── US1 + US2: worker failure path (the #122 "unknown error" / silent log) ──
|
||||
|
||||
def test_worker_failure_emits_structured_nonempty_reason():
|
||||
"""A task that raises with an EMPTY message must still surface a specific,
|
||||
non-empty reason + error_class + stage — not a bare/empty string — and log
|
||||
the real exception with a traceback (US2)."""
|
||||
|
||||
# Capture directly on the task logger — robust against the app's logging
|
||||
# config (propagate flags) and asyncio task boundaries.
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def emit(self, record): # noqa: D401
|
||||
records.append(record)
|
||||
|
||||
handler = _Capture(level=logging.ERROR)
|
||||
tlog = logging.getLogger("omnivoice.tasks")
|
||||
# The app may have run dictConfig(disable_existing_loggers=True) on import,
|
||||
# which leaves this logger disabled in the test process. Force it live so we
|
||||
# can assert the worker actually logs the traceback.
|
||||
tlog.disabled = False
|
||||
tlog.setLevel(logging.DEBUG)
|
||||
tlog.addHandler(handler)
|
||||
|
||||
try:
|
||||
events = asyncio.run(_drain_failing_task(_empty_boom))
|
||||
finally:
|
||||
tlog.removeHandler(handler)
|
||||
|
||||
errs = _error_events(events)
|
||||
assert errs, "worker must push a structured error event"
|
||||
evt = errs[-1]
|
||||
assert evt["reason"], "reason must be non-empty even for an empty-message exception"
|
||||
assert evt["error_class"] == "ValueError"
|
||||
assert evt["stage"] == "task"
|
||||
# US2: the real exception was logged with a traceback
|
||||
assert any(r.exc_info for r in records), "expected a logged traceback"
|
||||
|
||||
|
||||
# ── US1: extract-fails-on-bad-input (Test-matrix #1) ────────────────────────
|
||||
|
||||
def test_extract_failure_yields_structured_error(tmp_path):
|
||||
async def _run():
|
||||
events = []
|
||||
src = {"path": str(tmp_path / "does_not_exist.mp4")}
|
||||
async for ev in dp.ingest_pipeline("j_ext", str(tmp_path), src):
|
||||
events.append(ev)
|
||||
return events
|
||||
|
||||
errs = _error_events(asyncio.run(_run()))
|
||||
assert errs, "a failed extract must yield an error event"
|
||||
evt = errs[-1]
|
||||
assert evt["stage"] in ("extract", "ingest")
|
||||
assert evt["reason"]
|
||||
assert evt["error_class"] # structured, not a bare string
|
||||
|
||||
|
||||
# ── US1: remote/url ingest failure (Test-matrix #2) ─────────────────────────
|
||||
|
||||
def test_url_ingest_failure_yields_structured_error(tmp_path, monkeypatch):
|
||||
def _boom(*a, **k):
|
||||
raise RuntimeError("yt-dlp: Video unavailable")
|
||||
|
||||
monkeypatch.setattr(dp, "yt_download_sync", _boom)
|
||||
|
||||
async def _run():
|
||||
events = []
|
||||
src = {"kind": "url", "url": "https://example.com/watch?v=x"}
|
||||
async for ev in dp.ingest_pipeline("j_url", str(tmp_path), src):
|
||||
events.append(ev)
|
||||
return events
|
||||
|
||||
errs = _error_events(asyncio.run(_run()))
|
||||
assert errs, "a failed url ingest must yield an error event"
|
||||
evt = errs[-1]
|
||||
assert evt["stage"] in ("download", "ingest")
|
||||
assert evt["error_class"] == "RuntimeError"
|
||||
assert "unavailable" in evt["reason"].lower()
|
||||
|
||||
|
||||
# ── US3: fatal error event carries a sanitized diagnostic block ─────────────
|
||||
|
||||
def test_fatal_error_event_carries_sanitized_diagnostic():
|
||||
leaked = "hf_" + "C" * 36
|
||||
prev = os.environ.get("HF_TOKEN")
|
||||
os.environ["HF_TOKEN"] = leaked
|
||||
try:
|
||||
events = asyncio.run(_drain_failing_task(_runtime_boom))
|
||||
finally:
|
||||
if prev is None:
|
||||
os.environ.pop("HF_TOKEN", None)
|
||||
else:
|
||||
os.environ["HF_TOKEN"] = prev
|
||||
|
||||
errs = _error_events(events)
|
||||
assert errs, "fatal error must emit a structured event"
|
||||
evt = errs[-1]
|
||||
assert evt.get("diagnostic"), "fatal error must carry a copyable diagnostic block"
|
||||
assert "task" in evt["diagnostic"]
|
||||
assert leaked not in evt["diagnostic"], "diagnostic must not leak the HF token"
|
||||
@@ -0,0 +1,92 @@
|
||||
"""plan-04 (#131) — unit tests for the shared failure helper.
|
||||
|
||||
These are the foundational (Phase 2) tests: the non-empty-reason guarantee,
|
||||
redaction, classification, and the diagnostic block. Written RED before
|
||||
`core/failure.py` exists.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
from core import failure
|
||||
|
||||
|
||||
# ── Non-empty reason guarantee (the core fix) ───────────────────────────────
|
||||
|
||||
def test_reason_non_empty_when_exception_message_empty():
|
||||
evt = failure.build_failure(ValueError(""), stage="extract")
|
||||
assert evt["reason"], "reason must never be empty"
|
||||
assert evt["error_class"] == "ValueError"
|
||||
assert evt["stage"] == "extract"
|
||||
# Backward-compat mirror used by older frontends.
|
||||
assert evt["error"] == evt["reason"]
|
||||
|
||||
|
||||
def test_reason_uses_message_when_present():
|
||||
evt = failure.build_failure(FileNotFoundError("no such file: clip.mp4"), stage="extract")
|
||||
assert "no such file" in evt["reason"]
|
||||
assert evt["error_class"] == "FileNotFoundError"
|
||||
|
||||
|
||||
def test_accepts_plain_string_message():
|
||||
evt = failure.build_failure("preflight: ffmpeg not found", stage="preflight")
|
||||
assert evt["reason"] == "preflight: ffmpeg not found"
|
||||
assert evt["stage"] == "preflight"
|
||||
|
||||
|
||||
def test_build_failure_event_carries_type():
|
||||
evt = failure.build_failure_event(RuntimeError("boom"), stage="task")
|
||||
assert evt["type"] == "error"
|
||||
assert evt["reason"] == "boom"
|
||||
# warning variant for non-fatal degradations
|
||||
warn = failure.build_failure_event(RuntimeError("demucs down"), stage="demucs", event_type="warning")
|
||||
assert warn["type"] == "warning"
|
||||
|
||||
|
||||
# ── Redaction (Constitution I) ──────────────────────────────────────────────
|
||||
|
||||
def test_sanitize_redacts_hf_token():
|
||||
tok = "hf_" + "A" * 36
|
||||
out = failure.sanitize(f"auth failed using {tok} on download")
|
||||
assert tok not in out
|
||||
|
||||
|
||||
def test_sanitize_redacts_secret_env_values(monkeypatch):
|
||||
monkeypatch.setenv("OPENAI_API_KEY", "sk-supersecretvalue123456")
|
||||
out = failure.sanitize("request died: sk-supersecretvalue123456 rejected")
|
||||
assert "sk-supersecretvalue123456" not in out
|
||||
|
||||
|
||||
def test_sanitize_strips_home_path():
|
||||
home = str(Path.home())
|
||||
out = failure.sanitize(f"could not open {home}/Movies/clip.mp4")
|
||||
assert home not in out
|
||||
assert "~" in out
|
||||
|
||||
|
||||
# ── Diagnostic block (US3) ──────────────────────────────────────────────────
|
||||
|
||||
def test_diagnostic_has_context_and_no_secrets(monkeypatch):
|
||||
leaked = "hf_" + "B" * 36
|
||||
monkeypatch.setenv("HF_TOKEN", leaked)
|
||||
evt = failure.build_failure(RuntimeError("ffprobe exploded"), stage="extract")
|
||||
diag = evt["diagnostic"]
|
||||
assert "extract" in diag
|
||||
assert "RuntimeError" in diag
|
||||
assert leaked not in diag
|
||||
# carries an environment summary
|
||||
assert ("OS" in diag) or ("Python" in diag)
|
||||
|
||||
|
||||
# ── Classification → docs topic + hint (US1, FR-005) ────────────────────────
|
||||
|
||||
def test_docs_topic_and_hint_for_known_class():
|
||||
evt = failure.build_failure(
|
||||
ModuleNotFoundError("No module named 'pkg_resources'"), stage="task"
|
||||
)
|
||||
assert evt["docs_topic"] == "PKG_RESOURCES_MISSING"
|
||||
assert evt["hint"], "known classes must carry an actionable hint"
|
||||
|
||||
|
||||
def test_unknown_cause_has_empty_topic_but_still_non_empty_reason():
|
||||
evt = failure.build_failure(RuntimeError("totally novel failure xyz"), stage="task")
|
||||
assert evt["docs_topic"] == ""
|
||||
assert evt["reason"]
|
||||
Reference in New Issue
Block a user