feat(engines): real synthesis "Self-test" + copy-paste setup snippet for opt-in engines (#930)
* feat(engines): real synthesis "Self-test" + copy-paste setup snippet for opt-in engines Builds on #905's Engines-settings fixes (verified still green: license dialog mounts, matrix reloads on select, cpu_fallback routing toast, cpu-native → cpu_only). Two enhancements, no #905 behavior touched. Real "Self-test" for in-process TTS engines ------------------------------------------- The existing /engines/{id}/health probe only imports the package and reports "deps OK" for in-process engines — it never proves the engine can emit audio. New POST /engines/{id}/selftest runs a *tiny real synthesis* from a fixed short ASCII phrase and reports ok + duration + sample-rate + sample count, proving the engine actually produces audio. Guardrails keep it cross-platform-identical and CPU-cheap: TTS + available + in-process only, bounded wall-clock timeout (OMNIVOICE_SELFTEST_TIMEOUT_S, default 90s) that returns ok=false/timed_out instead of hanging the panel, a process-wide lock so a click-storm can't stack model loads, loopback-gated, and only ever on user click (never on load). The Compat Matrix gains a "Self-test" button (with cooldown) that renders "0.82s @ 24 kHz in 820 ms". HF tokens in a synth error are redacted like the health route. Verified end-to-end: kittentts synthesized 89,200 samples @ 24 kHz. Copy-paste setup snippet for path-gated opt-in engines ------------------------------------------------------ IndexTTS / MOSS-v1.5 / dots.tts / Confucius4 gate on an OMNIVOICE_*_DIR env var. list_backends() now emits a single-sourced `setup_snippet` (the exact `export VAR=/path/...` line) surfaced with a Copy button inside the matrix's "Why unavailable?" disclosure, so users don't reconstruct it from the docs. Also tightened the incomplete SelectEngineResponse TS type to include the routing echo (routing_status/effective_device/routing_reason) the post-select toast already reads at runtime. Tests: backend selftest success/subprocess-reject/unavailable/unknown/loopback/ exception-capture/timeout/HF-redaction + setup_snippet shape; frontend self-test render, timeout marker, subprocess+ASR gating, setup-snippet render. New route added to the API route snapshot. Full vitest (808) + backend engine/routing/asr/ route-inventory/no-CJK green; lint 0 errors; format + typecheck:ci clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): allow setup_snippet key in list_backends shape assertion The engine self-test PR added setup_snippet to each backend entry but only updated the route-shape test; test_list_backends_shape strict-asserts the key set. Add setup_snippet there too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: mergetest <test@local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Palash Debnath
co-authored by
Claude Fable 5
mergetest
Claude Fable 5
mergetest
parent
48a7154810
commit
5bd8968aea
@@ -15,6 +15,8 @@ Environment variables (`OMNIVOICE_TTS_BACKEND`, `OMNIVOICE_ASR_BACKEND`,
|
||||
`OMNIVOICE_LLM_BACKEND`) still win over the UI choice so power-users can pin
|
||||
a backend without Settings silently undoing it.
|
||||
"""
|
||||
import os
|
||||
import threading
|
||||
from time import perf_counter
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
@@ -261,6 +263,169 @@ def engine_health(engine_id: str):
|
||||
}
|
||||
|
||||
|
||||
# ── Real-synthesis self-test (in-process TTS engines) ──────────────────────
|
||||
#
|
||||
# ``/health`` above is a liveness/import probe — for an in-process backend it
|
||||
# only calls ``is_available()`` and the UI labels the result "deps OK". This
|
||||
# route goes one step further: for an AVAILABLE, IN-PROCESS TTS engine it runs
|
||||
# a *tiny real synthesis* from a fixed short phrase and reports duration +
|
||||
# sample-rate + sample count, proving the engine actually emits audio rather
|
||||
# than merely importing. The Compat Matrix's "Self-test" button calls it.
|
||||
#
|
||||
# Guardrails (kept identical across macOS/Windows/Linux per the default-feature
|
||||
# rule — the phrase, timeout and gating don't branch on OS):
|
||||
# * TTS family + available + in-process only. Subprocess engines keep their
|
||||
# spawn-and-ping ``health_check`` (a real synth there is a sidecar
|
||||
# cold-start — out of scope for a click-to-test affordance).
|
||||
# * Bounded wall-clock timeout (``OMNIVOICE_SELFTEST_TIMEOUT_S``, default 90s):
|
||||
# a runaway synth returns ``ok=False`` / ``timed_out=True`` instead of
|
||||
# hanging the Settings panel. The orphaned worker is best-effort daemon.
|
||||
# * A process-wide lock serialises self-tests so a click-storm can't stack
|
||||
# concurrent model loads.
|
||||
# * Only ever on user click (POST) — never on Settings load. Loopback-gated.
|
||||
|
||||
# Deliberately short + ASCII so the synth stays CPU-cheap and the phrase never
|
||||
# trips the no-hardcoded-CJK guard.
|
||||
_SELFTEST_PHRASE = "OmniVoice engine self test."
|
||||
_SELFTEST_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def _selftest_timeout_s() -> float:
|
||||
try:
|
||||
return max(1.0, float(os.environ.get("OMNIVOICE_SELFTEST_TIMEOUT_S", "90")))
|
||||
except (TypeError, ValueError):
|
||||
return 90.0
|
||||
|
||||
|
||||
def _sample_count(audio) -> int:
|
||||
"""Total sample count of an engine's ``generate()`` return, tolerant of
|
||||
torch.Tensor / numpy.ndarray / list shapes. 0 when it can't be measured."""
|
||||
try:
|
||||
shape = getattr(audio, "shape", None)
|
||||
if shape is not None and len(shape) > 0:
|
||||
return int(shape[-1])
|
||||
return int(len(audio))
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
def _run_synth_bounded(backend, timeout_s: float) -> dict | None:
|
||||
"""Run one tiny synthesis in a daemon thread, bounded by ``timeout_s``.
|
||||
|
||||
Returns ``{"audio": .., "duration_ms": ..}`` on success, ``{"error": exc}``
|
||||
on a synth exception, or ``None`` when the timeout elapsed (worker left
|
||||
running best-effort — Python threads can't be force-killed)."""
|
||||
box: dict = {}
|
||||
|
||||
def _worker():
|
||||
t0 = perf_counter()
|
||||
try:
|
||||
audio = backend.generate(_SELFTEST_PHRASE, language="en", num_step=8)
|
||||
box["audio"] = audio
|
||||
except Exception as exc: # noqa: BLE001 — surfaced to the caller as ok=False
|
||||
box["error"] = exc
|
||||
finally:
|
||||
box["duration_ms"] = (perf_counter() - t0) * 1000.0
|
||||
|
||||
th = threading.Thread(target=_worker, name="engine-selftest", daemon=True)
|
||||
th.start()
|
||||
th.join(timeout_s)
|
||||
if th.is_alive():
|
||||
return None
|
||||
return box
|
||||
|
||||
|
||||
class SelfTestResponse(BaseModel):
|
||||
id: str
|
||||
ok: bool
|
||||
message: str
|
||||
duration_ms: float
|
||||
sample_rate: int | None = None
|
||||
num_samples: int | None = None
|
||||
audio_seconds: float | None = None
|
||||
timed_out: bool = False
|
||||
|
||||
|
||||
@router.post(
|
||||
"/engines/{engine_id}/selftest",
|
||||
response_model=SelfTestResponse,
|
||||
dependencies=[Depends(require_loopback)],
|
||||
)
|
||||
def engine_selftest(engine_id: str):
|
||||
"""Run a bounded, real synthesis on an available in-process TTS engine.
|
||||
|
||||
404 for an unknown TTS id; 400 when the engine is subprocess-isolated or
|
||||
not currently available (a real synth on either is meaningless). Never
|
||||
raises through to a 500 on a synth failure — the exception is captured into
|
||||
``ok=False`` / ``message`` so the panel renders a per-row failure."""
|
||||
if engine_id not in tts_backend._REGISTRY:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"unknown TTS engine id: {engine_id!r}",
|
||||
)
|
||||
cls = tts_backend._REGISTRY[engine_id]
|
||||
if getattr(cls, "_is_subprocess_isolated", False):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
f"{engine_id} is subprocess-isolated — self-test runs real "
|
||||
"synthesis for in-process engines only. Use Test engine "
|
||||
"(spawn-and-ping) for subprocess engines."
|
||||
),
|
||||
)
|
||||
try:
|
||||
ok, msg = cls.is_available()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
ok, msg = False, f"{type(exc).__name__}: {exc}"
|
||||
if not ok:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
f"{engine_id} is not available: {tts_backend._mask_hf_tokens(msg)}. "
|
||||
"Install/enable the engine, then self-test."
|
||||
),
|
||||
)
|
||||
|
||||
timeout_s = _selftest_timeout_s()
|
||||
# Serialise so a click-storm can't stack concurrent model loads.
|
||||
with _SELFTEST_LOCK:
|
||||
backend = _get_engine_instance(cls)
|
||||
res = _run_synth_bounded(backend, timeout_s)
|
||||
|
||||
if res is None:
|
||||
return SelfTestResponse(
|
||||
id=engine_id,
|
||||
ok=False,
|
||||
message=f"timed out after {timeout_s:.0f}s (model still loading?)",
|
||||
duration_ms=timeout_s * 1000.0,
|
||||
timed_out=True,
|
||||
)
|
||||
if "error" in res:
|
||||
exc = res["error"]
|
||||
return SelfTestResponse(
|
||||
id=engine_id,
|
||||
ok=False,
|
||||
message=tts_backend._mask_hf_tokens(f"{type(exc).__name__}: {exc}"),
|
||||
duration_ms=res.get("duration_ms", 0.0),
|
||||
)
|
||||
|
||||
n = _sample_count(res.get("audio"))
|
||||
try:
|
||||
sr = int(getattr(backend, "sample_rate", 0) or 0) or None
|
||||
except Exception:
|
||||
sr = None
|
||||
secs = round(n / sr, 3) if (sr and n) else None
|
||||
return SelfTestResponse(
|
||||
id=engine_id,
|
||||
ok=n > 0,
|
||||
message="synthesized" if n > 0 else "engine returned no audio",
|
||||
duration_ms=res["duration_ms"],
|
||||
sample_rate=sr,
|
||||
num_samples=n or None,
|
||||
audio_seconds=secs,
|
||||
)
|
||||
|
||||
|
||||
class SelectEngineRequest(BaseModel):
|
||||
family: str # "tts" | "asr" | "llm"
|
||||
backend_id: str
|
||||
|
||||
@@ -1322,6 +1322,21 @@ _INSTALL_HINTS: dict[str, str] = {
|
||||
}
|
||||
|
||||
|
||||
# Copy-paste-ready setup line for opt-in engines gated behind a filesystem-path
|
||||
# env var (issue #498 / #590). The install_hint tells users a var exists; this
|
||||
# is the *exact* `export VAR=...` line to run, so they don't have to reconstruct
|
||||
# it from the docs. Surfaced verbatim in the Compat Matrix's "Why unavailable?"
|
||||
# disclosure with a Copy button. Single-sourced here so it can't drift from the
|
||||
# var each engine's is_available() actually reads. bash/zsh form (the dominant
|
||||
# clone-and-run workflow for these engines; dots.tts is *nix-only anyway).
|
||||
_SETUP_SNIPPETS: dict[str, str] = {
|
||||
"indextts2": "export OMNIVOICE_INDEXTTS_DIR=/path/to/index-tts",
|
||||
"moss-tts-v15": "export OMNIVOICE_MOSS_TTS_V15_DIR=/path/to/MOSS-TTS",
|
||||
"dots-tts": "export OMNIVOICE_DOTS_TTS_DIR=/path/to/dots.tts",
|
||||
"confucius4-tts": "export OMNIVOICE_CONFUCIUS4_TTS_DIR=/path/to/Confucius4-TTS",
|
||||
}
|
||||
|
||||
|
||||
def list_backends() -> list[dict]:
|
||||
"""Enumerate every registered backend with its availability state.
|
||||
|
||||
@@ -1333,6 +1348,7 @@ def list_backends() -> list[dict]:
|
||||
"available": bool,
|
||||
"reason": Optional[str], # message when not available
|
||||
"install_hint": Optional[str],
|
||||
"setup_snippet": Optional[str], # exact `export VAR=...` for path-gated opt-in engines
|
||||
"last_error": Optional[str], # cached most-recent failure
|
||||
"isolation_mode": "in-process" | "subprocess",
|
||||
"gpu_compat": list[str], # subset of {cuda, rocm, mps, xpu, cpu}
|
||||
@@ -1396,6 +1412,8 @@ def list_backends() -> list[dict]:
|
||||
"available": ok,
|
||||
"reason": None if ok else _mask_hf_tokens(msg),
|
||||
"install_hint": _INSTALL_HINTS.get(bid),
|
||||
# Exact `export VAR=...` line for path-gated opt-in engines, or None.
|
||||
"setup_snippet": _SETUP_SNIPPETS.get(bid),
|
||||
"last_error": _LAST_ERRORS.get(bid),
|
||||
"isolation_mode": isolation,
|
||||
"gpu_compat": list(gpu_compat),
|
||||
|
||||
Regular → Executable
@@ -3,6 +3,7 @@ import type {
|
||||
AllEnginesResponse,
|
||||
EngineFamily,
|
||||
EngineHealthResponse,
|
||||
EngineSelfTestResponse,
|
||||
SelectEngineResponse,
|
||||
} from './types';
|
||||
|
||||
@@ -64,6 +65,18 @@ export async function getEngineHealth(engineId: string): Promise<EngineHealthRes
|
||||
return apiJson<EngineHealthResponse>(`/engines/${encodeURIComponent(engineId)}/health`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a bounded, real tiny-synthesis on an AVAILABLE, IN-PROCESS TTS engine —
|
||||
* proves the engine actually emits audio (duration + sample-rate + samples),
|
||||
* not just that its package imports (`is_available()` liveness). The Compat
|
||||
* Matrix's "Self-test" button calls this; only ever on user click, never on
|
||||
* Settings mount. 400 for a subprocess-isolated or not-available engine, 404
|
||||
* for a non-TTS id. Never 500s on a synth failure — it lands in `ok:false`.
|
||||
*/
|
||||
export async function selfTestEngine(engineId: string): Promise<EngineSelfTestResponse> {
|
||||
return apiPost<EngineSelfTestResponse>(`/engines/${encodeURIComponent(engineId)}/selftest`, {});
|
||||
}
|
||||
|
||||
export async function listTranslationEngines(): Promise<TranslationEnginesResponse> {
|
||||
return apiJson<TranslationEnginesResponse>('/engines/translation');
|
||||
}
|
||||
|
||||
@@ -30,6 +30,9 @@ interface EngineBackend {
|
||||
available: boolean;
|
||||
reason: string | null;
|
||||
install_hint?: string | null;
|
||||
// Copy-paste-ready `export VAR=...` line for a path-gated opt-in engine
|
||||
// (IndexTTS / MOSS-v1.5 / dots.tts / Confucius4), else null/absent.
|
||||
setup_snippet?: string | null;
|
||||
last_error?: string | null;
|
||||
isolation_mode?: 'in-process' | 'subprocess';
|
||||
gpu_compat?: GPUTarget[];
|
||||
@@ -54,6 +57,12 @@ export interface SelectEngineResponse {
|
||||
family: EngineFamily;
|
||||
active: string;
|
||||
env_override: boolean;
|
||||
// Routing verdict for the picked engine on THIS host (#21) — the select echo
|
||||
// the post-select toast reads to warn on a cpu_fallback pick. Optional so a
|
||||
// legacy payload without them still types cleanly.
|
||||
routing_status?: RoutingStatus;
|
||||
effective_device?: EffectiveDevice;
|
||||
routing_reason?: string | null;
|
||||
}
|
||||
|
||||
export interface EngineHealthResponse {
|
||||
@@ -63,6 +72,20 @@ export interface EngineHealthResponse {
|
||||
latency_ms: number;
|
||||
}
|
||||
|
||||
// Real-synthesis self-test result for an available in-process TTS engine
|
||||
// (POST /engines/{id}/selftest). `ok` proves the engine emitted audio; the
|
||||
// rest quantify it. `timed_out` marks a synth that outran the bounded timeout.
|
||||
export interface EngineSelfTestResponse {
|
||||
id: string;
|
||||
ok: boolean;
|
||||
message: string;
|
||||
duration_ms: number;
|
||||
sample_rate?: number | null;
|
||||
num_samples?: number | null;
|
||||
audio_seconds?: number | null;
|
||||
timed_out?: boolean;
|
||||
}
|
||||
|
||||
// ── System / diagnostics ─────────────────────────────────────────────────
|
||||
export interface SystemInfo {
|
||||
app_version?: string;
|
||||
|
||||
@@ -8,10 +8,14 @@ import {
|
||||
CheckCircle2,
|
||||
RefreshCw,
|
||||
Layers,
|
||||
Volume2,
|
||||
Copy,
|
||||
Check,
|
||||
} from 'lucide-react';
|
||||
import { toastErrorWithReport } from '../utils/errorToast';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { listEngines, getEngineHealth } from '../api/engines';
|
||||
import { listEngines, getEngineHealth, selfTestEngine } from '../api/engines';
|
||||
import { copyText } from '../utils/copyText';
|
||||
import { ChevronRight } from 'lucide-react';
|
||||
import { Badge, Button, Segmented, Table } from '../ui';
|
||||
import { cn } from '@/lib/utils';
|
||||
@@ -128,6 +132,8 @@ function normalizeEntry(entry) {
|
||||
isolation_mode: entry.isolation_mode || 'in-process',
|
||||
gpu_compat:
|
||||
Array.isArray(entry.gpu_compat) && entry.gpu_compat.length > 0 ? entry.gpu_compat : ['cpu'],
|
||||
// Copy-paste `export VAR=...` line for a path-gated opt-in engine, or null.
|
||||
setup_snippet: entry.setup_snippet || null,
|
||||
// Routing (#21) — may be absent on a legacy/older backend payload, in
|
||||
// which case the matrix renders exactly as before (no routing badge).
|
||||
effective_device: entry.effective_device || null,
|
||||
@@ -136,6 +142,13 @@ function normalizeEntry(entry) {
|
||||
};
|
||||
}
|
||||
|
||||
/** Human duration: "0.4s" for ≥1 s, "820 ms" below — keeps the self-test
|
||||
* result compact whether a cold model load or a warm sub-second synth. */
|
||||
function fmtDuration(ms) {
|
||||
const n = Number(ms) || 0;
|
||||
return n >= 1000 ? `${(n / 1000).toFixed(1)}s` : `${Math.round(n)} ms`;
|
||||
}
|
||||
|
||||
export default function EngineCompatibilityMatrix({
|
||||
family = 'tts',
|
||||
onSelect = null,
|
||||
@@ -144,6 +157,7 @@ export default function EngineCompatibilityMatrix({
|
||||
// without resorting to module-level vi.mock incantations.
|
||||
apiListEngines = listEngines,
|
||||
apiGetEngineHealth = getEngineHealth,
|
||||
apiSelfTestEngine = selfTestEngine,
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
const [data, setData] = useState(null);
|
||||
@@ -158,6 +172,11 @@ export default function EngineCompatibilityMatrix({
|
||||
// { [id]: { inflight: boolean, ok?: boolean, message?: string,
|
||||
// latency_ms?: number, lastClickAt?: number } }
|
||||
const [healthByEngine, setHealthByEngine] = useState({});
|
||||
// Self-test (real tiny synthesis) state keyed by engine id, same shape as
|
||||
// health plus { duration_ms, sample_rate, audio_seconds, timed_out }.
|
||||
const [selfTestByEngine, setSelfTestByEngine] = useState({});
|
||||
// Which engine's setup snippet was just copied (transient ✓ affordance).
|
||||
const [copiedId, setCopiedId] = useState(null);
|
||||
|
||||
useEffect(() => {
|
||||
setActiveFamily(family);
|
||||
@@ -230,6 +249,48 @@ export default function EngineCompatibilityMatrix({
|
||||
[apiGetEngineHealth, healthByEngine],
|
||||
);
|
||||
|
||||
const runSelfTest = useCallback(
|
||||
async (id) => {
|
||||
const now = Date.now();
|
||||
const cur = selfTestByEngine[id];
|
||||
if (cur?.inflight) return;
|
||||
if (cur?.lastClickAt && now - cur.lastClickAt < TEST_COOLDOWN_MS) {
|
||||
// Click-storm cooldown — silently ignore. The backend also serialises
|
||||
// self-tests, so this is belt-and-braces against stacked model loads.
|
||||
return;
|
||||
}
|
||||
setSelfTestByEngine((prev) => ({
|
||||
...prev,
|
||||
[id]: { inflight: true, lastClickAt: now },
|
||||
}));
|
||||
try {
|
||||
const result = await apiSelfTestEngine(id);
|
||||
setSelfTestByEngine((prev) => ({
|
||||
...prev,
|
||||
[id]: { inflight: false, lastClickAt: now, ...result },
|
||||
}));
|
||||
} catch (e) {
|
||||
setSelfTestByEngine((prev) => ({
|
||||
...prev,
|
||||
[id]: {
|
||||
inflight: false,
|
||||
ok: false,
|
||||
message: e?.message || String(e),
|
||||
lastClickAt: now,
|
||||
},
|
||||
}));
|
||||
}
|
||||
},
|
||||
[apiSelfTestEngine, selfTestByEngine],
|
||||
);
|
||||
|
||||
const copySetup = useCallback(async (id, snippet) => {
|
||||
const ok = await copyText(snippet);
|
||||
if (!ok) return;
|
||||
setCopiedId(id);
|
||||
setTimeout(() => setCopiedId((c) => (c === id ? null : c)), 1500);
|
||||
}, []);
|
||||
|
||||
const COLUMNS = [
|
||||
{ key: 'name', label: t('engines.matrixTitle').split(' ')[0] || 'Engine', flex: 3 },
|
||||
{ key: 'status', label: t('engines.status'), width: 130, align: 'center' },
|
||||
@@ -322,6 +383,12 @@ export default function EngineCompatibilityMatrix({
|
||||
{backends.map((b) => {
|
||||
const isActive = b.id === activeBackendId;
|
||||
const health = healthByEngine[b.id];
|
||||
const selfTest = selfTestByEngine[b.id];
|
||||
// Real-synthesis self-test is TTS-only and meaningful only for an
|
||||
// available, in-process engine (subprocess engines keep spawn-and-
|
||||
// ping via "Test engine"; a real synth there is a sidecar cold-start).
|
||||
const canSelfTest =
|
||||
activeFamily === 'tts' && b.available && b.isolation_mode !== 'subprocess';
|
||||
return (
|
||||
<div
|
||||
key={b.id}
|
||||
@@ -386,6 +453,35 @@ export default function EngineCompatibilityMatrix({
|
||||
{t('engines.lastError', { error: b.last_error })}
|
||||
</span>
|
||||
)}
|
||||
{/* Copy-paste-ready setup line for a path-gated opt-in
|
||||
engine (IndexTTS/MOSS-v1.5/dots/Confucius4) — the
|
||||
exact `export VAR=…` so users don't hunt the docs. */}
|
||||
{b.setup_snippet && (
|
||||
<div
|
||||
className="engine-matrix__setup flex flex-col gap-[3px] mt-[2px]"
|
||||
data-testid={`setup-snippet-${b.id}`}
|
||||
>
|
||||
<span className="text-[11px] text-[color:var(--chrome-fg-muted,#888)]">
|
||||
{t('engines.setupSnippetLabel')}
|
||||
</span>
|
||||
<div className="flex items-center gap-[6px] flex-wrap">
|
||||
<code className="engine-matrix__setup-code font-mono text-[11px] px-[6px] py-[2px] rounded [background:var(--chrome-bg-inset,rgba(255,255,255,0.05))] text-[color:var(--chrome-fg,currentColor)] break-all">
|
||||
{b.setup_snippet}
|
||||
</code>
|
||||
<Button
|
||||
size="sm"
|
||||
variant="subtle"
|
||||
onClick={() => copySetup(b.id, b.setup_snippet)}
|
||||
leading={
|
||||
copiedId === b.id ? <Check size={11} /> : <Copy size={11} />
|
||||
}
|
||||
aria-label={t('engines.copySetup', { engine: b.display_name })}
|
||||
>
|
||||
{copiedId === b.id ? t('engines.copied') : t('engines.copy')}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</details>
|
||||
)}
|
||||
@@ -558,6 +654,42 @@ export default function EngineCompatibilityMatrix({
|
||||
: t('engines.failed')}
|
||||
</span>
|
||||
)}
|
||||
{/* Self-test: a real tiny synthesis proving the in-process TTS
|
||||
engine emits audio (not just imports). Guarded — TTS only,
|
||||
available + in-process only, user click only, cooldown +
|
||||
backend timeout bound it. */}
|
||||
{canSelfTest && (
|
||||
<Button
|
||||
size="sm"
|
||||
variant="subtle"
|
||||
onClick={() => runSelfTest(b.id)}
|
||||
disabled={!!selfTest?.inflight}
|
||||
loading={!!selfTest?.inflight}
|
||||
leading={!selfTest?.inflight && <Volume2 size={11} />}
|
||||
aria-label={`Self-test ${b.display_name}`}
|
||||
>
|
||||
{selfTest?.inflight ? t('engines.selfTesting') : t('engines.selfTest')}
|
||||
</Button>
|
||||
)}
|
||||
{canSelfTest && selfTest && !selfTest.inflight && (
|
||||
<span
|
||||
className={`engine-matrix__selftest-result text-[11px] font-mono ${selfTest.ok ? 'text-[color:var(--chrome-severity-ok,#98971a)]' : 'text-[color:var(--chrome-severity-err,#cc241d)]'}`}
|
||||
data-testid={`selftest-result-${b.id}`}
|
||||
title={selfTest.message}
|
||||
>
|
||||
{selfTest.ok
|
||||
? t('engines.selfTestOk', {
|
||||
seconds: Number(selfTest.audio_seconds ?? 0).toFixed(2),
|
||||
khz: selfTest.sample_rate
|
||||
? Math.round(selfTest.sample_rate / 1000)
|
||||
: '?',
|
||||
took: fmtDuration(selfTest.duration_ms),
|
||||
})
|
||||
: selfTest.timed_out
|
||||
? t('engines.selfTestTimedOut')
|
||||
: t('engines.selfTestFailed')}
|
||||
</span>
|
||||
)}
|
||||
{onSelect && b.available && !isActive && (
|
||||
<Button
|
||||
size="sm"
|
||||
|
||||
@@ -1558,6 +1558,15 @@
|
||||
"latencyMs": "{{ms}} ms",
|
||||
"depsOk": "deps OK",
|
||||
"failed": "failed",
|
||||
"selfTest": "Self-test",
|
||||
"selfTesting": "Synthesizing…",
|
||||
"selfTestOk": "{{seconds}}s @ {{khz}} kHz in {{took}}",
|
||||
"selfTestFailed": "Self-test failed",
|
||||
"selfTestTimedOut": "Self-test timed out",
|
||||
"setupSnippetLabel": "Point OmniVoice at the model directory, then restart:",
|
||||
"copy": "Copy",
|
||||
"copied": "Copied",
|
||||
"copySetup": "Copy setup command for {{engine}}",
|
||||
"acceptLicense": "Accept license",
|
||||
"noBackends": "No backends registered.",
|
||||
"switch_failed": "Failed to switch engine",
|
||||
|
||||
@@ -429,7 +429,9 @@ describe('EngineCompatibilityMatrix', () => {
|
||||
);
|
||||
await waitFor(() => screen.getByText('OmniVoice (test)'));
|
||||
const omniRow = screen.getByText('OmniVoice (test)').closest('[role="row"]');
|
||||
fireEvent.click(within(omniRow).getByRole('button', { name: /test omnivoice/i }));
|
||||
// Exact match: the row now also has a "Self-test OmniVoice" button, which a
|
||||
// loose /test omnivoice/i would ambiguously also match.
|
||||
fireEvent.click(within(omniRow).getByRole('button', { name: 'Test OmniVoice (test)' }));
|
||||
await waitFor(() => {
|
||||
expect(within(omniRow).getByTestId('health-result-omnivoice')).toHaveTextContent('deps OK');
|
||||
});
|
||||
@@ -537,4 +539,167 @@ describe('EngineCompatibilityMatrix', () => {
|
||||
expect(screen.getByText('Supertonic-3 — License Acceptance')).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
// ── Real-synthesis self-test (in-process TTS engines) ──────────────────
|
||||
it('clicking Self-test runs a real synthesis and renders audio seconds + sample rate', async () => {
|
||||
const apiListEngines = vi.fn().mockResolvedValue(makeEnginesResponse());
|
||||
const apiSelfTestEngine = vi.fn().mockResolvedValue({
|
||||
id: 'omnivoice',
|
||||
ok: true,
|
||||
message: 'synthesized',
|
||||
duration_ms: 820,
|
||||
sample_rate: 24000,
|
||||
num_samples: 19680,
|
||||
audio_seconds: 0.82,
|
||||
timed_out: false,
|
||||
});
|
||||
render(
|
||||
<EngineCompatibilityMatrix
|
||||
family="tts"
|
||||
apiListEngines={apiListEngines}
|
||||
apiGetEngineHealth={vi.fn()}
|
||||
apiSelfTestEngine={apiSelfTestEngine}
|
||||
/>,
|
||||
);
|
||||
await waitFor(() => screen.getByText('OmniVoice (test)'));
|
||||
const omniRow = screen.getByText('OmniVoice (test)').closest('[role="row"]');
|
||||
fireEvent.click(within(omniRow).getByRole('button', { name: /self-test omnivoice/i }));
|
||||
|
||||
await waitFor(() => expect(apiSelfTestEngine).toHaveBeenCalledWith('omnivoice'));
|
||||
await waitFor(() => {
|
||||
expect(within(omniRow).getByTestId('selftest-result-omnivoice')).toHaveTextContent(
|
||||
'0.82s @ 24 kHz in 820 ms',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it('renders a timed-out marker when the self-test outruns the timeout', async () => {
|
||||
const apiListEngines = vi.fn().mockResolvedValue(makeEnginesResponse());
|
||||
const apiSelfTestEngine = vi.fn().mockResolvedValue({
|
||||
id: 'omnivoice',
|
||||
ok: false,
|
||||
message: 'timed out after 90s (model still loading?)',
|
||||
duration_ms: 90000,
|
||||
timed_out: true,
|
||||
});
|
||||
render(
|
||||
<EngineCompatibilityMatrix
|
||||
family="tts"
|
||||
apiListEngines={apiListEngines}
|
||||
apiGetEngineHealth={vi.fn()}
|
||||
apiSelfTestEngine={apiSelfTestEngine}
|
||||
/>,
|
||||
);
|
||||
await waitFor(() => screen.getByText('OmniVoice (test)'));
|
||||
const omniRow = screen.getByText('OmniVoice (test)').closest('[role="row"]');
|
||||
fireEvent.click(within(omniRow).getByRole('button', { name: /self-test omnivoice/i }));
|
||||
await waitFor(() => {
|
||||
expect(within(omniRow).getByTestId('selftest-result-omnivoice')).toHaveTextContent(
|
||||
'Self-test timed out',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it('does not offer Self-test for a subprocess engine (spawn-and-ping only)', async () => {
|
||||
const apiListEngines = vi.fn().mockResolvedValue(makeEnginesResponse());
|
||||
render(
|
||||
<EngineCompatibilityMatrix
|
||||
family="tts"
|
||||
apiListEngines={apiListEngines}
|
||||
apiGetEngineHealth={vi.fn()}
|
||||
apiSelfTestEngine={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
await waitFor(() => screen.getByText('IndexTTS2 (test)'));
|
||||
const indexRow = screen.getByText('IndexTTS2 (test)').closest('[role="row"]');
|
||||
// "Test engine" (liveness) is present; the real-synth "Self-test" is not.
|
||||
expect(within(indexRow).getByRole('button', { name: /test indextts2/i })).toBeInTheDocument();
|
||||
expect(within(indexRow).queryByRole('button', { name: /self-test/i })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('does not offer Self-test on a non-TTS family (ASR)', async () => {
|
||||
const apiListEngines = vi.fn().mockResolvedValue({
|
||||
tts: { active: '', backends: [] },
|
||||
asr: {
|
||||
active: 'wx',
|
||||
backends: [
|
||||
{
|
||||
id: 'wx',
|
||||
display_name: 'WhisperX (test)',
|
||||
available: true,
|
||||
reason: null,
|
||||
install_hint: null,
|
||||
last_error: null,
|
||||
isolation_mode: 'in-process',
|
||||
gpu_compat: ['cpu'],
|
||||
},
|
||||
],
|
||||
},
|
||||
llm: { active: 'off', backends: [] },
|
||||
});
|
||||
render(
|
||||
<EngineCompatibilityMatrix
|
||||
family="asr"
|
||||
apiListEngines={apiListEngines}
|
||||
apiGetEngineHealth={vi.fn()}
|
||||
apiSelfTestEngine={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
await waitFor(() => screen.getByText('WhisperX (test)'));
|
||||
const row = screen.getByText('WhisperX (test)').closest('[role="row"]');
|
||||
expect(within(row).queryByRole('button', { name: /self-test/i })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
// ── Setup snippet for path-gated opt-in engines ────────────────────────
|
||||
it('renders the copy-paste setup snippet for a path-gated opt-in engine', async () => {
|
||||
const apiListEngines = vi.fn().mockResolvedValue({
|
||||
tts: {
|
||||
active: 'omnivoice',
|
||||
backends: [
|
||||
{
|
||||
id: 'indextts2',
|
||||
display_name: 'IndexTTS-2',
|
||||
available: false,
|
||||
reason: 'IndexTTS-2 venv not found. Set OMNIVOICE_INDEXTTS_DIR.',
|
||||
install_hint: 'git clone index-tts/index-tts',
|
||||
setup_snippet: 'export OMNIVOICE_INDEXTTS_DIR=/path/to/index-tts',
|
||||
last_error: null,
|
||||
isolation_mode: 'subprocess',
|
||||
gpu_compat: ['cuda', 'cpu'],
|
||||
},
|
||||
],
|
||||
},
|
||||
asr: { active: '', backends: [] },
|
||||
llm: { active: 'off', backends: [] },
|
||||
});
|
||||
render(
|
||||
<EngineCompatibilityMatrix
|
||||
family="tts"
|
||||
apiListEngines={apiListEngines}
|
||||
apiGetEngineHealth={vi.fn()}
|
||||
apiSelfTestEngine={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
await waitFor(() => screen.getByText('IndexTTS-2'));
|
||||
const snippet = screen.getByTestId('setup-snippet-indextts2');
|
||||
expect(snippet).toHaveTextContent('export OMNIVOICE_INDEXTTS_DIR=/path/to/index-tts');
|
||||
expect(
|
||||
within(snippet).getByRole('button', { name: /copy setup command/i }),
|
||||
).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('shows no setup snippet for a bundled engine', async () => {
|
||||
const apiListEngines = vi.fn().mockResolvedValue(makeEnginesResponse());
|
||||
render(
|
||||
<EngineCompatibilityMatrix
|
||||
family="tts"
|
||||
apiListEngines={apiListEngines}
|
||||
apiGetEngineHealth={vi.fn()}
|
||||
apiSelfTestEngine={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
await waitFor(() => screen.getByText('KittenTTS (test)'));
|
||||
// KittenTTS in the fixture carries no setup_snippet → no snippet block.
|
||||
expect(screen.queryByTestId('setup-snippet-kittentts')).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -329,6 +329,201 @@ def test_engine_health_caches_instance_across_calls(fresh_app, monkeypatch):
|
||||
)
|
||||
|
||||
|
||||
# ── /engines/{id}/selftest — real tiny synthesis (in-process TTS) ──────────
|
||||
|
||||
|
||||
def _register_fake_tts(tts_mod, engine_id, *, available=True, samples=100,
|
||||
raises=None, subprocess=False):
|
||||
"""Register a fresh in-process (or subprocess-marked) TTS stub whose
|
||||
generate() returns a `samples`-long list — torch-free so the shape test
|
||||
stays light. Returns (cls, restore_fn)."""
|
||||
_samples, _avail, _raises = samples, available, raises
|
||||
|
||||
class _Fake(tts_mod.TTSBackend):
|
||||
id = engine_id
|
||||
display_name = f"Fake {engine_id}"
|
||||
_is_subprocess_isolated = subprocess
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return 24000
|
||||
|
||||
@property
|
||||
def supported_languages(self):
|
||||
return ["en"]
|
||||
|
||||
@classmethod
|
||||
def is_available(cls):
|
||||
return (True, "ready") if _avail else (False, "deps missing (test)")
|
||||
|
||||
def generate(self, text, **kw):
|
||||
if _raises is not None:
|
||||
raise _raises
|
||||
return [0.0] * _samples
|
||||
|
||||
saved = dict(tts_mod._REGISTRY)
|
||||
tts_mod._REGISTRY[engine_id] = _Fake
|
||||
|
||||
def restore():
|
||||
tts_mod._REGISTRY.clear()
|
||||
tts_mod._REGISTRY.update(saved)
|
||||
|
||||
return _Fake, restore
|
||||
|
||||
|
||||
def test_selftest_in_process_success(fresh_app):
|
||||
from services import tts_backend as tts_mod
|
||||
|
||||
_, restore = _register_fake_tts(tts_mod, "fake-inproc", samples=1200)
|
||||
try:
|
||||
r = _client(fresh_app).post("/engines/fake-inproc/selftest")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
assert body["id"] == "fake-inproc"
|
||||
assert body["ok"] is True
|
||||
assert body["num_samples"] == 1200
|
||||
assert body["sample_rate"] == 24000
|
||||
# 1200 / 24000 = 0.05 s of audio.
|
||||
assert body["audio_seconds"] == 0.05
|
||||
assert isinstance(body["duration_ms"], (int, float))
|
||||
assert body["timed_out"] is False
|
||||
finally:
|
||||
restore()
|
||||
|
||||
|
||||
def test_selftest_rejects_subprocess_engine(fresh_app):
|
||||
from services import tts_backend as tts_mod
|
||||
|
||||
_, restore = _register_fake_tts(tts_mod, "fake-sub", subprocess=True)
|
||||
try:
|
||||
r = _client(fresh_app).post("/engines/fake-sub/selftest")
|
||||
assert r.status_code == 400
|
||||
assert "subprocess-isolated" in r.json()["detail"]
|
||||
finally:
|
||||
restore()
|
||||
|
||||
|
||||
def test_selftest_unavailable_engine_is_400(fresh_app):
|
||||
from services import tts_backend as tts_mod
|
||||
|
||||
_, restore = _register_fake_tts(tts_mod, "fake-down", available=False)
|
||||
try:
|
||||
r = _client(fresh_app).post("/engines/fake-down/selftest")
|
||||
assert r.status_code == 400
|
||||
assert "not available" in r.json()["detail"]
|
||||
finally:
|
||||
restore()
|
||||
|
||||
|
||||
def test_selftest_unknown_id_is_404(fresh_app):
|
||||
r = _client(fresh_app).post("/engines/nope-not-real/selftest")
|
||||
assert r.status_code == 404
|
||||
assert "unknown TTS engine id" in r.json()["detail"]
|
||||
|
||||
|
||||
def test_selftest_loopback_only(fresh_app):
|
||||
r = _client(fresh_app, host="10.0.0.9").post("/engines/omnivoice/selftest")
|
||||
assert r.status_code == 403
|
||||
assert r.json()["detail"] == "loopback origin required"
|
||||
|
||||
|
||||
def test_selftest_captures_synth_exception_without_500(fresh_app):
|
||||
from services import tts_backend as tts_mod
|
||||
|
||||
_, restore = _register_fake_tts(
|
||||
tts_mod, "fake-boom", raises=RuntimeError("model exploded"))
|
||||
try:
|
||||
r = _client(fresh_app).post("/engines/fake-boom/selftest")
|
||||
assert r.status_code == 200, r.text # never 500s on a synth failure
|
||||
body = r.json()
|
||||
assert body["ok"] is False
|
||||
assert "model exploded" in body["message"]
|
||||
assert body["num_samples"] is None
|
||||
finally:
|
||||
restore()
|
||||
|
||||
|
||||
def test_no_hf_token_leak_in_selftest_response(fresh_app):
|
||||
"""A synth exception carrying an HF token must be redacted in the body."""
|
||||
from services import tts_backend as tts_mod
|
||||
|
||||
_, restore = _register_fake_tts(
|
||||
tts_mod, "fake-tainted",
|
||||
raises=RuntimeError(f"401 for {SAMPLE_HF_TOKEN}"))
|
||||
try:
|
||||
r = _client(fresh_app).post("/engines/fake-tainted/selftest")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["ok"] is False
|
||||
assert not HF_TOKEN_RE.search(body["message"])
|
||||
assert "hf_***REDACTED***" in body["message"]
|
||||
finally:
|
||||
restore()
|
||||
|
||||
|
||||
def test_selftest_timeout_returns_timed_out(fresh_app, monkeypatch):
|
||||
"""A synth that outruns the bounded timeout returns ok=False/timed_out —
|
||||
the panel never hangs. Pin the timeout tiny and block generate briefly."""
|
||||
import threading as _threading
|
||||
|
||||
from api.routers import engines as engines_router
|
||||
from services import tts_backend as tts_mod
|
||||
|
||||
monkeypatch.setattr(engines_router, "_selftest_timeout_s", lambda: 0.05)
|
||||
gate = _threading.Event()
|
||||
|
||||
class _Slow(tts_mod.TTSBackend):
|
||||
id = "fake-slow"
|
||||
display_name = "Fake slow"
|
||||
|
||||
@property
|
||||
def sample_rate(self):
|
||||
return 24000
|
||||
|
||||
@property
|
||||
def supported_languages(self):
|
||||
return ["en"]
|
||||
|
||||
@classmethod
|
||||
def is_available(cls):
|
||||
return True, "ready"
|
||||
|
||||
def generate(self, text, **kw):
|
||||
gate.wait(2.0) # outruns the 50 ms timeout; released in finally
|
||||
return [0.0] * 10
|
||||
|
||||
saved = dict(tts_mod._REGISTRY)
|
||||
tts_mod._REGISTRY["fake-slow"] = _Slow
|
||||
try:
|
||||
r = _client(fresh_app).post("/engines/fake-slow/selftest")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
assert body["ok"] is False
|
||||
assert body["timed_out"] is True
|
||||
assert "timed out" in body["message"]
|
||||
finally:
|
||||
gate.set() # let the orphaned worker finish and exit
|
||||
tts_mod._REGISTRY.clear()
|
||||
tts_mod._REGISTRY.update(saved)
|
||||
|
||||
|
||||
# ── setup_snippet — copy-paste-ready env-var line for opt-in engines ────────
|
||||
|
||||
|
||||
def test_setup_snippet_present_for_path_gated_engines(fresh_app):
|
||||
client = _client(fresh_app)
|
||||
by_id = {b["id"]: b for b in client.get("/engines").json()["tts"]["backends"]}
|
||||
# Every entry carries the key (None for engines with no path gate).
|
||||
for entry in by_id.values():
|
||||
assert "setup_snippet" in entry
|
||||
# IndexTTS-2 is path-gated → exact export line, single-sourced in the backend.
|
||||
assert by_id["indextts2"]["setup_snippet"] == (
|
||||
"export OMNIVOICE_INDEXTTS_DIR=/path/to/index-tts"
|
||||
)
|
||||
# A bundled engine has no path gate → null.
|
||||
assert by_id["omnivoice"]["setup_snippet"] is None
|
||||
|
||||
|
||||
# ── HF-token leak prevention (T-02-12) ─────────────────────────────────────
|
||||
|
||||
|
||||
|
||||
@@ -151,6 +151,8 @@ def test_list_backends_shape(registry_sandbox):
|
||||
"id", "display_name", "available", "reason",
|
||||
"install_hint", "last_error", "isolation_mode", "gpu_compat",
|
||||
"effective_device", "routing_status", "routing_reason",
|
||||
# Copy-paste env-var line for path-gated opt-in engines (None otherwise).
|
||||
"setup_snippet",
|
||||
}
|
||||
for entry in out:
|
||||
assert set(entry.keys()) == required, (
|
||||
|
||||
Vendored
+1
@@ -159,6 +159,7 @@ POST /engines/sonitranslate/install
|
||||
POST /engines/sonitranslate/start
|
||||
POST /engines/sonitranslate/stop
|
||||
POST /engines/translation/{engine_id}/install
|
||||
POST /engines/{engine_id}/selftest
|
||||
POST /export
|
||||
POST /export/record
|
||||
POST /export/reveal
|
||||
|
||||
Reference in New Issue
Block a user