* test(probe): expand coverage — dubbing, i18n, engines, security, migration, dictation, design, coverage-critic Broadens the probe harness from one happy-path spec per layer to whole-app feature coverage (web, backend, dictation, clone, design), keeping the Actor/Judge split and offline-by-default + enable-on-demand for heavy paths. New specs + judges (one subprocess boot shared across backend-touching specs): - dubbing (L4): segment duration-ratio, SRT/VTT well-formed, export-archive contents, output language-ID (advisory) - i18n: locale files valid JSON (gate); orphan-keys + coverage (advisory). NOTE: surfaced a real bug — all 20 non-en locales carry gallery.cat_*/ bootstrap.lines keys absent from the en reference (reported, not gated). - engine matrix: active engine available + every unavailable engine explains why (11 TTS / 7 ASR backends via /engines/*) - loopback security: system routes reject non-loopback origins (403) - DB migration: alembic UPGRADE on the seeded omnivoice_data fixture - Coverage Critic: every declared layer still has a spec (gate) + API inventory - dictation: streaming-ASR WebSocket /ws/transcribe registered + handshake - voice design: reuses the audio-correctness ladder - real ASR round-trip: enable-on-demand (PROBE_E2E=1) Enriched _boot_runner.py to capture engines/asr/loopback/openapi/ws in ONE isolated boot (conftest boot_capture session fixture); added env.seeded_data_dir. 13 specs total. probe suite 74 passed / 5 skipped; full repo 687 passed, 0 failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(probe): address all 15 unresolved review findings on #247 - coverage.py:22 — use `with open(...)` context to close spec files after yaml.safe_load (file handle leak) - _boot_runner.py:80 — store only `type(exc).__name__` for WS errors; drop raw str(exc) that could leak home paths / secrets into capture JSON - _boot_runner.py:99 — snapshot DB files before boot; set db_created=True only when boot creates NEW files (not when fixture already had one) - dubbing.py:46 — FAIL segments_duration_ratio when validated==0 (guards against empty/corrupt segment list passing vacuously) - i18n.py:49 — FAIL locale_valid_json when locales_dir is empty/missing - i18n.py:7 — fix docstring: locale_no_orphan_keys is advisory, not blocking - test_probe_i18n.py:59 — assert r.passed is False, not just r.advisory - coverage_critic.probe.yaml:15 — add "meta" to required layers list - dub_export.probe.yaml:17 — capture dub_audio in steps before advisory reads it - migration.probe.yaml:13 — add path_exists(db_path) data-integrity check - test_probe_asr_e2e.py:33 — os.path.exists → os.path.isfile for PROBE_ASR_SAMPLE - test_probe_migration.py:24 — assert context["db_path"] (presence) not db_created (new creation), aligning with the boot_runner fix Two findings intentionally skipped with reasons (see review thread replies): test_probe_design.py:36 — offline pattern is intentional; actor step is bypassed by design throughout the probe suite for CI compatibility test_probe_engines.py:22 — whisperx pin is intentional; it verifies the shipped default ASR engine is available out-of-the-box Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(probe): ASCII x in dubbing detail (ruff) + run migration judges inside seeded dir Two regressions from the hardening pass: - dubbing.py: replace non-ASCII '×' with 'x' (Ruff ambiguous-unicode → Tests lint fail) - test_probe_migration: move run_judges inside the seeded_data_dir with-block so the new path_exists check sees the DB before the temp dir is torn down (was always failing) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
127 lines
4.3 KiB
Python
127 lines
4.3 KiB
Python
"""L5 env/first-run layer — boot the app in a fresh environment and capture the
|
|
signals the judges verify.
|
|
|
|
The core value OmniVoice is built around is "a first-run that actually works".
|
|
This layer tests exactly that: point the backend at an empty data dir, boot it,
|
|
and confirm it reaches health, initializes its database, and answers the
|
|
lowest-cost endpoints — the path a brand-new user hits.
|
|
|
|
The boot runs **in a subprocess** (``_boot_runner.py``) so it never mutates the
|
|
parent test session. Booting in-process would require purging and re-importing
|
|
the backend (``core.config`` caches DB_PATH at import), which corrupts state for
|
|
every other test in the suite. A subprocess is both safe and more faithful to a
|
|
real first run — a fresh process against a fresh data dir.
|
|
|
|
Docker / live-container boot is gated behind :func:`docker_available` and skips
|
|
cleanly where no daemon is present.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import contextlib
|
|
import json
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
from pathlib import Path
|
|
|
|
_REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
_BOOT_RUNNER = Path(__file__).resolve().parent / "_boot_runner.py"
|
|
|
|
|
|
@contextlib.contextmanager
|
|
def fresh_data_dir(tmp_path: str | os.PathLike | None = None):
|
|
"""Yield a brand-new, empty OMNIVOICE_DATA_DIR (true first-run state)."""
|
|
created = None
|
|
if tmp_path is None:
|
|
created = tempfile.mkdtemp(prefix="probe-firstrun-")
|
|
target = created
|
|
else:
|
|
target = str(tmp_path)
|
|
os.makedirs(target, exist_ok=True)
|
|
try:
|
|
yield Path(target)
|
|
finally:
|
|
if created:
|
|
import shutil
|
|
|
|
shutil.rmtree(created, ignore_errors=True)
|
|
|
|
|
|
@contextlib.contextmanager
|
|
def seeded_data_dir():
|
|
"""Yield a temp data dir pre-populated with the checked-in regression fixture
|
|
(tests/fixtures/omnivoice_data) — for testing the alembic UPGRADE path on
|
|
existing user data, not a clean first run."""
|
|
import shutil
|
|
import tempfile
|
|
|
|
fixture = _REPO_ROOT / "tests" / "fixtures" / "omnivoice_data"
|
|
if not fixture.exists():
|
|
raise RuntimeError(f"regression fixture missing at {fixture}")
|
|
tmp = tempfile.mkdtemp(prefix="probe-seeded-")
|
|
try:
|
|
shutil.copytree(fixture, tmp, dirs_exist_ok=True)
|
|
yield Path(tmp)
|
|
finally:
|
|
shutil.rmtree(tmp, ignore_errors=True)
|
|
|
|
|
|
def capture_first_run(data_dir: str | os.PathLike, timeout: float = 180.0) -> dict:
|
|
"""Boot the backend against ``data_dir`` in a subprocess and return the
|
|
captured first-run context (endpoint status/body/latency + on-disk
|
|
artifacts). Raises RuntimeError if the boot fails or times out.
|
|
|
|
The parent process's modules and environment are left completely untouched.
|
|
"""
|
|
fd, out_path = tempfile.mkstemp(suffix=".json", prefix="probe-capture-")
|
|
os.close(fd)
|
|
child_env = dict(os.environ)
|
|
child_env.update(
|
|
OMNIVOICE_MODEL="test",
|
|
OMNIVOICE_DISABLE_FILE_LOG="1",
|
|
OMNIVOICE_DATA_DIR=str(data_dir),
|
|
)
|
|
try:
|
|
proc = subprocess.run(
|
|
[sys.executable, str(_BOOT_RUNNER), str(data_dir), out_path],
|
|
cwd=str(_REPO_ROOT),
|
|
env=child_env,
|
|
capture_output=True,
|
|
text=True,
|
|
timeout=timeout,
|
|
)
|
|
if proc.returncode != 0:
|
|
raise RuntimeError(
|
|
f"first-run boot subprocess failed (rc={proc.returncode}):\n"
|
|
+ (proc.stderr or "")[-2000:]
|
|
)
|
|
with open(out_path, encoding="utf-8") as fh:
|
|
return json.load(fh)
|
|
finally:
|
|
with contextlib.suppress(OSError):
|
|
os.remove(out_path)
|
|
|
|
|
|
# ── optional runtimes (skip cleanly when absent) ────────────────────────────────
|
|
|
|
|
|
def docker_available() -> bool:
|
|
"""True only if a Docker CLI *and* a responsive daemon are present."""
|
|
import shutil
|
|
|
|
if shutil.which("docker") is None:
|
|
return False
|
|
try:
|
|
return (
|
|
subprocess.run(["docker", "info"], capture_output=True, timeout=10).returncode == 0
|
|
)
|
|
except Exception: # noqa: BLE001
|
|
return False
|
|
|
|
|
|
def compose_file() -> Path:
|
|
"""Path to the project's docker-compose (the L5 Docker Actor target)."""
|
|
return _REPO_ROOT / "deploy" / "docker-compose.yml"
|