Files
VoiceStudio/tests/probe/env.py
T
Palash DebnathandClaude Opus 4.8 ef98aae4db test(probe): whole-app coverage — dubbing, i18n, engines, security, migration, dictation, design (#247)
* test(probe): expand coverage — dubbing, i18n, engines, security, migration, dictation, design, coverage-critic

Broadens the probe harness from one happy-path spec per layer to whole-app
feature coverage (web, backend, dictation, clone, design), keeping the
Actor/Judge split and offline-by-default + enable-on-demand for heavy paths.

New specs + judges (one subprocess boot shared across backend-touching specs):
- dubbing (L4): segment duration-ratio, SRT/VTT well-formed, export-archive
  contents, output language-ID (advisory)
- i18n: locale files valid JSON (gate); orphan-keys + coverage (advisory).
  NOTE: surfaced a real bug — all 20 non-en locales carry gallery.cat_*/
  bootstrap.lines keys absent from the en reference (reported, not gated).
- engine matrix: active engine available + every unavailable engine explains
  why (11 TTS / 7 ASR backends via /engines/*)
- loopback security: system routes reject non-loopback origins (403)
- DB migration: alembic UPGRADE on the seeded omnivoice_data fixture
- Coverage Critic: every declared layer still has a spec (gate) + API inventory
- dictation: streaming-ASR WebSocket /ws/transcribe registered + handshake
- voice design: reuses the audio-correctness ladder
- real ASR round-trip: enable-on-demand (PROBE_E2E=1)

Enriched _boot_runner.py to capture engines/asr/loopback/openapi/ws in ONE
isolated boot (conftest boot_capture session fixture); added env.seeded_data_dir.

13 specs total. probe suite 74 passed / 5 skipped; full repo 687 passed, 0 failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(probe): address all 15 unresolved review findings on #247

- coverage.py:22 — use `with open(...)` context to close spec files after
  yaml.safe_load (file handle leak)
- _boot_runner.py:80 — store only `type(exc).__name__` for WS errors; drop
  raw str(exc) that could leak home paths / secrets into capture JSON
- _boot_runner.py:99 — snapshot DB files before boot; set db_created=True
  only when boot creates NEW files (not when fixture already had one)
- dubbing.py:46 — FAIL segments_duration_ratio when validated==0 (guards
  against empty/corrupt segment list passing vacuously)
- i18n.py:49 — FAIL locale_valid_json when locales_dir is empty/missing
- i18n.py:7 — fix docstring: locale_no_orphan_keys is advisory, not blocking
- test_probe_i18n.py:59 — assert r.passed is False, not just r.advisory
- coverage_critic.probe.yaml:15 — add "meta" to required layers list
- dub_export.probe.yaml:17 — capture dub_audio in steps before advisory reads it
- migration.probe.yaml:13 — add path_exists(db_path) data-integrity check
- test_probe_asr_e2e.py:33 — os.path.exists → os.path.isfile for PROBE_ASR_SAMPLE
- test_probe_migration.py:24 — assert context["db_path"] (presence) not
  db_created (new creation), aligning with the boot_runner fix

Two findings intentionally skipped with reasons (see review thread replies):
  test_probe_design.py:36 — offline pattern is intentional; actor step is
    bypassed by design throughout the probe suite for CI compatibility
  test_probe_engines.py:22 — whisperx pin is intentional; it verifies the
    shipped default ASR engine is available out-of-the-box

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(probe): ASCII x in dubbing detail (ruff) + run migration judges inside seeded dir

Two regressions from the hardening pass:
- dubbing.py: replace non-ASCII '×' with 'x' (Ruff ambiguous-unicode → Tests lint fail)
- test_probe_migration: move run_judges inside the seeded_data_dir with-block so the
  new path_exists check sees the DB before the temp dir is torn down (was always failing)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 22:41:21 +05:30

127 lines
4.3 KiB
Python

"""L5 env/first-run layer — boot the app in a fresh environment and capture the
signals the judges verify.
The core value OmniVoice is built around is "a first-run that actually works".
This layer tests exactly that: point the backend at an empty data dir, boot it,
and confirm it reaches health, initializes its database, and answers the
lowest-cost endpoints — the path a brand-new user hits.
The boot runs **in a subprocess** (``_boot_runner.py``) so it never mutates the
parent test session. Booting in-process would require purging and re-importing
the backend (``core.config`` caches DB_PATH at import), which corrupts state for
every other test in the suite. A subprocess is both safe and more faithful to a
real first run — a fresh process against a fresh data dir.
Docker / live-container boot is gated behind :func:`docker_available` and skips
cleanly where no daemon is present.
"""
from __future__ import annotations
import contextlib
import json
import os
import subprocess
import sys
import tempfile
from pathlib import Path
_REPO_ROOT = Path(__file__).resolve().parents[2]
_BOOT_RUNNER = Path(__file__).resolve().parent / "_boot_runner.py"
@contextlib.contextmanager
def fresh_data_dir(tmp_path: str | os.PathLike | None = None):
"""Yield a brand-new, empty OMNIVOICE_DATA_DIR (true first-run state)."""
created = None
if tmp_path is None:
created = tempfile.mkdtemp(prefix="probe-firstrun-")
target = created
else:
target = str(tmp_path)
os.makedirs(target, exist_ok=True)
try:
yield Path(target)
finally:
if created:
import shutil
shutil.rmtree(created, ignore_errors=True)
@contextlib.contextmanager
def seeded_data_dir():
"""Yield a temp data dir pre-populated with the checked-in regression fixture
(tests/fixtures/omnivoice_data) — for testing the alembic UPGRADE path on
existing user data, not a clean first run."""
import shutil
import tempfile
fixture = _REPO_ROOT / "tests" / "fixtures" / "omnivoice_data"
if not fixture.exists():
raise RuntimeError(f"regression fixture missing at {fixture}")
tmp = tempfile.mkdtemp(prefix="probe-seeded-")
try:
shutil.copytree(fixture, tmp, dirs_exist_ok=True)
yield Path(tmp)
finally:
shutil.rmtree(tmp, ignore_errors=True)
def capture_first_run(data_dir: str | os.PathLike, timeout: float = 180.0) -> dict:
"""Boot the backend against ``data_dir`` in a subprocess and return the
captured first-run context (endpoint status/body/latency + on-disk
artifacts). Raises RuntimeError if the boot fails or times out.
The parent process's modules and environment are left completely untouched.
"""
fd, out_path = tempfile.mkstemp(suffix=".json", prefix="probe-capture-")
os.close(fd)
child_env = dict(os.environ)
child_env.update(
OMNIVOICE_MODEL="test",
OMNIVOICE_DISABLE_FILE_LOG="1",
OMNIVOICE_DATA_DIR=str(data_dir),
)
try:
proc = subprocess.run(
[sys.executable, str(_BOOT_RUNNER), str(data_dir), out_path],
cwd=str(_REPO_ROOT),
env=child_env,
capture_output=True,
text=True,
timeout=timeout,
)
if proc.returncode != 0:
raise RuntimeError(
f"first-run boot subprocess failed (rc={proc.returncode}):\n"
+ (proc.stderr or "")[-2000:]
)
with open(out_path, encoding="utf-8") as fh:
return json.load(fh)
finally:
with contextlib.suppress(OSError):
os.remove(out_path)
# ── optional runtimes (skip cleanly when absent) ────────────────────────────────
def docker_available() -> bool:
"""True only if a Docker CLI *and* a responsive daemon are present."""
import shutil
if shutil.which("docker") is None:
return False
try:
return (
subprocess.run(["docker", "info"], capture_output=True, timeout=10).returncode == 0
)
except Exception: # noqa: BLE001
return False
def compose_file() -> Path:
"""Path to the project's docker-compose (the L5 Docker Actor target)."""
return _REPO_ROOT / "deploy" / "docker-compose.yml"