Palash DebnathandClaude Opus 4.8
ef98aae4db
* test(probe): expand coverage — dubbing, i18n, engines, security, migration, dictation, design, coverage-critic Broadens the probe harness from one happy-path spec per layer to whole-app feature coverage (web, backend, dictation, clone, design), keeping the Actor/Judge split and offline-by-default + enable-on-demand for heavy paths. New specs + judges (one subprocess boot shared across backend-touching specs): - dubbing (L4): segment duration-ratio, SRT/VTT well-formed, export-archive contents, output language-ID (advisory) - i18n: locale files valid JSON (gate); orphan-keys + coverage (advisory). NOTE: surfaced a real bug — all 20 non-en locales carry gallery.cat_*/ bootstrap.lines keys absent from the en reference (reported, not gated). - engine matrix: active engine available + every unavailable engine explains why (11 TTS / 7 ASR backends via /engines/*) - loopback security: system routes reject non-loopback origins (403) - DB migration: alembic UPGRADE on the seeded omnivoice_data fixture - Coverage Critic: every declared layer still has a spec (gate) + API inventory - dictation: streaming-ASR WebSocket /ws/transcribe registered + handshake - voice design: reuses the audio-correctness ladder - real ASR round-trip: enable-on-demand (PROBE_E2E=1) Enriched _boot_runner.py to capture engines/asr/loopback/openapi/ws in ONE isolated boot (conftest boot_capture session fixture); added env.seeded_data_dir. 13 specs total. probe suite 74 passed / 5 skipped; full repo 687 passed, 0 failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(probe): address all 15 unresolved review findings on #247 - coverage.py:22 — use `with open(...)` context to close spec files after yaml.safe_load (file handle leak) - _boot_runner.py:80 — store only `type(exc).__name__` for WS errors; drop raw str(exc) that could leak home paths / secrets into capture JSON - _boot_runner.py:99 — snapshot DB files before boot; set db_created=True only when boot creates NEW files (not when fixture already had one) - dubbing.py:46 — FAIL segments_duration_ratio when validated==0 (guards against empty/corrupt segment list passing vacuously) - i18n.py:49 — FAIL locale_valid_json when locales_dir is empty/missing - i18n.py:7 — fix docstring: locale_no_orphan_keys is advisory, not blocking - test_probe_i18n.py:59 — assert r.passed is False, not just r.advisory - coverage_critic.probe.yaml:15 — add "meta" to required layers list - dub_export.probe.yaml:17 — capture dub_audio in steps before advisory reads it - migration.probe.yaml:13 — add path_exists(db_path) data-integrity check - test_probe_asr_e2e.py:33 — os.path.exists → os.path.isfile for PROBE_ASR_SAMPLE - test_probe_migration.py:24 — assert context["db_path"] (presence) not db_created (new creation), aligning with the boot_runner fix Two findings intentionally skipped with reasons (see review thread replies): test_probe_design.py:36 — offline pattern is intentional; actor step is bypassed by design throughout the probe suite for CI compatibility test_probe_engines.py:22 — whisperx pin is intentional; it verifies the shipped default ASR engine is available out-of-the-box Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(probe): ASCII x in dubbing detail (ruff) + run migration judges inside seeded dir Two regressions from the hardening pass: - dubbing.py: replace non-ASCII '×' with 'x' (Ruff ambiguous-unicode → Tests lint fail) - test_probe_migration: move run_judges inside the seeded_data_dir with-block so the new path_exists check sees the DB before the temp dir is torn down (was always failing) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
38 lines
1.5 KiB
Python
38 lines
1.5 KiB
Python
"""Engine matrix — TTS/ASR registry verified against the real running backend
|
|
(shared boot), plus offline judge unit tests."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
from . import spec as probe_spec
|
|
from .judges import engine as E
|
|
|
|
_SPEC = os.path.join(os.path.dirname(__file__), "specs", "engines.probe.yaml")
|
|
|
|
|
|
def test_engine_matrix(probe_report, boot_capture):
|
|
spec = probe_spec.load_spec(_SPEC)
|
|
ctx = {"engines_tts": boot_capture["engines_tts"], "engines_asr": boot_capture["engines_asr"]}
|
|
results = probe_spec.run_judges(spec, ctx)
|
|
probe_report.record(spec, results)
|
|
assert probe_spec.blocking_failures(results) == [], "\n".join(str(r) for r in results)
|
|
# Real-world sanity: the shipped defaults are available out of the box.
|
|
assert E.active_engine_available(boot_capture["engines_tts"]).passed is True
|
|
assert E.engine_available(boot_capture["engines_asr"], "whisperx").passed is True
|
|
|
|
|
|
def test_unavailable_engines_explained_synthetic():
|
|
payload = {"active": "a", "backends": [
|
|
{"id": "a", "available": True},
|
|
{"id": "b", "available": False, "reason": "pip install b"},
|
|
]}
|
|
assert E.unavailable_engines_explained(payload).passed is True
|
|
silent = {"backends": [{"id": "c", "available": False, "reason": ""}]}
|
|
assert E.unavailable_engines_explained(silent).passed is False
|
|
|
|
|
|
def test_active_engine_unavailable_fails():
|
|
payload = {"active": "ghost", "backends": [{"id": "real", "available": True}]}
|
|
assert E.active_engine_available(payload).passed is False
|