Palash DebnathandClaude Opus 4.8
ef98aae4db
* test(probe): expand coverage — dubbing, i18n, engines, security, migration, dictation, design, coverage-critic Broadens the probe harness from one happy-path spec per layer to whole-app feature coverage (web, backend, dictation, clone, design), keeping the Actor/Judge split and offline-by-default + enable-on-demand for heavy paths. New specs + judges (one subprocess boot shared across backend-touching specs): - dubbing (L4): segment duration-ratio, SRT/VTT well-formed, export-archive contents, output language-ID (advisory) - i18n: locale files valid JSON (gate); orphan-keys + coverage (advisory). NOTE: surfaced a real bug — all 20 non-en locales carry gallery.cat_*/ bootstrap.lines keys absent from the en reference (reported, not gated). - engine matrix: active engine available + every unavailable engine explains why (11 TTS / 7 ASR backends via /engines/*) - loopback security: system routes reject non-loopback origins (403) - DB migration: alembic UPGRADE on the seeded omnivoice_data fixture - Coverage Critic: every declared layer still has a spec (gate) + API inventory - dictation: streaming-ASR WebSocket /ws/transcribe registered + handshake - voice design: reuses the audio-correctness ladder - real ASR round-trip: enable-on-demand (PROBE_E2E=1) Enriched _boot_runner.py to capture engines/asr/loopback/openapi/ws in ONE isolated boot (conftest boot_capture session fixture); added env.seeded_data_dir. 13 specs total. probe suite 74 passed / 5 skipped; full repo 687 passed, 0 failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(probe): address all 15 unresolved review findings on #247 - coverage.py:22 — use `with open(...)` context to close spec files after yaml.safe_load (file handle leak) - _boot_runner.py:80 — store only `type(exc).__name__` for WS errors; drop raw str(exc) that could leak home paths / secrets into capture JSON - _boot_runner.py:99 — snapshot DB files before boot; set db_created=True only when boot creates NEW files (not when fixture already had one) - dubbing.py:46 — FAIL segments_duration_ratio when validated==0 (guards against empty/corrupt segment list passing vacuously) - i18n.py:49 — FAIL locale_valid_json when locales_dir is empty/missing - i18n.py:7 — fix docstring: locale_no_orphan_keys is advisory, not blocking - test_probe_i18n.py:59 — assert r.passed is False, not just r.advisory - coverage_critic.probe.yaml:15 — add "meta" to required layers list - dub_export.probe.yaml:17 — capture dub_audio in steps before advisory reads it - migration.probe.yaml:13 — add path_exists(db_path) data-integrity check - test_probe_asr_e2e.py:33 — os.path.exists → os.path.isfile for PROBE_ASR_SAMPLE - test_probe_migration.py:24 — assert context["db_path"] (presence) not db_created (new creation), aligning with the boot_runner fix Two findings intentionally skipped with reasons (see review thread replies): test_probe_design.py:36 — offline pattern is intentional; actor step is bypassed by design throughout the probe suite for CI compatibility test_probe_engines.py:22 — whisperx pin is intentional; it verifies the shipped default ASR engine is available out-of-the-box Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(probe): ASCII x in dubbing detail (ruff) + run migration judges inside seeded dir Two regressions from the hardening pass: - dubbing.py: replace non-ASCII '×' with 'x' (Ruff ambiguous-unicode → Tests lint fail) - test_probe_migration: move run_judges inside the seeded_data_dir with-block so the new path_exists check sees the DB before the temp dir is torn down (was always failing) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
33 lines
1.2 KiB
Python
33 lines
1.2 KiB
Python
"""Coverage Critic — enumerate the OpenAPI surface + probe specs, gate that every
|
|
declared layer still has a spec, and report (advisory) the API inventory."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
from . import spec as probe_spec
|
|
from .judges import coverage as C
|
|
|
|
_SPECS_DIR = os.path.join(os.path.dirname(__file__), "specs")
|
|
_SPEC = os.path.join(_SPECS_DIR, "coverage_critic.probe.yaml")
|
|
|
|
|
|
def test_coverage_critic(probe_report, boot_capture):
|
|
specs = C.scan_specs(_SPECS_DIR)
|
|
spec = probe_spec.load_spec(_SPEC)
|
|
ctx = {"openapi_paths": boot_capture["openapi_paths"], "specs": specs}
|
|
results = probe_spec.run_judges(spec, ctx)
|
|
probe_report.record(spec, results)
|
|
assert probe_spec.blocking_failures(results) == [], "\n".join(str(r) for r in results)
|
|
# The critic actually enumerated a real surface.
|
|
assert len(boot_capture["openapi_paths"]) > 100
|
|
assert len(specs) >= 10
|
|
|
|
|
|
def test_layers_have_specs_detects_gap():
|
|
specs = [{"layer": "media"}, {"layer": "env"}]
|
|
ok = C.layers_have_specs(specs, ["media", "env"])
|
|
assert ok.passed is True
|
|
gap = C.layers_have_specs(specs, ["media", "env", "desktop"])
|
|
assert gap.passed is False and "desktop" in gap.detail
|