Palash DebnathandClaude Opus 4.8
ef98aae4db
* test(probe): expand coverage — dubbing, i18n, engines, security, migration, dictation, design, coverage-critic Broadens the probe harness from one happy-path spec per layer to whole-app feature coverage (web, backend, dictation, clone, design), keeping the Actor/Judge split and offline-by-default + enable-on-demand for heavy paths. New specs + judges (one subprocess boot shared across backend-touching specs): - dubbing (L4): segment duration-ratio, SRT/VTT well-formed, export-archive contents, output language-ID (advisory) - i18n: locale files valid JSON (gate); orphan-keys + coverage (advisory). NOTE: surfaced a real bug — all 20 non-en locales carry gallery.cat_*/ bootstrap.lines keys absent from the en reference (reported, not gated). - engine matrix: active engine available + every unavailable engine explains why (11 TTS / 7 ASR backends via /engines/*) - loopback security: system routes reject non-loopback origins (403) - DB migration: alembic UPGRADE on the seeded omnivoice_data fixture - Coverage Critic: every declared layer still has a spec (gate) + API inventory - dictation: streaming-ASR WebSocket /ws/transcribe registered + handshake - voice design: reuses the audio-correctness ladder - real ASR round-trip: enable-on-demand (PROBE_E2E=1) Enriched _boot_runner.py to capture engines/asr/loopback/openapi/ws in ONE isolated boot (conftest boot_capture session fixture); added env.seeded_data_dir. 13 specs total. probe suite 74 passed / 5 skipped; full repo 687 passed, 0 failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(probe): address all 15 unresolved review findings on #247 - coverage.py:22 — use `with open(...)` context to close spec files after yaml.safe_load (file handle leak) - _boot_runner.py:80 — store only `type(exc).__name__` for WS errors; drop raw str(exc) that could leak home paths / secrets into capture JSON - _boot_runner.py:99 — snapshot DB files before boot; set db_created=True only when boot creates NEW files (not when fixture already had one) - dubbing.py:46 — FAIL segments_duration_ratio when validated==0 (guards against empty/corrupt segment list passing vacuously) - i18n.py:49 — FAIL locale_valid_json when locales_dir is empty/missing - i18n.py:7 — fix docstring: locale_no_orphan_keys is advisory, not blocking - test_probe_i18n.py:59 — assert r.passed is False, not just r.advisory - coverage_critic.probe.yaml:15 — add "meta" to required layers list - dub_export.probe.yaml:17 — capture dub_audio in steps before advisory reads it - migration.probe.yaml:13 — add path_exists(db_path) data-integrity check - test_probe_asr_e2e.py:33 — os.path.exists → os.path.isfile for PROBE_ASR_SAMPLE - test_probe_migration.py:24 — assert context["db_path"] (presence) not db_created (new creation), aligning with the boot_runner fix Two findings intentionally skipped with reasons (see review thread replies): test_probe_design.py:36 — offline pattern is intentional; actor step is bypassed by design throughout the probe suite for CI compatibility test_probe_engines.py:22 — whisperx pin is intentional; it verifies the shipped default ASR engine is available out-of-the-box Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(probe): ASCII x in dubbing detail (ruff) + run migration judges inside seeded dir Two regressions from the hardening pass: - dubbing.py: replace non-ASCII '×' with 'x' (Ruff ambiguous-unicode → Tests lint fail) - test_probe_migration: move run_judges inside the seeded_data_dir with-block so the new path_exists check sees the DB before the temp dir is torn down (was always failing) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
81 lines
3.3 KiB
Python
81 lines
3.3 KiB
Python
"""probe pytest plumbing: a session-scoped result recorder that emits an HTML
|
|
report (and opens it) when the probe suite finishes.
|
|
|
|
Tests record outcomes via the ``probe_report`` fixture. At session end the
|
|
recorded outcomes are rendered to ``tests/probe/reports/report-*.html`` and
|
|
opened in the browser (suppressed in CI / headless / when ``PROBE_NO_OPEN``).
|
|
|
|
This conftest is imported by pytest as ``probe.conftest`` (the probe package),
|
|
so relative imports resolve; a sys.path fallback covers odd invocations.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
try: # normal case: loaded as part of the `probe` package
|
|
from .report import Report, SpecOutcome, save_and_open
|
|
from .spec import JudgeResult, Spec
|
|
except ImportError: # pragma: no cover - defensive for unusual rootdirs
|
|
import os
|
|
import sys
|
|
|
|
sys.path.insert(0, os.path.dirname(__file__))
|
|
from report import Report, SpecOutcome, save_and_open # type: ignore
|
|
from spec import JudgeResult, Spec # type: ignore
|
|
|
|
|
|
class ProbeRecorder:
|
|
"""Collects per-spec outcomes during a session for the final HTML report."""
|
|
|
|
def __init__(self) -> None:
|
|
self.outcomes: list[SpecOutcome] = []
|
|
|
|
def record(self, spec: Spec, results: list[JudgeResult], duration_s: float = 0.0) -> None:
|
|
self.outcomes.append(SpecOutcome.from_spec(spec, results, duration_s=duration_s))
|
|
|
|
def record_group(self, name: str, results: list[JudgeResult], *, layer: str = "", duration_s: float = 0.0) -> None:
|
|
self.outcomes.append(SpecOutcome(name=name, feature=name, layer=layer, results=list(results), duration_s=duration_s))
|
|
|
|
|
|
def pytest_configure(config: pytest.Config) -> None:
|
|
config.addinivalue_line("markers", "api_fuzz: L1 Schemathesis property fuzzing of the API")
|
|
if not hasattr(config, "_probe_recorder"):
|
|
config._probe_recorder = ProbeRecorder() # type: ignore[attr-defined]
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def probe_report(request: pytest.FixtureRequest) -> ProbeRecorder:
|
|
return request.config._probe_recorder # type: ignore[attr-defined]
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def boot_capture() -> dict:
|
|
"""Boot the backend ONCE (fresh data dir, subprocess-isolated) and share the
|
|
rich capture — first-run endpoints, engine/ASR matrices, loopback-reject
|
|
status, and the OpenAPI inventory — across the engine/security/coverage
|
|
probes so the suite pays for only one boot."""
|
|
from . import env
|
|
|
|
with env.fresh_data_dir() as data_dir:
|
|
return env.capture_first_run(data_dir)
|
|
|
|
|
|
def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None:
|
|
recorder: ProbeRecorder | None = getattr(session.config, "_probe_recorder", None)
|
|
if not recorder or not recorder.outcomes:
|
|
return
|
|
report = Report(outcomes=recorder.outcomes)
|
|
# When there are blocking failures, the Triager drafts a prefilled GitHub
|
|
# issue URL and the report renders a one-click "Draft GitHub issue" button.
|
|
if report.failed:
|
|
try:
|
|
from .triage import triage
|
|
|
|
report.issue_url = triage(report).url
|
|
except Exception: # noqa: BLE001 — triage is best-effort, never break reporting
|
|
pass
|
|
# Writes the HTML and opens it in the browser. Opening is auto-suppressed in
|
|
# CI / headless / when PROBE_NO_OPEN is set (see report._should_open).
|
|
save_and_open(report)
|