* test(probe): add spec-driven AI-agent test harness (L1/L2/L4/L5 + report + triage) Introduces `tests/probe/`, a portable, mostly-deterministic test harness built on the Actor/Judge split: AI agents may drive and self-heal, but verdicts are always deterministic code + metrics — no LLM on the verdict path. Layers: - L1 API: Schemathesis property-fuzz over in-process ASGI (enable-on-demand). - L2 web: Playwright Driver + deterministic self-heal (id→test-id→text, loosened CSS) → pluggable Healer; LLMHealer/anthropic_healer for genuine agentic heal. Judges + self-heal logic unit-tested offline via FakePage; live browser skips. - L4 media: audio correctness — exists/decode/duration/not-silent/clipping/NaN, round-trip ASR WER (pure-python, faster-whisper backend), speaker similarity. No golden-WAV (device-stable metrics only); naturalness is advisory-only. - L5 env/first-run: fresh-data-dir backend boot in a SUBPROCESS (no session contamination), asserts health + DB init + endpoint reachability. Docker gated. Plus: hybrid YAML spec engine + JudgeResult/registry; self-contained HTML report that auto-opens (suppressed in CI/headless/PROBE_NO_OPEN); Triager that clusters failures and drafts a prefilled GitHub issue URL (sanitized, no auto-submit) with a one-click button in the report. Dependency-light: runs in the base venv; schemathesis/resemblyzer/playwright/ anthropic are enable-on-demand and skip cleanly. Generated reports gitignored. Full suite green (657 passed); no contamination of existing tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(probe): add L3 desktop layer (Tauri config-integrity + guarded launch) Per the architecture decision, desktop E2E is substituted by backend-over-HTTP (L5) + browser (L2) since Tauri has no official macOS WebDriver. L3 guards the packaging/shell contract a browser test can't see, against the real tauri.conf.json (with platform-override merge), running on any platform with no Tauri toolchain: - version parity between tauri.conf.json and pyproject (release integrity) - dev/build wiring (devUrl matches the Vite frontend, frontendDist, before* cmds) - bundled binaries first-run depends on (uv / ffmpeg / ffprobe in externalBin) - CSP actually permits the local backend origins (desktop-only failure mode: packaged app can't reach :3900 while the browser build works) Adds desktop.py (config load + platform deep-merge + bundle discovery + launch guard), judges/desktop.py (config_present/config_eq/config_contains/csp_allows), desktop_smoke.probe.yaml, and tests covering integrity, platform-merge replace semantics, and a live bundle launch that skips without a built bundle/display. Full suite green (662 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
46 lines
1.8 KiB
Python
46 lines
1.8 KiB
Python
"""L1 API contract/property fuzzing via Schemathesis (enable-on-demand).
|
|
|
|
Schemathesis reads the FastAPI app's own OpenAPI schema and autonomously
|
|
generates inputs that probe every operation for: unhandled 500s, responses that
|
|
violate the declared schema, and validation bypasses. It is the highest-ROI
|
|
*autonomous* bug finder available (independent studies rank it best/second-best
|
|
among API fuzzers) — and it needs no hand-written assertions.
|
|
|
|
It runs against the app **in-process over ASGI** (no live server, no port), the
|
|
same way tests/test_api.py uses FastAPI's TestClient.
|
|
|
|
Enable it with one command:
|
|
|
|
uv add schemathesis # or: uv pip install schemathesis
|
|
|
|
Until then, tests/probe/test_api_fuzz.py skips cleanly so CI stays green.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
|
|
def load_app():
|
|
"""Return the FastAPI ASGI app. Mirrors the import path used across the
|
|
suite (conftest puts ``backend/`` on sys.path with ``--app-dir backend``)."""
|
|
from main import app # noqa: PLC0415 — must import after conftest sys.path setup
|
|
|
|
return app
|
|
|
|
|
|
def build_schema(app=None):
|
|
"""Build a Schemathesis schema from the in-process ASGI app, tolerating the
|
|
API rename between Schemathesis 3.x and 4.x. Raises ImportError if the
|
|
package isn't installed (callers guard with pytest.importorskip)."""
|
|
import schemathesis
|
|
|
|
if app is None:
|
|
app = load_app()
|
|
|
|
# 4.x: schemathesis.openapi.from_asgi(path, app); 3.x: schemathesis.from_asgi
|
|
from_asgi = getattr(getattr(schemathesis, "openapi", None), "from_asgi", None)
|
|
if from_asgi is None:
|
|
from_asgi = getattr(schemathesis, "from_asgi", None)
|
|
if from_asgi is None: # pragma: no cover - version we don't recognise
|
|
raise RuntimeError("Schemathesis present but no from_asgi entrypoint found")
|
|
return from_asgi("/openapi.json", app)
|