Ports Patter's eval harness (MIT, attribution headers) into tests/evals/
with the judge transport swapped to services/llm_backend.py — the judge
runs against whatever local Ollama/LM Studio/OpenAI-compat endpoint the
user configured, keeping local-first. Both Patter hardening details kept
verbatim: verdict recomputed locally from the score (hallucinated
'passed: true' at score 0.2 fails), and tolerant JSON parsing (fences
stripped, invalid JSON -> fail-with-reasoning). Per-case containment:
agent exceptions keep the partial transcript and still judge it; a judge
failure records score 0 instead of aborting the suite.
HARD RULE preserved: LLM judges never gate CI. The scheduled workflow
(weekly + dispatch) is continue-on-error with the JSON report as artifact;
run_evals.py exits 0 always and skips cleanly when the active LLM backend
is 'off'. Deterministic probe judges remain the only gates; the harness
unit tests (10, no LLM needed) do run in gating CI.
First suite: dub translation naturalness v1 (4 cases) driving the real
cinematic_refine_sync reflect+adapt chain. The telephony-specific
session/assertions layers were deliberately not ported. The
dictation-refinement suite lands with Wave 1.1/2.1.
Spec: docs/competitive-analysis.md Spec 9b / parity program Wave 0.3.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
