Palash DebnathandClaude Fable 5
1195b4e0dd
Ports Patter's eval harness (MIT, attribution headers) into tests/evals/ with the judge transport swapped to services/llm_backend.py — the judge runs against whatever local Ollama/LM Studio/OpenAI-compat endpoint the user configured, keeping local-first. Both Patter hardening details kept verbatim: verdict recomputed locally from the score (hallucinated 'passed: true' at score 0.2 fails), and tolerant JSON parsing (fences stripped, invalid JSON -> fail-with-reasoning). Per-case containment: agent exceptions keep the partial transcript and still judge it; a judge failure records score 0 instead of aborting the suite. HARD RULE preserved: LLM judges never gate CI. The scheduled workflow (weekly + dispatch) is continue-on-error with the JSON report as artifact; run_evals.py exits 0 always and skips cleanly when the active LLM backend is 'off'. Deterministic probe judges remain the only gates; the harness unit tests (10, no LLM needed) do run in gating CI. First suite: dub translation naturalness v1 (4 cases) driving the real cinematic_refine_sync reflect+adapt chain. The telephony-specific session/assertions layers were deliberately not ported. The dictation-refinement suite lands with Wave 1.1/2.1. Spec: docs/competitive-analysis.md Spec 9b / parity program Wave 0.3. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
18 lines
907 B
Python
18 lines
907 B
Python
"""LLM-judge eval tier (parity program Wave 0.3, Spec 9b).
|
|
|
|
Semantic evaluation of outputs that deterministic probe judges can't score
|
|
(dub translation naturalness, dictation-refinement quality). HARD RULE:
|
|
these evals NEVER gate CI — they run as a separate non-blocking scheduled
|
|
job (.github/workflows/evals.yml) whose report lands as an artifact.
|
|
Deterministic probe judges (tests/probe/judges/) remain the only gates.
|
|
|
|
Harness adapted from Patter (https://github.com/PatterAI/Patter),
|
|
MIT License, Copyright (c) 2026 Patter Contributors. The telephony-specific
|
|
session/assertions layers were intentionally not ported; the judge backend
|
|
is swapped to OmniVoice's local-first LLM adapter (services.llm_backend).
|
|
"""
|
|
|
|
from .case import EvalCase, EvalResult, EvalTurn, JudgeResult # noqa: F401
|
|
from .judge import LLMJudge # noqa: F401
|
|
from .runner import EvalRunner, EvalSuite, load_suite # noqa: F401
|