test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)
Ports Patter's eval harness (MIT, attribution headers) into tests/evals/ with the judge transport swapped to services/llm_backend.py — the judge runs against whatever local Ollama/LM Studio/OpenAI-compat endpoint the user configured, keeping local-first. Both Patter hardening details kept verbatim: verdict recomputed locally from the score (hallucinated 'passed: true' at score 0.2 fails), and tolerant JSON parsing (fences stripped, invalid JSON -> fail-with-reasoning). Per-case containment: agent exceptions keep the partial transcript and still judge it; a judge failure records score 0 instead of aborting the suite. HARD RULE preserved: LLM judges never gate CI. The scheduled workflow (weekly + dispatch) is continue-on-error with the JSON report as artifact; run_evals.py exits 0 always and skips cleanly when the active LLM backend is 'off'. Deterministic probe judges remain the only gates; the harness unit tests (10, no LLM needed) do run in gating CI. First suite: dub translation naturalness v1 (4 cases) driving the real cinematic_refine_sync reflect+adapt chain. The telephony-specific session/assertions layers were deliberately not ported. The dictation-refinement suite lands with Wave 1.1/2.1. Spec: docs/competitive-analysis.md Spec 9b / parity program Wave 0.3. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Palash Debnath
co-authored by
Claude Fable 5
Claude Fable 5
parent
11c498eeb5
commit
1195b4e0dd
@@ -0,0 +1,55 @@
|
||||
# LLM-judge evals — semantic quality suites, NEVER a gate.
|
||||
#
|
||||
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
|
||||
# judges never gate CI. This workflow is scheduled + manual only, the eval
|
||||
# step is continue-on-error, and the JSON report is the deliverable
|
||||
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
|
||||
# the only gates.
|
||||
#
|
||||
# On the hosted runner there is no local LLM endpoint, so the run usually
|
||||
# reports "skipped — no LLM backend configured"; the workflow exists so the
|
||||
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
|
||||
# (e.g. a self-hosted runner with Ollama).
|
||||
|
||||
name: evals
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Weekly, Sundays 04:00 UTC.
|
||||
- cron: "0 4 * * 0"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
evals:
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- uses: astral-sh/setup-uv@v3
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
|
||||
- name: Install deps
|
||||
run: uv sync
|
||||
|
||||
- name: Run eval suites (non-gating)
|
||||
continue-on-error: true
|
||||
env:
|
||||
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
|
||||
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
|
||||
run: uv run python tests/evals/run_evals.py --output eval-report.json
|
||||
|
||||
- name: Upload report artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: eval-report
|
||||
path: eval-report.json
|
||||
if-no-files-found: warn
|
||||
@@ -0,0 +1,17 @@
|
||||
"""LLM-judge eval tier (parity program Wave 0.3, Spec 9b).
|
||||
|
||||
Semantic evaluation of outputs that deterministic probe judges can't score
|
||||
(dub translation naturalness, dictation-refinement quality). HARD RULE:
|
||||
these evals NEVER gate CI — they run as a separate non-blocking scheduled
|
||||
job (.github/workflows/evals.yml) whose report lands as an artifact.
|
||||
Deterministic probe judges (tests/probe/judges/) remain the only gates.
|
||||
|
||||
Harness adapted from Patter (https://github.com/PatterAI/Patter),
|
||||
MIT License, Copyright (c) 2026 Patter Contributors. The telephony-specific
|
||||
session/assertions layers were intentionally not ported; the judge backend
|
||||
is swapped to OmniVoice's local-first LLM adapter (services.llm_backend).
|
||||
"""
|
||||
|
||||
from .case import EvalCase, EvalResult, EvalTurn, JudgeResult # noqa: F401
|
||||
from .judge import LLMJudge # noqa: F401
|
||||
from .runner import EvalRunner, EvalSuite, load_suite # noqa: F401
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Eval case data model.
|
||||
|
||||
Adapted from Patter (https://github.com/PatterAI/Patter), MIT License,
|
||||
Copyright (c) 2026 Patter Contributors.
|
||||
|
||||
An :class:`EvalCase` is either a scripted conversation (``turns``) or — the
|
||||
OmniVoice extension — a structured ``input`` mapping handed verbatim to the
|
||||
system under test (e.g. a dub segment with source/literal/langs). Both shapes
|
||||
produce a role-tagged transcript that the judge LLM scores against
|
||||
``expected_behavior`` + ``rubric``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvalTurn:
|
||||
"""A single user utterance in a scripted conversation."""
|
||||
|
||||
user: str
|
||||
# Optional substrings the reply should contain — a cheap pre-filter
|
||||
# logged before the judge runs (the judge still decides).
|
||||
expected_contains: tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvalCase:
|
||||
"""A complete evaluation scenario."""
|
||||
|
||||
name: str
|
||||
expected_behavior: str
|
||||
rubric: str
|
||||
turns: tuple[EvalTurn, ...] = field(default_factory=tuple)
|
||||
# OmniVoice extension: structured input for non-conversational systems
|
||||
# under test (translator, refiner). When set, ``turns`` is ignored and
|
||||
# the agent callable receives this mapping.
|
||||
input: dict[str, Any] = field(default_factory=dict)
|
||||
tags: tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class JudgeResult:
|
||||
"""The judge's verdict on one case."""
|
||||
|
||||
score: float # 0.0-1.0
|
||||
passed: bool
|
||||
reasoning: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvalResult:
|
||||
"""The result of running a single :class:`EvalCase`."""
|
||||
|
||||
case_name: str
|
||||
transcript: tuple[dict[str, str], ...] # [{"role": "user"|"agent", "text": ...}]
|
||||
judge: JudgeResult
|
||||
duration_s: float
|
||||
error: str | None = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"case": self.case_name,
|
||||
"score": self.judge.score,
|
||||
"passed": self.judge.passed,
|
||||
"reasoning": self.judge.reasoning,
|
||||
"transcript": list(self.transcript),
|
||||
"duration_s": round(self.duration_s, 3),
|
||||
"error": self.error,
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
"""LLM-as-judge scoring for eval cases.
|
||||
|
||||
Adapted from Patter (https://github.com/PatterAI/Patter), MIT License,
|
||||
Copyright (c) 2026 Patter Contributors. Two hardening details are ported
|
||||
verbatim by design:
|
||||
|
||||
* the verdict is recomputed LOCALLY (``passed = score >= threshold``) —
|
||||
trusting the model's self-reported ``passed`` once let a hallucinated
|
||||
``passed: true`` with ``score: 0.2`` record a pass;
|
||||
* JSON parsing is tolerant (code fences stripped; invalid JSON becomes a
|
||||
fail-with-reasoning, never a crash).
|
||||
|
||||
The OpenAI-specific client is replaced by OmniVoice's local-first LLM
|
||||
adapter (``services.llm_backend``) — the judge runs against whatever
|
||||
Ollama/LM Studio/OpenAI-compat endpoint the user configured, keeping the
|
||||
no-required-cloud guarantee. Any object exposing ``judge(prompt) -> str``
|
||||
(async) can be injected via ``backend=`` for tests.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from .case import EvalCase, JudgeResult
|
||||
|
||||
logger = logging.getLogger("omnivoice.evals")
|
||||
|
||||
|
||||
_JUDGE_SYSTEM = (
|
||||
"You are a strict but fair evaluator of a local voice studio's text "
|
||||
"outputs (translations, cleaned-up transcripts). You will be given: "
|
||||
"(1) the expected behavior, (2) a rubric, (3) a transcript of inputs "
|
||||
"and the system's output. "
|
||||
"Return a JSON object with exactly three keys:\n"
|
||||
' - "score": float between 0.0 and 1.0\n'
|
||||
' - "passed": boolean (true when score >= threshold)\n'
|
||||
' - "reasoning": short string explaining the score\n'
|
||||
"Do not return any text outside the JSON object."
|
||||
)
|
||||
|
||||
|
||||
class _LLMBackendJudge:
|
||||
"""Default judge transport: OmniVoice's active LLM backend."""
|
||||
|
||||
def __init__(self, timeout: float = 120.0) -> None:
|
||||
self._timeout = timeout
|
||||
self._backend: Any = None
|
||||
|
||||
def _resolve(self):
|
||||
if self._backend is None:
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
|
||||
self._backend = get_active_llm_backend()
|
||||
return self._backend
|
||||
|
||||
async def judge(self, prompt: str) -> str:
|
||||
backend = self._resolve()
|
||||
# LLMBackend.chat is sync; keep the judge loop responsive.
|
||||
return await asyncio.to_thread(
|
||||
backend.chat, system=_JUDGE_SYSTEM, user=prompt, timeout=self._timeout
|
||||
)
|
||||
|
||||
|
||||
class LLMJudge:
|
||||
"""Scores case transcripts against a rubric via the configured LLM."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
pass_threshold: float = 0.7,
|
||||
backend: Any = None,
|
||||
) -> None:
|
||||
self.pass_threshold = pass_threshold
|
||||
self._backend = backend or _LLMBackendJudge()
|
||||
|
||||
async def judge_case(
|
||||
self, case: EvalCase, transcript: list[dict[str, str]]
|
||||
) -> JudgeResult:
|
||||
prompt = self._build_prompt(case, transcript)
|
||||
raw = await self._backend.judge(prompt)
|
||||
return self._parse(raw)
|
||||
|
||||
def _build_prompt(self, case: EvalCase, transcript: list[dict[str, str]]) -> str:
|
||||
lines = [
|
||||
f"EXPECTED BEHAVIOR: {case.expected_behavior}",
|
||||
f"RUBRIC: {case.rubric}",
|
||||
f"PASS THRESHOLD: {self.pass_threshold}",
|
||||
"TRANSCRIPT:",
|
||||
]
|
||||
for turn in transcript:
|
||||
lines.append(f" {turn.get('role', '?')}: {turn.get('text', '')}")
|
||||
return "\n".join(lines)
|
||||
|
||||
def _parse(self, raw: str) -> JudgeResult:
|
||||
text = (raw or "").strip()
|
||||
if text.startswith("```"):
|
||||
text = re.sub(r"^```(?:json)?\s*", "", text)
|
||||
text = re.sub(r"\s*```$", "", text)
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("LLMJudge: invalid JSON, defaulting to fail: %r", raw)
|
||||
return JudgeResult(
|
||||
score=0.0,
|
||||
passed=False,
|
||||
reasoning=f"Judge returned invalid JSON: {(raw or '')[:200]}",
|
||||
)
|
||||
try:
|
||||
score = float(data.get("score", 0.0))
|
||||
except (TypeError, ValueError):
|
||||
score = 0.0
|
||||
score = max(0.0, min(1.0, score))
|
||||
# Verdict computed locally — never trust the model's own `passed`.
|
||||
passed = score >= self.pass_threshold
|
||||
return JudgeResult(score=score, passed=passed, reasoning=str(data.get("reasoning", "")))
|
||||
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the LLM-judge eval suites and write a JSON report artifact.
|
||||
|
||||
NEVER a CI gate (parity program Wave 0.3 hard rule): exits 0 whether cases
|
||||
pass or fail — the report is the deliverable. Exits 0 with a skip notice
|
||||
when no LLM backend is configured (the scheduled CI runner has none; the
|
||||
suites are meant for machines with a local Ollama/LM Studio endpoint).
|
||||
|
||||
Usage:
|
||||
uv run python tests/evals/run_evals.py --output eval-report.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(_HERE.parent)) # tests/ -> import evals.*
|
||||
sys.path.insert(0, str(_HERE.parents[1] / "backend")) # services.*
|
||||
|
||||
SUITES_DIR = _HERE / "suites"
|
||||
|
||||
|
||||
def _llm_ready() -> tuple[bool, str]:
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
|
||||
backend = get_active_llm_backend()
|
||||
# OffBackend reports available=True (it is a valid no-op choice) but its
|
||||
# chat() raises — for evals "ready" means a real endpoint is configured.
|
||||
if backend.id == "off":
|
||||
return False, "active LLM backend is 'off'"
|
||||
ok, msg = type(backend).is_available()
|
||||
return ok, f"{backend.display_name}: {msg}"
|
||||
|
||||
|
||||
async def _translator_agent(case_input: dict) -> str:
|
||||
"""System under test for the dub-naturalness suite."""
|
||||
from services.translator import cinematic_refine_sync
|
||||
|
||||
result = await asyncio.to_thread(
|
||||
cinematic_refine_sync,
|
||||
case_input["source"],
|
||||
case_input["literal"],
|
||||
source_lang=case_input.get("source_lang", "en"),
|
||||
target_lang=case_input["target_lang"],
|
||||
)
|
||||
if result.get("error"):
|
||||
raise RuntimeError(f"translator error: {result['error']}")
|
||||
return result["text"]
|
||||
|
||||
|
||||
_AGENTS = {
|
||||
"dub_translation_naturalness": lambda: _translator_agent,
|
||||
}
|
||||
|
||||
|
||||
async def _main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--suite", default="all", help="suite stem or 'all'")
|
||||
parser.add_argument("--output", type=Path, default=Path("eval-report.json"))
|
||||
parser.add_argument("--pass-threshold", type=float, default=0.7)
|
||||
args = parser.parse_args()
|
||||
|
||||
ok, detail = _llm_ready()
|
||||
if not ok:
|
||||
print(f"evals: SKIPPED — no LLM backend configured ({detail}).")
|
||||
print("Configure TRANSLATE_BASE_URL (Ollama/LM Studio/OpenAI-compat) to run.")
|
||||
args.output.write_text(
|
||||
json.dumps({"skipped": True, "reason": detail}, indent=2), encoding="utf-8"
|
||||
)
|
||||
return 0
|
||||
|
||||
from evals.judge import LLMJudge
|
||||
from evals.runner import EvalRunner, load_suite
|
||||
|
||||
stems = (
|
||||
sorted(p.stem for p in SUITES_DIR.glob("*.yaml"))
|
||||
if args.suite == "all"
|
||||
else [args.suite]
|
||||
)
|
||||
reports = []
|
||||
for stem in stems:
|
||||
factory = _AGENTS.get(stem)
|
||||
if factory is None:
|
||||
print(f"evals: no agent registered for suite {stem!r}, skipping.")
|
||||
continue
|
||||
suite = load_suite(SUITES_DIR / f"{stem}.yaml")
|
||||
runner = EvalRunner(judge=LLMJudge(pass_threshold=args.pass_threshold))
|
||||
results = await runner.run(suite, factory)
|
||||
report = json.loads(runner.report(suite, results))
|
||||
reports.append(report)
|
||||
print(
|
||||
f"evals: {suite.name} — {report['passed']}/{report['total']} passed "
|
||||
f"(rate {report['pass_rate']:.2f})"
|
||||
)
|
||||
|
||||
args.output.write_text(json.dumps({"suites": reports}, indent=2), encoding="utf-8")
|
||||
print(f"evals: report written to {args.output}")
|
||||
return 0 # informational by design — failures live in the report
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(_main()))
|
||||
@@ -0,0 +1,179 @@
|
||||
"""Eval runner — executes an :class:`EvalSuite` and produces a JSON report.
|
||||
|
||||
Adapted from Patter (https://github.com/PatterAI/Patter), MIT License,
|
||||
Copyright (c) 2026 Patter Contributors. The real-pipeline ``EvalSession``
|
||||
path (telephony-specific) was not ported; OmniVoice cases either script
|
||||
``turns`` against an async ``reply(text) -> str`` callable, or carry a
|
||||
structured ``input`` mapping handed to an async ``run(input) -> str``
|
||||
callable (the system under test: translator, refiner, ...).
|
||||
|
||||
Per-case error containment is preserved verbatim: a mid-case exception keeps
|
||||
the partial transcript and still judges it; a judge failure records
|
||||
``score 0 + reasoning`` instead of aborting the whole suite.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Awaitable, Callable
|
||||
|
||||
from .case import EvalCase, EvalResult, EvalTurn, JudgeResult
|
||||
from .judge import LLMJudge
|
||||
|
||||
logger = logging.getLogger("omnivoice.evals")
|
||||
|
||||
# ``turns`` cases: factory returns an async ``reply(text) -> str``.
|
||||
# ``input`` cases: factory returns an async ``run(input: dict) -> str``.
|
||||
AgentCallable = Callable[[Any], Awaitable[str]]
|
||||
AgentFactory = Callable[[], AgentCallable]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvalSuite:
|
||||
"""A named collection of :class:`EvalCase` to run together."""
|
||||
|
||||
name: str
|
||||
cases: tuple[EvalCase, ...]
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class EvalRunner:
|
||||
def __init__(self, judge: LLMJudge | None = None) -> None:
|
||||
self.judge = judge or LLMJudge()
|
||||
|
||||
async def run(self, suite: EvalSuite, agent_factory: AgentFactory) -> list[EvalResult]:
|
||||
return [await self.run_case(case, agent_factory) for case in suite.cases]
|
||||
|
||||
async def run_case(self, case: EvalCase, agent_factory: AgentFactory) -> EvalResult:
|
||||
start = time.monotonic()
|
||||
transcript: list[dict[str, str]] = []
|
||||
error: str | None = None
|
||||
|
||||
try:
|
||||
agent = agent_factory()
|
||||
if case.input:
|
||||
# Structured-input path: render the input for the judge,
|
||||
# then hand the mapping to the system under test.
|
||||
rendered = "\n".join(f"{k}: {v}" for k, v in case.input.items())
|
||||
transcript.append({"role": "user", "text": rendered})
|
||||
reply = await agent(dict(case.input))
|
||||
transcript.append({"role": "agent", "text": reply or ""})
|
||||
else:
|
||||
for turn in case.turns:
|
||||
transcript.append({"role": "user", "text": turn.user})
|
||||
reply = await agent(turn.user)
|
||||
transcript.append({"role": "agent", "text": reply or ""})
|
||||
self._log_missing_expected(case, turn, reply or "")
|
||||
except Exception as exc: # noqa: BLE001 — containment is the point
|
||||
error = f"{type(exc).__name__}: {exc}"
|
||||
logger.exception("case=%r raised", case.name)
|
||||
|
||||
if error and not transcript:
|
||||
return EvalResult(
|
||||
case_name=case.name,
|
||||
transcript=tuple(transcript),
|
||||
judge=JudgeResult(score=0.0, passed=False, reasoning=error),
|
||||
duration_s=time.monotonic() - start,
|
||||
error=error,
|
||||
)
|
||||
|
||||
try:
|
||||
judge_result = await self.judge.judge_case(case, transcript)
|
||||
except Exception as exc: # noqa: BLE001 — judge 429/timeout/missing key
|
||||
# One transient judge failure must not abort the whole suite.
|
||||
return EvalResult(
|
||||
case_name=case.name,
|
||||
transcript=tuple(transcript),
|
||||
judge=JudgeResult(score=0.0, passed=False, reasoning=f"judge error: {exc}"),
|
||||
duration_s=time.monotonic() - start,
|
||||
error=f"judge error: {exc}",
|
||||
)
|
||||
return EvalResult(
|
||||
case_name=case.name,
|
||||
transcript=tuple(transcript),
|
||||
judge=judge_result,
|
||||
duration_s=time.monotonic() - start,
|
||||
error=error,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _log_missing_expected(case: EvalCase, turn: EvalTurn, reply: str) -> None:
|
||||
for needle in turn.expected_contains:
|
||||
if needle.lower() not in reply.lower():
|
||||
logger.info("case=%r expected_contains=%r missing in reply", case.name, needle)
|
||||
|
||||
def report(self, suite: EvalSuite, results: list[EvalResult]) -> str:
|
||||
"""Render a JSON report suitable for CI artifacts. Never a gate."""
|
||||
total = len(results)
|
||||
passed = sum(1 for r in results if r.judge.passed)
|
||||
payload = {
|
||||
"suite": suite.name,
|
||||
"total": total,
|
||||
"passed": passed,
|
||||
"failed": total - passed,
|
||||
"pass_rate": (passed / total) if total else 0.0,
|
||||
"cases": [r.to_dict() for r in results],
|
||||
}
|
||||
return json.dumps(payload, indent=2)
|
||||
|
||||
|
||||
def load_suite(path: Path) -> EvalSuite:
|
||||
"""Load a suite from YAML or JSON.
|
||||
|
||||
Schema (YAML)::
|
||||
|
||||
name: "dub translation naturalness v1"
|
||||
cases:
|
||||
- name: "idiom is adapted, not translated"
|
||||
expected_behavior: "The adapted line replaces the idiom ..."
|
||||
rubric: "Pass if ..."
|
||||
input:
|
||||
source: "It's raining cats and dogs."
|
||||
literal: "..."
|
||||
"""
|
||||
text = path.read_text(encoding="utf-8")
|
||||
if path.suffix.lower() in {".yaml", ".yml"}:
|
||||
import yaml
|
||||
|
||||
data = yaml.safe_load(text)
|
||||
else:
|
||||
data = json.loads(text)
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"Eval suite {path} must be a mapping, got {type(data).__name__}")
|
||||
cases_raw = data.get("cases", [])
|
||||
if not isinstance(cases_raw, list):
|
||||
raise ValueError(f"Eval suite {path}: 'cases' must be a list")
|
||||
|
||||
cases: list[EvalCase] = []
|
||||
for i, c in enumerate(cases_raw):
|
||||
if not isinstance(c, dict):
|
||||
raise ValueError(f"Eval suite {path}: case {i} must be a mapping")
|
||||
turns = tuple(
|
||||
EvalTurn(
|
||||
user=str(t.get("user", "")),
|
||||
expected_contains=tuple(t.get("expected_contains", []) or []),
|
||||
)
|
||||
for t in c.get("turns", []) or []
|
||||
if isinstance(t, dict)
|
||||
)
|
||||
cases.append(
|
||||
EvalCase(
|
||||
name=str(c.get("name", f"case_{i}")),
|
||||
turns=turns,
|
||||
input=dict(c.get("input", {}) or {}),
|
||||
expected_behavior=str(c.get("expected_behavior", "")),
|
||||
rubric=str(c.get("rubric", "")),
|
||||
tags=tuple(c.get("tags", []) or []),
|
||||
)
|
||||
)
|
||||
|
||||
return EvalSuite(
|
||||
name=str(data.get("name", path.stem)),
|
||||
cases=tuple(cases),
|
||||
metadata=dict(data.get("metadata", {}) or {}),
|
||||
)
|
||||
@@ -0,0 +1,70 @@
|
||||
# Dub translation naturalness v1 — semantic regression suite for the
|
||||
# cinematic translate chain (services/translator.py reflect+adapt pass).
|
||||
#
|
||||
# System under test: cinematic_refine_sync(source, literal, ...) — each case
|
||||
# feeds a source line + a deliberately stiff literal translation; the judge
|
||||
# scores whether the adapted line reads like natural dialogue. NEVER a CI
|
||||
# gate (parity program Wave 0.3 hard rule).
|
||||
name: dub translation naturalness v1
|
||||
metadata:
|
||||
system_under_test: services.translator.cinematic_refine_sync
|
||||
cases:
|
||||
- name: english idiom adapts to spanish equivalent
|
||||
tags: [idiom, es]
|
||||
input:
|
||||
source: "It's raining cats and dogs out there, take an umbrella."
|
||||
literal: "Esta lloviendo gatos y perros alla afuera, lleva un paraguas."
|
||||
source_lang: en
|
||||
target_lang: es
|
||||
expected_behavior: >
|
||||
The adapted Spanish line replaces the English idiom with a natural
|
||||
Spanish equivalent (e.g. "esta lloviendo a cantaros") instead of the
|
||||
word-for-word animal phrase, and keeps the umbrella advice.
|
||||
rubric: >
|
||||
Fail if the output still contains a literal cats-and-dogs construction.
|
||||
Pass if a fluent Spanish speaker would find the line natural and the
|
||||
meaning (heavy rain + take an umbrella) is preserved.
|
||||
|
||||
- name: register matches casual dialogue in german
|
||||
tags: [register, de]
|
||||
input:
|
||||
source: "No way, you've got to be kidding me!"
|
||||
literal: "Kein Weg, du musst mich verspotten!"
|
||||
source_lang: en
|
||||
target_lang: de
|
||||
expected_behavior: >
|
||||
The adapted German line is a natural colloquial exclamation (e.g.
|
||||
"Das gibt's doch nicht!" or "Das kann nicht dein Ernst sein!"), not the
|
||||
ungrammatical word-for-word rendering.
|
||||
rubric: >
|
||||
Fail if "Kein Weg" or "verspotten" survive. Pass if the line is
|
||||
idiomatic spoken German with the same incredulous register.
|
||||
|
||||
- name: contraction-heavy line stays speakable in french
|
||||
tags: [speakability, fr]
|
||||
input:
|
||||
source: "I dunno, it's kinda late — let's just call it a day."
|
||||
literal: "Je ne sais pas, c'est un peu tard - appelons-le simplement un jour."
|
||||
source_lang: en
|
||||
target_lang: fr
|
||||
expected_behavior: >
|
||||
The adapted French line drops the calqued "appelons-le un jour" and uses
|
||||
a natural equivalent (e.g. "on arrete la pour aujourd'hui"), keeping the
|
||||
casual hedging tone.
|
||||
rubric: >
|
||||
Fail if the call-it-a-day calque survives. Pass if the line is natural
|
||||
spoken French of similar length and casualness.
|
||||
|
||||
- name: honorific register preserved in formal speech
|
||||
tags: [register, formal, es]
|
||||
input:
|
||||
source: "Ladies and gentlemen, thank you for joining us this evening."
|
||||
literal: "Damas y caballeros, gracias por unirse a nosotros esta noche."
|
||||
source_lang: en
|
||||
target_lang: es
|
||||
expected_behavior: >
|
||||
The adapted line keeps the formal register of a public address — it must
|
||||
NOT become casual — while staying natural.
|
||||
rubric: >
|
||||
Fail if the register drops to informal. Pass if a native speaker would
|
||||
accept it as a natural formal opening of similar length.
|
||||
@@ -0,0 +1,163 @@
|
||||
"""Deterministic unit tests for the eval harness (no LLM required).
|
||||
|
||||
These DO run in gating CI — they test the harness mechanics, not semantic
|
||||
quality. The semantic suites themselves run only in the non-gating
|
||||
scheduled workflow (.github/workflows/evals.yml).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from evals.case import EvalCase, EvalTurn
|
||||
from evals.judge import LLMJudge
|
||||
from evals.runner import EvalRunner, EvalSuite, load_suite
|
||||
|
||||
SUITES_DIR = Path(__file__).resolve().parent / "suites"
|
||||
|
||||
|
||||
class FakeJudgeBackend:
|
||||
"""Canned judge transport: returns queued raw strings."""
|
||||
|
||||
def __init__(self, *responses: str):
|
||||
self._responses = list(responses)
|
||||
|
||||
async def judge(self, prompt: str) -> str:
|
||||
return self._responses.pop(0)
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.run(coro)
|
||||
|
||||
|
||||
def _echo_factory():
|
||||
async def reply(payload):
|
||||
return f"echo: {payload}"
|
||||
return reply
|
||||
|
||||
|
||||
# ── Judge hardening (ported from Patter verbatim — keep these honest) ──────
|
||||
|
||||
def test_verdict_recomputed_locally_ignores_hallucinated_passed():
|
||||
judge = LLMJudge(pass_threshold=0.7,
|
||||
backend=FakeJudgeBackend('{"score": 0.2, "passed": true, "reasoning": "nope"}'))
|
||||
case = EvalCase(name="c", expected_behavior="x", rubric="y",
|
||||
turns=(EvalTurn(user="hi"),))
|
||||
result = _run(judge.judge_case(case, [{"role": "user", "text": "hi"}]))
|
||||
assert result.passed is False # 0.2 < 0.7 regardless of the model's claim
|
||||
assert result.score == 0.2
|
||||
|
||||
|
||||
def test_judge_strips_code_fences():
|
||||
judge = LLMJudge(backend=FakeJudgeBackend('```json\n{"score": 0.9, "reasoning": "ok"}\n```'))
|
||||
case = EvalCase(name="c", expected_behavior="x", rubric="y")
|
||||
result = _run(judge.judge_case(case, []))
|
||||
assert result.passed is True and result.score == 0.9
|
||||
|
||||
|
||||
def test_invalid_judge_json_fails_with_reasoning_not_crash():
|
||||
judge = LLMJudge(backend=FakeJudgeBackend("I think it passes!"))
|
||||
case = EvalCase(name="c", expected_behavior="x", rubric="y")
|
||||
result = _run(judge.judge_case(case, []))
|
||||
assert result.passed is False
|
||||
assert "invalid JSON" in result.reasoning
|
||||
|
||||
|
||||
def test_score_clamped_to_unit_interval():
|
||||
judge = LLMJudge(backend=FakeJudgeBackend('{"score": 7, "reasoning": ""}'))
|
||||
result = _run(judge.judge_case(EvalCase(name="c", expected_behavior="", rubric=""), []))
|
||||
assert result.score == 1.0
|
||||
|
||||
|
||||
# ── Runner containment ──────────────────────────────────────────────────────
|
||||
|
||||
def test_agent_exception_keeps_partial_transcript_and_still_judges():
|
||||
calls = []
|
||||
|
||||
class SpyJudge(LLMJudge):
|
||||
async def judge_case(self, case, transcript):
|
||||
calls.append(list(transcript))
|
||||
return await super().judge_case(case, transcript)
|
||||
|
||||
judge = SpyJudge(backend=FakeJudgeBackend('{"score": 0.0, "reasoning": "partial"}'))
|
||||
|
||||
def factory():
|
||||
state = {"n": 0}
|
||||
|
||||
async def reply(text):
|
||||
state["n"] += 1
|
||||
if state["n"] == 2:
|
||||
raise RuntimeError("boom")
|
||||
return "ok"
|
||||
return reply
|
||||
|
||||
case = EvalCase(name="c", expected_behavior="x", rubric="y",
|
||||
turns=(EvalTurn(user="one"), EvalTurn(user="two")))
|
||||
result = _run(EvalRunner(judge=judge).run_case(case, factory))
|
||||
assert result.error == "RuntimeError: boom"
|
||||
# Partial transcript (turn one + its reply + turn two) was judged.
|
||||
assert calls and len(calls[0]) == 3
|
||||
|
||||
|
||||
def test_judge_failure_records_zero_not_suite_abort():
|
||||
class ExplodingBackend:
|
||||
async def judge(self, prompt):
|
||||
raise TimeoutError("judge LLM timed out")
|
||||
|
||||
suite = EvalSuite(name="s", cases=(
|
||||
EvalCase(name="a", expected_behavior="x", rubric="y", turns=(EvalTurn(user="hi"),)),
|
||||
EvalCase(name="b", expected_behavior="x", rubric="y", turns=(EvalTurn(user="hi"),)),
|
||||
))
|
||||
runner = EvalRunner(judge=LLMJudge(backend=ExplodingBackend()))
|
||||
results = _run(runner.run(suite, _echo_factory))
|
||||
assert len(results) == 2 # second case still ran
|
||||
assert all("judge error" in r.judge.reasoning for r in results)
|
||||
assert all(r.judge.passed is False for r in results)
|
||||
|
||||
|
||||
def test_structured_input_case_renders_transcript():
|
||||
judge = LLMJudge(backend=FakeJudgeBackend('{"score": 1.0, "reasoning": "ok"}'))
|
||||
case = EvalCase(name="c", expected_behavior="x", rubric="y",
|
||||
input={"source": "hello", "target_lang": "es"})
|
||||
result = _run(EvalRunner(judge=judge).run_case(case, _echo_factory))
|
||||
assert result.judge.passed is True
|
||||
assert "source: hello" in result.transcript[0]["text"]
|
||||
assert result.transcript[1]["text"].startswith("echo: ")
|
||||
|
||||
|
||||
def test_report_shape():
|
||||
judge = LLMJudge(backend=FakeJudgeBackend(
|
||||
'{"score": 1.0, "reasoning": "ok"}', '{"score": 0.1, "reasoning": "bad"}'))
|
||||
suite = EvalSuite(name="s", cases=(
|
||||
EvalCase(name="a", expected_behavior="x", rubric="y", turns=(EvalTurn(user="hi"),)),
|
||||
EvalCase(name="b", expected_behavior="x", rubric="y", turns=(EvalTurn(user="hi"),)),
|
||||
))
|
||||
runner = EvalRunner(judge=judge)
|
||||
results = _run(runner.run(suite, _echo_factory))
|
||||
report = json.loads(runner.report(suite, results))
|
||||
assert report == {
|
||||
"suite": "s", "total": 2, "passed": 1, "failed": 1, "pass_rate": 0.5,
|
||||
"cases": report["cases"],
|
||||
}
|
||||
assert report["cases"][0]["case"] == "a"
|
||||
|
||||
|
||||
# ── Suite loading ────────────────────────────────────────────────────────────
|
||||
|
||||
def test_load_shipped_dub_suite():
|
||||
suite = load_suite(SUITES_DIR / "dub_translation_naturalness.yaml")
|
||||
assert suite.cases, "shipped suite must not be empty"
|
||||
for case in suite.cases:
|
||||
assert case.input.get("source") and case.input.get("target_lang")
|
||||
assert case.expected_behavior and case.rubric
|
||||
|
||||
|
||||
def test_load_suite_rejects_non_mapping(tmp_path):
|
||||
bad = tmp_path / "bad.yaml"
|
||||
bad.write_text("- just\n- a list\n", encoding="utf-8")
|
||||
with pytest.raises(ValueError, match="must be a mapping"):
|
||||
load_suite(bad)
|
||||
Reference in New Issue
Block a user