Palash DebnathandClaude Fable 5
1195b4e0dd
Ports Patter's eval harness (MIT, attribution headers) into tests/evals/ with the judge transport swapped to services/llm_backend.py — the judge runs against whatever local Ollama/LM Studio/OpenAI-compat endpoint the user configured, keeping local-first. Both Patter hardening details kept verbatim: verdict recomputed locally from the score (hallucinated 'passed: true' at score 0.2 fails), and tolerant JSON parsing (fences stripped, invalid JSON -> fail-with-reasoning). Per-case containment: agent exceptions keep the partial transcript and still judge it; a judge failure records score 0 instead of aborting the suite. HARD RULE preserved: LLM judges never gate CI. The scheduled workflow (weekly + dispatch) is continue-on-error with the JSON report as artifact; run_evals.py exits 0 always and skips cleanly when the active LLM backend is 'off'. Deterministic probe judges remain the only gates; the harness unit tests (10, no LLM needed) do run in gating CI. First suite: dub translation naturalness v1 (4 cases) driving the real cinematic_refine_sync reflect+adapt chain. The telephony-specific session/assertions layers were deliberately not ported. The dictation-refinement suite lands with Wave 1.1/2.1. Spec: docs/competitive-analysis.md Spec 9b / parity program Wave 0.3. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
56 lines
1.6 KiB
YAML
56 lines
1.6 KiB
YAML
# LLM-judge evals — semantic quality suites, NEVER a gate.
|
|
#
|
|
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
|
|
# judges never gate CI. This workflow is scheduled + manual only, the eval
|
|
# step is continue-on-error, and the JSON report is the deliverable
|
|
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
|
|
# the only gates.
|
|
#
|
|
# On the hosted runner there is no local LLM endpoint, so the run usually
|
|
# reports "skipped — no LLM backend configured"; the workflow exists so the
|
|
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
|
|
# (e.g. a self-hosted runner with Ollama).
|
|
|
|
name: evals
|
|
|
|
on:
|
|
schedule:
|
|
# Weekly, Sundays 04:00 UTC.
|
|
- cron: "0 4 * * 0"
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
evals:
|
|
runs-on: ubuntu-22.04
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.11"
|
|
|
|
- uses: astral-sh/setup-uv@v3
|
|
with:
|
|
enable-cache: true
|
|
cache-dependency-glob: "uv.lock"
|
|
|
|
- name: Install deps
|
|
run: uv sync
|
|
|
|
- name: Run eval suites (non-gating)
|
|
continue-on-error: true
|
|
env:
|
|
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
|
|
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
|
|
run: uv run python tests/evals/run_evals.py --output eval-report.json
|
|
|
|
- name: Upload report artifact
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: eval-report
|
|
path: eval-report.json
|
|
if-no-files-found: warn
|