Palash DebnathandClaude Fable 5
1195b4e0dd
Ports Patter's eval harness (MIT, attribution headers) into tests/evals/ with the judge transport swapped to services/llm_backend.py — the judge runs against whatever local Ollama/LM Studio/OpenAI-compat endpoint the user configured, keeping local-first. Both Patter hardening details kept verbatim: verdict recomputed locally from the score (hallucinated 'passed: true' at score 0.2 fails), and tolerant JSON parsing (fences stripped, invalid JSON -> fail-with-reasoning). Per-case containment: agent exceptions keep the partial transcript and still judge it; a judge failure records score 0 instead of aborting the suite. HARD RULE preserved: LLM judges never gate CI. The scheduled workflow (weekly + dispatch) is continue-on-error with the JSON report as artifact; run_evals.py exits 0 always and skips cleanly when the active LLM backend is 'off'. Deterministic probe judges remain the only gates; the harness unit tests (10, no LLM needed) do run in gating CI. First suite: dub translation naturalness v1 (4 cases) driving the real cinematic_refine_sync reflect+adapt chain. The telephony-specific session/assertions layers were deliberately not ported. The dictation-refinement suite lands with Wave 1.1/2.1. Spec: docs/competitive-analysis.md Spec 9b / parity program Wave 0.3. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
71 lines
3.1 KiB
YAML
71 lines
3.1 KiB
YAML
# Dub translation naturalness v1 — semantic regression suite for the
|
|
# cinematic translate chain (services/translator.py reflect+adapt pass).
|
|
#
|
|
# System under test: cinematic_refine_sync(source, literal, ...) — each case
|
|
# feeds a source line + a deliberately stiff literal translation; the judge
|
|
# scores whether the adapted line reads like natural dialogue. NEVER a CI
|
|
# gate (parity program Wave 0.3 hard rule).
|
|
name: dub translation naturalness v1
|
|
metadata:
|
|
system_under_test: services.translator.cinematic_refine_sync
|
|
cases:
|
|
- name: english idiom adapts to spanish equivalent
|
|
tags: [idiom, es]
|
|
input:
|
|
source: "It's raining cats and dogs out there, take an umbrella."
|
|
literal: "Esta lloviendo gatos y perros alla afuera, lleva un paraguas."
|
|
source_lang: en
|
|
target_lang: es
|
|
expected_behavior: >
|
|
The adapted Spanish line replaces the English idiom with a natural
|
|
Spanish equivalent (e.g. "esta lloviendo a cantaros") instead of the
|
|
word-for-word animal phrase, and keeps the umbrella advice.
|
|
rubric: >
|
|
Fail if the output still contains a literal cats-and-dogs construction.
|
|
Pass if a fluent Spanish speaker would find the line natural and the
|
|
meaning (heavy rain + take an umbrella) is preserved.
|
|
|
|
- name: register matches casual dialogue in german
|
|
tags: [register, de]
|
|
input:
|
|
source: "No way, you've got to be kidding me!"
|
|
literal: "Kein Weg, du musst mich verspotten!"
|
|
source_lang: en
|
|
target_lang: de
|
|
expected_behavior: >
|
|
The adapted German line is a natural colloquial exclamation (e.g.
|
|
"Das gibt's doch nicht!" or "Das kann nicht dein Ernst sein!"), not the
|
|
ungrammatical word-for-word rendering.
|
|
rubric: >
|
|
Fail if "Kein Weg" or "verspotten" survive. Pass if the line is
|
|
idiomatic spoken German with the same incredulous register.
|
|
|
|
- name: contraction-heavy line stays speakable in french
|
|
tags: [speakability, fr]
|
|
input:
|
|
source: "I dunno, it's kinda late — let's just call it a day."
|
|
literal: "Je ne sais pas, c'est un peu tard - appelons-le simplement un jour."
|
|
source_lang: en
|
|
target_lang: fr
|
|
expected_behavior: >
|
|
The adapted French line drops the calqued "appelons-le un jour" and uses
|
|
a natural equivalent (e.g. "on arrete la pour aujourd'hui"), keeping the
|
|
casual hedging tone.
|
|
rubric: >
|
|
Fail if the call-it-a-day calque survives. Pass if the line is natural
|
|
spoken French of similar length and casualness.
|
|
|
|
- name: honorific register preserved in formal speech
|
|
tags: [register, formal, es]
|
|
input:
|
|
source: "Ladies and gentlemen, thank you for joining us this evening."
|
|
literal: "Damas y caballeros, gracias por unirse a nosotros esta noche."
|
|
source_lang: en
|
|
target_lang: es
|
|
expected_behavior: >
|
|
The adapted line keeps the formal register of a public address — it must
|
|
NOT become casual — while staying natural.
|
|
rubric: >
|
|
Fail if the register drops to informal. Pass if a native speaker would
|
|
accept it as a natural formal opening of similar length.
|