Files
VoiceStudio/backend/schemas/requests.py
T
29269b9cf0 feat: LLM Providers page + Autofit translation quality (fit-to-segment-time) (#838) (#854)
* feat(llm): multi-provider LLM registry + encrypted key storage + settings API (v0.3.8, phase 1)

Foundation for the LLM Providers settings page and timing-aware (Autofit)
translation. Every provider in the shipped .env is OpenAI-compatible, so one
client drives all of them via a registry instead of a class-per-provider.

- llm_providers.py: registry of 16 providers (OpenAI, OpenRouter, Groq,
  Cerebras, Google AI, Mistral, Cohere, NVIDIA, GitHub Models, Cloudflare,
  HuggingFace, SambaNova, SiliconFlow, + local Ollama/LM Studio + Custom).
  Field resolution precedence env → encrypted store → default; active-provider
  selection (LLM_DEFAULT_PROVIDER → stored → first keyed remote; local requires
  explicit pick so we never assume a local server is up). Legacy TRANSLATE_*
  maps to the Custom provider (keyless-with-base_url preserved).
- settings_store.py: generic ENCRYPTED secrets (get/set/clear_secret,
  list_secret_names) reusing the HF-token Fernet path; get_text/set_text now
  refuse the secret namespace (no ciphertext leak).
- llm_backend.py: OpenAICompatBackend resolves the active provider's
  base_url/key/model from the registry. Backward-compatible.
- settings API: GET /llm-providers, PUT /llm-providers/{id} (encrypted key +
  overrides), POST /llm-providers/active, POST /llm-providers/{id}/test.
  Loopback-gated; never returns key material.
- 11 registry tests; existing llm-endpoint/openai-available tests still green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(settings): LLM Providers page — configure any provider's key/URL/model + Test + set active (v0.3.8, phase 2)

New Settings → System → LLM Providers pane (Brain icon, searchable). Lists all
16 registry providers; pick one to configure its encrypted API key, base URL,
model (and Cloudflare account id), Test the connection with one round-trip, and
'Save & use for translation' to make it the active provider for Cinematic/
Autofit. Keys are write-only from the UI (masked placeholder, never echoed);
env-set keys show as read-only. Local providers (Ollama/LM Studio) need no key.

- LLMProvidersPanel.jsx: provider selector + per-provider config + Test/activate,
  following the LLMEndpointPanel pattern (apiJson/apiFetch/apiPost, SettingsSection
  primitives).
- settingsCategories.jsx: new 'llm-providers' category under System + Brain icon.
- Settings.jsx: route the category to the panel.
- en.json: settings.llm_providers label.
- Frontend build passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(translate): Autofit quality style + one-click LLM setup from the dub menu (v0.3.8, phases 3-4)

Autofit = Cinematic + a strict 'never exceed the segment time' fit. The LLM
rewrites each translated line so its target-language reading time fits within
the slot, preserving the video timing without harsh audio time-stretch.

Backend:
- speech_rate.adjust_for_slot(strict=): strict caps the accepted upper ratio at
  1.0 (fit within slot) vs Cinematic's 1.08; best-effort, degrades gracefully
  with no LLM.
- dub_translate: quality='autofit' takes the LLM refine path and runs the fit
  pass with strict=True; reports quality_used accurately.
- TranslateRequest.quality doc note.

Frontend:
- 'autofit' added to the quality control (Settings Translation + dub menu) and
  the TranslateQuality type.
- Dub menu: picking Cinematic/Autofit with no LLM no longer dead-ends on a toast
  — it offers a one-click 'Set up' that routes to Settings → LLM Providers,
  with copy about fitting translations to segment time (#838).

- 4 strict-fit tests; frontend build green; i18n keys added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(translate): document Autofit quality + the LLM Providers page (v0.3.8, phase 5)

- CHANGELOG [0.3.8] Added: Autofit style + LLM Providers page.
- docs/dubbing/translation-engines.md: Fast/Autofit/Cinematic quality section
  and an LLM Providers setup section (16 providers, encrypted keys, offline
  Ollama/LM Studio, env overrides).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(api): add /api/settings/llm-providers routes to the route-inventory snapshot

Regenerated tests/fixtures/api_routes.txt for the 4 new LLM-providers endpoints
so test_route_inventory_matches_snapshot passes (keep-main-green).

* style(frontend): oxfmt the LLM Providers panel + dub quality control (format:check green)

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:55:59 +05:30

158 lines
7.6 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
from pydantic import BaseModel, field_validator
from typing import List, Literal, Optional
from services.audio_dsp import EFFECT_PRESETS
class ExportRequest(BaseModel):
source_filename: str
destination_path: str
mode: str = "history"
class ExportRecordRequest(BaseModel):
filename: str
destination_path: str = "~/Downloads"
mode: str = "file"
class RevealRequest(BaseModel):
path: str
class DubSegment(BaseModel):
start: float
end: float
text: str
instruct: str = "" # Per-segment voice override
profile_id: str = "" # Per-segment voice profile
speed: Optional[float] = None
gain: Optional[float] = None # Per-segment volume (0.0 - 2.0, default 1.0)
target_lang: Optional[str] = None # Per-segment language override (ISO code)
# Phase 4.2 free-form directorial note ("urgent, whispered…"). The client
# has always sent this; without the field pydantic silently dropped it,
# so directions never reached TTS and never entered the regen
# fingerprint (#281).
direction: Optional[str] = None
effect_preset: str = "broadcast" # NEW: DSP preset id (default: broadcast)
@field_validator("effect_preset")
@classmethod
def validate_effect_preset(cls, v: str) -> str:
if v not in EFFECT_PRESETS:
raise ValueError(
f"Unknown effect preset: {v!r}. "
f"Valid: {list(EFFECT_PRESETS.keys())}"
)
return v
class FitOptions(BaseModel):
"""Optional knob overrides for the `smart_fit` timing strategy.
All fields default to None — the server fills in the canonical
defaults (services.fit_planner.FitParams) so old clients and sparse
payloads behave identically to a fully-populated default payload.
"""
max_audio_only_rate: Optional[float] = None # default 1.2
audio_rate_cap: Optional[float] = None # default 1.5
video_slow_cap: Optional[float] = None # default 2.0
gap_guard_s: Optional[float] = None # default 0.05
allow_video_retime: Optional[bool] = None # default True
class DubRequest(BaseModel):
segments: List[DubSegment]
language: str = "Auto"
language_code: str = "und" # ISO 639-1 for ffmpeg metadata (e.g. "es", "fr", "de")
instruct: str = ""
num_step: int = 16
guidance_scale: float = 2.0
speed: float = 1.0
# Phase 4.1 — partial regen. Parallel lists by index with `segments`.
# When `regen_only` is set, only listed segment ids re-run TTS; others
# reuse their on-disk seg_N.wav. `segment_ids` lets the client bind
# each segment to a stable id across regen cycles.
segment_ids: Optional[List[str]] = None
regen_only: Optional[List[str]] = None
# Fast-preview mode for interactive edits. When true, TTS runs at
# num_step=8 (~2× faster, ~10-20% quality drop). Client is responsible
# for re-rendering preview segs at full quality before final export.
preview: Optional[bool] = False
# How to handle segs whose TTS audio is longer than its slot (the
# "ghost lang" overlap bug otherwise). Options:
# "time_stretch" — phase-vocoder stretch to fit, preserves pitch (default).
# "trim" — hard-clip to slot length + fade out (cheap, may cut mid-word).
# "off" — no fit; mix layers with += (legacy behaviour, may overlap).
# Legacy knob. When `timing_strategy` is set, it takes precedence and this is ignored.
slot_fit: Optional[str] = "time_stretch"
# High-level timing strategy. Replaces the audio-compression default that
# produced chipmunk/alien artefacts on high-density target languages
# (Bengali, Hindi, Arabic…). Three modes:
# "concise" — never compress TTS audio. Trim text up-front via
# speech_rate so it fits naturally; if it still
# overflows, hard-trim at slot with a short fade and
# surface fit_status="overflows" so the UI can prompt
# the user to shorten the segment. DEFAULT.
# "stretch_video" — never compress TTS audio. Re-lay the timeline so
# each segment's video portion is stretched (via
# ffmpeg setpts) to fit the natural-rate dub audio.
# Audio plays at 1.0×; total video duration grows.
# "smart_fit" — dub-length fitting v2: split the burden between a
# mild pitch-preserving audio speed-up (≤1.2× alone,
# ≤1.5× in hybrid) and a mild per-segment video
# slow-down (≤2.0×), per services/fit_planner.py.
# Residual overflow is trimmed and surfaced.
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
# when audio > slot). Kept for back-compat.
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot", "smart_fit"]] = "concise"
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
# gap absorption + this much extra time has been consumed.
overflow_budget_s: Optional[float] = 0.0
# Knob overrides for `smart_fit` (ignored by other strategies). Omitted
# fields default server-side to fit_planner.FitParams values.
fit_options: Optional[FitOptions] = None
class TranslateSegment(BaseModel):
id: str
text: str
target_lang: Optional[str] = None
# Free-form delivery direction ("urgent, whispering") — feeds the
# cinematic reflect/adapt prompts. The frontend has sent this since
# Phase 4.2 but pydantic silently dropped it as an undeclared extra,
# so the per-segment direction hint never reached the LLM.
direction: Optional[str] = None
# Available time slot (end - start, seconds) for rate-ratio prediction
# and the cinematic slot-fit pass. Same silent-drop fix as `direction`.
slot_seconds: Optional[float] = None
class TranslateRequest(BaseModel):
segments: List[TranslateSegment]
target_lang: str # ISO 639-1 code like "es", "fr"
provider: Optional[str] = None
source_lang: Optional[str] = None # ISO 639-1; overrides job detection
job_id: Optional[str] = None # Dub job id, used to resolve detected source_lang
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect→adapt) | "autofit" (cinematic + strict fit-to-slot)
glossary: Optional[List[dict]] = None # [{"source": "...", "target": "...", "note": "..."}]
# Optional regional dialect (BCP-47, e.g. "es-AR", "pt-BR") — #280 item 2.
# Applied by LLM-backed paths (provider="openai" or quality="cinematic"):
# the prompt asks for that region's vocabulary/grammar (e.g. Argentinian
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
# Google) can't honor it; the response then carries dialect_applied=false.
dialect: Optional[str] = None
class DubIngestUrlRequest(BaseModel):
url: str
job_id: Optional[str] = None
# When true and the URL is a caption-bearing host (YouTube, Vimeo, TED…),
# ask yt-dlp to also download the original-language + any additional
# sub_langs as VTT. The UI uses this to seed a transcript without running
# Whisper, and optionally to skip the Translate step for languages that
# YouTube auto-translates for us.
fetch_subs: Optional[bool] = False
sub_langs: Optional[List[str]] = None
class ProjectSaveRequest(BaseModel):
name: str
video_path: Optional[str] = None
audio_path: Optional[str] = None
duration: Optional[float] = None
state: dict # Full JSON blob: segments, settings, tracks, etc.