Compare commits

..
160 Commits
Author SHA1 Message Date
6044765f09 chore(version): bring frontend/package.json into the version lockstep (0.3.6) (#497)
Pre-v0.3.6 release sweep found frontend/package.json stuck at 0.3.5 while the
other three version files were 0.3.6. package.json drives the runtime
`__APP_VERSION__` (vite.config.js), so a v0.3.6 build was calling itself "v0.3.5"
in the first-run footer AND in every auto bug report (undercutting the bug-report
feature). Root cause: the release.yml version-bump job only bumped the trio
(tauri.conf.json / Cargo.toml / pyproject.toml), never package.json, and no test
guarded the lockstep.

- Bump frontend/package.json 0.3.5 → 0.3.6 (matches the trip; `--frozen-lockfile`
  still passes — the version field doesn't affect the bun lock graph).
- Add frontend/package.json to the release.yml version-bump job (set absolutely
  via jq so any prior drift self-heals on the next release).
- Add tests/test_app_version.py::test_all_version_files_in_lockstep — fails CI if
  the four files ever diverge again.
- CLAUDE.md versioning rule updated: it's now FOUR lockstep files, not three
  (docs-sync).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 17:59:44 +05:30
f46ea1fbf6 docs(planning): commit singing-mode + donate-cta specs + SPIKE-02 supersession (#496)
Persist the planning artifacts produced this cycle:
- specs/006-dubbing-singing-mode/ (SoulX-Singer SVS evaluation + plan; supersedes
  SPIKE-02, which is marked superseded here).
- specs/007-donate-cta/ ("Fund Claude Max" goal bar + kawaii postcard design,
  conversion strategy, frequency state machine, Discord surface).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 17:06:11 +05:30
af3da58584 fix(bootstrap): force-reinstall setuptools so pkg_resources repair actually works (#248) (#495)
The auto-repair ran `uv pip install setuptools>=75,<80`, which `uv` treats as
"already satisfied" (no-op, "Checked 1 package in 5ms") whenever setuptools'
*metadata* is present but its `pkg_resources` files are gone — the common cause
being Windows Defender quarantining `pkg_resources/`, or a partial extract on a
restricted network. So the repair never restored the files, the post-check
failed, and users hit the #248 dead-end. The error message *also* told them to
run the same no-op command, so the suggested manual fix didn't work either
(reported on Discord, Win11 + RTX 5070 Ti).

Fix: both repair sites in bootstrap.rs now use `--reinstall` (the flag already
used for the ROCm torch repair), which force re-extracts pkg_resources even when
uv thinks setuptools is satisfied. The fail() message and the failure.py hint now
suggest `uv pip install --reinstall 'setuptools>=75,<80'` + an antivirus-exclusion
note, and docs/install/troubleshooting.md (#pkg_resources-missing) is updated with
the real cause (metadata-present/files-missing) + AV guidance.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 17:06:05 +05:30
62637cad95 feat(donate): "Fund Claude Max" goal bar + kawaii postcard + milestones (#007) (#494)
Problem
  OmniVoice's donate surface was a static link page. There was no sense of
  shared progress toward a concrete funding goal, and no gentle, success-only
  ask after a user got value — only an always-on footer heart.

Design
  Phase 1 — Goal bar + data (Option B):
    - frontend/public/donation_progress.json (committed snapshot) + a bundled
      offline fallback in api/donation.ts. loadDonationProgress() best-effort
      fetches a fresher copy and gracefully falls back to the bundle on any
      failure (offline / non-2xx / bad JSON). Never throws.
    - <GoalBar> (page + `mini` variant), --goal-pct-driven fill, Pip mascot
      perched on the fill, ONE shimmer pass, reduced-motion guard on every
      animation. Added to SupportPage above the payment cards with
      "Join {n} supporters" social proof + suggested amounts ($3/$5/$10/Custom,
      middle flagged "most common", NONE pre-selected).
  Phase 2 — Pip + postcard + state machine:
    - Pip.jsx (currentColor->accent, pipBob/pipWave idle, reduced-motion off).
    - donationSlice.ts composed into the store: added to partialize (all EXCEPT
      shownThisSession), version 5->6 with a pass-through migrate branch.
      shouldShow rules: first-3 grace, <=1/session, escalating 7d/14d/30d/75d
      cooldowns, optedOut terminal, success-only.
    - Postcard.jsx rendered via react-hot-toast as a NON-BLOCKING custom toast
      (no backdrop, no focus steal, ~12s auto-dismiss, pause on hover) with the
      perforation / dot-grain / stampThunk / postcardIn / .is-leaving art,
      a mini GoalBar, and Chip in / Maybe later / quiet Don't ask again /
      free Star on GitHub actions.
    - One shared evaluateDonationPrompt() called right after each SUCCESS
      (dub-complete, clone-save resolve, longform export) — never on the
      error / in-progress / setup / first-run paths.
  Phase 3 — Milestones + pill:
    - Milestone eval (1st clone / 10th dub / 30-day sustained, each once-ever,
      same cooldowns + opt-out) inside the shared evaluator.
    - Quiet nav-rail .donate-pill (🩷 Support) that warms to the accent on
      hover and opens setMode('donate').

Tests (vitest, all green: 60 files / 533 tests)
  - donationSlice.test.ts: shouldShow truth table with injected `now` — grace,
    each cooldown rung, session cap, opted-out terminal, success-only.
  - GoalBar.test.jsx: renders from injected JSON, offline fallback to bundle,
    goal-met state, mini variant; plus the data module's clamp/normalize/fetch.
  - evaluateDonationPrompt.test.jsx: gating + that the postcard never fires on
    the error path (success-only contract).
  bun run typecheck:ci clean; vite build green; root bun.lock untouched
  (frozen install verified); i18n: all user-facing strings via t('donate.…')
  with English defaultValue fallbacks — no hardcoded CJK.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 16:59:24 +05:30
3656f0a4ef fix(profiles): decouple design-profile save from TTS render (#476) (#488)
* fix(profiles): decouple design-profile save from TTS render (#476)

Saving a design voice profile forced a full TTS model load + inference to
render a deterministic identity sample. On a fresh model-less image (Docker
first-run) that 503'd, so the save failed. A secondary guard also rejected an
all-Auto design (empty instruct) with a 422.

Saving a design profile is now a pure persistence operation:
- The seed-42 identity sample render is attempted opportunistically but is
  non-fatal — if the engine isn't ready the row is persisted with
  ref_audio_path=NULL (sample pending). The row's vd_states + instruct already
  make the voice fully usable (generation.py falls back to instruct-only
  conditioning for design profiles with no ref audio).
- The sample is rendered lazily + cached on the first GET /profiles/{id}/audio
  request; if the engine is still unavailable that path returns a precise
  "model not ready — finish setup / download a model" 503.
- The all-Auto (empty-instruct) design is now saveable (vd_states still
  required).

Adds tests/test_profile_design_save_decouple.py (top-level tests/, asyncio.run
per test) covering: design save with model unavailable creates the row instead
of 503-ing; all-Auto design is saveable; the pending sample materializes on
first /audio request. Updates the unification spec (docs-sync).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(profiles): contain profile-audio paths under VOICES_DIR (CodeQL CWE-22)

The lazy design-sample path was built as `os.path.join(VOICES_DIR,
f"{profile_id}.wav")` / `os.path.join(VOICES_DIR, audio_file)` where profile_id
is the request path param — CodeQL flagged 5 high-severity path-injection alerts
(profiles.py + the taint flowing into archetypes.py's torchaudio save). Add
`_safe_voice_path()` (basename + safe-char sanitise + realpath containment,
mirroring core.config.dub_seg_path) and route both the read and lazy-render
sites through it; a traversal id now 404s instead of escaping VOICES_DIR.
Regression test covers the containment guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(profiles): use CodeQL-recognized path-injection guards (CWE-22)

The previous `_safe_voice_path()` helper was correct (basename + realpath
containment) but CodeQL's taint tracking didn't propagate the barrier through
the function return, so the 5 path-injection alerts persisted. Switch to guards
CodeQL recognizes, inline at each file-op site:
- validate `profile_id` against the generated-id charset (`[A-Za-z0-9_-]{1,64}`)
  with `re.fullmatch` and 404 on mismatch (covers the `f"{profile_id}.wav"`
  render path);
- read only `os.path.join(VOICES_DIR, os.path.basename(name))` so a stored/derived
  filename is always a direct child of VOICES_DIR (covers the read + the taint
  flowing into archetypes.py's torchaudio save).
Drop the helper. Test now asserts a traversal/separator/NUL profile_id 404s at
the guard. Same security property, recognized by CodeQL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(profiles): inline realpath+commonpath containment for CodeQL (CWE-22)

CodeQL didn't recognize the earlier sanitizers — neither the helper (barrier
hidden behind a function return) nor os.path.basename / a cross-function regex
guard cleared the 5 path-injection alerts. Use the canonical, CodeQL-recognized
form INLINE at each file-op site: resolve the path with os.path.realpath (which
collapses any `..`) and confirm os.path.commonpath((base, path)) == base before
the read / the render, returning 404 / raising on escape. Same property the
helper had, now in a shape CodeQL's taint tracking follows. Keeps the profile_id
charset guard as defense-in-depth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(profiles): route design-sample path through shared _voices_path guard (#476)

The inline realpath+commonpath containment in get_profile_audio and
_materialize_design_sample wasn't recognized by CodeQL as a path-injection
sanitizer (5 new high-severity py/path-injection alerts at the file-op sites,
incl. archetypes.py mkdir via the rendered Path). Both now reuse the existing
_voices_path() helper, which applies the os.path.basename() barrier plus
symlink-resolved containment — the same guard the consent endpoint uses and
that CodeQL already accepts. Behavior is unchanged: the DB columns only ever
hold bare {profile_id}.wav filenames, so basename() is a no-op here.

Tests: tests/test_profile_design_save_decouple, test_profile_unification,
test_profile_consent, test_archetype_blank_guard — 25 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 16:58:17 +05:30
392b573cdf docs(conventions): add "Fix quality" + "Keep main green" hard rules (#493)
Owner-set 2026-06-16. (1) Fix issues properly/future-maintenance-proof — fix the
whole class, add a regression test, harden against recurrence; extra effort, not
extra verbosity. (2) A merge must never break main's CI — verify the full CI
matrix (every .github/workflows/* AND deploy/Dockerfile) before landing, with
explicit guidance that frontend/ is a bun workspace monorepo whose root bun.lock
must be regenerated on any frontend/package.json change (Docker uses
--frozen-lockfile; plain bun install in ci.yml tolerates drift). Motivated by the
#485 bun.lock incident that reddened main's Docker workflow.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 15:39:40 +05:30
cfe00c5c1c fix(ui): clone popover/CTA clipping + textarea resize (#481, #476) (#489)
Bug #481 — Clone "+Insert" popover was clipped offscreen and the script
textarea couldn't be resized:
- Apply the existing `.clone-panel--overflow-visible` helper to the script
  `.studio-panel` so the upward-opening popover escapes the panel's
  `overflow:auto` box instead of being shoved into its scroll region.
- Cap the popover at `max-width: min(360px, calc(100vw - 16px))` so the
  14-chip grid can never spill past the viewport edge.
- Re-enable the textarea corner grip (`resize: vertical`, matching the base
  `textarea.input-base`) and lift the ⊕ Insert button off the bottom-right so
  it no longer physically covers the drag handle.

Bug #476 — the design-mode "Synthesize Audio" CTA dropped below the fold on
narrow shells:
- Replace the raw `@media (max-width: 900px)` reflow rules with the app's
  shell-width classes (`.shell-narrow` / `.shell-mini`, set in App.jsx from
  `app-container.clientWidth`). The shell scales via `zoom`, so a viewport
  media query fired at the wrong threshold whenever `--ui-scale ≠ 1`.
- When stacked, let `.studio-with-history__main` grow (drop its `overflow:hidden`
  clip) and pin the action bar `position: sticky; bottom: 0` so the Synthesize
  CTA stays on-screen.

Pure CSS + one className; no component restructuring. Added a regression test
guarding the shell-class reflow + sticky CTA against the viewport-`@media`
anti-pattern. typecheck:ci clean; vitest 506/506 green.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 15:39:34 +05:30
1650a121db fix(dub): auto-assign per-speaker voices in multi-speaker dubbing (#486) (#490)
Multi-speaker dubs detected speakers and built per-speaker clones (Voice
dropdown showed "From Video → Speaker N"), but most segments stayed on
"Default" voice and had to be set by hand — inconsistently across runs.

Root cause: after diarization, dub_core stamped each long line (the
default-on per-segment-ref path) with `auto-seg:{id}` as its profile_id.
The dub editor's Voice <select> (and the Cast panel) only render `auto:`
options, so an `auto-seg:` value matched no <option> and silently showed
"Default". Short lines (<3s) fell through to `auto:{speaker}`, which DID
render — hence "sometimes the cloned voice is picked".

Fix: bind every segment to the UI-visible `auto:{speaker}` whenever its
detected speaker has a clone; only fall back to `auto-seg:{id}` when the
speaker has no per-speaker clone at all. The per-segment-ref quality win
is preserved: dub_generate's `auto:` branch now transparently prefers
THIS segment's own per-segment ref (segment_clones[seg_id]) when present,
else the per-speaker clone. Manual overrides and the no-clone path are
untouched; existing jobs that persisted `auto-seg:` ids still resolve.

Tests: tests/test_dub_multispeaker_voice_486.py — assignment binds to
auto:{speaker} (not auto-seg:), never clobbers manual overrides, falls
back to auto-seg: only when the speaker has no clone; generate-time
resolution prefers per-segment ref then per-speaker clone. Green
alongside the existing dub generate/incremental/segmentation suites.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 15:39:28 +05:30
bc15fdc990 fix(deps): re-sync root bun.lock after #485 (fixes main Docker red) (#492)
* fix(deps): re-sync root bun.lock after #485 frontend floor bumps (main Docker red)

#485 bumped ~25 dependency floors in `frontend/package.json` but didn't
regenerate the repo-root `bun.lock` (this is a bun *workspace* monorepo — the
lockfile lives at root and embeds the frontend member's ranges). The Docker
workflow runs `bun install --frozen-lockfile`, which failed on the drift
("lockfile had changes, but lockfile is frozen") — turning main red on commit
4bcbc74. `ci.yml` uses a plain `bun install`, so it tolerated the drift and went
green, which is why only Docker caught it.

Regenerate `bun.lock` so its embedded frontend snapshot matches the manifest;
`bun install --frozen-lockfile` now passes (verified locally, bun 1.3.14, the
same version Docker uses). Lockfile-only change.

Follow-up (separate): switch `ci.yml`'s frontend `bun install` to
`--frozen-lockfile` so this drift class fails fast in CI, not only in Docker.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci: use --frozen-lockfile for frontend install so lockfile drift fails fast

Recurrence-proofing for the #485 incident: ci.yml's plain `bun install` silently
tolerated the root bun.lock drifting from frontend/package.json, so CI went green
while only the Docker build (which already uses --frozen-lockfile) caught it and
reddened main. Both frontend install steps now use --frozen-lockfile, so a
package.json change that forgets to regenerate root bun.lock fails in CI fast.
Verified `bun install --frozen-lockfile` passes from frontend/ against the
re-synced lockfile (bun 1.3.14).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 15:39:21 +05:30
4bcbc74e86 chore(deps): conservative refresh — backend, UI, Tauri (#485)
* chore(deps): refresh backend HTTP/cert/security leaf packages

Conservative, targeted refresh (no blanket re-resolve). Bumps only low-risk leaf
packages — yt-dlp 2026.3.17→2026.6.9 (extractor currency), aiohttp, requests,
urllib3, idna, certifi, charset-normalizer, pillow (HTTP/cert/security). No
major bumps, no downgrades, no transitive removals; the 91-package full
`--upgrade` was rejected because it downgraded numpy/pandas/av and pulled a
starlette 1.x major that broke WS route introspection. Full backend suite: 1674
passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(deps): bump UI deps to within-major latest

`bun update --latest` floors raised to current within-major releases — react
19.2.7, react-dom 19.2.7, vite 8.0.16, tailwindcss/@tailwindcss/vite 4.3.1,
@tanstack/react-query 5.101, @radix-ui/* minors, lucide-react 1.18, zustand
5.0.14, i18next 26.3.1, plus dev tooling (vitest 4.1.9, eslint 10.5, playwright
1.61, @tauri-apps/cli 2.11.2). Verified no major version crossings. typecheck:ci
clean; vitest 503/503.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(deps): cargo update Tauri crates within range

`cargo update` — 77 crates locked to latest semver-compatible versions (patch/
minor: bitflags, chrono, hyper, reqwest, regex, rustls-native-certs, etc.; one
in-range 0.x bump global-hotkey 0.7→0.8). No Cargo.toml range changes. `cargo
check` compiles clean (0 errors).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 14:56:33 +05:30
a96af2979d fix(deps): bundle openai so Cinematic/LLM features work out of the box (#484)
Cinematic dub refinement, glossary auto-extract, and LLM-based translation all
`from openai import OpenAI` (services.llm_backend / translator / dub_translate /
glossary), but `openai` was declared nowhere in pyproject — not in dependencies,
not in any optional extra, and no setup script installed it. So a fresh `uv sync`
never installed it, and these features were dead-on-arrival on every source
install: picking Cinematic showed "Cinematic needs an LLM" even with Ollama
running and correctly configured, because `OpenAICompatBackend.is_available()`
returned "openai package missing". The UI's "pip install openai" hint is a trap
on a managed venv — users (Discord report) installed it into system Python, not
the app's `.venv`, so it still didn't take.

Add `openai>=1.40` to dependencies (resolves to 2.41.1; verified the code's
`OpenAI(...)` + `chat.completions.create(model=, messages=)` call shapes are
unchanged in 2.x). Pure-Python, no native deps → identical on macOS/Windows/Linux
(default-parity rule). Cinematic + any OpenAI-compatible endpoint (OpenAI, Ollama,
LM Studio, vLLM) now work after `uv sync`, no manual package install.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 14:36:59 +05:30
a245d96684 fix(settings): make in-app models dir authoritative over launcher-injected env (#480) (#483)
Changing the model download location in Settings had no effect: after the
prompted restart, new downloads still went to the old folder and "Effective
location" stayed stuck on it.

Two stores hold the models dir. The in-app Settings panel writes the new path to
the durable per-user env file (`~/.config/omnivoice/env`, OMNIVOICE_CACHE_DIR),
but the desktop launcher injects the OLD value from its own Tauri config into the
backend's environment before startup — and main.py loaded the per-user file with
`override=False`, so the launcher's stale value always won. main.py then maps
OMNIVOICE_CACHE_DIR → HF_HOME/HF_HUB_CACHE/TORCH_HOME, pointing downloads at the
old dir; `_effective_models_dir()` reads that live env, so the UI faithfully
reported the old path as if the change had failed.

Fix: load the per-user env file with override so it beats launcher-injected
defaults — restoring this file's documented "values written here take effect on
the next backend launch" contract. Centralized as `user_env.load_into_environ()`
(the file is the in-app Settings source of truth) and called from main.py. Both
keys this file can hold (OMNIVOICE_CACHE_DIR, HF_ENDPOINT) are the user's explicit
Settings choice and should beat the launcher default, so the override is correct
for both (this also fixes the same latent bug for a Settings-set HF mirror).
HF_TOKEN isn't launcher-injected, so its behavior is unchanged.

Follow-up (separate PR): add a Tauri `set_models_dir` command so the launcher's
config.json stays in sync, covering the reset-to-default edge too.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 13:53:34 +05:30
dd092c95cb fix(asr): decode WhisperX audio via validated ffmpeg, not bare PATH lookup (#479) (#482)
WhisperX transcription called `whisperx.load_audio()`, which shells out to a
literal `"ffmpeg"` resolved against the OS PATH. On Windows that resolves to a
WindowsApps alias stub or a corrupt/wrong-arch binary — passing `which` but
exploding at spawn with `[WinError 193] %1 is not a valid Win32 application`.
whisperx only catches `CalledProcessError`, so the spawn-time `OSError` escaped
and the dub/batch path reported the opaque "Transcription produced no segments".

#377 added ffmpeg validation but only for the dub-export path; the transcription
path never went through the validated resolver. Since WhisperX is a default ASR
engine, this is a P0 platform-parity break (works on mac/Linux, fails on Windows).

Fix: decode the audio ourselves in `WhisperXBackend.transcribe` via
`find_ffmpeg()` (which `-version`-probes each candidate and returns the bundled
imageio-ffmpeg / Tauri sidecar) and hand WhisperX the array — bypassing the bare
PATH lookup entirely. This is more robust than a PATH-prepend, which couldn't
fix the imageio case (its binary is named `ffmpeg-<plat>-vN.exe`, not `ffmpeg`).
If no runnable ffmpeg exists, raise a clear, locale-independent error instead of
"no segments". Fixes both the dub and batch transcription paths.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 13:26:26 +05:30
4e3136c1e0 fix(translate): guess source language from text instead of defaulting to "en" (#478)
When neither the request nor the job carries a detected source language,
_resolve_source_lang() silently fell back to "en". For non-English audio
(e.g. Korean) this produced en -> en, which has no Argos package and failed
every segment — even though WhisperX had detected the language correctly
(e.g. "Detected language: ko (0.98)").

Add a last-resort script-based guess (ko/ja/zh/ru/ar) from the segment text
so the bare "en" fallback no longer breaks non-English dubbing.

Co-authored-by: stronghamjji <289942360+stronghamjji@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 11:04:10 +05:30
Palash Debnath 46d4a94479 fix(bootstrap): stall watchdog so a stuck backend isn't a buttonless dead-end (#474) (#475)
The entire main UI is gated behind `bootstrapStage === 'ready'` (App.jsx) — until
the Python backend reports ready, only the BootstrapSplash shows. If the backend
hangs in a non-terminal stage and never reaches ready (e.g. a failed from-source
backend spawn on Windows: uv/Python not on PATH), useBootstrapStage polled
forever, trapping the user on a splash with no Settings / Start / Clone / Extract
buttons — which is exactly what #474 reports (verified: no backend-startup
regression; every startup-imported router imports cleanly).

- useBootstrapStage: add a per-stage stall watchdog. Track when (stage,message)
  last changed; if a non-terminal stage sits past its budget (installing_deps
  gets 20 min since it legitimately runs 5–10 min; everything else 120 s), flip
  to the existing `failed` state — which already surfaces actionable hints, the
  live log panel, and Retry / Clean-&-Retry. Any change resets the clock, so a
  live install never trips it.
- detectHints + bootstrap.hint_stuck: a targeted hint for the stuck case
  (run `uv sync`, check uv/Python on PATH, read the log / Settings → Logs).
- CONTRIBUTING.md: document `bun run desktop-prod` (the prod desktop command the
  reporter typo'd as `desktop=prod`), note both desktop scripts auto-run
  `uv sync` + start the backend, and add a "stuck on the setup splash" pointer.

No backend code change. Frontend suite green (503); CJK guard green.
2026-06-15 00:11:24 +05:30
Palash Debnath 18793a99a7 feat(audiobook): durable crash-resume for interrupted longform renders (#470)
* feat(audiobook): durable crash-resume for interrupted longform renders

Chapter WAVs were already content-addressed (a re-run reused finished chapters),
but resume only worked if the user could re-submit the EXACT script — impossible
for Stories, whose plan is compiled from cast+lines. This persists the plan
itself so an interrupted render is resumable without the original input.

- New services/longform_resume.py (pure file/JSON): on render start, write a
  resume.json manifest (compiled plan + render params + title) into the job work
  dir, atomically; clear it on successful completion. read/has/clear/build
  helpers, schema-versioned (a foreign/corrupt manifest is ignored, never
  resumed).
- _render_longform_sse: accepts an optional job_id + resume flag (resume reuses
  the original job row + cached chapters instead of creating a new one); writes
  the manifest at start, clears it on done. Both front doors (/audiobook,
  /longform/render) unchanged for callers.
- GET /audiobook/jobs — lists interrupted renders (running/failed longform jobs
  that still have a manifest; a job left "running" across an app restart is
  interrupted by definition), with title + total/done chapter counts for the UI.
- POST /audiobook/resume/{job_id} — rebuilds the plan from the manifest and
  replays _render_longform_sse under the original job_id; the content-addressed
  cache makes finished chapters instant, so only the unrendered ones synthesize.
  404 on unknown id / missing manifest.

Resume durability is best-effort — a manifest failure never blocks the render.
The resume UI affordance is a follow-up (the endpoints are ready for it).

Tests: tests/test_longform_resume.py (7, pure manifest round-trip / version &
corrupt rejection / atomic write — monkeypatches OUTPUTS_DIR, no global
core.config stub so the shared tests/ session isn't polluted) +
backend/tests/test_audiobook_resume_api.py (6, config-stub: jobs-list with
progress, failed-included, done/manifestless/non-longform excluded, resume
404s). 13 passed. CJK green. Stale module docstring updated.

* fix(audiobook): confine resume paths — py/path-injection (CodeQL) + quality

The default-setup CodeQL (security-and-quality suite) flagged the crash-resume
work: longform_resume built filesystem paths from job_id, which on the
POST /audiobook/resume/{job_id} endpoint is a request-supplied path param →
py/path-injection (10 high-severity sinks: open/replace/remove/makedirs/isfile).

- longform_resume.work_dir now confines like profiles._voices_path: reject an
  unknown job_type or an id that isn't a bare safe token (^[A-Za-z0-9_-]{1,64}$),
  then realpath + startswith(OUTPUTS_DIR + os.sep) — a crafted id (`../`, NUL,
  separators) can never escape OUTPUTS_DIR. Returns None on violation; all
  callers (manifest_path/read/write/clear/has) degrade gracefully.
- The resume endpoint also gates the path-param id up front (404 on a bad
  token) — barrier at the source as well as the sink.

Also cleared the quality alerts the same diff introduced:
- py/repeated-import: the 4 inline `from services import longform_resume` calls
  collapse to one module-top import (it's pure, no torch).
- py/empty-except: the best-effort manifest blocks now logger.debug instead of
  a bare `pass`.

13 resume tests still pass; all job ids in tests are safe tokens.

* fix(audiobook): sanitize resume job_id at the source (path + log injection)

The first CodeQL pass wasn't enough: resume made job_id request-controlled, so
it tainted not just the manifest paths but the EXISTING work-dir join and the
progress log lines too (py/path-injection + py/log-injection, ~14 alerts).

Fix at the source so the whole dataflow is clean:
- _render_longform_sse strips job_id to a safe token (`re.sub` removing anything
  but [A-Za-z0-9_-], capped 64) right after it's resolved — no path separator,
  no CR/LF can survive, whether the id came from the resume path param or a
  fresh uuid.
- The work dir now routes through longform_resume.work_dir, which adds the
  proven os.path.basename(seg)==seg barrier (the shape CodeQL accepts in
  _voices_path) on top of the realpath+startswith confinement — so the join and
  every path derived from it (meta/concat/out) is sanitized.
- The best-effort manifest-write log no longer interpolates the raw exception
  (uses exc_info); clear_manifest's OSError handler returns instead of bare pass
  (py/empty-except).

13 resume tests still pass.

* fix(audiobook): launder resume job_id via trusted FS scan (CodeQL path/log-injection)

The custom realpath/regex barriers weren't in CodeQL's recognized sanitizer set,
so the request-supplied resume job_id kept tainting the work-dir/manifest paths
and the progress logs. Switch to the pattern CodeQL does accept — launder the id
through a trusted filesystem enumeration:

- longform_resume.scan_resumable() lists resumable jobs by scanning OUTPUTS_DIR
  for <type>_<id>/resume.json; every id it returns is sourced from os.listdir
  (never request input).
- POST /audiobook/resume/{job_id} now only resumes an id that scan_resumable()
  reports (membership match), and uses the (job_type, job_id) pair FROM that
  trusted list for everything downstream — so nothing request-controlled reaches
  a filesystem path or a log line.
- GET /audiobook/jobs lists from scan_resumable() too (filesystem-sourced ids).

work_dir keeps the realpath+startswith+basename confinement as genuine defense;
the render path's job_id is now always either a fresh uuid or a laundered id.
13 resume tests still pass.

* fix(audiobook): exact-match allowlist on the work-dir name (CodeQL path-injection)

The remaining 4 path-injection alerts were inside work_dir: I validated job_id
with an anchored regex but then joined a DIFFERENT f-string (`{job_type}_{job_id}`),
so CodeQL didn't carry the sanitization to the joined value. Mirror the pattern
the repo's _safe_cover_path uses (which CodeQL accepts): validate the WHOLE
joined component against an exact-match allowlist regex (_SAFE_SEG_RE), then
confine with os.path.commonpath containment (the recognized barrier) instead of
startswith. 13 resume tests still pass.

* fix(audiobook): basename-sanitize the work-dir name for CodeQL path-injection

The exact-match regex alone wasn't credited; route the joined value through os.path.basename() first — the sanitizer CodeQL recognizes (mirrors _safe_cover_path) — then the regex + commonpath. Functionally identical (no separator in the name) but clears the 4 remaining alerts. 13 tests pass.

* fix(audiobook): allow-list membership guard launders resume job_id (CodeQL)

The next(... if pair[1]==job_id) comparison-select didn't sanitize for CodeQL. Build a dict of resumable ids from the trusted scan and gate with 'if job_id not in resumable' — the membership barrier CodeQL recognizes — then use job_id directly downstream. 13 tests pass.

* fix(audiobook): eliminate request→path flow in resume (definitive CodeQL fix)

Five rounds of recognized path-injection barriers (regex, basename, exact-match,
commonpath, membership-guard) still left CodeQL flagging the resume job_id →
work-dir/manifest/log flow. Remove the flow entirely instead of guarding it:

- scan_resumable() now returns {job_type, job_id, manifest_path} where
  manifest_path is built from the os.listdir dir name (trusted), plus
  load_manifest_file(path) / discard_manifest_file(path) that operate on those
  trusted paths. The request job_id is used ONLY to *select* a scan entry, never
  to build a path.
- POST /audiobook/resume/{job_id} reads the manifest via the trusted scan path
  and renders under a FRESH server uuid (job_id=None). The chapter cache is
  content-addressed (keyed by chapter content, not the job id), so finished
  chapters still hit instantly — resume works, but the request's id never names
  a work dir, output file, or log line.
- The interrupted job's manifest is discarded (trusted path) once the fresh-id
  resume kicks off, so it stops showing as resumable.

Net: no request-controlled value reaches any file operation or log on the
render path (job_id there is always a server uuid). work_dir keeps its
confinement barriers as defence-in-depth. 13 resume tests pass.
2026-06-14 23:30:11 +05:30
Palash Debnath 4a0d18f510 perf(omnivoice): cache voice-clone prompt embeddings (#427) (#473)
Every cloned generation re-encoded the reference audio from scratch — a fixed
per-request latency that compounds on batch / long-form / dataset workloads that
reuse one saved voice across many calls.

The OmniVoice model already exposes the fast path (create_voice_clone_prompt →
VoiceClonePrompt, generate(voice_clone_prompt=)); the Studio backend just wasn't
using it. OmniVoiceBackend.generate now:
- builds a VoiceClonePrompt once per reference and caches it (bounded LRU, max 8,
  keyed by ref path + mtime + ref_text; thread-safe — generation runs in a GPU
  thread pool), then passes voice_clone_prompt= to skip the re-encode;
- falls back to the inline ref_audio/ref_text path on ANY cache miss or error,
  so output is identical either way (the model documents the two as equivalent)
  — this is purely a latency optimization, never a behaviour change;
- the design/instruct path (no ref_audio) is untouched.
- unload() clears the cache so a flush / engine-switch frees the prompt tensors.

tests/test_clone_prompt_cache.py: 6 cases (encode-once-then-hit, ref_text +
mtime invalidation, LRU eviction at the cap, encode-failure → None fallback,
clear). 6 passed.

Closes #427.
2026-06-14 22:08:35 +05:30
Palash Debnath fe7b59eeef feat(stories): global reading-speed control (#415) (#472)
The Stories editor only had a per-track speed slider; long scripts had no way to
set one speed for the whole thing. Add a global speed control that applies to
every line WITHOUT its own per-track override (the per-track slider still wins).

- storyToSpans(tracks, cast, globalSpeed): per-track speed wins, else the global
  speed, else engine default. 1.0× (and null) is treated as "no override" so a
  resting control never stamps an explicit speed on every span. Builds on the
  #27 default_speed plumbing already in the canonical parser.
- StoriesEditor: a global speed slider in the toolbar (0.5–2.0×, with reset),
  persisted to localStorage (UI preference — no project-state/slice migration).
- i18n: stories.global_speed / global_speed_hint in en.json.

Tests: storyToSpans.test.js +2 (global applies to un-overridden lines, per-track
wins; 1.0×/null/default-arg = no override). 17 file / 120 suite pass; CJK green.

Closes #415.
2026-06-14 21:58:36 +05:30
Palash Debnath 2b8c8aec7c fix: actionable errors for non-executable engine binary (#437) + unreachable backend (#438/#454/#466) (#471)
Two reliability bugs from open issues, both first-run papercuts where the error
told the user the wrong thing.

#437 — `[Errno 13] Permission denied: bin/omnivoice-tts-linux-x86_64`: a git
clone / zip extract on POSIX can drop the bundled binary's execute bit. It only
surfaced at spawn time, and the generic synth handler then mislabeled it as
"ran out of memory" and told the user to flush the model.
- omnivoice_gguf.is_available() now self-heals: after the SHA check confirms the
  binary is the right file, it adds +x (best-effort) on POSIX; if it can't, it
  returns a clear "isn't executable — run chmod +x <path>" message instead of a
  spawn-time crash. No-op on Windows.
- generation.py classifies PermissionError / EACCES / "Permission denied" as its
  own case ("a bundled binary lost its execute bit — reinstall or chmod +x"),
  so it never again masquerades as OOM.

#438/#454/#466 — bare "Failed to fetch" / "NetworkError": when the local backend
is still starting, crashed, or the dev server dropped, fetch() throws a TypeError
that propagated raw to the user.
- client.ts apiFetch now catches the thrown fetch and raises an ApiError with an
  actionable message ("Can't reach the local OmniVoice backend — it may still be
  starting up… restart the app or check Settings → Logs"), status:0 to mark a
  transport failure vs an HTTP error.

Tests: client.test.ts +1 (thrown fetch → ApiError status 0 + actionable text);
3 pass. CJK guard green.
2026-06-14 21:46:52 +05:30
Palash Debnath 129beb0ee6 test(settings): de-flake the at-rest-encryption assertion (#469)
test_stored_value_is_encrypted_not_plaintext asserted `"hf_" not in raw`, but
the stored value is Fernet URL-safe base64 whose alphabet includes `_`, so a
random ciphertext occasionally contains the substring `hf_` by chance — a
false failure that bit unrelated PRs on CI (~1 in N runs).

Replace the 3-char-prefix substring check (weak AND flaky) with stronger,
deterministic guarantees:
- the full token is absent from the raw column (kept),
- a 16-char leading chunk is absent (no partial leak; 62^16 ≈ never collides),
- and the value round-trips via get_hf_token() — proving it's genuinely
  encrypted, not merely absent/empty.

Verified non-flaky: the target test passed 8/8 consecutive runs.
2026-06-14 18:58:10 +05:30
Palash Debnath b998c383e7 feat(longform): JS canonical port + frontend convergence (#27 slice B) (#467)
Mechanically-mirrored JS twin of the Python parser, verified byte-for-byte against the shared golden corpus. See PR body.
2026-06-14 18:39:28 +05:30
Palash Debnath faa1b87226 docs(longform): retire the hand-sync comment now the corpus enforces parity (#27 slice C) (#468)
The SSML-lite client port header said "keep in sync with
backend/services/ssml_lite.py" — a manual contract with no test behind it.
After #27 the canonical longform grammar (incl. SSML-lite via the
longformParser.js → storyToSpans path) is asserted byte-for-byte against the
Python parser through the shared golden corpus
(tests/fixtures/longform_parser_cases.json), so drift between the two SSML impls
now fails CI. Update the comment to point at that enforcement.

No user-facing docs document the marker dialect (verified by grep: only the
internal competitive-analysis planning doc references it), so no docs-sync
update is required for the converged behaviour.
2026-06-14 18:36:45 +05:30
Palash Debnath 276875c397 feat(longform): canonical Python parser + golden corpus (#27 slice A) (#465)
The longform marker dialect (# heading / [voice:] / [pause] / SSML-lite) was
parsed by three independent code paths that already disagreed (client vs server
on [pause] units, [voice:] empty, H1-only chapters). This lands the single
canonical Python parser; the JS port + cross-impl test follow in slice B.

- New backend/services/longform_parser.py — parse_script_to_spans(text, *,
  default_voice, default_speed) + _parse_chapter_body (the reusable voice→pause
  →SSML layering the JS twin mirrors). Moves the H1/voice regexes verbatim from
  audiobook.py (already CodeQL-cleared), reuses parse_pause_markers + ssml_lite
  unchanged. Coerces None→"" and normalizes CRLF/CR→LF at entry (cross-platform
  parity so Windows-authored scripts never carry a stray \r). Adds default_speed
  plumbing (inline SSML speed overrides the per-line default).
- audiobook.py: parse_audiobook_script is now a thin wrapper that wraps the
  canonical span dicts in Span/Chapter/AudiobookPlan — public return type and
  .to_dict() shape unchanged, all four router call sites untouched. Deleted
  _parse_spans / _HEADING_RE / _VOICE_RE and the now-dead `import re` +
  parse_pause_markers import.
- tests/fixtures/longform_parser_cases.json — 78-case golden corpus (≥40
  required) covering §A–I: H1-only chapters (H2–H6 + `# ` no-title → body), the
  full pause dialect incl. the NO-MATCH boundary, banker's-rounding ties
  ([pause 0.5]→0, [pause 1.5]→2), [voice:] empty→default, [voice:[nested]]
  literal, SSML nesting/spell/unknown-tag, speed override, CRLF, combined
  precedence. Generated from actual parser output (the truth the JS port must
  match).
- tests/test_longform_parser.py — parametrized over the corpus + None-input +
  ReDoS-linearity (5000× repeats < 1 s).

130 passed (corpus + test_audiobook + test_pause_markers + test_ssml_lite all
green); CJK guard green.
2026-06-14 18:26:17 +05:30
Palash Debnath d0e1c19e88 docs(persona): document the .ovsvoice portable format (#29 slice D) (#464)
- docs/persona-format.md: export (privacy/include-reference, watermarked
  preview), import (consent/verification non-forgeability rule), the ZIP layout
  table, SPDX-license semantics (metadata only), and the local-first / zero-
  network guarantee. Notes legacy .omnivoice compatibility.
- CHANGELOG.md: [Unreleased] → Added entry for portable personas.

Satisfies the docs-sync hard rule for the new bundle format.
2026-06-14 18:04:59 +05:30
Palash Debnath 0338d4c900 feat(persona): export/import UI for .ovsvoice bundles (#29 slice C) (#462)
* feat(persona): .ovsvoice build/parse core + embed_watermark(force=) (#29 slice A)

Extends the merged persona-bundle nucleus (constants, normalize_spdx,
build_manifest, build_consent_json) with the model-coupled core that the
export/import router (next slice) will sit on:

- `build_persona_bundle(profile, *, license_spdx, tags, include_reference,
  embed_fn, …)` → assembles the .ovsvoice ZIP in memory: a watermarked
  preview.wav (24 kHz mono 16-bit, downmixed + resampled + trimmed ≤8 s),
  manifest.json, a legacy-shaped metadata.json (so an older OmniVoice can still
  import the ref audio), optional consent.json, and the raw ref/locked/consent
  members unless include_reference=False (privacy / preview-only, A12). Raises
  NoPreviewSource (router → 503) when no source clip is readable (A2-A5).
- `parse_persona_bundle(bytes)` → validates the ZIP, prefers manifest.json and
  falls back to legacy metadata.json, resolves audio members by prefix
  (last-wins, B9; member names never build paths — zip-slip safe), normalizes
  the SPDX id, flags preview-only / future-schema_version. Raises
  BundleError(400|413) for B1-B11. No DB, no file writes.
- `ParsedPersona` dataclass with `extract_member(prefix, dest_path)` — the
  router derives dest_path from the server-generated id, never the member name.
- `embed_watermark(..., *, force=False)`: keyword-only flag that bypasses the
  user's invisible-watermark preference for the mandatory persona preview, but
  still no-ops without AudioSeal. All existing positional call sites are
  unchanged (default force=False) — default cross-platform behaviour identical.

All heavy imports (torch/torchaudio/watermark/audio_io) are lazy so the module
stays model-free at collection (avoids the local torch/Triton segfault).

tests/test_persona_bundle.py: +31 cases — parse validation (manifest/legacy
selection, preview-only, future-schema, missing/malformed/no-audio → 400,
oversize → 413, bad-SPDX normalize, last-wins dup, advisory consent), build
round-trip (identity fields, metadata sibling, no-source → NoPreviewSource,
include_reference=False, stereo/off-rate downmix+resample), and the force=
unit (D1/D3). 25 pure cases pass locally; the 6 torchaudio-coupled cases run on
CI (local torch+pytest segfault is pre-existing). CJK guard green.

* feat(persona): /personas export·import·inspect router + wiring (#29 slice B)

Thin HTTP layer over the persona_bundle service (slice A), registered in main.py
next to the legacy marketplace router:

- POST /personas/export/{id} → builds the .ovsvoice off the event loop
  (run_in_executor) and streams it (application/zip, .ovsvoice filename;
  empty name → persona_<id>). 404 when the profile is missing;
  NoPreviewSource → 503 (no readable source audio); any other build error → 503
  with a generic message (no raw exception text in the body).
- POST /personas/import → parse (BundleError → its HTTP status), extract audio
  members to server-named files ({id}{ext}/{id}_locked{ext}/{id}_consent{ext} —
  never the member name, zip-slip safe via profiles._voices_path), 17-column
  INSERT (legacy 13 + the 4 consent columns), event_bus emit after commit.
  Verified-own-voice is granted ONLY with a real recording ≥ floor AND non-empty
  consent_text AND consent.json present (forgery guard, B12-B16). Rollback:
  every written file is deleted on any extraction/INSERT failure; id-collision
  retries once (renaming the on-disk files to the new id). Accepts legacy
  .omnivoice too (case-insensitive extension guard).
- POST /personas/inspect → manifest + consent summary with NO DB row and NO
  file extracted (import-preview UI).

backend/tests/test_personas_api.py: 13 cases (config-stub pattern → mounts only
the router, no main/torch import) — export 404; import bad-ext/non-zip/missing-
manifest 400; round-trip row+file under server name; case-insensitive ext;
forgery-unverified; verified-with-recording; short-recording-unverified;
preview-only-as-ref; legacy .omnivoice; inspect no-write + consent summary.
13 passed locally. CJK guard green.

* feat(persona): export/import UI for .ovsvoice bundles (#29 slice C)

Wires the persona endpoints (slice B) into the voice UI:

- api/profiles.ts: exportPersona (blob download, builds the license/tags/
  include_reference query), importPersona, inspectPersona; PersonaImportResult
  + PersonaBundleMeta types in types.ts.
- VoiceProfile.jsx: "Export persona" toolbar action + a privacy "Include voice
  clip" checkbox (default ON; off → preview-only bundle, no raw reference clip).
  Triggers a blob download named <voice>.ovsvoice; distinct toast for the 503
  no-audio case vs a generic failure.
- VoiceGallery.jsx (My Imports): an Import-persona button next to Upload, accept
  ".ovsvoice,.omnivoice", that POSTs to /personas/import and refreshes the
  voice list. Surfaces the 413 too-large case distinctly; flags an unverified
  import in the success message.
- i18n: voice_profile.persona_* + gallery.persona_*/import_persona keys in
  en.json only (fallbackLng=en covers other locales).

Tests: frontend/src/api/profiles.persona.test.ts (7 cases — export query
construction incl. include_reference omitted-when-true, non-ok → throws status,
blob passthrough; import/inspect post FormData to the right path). Full suite
408 passing; en.json valid; CJK guard green. No new tsc errors in the changed
files (pre-existing errors elsewhere are unaffected).
2026-06-14 17:56:39 +05:30
Palash Debnath 35c063ae52 feat(persona): /personas export·import·inspect router + wiring (#29 slice B) (#461)
* feat(persona): .ovsvoice build/parse core + embed_watermark(force=) (#29 slice A)

Extends the merged persona-bundle nucleus (constants, normalize_spdx,
build_manifest, build_consent_json) with the model-coupled core that the
export/import router (next slice) will sit on:

- `build_persona_bundle(profile, *, license_spdx, tags, include_reference,
  embed_fn, …)` → assembles the .ovsvoice ZIP in memory: a watermarked
  preview.wav (24 kHz mono 16-bit, downmixed + resampled + trimmed ≤8 s),
  manifest.json, a legacy-shaped metadata.json (so an older OmniVoice can still
  import the ref audio), optional consent.json, and the raw ref/locked/consent
  members unless include_reference=False (privacy / preview-only, A12). Raises
  NoPreviewSource (router → 503) when no source clip is readable (A2-A5).
- `parse_persona_bundle(bytes)` → validates the ZIP, prefers manifest.json and
  falls back to legacy metadata.json, resolves audio members by prefix
  (last-wins, B9; member names never build paths — zip-slip safe), normalizes
  the SPDX id, flags preview-only / future-schema_version. Raises
  BundleError(400|413) for B1-B11. No DB, no file writes.
- `ParsedPersona` dataclass with `extract_member(prefix, dest_path)` — the
  router derives dest_path from the server-generated id, never the member name.
- `embed_watermark(..., *, force=False)`: keyword-only flag that bypasses the
  user's invisible-watermark preference for the mandatory persona preview, but
  still no-ops without AudioSeal. All existing positional call sites are
  unchanged (default force=False) — default cross-platform behaviour identical.

All heavy imports (torch/torchaudio/watermark/audio_io) are lazy so the module
stays model-free at collection (avoids the local torch/Triton segfault).

tests/test_persona_bundle.py: +31 cases — parse validation (manifest/legacy
selection, preview-only, future-schema, missing/malformed/no-audio → 400,
oversize → 413, bad-SPDX normalize, last-wins dup, advisory consent), build
round-trip (identity fields, metadata sibling, no-source → NoPreviewSource,
include_reference=False, stereo/off-rate downmix+resample), and the force=
unit (D1/D3). 25 pure cases pass locally; the 6 torchaudio-coupled cases run on
CI (local torch+pytest segfault is pre-existing). CJK guard green.

* feat(persona): /personas export·import·inspect router + wiring (#29 slice B)

Thin HTTP layer over the persona_bundle service (slice A), registered in main.py
next to the legacy marketplace router:

- POST /personas/export/{id} → builds the .ovsvoice off the event loop
  (run_in_executor) and streams it (application/zip, .ovsvoice filename;
  empty name → persona_<id>). 404 when the profile is missing;
  NoPreviewSource → 503 (no readable source audio); any other build error → 503
  with a generic message (no raw exception text in the body).
- POST /personas/import → parse (BundleError → its HTTP status), extract audio
  members to server-named files ({id}{ext}/{id}_locked{ext}/{id}_consent{ext} —
  never the member name, zip-slip safe via profiles._voices_path), 17-column
  INSERT (legacy 13 + the 4 consent columns), event_bus emit after commit.
  Verified-own-voice is granted ONLY with a real recording ≥ floor AND non-empty
  consent_text AND consent.json present (forgery guard, B12-B16). Rollback:
  every written file is deleted on any extraction/INSERT failure; id-collision
  retries once (renaming the on-disk files to the new id). Accepts legacy
  .omnivoice too (case-insensitive extension guard).
- POST /personas/inspect → manifest + consent summary with NO DB row and NO
  file extracted (import-preview UI).

backend/tests/test_personas_api.py: 13 cases (config-stub pattern → mounts only
the router, no main/torch import) — export 404; import bad-ext/non-zip/missing-
manifest 400; round-trip row+file under server name; case-insensitive ext;
forgery-unverified; verified-with-recording; short-recording-unverified;
preview-only-as-ref; legacy .omnivoice; inspect no-write + consent summary.
13 passed locally. CJK guard green.
2026-06-14 17:56:01 +05:30
Palash Debnath 4500dcb6b4 feat(persona): .ovsvoice build/parse core + embed_watermark(force=) (#29 slice A) (#460)
Extends the merged persona-bundle nucleus (constants, normalize_spdx,
build_manifest, build_consent_json) with the model-coupled core that the
export/import router (next slice) will sit on:

- `build_persona_bundle(profile, *, license_spdx, tags, include_reference,
  embed_fn, …)` → assembles the .ovsvoice ZIP in memory: a watermarked
  preview.wav (24 kHz mono 16-bit, downmixed + resampled + trimmed ≤8 s),
  manifest.json, a legacy-shaped metadata.json (so an older OmniVoice can still
  import the ref audio), optional consent.json, and the raw ref/locked/consent
  members unless include_reference=False (privacy / preview-only, A12). Raises
  NoPreviewSource (router → 503) when no source clip is readable (A2-A5).
- `parse_persona_bundle(bytes)` → validates the ZIP, prefers manifest.json and
  falls back to legacy metadata.json, resolves audio members by prefix
  (last-wins, B9; member names never build paths — zip-slip safe), normalizes
  the SPDX id, flags preview-only / future-schema_version. Raises
  BundleError(400|413) for B1-B11. No DB, no file writes.
- `ParsedPersona` dataclass with `extract_member(prefix, dest_path)` — the
  router derives dest_path from the server-generated id, never the member name.
- `embed_watermark(..., *, force=False)`: keyword-only flag that bypasses the
  user's invisible-watermark preference for the mandatory persona preview, but
  still no-ops without AudioSeal. All existing positional call sites are
  unchanged (default force=False) — default cross-platform behaviour identical.

All heavy imports (torch/torchaudio/watermark/audio_io) are lazy so the module
stays model-free at collection (avoids the local torch/Triton segfault).

tests/test_persona_bundle.py: +31 cases — parse validation (manifest/legacy
selection, preview-only, future-schema, missing/malformed/no-audio → 400,
oversize → 413, bad-SPDX normalize, last-wins dup, advisory consent), build
round-trip (identity fields, metadata sibling, no-source → NoPreviewSource,
include_reference=False, stereo/off-rate downmix+resample), and the force=
unit (D1/D3). 25 pure cases pass locally; the 6 torchaudio-coupled cases run on
CI (local torch+pytest segfault is pre-existing). CJK guard green.
2026-06-14 17:55:17 +05:30
Palash Debnath ca8a2e8eb8 feat(audiobook): PDF ingest for /audiobook/import (ebook-in core value) (#459)
The audiobook importer accepted .txt/.md/.epub but not PDF — the single most
common "ebook in" format. Add a pure `pdf_to_chapter_script(data)` that
extracts the text layer page-by-page and runs it through the existing
chapterizer, so PDFs land in the same `# Heading` + body grammar EPUB and
plaintext already produce (one front door onto the unchanged render pipeline).

- Dep: `pypdf>=4.0` — pure-Python, MIT, zero native deps, so PDF import behaves
  identically on macOS/Windows/Linux (default-feature cross-platform rule).
  EPUB + plaintext stay stdlib-only; only PDF needs a real parser.
- Robustness, surfaced as actionable 400s rather than silent empty imports:
  corrupt file, password-protected (empty-password decrypt attempted first),
  scanned/image-only (no text layer → clear "scanned PDF" message), and a
  page-count ceiling. A single unparseable page is skipped, not fatal.
- Route: `.pdf` branch in audiobook_import; frontend accept filter +
  api-client doc updated to `.txt,.md,.epub,.pdf`.

tests/test_longform_import.py: 5 PDF cases (extract+chapterize, no-marker
single chapter, corrupt, image-only, page-cap) using a hand-built in-memory
PDF — no PDF-authoring test dep, mirroring the in-memory-EPUB approach.
16 passed; frontend suite 401; CJK guard green.
2026-06-14 17:14:28 +05:30
Palash Debnath 142b4bc25a feat(dub): wire second-pass timing QC into the dub editor UI (#458)
The Wave 3.3 QC backend was complete but unreachable from the UI: the
`POST /dub/qc/{job_id}` route (re-recognizes the dubbed audio, scores per-line
drift vs the target text, annotates segments with qc_drift/qc_flagged/
qc_recognized/qc_measured_start-end), the `dubQc()` API client, and the
DubSegmentRow "Verify" badge all existed — but nothing ever called the route,
so the badge never lit and the measured timings were never surfaced.

Add a "Verify dub timing" action to the dub editor header (shown once
dubStep === 'done'):
- Calls `dubQc(jobId, lang)` for the currently-previewed language.
- Merges the returned per-segment scores back onto dubSegments by id, so
  flagged lines light their re-listen badge and carry the measured onsets.
- Toast summary: "{flagged} of {total} lines may need a re-listen", or a
  clean-pass success when nothing drifted. Loading + error states handled;
  non-destructive (generated text untouched).

i18n: dub.qc_btn / qc_running / qc_result / qc_clean / qc_failed in en.json
(fallbackLng=en covers other locales). Frontend suite green (401).
2026-06-14 17:07:04 +05:30
Palash Debnath 4531e999b1 feat(capture): opt-in LLM refinement on REST /transcribe (parity with live dictation) (#457)
The live-dictation socket (capture_ws) already runs the final transcript
through the configured local LLM (disfluency/self-correction/punctuation
cleanup, Wave 2.1). The REST /transcribe endpoint — the MCP / CLI / file-upload
surface — only did the always-on hallucination-loop collapse, so agentic and
batch callers couldn't get the same cleaned output.

Add an opt-in `refine` form flag that runs the identical `maybe_refine`
pipeline off-thread:
- OFF by default → existing MCP/CLI callers keep raw-only output and pay no
  LLM latency (backward-compatible).
- Honours the user's Settings → Dictation-refinement config and silently
  passes through when no LLM backend is configured (cross-platform default
  parity — identical no-op everywhere with no LLM).
- Raw `text` is always returned; `refined_text` is added only when the LLM
  actually changed the text — same contract the socket emits.

tests/test_capture_refine.py: 13 cases — flag-off no-call, refined_text on
change, no-op/identical omission, and flag parsing. maybe_refine is patched at
its source module since the handler imports it lazily.
2026-06-14 17:00:23 +05:30
Palash Debnath 875f840d8e chore(issues): structured GitHub Issue Forms (bug / install / feature) + config (#456)
Replace the two flat markdown templates with validated YAML Issue Forms and a
chooser config, so reports arrive with the diagnostic fields triage actually
needs and "how do I…" traffic routes to chat instead.

- `bug_report.yml` — dup-search + latest-version checkboxes; required
  what/repro/expected; OS / install-method / version / compute-device dropdowns
  (incl. ROCm + XPU); active-engine; logs (render: text) with the diagnostic-
  bundle + `--diagnose` tip up top.
- `install_problem.yml` — NEW, for the "first-run that just works" core value:
  a failure-stage dropdown (launch / uv-bootstrap / model-download / engine-
  install / first-synth), required error + OS/install/version, and a
  network-conditions dropdown (proxy / restricted-region / offline) since
  restricted networks are a known bootstrap failure mode.
- `feature_request.yml` — problem/solution/alternatives + an Area dropdown, with
  a local-first/cross-platform constraints note so proposals fit.
- `config.yml` — `blank_issues_enabled: false`; contact links to Discord,
  Discussions, and the private security policy.

Removes bug_report.md / feature_request.md (superseded). Forms validated (yaml
parse); SECURITY.md backs the security link; CJK guard green.
2026-06-14 16:05:20 +05:30
Palash DebnathandClaude Opus 4.8 d20c24e1e1 feat(longform): two-pass loudnorm measure orchestrator + wiring (#28 slice 2) (#455)
* feat(longform): two-pass loudnorm measure orchestrator + wiring (#28 slice 2)

Completes accurate ACX/podcast mastering end-to-end (builds on the pure builders
from #28 slice 1).

- `services/loudness.py` — `measure_loudness(ffmpeg, concat, preset, *, job_id)`:
  runs ffmpeg's measure pass, parses the loudnorm JSON → MeasuredLoudness.
  **Never raises** — skip / non-zero rc / rc None / asyncio.TimeoutError / spawn
  OSError / empty or unparseable stderr / silent program all WARN + return None
  → single-pass fallback (a slow/broken measure degrades the master, never
  aborts the render). Logs rc + a static message only, never the raw stderr
  (path-safe / local-first). UTF-8 decode with replacement (Windows-cp safe).
- `_render_longform_sse` (audiobook.py): between the concat write and the mux,
  when `loudness` is a known preset (acx/podcast; same `.lower()`/no-strip gate
  as the builders) → emit a `mastering` event, measure, and pass `measured` into
  `build_render_cmd` (two-pass apply; `None` → single-pass). `done` gains a
  `loudness` block {preset, target_i, target_tp, two_pass, measured_i} ONLY for
  a requested preset — off/None paths keep the byte-identical legacy `done`
  shape. Both front doors (/audiobook + /longform/render) get it via the shared
  generator. Chapter cache key is deliberately untouched (loudness-agnostic →
  acx/off reuse the same cached WAVs; no re-render, no cache-layout break).

Tests: `test_loudness.py` (14 — happy fixture, skip-without-spawn for off/
unknown/whitespace/None, non-zero/None rc, timeout-not-propagated, OSError,
empty/unparseable stderr, non-UTF-8 stderr, job_id+argv forwarding) + 2 e2e
cases (mastering event + done.loudness present for acx; absent for off). Orch
tests run locally (stubbed run_ffmpeg, no torch); e2e on CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(loudness): lazy-import run_ffmpeg so the measure stub survives sys.modules purges

test_loudness monkeypatched services.loudness.run_ffmpeg, but the route-shape
fresh_app fixture purges services.* from sys.modules, so under the full-suite
ordering the patch missed the re-imported module → real ffmpeg ran → 3 failures.
Lazy-import run_ffmpeg inside measure_loudness and patch it at its source
(services.ffmpeg_utils.run_ffmpeg) so the stub is always picked up at call time.
Verified by running the purging suite + test_loudness together (31 pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 16:03:28 +05:30
Palash DebnathandClaude Opus 4.8 4a75d694e6 feat(persona): .ovsvoice manifest + SPDX + consent core (#29 / parity §R3 G1, pure) (#453)
The model-free nucleus of the portable .ovsvoice persona-bundle format: format
constants, SPDX normalization, and the manifest/consent builders — all pure (no
torch, no I/O), fully locally testable. The audio preview + ZIP pack/unpack +
watermark `force=` param + router + frontend are follow-on slices.

- Constants: OVSVOICE_FORMAT/SCHEMA_VERSION, MAX_BUNDLE_BYTES (100 MB),
  DEFAULT_LICENSE (`LicenseRef-OmniVoice-Personal`), the SPDX allowlist.
- `normalize_spdx()` — membership + `LicenseRef-` prefix; junk/None/injection →
  DEFAULT_LICENSE, never raises/400s. No regex over the SPDX string (CodeQL-clean).
- `build_manifest()` — mirrors the legacy `_bundle_metadata` persona fields into
  the manifest + format discriminator + normalized license + tags + engine /
  preview / members blocks. seed/vd_states pass through (None-safe; vd_states is
  a JSON string, never re-parsed). `BundleError(status, detail)` for the router.
- `build_consent_json()` — designed-synthetic for `kind='design'`, self-recorded
  for an attested clone, None when nothing to attest; `recorded_at` coerced.
  Fields are advisory by design — real verification needs the actual consent
  audio member, so verified-own-voice can't be forged by editing a manifest.

Tests: 14 cases (SPDX allowlist/prefix/junk/strip, manifest schema + field
mirror + None-passthrough + bad-license-normalized, consent design/clone/none/
coerce). Backend pytest green; CJK guard green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 13:57:39 +05:30
Palash Debnath 53c6845784 fix(ui): app-shell scales via zoom and always fills the viewport — permanent black-band fix (#452)
Root cause: uiScale DEFAULTS to 1.3, so the shell's `width: calc(100vw/--ui-scale)`
+ `transform: scale(--ui-scale)` path is active for every user. On WebKitGTK
(the Linux webview) the transform wasn't magnifying the shrunk shell, so
`calc(100vw/1.3)` left ~⅓ of the window black — on EVERY view, by default.
(The earlier #445 fix addressed the responsive breakpoints, not this — wrong
layer.)

Permanent fix: scale via `zoom` and keep the shell at full `100vw × 100vh`
(drop the `calc(…/scale)` shrink + the `transform`):
- Chromium (mac/win): `zoom` magnifies AND fills (standard browser zoom — same
  mechanism the bootstrap/wizard wrappers already use).
- WebKitGTK (Linux): `zoom` is a no-op → UI renders at 1.0× but the shell is a
  plain 100vw×100vh element → it FILLS, no band. A missed magnification now
  degrades to "unscaled but full", never "shrunk + black band".

Regression-proofed: `src/test/appShellScale.test.js` fails CI if anyone
reintroduces `width: calc(100vw/var(--ui-scale))` or
`transform: scale(var(--ui-scale))` on the shell, or drops the zoom/100vw/100vh
contract — so a future change can't silently bring the band back. The fix +
guard are documented inline in the `.app-container` rule.

Full vitest green (398, incl. the 3-case guard); typecheck:ci + vite build clean.
2026-06-14 13:47:28 +05:30
8e3c1a8bcc fix(realtime): probe auth-exempt /health, not gated /model/status (#450) (#451)
The cold-start health probe added in #439 used a raw fetch() to
/model/status. Raw fetch does not carry the LAN PIN / remote API-key
headers that apiFetch attaches, and /model/status is not in the backend
_SHELL_PATHS allowlist, so it is gated by NetworkAccessMiddleware and
BearerKeyMiddleware. In LAN-share / remote-API mode the probe gets 401,
rejects forever, and the realtime-events WebSocket never opens.

Probe /health instead — the auth-exempt liveness endpoint (in
_SHELL_PATHS) that returns 200 as soon as Uvicorn is up. Using
apiUrl('/health') also avoids a double-slash when the API base has a
trailing slash. Default loopback desktop use is unaffected.

Adds a regression test asserting the probe targets /health (not a gated
path) and only opens the WebSocket after the probe succeeds.

Fixes #450

Co-authored-by: mergetest <hashduch@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 13:46:59 +05:30
Palash DebnathandClaude Opus 4.8 e1c8c3bc0d feat(longform): two-pass loudnorm builders + parser (#28 slice 1 — pure) (#449)
Groundwork for accurate ACX mastering: the pure, ffmpeg-free pieces of the
two-pass loudnorm upgrade, layered over the existing single-pass builders
(which stay). The async measure orchestrator + SSE wiring into the render path
is slice 2.

- `MeasuredLoudness` (frozen dataclass: the 5 measure-pass floats).
- `build_loudnorm_measure_filter(preset)` — first pass (+print_format=json);
  mirrors build_loudnorm_filter's lookup (no strip) so the same inputs map to
  "no filter".
- `parse_loudnorm_measure(stderr)` — extracts the LAST balanced {...} via a
  linear brace-depth scan (NO regex → CodeQL-safe), json.loads + coerces the 5
  keys to finite floats; returns None on the full failure matrix (absent/empty/
  unbalanced/malformed/missing-key/non-numeric/non-finite "-inf"/array/scalar).
  Rejecting "-inf" is the silent-clip path → single-pass fallback.
- `build_loudnorm_apply_filter(preset, measured)` — second pass feeding
  measured_*/offset back in with linear=true; None for off/unknown OR measured
  is None.
- `build_loudnorm_measure_cmd(ffmpeg, concat, filt)` — exact 16-element argv,
  input segment byte-identical to build_render_cmd (measured == muxed),
  portable `-f null -` sink (no /dev/null or NUL).
- `build_render_cmd` gains `measured: Optional[MeasuredLoudness] = None`: apply
  two-pass when present, else single-pass; off-render still emits no -af. The
  `measured=None` default keeps every existing caller + argv byte-identical.

Loudness stays opt-in (default None) → default cross-platform behavior unchanged.

Tests: 28 cases — measure-filter goldens + off/unknown/whitespace; parser
success (last-block-wins, ignores extra keys) + full failure matrix +
non-finite rejection; apply-filter golden + None cases; exact measure argv;
build_render_cmd two-pass/single-pass/off branches. Backend pytest green (71).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 13:10:01 +05:30
Palash DebnathandClaude Opus 4.8 e297cbfee3 feat(longform): TranscriptionPicker + shared reader util (#23 slices 1–2) (#448)
Groundwork for "import from a past dictation": the shared store reader + the
reusable picker modal, fully unit/RTL-tested. The two-tab wiring (Audiobook
Replace/Append prompt + Stories split-panel routing) is slice 3 — deferred for
visual verification.

Slice 1 — shared reader (`utils/transcriptionsStore.js`):
- `loadTranscriptions()` (parse + Array.isArray guard, [] on
  absent/empty/malformed/non-array/blocked-storage) + `TRANSCRIPTIONS_KEY` /
  `TRANSCRIPTION_EVENT` consts. Kills the third copy of the localStorage parse.
- Refactored `Transcriptions.jsx` + `Projects.jsx` onto it (behavior-preserving;
  the Array.isArray guard is a superset that only hardens against corrupt
  blobs). Storage key/shape/200-cap unchanged → no migration.

Slice 2 — `components/TranscriptionPicker.jsx`:
- Controlled modal wrapping the shared `ui/Dialog` (Radix → focus trap, ESC,
  backdrop, ARIA inherited). Reads on open, subscribes to the add-event only
  while open. Per-row display normalization, hides empty-text rows, distinct
  empty vs empty-search states, case-insensitive `String.includes` search (no
  RegExp → no ReDoS surface), keyboard-activatable `<button>` rows, Invalid-Date
  guard. `onPick` gets the original un-normalized entry. Every string via t().

Tests: util edge matrix (3) + picker RTL (7: empty, list+hide-empty,
click→onPick+onClose, keyboard rows, search filter + empty-search, bad-timestamp
chip omitted, live-refresh on event). Full vitest green; typecheck:ci clean;
CJK guard green (new files i18n-only).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 12:54:07 +05:30
baoyu0 4817fd1c1f fix: poll backend HTTP before WebSocket connect to avoid startup ECONNREFUSED (#439)
The frontend mounts faster than the Python backend (which takes ~14s to
import torch/fastapi before Uvicorn starts).  useRealtimeEvents was
creating a WebSocket immediately, which always failed with code 1006
on the first attempt, triggering an unnecessary exponential-backoff
reconnect.

Fix: poll /model/status via HTTP fetch before creating the WebSocket.
Once the backend responds 200, proceed to open the WS.  If the health
check fails, schedule a reconnect using the same backoff — but without
the noisy 'closed (code=1006)' log.

The /model/status endpoint is chosen because it's already polled by the
TanStack Query hooks and always returns 200 once Uvicorn is running,
even before models are loaded.
2026-06-14 12:43:57 +05:30
Palash DebnathandClaude Opus 4.8 95289b8192 fix(ui): scale-aware shell breakpoints — no more cramped/black layout at narrow widths (#445)
The app shell is sized `width: calc(100vw / --ui-scale)` then `transform:
scale(--ui-scale)` (the WebKitGTK fix, #407), so its grid lays out against
`100vw / scale`. But the responsive collapse used viewport `@media (max-width)`
queries, which fire on raw `100vw` — so at any `--ui-scale ≠ 1` they trip at the
wrong threshold. In a narrow window the 3-column grid was kept, the sidebar's
`min 180px` crushed the main column toward 0, and the content ended up jammed
into a left sliver with a black band filling the rest.

Fix: drive the breakpoints off the shell's OWN width. A ResizeObserver on the
app-container reads `el.clientWidth` (= the pre-transform layout width =
100vw/scale; transforms don't change the layout box) and toggles `shell-narrow`
(≤1100) / `shell-mini` (≤600) classes; the `@media` queries become equivalent
`.app-container.shell-*` rules. Correct on every engine and at every UI scale.
Observer fires on both window resize and scale change (the calc width changes).

Needs a visual check in the running app at a couple of window sizes + UI scales.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 12:43:53 +05:30
Palash DebnathandClaude Opus 4.8 52eeb0b194 feat(longform): Story⇄Audiobook convert transforms (#24 slice 1 — pure utils) (#447)
The render-faithful interchange between the two long-form editors, as pure,
unit-tested functions (no UI/store yet — that's slice 2). The store seam for
this (convertMode/projectMode) already shipped in #31a.

- `storyToScript(tracks, cast, {projectName})` → `{script, defaultVoice,
  metadata}`. Emits **profile-id** `[voice:]` tags (the backend resolver keys on
  id, not display name) so the script renders identically through
  /longform/render from either door. Most-used effective voice → defaultVoice
  (no tag), deterministic earliest-occurrence tie-break; tags emitted only on
  voice change; single-# un-indented headings; inline markup ([pause], SSML-lite,
  emotion) passes through verbatim — never re-tokenized (no drift vs the backend
  parser). Respects the three client/server divergences (heading depth,
  [voice:default] semantics, [pause] dialect): it never synthesizes a pause and
  never emits [voice:default].
- `scriptToStory(text, profiles)` → `{tracks, cast}` (persisted StoryTrack shape;
  cast always ≥ a narrator clone). One physical line = one track; a leading
  [voice:id] becomes the track override + a cast member (named from profiles or
  the raw id, which is kept as profileId so it round-trips); mid-line markup +
  body text preserved byte-for-byte; CRLF normalized; slug-collision-safe cast
  ids; sequential numeric ids.
- No new regex over user input (leading-voice detection is string ops) —
  CodeQL-clean; render output stays identical across both doors (the invariant).

Tests: 19 cases incl. the edge matrix + **round-trip equivalence** both
directions (script→story→script reproduces; story→script→story preserves spoken
text + voice mapping). Full vitest green; typecheck:ci clean; CJK guard green.

Deferred (slice 2, needs visual verify): the two UI buttons, store prefill
fields, mount read-clear effects, AppMode 'audiobook' fix, i18n.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 12:42:28 +05:30
Palash DebnathandClaude Opus 4.8 c1e3031cfa feat(audiobook): use shared VoiceSelector for the default-voice picker (#22 migration 1/N) (#446)
First call-site migration onto the shared <VoiceSelector> (#22): the Audiobook
default-voice <select> becomes the searchable, grouped picker. Value contract is
unchanged ('' = engine default | profileId), already store-bound (#31b), so no
behavior or data change — just search + clone/designed grouping. `defaultLabel`
preserves the existing "engine default" row label.

Stories cast / per-line track / Dub segment pickers are intricate live layouts
(custom select CSS, row composition) — deferred to follow-up migrations that can
be visually verified, rather than blind-swapped.

vitest green (357); typecheck:ci clean; vite build clean.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 12:29:43 +05:30
Palash DebnathandClaude Opus 4.8 b6f0c73f7a feat(audiobook): persist book metadata/script/prefs via LongformProject store (#31b) (#444)
Audiobook's script, default voice, output format, loudness, book metadata
(title/author/narrator/genre/year/description) and pronunciation lexicon now
bind to the unified store (#31a) instead of component useState — so they
**survive a tab switch / reload** (previously all lost). The headline #31 win.

- text→script, defaultVoice, format→outputFormat, loudness, meta→setProjectMeta,
  bound to store selectors. `meta` is default-filled so an empty record never
  flips a controlled input to uncontrolled.
- Lexicon rows stay LOCAL (half-typed rows aren't junk-persisted); the filtered
  dict flushes to the store on change and hydrates back into rows on mount.
- Transient state (plan, generating, progress, output, chapter previews) stays
  component-local — correctly NOT persisted.

Deferred (noted): coverRef persistence (a File/blob can't go to localStorage);
the "Save as named project" affordance + Projects-list card + App `onOpenStory`
mode-aware routing (criterion 4 — re-open from Projects). This slice lands the
working-state persistence (criterion 3); save/reopen is the next slice.

Full frontend vitest green (357); typecheck:ci clean; CJK guard green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 12:18:20 +05:30
Palash DebnathandClaude Opus 4.8 0a72a75ed2 feat(ui): shared VoiceSelector component + SearchableSelect grouping (#22) (#442)
A single searchable, grouped voice picker to replace the per-tab <select>s
across Stories / Audiobook / Dub. This slice ships the COMPONENT + the two
backward-compatible SearchableSelect extensions it needs; the call-site
migrations are a follow-up slice (component lands first, tested in isolation).

- `SearchableSelect` gains two opt-in, back-compat props (the two existing
  call sites are untouched, both render-identically):
  - `renderGroupHeaders` (default false) — emits a `.ss-group-label` header on
    the first MAIN row of each new `option.group` with a non-empty `groupLabel`
    (pinned recent/popular rows never trigger one; empty groups never emit a
    stray header).
  - `isRecentable` (default `() => true`) — gates which committed values get
    recorded as recents.
- `VoiceSelector` builds a group-ordered options array (default → fromVideo →
  clone → designed → preset) over the EXISTING value contract
  ('' | id | preset:<id> | auto:<slug>) — byte-identical to what every call
  site already sends, so project data stays compatible. Clone-vs-designed
  splits on the runtime `.instruct` string (matching VoicePreview), not
  `.kind`. Renders optional preview / gallery-jump / create adornments (the
  component owns no audio and makes no API call — it only emits the value and
  fires the parent's callbacks). A deleted-but-referenced voice renders a
  "Voice not found (re-pick)" ghost row WITHOUT auto-clearing the value.
  `isRecentable` excludes '' / preset: / auto: so only real voices are recents.
- i18n keys under `voiceSelector.*` (en.json; other locales fall back via
  fallbackLng, matching the project's established pattern).

Tests: 9 RTL cases — grouping/headers, value contract for id/preset/auto,
from-video slug parity, ghost row (no auto-clear), recents guard (sentinels
excluded, real ids kept), preview button presence/value/loading. Full frontend
vitest green (362); typecheck:ci clean; CJK guard green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 11:28:15 +05:30
Palash DebnathandClaude Opus 4.8 83dcad5878 feat(store): unified LongformProject store + v4→v5 migration (#31a) (#443)
Introduces one project concept both long-form editors bind to: Stories
(cast+tracks) and Audiobook (raw script + book metadata), discriminated by a
`projectMode`. Store-only, no UI behavior change — Audiobook is not yet bound
(its inputs still use local state; that's the #31b follow-up). Ships the data
model + migration + the `convertMode` seam #24 will consume.

- `storiesSlice.ts` → `longformSlice.ts`: `StoryProject` → `LongformProject`
  (gains mode/script/meta/lexicon/coverRef/outputFormat/loudness/defaultVoice);
  new working fields + actions (setScript, setProjectMeta [merge], setLexicon
  [replace], setOutputPrefs [merge], setCoverRef, convertMode). `loadProject`
  restores the FULL surface default-filled (old records never surface undefined
  to a controlled input); `newProject(mode?)` clears it. `SLICE_DEFAULTS` +
  `genProjectId` exported (the migrate fn imports genProjectId). Deprecated
  aliases (`StoryProject`/`StoriesSlice`/`createStoriesSlice`) re-exported so the
  rename breaks no import.
- **Field names kept** (`storyProjects`/`storyTracks`/`cast`) so all 6 consumers
  and every existing localStorage blob keep working with zero change — the
  persisted KEY is unchanged; only the per-project SHAPE is enriched.
- The project-mode working field is named **`projectMode`**, NOT `mode` — `mode`
  is already the app navigation field (uiSlice/AppMode); the spec's `mode` would
  collide (TS error + duplicate partialize key). The stored
  `LongformProject.mode` (nested) keeps its name.
- persist `version: 4 → 5` + a `version < 5` migrate branch (the localStorage
  analog of an alembic upgrade): enriches each saved project with defaults
  (spread `...sp` last so id/name/cast/tracks/updatedAt win), drops malformed
  entries, never throws. v4 users see the same projects, same names/cast/tracks.

Tests: ported the back-compat suite (Stories unchanged) + new coverage —
default-fill on a v4-shaped record, no-stale-carryover, merge-vs-replace
semantics, convertMode idempotency/guard, snapshot+restore of the new fields.
Full frontend vitest green (357); typecheck:ci clean; CJK guard green (new
slice scanned). No app version-file change (the persist version is the
localStorage schema, not the release).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 11:27:13 +05:30
Palash DebnathandClaude Opus 4.8 2a1c3eee3d feat(routing): synth-time no-silent-fallback gating at all TTS entry points (#21 follow-up) (#440)
Closes the last #21 gap: a per-request engine=/model= override bypasses the
/engines/select host-gate, so an engine that can't use this host's GPU could
still be triggered at synth time and silently fall back to CPU (or die mid-
synth). Now enforced at every TTS synth entry point, reusing the SAME probe +
resolver — never re-deriving routing.

Shared helpers (services/engine_routing.py):
- `routing_notice(result)` → (status, reason) to surface, or None. Fires for
  cpu_fallback (always) and accelerated-with-caveat (driver/arch); silent for
  cpu_only / clean-accelerated / n/a.
- `header_safe_reason(reason)` → scrubbed + ASCII-sanitized (headers are
  latin-1; a non-ASCII device name would 500 otherwise) + ≤256 chars. No regex.

Entry points:
- REST `POST /generate` (generation.py): after engine resolution, resolve
  routing once; `unavailable` → 400; cpu_fallback / accelerated-caveat → 200 +
  `X-OmniVoice-Routing` + `X-OmniVoice-Routing-Reason` headers on the WAV
  StreamingResponse; benign → no headers. Covers OmniVoice + adapter branches.
- OpenAI-compat `POST /v1/audio/speech` (openai_compat.py): same gate + same
  headers; the tts-1/tts-1-hd alias inherits the active engine's routing.
- WebSocket `/ws/tts` (tts_stream.py): no headers → frames. `unavailable` →
  `{"type":"error",...}` + skip stream; cpu_fallback / caveat → one
  `{"type":"routing","status","reason"}` frame before any audio.
- `select_engine` response now echoes routing_status / effective_device /
  routing_reason (PR #432 added the gate; this adds the fields so the UI can
  warn on a cpu_fallback pick). New fields on SelectEngineResponse.

Frontend: `useTTS` reads the X-OmniVoice-Routing header and shows a one-time,
non-blocking toast (in-memory de-dup by status — a 50-clip batch fires once,
no localStorage). i18n keys `tts.routingFallback`/`tts.routingCaveat`.

Tests: routing_notice + header_safe_reason (ASCII/length/scrub) unit tests;
REST synth gate (unavailable→400, cpu_fallback→headers, cpu_only→none) via the
fake-engine harness with a mocked host; select response routing fields.

Deferred (small follow-up): dub-pipeline ASR routing note on the preflight_error
SSE channel — separate path, not a TTS synth entry point. No frontend /ws/tts
client exists today (the routing frame serves external API consumers).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 11:18:37 +05:30
Palash DebnathandClaude Opus 4.8 63a4b897ee test(longform): real-ffmpeg + stub-TTS e2e for the chapterized renderer (#34) (#441)
#34 runtime-verify Layer 1 — the cheap regression net over the audiobook /
stories convergence. Drives the REAL `_render_longform_sse` generator + REAL
ffmpeg with a stub CPU-tone synth (no GPU/model), and ffprobes the muxed output.

Covers happy m4b (full SSE sequence + 2 tagged chapters), mp3 container,
per-chapter partial failure (chapter_error isolates ch.0, surviving chapter
still muxes), total failure (error + NO file), empty plan, and the no-ffmpeg
branch. Gated on ffmpeg present (skip otherwise; runs in CI).

Like the other endpoint tests it imports the app+torch stack, so it's validated
on CI (local pytest segfaults on the pre-existing torch/Triton import).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 11:07:04 +05:30
Palash DebnathandClaude Opus 4.8 747507ff61 feat(ui): Engine Compatibility Matrix routing display (#21 PR 5/5) (#434)
Surfaces the /engines routing data (PR 3) in the matrix so users see the
device each engine will actually use on THIS machine.

- The chip matching `effective_device` is highlighted (accent ring + bold),
  with a "Runs on X on this machine" tooltip.
- A status-toned routing badge: accelerated→success "GPU active",
  cpu_fallback→warn "CPU fallback" (reason in tooltip), cpu_only→neutral
  "CPU". The badge is SUPPRESSED for unavailable rows (the availability badge
  already says so) and for legacy payloads with no routing_status (renders
  exactly as before). An unknown/future status falls back to a neutral
  "Unknown" badge.
- LLM rows (routing 'n/a') render a single neutral "Remote" badge instead of
  device chips — no false GPU claim.
- types.ts: EngineBackend gains effective_device / routing_status /
  routing_reason; GPUTarget gains `xpu`; new EffectiveDevice + RoutingStatus
  unions. Corrected the stale "only TTS migrated" comment (all 3 families now
  emit the full shape).
- i18n keys in en.json (other locales fall back to en via fallbackLng until
  translated — no key-parity gate). xpu chip color in the matrix CSS.

Tests: 5 new RTL cases (accelerated highlight+badge, cpu_fallback badge,
unavailable suppression, legacy no-badge, LLM Remote). Full frontend vitest
green (350); typecheck:ci clean.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 10:25:56 +05:30
Palash DebnathandClaude Opus 4.8 c6a55794da feat(routing): active-engine GPU verdict in preflight + diagnose (#21 PR 4/5) (#433)
Surfaces a routing verdict for the CURRENTLY-SELECTED TTS engine in the two
system-health surfaces, so a CPU fallback / unavailable-GPU is heard about
before a slow or failed synth — the no-silent-fallback contract, read-only.

- `tts_backend.active_routing()` + `gpu_routing_verdict()`: the active engine's
  routing derived from list_backends() (byte-identical to the matrix) plus the
  host compute summary (family + VRAM from the canonical probe). Never raise.
- `/system/diagnose` gains a `gpu_routing` check: accelerated→ok,
  accelerated-with-caveat / cpu_fallback→warn (+ actionable hint), cpu_only→ok
  (no-GPU host is the expected normal state — never noise-warns), unavailable→
  fail, no-engine→warn. ASCII-safe detail strings (the text dump enforces ASCII).
- `/setup/preflight` gains an "Active engine routing" check + an explicit
  `gpu_routing` object on PreflightResponse (a real field — the response has no
  extra="allow", so it would otherwise be dropped). `device` gains `gpu_family`
  (ROCm-vs-CUDA aware) + `vram_gb`. New `GpuRouting` schema.

Tests: gpu_routing_verdict (host + active-engine + degraded), diagnose status
mapping across all 6 states + never-raises, preflight gpu_routing object +
check + device.gpu_family. Existing diagnose/preflight tests stay green (checks
are additive; the report's top-level key set is unchanged).

Deferred (documented): synth-time routing headers/WS-frames at the 3 synth
entry points. Selection is already hard-gated (PR 3 select_engine), and the
matrix (PR 5) + this preflight/diagnose verdict surface the situation — the
synth-time signal is incremental belt-and-suspenders for the env-var-pinned
edge and is best validated interactively. Tracked as a #21 follow-up.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 02:07:54 +05:30
Palash DebnathandClaude Opus 4.8 8c8d525397 feat(routing): wire effective-device into /engines + select gate (#21 PR 3/5) (#432)
* feat(routing): wire effective-device + routing_status into /engines (#21 PR 3/5)

Surfaces the PR-1 probe + resolver through the engine registries so the
matrix UI (PR 5) and the no-silent-fallback gates can consume it.

- `engine_routing.routing_fields()`: shared helper returning the three
  serialization-ready keys, centralizing the scrub rule — routing_reason is
  scrubbed via `core.scrub.scrub_text` only when truthy, so a None reason
  stays JSON `null` (never coerced to "").
- TTS/ASR `list_backends()` each gain `effective_device` / `routing_status` /
  `routing_reason`, computed from a SINGLE `detect_host_caps()` call per
  request (host caps are constant per process). ASR is brought to full TTS
  parity: it now also carries `install_hint` / `last_error` / `isolation_mode`
  and a SCRUBBED `reason` (closing a pre-existing ASR token-leak gap) — an
  identical 11-key shape across families. ASR also gains the same
  is_available()-raises resilience TTS has (degrade to available:false, never
  500).
- LLM `list_backends()` reaches 11-key parity too but emits literal
  `effective_device:"network"` / `routing_status:"n/a"` / `routing_reason:null`
  (NOT via resolve_routing — LLM runs no local GPU model). `LLMBackend.gpu_compat
  = ()`. "network" is a label, not a probe — nothing here touches the network.
- `select_engine` host-routing gate: refuses a pick whose `routing_status` is
  `unavailable` on this host (400 with an actionable detail), while ALLOWING
  `cpu_fallback` (it runs, just slower). LLM is never gated. Defensive `.get`
  so legacy payloads still select. New typed `SelectEngineResponse`.

Tests: 11-key shape across all 3 families, well-formed tts/asr routing keys
(+ None-not-"" contract), LLM network/n/a labels, select gate (block
unavailable / allow cpu_fallback / never-gate LLM). Updated the registry
exact-shape test for the 3 new keys.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(cjk): allowlist docs/specs/ in the hardcoded-CJK guard

PR #429 merged the longform design specs, which legitimately quote functional
CJK (test-fixture descriptions, CosyVoice speaker IDs, multilingual sample
text). The CJK guard scans every tracked file, so those docs turned main red.
Specs are documentation, not shipped UI strings — allowlist the docs/specs/
prefix, matching the individually-allowlisted docs already in the set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 01:59:33 +05:30
Palash DebnathandClaude Opus 4.8 e0b59f3984 docs(longform): implementation specs for the 14 roadmap tasks (#21–#34) (#429)
* docs(longform): implementation specs for the 14 roadmap/integration tasks (#21–#34)

Per-task implementation specs under docs/specs/longform/ for the remaining
longform + #346-roadmap work: GPU compat matrix, shared VoiceSelector,
Transcriptions import, Story⇄Audiobook export, inline Create Voice, gallery
handoff, parser unification, two-pass ACX, .ovsvoice format, Dub→Stories,
unified LongformProject store, phone calls, cue-sheet, runtime-verify.

Authored by a draft + iterative-refinement workflow (codebase-grounded: exact
file:line anchors, API/data shapes, test plans, constraints, deps, risk, PR
slices). NOTE: the 10-round refinement was cut to ~rounds 4–5 by an account
session limit; rounds 5–10 (incl. the final de-bloat/polish pass) are pending —
the specs carry per-round revision-note preambles that the polish round trims.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(longform): strip accreted (this-revision) note preambles from specs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 01:43:24 +05:30
Palash DebnathandClaude Opus 4.8 c3b2346759 fix(engines): MLX platform gate (#390) + ASR gpu_compat + IndexTTS2 (#21 PR 2/5) (#431)
Builds on the device probe from PR 1. Backend-only; the routing keys are
wired into /engines in PR 3.

- #390 closed: MLXAudioBackend / MLXWhisperBackend now call the shared
  `core.device_caps.mlx_supported()` gate FIRST, before importing the
  package. On Linux/Windows/mac-Intel they report unavailable and never
  advertise a usable `mps` route, even with a stray mlx wheel installed.
  Replaces the ASR backend's ad-hoc inline MPS check with the one shared
  rule. (The Wave-4.4 OSError/RuntimeError import-guard is preserved — it
  now lives behind the platform gate; its test forces the gate open so the
  guard stays the path under test.)
- `ASRBackend` ABC gains `gpu_compat: tuple[str, ...] = ("cpu",)` mirroring
  TTSBackend, and each subclass declares its real targets:
  whisperx/faster-whisper → (cuda,cpu); mlx-whisper → (mps,cpu);
  pytorch-whisper → (cuda,mps,cpu); nemo/funasr → (cuda,cpu);
  moonshine → (cpu,). Inert until PR 3 serializes them.
- IndexTTS2 declares `gpu_compat = ("cuda","cpu")` so it stops advertising
  the inherited CPU-only default.
- ROCm is deliberately NOT claimed for any ASR engine (or for IndexTTS2):
  CTranslate2 has no upstream HIP build, and an unverified `rocm` claim
  would route ROCm hosts to a broken GPU path — strictly worse than the
  honest `cpu_fallback` the resolver already emits ("declares CUDA only;
  ROCm not in its compat set"). The per-engine TTS ROCm audit is a tracked
  follow-up that will verify each path before claiming it.

Tests: MLX gate regression (both backends, on/off Apple), ASR gpu_compat
tuples + no-false-rocm invariant, IndexTTS2 override; existing MLX
import-guard test updated for the new gate ordering.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 01:43:21 +05:30
b62d1f5073 refactor(longform): share the SSE stream consumer across Stories + Audiobook (#436)
Stories and Audiobook are two authoring frontends over one server-side
renderer (_render_longform_sse), emitting the same chapter-progress events.
Both hand-rolled the identical read/decode/splitSSEBuffer/parseSSELine loop.

Extract utils/longformStream.consumeLongformStream(res, onEvent, {isAborted}):
one place owns the SSE protocol; each editor keeps only its own per-event state
handling (Stories: export %; Audiobook: {current,total,title,assembling,done}).
Behaviour unchanged — Audiobook keeps its abort check via isAborted.

The rest of the two editors stay distinct on purpose (cast/dialogue vs
manuscript/EPUB authoring), per docs/specs/2026-06-13-stories-audiobook-maturity.

Tests: frontend/src/test/longformStream.test.js (chunk-boundary parsing, abort,
no-body). Full vitest: 348 passed; typecheck:ci clean.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 23:59:00 +05:30
000010ebb8 feat(dub): dedicated Dub home (projects/history) + project rename (#435)
The dub Projects + History rail (WorkspaceProjects/WorkspaceHistory) used to
sit beside the editor at all times. Now it's a landing: shown only when no
project is being edited (dubStep === 'idle'); opening/creating one switches to
a full-width editor. (The global Sidebar is already hidden in dub mode, so the
studio-right rail is the only surface — no Sidebar change needed.)

Adds project rename:
- backend: PATCH /projects/{id} updates just the name (400 on empty, 404 on
  missing) — lighter than PUT which rewrites the whole state blob.
- api: renameProject(id, name); App.jsx renameProject handler (updates the
  active-project label + refreshes the list).
- UI: inline rename on each project card (pencil → edit → Enter/Save / Esc).

Verified: PATCH create→rename→list / 400 / 404; frontend typecheck:ci clean.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 23:46:22 +05:30
Palash DebnathandClaude Opus 4.8 e61665fe34 feat(routing): host device probe + routing resolver (#21 PR 1/5) (#430)
* feat(routing): canonical host device probe + routing resolver (#21 PR 1/5)

Foundational, backend-only slice of the GPU compatibility matrix (#21).
No API or UI change — wiring lands in PRs 3–5.

- `core/device_caps.py`: single source of truth for host accelerator
  capability. `detect_host_caps()` distinguishes ROCm from CUDA (unlike
  the gguf hardware_probe), never raises, makes no network call, stays
  kernel-free on cold start, and caches per process. Enumerates the full
  degradation contract (torch-unimportable→probe_ok=False, CUDA-init
  raises, device_count==0, multi-GPU, mem_get_info failure, arch
  mismatch, MPS, XPU, DirectML). Plus shared `mlx_supported()` gate
  (#390 groundwork) — exact-string platform check, no regex.
- `services/engine_routing.py`: pure `resolve_routing(gpu_compat, caps)`
  → `{effective_device, routing_status, routing_reason}`; deterministic
  and byte-identical across OSes. Rules for accelerated / cpu_fallback
  (the no-silent-fallback signal) / cpu_only / unavailable, incl. the
  ROCm-not-in-set, DirectML-neutral, and XPU edges.
- `get_best_device()` delegates its family decision to the probe so the
  loader and probe can never disagree; keeps the ROCm HSA env override
  and DirectML device-string return (probe reads, loader writes). String
  contract unchanged.
- 39 unit tests (probe / resolver / mlx gate / reason-scrub contract);
  no new regex (CodeQL-clean), English-only (CJK guard green).

The gguf hardware_probe rebase is a deliberate follow-up: it has its own
torch-mocked suite and a VRAM-driven quant table unaffected by the family
rename, so it stays out of this zero-risk slice.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routing): address review — full available_families + empty-except comments

CodeRabbit / CodeQL review on PR 1:
- `available_families` no longer drops secondary accelerators on hybrid hosts
  (e.g. NVIDIA + Intel-iGPU-via-IPEX). The probe now detects every accelerator
  independently and picks `family` by priority at the end, instead of
  short-circuiting after the first hit. Routing is unaffected (it keys off
  `family`), but the field is now honest. + hybrid-host test.
- Annotated every `except: pass` in device_caps with an explanatory comment
  (CodeQL py/empty-except).
- Removed the unused `_MIN_NVIDIA_DRIVER` constant — the driver-version check
  stays in wizard preflight (no subprocess on the probe path); documented why.
- `get_best_device()` now checks MPS before DirectML, mirroring the probe's
  family-priority order so loader and probe never disagree.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 23:03:22 +05:30
dc1d36fe5f refactor(models): model-management v2 cleanup (mm2, all tiers) (#428)
One coherent lifecycle surface over the in-process model, diarization, and
subprocess sidecars; fixes the engine-switch VRAM leak; tightens download
robustness. Backend-only, response shapes preserved, no new deps.

Tier 1 — correctness:
- MM2-01: get_active_tts_backend() caches one instance per backend id and
  unload()s the outgoing engine on switch (fixes the VRAM leak behind #278);
  adds reset_active_backend().
- MM2-02: OmniVoiceBackend.unload() releases the shared model_manager singleton
  + free_vram(); SubprocessBackend.unload() -> unload_sidecar(self.id),
  inherited by all sidecar engines. Idempotent + preload-safe.
- MM2-03: /model/loaded ASR row reports the real device + a note explaining the
  disabled unload button.

Tier 2 — single surface:
- MM2-04: new services/model_lifecycle.py owns list_loaded/unload/unload_all/
  free_vram; system.py routers are thin delegations (shapes unchanged).
- MM2-05: idle timeouts (in-process + sidecar) resolve via prefs.resolve
  (env wins, no restart); removed the duplicated _IDLE_TIMEOUT_SECONDS.

Tier 3 — robustness/observability:
- MM2-06: _install_cooldowns swept (1h TTL) + cleared on success — bounded.
- MM2-07: per-extension weight floors (onnx 64KB, tensors 5MB) OR the original
  >=5MB catch — small ONNX no longer false-flagged, #352 still caught.
- MM2-08: indextts GPU sidecar self-reports vram_mb in pong; parent surfaces it
  in list_live_sidecars (0 = CPU/unmeasured).
- MM2-09: is_cached scan_cache_dir->disk fallback logs WARNING w/ exc type
  (#117/#118), was invisible at DEBUG.

Tests: tests/test_mm2_lifecycle.py (15). Full suite: 1379 passed.
Plan/summary: .planning/quick/260613-mm2-clean-model-management-v2/.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 20:45:32 +05:30
4cc55ab852 Fast model downloads: Xet fast path + accurate progress (FDL W0–W2 + W4) (#424)
* feat(downloads): Xet fast path + accurate progress (FDL W0–W2)

Make model downloads fast and show accurate downloaded/remaining/speed.
Research confirmed hf-xet already implements the IDM/uGet technique
(content-defined chunking, parallel byte-range gets, dedup, resume), and
the spike found all 25 catalog repos are Xet-backed — so the win is
driving Xet well + accurate progress, not a custom downloader.

W1 — maximize + guarantee Xet:
- pin huggingface_hub>=1.7 + hf-xet>=1.1 (was transitive); no hf_transfer
- drive snapshot_download with explicit tqdm_class + max_workers + endpoint
- opt-in HF_XET_HIGH_PERFORMANCE / HDD sequential-write knobs (default off)
- /system/info reports fast_download {xet_enabled, xet_version, high_perf}

W2 — accurate progress:
- dry_run preflight -> install_plan event (exact total/cached/remaining)
- utils/download_aggregator.py: one overall bar; byte bars (by id) vs the
  "Fetching N files" count bar; windowed rate; emits one 'aggregate' event
- frontend overall bar (speed/remaining/ETA), cached-skip,  fast badge

Known limit (verified live): under Xet+hf_hub 1.7.2 per-file byte bars
never advance/close via tqdm, so mid-download the bar is file-granular and
bytes flush to the exact total on completion. Classic-LFS/mirror repos get
true byte progress (W4).

Drive-by: download.py used os.walk without importing os (latent NameError
in _validate_snapshot_has_weights on every install) — fixed.

Tests: tests/backend/setup/test_download_preflight.py (10). Spike + plan
under .planning/quick/260613-fdl-fast-model-downloads/.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(downloads): opt-in mirror + cancel + docs (FDL W4)

- mirror (FDL-10): snapshot_download(endpoint=) honours prefs hf_endpoint /
  env HF_ENDPOINT on preflight + download (per-call, no process-wide env).
  Documented as the classic-LFS path (no Xet) for restricted networks.
- cancel (FDL-11): POST /models/install/cancel {repo_id} stops further
  retries at the next boundary, emits install_cancelled, clears the cooldown
  (cancel is intent, not failure). Frontend treats it as a terminator.
- docs (FDL-12): docs/downloading-models.md (Xet fast path, progress
  semantics + byte-speed limitation, opt-in tuning, mirror, cancel,
  troubleshooting) + README pointer. Docs-sync rule satisfied.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(planning): model-management v2 cleanup plan (mm2)

GSD plan for cleaning the model-management subsystem: registry unload-on-
switch + per-engine unload() (fixes VRAM leak), model_lifecycle facade,
unified idle/timeout config, bounded cooldowns, sidecar VRAM self-report,
cache-fallback logging. Planning artifact only — no code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(downloads): reconcile with main's HF_HUB_DISABLE_XET; honest status

Rebasing onto main surfaced that main forces HF_HUB_DISABLE_XET=1 (classic
LFS) because Xet progress bypasses the tqdm hook — the same limitation found
here. Reconcile instead of fight:

- /system/info fast_download now reports runtime truth: xet_installed +
  xet_active (installed AND not HF_HUB_DISABLE_XET) + xet_enabled alias. The
   badge only shows when Xet actually runs; startup log says
  "downloads: Xet disabled → legacy LFS".
- complete(): clear the rate window before the final flush so crediting the
  full size in one step can't emit an absurd instantaneous rate.
- docs/downloading-models.md rewritten: default is legacy LFS for accurate
  progress; Xet is opt-in via HF_HUB_DISABLE_XET=0. hf-xet pin stays (ready
  for a future Xet progress hook).

W2 (preflight total/remaining + aggregate bar + exact completion) is the
value on either path; W1's "maximize Xet" is dormant by main's design.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(downloads): opt-in segmented multi-connection accelerator (FDL W3)

Since main forces Xet off (HF_HUB_DISABLE_XET=1), the default path is
single-stream legacy LFS — so a segmented downloader is the way to get BOTH
parallel speed and live byte progress.

- services/segmented_download.py: async multi-connection Range downloader for
  one file — parallel byte-ranges, resume (.part + manifest), per-segment
  short-read truncation guard, optional sha256/etag verify, cancel, and a
  single-stream fallback when the server won't range. Auth-safe: the HF
  Authorization header is sent only to huggingface.co/hf.co and never
  forwarded to a CDN host on redirect (unit-tested).
- dispatch (download.py): opt-in via prefs segmented_downloader / env
  OMNIVOICE_SEGMENTED_DOWNLOAD (default off). When on and Xet inactive,
  fetches each file into the HF cache mirroring hf_hub_download (blobs +
  snapshot symlinks + refs/main), feeding real bytes to the aggregator. Any
  failure falls back to snapshot_download — never breaks a correct install.
- fix: complete() was adding a full total on top of accumulated segmented
  bytes (2x); now replaces byte bars so the sum is exactly total.

Verified live (accelerator on): real byte progress to ~16.6 MB/s, final
bytes==total, /models installed=True, delete frees correctly.

Tests: test_segmented_download.py (7) + aggregator double-count regression.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(downloads): relocate FDL tests to top-level; loop-isolate segmented test

CI runs the full suite, which exposed a pre-existing test-isolation leak:
several tests/backend/** fixtures purge core.*/services.* from sys.modules
under a temp OMNIVOICE_DATA_DIR and never restore, leaving core.config/core.db
bound to a dead temp dir. It only bites when collection order puts a purging
test ahead of a real-DB reader (test_longform_jobs). Adding tests under
tests/backend/setup/ reordered collection and tripped it.

Fix without touching the shared (fragile) fixtures or risking class-identity
breakage from a blanket sys.modules restore:
- move the two FDL test files to top-level tests/ (tests/test_fdl_*.py) so
  tests/backend/** collection order is identical to main — longform passes.
- rewrite the segmented test to run each case under asyncio.run() (fresh loop)
  instead of asyncio.get_event_loop(), which an earlier async test can leave
  closed in the full suite.

Full suite green locally: 1364 passed, 0 failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 20:15:15 +05:30
Palash DebnathandClaude Opus 4.8 df94af888f feat(longform): cohesion quick-wins — Audiobook launchpad card + Stories in Projects (#426)
* fix(ui): align Audiobook + Stories controls to the design tokens

The hand-written tabs used a bare `.btn` class (which has NO CSS rule → bright
white browser-default buttons) and an unstyled `.field-label`, so the buttons,
labels, and selects looked off-theme. (Other tabs use the Button/ui-btn system,
which is why only these looked wrong.) Found via a design-token audit workflow.

AudiobookTab:
- Import / Preview plan / Add word / Add cover / Download → `ui-btn ui-btn--subtle`;
  Create → `ui-btn ui-btn--primary`; cover-remove / lexicon-remove / chapter-play
  → `ui-btn ui-btn--icon` (the app's themed button variants from ui/Button.css).
- AudiobookTab.css: define `.audiobook-tab .field-label` (chrome mono/uppercase
  via --chrome-* tokens) + header serif title / muted subtitle (--font-serif,
  --text-xl, --color-fg/-muted). Selects/inputs already used `.input-base` (the
  canonical chrome look) — left as-is.

StoriesEditor:
- Format `<select>` now uses `.input-base` (canonical chrome select + arrow);
  trimmed the bespoke `.stories-editor__format` rule to just the toolbar sizing.

Build clean; 345 frontend tests green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(longform): cohesion quick-wins — Audiobook launchpad card + Stories in Projects

Make Stories/Audiobook feel wired into the app (integration-map plan, quick-win tier):
- Launchpad: an Audiobook ActionCard (was NavRail-only; Stories already had one).
- Projects/OmniDrive: saved Stories projects now appear as a "Stories" category
  (line + voice counts) and open via onOpenStory → loadProject + setMode('stories'),
  mirroring onOpenDub. App.jsx reads storyProjects/loadProject from storiesSlice.
- Live profile sync (QW1) confirmed already working: both tabs map the `profiles`
  prop in render (no mount snapshot), so a voice cloned/designed/imported anywhere
  shows up live in the cast/default pickers — no code needed.

Deferred (no trigger yet): QW4 create-voice handoff to these tabs needs an inline
create/gallery "use here" affordance first (QW3/M3).

Build clean; 345 frontend tests green; en.json valid.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 17:45:34 +05:30
Palash DebnathandClaude Opus 4.8 8bd2f149a0 fix(ui): align Audiobook + Stories controls to the design tokens (#425)
The hand-written tabs used a bare `.btn` class (which has NO CSS rule → bright
white browser-default buttons) and an unstyled `.field-label`, so the buttons,
labels, and selects looked off-theme. (Other tabs use the Button/ui-btn system,
which is why only these looked wrong.) Found via a design-token audit workflow.

AudiobookTab:
- Import / Preview plan / Add word / Add cover / Download → `ui-btn ui-btn--subtle`;
  Create → `ui-btn ui-btn--primary`; cover-remove / lexicon-remove / chapter-play
  → `ui-btn ui-btn--icon` (the app's themed button variants from ui/Button.css).
- AudiobookTab.css: define `.audiobook-tab .field-label` (chrome mono/uppercase
  via --chrome-* tokens) + header serif title / muted subtitle (--font-serif,
  --text-xl, --color-fg/-muted). Selects/inputs already used `.input-base` (the
  canonical chrome look) — left as-is.

StoriesEditor:
- Format `<select>` now uses `.input-base` (canonical chrome select + arrow);
  trimmed the bespoke `.stories-editor__format` rule to just the toolbar sizing.

Build clean; 345 frontend tests green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 17:36:29 +05:30
Palash DebnathandClaude Opus 4.8 e196d790cc fix(longform): evict oldest chapters from the render cache (review fast-follow) (#423)
The content-addressed longform_cache/ accumulated uncompressed chapter WAVs
across every render with no bound (a review finding). Add prune_cache_dir() —
LRU-by-mtime eviction down to a 2 GB ceiling (OMNIVOICE_LONGFORM_CACHE_MAX_GB);
best-effort, never raises. Called at the start of each render job, before its
chapters are written, so the fresh ones are never the eviction target.

Tests: under-cap no-op, evicts-oldest-keeps-newest, missing-dir safe. 38 green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 17:12:08 +05:30
Palash DebnathandClaude Opus 4.8 0c761ee991 feat(audiobook): pronunciation editor + markup reference UI (#422)
Makes the lexicon backend (#419) and SSML-lite markup (#421) usable from the tab.

- A "Pronunciation" editor in the full-width side pane: add/remove {word → say
  it as…} rows, compiled to a lexicon dict sent with both the full render and
  per-chapter preview (so previews match the final output).
- A collapsible "Markup reference" listing the script syntax (# chapter,
  [voice:], [pause], [slow]/[fast]/[emphasis]/[spell]).
- api/audiobook.ts: lexicon field on the generate + preview bodies.

Build clean; 345 frontend tests green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 17:07:44 +05:30
Palash DebnathandClaude Opus 4.8 62b2a6fab9 feat(longform): SSML-lite prosody markup — [slow]/[fast]/[emphasis]/[spell] (PR 8b) (#421)
Inline delivery hints within a narration line, wired into BOTH front doors so
Audiobook and Stories behave identically.

- services/ssml_lite.py (parallel-built, 18 tests): parse_ssml_lite splits a
  line into {text, speed, spell, emphasis} segments — nesting (innermost wins),
  unclosed-to-EOL, stray-close ignored, adjacent-merge; ReDoS-safe literal
  alternation. + spell_out().
- _parse_spans (audiobook script path) now applies SSML-lite as the innermost
  layer (precedence: [voice:] → [pause] → SSML); each segment becomes a Span
  with its speed (threaded to the renderer) and spelled-out text for [spell].
  Trailing pause attaches to the run's last segment.
- frontend/src/utils/ssmlLite.js: client port (kept in sync with the .py) +
  storyToSpans applies it per chunk — inline speed OVERRIDES the per-line slider,
  falls back to it otherwise.

Tests: parse_ssml_lite (18 py + 10 js), script-level prosody parse, Stories
SSML compile (override + spell). 70 backend + 345 frontend green; build clean.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 16:56:55 +05:30
Palash DebnathandClaude Opus 4.8 8555c510b8 fix(ui): full-width/height Audiobook + Stories layouts (match other tabs) (#420)
Both tabs rendered as narrow centered columns (Audiobook maxWidth:860, Stories
max-width:1040 margin-auto) while the rest of the studio is full-bleed.

- AudiobookTab: rebuilt into a full-height two-pane layout (new AudiobookTab.css)
  — header with the action buttons, a left script editor that grows to fill the
  window height, and a right settings+results pane (voice/format/loudness, cover
  & metadata, progress/output/plan) that scrolls independently. Collapses to one
  column under 900px. Removed the inline 860px cap.
- StoriesEditor: dropped the `max-width:1040px; margin-inline:auto` cap → fills
  edge-to-edge like the dub/projects/transcripts tabs.

Build clean; 334 frontend tests green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 16:52:15 +05:30
Palash DebnathandClaude Opus 4.8 dde43de5a4 feat(longform): pronunciation lexicon — per-render word respelling (PR 8a) (#419)
Lets a render correct hard-to-say words (e.g. {"GIF":"jiff","Dr":"Doctor"}).
Backend wiring; the editor UI folds into the full-width Audiobook redesign.

- services/pronunciation.py (parallel-built, 19 tests): apply_lexicon —
  whole-word, case-insensitive, longest-first, word-boundary, single ReDoS-safe
  re.sub pass; + normalize/load/save_lexicon (JSON).
- synthesize_chapter gains a `lexicon` kwarg, applied to each span's text before
  chunk splitting (None/empty = no-op → backward compatible).
- _render_chapter_cached folds the normalized lexicon into the chapter cache key
  (a lexicon edit re-renders); threaded through _render_longform_sse + the
  /audiobook, /audiobook/preview, /longform/render request models.

Tests: synthesize_chapter respells via lexicon; pronunciation module (19);
75 related backend tests green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 16:43:48 +05:30
Palash DebnathandClaude Opus 4.8 18e4c2347a fix(longform): correctness + robustness fixes from adversarial review (#418)
* fix(longform): correctness + robustness fixes from adversarial review

Fixes the confirmed findings from a multi-agent review of the convergence:

HIGH (correctness/output):
- MP3 + cover produced a corrupt file (-map 2:v -c:v copy is invalid for mp3).
  Cover art is now embedded for M4B only; mp3 skips it (m4b is the cover format).
- Chapter cache key omitted ref_text — editing only a profile's ref_text served
  stale audio. ref_text is now part of the voice signature.
- Preview wrote audiobook_cache/ but the render reads longform_cache/ (rename
  missed in PR 5) → cache-warming silently broke. Unified to longform_cache/.

Robustness (DoS/OOM guards):
- /audiobook/import caps upload at 64 MB; epub_to_chapter_script bounds per-entry
  (25 MB) and cumulative (300 MB) uncompressed reads (zip-bomb guard).
- /longform/render rejects > 10,000 chapters (422).

Frontend leaks:
- StoriesEditor.removeTrack revokes the line's preview blob URL.
- AudiobookTab revokes the cover blob URL on replace/unmount.

Deferred fast-follows (also from review): render-cache disk eviction; restoring
the standalone chapter cue-sheet export (needs chapter times in the done event).

Tests: mp3-drops-cover, epub entry/total caps, import + chapter-count limits;
updated the cache-hit test for the 4-field voice sig. 70 backend + 334 frontend green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(longform): pass EPUB caps as params, not monkeypatch (CI import-path fix)

The cap tests monkeypatched module constants, but in the full-suite CI context
the module loads under a different import path so the patch missed the function
(it used the real 300 MB cap → tests failed). epub_to_chapter_script now takes
max_entry_bytes/max_total_bytes kwargs (default to the constants); tests pass
small values directly — deterministic regardless of import path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 16:31:13 +05:30
Palash DebnathandClaude Opus 4.8 36e7fb12fc feat(longform): job library — finished books/stories in Projects (PR 7/8) (#417)
Surfaces finished Audiobook + Story renders so they're re-downloadable from the
Projects view — closing the resume/history loop of the convergence.

Backend (new, no migration — reads existing job_store rows):
- routers/longform_jobs.py: GET /longform/jobs lists finished audiobook/story
  jobs newest-first, recovering output/chapters/duration from each job's
  persisted 'done' SSE event. Pure build_longform_library() over the job_store
  callables; defensive (skips unparseable jobs, never 500s). Registered in main.py.

Frontend:
- Projects.jsx: new "Audiobooks" category fed by /longform/jobs; each row opens
  the rendered file (/audio/<output>) with type/chapters/duration. Offline-safe
  (empty on fetch failure). en.json keys added.

Built via parallel worktree agent; backend tests/test_longform_jobs.py (9) green;
334 frontend tests + build clean.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 16:17:28 +05:30
Palash DebnathandClaude Opus 4.8 00f400e4c7 fix(stories): thread per-line speed through the shared renderer (PR 6/8) (#416)
PR 5 moved Stories' full export to /longform/render but dropped per-line
**speed** — the old client export sent each line's speed to /generate; the
converged path silently ignored it. This restores it end-to-end.

- Span gains an optional `speed`; synthesize_chapter passes it to the injected
  synth (signature now `synth(text, voice_id, speed)`); both engine paths
  (OmniVoice model + generic TTSBackend) forward it to generate(speed=…).
- chapter_cache_key now includes speed (a speed change re-renders; tuples accept
  an optional 4th element so existing 3-tuple callers/tests still work).
- LongformSpan + /longform/render carry speed; storyToSpans emits each line's
  speed onto its spans.

Emotion note: per-line tone is already model-native via inline tags
([laughter] etc.) inserted into the text, so no separate emotion→instruct
plumbing is needed — the dead `emotion` store field stays unused/superseded.

Tests: storyToSpans speed passthrough (8); cache-key speed sensitivity; synth
stubs updated for the 3-arg signature. 65 backend + 334 frontend green; build clean.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 16:02:21 +05:30
Palash DebnathandClaude Opus 4.8 0f67895585 feat(stories): full export → shared server-side renderer (PR 5/8) (#413)
* feat(stories): full export → shared server-side renderer (PR 5/8)

The convergence core. Stories' full export no longer stitches audio in the
browser (Web Audio, capped by RAM, no resume/loudness/markers) — it compiles
cast + lines into a chapter/span plan and streams through the same chapterized
renderer the Audiobook tab uses.

Backend:
- Extracted the audiobook SSE job into a shared `_render_longform_sse(plan, …)`
  generator (resume cache, per-chapter fault isolation, mux). /audiobook is now
  a thin caller.
- New POST /longform/render — accepts a pre-built {chapters:[{title,spans:
  [{voice_id,text,pause_ms_after}]}]} plan (+ format/loudness/cover/metadata) and
  renders it. Pause-only spans (empty text) are kept as silence. job_type=story.
- Shared content-addressed cache renamed longform_cache (one render per unique
  chapter across both front doors).

Frontend:
- storyToSpans(tracks, cast) — pure compiler: `# ` lines → chapters; each line
  resolves its cast/override voice; inline [voice:]/[pause] split into spans;
  pauses fold into the previous span.
- StoriesEditor.generateAll now posts via longformRender and downloads the
  server file (chaptered M4B / MP3). Single-line preview stays client-side;
  stems export unchanged. Format select WAV→M4B.

Deferred to PR 6 (with the component split): per-line regenerate, emotion→instruct.

Tests: storyToSpans (7) — cast resolution, chapters, per-line + inline voice,
pause folding, empty-drop. 64 backend + 333 frontend green; build clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audiobook): confine cover_path to OUTPUTS_DIR + don't leak exception text (CodeQL)

- _safe_cover_path() restricts the user-supplied cover to OUTPUTS_DIR before it
  reaches ffmpeg (py/path-injection).
- SSE error events now emit a generic message and log the detail server-side
  (py/stack-trace-exposure); empty best-effort excepts annotated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audiobook): cover path via basename+fixed dir (clears CodeQL py/path-injection)

CodeQL didn't recognize realpath+startswith as a barrier; os.path.basename is a
recognized sanitizer. Covers only come from /audiobook/cover (OUTPUTS_DIR/
audiobook_covers), so rebuilding from the basename onto that fixed dir is both
CodeQL-clean and strictly tighter — no caller path can escape it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audiobook): regex-allowlist cover filename (clears CodeQL py/path-injection)

basename alone wasn't a barrier CodeQL credits. Restrict the cover name to the
exact pattern /audiobook/cover emits (12 hex + jpg/jpeg/png) before joining onto
the fixed covers dir — an anchored-regex guard CodeQL recognizes as sanitizing,
and strictly tighter than before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audiobook): commonpath-confine resolved cover path (CodeQL py/path-injection)

Add an os.path.realpath + os.path.commonpath containment check on the resolved
cover path (the barrier static analysis recognizes), on top of the regex
allowlist + basename. Defense in depth; the path provably cannot escape
OUTPUTS_DIR/audiobook_covers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 15:47:03 +05:30
Palash DebnathandClaude Opus 4.8 d674084510 fix(ci): make Docker Hub description sync non-fatal (#414)
The image build+push succeeds, but the "Update Docker Hub description" step
403s (Forbidden) — DOCKERHUB_TOKEN can push yet lacks description-edit scope, a
common limitation of fine-grained Docker Hub tokens. That cosmetic overview
sync was failing the whole Docker (GHCR) run on main.

Mark the step continue-on-error so a creds-scope mismatch no longer reds-out an
otherwise-successful build. To actually sync the overview, the token needs
read/write (incl. description) scope, or use the account password.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 14:59:32 +05:30
Palash DebnathandClaude Opus 4.8 ea6833138b feat(audiobook): text + EPUB import → auto-chapter (PR 4/8) (#412)
Spec PR 4. A front door onto the existing chapter parser: import a file, get a
chapter-delimited script in the editor.

Backend (new services/longform_import.py — pure, stdlib only, no new dep):
- chapterize_plaintext(text): inserts `# ` headings ahead of short standalone
  chapter-title lines (Chapter/Part/Prologue/…); no-op if the text already has
  H1s; long "Chapter …" sentences stay prose. ReDoS-safe (anchored, per-line).
- epub_to_chapter_script(bytes): parses EPUB (zipfile + ElementTree +
  html.parser) in spine order → `# Title` + stripped body per document; skips
  empty/nav pages; the heading becomes the chapter title (not narrated). Raises
  ValueError on a malformed EPUB. ET.fromstring annotated `# nosec B314` (local
  user file, no external-entity expansion).
- POST /audiobook/import (UploadFile) → {text, chapters}.

Frontend: an Import button (.txt/.md/.epub) that fills the script editor.

Tests: tests/test_longform_import.py (9) incl. an in-memory synthetic EPUB
(spine order, empty-doc skip, tag stripping, bad-zip). 64 backend + 326 frontend
green; build clean; en.json valid.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 14:33:33 +05:30
Palash DebnathandClaude Opus 4.8 bd62659b9f docs(docker): maintain Docker Hub overview in-repo + auto-sync on main (#410)
The hub.docker.com/r/palashdeb/omnivoice-studio overview was managed by
hand and had gone stale (stuck at the sha-f86beb0 era, missing the tag
table, audiobook/long-form, Supertonic-3, server-mode networking notes).

Add deploy/dockerhub-overview.md as the source of truth and a
peter-evans/dockerhub-description step in docker.yml that pushes it to
Docker Hub on main pushes. Gated identically to the image push: only when
DOCKERHUB_TOKEN is set, so forks / GHCR-only runs are unaffected.

Overview adds the :latest=preview / :stable=release tag semantics (matching
docs/install/docker.md), the current feature set, server-mode + LAN
networking notes, and shields badges. Short description is 98/100 chars.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 14:26:40 +05:30
Palash DebnathandClaude Opus 4.8 7af5143fac feat(audiobook): per-chapter preview + resume + chapter fault-isolation (PR 3/8) (#411)
* feat(audiobook): per-chapter preview + resume + chapter fault-isolation (PR 3/8)

Builds on the shared core (#408) and metadata UI (#409). Chapter-level control,
the spec's PR 3.

Shared core:
- chapter_cache_key(spans, sr, engine_id, voice_sig) — deterministic content
  hash of a chapter's audio inputs. Same inputs → reuse; any change (text,
  voice, order, pauses, sr, engine, resolved-voice signature) → re-render.

Backend (audiobook router):
- Chapter WAVs are now content-addressed in OUTPUTS_DIR/audiobook_cache. A
  re-run after a failure/interruption reuses already-rendered chapters and only
  synthesizes the missing/changed ones (resume). Job emits `cached` per chapter
  and `cached_chapters`/`failed_chapters` on done.
- Per-chapter fault isolation: a chapter that throws emits `chapter_error` and
  the job continues; the m4b assembles from the successful chapters. Re-running
  retries only the failed (un-cached) chapters.
- POST /audiobook/preview — render a single chapter to audition it; shares the
  same cache so a preview warms the full run and a re-preview is instant.
- _build_synth now exposes resolve + engine_id; _prepare_synth unifies the
  omnivoice/generic paths for both the job and preview.

Frontend:
- Plan view: a ▶ preview button per chapter with inline playback.
- Done panel: "reused N chapters" + "N failed — click Create to retry" notes.

Tests: chapter_cache_key determinism + sensitivity (8); preview validation +
cache-hit-skips-synth (3). 55 backend + 326 frontend green; build clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audiobook): mark cache-key SHA1 usedforsecurity=False (bandit B324)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 14:19:48 +05:30
Palash DebnathandClaude Opus 4.8 086ac08592 feat(audiobook): metadata, cover art, format + loudness UI (PR 2/8) (#409)
Surfaces the shared-render-core capabilities (PR 1, #408) in the Audiobook tab.

Backend:
- POST /audiobook/cover — multipart cover upload (jpg/png, 8 MB cap), returns a
  server-side path passed back as cover_path. Unit-tested via the handler
  directly (no main+torch import).

Frontend:
- api/audiobook.ts: AudiobookGenerateBody (format/loudness/cover_path/metadata)
  + audiobookUploadCover(file).
- AudiobookTab: format select (M4B/MP3), loudness select (off/ACX/podcast,
  default off), and a "Cover & details" panel — cover picker with preview +
  title/author/narrator/year/genre/description. On create, the cover uploads
  first, then the job runs with metadata + format + loudness.
- en.json: audiobook.* keys for the new controls.

Tests: tests/test_audiobook_cover.py (4) green; frontend vitest 326 green; prod
build clean; CJK + i18n-parity gates pass.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 14:02:10 +05:30
Palash DebnathandClaude Opus 4.8 e9481ef307 feat(longform): shared render core — loudness, metadata, cover art (PR 1/8) (#408)
First slice of the Stories+Audiobook convergence (spec:
docs/specs/2026-06-13-stories-audiobook-maturity.md). Both features will compile
to one server-side chapterized renderer; this lands the shared pure builders and
wires them behind Audiobook.

New `backend/services/longform_render.py` (all pure, unit-tested without
ffmpeg/torch):
- build_ffmetadata(chapters, global_meta) — FFMETADATA1 with an optional global
  tag block (title/author→artist/narrator→composer/year→date/genre/description→
  comment) + chapter table.
- build_loudnorm_filter(preset) — `-af loudnorm` for ACX (~-19 LUFS, -3 dBTP) or
  podcast (-16 LUFS); off/unknown → None. Opt-in, so default behavior stays
  platform-identical.
- validate_cover_image — jpg/png + 8 MB cap guard.
- build_render_cmd — generalizes the m4b mux: m4b|mp3, optional cover
  (attached_pic) + loudness, bitrate validated.
- build_concat_list — moved here.

`services/audiobook.py`: build_chapter_ffmetadata / build_m4b_cmd / build_concat_list
are now backward-compatible wrappers over the core (existing imports + tests
unchanged).

`POST /audiobook`: now accepts optional `format` (m4b|mp3), `loudness`,
`cover_path`, and `metadata` and passes them through — backend-complete; the UI
for these lands in PR 2.

Tests: tests/test_longform_render.py (28) + existing test_audiobook.py (11) green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 13:49:34 +05:30
Palash DebnathandClaude Opus 4.8 f86beb041c fix(ui): UI scale via transform:scale, not zoom — fixes WebKitGTK black bands (#407)
CSS `zoom` is honoured by Chromium (the macOS/Windows webview) but IGNORED by
WebKitGTK (the Linux webview). The shell sized itself to `100vw/scale` ×
`100vh/scale` expecting `zoom` to magnify it back to full size; on Linux the
magnification never happened, so at the default uiScale of 1.3 the whole app
rendered at 1/1.3 ≈ 77% of the window, leaving black bands on the right and
bottom (a cross-platform default-parity P0 — 1.3 ships out of the box).

Switch to `transform: scale(var(--ui-scale))` + `transform-origin: top left`,
which scales identically on every engine and doesn't alter how vw/vh resolve,
so `declared (100vw/scale) × scale` fills the viewport exactly. Drop the inline
`zoom` (keep setting the `--ui-scale` CSS var the transform reads).

Verified on the real WebKitGTK webview (Tauri debug build, localStorage
uiScale=1.3): shell now fills edge-to-edge — header, content, and logs footer
all reach the window edges; no black bands.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 13:35:36 +05:30
Palash DebnathandClaude Opus 4.8 599f3bcc5c feat(engines): on-demand unload of subprocess-engine sidecars (Action 13) (#406)
Completes the dynamic engine load/unload slice. The idle reaper (#401) frees
sidecar VRAM after 5 min; this adds a user-initiated "free VRAM now" path so
multi-engine users don't have to wait:

- subprocess_backend: `list_live_sidecars()`, `unload_sidecar(id)`,
  `unload_all_sidecars()` via a shared `_force_reap(predicate)` — busy-guarded
  exactly like the idle reaper (non-blocking lock; a sidecar mid-synth is
  skipped, never interrupted; next request respawns it).
- system.py: `/model/loaded` now surfaces live sidecars as unloadable rows;
  `/model/unload/{sidecar:<id>|sidecars}` frees one or all. The existing
  generic flush panel picks these up with zero frontend change.

Also refresh CLAUDE.md stale version notes: main is 0.3.6 (latest release
v0.3.5 + 1 patch); the v0.3.0-as-unreleased framing in the project/cadence
notes is corrected to the v0.3.x continuous-to-main reality.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 13:05:25 +05:30
Palash DebnathandClaude Opus 4.8 6704d062fc fix(persona): preserve design kind + vd_states across share/import (Wave 5 §R3) (#405)
The persona-gallery surface already exists (VoiceGallery Community zone +
community.py manifest + marketplace .omnivoice bundles). The blocker for §R3's
'synthetic-only' gate was data integrity: a *designed* persona lost its
kind='design' (and vd_states) when imported from the community gallery or
round-tripped through a bundle — silently demoting it to a clone.

- community.py /use: a 'preset' (rendered from instruct) imports as
  kind='design'; a 'voice' (real reference clip) as 'clone'.
- marketplace.py: extract a pure _bundle_metadata() (dedupes export+publish)
  that captures kind + vd_states; import restores them. Old bundles without
  the keys import as 'clone' (backward-compatible).

This makes 'accept only designed/synthetic voices' enforceable instead of
everything defaulting to clone. No new persona-gallery feature was built — that
would duplicate the existing community/marketplace surface.

4 torch-free tests (isolated DB): _bundle_metadata captures design + defaults
to clone; import round-trip preserves design kind+vd_states; legacy bundle →
clone. docs §R3 status updated.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 02:21:43 +05:30
Palash DebnathandClaude Opus 4.8 151f73f794 feat(audiobook): Audiobook tab — script → plan → m4b (Wave 5 UI) (#404)
Frontend for the audiobook backend (#402/#403): a dedicated Audiobook tab.

- pages/AudiobookTab.jsx: script textarea + default-voice picker (reuses the
  app's profiles), 'Preview plan' (POST /audiobook/plan → chapter list) and
  'Create' (POST /audiobook → reads the SSE stream, shows per-chapter progress
  + assembling, then an <audio> player + m4b download via the /audio mount).
- api/audiobook.ts: typed plan() + generate() (returns the raw streaming
  Response).
- utils/sseParse.js: pure splitSSEBuffer/parseSSELine helpers for reading the
  POST event-stream (EventSource is GET-only) — unit-tested (the buffer/line
  handling is the easy thing to get subtly wrong).
- NavRail + App.jsx wiring (lazy tab, hideSidebar); i18n keys in en.json.

All strings via i18n (CJK gate green). 7 new SSE tests; full vitest 326 +
vite build green. Runtime-unverifiable here (Tauri webview) — wants an in-app
pass. docs §R3 updated.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 02:02:25 +05:30
Palash DebnathandClaude Opus 4.8 9441274ab6 feat(audiobook): synth job → chapterized m4b, SSE progress (Wave 5) (#403)
Completes the audiobook backend: POST /audiobook renders each chapter through
the active TTS engine (synthesize_chapter + chunked_tts), writes per-chapter
WAVs, then muxes a chapterized m4b (FFMETADATA1 chapters via build_m4b_cmd +
concat demuxer). Progress streams as SSE (started/chapter/assembling/done/
error), recorded to job_store. ffmpeg-gated — emits an error event and stops
when ffmpeg is absent (m4b is the only output).

- services/audiobook.build_concat_list: pure ffmpeg concat-list builder with
  proper single-quote escaping (no arg injection). Unit-tested.
- router: voice resolution (compact form of generation.py's locked/design/
  clone cases) cached per id; OmniVoice native model path + generic TTSBackend
  path; chapter synthesis runs on the GPU pool, ffmpeg via run_ffmpeg.

Reuses the tested building blocks from #402 (parser, synthesize_chapter,
FFMETADATA + m4b argv builders) — the new router glue is thin and
import-checked by CI. Deferred: epub/pdf ingest, ACX loudnorm mastering,
crash-resume, UI. 15 audiobook tests (added concat-list); docs §R3 updated.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 01:29:41 +05:30
Palash DebnathandClaude Opus 4.8 34b47282af feat(audiobook): chapterized audiobook core + plan preview (Wave 5) (#402)
* feat(audiobook): chapterized audiobook core + plan preview (Wave 5)

First cut of the long-form vertical (parity §R3). Engine-agnostic core in
services/audiobook.py:

- parse_audiobook_script: pure parser. Markdown '# H1' headings → chapters;
  inline [voice:NAME] switches the narrator; [pause …] is delegated to the
  shared omnivoice.utils.text.parse_pause_markers so audiobooks and single-shot
  synthesis keep one pause dialect. Returns a chapter/span plan.
- synthesize_chapter: orchestration via an injected synth(text, voice) callable
  (reuses chunked_tts split + crossfade, stitches inter-span silence) — so it's
  unit-testable with a stub backend, no model/GPU.
- build_chapter_ffmetadata + build_m4b_cmd: pure FFMETADATA1 [CHAPTER] builder
  and faststart-m4b concat-demux argv (bitrate-validated, no injection).

POST /audiobook/plan returns the parsed plan (no TTS/ffmpeg, no side effects).

Deferred (follow-ups): the streaming synth job + chapterized-m4b run, epub/pdf
ingest (new dep), ACX loudnorm mastering, crash-resume, UI.

14 tests: parser (chapters/voice/pause/intro/empties/to_dict), FFMETADATA
offsets+escaping, m4b argv + bitrate guard, and stub-synth orchestration
(span+silence stitching, voice threading). docs §R3 status updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(audiobook): linear-time regexes (CodeQL ReDoS)

CodeQL flagged polynomial backtracking on user-provided input in three
regexes reachable from the new POST /audiobook/plan endpoint:

- _VOICE_RE: \s*(...)\s* → single [^\]]* class, stripped in code.
- _HEADING_RE: trailing [ \t]* removed; title captured greedily + stripped.
- _PAUSE_RE (omnivoice/utils/text.py): the numeric spec is now an atomic
  group (?>…) so its leading \s+ can't backtrack against the trailing \s*.
  Behavior-preserving (Python >=3.11 already required); 14 pause tests + 14
  audiobook tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(audiobook): require non-space heading title start (CodeQL ReDoS)

The previous _HEADING_RE '[ \t]+(.+)' still let the leading whitespace class
and the title '.+' both match the same tab run (overlap → polynomial). Anchor
the title capture with \S so the two can't overlap. 14 audiobook tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(audiobook): exclude '[' from voice-tag content (CodeQL ReDoS)

[^\]]* still matched '[', so a run of nested [voice: prefixes produced
overlapping finditer match attempts → O(n^2). Excluding both brackets
([^\]\[]) makes matches non-overlapping and linear. A voice name never
contains a bracket. 14 audiobook tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 01:17:01 +05:30
Palash DebnathandClaude Opus 4.8 34c8ab2409 feat(engines): idle-reap subprocess-engine sidecars to free VRAM (Wave 13) (#401)
Parity Action 13 (dynamic load/unload), subprocess-engine half. A subprocess
engine's sidecar holds a process — and, for GPU engines, VRAM — for the life
of the backend, even after the user switches engines. The default in-process
OmniVoice model already idle-unloads (model_manager.idle_worker); this gives
the subprocess engine class the same treatment.

subprocess_backend gains a background reaper (lazy daemon thread, started on
first spawn) that shuts down sidecars idle past OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S
(default 300 s; <= 0 disables). The next request transparently respawns one via
the existing dead-process relaunch. Safety: the reaper only acts while holding
the per-backend lock acquired NON-blockingly, so it can never run mid-op — if
an op holds the lock it skips that backend this round. Reuses the idempotent
shutdown() (which doesn't take the lock, so no re-entrancy). Each backend tracks
last-use and registers in a weak live-set.

Scope: subprocess engines only (the heavy, VRAM-holding, process-isolated
class). In-process non-default engines and cross-engine VRAM preemption remain
TODO — get_active_tts_backend returns a fresh instance per call, so those need
an instance-tracking refactor.

6 reaper tests via the stdlib echo sidecar (no torch): kills idle, respawns,
skips busy (lock held), recent-use kept, disabled at <=0, ignores dead. The 3
subprocess suites pass together (24).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 00:38:14 +05:30
Palash DebnathandClaude Opus 4.8 7d7d07c8fc feat(dictation): wire AEC end-to-end in the frontend (Wave 8, opt-in) (#400)
Completes Action 8: dictate-over-playback echo cancellation now works
end-to-end, gated behind a new off-by-default 'aecEnabled' pref so the
standard dictation + playback paths are untouched when off.

- utils/aec/{pcm,farEndBus,micCapture,playbackTap}.js + public/aec-worklet.js:
  AudioWorklet captures the mic as raw int16 PCM; a player tap routes playback
  output through Web Audio to a singleton far-end bus. Pure framing/encode
  helpers are unit-tested.
- CaptureWidget: when aecEnabled, opens /ws/transcribe?aec=1, streams tagged
  PCM (0x00 mic / 0x01 far-end) instead of MediaRecorder/WebM. Default path
  unchanged; no POST fallback in AEC mode (the WS is the sole channel).
- WaveformPlayer: while actually playing AND aecEnabled, taps its decoded
  output as the echo reference. Gated on isPlaying so only the one active
  player holds an AudioContext (well under the browser cap); audio stays
  audible (source always reconnected to destination).
- Settings → Capture: AecPanel toggle. prefsSlice: aecEnabled (persisted).

Runtime-unverifiable here (jsdom has no Web Audio); needs in-app testing in
the Tauri shell. 7 new pure-helper tests; full vitest (319) + vite build green.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 23:52:02 +05:30
Palash DebnathandClaude Opus 4.8 e8705a106d feat(dictation): opt-in NLMS AEC for dictate-over-playback (Wave 8b) (#399)
* feat(dictation): opt-in NLMS AEC for dictate-over-playback (Wave 8b)

Dictating while OmniVoice plays audio (TTS preview, dub, video) leaks the
loudspeaker signal into the mic, and the streaming ASR transcribes that
bleed. Browser echoCancellation varies per platform/webview — it can't be a
cross-platform default — so this adds a server-side canceller that behaves
identically everywhere.

services/aec.py ports Patter's NlmsEchoCanceller (MIT): a time-domain NLMS
adaptive filter with a Geigel double-talk detector, warm-up step ramp, and
far-end staleness pass-through. /ws/transcribe gains an opt-in '?aec=1[&sr=]'
mode: frames are raw int16 mono PCM tagged with a 1-byte prefix (0x00 mic,
0x01 playback reference); the mic is cleaned against the reference before
buffering, and the cleaned PCM is muxed via stdlib wave (not ffmpeg). Without
the param the protocol and behaviour are byte-for-byte unchanged.

Backend ships dark (no new deps — numpy already pinned); frontend far-end
streaming is a follow-up. Tests cover echo attenuation, double-talk
preservation, cold/stale pass-through, param validation, and the framing
helpers — all pure-numpy/stdlib so they skip the torch ASR stack.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(capture_ws): stubs accept the new pcm_sr kwarg

_transcribe_buffer/_transcribe_buffer_full gained an optional pcm_sr kwarg
for the AEC PCM path; the protocol-test stubs had fixed signatures and
raised TypeError on it, so the handler sent 'error' instead of 'final'.
Accept **kw in the stubs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 23:08:45 +05:30
Palash DebnathandClaude Opus 4.8 e862f0faf0 feat(asr): crash-isolated faster-whisper subprocess backend (Wave 4.2) (#393)
* feat(asr): crash-isolated faster-whisper subprocess backend (Wave 4.2)

Native ASR engines (faster-whisper / CTranslate2) can segfault on GPU
teardown — a process-level crash that kills the whole backend. Running the
engine in a child process turns that into a failed job: the sidecar dies,
the parent raises a decorated error (engine id + device), and the next
request respawns a fresh sidecar.

- services/subprocess_asr.py: SubprocessASRBackend reuses
  SubprocessBackend's wire protocol + lifecycle — including
  respawn-on-dead-process (_spawn relaunches when the child isn't alive) and
  GPU-slot acquire/release — adding a 'transcribe' op (the TTS 'generate'
  surface is stubbed). IsolatedFasterWhisperBackend wraps faster-whisper
  using the PARENT venv (already a dep — only the process boundary is new);
  opt-in via OMNIVOICE_ASR_BACKEND=faster-whisper-isolated.
- engines/_asr_sidecar/main.py: the faster-whisper runner (stdlib wire
  protocol; torch/CT2 import lazily so the ready handshake fits the timeout).
- engines/_echo/main.py: a 'transcribe' echo op so the round-trip + crash
  recovery are testable without a real engine.
- asr_backend._REGISTRY is now a lazy dict (mirrors the TTS registry) so the
  isolated backend lists/resolves without importing the subprocess stack
  unless selected.

Tests (echo sidecar, stdlib-only): round-trip, single long-lived sidecar
across calls, crash-mid-transcribe → decorated error + backend healthy +
next call respawns, registry exposure, generate-not-supported.

Spec 7 / parity program Wave 4.2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(asr): deterministic crash test + drift marker for lazy ASR registry (Wave 4.2 CI)

CI surfaced two issues:
- The echo crash test relied on the crash-AFTER-reply hook, whose reply
  may still reach the parent (timing-dependent) — and a leaked
  OMNIVOICE_ECHO_CRASH from a sibling subprocess test poisoned the
  non-crash tests. Fix: a deterministic OMNIVOICE_ECHO_CRASH_NO_REPLY hook
  that exits BEFORE replying (guaranteed dead pipe → decorated error), and
  the asr fixture clears both crash envs so the round-trip/two-call tests
  can't inherit a leak.
- check-docs-drift's _ASR_MARKER didn't match the new lazy registry line
  (_LazyASRRegistry({); updated the marker + the self-test fixture.

Verified the no-reply crash hook by driving the sidecar directly
(reply=None, exit 1); drift self-test + real-repo check green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(asr): allowlist the 'segments' op so transcribe replies aren't dropped (Wave 4.2 CI)

The parent's PARENT_INBOUND_OPS frozenset gated inbound sidecar frames but
never included 'segments' — the ASR transcribe reply op. _recv() dropped the
frame as disallowed, tail-recursed, hit EOF, and returned None, so every
transcribe surfaced as a bogus 'sidecar crashed mid-transcription'. TTS
('audio') was allowlisted; ASR ('segments') was missed. Add it (and list
'transcribe' in the informational SIDECAR_INBOUND_OPS), update the exact-shape
allowlist test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 21:50:42 +05:30
a330c9774c docs(spec): Voice Console 10/10 polish spec (#394)
* docs(spec): Voice Console 10/10 — pinned action bar, two-kicker hierarchy, unified presets, identity-first right rail

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(spec): ASCII '+' in wireframes — clears the CJK gate

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 21:07:34 +05:30
4270645469 fix(layout): rail-right + hidden-sidebar left a phantom 48px gap (#398)
The 2-column sidebar-hidden template still sent the nav rail to
grid-column 3 — it overflowed into an implicit column and the reserved
48px slot rendered as a dead black band beside it. Rail now maps to
column 2 under that combo (and history-panel to column 1 under
rail-right+collapsed). Verified in WebKit: main/footer edges meet the
rail exactly.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 20:58:24 +05:30
3099e5de91 feat(studio): Voice Console 10x P4 — contrast, radiogroups, focus rings, reduced motion (#397)
Craft pass per docs/specs/voice-console-10x.md §3 (a11y gate, 8-pt
rhythm) and §4 acceptance:

- Contrast: solarized --chrome-fg-muted #657b83 (2.92:1 on --chrome-bg)
  → #899da4 (4.59:1, same hue); all other themes already pass. Readable
  kickers/labels that rode the decorative dim token (identity-line
  kicker, starting-points label, wv active kicker, slider kicker,
  describe hint) switch to the muted token. --chrome-fg-dim itself
  stays decorative-only.
- Radiogroups: design category chip groups are role="radiogroup"
  (aria-label = category name) with role="radio" + aria-checked chips,
  roving tabindex, and ArrowLeft/ArrowRight selection. The shared
  Segmented control already ships radio semantics via Radix ToggleGroup
  (role="radio" items + RovingFocusGroup) — left untouched.
- Focus: one shared :focus-visible rule (outline 2px chrome-accent,
  offset 1px) for the 10x controls; verified none of them suppressed
  outlines without replacement.
- Reduced motion: dub-skel-shimmer / dub-pulse / dub-stepper-spin,
  heart-glow / logs-spin, wf-spin, and the FloatingPill dot-pulse /
  progress-sweep now stop under prefers-reduced-motion (FirstRunSetup's
  frs-alarm / frs-hw-pulse coverage verified pre-existing).
- aria-live: FloatingPill already carries role="status"
  aria-live="polite" (verified); the action bar gains a persistent
  sr-only polite status region announcing generation start/finish.
- 8-pt audit (CloneDesignTab.css): 5→4 gap, 5px 10px→4px 10px and
  3px 9px→4px 8px paddings, 7→8 grid gap.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 20:53:34 +05:30
b90b0f13ab feat(studio): Voice Console 10x P3 — identity recipe line, Active-voice card, empty-state verbs (#396)
- category chips collapse behind an 'Identity' recipe line (male · elderly
  · …) that the describe box rewrites live; all-Auto starts expanded
- right rail leads with an ACTIVE VOICE card: name, kind badge, recipe,
  identity sample player, + New; empty card carries verbs
- empty saved-voices states point at the action ('Describe one in Voice ←')
- script column stacks naturally (no void before VOICE)

Spec: docs/specs/voice-console-10x.md §1.5, §2.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 20:36:48 +05:30
d4af112975 feat(studio): Voice Console 10x — P1 pinned action bar + P2 hierarchy/presets/insert popover (#395)
P1 (fold): language, steps, and the overrides disclosure move into a
pinned action bar with SYNTHESIZE — the primary CTA is visible at every
window size (verified 1280×720 and 1400×900 in WebKit); Cmd/Ctrl+Enter
synthesizes from anywhere; overrides expand upward above the bar.

P2 (hierarchy/consistency): two kickers only (SCRIPT, VOICE — method
toggle inline); the four redundant headers removed; the old PROMPT preset
chips merge with personalities into one edge-faded scrollable 'Starting
points' lane; the 14-chip tag wall becomes a ⊕ Insert popover at the
script corner (click-outside dismiss).

Spec: docs/specs/voice-console-10x.md §1.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 20:28:21 +05:30
Palash DebnathandClaude Opus 4.8 4aa4d983aa feat(settings): Hugging Face mirror (HF_ENDPOINT) for restricted networks (Wave 4.3) (#391)
The model manager already lists/deletes cached models; this adds the
remaining high-value slice — an in-app HF mirror setting so users behind
restricted networks (e.g. the Great Firewall) can route downloads through
hf-mirror.com or any HF_ENDPOINT. Persisted to the durable per-user env
(survives Tauri/Finder launches); HF reads HF_ENDPOINT at import, so the
override applies on restart (surfaced in the UI).

- GET/PUT /api/settings/hf-mirror (loopback-gated): presets (official +
  hf-mirror.com), http(s) validation, empty clears to official.
- Models-tab panel with quick-picks + free-text field + restart note.

(Skipped 'hf cache verify' — version-fragile across huggingface_hub
releases and low value vs the mirror, which the China/Russia network
research flagged as the real gap.)

3 endpoint tests (default, set+trim+clear, non-http rejection).

Spec §R4(c) / parity program Wave 4.3.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 19:06:56 +05:30
Palash DebnathandClaude Opus 4.8 b1ffdf2387 fix(mlx): harden import guards against PyInstaller dylib failures (Wave 4.4) (#390)
MLXWhisperBackend / MLXAudioBackend is_available() caught only ImportError.
In a PyInstaller bundle mlx's native dylib/metallib can fail to load even
when the package imports, raising OSError/RuntimeError — which would
propagate and crash the registry scan instead of reporting the backend
unavailable. Broaden to (ImportError, OSError, RuntimeError) so the picker
falls back cleanly. 6 tests across all three exception types.

The capture ASR path already prefers MLX Turbo on Apple Silicon
(get_capture_asr_backend), so this hardening is the remaining slice of
Spec 6 / Wave 4.4.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 19:06:51 +05:30
5361264d12 fix(history): real 2-line title clamp + de-noised display + click-to-expand (#389)
The old max-height:3em guillotined the third line mid-glyph. Now a true
-webkit-line-clamp with ellipsis, leading [tag] control tokens stripped
from the display (full text stays in the tooltip and restore flows), and
clicking the title toggles the full prompt.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 19:01:33 +05:30
Palash DebnathandClaude Opus 4.8 b60cceb3e6 docs(engines): uv dedupe + sidecar torch-pin disk-usage policy (Wave 4.5) (#392)
Explain why dedicated-venv engines (IndexTTS2) add disk (a second torch +
CUDA libs: Linux cu128 ~0.83 GiB, Windows ~3.2 GiB), and how uv's link-mode
dedup (clone on macOS/Linux, hardlink on Windows) shares identical wheels
for free — provided UV_CACHE_DIR and the venvs are on the same filesystem.
Key policy: pin the same torch build as the parent whenever the engine
allows, since only identical wheels dedupe; UV_LINK_MODE=hardlink on Linux
ext4. Linked from the IndexTTS engine doc.

Spec §R4(a) / parity program Wave 4.5.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 19:00:49 +05:30
Palash DebnathandClaude Opus 4.8 6306f6edae docs: official Docker Hub image palashdeb/omnivoice-studio (#388)
Link the published Docker Hub repo (https://hub.docker.com/r/palashdeb/
omnivoice-studio) as an official image alongside GHCR in the README install
list and docker.md header. Same images, same tags.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 19:00:45 +05:30
3957d333b9 perf(dub): retime batches seek to their window instead of decoding from frame 0 (#387)
Each batch now uses input seeking (-ss before -i, frame-accurate under
re-encode) plus a bounded read (-t window+0.5s), with chunk times shifted
into window-relative coordinates — long-video Smart Fit exports drop from
O(n²) decode cost to O(n).

Fixes #382

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 18:23:26 +05:30
a93abcfc5e chore(i18n): backfill 36 studio-overhaul keys into en.json + all 20 locales (#386)
Canonical English added from the t() defaultValues introduced by the
overhaul PRs (#374-#381); 20 parallel translation passes added each key
to every locale (placeholders, product names, and existing per-locale
terminology preserved). All locale files parse; CJK gate + 312 frontend
tests green.

Fixes #383

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 18:11:19 +05:30
3161166328 fix: stale-chunk preload recovery (#380) + surface unsupported-GPU-arch in notifications (#284) (#385)
- #380: vite:preloadError (old hashed assets after an update) triggers a
  one-time reload to pick up the fresh manifest; session flag prevents loops
- #284: check_device_compatibility's warning (e.g. Blackwell sm_120 on a
  pre-cu128 torch) now appears in the notification panel as an error with
  the pip fix — a log line never reached affected users while synthesis
  silently produced noise. Cached once per process.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:59:29 +05:30
77d6477fc9 fix(player): WaveformPlayer paused itself on play — idempotent claim + hard listener teardown (#384)
Live-debugged in Playwright WebKit with a pause() stack hook: the media
'play' event fired twice (a stale WaveSurfer instance's listeners survive
a destroy() that throws mid-teardown under StrictMode double-mount), so
the second claimPlayback stopped the current owner — this very element.
play → instant self-pause → 'click does nothing'.

- 'play' handler only claims when it doesn't already own the slot
- per-instance stale flag inert-izes leaked handlers
- cleanup detaches handlers (unAll) BEFORE destroy so a throwing destroy
  can't leak them

Verified in WebKit: paused=false, currentTime advancing, 0 stray pause calls.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 17:33:04 +05:30
f5579b40aa fix: issue-triage batch — timeline box flicker, truncated-model detection, stale history pruning (#381)
- #373: drop will-change:transform on the segment lane (persistent
  compositor layer made the semi-transparent boxes vanish during
  playback/drag on some Windows GPUs) + raise region alpha 0.30→0.45
- #352: validate a finished snapshot actually contains weights (>5 MB
  file) so interrupted downloads fail at install time with a re-download
  hint; loader translates the opaque transformers error into the same
  guidance
- GET /history prunes rows whose audio file is gone instead of serving
  dead 404 players forever

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 16:51:00 +05:30
171febfd59 fix(player): WaveformPlayer click did nothing — media element never got a src (#379)
With an external `media`, wavesurfer's `url` option only fetches for peak
decoding and never assigns the element's src — so the waveform drew but
play() had nothing to play. Set src on the in-DOM <audio> via JSX (same
pattern as WaveformTimeline) and stop passing `url`. Also surface
playPause() rejections instead of swallowing them.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 16:17:50 +05:30
35de7d03b4 feat(studio): consolidate Clone + Design into one Voice workspace (spec P4) (#378)
One 'studio' navigation mode replaces the clone/design pair; the split
lives on as a 'Define voice' toggle (From audio / By design) at the top
of Voice Source. Selecting a saved profile sets the method from its kind.

- uiSlice: AppMode + 'studio'; defineMethod ('audio'|'design') persisted
- legacy shims: localStorage mode + restoreHistory map clone/design →
  studio + method; history mode VALUES unchanged
- NavRail/Header: single Voice entry (Fingerprint, #d3869b)
- CloneDesignTab/WorkspaceVoices/useTTS/useProfiles/Gallery/Launchpad/
  Sidebar: definition-method semantics moved off the navigation mode

Build clean; 312/312 tests; tsc clean; no setMode('clone'|'design') left.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 16:17:09 +05:30
6140f888e1 fix(dub+win): dialect↔cinematic guidance loop + WinError 193 ffmpeg validation (#377)
* fix(dub): break the dialect↔cinematic guidance loop (#372, #373)

- Cinematic toggle refuses the pick when no LLM endpoint is configured,
  pointing at Settings → Credentials → LLM endpoint
- backend Fast fallback now syncs the quality toggle to 'fast'
- the dialect warning no longer fires alongside the cinematic-no-LLM
  warning (the pair formed the loop), and both messages point at the
  LLM endpoint settings instead of each other

Fixes #372

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ffmpeg): validate resolved ffmpeg/ffprobe actually runs — fall through on WinError 193 (#360, #361, #362)

A corrupt or wrong-arch imageio-ffmpeg download (and WindowsApps alias
stubs) passes os.path.isfile/shutil.which but explodes at spawn with
'[WinError 193] %1 is not a valid Win32 application', killing
transcription with an opaque 500. Every resolution step now probes the
candidate with '-version' (cached per process), logs the rejected
basename, and falls through to the next source.

Fixes #362
Fixes #361
Fixes #360

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 15:54:47 +05:30
66f2ea7e50 feat(profiles): unified profile model — kind discriminator + stored design params (spec P3) (#376)
Migration 0005_unified_profiles (0004 taken by mcp bindings):
- voice_profiles.kind TEXT DEFAULT 'clone' ('clone' | 'design'), backfilled
- voice_profiles.vd_states TEXT NULL — JSON of design category picks
- mirrored in _BASE_SCHEMA; idempotent _has_column guards; downgrade drops

POST /profiles:
- ref_audio now optional; kind + vd_states form fields with validation
  (clone requires audio; design requires vd_states JSON object + instruct)
- design profiles render a deterministic identity sample (seed 42) through
  the shared archetype renderer — one TTS code path

POST /generate:
- profile resolution branches on profile.kind (authoritative) instead of
  the brittle is_locked/instruct inference; legacy pre-0005 rows keep the
  old inference as fallback; history.mode records profile.kind

Frontend:
- 'Save design as profile' in the Design tab (vd_states + buildDesignInstruct)
- selecting a design profile restores its sliders (vd_states) for re-editing

Also unforks the alembic chain (0004_mcp + my 0004 both revised 0003 →
multiple heads broke alembic upgrade head and the 0003 migration tests).

Tests: tests/test_profile_unification.py — validation, design-create with
mocked renderer, migration up/backfill/downgrade. 18/18 profile tests,
312/312 frontend, related backend suite green.

Note: docs/specs/voice-studio-unification.md (on feat/studio-ux-overhaul)
still says 0004 — renumber to 0005 when branches meet.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 14:29:18 +05:30
851ca4e012 feat(studio): workspace UX overhaul — right-side panels, shared waveform player, dub pipeline UX, setup polish (#374)
* feat(studio): workspace UX overhaul — right-side panels, shared waveform player, dub pipeline UX, setup polish, UI-wide fixes

Voice workspace (specs: docs/specs/voice-studio-unification.md, workspace-connectivity.md):
- Right-side panels replace the left sidebar for clone/design and dub:
  WorkspaceVoices (saved profiles), WorkspaceHistory (scoped history with
  All/Clone/Design filters), WorkspaceProjects (dub projects)
- Prompt restacked over Voice Source in one definition column (spec §1)
- Gallery "Use voice" now hands off via pendingProfileId and lands in clone
- Shared <WaveformPlayer> (wavesurfer + in-DOM media element for Tauri
  WebKit, blob routing via preview endpoint, 404 -> "audio file missing")
  replaces every bare <audio controls>; lazy-mounted via IntersectionObserver

Dub:
- Pipeline stepper (Upload -> Prepare -> Transcribe -> Edit -> Generate -> Export)
- Multi-language preview switcher pills (Original + per-track, ElevenLabs-style)
- Batch multi-language generation via langOverride loop
- FloatingPill: bottom-center, suppressed on its homeMode tab (no dup progress)
- Transcript skeleton shimmer (no fake data), progress overlays the video,
  exports demoted behind Generate, empty right-panels collapse

Chrome/layout:
- Nav rail is full-window-height; content yields to the logs footer via
  padding-bottom; footer joins the rail edge (no overlap at any UI scale)
- UI scale 60–175% slider with zoom-compensated container sizing
- LogsFooter: merged single Logs tab when collapsed, per-source tabs on
  expand; Updates chip lives with the logs tabs
- Gallery: three independently scrollable filter lanes, uniform 26px controls
- Font picker as live-preview grid; double-click titlebar maximize fixed
  (single mousedown detail-2 handler)

First-run:
- Setup wizard: pinned action row + scrollable content at every window size,
  one-line head-ellipsized paths, height budget for short windows, library
  rows back to one-line grammar, raw i18n key + duplicate host fixed

Performance/i18n/consistency sweep (10-agent scan, 47 fixes):
- i18n locales lazy-loaded per language (i18n chunk 1.84 MB -> 76 kB)
- Undefined CSS vars replaced with real tokens across 8 stylesheets;
  hardcoded hexes tokenized; emoji swept to lucide icons app-wide
- Poll throttling (sysinfo subscription scoped to Header, logs 45s when
  collapsed, rAF only during playback), hardcoded strings moved to t()

Build clean; 312/312 tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(studio): re-flow clone/design columns (grid rows collapsed in restack) + strip placeholder emoji across locales

The base .studio-column grid (minmax(0,1fr) rows) collapsed to 0 height
inside the new auto-height definition column, overlapping every panel in
design mode — found via Playwright visual pass. Columns now re-flow as
natural-height flex stacks. Also removed the leftover pencil emoji from
clone.prompt_placeholder in all 21 locales.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(design): compact the design control stack — 2-up facet selects, scrollable tag row, tighter rhythm

English accent + Chinese dialect dropdowns share one row (full-width on
narrow), insertable tag chips collapse from three wrapped rows to one
scrollable line, and describe/personality spacing tightens — the whole
design stack now fits a single viewport.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(spec): unification migration renumbered 0004 — upstream 0003 is voice-profile consent

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ci): clear hardcoded-CJK gate — ASCII '+' in spec wireframes, reword voiceIcons comment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(spec): migration is 0005 — 0004 taken by mcp bindings upstream

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 14:17:38 +05:30
Palash DebnathandClaude Opus 4.8 1561ff4428 ci(docker): also publish to Docker Hub palashdeb/omnivoice-studio (#375)
Push the same images (same tag set: :latest rolling main, :stable/:X.Y.Z
releases, :sha-) to docker.io/palashdeb/omnivoice-studio alongside GHCR.
Gated on DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets — without them the
build still publishes to GHCR only. Docs-sync: docker.md mirror note.

Requires repo secrets: DOCKERHUB_USERNAME, DOCKERHUB_TOKEN.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 14:08:46 +05:30
d6562d6f30 feat(dub): Smart Fit phase B — per-segment video retime export, drift absorption, fitted subtitles (#350)
* feat(dub): Smart Fit phase B — per-segment video retime export, drift absorption, fitted subtitles

Executes the video side of the Smart Fit plans persisted by Phase A
(job["fit_plans"], #347) at export and preview time.

Backend:
- services/video_retime.py (new, clean-room): two-tier retime executor.
  ≤48 chunks → the proven single-pass split/trim/setpts/concat
  filter_complex; above → batches of 40 chunks rendered to intermediate
  slices (identical libx264 medium/crf20 params, keyframe at t=0) joined
  losslessly with the concat demuxer. Slices are CFR-resampled (fps=)
  because setpts leaves VFR-ish timestamps that broke tpad and drifted a
  frame per retimed chunk on ffmpeg 7.x. Temp slices cleaned on success
  AND failure/abort.
- Drift absorption: fitted track longer than retimed video → freeze-frame
  tail (tpad=stop_mode=clone) predicted into the last slice / single-pass
  graph, with residual mux-side tpad; video longer → silence-pad the dub
  audio chain (apad=whole_dur). ±50 ms tolerance.
- VFR guard: probe r_frame_rate vs avg_frame_rate; normalise with fps=
  before trim/setpts; probe failure degrades gracefully.
- Plan resolution: _video_retime_plan_for spans legacy video_stretch_plans
  (byte-identical resolution + command construction) and fit_plans, gated
  on the track's own timing_strategy so stale plans never retime a track
  re-generated under another strategy.
- Fitted subtitles: /dub/srt + /dub/vtt accept ?lang= and serve cue times
  from fitted_segments for Smart Fit tracks; _write_burn_srt does the
  same for burn-in. burn_subs+retime is now allowed for smart_fit (burn
  runs AFTER the retime graph); still rejected for legacy stretch_video.
- /dub/preview-video resolves the same plan so in-app preview matches
  export.
- Fallback ladder: batch encode failure/timeouts → un-retimed export with
  a structured core.failure warning (X-Dub-Export-Warning header +
  job["last_export_warning"]); concat join rejection → one single-pass
  retry while ≤96 chunks; abort → 409 + proc kill via run_ffmpeg job_id
  registration (/dub/abort reaches export encodes now) + temp cleanup.

Frontend:
- Export drawer passes ?lang= on subtitle exports and shows an i18n'd
  re-encode cost note (~0.5–2× video length on CPU) when a retiming
  strategy is active — translated in all 21 locales.

Tests: tests/test_smart_fit_export.py — plan resolution, batch math,
graph parity + new stages, fitted-cue SRT/VTT/burn selection, burn
policy, VFR detection; ffmpeg-gated integration renders both executor
tiers (batch size forced to 2) and the real /dub/download endpoint,
ffprobing durations within ±50 ms across both pad branches. All existing
dub export/subtitle/preview/timing tests pass unchanged.

Refs docs/competitive-analysis.md Action 1 (dub-length fitting v2);
completes Smart Fit (Phase A = #347).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): sanitize Smart Fit retime work paths at every sink (CodeQL py/path-injection)

The job_id-derived retime work path (retimed_*.mp4 / preview_retimed_*.tmp.mp4)
flowed unguarded from dub_export into prepare_smart_fit_video /
render_retimed_video and their derived slice/concat paths and ffmpeg argv.
Apply the repo's proven inline realpath+startswith containment pattern
(helpers/commonpath are not recognized — see #309/#328/#329/#348):

- dub_export.py: validate work_path against DUB_DIR at both construction
  sites (export + preview) and pass the validated realpath onward.
- video_retime.py: make both entry points self-defending — realpath +
  DUB_DIR containment on out_path/work_path before any derivation, raising
  RetimeError(stage="plan") on escape; slices_dir/slice_path/list_path and
  RetimeDecision.file_path now all derive from the sanitized value. DUB_DIR
  is read via module attribute so test fixtures reloading core.config work.
- ffmpeg_utils.py: document that all caller-assembled argv paths are
  realpath-validated upstream.
- tests: sandbox DUB_DIR in the executor integration tests (tmp_path) so
  the new guard sees the test workspace.

No behavior change for valid (server-built) paths — the guard only fires
on traversal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(smart-fit): patch DUB_DIR on video_retime's own config ref — survives suite-wide reload

The retime guard reads video_retime._config.DUB_DIR at call time; the
sandbox fixture patched a fresh 'import core.config' instead. Another
test reloads core.config in the full suite, so the two module refs
diverged — the patch missed and the guard rejected the test's tmp paths
(green in isolation, red in CI's full run). Patch the exact ref the
guard dereferences.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dub): resolve DUB_DIR live at call time in retime guards — survive full-suite reload

The path-containment guards bound DUB_DIR via a module-level
'from core import config as _config'. Other tests importlib.reload()
core.config (sandboxing OMNIVOICE_DATA_DIR), after which the guard
checked containment against a stale DUB_DIR while dub_export built the
path under the reloaded one — every retime path then 'escaped the dub
workspace' (green file-alone, red full-suite: the 5 integration
failures CI hit). Re-import DUB_DIR locally in each guard so it always
reads the current sys.modules value; simplify the sandbox fixture to
patch the canonical module. Verified: full backend suite green on the
Smart Fit tests (the 2 remaining settings_store failures are
pre-existing on main, unrelated — local data-dir artifact).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): clear CodeQL alerts on Smart Fit export — job_id allowlist, proc-registry decouple

- py/path-injection (8, video_retime.py): validate job_id with a strict
  inline regex allowlist (re.fullmatch [A-Za-z0-9_-]{1,64}) at the entry
  of dub_download and dub_preview_video, before it reaches any filesystem
  path or ffmpeg argv. The existing realpath containment guards stay as
  defense-in-depth; the regex barrier is the sanitizer CodeQL recognizes
  through the service-module call chain.
- py/log-injection (4): newline-strip job_id inline at the logger calls
  in ffmpeg_utils.run_ffmpeg and the two retime-fallback logger.error
  sites in dub_export.
- py/empty-except (3): best-effort cleanup os.remove handlers now log
  the OSError at debug instead of bare pass (video_retime + both
  dub_export mux finally blocks; _discard_tmp too for consistency).
- py/cyclic-import (2): break the dub_pipeline <-> ffmpeg_utils cycle
  for real — the subprocess registry (register_proc/unregister_proc/
  kill_job_procs/has_active_procs + state) moves to a new stdlib-only
  leaf module services/proc_registry.py. ffmpeg_utils now imports it at
  module top (no lazy import); dub_pipeline re-exports every name so
  dub_core aliases and tests keep working unchanged.

No behavior change for valid inputs; invalid job ids now get a clean
400 instead of a 404/containment error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dub): address #350 review — cancelled-vs-failed retime, logged best-effort excepts, redacted probe logs, narrowed test assert

- rc<0 (killed by user cancel) now raises RetimeError(stage='aborted')
  instead of reporting an ordinary render failure (CodeRabbit)
- best-effort cleanup/QC-event excepts log at debug instead of bare pass
  (CodeQL empty-except x3)
- probe failure logs use basename, not full user paths (CodeRabbit/CodeQL)
- test_render_cleans_slices_on_failure asserts RetimeError, not Exception

Rebuttals (no change needed, see PR comment): fitted-cue subtitles track
the fitted AUDIO timeline which is correct even on retime fallback;
the planner only emits stretch ratios >1 so the early-exit guard is a
true no-op check; '\'' is ffmpeg's own utility quoting for concat lists;
has_active_procs is an intentional re-export (noqa'd).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 14:06:00 +05:30
Palash DebnathandClaude Fable 5 825f4f7ac6 feat(dub): regenerate subtitle timeline on the fitted timeline (Wave 3.1) (#371)
Smart Fit Phase A (planner) + the export-side video retime + audio stretch
already shipped (#347 + dub_export stretch filter). The last piece of
Spec 1 was the subtitle timeline: under stretch_video the dubbed audio
plays at FITTED positions, but the standalone SRT/VTT export still used the
original segment times — so external subtitles drifted against the dubbed
video.

- services/fitted_subtitles.py (pure, tested): map_time_to_fitted() +
  fitted_cues() remap original cue times onto the same per-chunk
  {orig→new, stretch_ratio} plan the video stretch uses, with a
  monotonicity guard.
- dub_export SRT + VTT endpoints: when a job used stretch_video, cues are
  regenerated from the plan (subtitles track actual dub placement); no
  plan → original times, unchanged. New optional ?lang= selects the track.

7 pure tests (chunk-bound mapping, linear interpolation, unit-rate tail,
fitted cues, monotonicity, empty-plan identity).

Spec 1 (remaining) / parity program Wave 3.1.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 13:03:00 +05:30
Palash DebnathandClaude Fable 5 a12492af07 feat(dub): second-pass ASR QC — flag lines whose dub drifts from target (Wave 3.3) (#370)
After a dub is generated, re-recognize the synthetic audio and compare what
the ASR heard against what we asked the TTS to say. Lines that drift are
flagged for the user to re-listen / re-dub — turning subtitle timing and
pronunciation from trusted math into measured truth, and doubling as an
automatic dub-quality check.

Design delta from pyvideotrans (which lets recognized text REPLACE the
subtitles wholesale): we keep the generated text authoritative and use the
second pass only for MEASUREMENT — a per-line drift score + measured
start/end that feed the incremental re-dub loop, never silently overwriting
the translation.

- services/dub_qc.py (pure, tested): word_error_rate (normalized token edit
  distance, case/punct-insensitive, script-agnostic) + score_dub (matches
  recognized segments to dub segments by time overlap, concatenates the
  hypothesis, scores drift, derives measured bounds).
- POST /dub/qc/{job_id}: runs the active ASR backend on the dubbed track in
  the GPU pool, annotates each segment with qc_drift/qc_flagged/
  qc_recognized/qc_measured_start-end (non-destructive — content untouched),
  persists, emits a qc_done job event. Opt-in, never fatal.
- Frontend: dubQc() API fn + a red 'Verify' badge on flagged segment rows
  (en.json keys; other locales fall back).

12 pure scoring tests (identical/substitution/empty/no-overlap/multi-segment
matching/measured-timing); endpoint validated in CI.

Spec 5 / parity program Wave 3.3.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 12:50:39 +05:30
Palash DebnathandClaude Fable 5 8cce99298a feat(dub): per-segment clone references (Wave 3.2) (#369)
Cut each long-enough dub segment's clone reference from the isolated vocals
at that segment's own timestamps, so the dub of each line carries the
prosody/emotion of its source line — finer than one reference per speaker.
Reimplemented from the clean-room spec (pyvideotrans per-line ref idea); our
design delta is a quality floor with fallback.

- services/speaker_clone.py: extract_segment_refs() keyed by segment id;
  reference transcript is the SOURCE text (text_original), since the vocals
  slice is source-language audio. Floor at MIN_SEGMENT_REF_DURATION_S=3.0
  (not the per-speaker 5.0, which most dialogue lines fall under) — shorter
  lines are omitted and fall back to the per-speaker clone, so it's a strict
  improvement, never a regression.
- dub_core: run extraction at transcribe (per_segment_refs query param,
  default on), store job['segment_clones'], default each unassigned
  segment's profile_id to 'auto-seg:{id}' when it has its own ref, else the
  existing 'auto:{speaker}'. Forcing per-speaker (per_segment_refs=false)
  is supported for long-form consistency.
- dub_generate _gen: resolve 'auto-seg:' from segment_clones, ahead of the
  per-speaker 'auto:' path. profile_id is already a fingerprint field, so
  flipping the mode re-dubs automatically (no _GEN_INPUT_FIELDS change).

7 pure tests over a synthetic vocals wav (own-ref for long lines,
short-line omission/fallback, source-text transcript, bounds clamping,
floor boundary). Pipeline wiring validated in CI.

Spec 4 / parity program Wave 3.2.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 12:50:35 +05:30
Palash DebnathandClaude Fable 5 99357e8c5b feat(mcp): MCP server v1 — mount on /mcp, per-agent voice binding, stdio shim (Wave 2.2) (#368)
* feat(mcp): MCP server v1 — mount on /mcp, per-agent voice binding, stdio shim (Wave 2.2)

The FastMCP server (previously dead code, never mounted) is now mounted on
the main FastAPI app at /mcp via Streamable HTTP, with its session manager
composed into the app lifespan through an AsyncExitStack (best-effort: a
missing mcp package or OMNIVOICE_MCP_DISABLE=1 never breaks startup).
streamable_http_path set to '/' so the sub-mount lands at /mcp, not
/mcp/mcp. Adds the 'mcp' dependency (1.27.x).

Per-agent voice binding (Spec 2 headline): each MCP client sends an
X-OmniVoice-Client-Id header; generate_speech resolves the voice as
explicit arg > the client's binding > global default > app default. New
mcp_client_bindings table (alembic 0004 + _BASE_SCHEMA, additive/idempotent),
services/mcp_bindings.py (CRUD + resolve_voice + best-effort last_seen),
and a loopback-gated REST router (/api/mcp/bindings) the Settings panel
drives.

New transcribe tool (base64 audio in, 200 MB cap). Stdio shim
(backend/mcp_shim, httpx-only, ported from voicebox MIT) proxies stdio
clients to the mounted endpoint and forwards OMNIVOICE_CLIENT_ID as the
binding header. Settings → Sharing gains an MCP bindings panel. Docs:
docs/mcp.md (both connection modes + binding REST) and docs/mcp.json
updated to the shim form.

Tests: bindings service + resolution precedence + migration up/down (pure,
run locally); REST CRUD + mount-not-404 + disable-flag (main-importing,
validated in CI). MCP build + mount + initialize handshake verified
out-of-band (no torch).

Spec: docs/competitive-analysis.md Spec 2 / parity program Wave 2.2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(mcp): assert /mcp mount via app.routes, not a lifespan client

The two main-importing mount tests ran the app lifespan, which now starts
the FastMCP session manager and binds asyncio queues to the test loop —
contaminating later lifespan-running tests ('bound to a different event
loop'). The mount happens at import time, so inspecting app.routes for the
/mcp Mount is the correct loop-free assertion. Same fix shape as the
Wave 0.2 consent tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(mcp): stop reload-main poisoning across the MCP test files

Root cause of the CI failure: the bindings REST fixture set
OMNIVOICE_MCP_DISABLE=1 and reloaded main but never restored it, so a
later 'from main import app' in test_mcp_mount saw /mcp un-mounted
({'/audio','/voice_audio'}). Reloading main mutates the shared module for
every subsequent test.

- REST fixture: drop the disable flag (the mount is harmless without a
  lifespan), yield the client, and restore main (+ core.config/db) to the
  default data dir in teardown so the global module is clean again.
- test_main_mounts_mcp_route: reload main with the disable flag cleared so
  the assertion is independent of any earlier reload.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 11:56:19 +05:30
Palash DebnathandClaude Fable 5 c8fdcb619a fix(settings): remove stray rebase conflict marker in settings.py (#367)
A '>>>>>>>' marker from the #365 rebase was committed at the tail of the
LLM-endpoint block, making the module unparseable. Strip it; settings.py
parses clean and the endpoint tests pass.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 04:08:45 +05:30
Palash DebnathandClaude Fable 5 d0b46f249e feat(settings): remote LLM endpoint UI — Ollama/vLLM/LM Studio (Wave 2.4) (#365)
A focused Settings panel for the OpenAI-compatible LLM that powers
cinematic translate, glossary auto-extract, and dictation refinement
(Wave 2.1). Persistence reuses the existing TRANSLATE_BASE_URL /
TRANSLATE_MODEL / TRANSLATE_API_KEY env vars (already in system.py
PERSISTENT_KEYS, restored at startup), so llm_backend/translator
resolution is unchanged — vLLM is a verified drop-in, Ollama ignores the
key, vLLM/LM Studio require it.

- GET/PUT /api/settings/llm-endpoint (loopback-gated): read shape returns
  base_url, model, masked key, and live availability; PUT treats a null
  field as unchanged and an empty string as clear (so the key isn't wiped
  by a base-url-only save). Key is masked to last-4 in the read path,
  never echoed.
- Credentials-tab panel with one-click presets (Ollama/LM Studio/vLLM/
  OpenAI), base URL + model + optional key fields, and a reachable/not
  status badge.

6 endpoint tests (read shape, set+mask, null-unchanged, empty-clears,
local-url-no-key, short-key masking); availability assertions guarded on
openai being installed.

Spec: parity program Wave 2.4 / competitive-analysis §R2 rung 4.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 03:58:44 +05:30
Palash DebnathandClaude Fable 5 9b6d1d0863 docs(agentic): OmniVoice as a TTS/STT provider for pipecat/LiveKit (Wave 2.5) (#366)
Agentic v1: OmniVoice is a provider, not the orchestrator. Its existing
OpenAI-compatible API already serves everything pipecat/LiveKit need
(POST /v1/audio/speech with pcm/wav, voice-profile id, speed; default
24 kHz output matching pipecat's OpenAITTSService) — so this is docs + an
example + a contract test, no new endpoint.

- docs/agentic-voice.md: the provider recipe for pipecat (base_url to
  :3900/v1) and LiveKit, the remote-backend note (bearer from 2.3), the
  consent-locked-voice nudge (0.2), and an explicit telephony-is-deferred
  scope box.
- examples/agentic/pipecat_minimal.py: lazy-import skeleton wiring the
  OmniVoice STT/TTS services (importable without pipecat installed).
- tests/test_agentic_provider_contract.py: pins the /v1/audio/speech
  request shape pipecat sends (pcm + wav formats, voice-profile passthrough,
  speed) so the documented recipe can't silently break. Validated in CI.

Spec: Action 15 / §R1 v1 / parity program Wave 2.5.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 03:57:26 +05:30
Palash DebnathandClaude Fable 5 22ba348f17 feat(remote): backend URL + bearer key + Tailscale docs (Wave 2.3) (#364)
Run inference on a remote GPU box, drive it from the desktop app — opt-in,
off by default (loopback-only is unchanged when no key is set).

Backend:
- BearerKeyMiddleware (main.py): when OMNIVOICE_API_KEY is set, every
  non-loopback HTTP + WebSocket request must present it (Authorization:
  Bearer, ?api_key=, or the ov_key cookie set on first auth). Pure ASGI
  (no response buffering), loopback always bypasses, SPA shell stays
  reachable. Constant-time compare, never logged.
- ws_remote_authorized() in dependencies; capture_ws lets a keyed
  non-loopback client through its inline loopback guard (the thin-client
  dictation case: mic local, GPU remote).

Frontend:
- api/client.ts: ov_backend_url (localStorage) is the top-precedence base
  override; new wsUrl() derives ws scheme + host from the API base (not
  window.location, which lies in the Tauri webview) and appends ?api_key.
  apiFetch attaches the bearer header. Both WS call sites (dictation,
  events) routed through wsUrl; the HTTP transcribe fallback through
  apiFetch.
- Settings > Sharing > Remote backend panel: URL + key fields, a
  test-connection probe against {url}/health, save-and-reload.

Docs: docs/remote-gpu.md — the Tailscale recipe (MagicDNS + Serve, never
Funnel, headscale note, plain-HTTP-is-sniffable warning, PIN-vs-key split).

Tests: 10 bearer-middleware cases (inert without env, loopback bypass,
401 without/pass with key via header+query, wrong key, shell exemption,
plain-ASGI guard, WS handshake reject/accept). Validated in CI.

Spec: parity program Wave 2.3 / competitive-analysis §R2 rungs 1-3.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 03:57:20 +05:30
Palash DebnathandClaude Fable 5 10806fea4f feat(dictation): optional local-LLM refinement of finals (Wave 2.1) (#363)
Phase 2 of Spec 3, on top of Wave 1.1's deterministic collapse. Prompt
design ported from voicebox (MIT): 'text filter, not an assistant' base
instruction + three toggleable sections (smart_cleanup, self_correction,
preserve_technical) + 7 few-shot examples passed as STRUCTURED chat turns
(small local models echo inline examples). Runs through the user's own
Ollama/LM Studio/OpenAI-compat endpoint via llm_backend — new additive
chat_messages() on the adapter; chat() now delegates to it.

Pass-through is the contract: with no LLM configured (backend 'off'),
on any error/timeout, or on an empty reply, the raw transcript stands —
identical default behavior on every platform. Refinement runs off-thread
on FINALS only; the WS final dict gains optional refined_text and the
dictation pill pastes refined_text ?? text (raw kept in history).

Settings: GET/PUT /api/settings/dictation-refinement (loopback-gated,
persisted in the settings table) + a Capture-tab panel with the master
switch + per-flag toggles and a 'no LLM configured' hint.

15 new unit tests: prompt sections per flag, structured few-shot message
shape, and the full maybe_refine pass-through matrix (off backend,
disabled config, LLM failure, empty reply, empty input).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 00:13:56 +05:30
Palash DebnathandClaude Fable 5 ac8bdecfa6 test(api): pin the /generate surface pyvideotrans's integration consumes (Wave 1.3) (#359)
pyvideotrans drives OmniVoice as a per-line clone backend (their
videotrans/tts/_omnivoice.py — being replaced upstream with a REST
integration against POST /generate). This contract suite pins the exact
multipart shape that integration sends (text + uploaded ref_audio +
ref_text + language name + num_step/guidance_scale/speed/denoise/
postprocess flags -> audio/wav with X-Audio-Duration) so a /generate
change that would silently break the 17.9k-star upstream fails our CI —
the engine-compat constraint extended to an external consumer.

Engine stubbed; validated in CI (local torch/Triton segfault on
main-importing tests, see project memory).

Spec 11 / parity program Wave 1.3 (our-repo half).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 23:06:49 +05:30
Palash DebnathandClaude Fable 5 9162f2b9e7 feat(stream): sentence-by-sentence /ws/tts via ported chunker (Wave 1.4) (#358)
Ports Patter's SentenceChunker (MIT, attribution header) behavior-identical
— all 61 upstream golden parity scenarios ship as fixtures and pass,
including documented quirks (current_behavior xfail semantics mirrored from
their parity runner). Terminator tables carry functional CJK; file added to
the test_no_hardcoded_cjk allowlist per convention.

/ws/tts now splits the request into sentences and synthesizes each in turn,
streaming the first sentence's PCM while later sentences are still
generating — the time-to-first-audio win on multi-sentence input.
Single-sentence requests behave exactly like the old single-shot path;
'start' metadata still waits for the first generation so lazy-loading
engines report their true sample rate. Italian comma-decimal guard
hard-disables aggressive first-clause flush per upstream.

Spec 8a (docs/competitive-analysis.md) / parity program Wave 1.4.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 23:04:00 +05:30
Palash DebnathandClaude Fable 5 454affb6e9 feat(tts): unlimited-length generation — sentence-boundary chunking + crossfade (Wave 1.2) (#357)
Ports voicebox's chunked TTS (MIT, attribution header) with two deliberate
changes: the concat half is reworked for torch tensors (matching what our
inference helpers feed the effect chain, incl. multi-channel on the last
axis), and the sample rate comes from the engine's declared rate instead
of the first chunk (fixes a latent upstream bug).

Long text (> max_chunk_chars, default 800) splits at sentence boundaries
(abbreviation/decimal-aware, bracket tags atomic, fullwidth enders via
unicode escapes for the CJK gate) -> per-chunk generation with
deterministic seed variation (seed+i) -> linear crossfade join (default
50 ms, 0 = hard cut) -> effect chain + watermark once on the joined audio.
Wired into BOTH inference paths (OmniVoice-native _run_inference and the
engine-adapter _run_backend_inference) beside the existing [pause]
stitcher; [pause] inputs keep their dedicated path. Short text is
byte-for-byte the old single-shot path; max_chunk_chars=0 disables.

New /generate form params: max_chunk_chars (>=0, default 800),
crossfade_ms (0-1000, default 50).

Tests: 15 model-free unit tests (split priorities, abbreviation/decimal/
tag guards, crossfade math incl. multichannel + clamping) + 3 stubbed-
engine endpoint tests (long text fans out with no words lost, short text
single-shot, 0 disables). Endpoint tests validated in CI — this machine
has a pre-existing local torch/Triton segfault on any main-importing test.

Spec: voicebox deep dive 1 / parity program Wave 1.2 / #346
unlimited-length item.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 23:03:56 +05:30
Palash DebnathandClaude Fable 5 93723c2789 feat(dictation): collapse Whisper hallucination loops in final transcripts (Wave 1.1) (#356)
Deterministic pre-pass ported from voicebox (MIT, attribution header):
word-level (token repeated >=6x, punctuation-normalized) + character-level
(2-60-char unit repeated >=6x, catches multi-word and no-space-script
loops). Rhetorical repeats below 6 survive; no LLM involved; identical on
every platform. Applied to the FINAL text in /ws/transcribe and POST
/transcribe — segments keep raw recognition so timings stay truthful.

Phase 1 of Spec 3 (docs/competitive-analysis.md); the optional local-LLM
refinement pass (phase 2) lands with parity program Wave 2.1 in the same
module.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:30:57 +05:30
Palash DebnathandClaude Fable 5 7422f20a63 feat(profiles): consent-locked voice profiles — verified_own_voice + spoken consent flow (Wave 0.2) (#354)
* feat(profiles): consent-locked voice profiles — verified_own_voice + spoken consent flow (Wave 0.2)

A profile becomes 'verified own voice' when its owner records themselves
reading a consent statement (spoken attestation, not a checkbox). Agentic
features and gallery sharing will gate on the flag; plain local synthesis
never does.

- alembic 0003 (additive, PRAGMA-guarded, downgrade supported) +
  _BASE_SCHEMA columns: verified_own_voice, consent_text,
  consent_audio_path, consent_recorded_at
- POST/DELETE /profiles/{id}/consent — stores the recording as provenance
  in VOICES_DIR ({id}_consent.*), replaces on re-record, cleans up on
  revoke and on profile delete; 422 on empty statement / too-short audio
- VoiceProfile page: Verified badge + Voice ownership panel (record via
  the existing useRecording denoise flow, revoke with confirm); en.json
  keys only (other locales fall back per the advisory i18n parity policy)

Spec: docs/competitive-analysis.md Action 22 / parity program Wave 0.2.
Prerequisite for agentic v2/v3 and the persona gallery.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(profiles): harden consent paths against py/path-injection; drop lifespan in tests

- _voices_path(): resolve DB-stored filenames strictly inside VOICES_DIR
  (bare-filename check + realpath containment); extension whitelist on the
  uploaded consent filename (fallback .wav) so a crafted filename can never
  steer the on-disk path. Applied to write, re-record cleanup, revoke, and
  profile-delete cleanup. New test: malicious upload filename falls back.
- Test fixture no longer runs the app lifespan: startup/shutdown touched
  module-level asyncio primitives bound to another module's event loop,
  making the suite order-dependent in full-suite CI. init_db() is called
  directly; endpoints under test need only the schema.

Fixes the CodeQL (3x py/path-injection high) and full-suite event-loop
failures on PR #354.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:10:30 +05:30
Palash DebnathandClaude Fable 5 1195b4e0dd test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)
Ports Patter's eval harness (MIT, attribution headers) into tests/evals/
with the judge transport swapped to services/llm_backend.py — the judge
runs against whatever local Ollama/LM Studio/OpenAI-compat endpoint the
user configured, keeping local-first. Both Patter hardening details kept
verbatim: verdict recomputed locally from the score (hallucinated
'passed: true' at score 0.2 fails), and tolerant JSON parsing (fences
stripped, invalid JSON -> fail-with-reasoning). Per-case containment:
agent exceptions keep the partial transcript and still judge it; a judge
failure records score 0 instead of aborting the suite.

HARD RULE preserved: LLM judges never gate CI. The scheduled workflow
(weekly + dispatch) is continue-on-error with the JSON report as artifact;
run_evals.py exits 0 always and skips cleanly when the active LLM backend
is 'off'. Deterministic probe judges remain the only gates; the harness
unit tests (10, no LLM needed) do run in gating CI.

First suite: dub translation naturalness v1 (4 cases) driving the real
cinematic_refine_sync reflect+adapt chain. The telephony-specific
session/assertions layers were deliberately not ported. The
dictation-refinement suite lands with Wave 1.1/2.1.

Spec: docs/competitive-analysis.md Spec 9b / parity program Wave 0.3.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 22:00:30 +05:30
Palash DebnathandClaude Fable 5 11c498eeb5 ci(docs): daily docs-drift job — canonical inventory vs README/docs/registries (Wave 0.1) (#353)
docs/features.yaml is the curated single source of truth (12 features,
11 TTS + 7 ASR engine ids, required install docs). scripts/check-docs-drift.py
diffs it against README.md, docs/, and the engine registries — parsing
registry keys from source so the CI runner never imports torch. The daily
workflow updates ONE rolling 'docs-drift' issue in place and auto-closes it
when clean (pattern adapted from Patter, MIT). Self-test includes a
real-repo-is-clean gate, so any PR that changes engines/features without
updating the inventory fails CI too.

Spec: docs/competitive-analysis.md Spec 9a / parity program Wave 0.1.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 21:02:07 +05:30
Palash DebnathandClaude Fable 5 73de4f9277 docs(specs): ElevenLabs-parity program — waved implementation plan from #346 + #345 (#349)
Turns the discussion #346 roadmap and the competitive-analysis research (#345)
into an executable program of small PRs: 6 waves, dependency-aware, each item
citing its Spec/§R section with effort and acceptance criteria. Accounts for
Smart Fit Phase A (#347), the timeline editor (#348), and Scalar (#307) having
already shipped. Telephony explicitly deferred behind guardrails + two spikes.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 19:22:43 +05:30
Palash DebnathandClaude Fable 5 eea2053a5e docs: competitive analysis v2 — second-tier landscape, source deep dives, action specs, market sentiment (#345)
* docs: expand competitive analysis — second-tier landscape, deep dives, action specs, market sentiment

Second research pass over PR #339's analysis (six parallel agents):
- Second-tier landscape: 13 projects surveyed, 7 profiled; KrillinAI/KlicStudio
  promoted to direct-competitor status
- Source-level deep dives: voicebox + Patter (MIT, portable briefs) and
  pyvideotrans (GPL, clean-room functional specs incl. the full _rate.py
  decision tree with verified constants)
- pyvideotrans's OmniVoice integration verified broken (Gradio /_clone_fn vs
  our FastAPI :3900) — Action 11 reframed as fix-the-bridge
- Implementation specs mapping all ranked actions onto our codebase
- User-sentiment + market-positioning research (issue clustering, ElevenLabs
  pricing pressure, honest verdicts on our five differentiators, name-collision
  risk, four positioning moves)
- Three stale matrix grades corrected (docs-drift CI, eval harness, MCP)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: ground the #346 roadmap in research — agentic voice, remote GPU, audiobooks, persona gallery, model/env management

Third research pass (four agents + five verification sub-agents) adding a
'Roadmap directions' section that maps every item from discussion #346 to
either an existing spec or new research:

- Agentic voice workflow: pipecat (BSD-2) as the license-clean in-process
  runtime; honest telephony constraints (no local PSTN path — opt-in carrier
  creds only); FCC/TCPA, Texas SB 140, ELVIS Act, EU AI Act Art 50
  (2026-08-02, OSS exemption does not cover it); six concrete guardrails;
  v1/v2/v3 scope ladder
- Remote GPU/Tailscale/remote API: base-URL + bearer-token consensus pattern;
  175k-exposed-Ollama cautionary tale; Tailscale rung (a) docs-only; vLLM
  drop-in for llm_backend; Scalar already shipped (#307), remaining work is
  OpenAPI hygiene
- Audiobook creator + persona gallery: ACX technical-spec mastering bar;
  ebooklib/PyMuPDF/mobi AGPL/GPL parser traps with clean alternatives;
  unoccupied consent-aware-gallery territory; .ovsvoice portable format
- Model/env + GPU compat: uv link-mode dedupe math (measured wheel sizes);
  two-dimensional (torch x cuda-variant) -> sm_XX compat matrix; HF cache as
  single source of truth (hf cache ls/rm/verify); preflight gate + loud
  CPU-fallback banner vs the Ollama/voicebox silent-fallback antipattern
- Eight consolidated new actions (15-22)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:35:52 +05:30
65fc5245dc feat(dub): timeline segment editor — drag, snap-to-onset, keyboard a11y (#280) (#348)
* feat(dub): full-track speech-onset detection + GET /dub/onsets/{job_id} (#280)

detect_speech_onsets() lists every speech rise across the track (frame RMS,
adaptive threshold, 150ms hysteresis) — powers the timeline editor's
snap-to-onset ticks. Route prefers the Demucs vocals stem, falls back to the
mix, and caches onsets.json per job (mtime-invalidated).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(dub): timeline editor math core — windowing, snap, clamp, fingerprint-safe commit (#280)

Pure helpers for the segment track: binary-search windowing, snapTime with
deterministic ties, neighbour/min-duration clamps with Alt-overlap (<=200ms),
commitMoveResize with fingerprint parity (move touches only start/end; resize
sets speed exactly like the old Regions handler and DELETES the key at 1.0 so
_canon_value's missing-vs-1.0 hashing can't mark untouched segments stale),
and overlap detection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(dub): SegmentTrack editing lane replaces the Regions plugin (#280)

Custom DOM segment boxes (6px edge handles, body-drag move, speaker colors,
stale/fresh tint, hatched overlap warning) virtualized by time over a single
{pxPerSec, scrollLeft} alignment source read off WaveSurfer's wrapper.
Snap-to-onset ticks on a viewport-sized canvas light up in snap range;
Ctrl/Cmd-wheel zooms centered on the cursor; double-click plays the slot via
playRange (timeupdate watcher pauses at slot end). Roving-tabindex listbox
keyboard model (arrows / Enter / Shift / Alt / Delete / S) with polite
aria-live announcements. WebKit fallback keeps a self-scrolling lane at a
fixed px/sec. timeline.* strings translated in all 21 locales.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(dub): wire timeline editor — per-gesture undo, id fix, table selection sync (#280)

segmentMoveResize() pushes undo ONCE per gesture (drag commits on pointerup;
keyboard nudges coalesce per focus session) and matches by String(id) — the
old parseInt('seg-3_a') path edited the wrong segment after a split. Commits
go through commitMoveResize for fingerprint parity, and the existing
recomputeIncremental effect picks up every commit. Clicking a timeline box
scrolls + highlights its row in DubSegmentTable; 'preview dub here' parks
the player at the slot start, then synthesizes the line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dub): inline the onsets-cache containment guard — CodeQL can't track helpers

Same lesson as #328/#329: the realpath+startswith sanitizer must sit at
the sink, not behind a function return. Unused helper removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:21:07 +05:30
4b21f82619 feat(dub): Smart Fit timing strategy — planner, fingerprints, generate path (phase A) (#347)
* feat(dub): Smart Fit planner, fit fingerprints, shared ffmpeg stretch helpers

- services/fit_planner.py: pure, I/O-free planner for dub-length fitting
  v2 — slack absorption (gap guard), audio-only band (<=1.2x), geometric
  50/50 audio/video split capped at 1.5x / 2.0x, residual overflow
  accounting, and a stretch_video-compatible video_plan + fitted timeline
  cursor. Clean-room reimplementation from a published description.
- services/incremental.py: fit_fingerprint() over the fit params with the
  same _canon_value canonicalisation as segment hashes (#281 class).
  Fit params stay OUT of segment_fingerprint — a fit change re-mixes,
  never re-TTSes.
- services/ffmpeg_utils.py: move _atempo_chain/_pitch_preserving_stretch
  out of the dub_generate router (lazy torch/numpy imports) so the Phase B
  export pipeline can reuse them; add probe_duration() ffprobe helper.
- schemas/requests.py: timing_strategy gains "smart_fit"; optional
  fit_options knob overrides default server-side.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(dub): smart_fit branch in the generate path

TTS loop unchanged (dur_s=None, natural-rate WAVs on disk). After the
loop, plan_fit() decides per segment; the mix loop applies audio_rate via
the pitch-preserving atempo pipe (linear-interp fallback), trims residual
overflow with the existing fades, and places audio at the planned
new_start on a fitted-length canvas. Truthful fit_status entries
(audio_rate / video_ratio / overflow_s) feed the row badges.

Persists job["fit_plans"][lang] = {plan (exact
_build_video_stretch_filter_graph shape), fitted_segments (cue times from
ACTUAL stretched sample positions), total/orig duration, params, fit_fp}
and mirrors fit_fp on dubbed_tracks[lang]. video_stretch_plans untouched.

Strategy-transition guard: job["seg_wav_kind"] records whether on-disk
seg WAVs are natural or slot-squeezed; a smart_fit partial regen over
slotted (or unknown) WAVs forces one full regen instead of
double-compressing. Old strategies and old persisted jobs are
byte-identical (all new reads via .get()).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ui): Smart Fit option in the dub timing picker (all 21 locales)

- prefsSlice: TimingStrategy union gains 'smart_fit'; optional FitOptions
  overrides (null by default — backend defaults apply identically on
  every platform); persisted alongside timingStrategy.
- DubTab: Segmented gains Smart Fit with i18n label + tooltip.
- useDubWorkflow: sends fit_options only when set and strategy is
  smart_fit. Default strategy stays 'concise' — no default behaviour
  change on any platform.
- locales: dub.timing_smart_fit{,_title} translated in all 21 languages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(dub): fit planner unit + golden suites, smart_fit generate-path integration

- test_fit_planner.py: threshold boundaries (0.9/1.0/1.2/1.21/4.0), cap
  saturation -> overflow, slack absorption incl. gap guard, last-segment
  tail, cursor monotonicity, allow_video_retime=False, video_plan fed
  straight into _build_video_stretch_filter_graph, fit_fingerprint
  canonicalisation (int vs float, omitted vs default — the #281 class)
  and a pinned stable digest.
- tests/fixtures/fit_planner/*.json: 4 golden FitPlans; algorithm drift
  is a deliberate fixture diff, never a silent change.
- test_smart_fit_generate.py: hermetic end-to-end runs (mock TTS, no
  ffmpeg) covering audio-only stretch, hybrid timeline growth +
  persisted plan shape, fit_options override, strict_slot->smart_fit
  forced regen then zero-TTS fit-only re-mix, and concise back-compat.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(competitive): dub-length fitting row reflects Smart Fit Phase A

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(incremental): mark fingerprint hashes usedforsecurity=False — dedup keys, not security (Bandit)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:20:45 +05:30
4288863f50 docs: model-source support policy — verifiable public sources only (#310) (#344)
* docs: model-source support policy — verifiable public sources only

Owner decision (issue #310): the local-loading mechanism stays, but
official support covers only models from verifiable public sources
(HF repos, official releases with license + checksums). Privately
distributed / paywalled model files are use-at-your-own-risk; never
run bundled executables. Mirrored in SECURITY.md as a supply-chain
note. Per the docs-sync rule, shipped alongside the policy decision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: firm up model-source policy — open, public, verifiable only; no private/paid models

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:29:16 +05:30
2574fccaf6 docs: community docs refresh — README, CONTRIBUTING, SECURITY, SUPPORT, Docker/macOS install (#341)
* docs: refresh community docs to match the project's current reality

- README: download badges now point to releases/latest (were frozen at
  v0.2.7); Intel-Mac note (pre-built bundle is Apple Silicon; source
  works on Intel; pre-built Intel tracked in #279)
- SECURITY: supported-versions table 0.2.x -> 0.3.x + 0.2.7 legacy row
- docs/install/docker.md: tag mapping matches docker.yml after #338 —
  :latest is the rolling main preview, :stable (new) pins releases
- PR template: removed the abolished two-RC/48h-soak ceremony; documents
  continuous-to-main
- CONTRIBUTING: new sections — what bot review looks like (CodeRabbit +
  Greptile), conventional-commit + issue-link expectations, the quality
  gates (cross-platform parity, 21-locale i18n + CJK allowlist, alembic,
  engine back-compat, local-first, loopback security posture), and a
  contribution-licensing grant that keeps the AGPL + commercial
  dual-license viable
- SUPPORT.md: new — channels, before-you-file checklist, expectations
- docs/install/macos.md: Intel caveat aligned with reality

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: codify the docs-sync hard rule — behavior changes update their docs in the same PR

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(agents): rtk rules for Antigravity — token-compressed tool output

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 13:58:43 +05:30
101cf2a6e7 ci(release): reinstate macOS Intel (x86_64) build target on macos-15-intel (#342)
Intel MacBook users had no installable artifact: the release matrix only
built aarch64-apple-darwin, and Rosetta 2 cannot run arm64 apps on Intel
(it only translates the other direction) — the rationale in the old
"Intel dropped" comment was backwards. Refs #279.

- Add a native `macos-15-intel` matrix leg (GitHub's designated x86_64
  migration target after macos-13 retired Dec 2025; standard image,
  supported through Aug 2027) building --target x86_64-apple-darwin
  with app,dmg,updater bundles.
- Existing per-TRIPLE steps already carry x86_64-apple-darwin cases
  (uv sidecar tar.gz, evermeet.cx ffmpeg/ffprobe — x86_64 Mach-O,
  natively correct on Intel), so the leg flows through the same
  Bundle/Build/Smoke/Verify steps untouched.
- The PR #290 signing path applies automatically: ad-hoc seal from
  tauri.conf.json signingIdentity "-", opt-in APPLE_* stable signing,
  and scripts/verify-macos-signing.sh both gated on runner.os == macOS.
- tauri-action includeUpdaterJson merges the new darwin-x86_64 platform
  key into latest.json alongside darwin-aarch64, so Intel installs
  auto-update on both Stable and Preview channels.
- docs/install/macos.md: table telling users which DMG (aarch64 vs x64)
  matches their Mac, and the from-source fallback for old releases.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 13:58:28 +05:30
acb7c90083 fix(tts): pin cudagraph-compiled model inference to one dedicated thread (#315) (#343)
torch.compile(mode="reduce-overhead") captures CUDA graphs whose state is
thread-local (torch/_inductor/cudagraph_trees keys its tree manager off the
capturing thread). The _gpu_pool ThreadPoolExecutor runs up to 4 workers, so
the first render captured the graph on worker A and a later render dispatched
to worker B replayed mismatched cudagraph state — silently corrupting the
audio (static noise + slowed playback from the second render onward, no
exception, so the #327 eager fallback never fired).

Fix: when the model is compiled with a cudagraph mode, wrap model.generate
(the same single choke point #327 uses) so every call hops to a dedicated
1-thread "compiled-infer" executor — capture and replay always happen on the
same thread, deterministically. A thread-ident re-entrancy guard runs inline
when already on that thread (a 1-worker executor submitting to itself would
deadlock). Installed after the #327 fallback wrapper, so the eager retry path
also runs on the dedicated thread.

No behavior change for CPU / MPS / Windows-no-Triton / compile-disabled
paths: should_torch_compile() gates exactly as before and uncompiled models
keep the full pool.

Closes #315

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 13:58:21 +05:30
Palash DebnathandClaude Opus 4.8 948bc76543 macOS: ad-hoc sign so users open without Terminal + signing/notarization verification (#290)
* chore(release): add macOS signing/Gatekeeper/notarization verification

Codify and enforce the macOS build-signing requirements. The release
pipeline built bundles and had opt-in Apple signing, but never verified
codesign/spctl/notarization — unsigned or broken bundles could ship silently.

- scripts/verify-macos-signing.sh: runs codesign --verify --deep --strict,
  spctl Gatekeeper assessment, per-nested-Mach-O signature check, stapler
  validate, and (opt-in) notarytool history. Report-only by default (unsigned
  dev/preview is expected); --require-signed fails on any unsigned/un-notarized
  component so a broken release stops instead of publishing an unsigned artifact.
- scripts/macos-dev-unquarantine.sh: local-dev-only quarantine stripper, with a
  loud "never a substitute for notarization" warning.
- release.yml: new "Verify macOS signing" step on the macOS leg — report-only on
  unsigned paths, STRICT on the opt-in signed stable path (same condition as
  "Configure Apple signing"), so signing/notarization failures fail the job.
- docs/macos-signing-verification.md: the canonical 10-point requirements +
  how-to-verify checklist, cross-linked to docs/install/macos.md and DESKTOP_RELEASE.md.

Verified locally: report-only PASS (exit 0) and --require-signed FAIL (exit 1)
against the real unsigned debug .app; release.yml parses as valid YAML.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(macos): ad-hoc sign bundle so users open it without Terminal (no Apple ID)

The "app is damaged and can't be opened" error is caused by a broken/incomplete
code-signature seal (codesign --verify failed: "code has no resources but
signature indicates they must be present") on the quarantined download — there
is no GUI bypass for that variant on modern macOS, forcing users to run `xattr`.

Give the bundle a VALID ad-hoc signature at build time (free, no Apple Developer
account) via tauri.conf.json bundle.macOS.signingIdentity = "-". Verified through
a real `tauri build`: the produced .app is now flags=adhoc,runtime and passes
codesign --verify --deep --strict. A valid seal flips the Gatekeeper prompt from
the un-bypassable "damaged" to the GUI-bypassable "unidentified developer", which
users clear with right-click → Open / Settings → "Open Anyway" — no Terminal.

Still not notarized (that needs the paid Apple ID), so there's a one-time
confirmation rather than a clean double-click. The opt-in Developer-ID path is
unchanged: APPLE_SIGNING_IDENTITY (env) overrides the "-" default on the signed
stable release.

- tauri.conf.json: signingIdentity "-" (ad-hoc default).
- verify-macos-signing.sh: detect ad-hoc tier; report the no-Terminal GUI path
  in report-only, still FAIL it under --require-signed (production must notarize).
- docs/install/macos.md: lead the Gatekeeper section with right-click → Open;
  keep xattr as fallback for the harsher "damaged"/corrupted-download case.
- docs/macos-signing-verification.md: signing-tiers table + ad-hoc default note.
- release.yml: comment the ad-hoc default + env override on the signed path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 13:26:04 +05:30
3c780dced9 feat(dub): speech-onset alignment + regional dialect targeting (#280) (#330)
Items 1 and 2 from the improvement list:

1. Synchronization — Whisper-family ASR stretches segment starts back
   over leading non-speech (intro music, silence), so the dub starts at
   0:00 while the speaker starts at 0:02-0:03. New onset_align service
   snaps each segment start forward to the first audible vocal onset
   (adaptive RMS threshold over the Demucs-isolated vocals when
   available). Forward-only and conservative: never moves a start
   earlier, ignores sub-100ms shifts, preserves minimum duration,
   leaves silent-window segments untouched. Pure NumPy — identical
   across platforms.

2. Accent/vocabulary by country — a Dialect picker in the Dub panel
   (BCP-47 codes per target language) injects a regional instruction
   into LLM translation prompts (OpenAI/Ollama engines and the
   Cinematic refine pass): Argentina yields 'Vos sos muy listo', not
   'Tú eres muy listo'. Non-LLM engines show a clear hint that the
   dialect needs an LLM. New i18n keys translated in all 21 locales.

Item 3 (segment rectangles: move/crop/stretch on the timeline) is a
larger editor feature and stays open on #280.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: mergetest <test@local>
2026-06-11 13:09:40 +05:30
c0924f5eba fix(tts): torch.compile failures fall back to eager — generation never fails on unsupported GPUs (#278) (#327)
* fix(tts): torch.compile failures fall back to eager — generation never fails on unsupported GPUs (#278)

On GPU architectures the bundled Triton doesn't support (e.g. Blackwell
sm_120 / RTX 5060), the compiled model dies mid-generation inside the
Dynamo/Inductor/Triton/cudagraph stack — previously surfaced as a fake
'ran out of memory' error and a dead Archetype preview. Now:

- up-front arch gate: skip compile when the GPU's compute capability is
  not in this torch build's arch list (OMNIVOICE_FORCE_TORCH_COMPILE=1
  overrides for PTX forward-compat setups)
- runtime fallback: model.generate is wrapped once; a compile-stack
  failure (classified by exception chain: module, message, traceback
  paths — the cudagraph case is a bare AssertionError) logs a warning,
  restores the eager module, disables compile for the session, resets
  dynamo state, and retries eagerly. Non-compile errors propagate
  unchanged.
- the /generate OOM handler no longer mislabels compile crashes as OOM
  and points users at the actual remedy.

Fixes #278

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Potential fix for pull request finding 'CodeQL / Empty except'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Empty except'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Update backend/api/routers/generation.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: mergetest <test@local>
2026-06-11 13:09:18 +05:30
e2027c1291 ci(security): never cancel main scans — merge trains left red ✗ on every intermediate commit (#340)
PR branches keep cancel-in-progress (superseded scans are wasted work).
On main each commit gets its own concurrency group, so a burst of merges
runs every scan to completion instead of cancelling all but the last —
'cancelled' renders as a permanent red ✗ in the commit history even
though nothing failed.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:54:33 +05:30
Palash DebnathandClaude Fable 5 a949b2c78a chore(version): main is always latest release + 1 — rule, bump to 0.3.6, Docker retag, auto-bump job (#338)
Versioning hard rule (owner-set 2026-06-11), codified in CLAUDE.md:
- main's three version sources (tauri.conf.json, Cargo.toml,
  pyproject.toml) always carry last release + 1 patch; bumped 0.3.5 ->
  0.3.6 now.
- Preview builds stamp BASE-N which now sorts ABOVE the last stable
  (0.3.6-N > 0.3.5) — the updater ordering becomes natural and the
  Windows MSI ProductVersion wrinkle disappears.
- Docker: :latest = rolling main preview; :stable + :X.Y.Z + :X.Y =
  tagged releases. workflow_dispatch still only emits throwaway :sha-.
- release.yml gains a version-bump job: on every stable v* tag it
  bumps main to the next patch automatically.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:16:03 +05:30
853b9eefc7 fix(dub): burn translated subtitles, fix subtitle save JSON error (#309) (#328)
* fix(dub): burn translated subtitles, fix subtitle save JSON error (#309)

Two symptoms, one root: the job kept the original-language ASR transcript
while the editor only sent translated/edited text in the generate request.

- dub_generate now persists the segments the dub was actually generated
  from back onto the job (metadata carried over by stable id, fallback
  index; text_original retained for dual-subtitle layouts) — SRT/VTT
  export and ffmpeg burn-in now render the dub language, not the source.
- The SRT/VTT export endpoints honor the save_path query param the Tauri
  save dialog appends (like every other export) and return the standard
  JSON envelope — previously they ignored it and returned the raw body,
  so the frontend's JSON.parse choked on the SRT cue index ('Unexpected
  non-whitespace character after JSON').
- Frontend guards the save response content-type so any future raw-body
  response surfaces as a clear error.

Fixes #309

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Potential fix for pull request finding 'CodeQL / Uncontrolled data used in path expression'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(dub): use the file's established realpath+startswith containment idiom (CodeQL)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dub): write subtitle saves from the Tauri process, not the backend (#309)

The backend save_path variant on /dub/srt and /dub/vtt routed a
user-controlled destination through the loopback HTTP surface — six new
CodeQL path-injection flows plus two log-injection flows. Subtitles are
small text bodies, so the frontend now fetches them raw and writes the
file via a new save_text_file Tauri command: the OS save dialog in the
trusted process is the write authorization, and the backend never sees
a destination path. Binary exports keep the established save_path flow.
Also strips newlines from user-derived values in the two flagged log
lines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dub): leave _native_save byte-identical to main

The newline-strip on the log line moved a path sink onto a changed line,
which made CodeQL re-attribute the long-standing binary-export flow to
this PR as a new alert. The subtitle endpoints no longer feed this
function at all, so restore the exact original line — the baseline alert
stays baseline, and hardening pre-existing flows belongs in its own PR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-06-11 12:15:53 +05:30
1ed22af6ca docs: competitive analysis — voicebox, pyvideotrans, Patter (feature matrix + ranked adoption plan) (#339)
* docs: competitive analysis — voicebox, pyvideotrans, Patter

Feature matrix vs our self-inventoried maturity grades, license-aware
reuse verdicts (MIT = port with attribution, GPL-3.0 = reimplement only
— copied GPL files would break the AGPL + commercial dual-license), and
an 11-item ranked action plan with effort estimates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: append Chatterbox engine evaluation to the competitive analysis

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: mergetest <test@local>
2026-06-11 12:15:44 +05:30
Palash DebnathandClaude Fable 5 d0517fdb87 chore(review-bots): diagrams + ASCII UI sketches in every PR walkthrough (#337)
* chore(review-bots): visual walkthroughs — diagrams for mechanics, ASCII sketches for UI

CodeRabbit: enable sequence_diagrams explicitly and instruct the
high-level summary to sketch UI changes as compact ASCII before/after
and behavior changes as a small mermaid flow. Greptile: new repo-level
greptile.json turning on the sequence-diagram and summary sections with
matching instructions, plus the project's local-first and cross-platform
hard rules so both bots review against them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(review-bots): expert-panel review rubrics, pre-merge rule audits, knowledge base

Encode one senior-domain-expert lens per subsystem (ML inference for
backend/services, product frontend for src, desktop systems for
src-tauri, test infra for tests) as path instructions; add non-gating
pre-merge checks for the project's four hard rules (cross-platform
default parity, 21-locale i18n completeness, local-first guarantee,
backward compatibility); feed CLAUDE.md and docs into CodeRabbit's
knowledge base; mirror it all in greptile.json with customContext rules
and strictness tuning.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:15:38 +05:30
668d824e86 feat(setup): unified first-run journey — install gate, studio-console wizard, platform awareness (#295)
* feat(setup): first-run install gate — nothing installs until the user confirms a plan

New `setup` module parks first runs in BootstrapStage::AwaitingSetup instead
of auto-installing. complete_setup validates the user's InstallPlan and only
then starts the existing bootstrap:

- install modes: installed (platform dirs) / portable (one folder next to
  the exe / AppImage, config.json travels with it)
- user-chosen storage: env dir, data dir (OMNIVOICE_DATA_DIR), model cache
  (OMNIVOICE_CACHE_DIR) — None = legacy default, byte-identical behavior
- minimum-space gate: per-volume free-space check (fs4 statvfs), grouped by
  filesystem so dirs sharing a disk sum their requirements; install refused
  when short (9 GiB env + 7 GiB models + 1 GiB data, measured + headroom)
- custom mirrors (PyPI index, HF endpoint, python-build-standalone) take
  precedence over region presets in the venv/sync/backend env wiring
- ROCm torch variant selectable via config (env var still wins)
- existing installs migrate silently: venv present → setup_complete=true,
  no questions re-asked; dev trees skip the gate entirely

19 unit tests (disk probing, space grouping, mirror validation, legacy
config compat).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): first-run setup screen — mode, storage with space gate, mirrors, compute

FirstRunSetup renders when the Rust side reports awaiting_setup (lazy-loaded;
regular launches pay nothing). One screen, defaults all work:

- language picker first (rest re-renders translated), 21 locales shipped
- Installed / Portable mode cards (portable disabled with reason when the
  exe-adjacent folder isn't writable)
- storage rows with live per-path free-space probes (debounced
  check_install_target), 'needs ~X / Y free' readouts, folder pickers
- client mirrors the Rust per-volume space gate: Start installation is
  disabled with an explicit reason until every volume fits
- compute (CUDA-auto / ROCm), update channel, region + custom mirror URLs
- complete_setup errors surface inline; on success the normal bootstrap
  progress UI takes over on the next status poll

Verified on a wiped machine: gate parks (no spawn, no downloads), screen
renders, 450 GB ≥ 17 GB requirement → Start enabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): studio-console redesign of the first-run screen

The setup screen now reads as powering on studio hardware rather than a web
form — true to a voice studio, and self-sufficient offline (every font and
asset is bundled; a first run may be on a restricted network):

- breathing waveform masthead (CSS-only, deterministic speech-cadence
  silhouette, staggered per-bar delays)
- Source Serif 4 display headline + engraved IBM Plex Mono panel labels +
  Inter body — the three faces the app already ships
- rack-unit panels with corner screws, engraved title rules, serial plate
  (OVS · vX.Y.Z)
- disk space as segmented LED capacity meters: lit = what the install
  consumes, alarm-blink red on insufficient volumes
- mode cards with indicator LEDs; 'armed' Start button — LED lights and a
  halo pulses only once every volume passes the space gate
- atmosphere: corner accent glows + SVG film grain; staggered rise-in
  choreography on load
- all motion transform/opacity only; prefers-reduced-motion holds every
  frame still; theme-token derived colors; focus-visible rings throughout

No logic changes: same IPC calls, same i18n keys, same space-gate math.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): wide desktop deck, hardware-aware Compute + Update channel cards

Three pieces of feedback addressed:

- width: the console is now a 1240px two-column deck (storage rail left,
  decision rail right) that uses desktop real estate; collapses to one
  column under 980px and stacks fully under 620px
- no outer chassis box: panels float directly on the atmospheric backdrop,
  each carrying its own rack-unit treatment
- Compute and Update channel split into separate cards with real
  information: get_setup_state now detects hardware (nvidia-smi → CUDA
  name, /sys/class/drm vendor 0x1002 → AMD/ROCm, Apple Silicon → MPS,
  CPU cores + RAM via sysinfo; best-effort, never blocks) — the Compute
  card shows a live 'Detected: …' readout, badges the option that matches
  the machine, and pre-selects ROCm on AMD boxes; both cards use LED
  radio options with full descriptions (6 new i18n keys × 21 locales)

Also pins playwright-core as an explicit devDep — bun did not materialize
it through @playwright/test, breaking programmatic browser use.

20/20 Rust tests · vite build · CJK guard green. Verified live (gate
engaged, responsive single-column) and at 1600×1000 via mocked-IPC
browser shot (two-column deck).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): move network (region + mirrors) into the masthead with language

Language and download region are the two 'where am I' choices — they now
sit together top-right of the masthead, with the custom-mirrors disclosure
tucked beneath the subtitle. The Network panel is gone, leaving a balanced
deck: Install mode + Storage left, Compute + Update channel right.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): strip the boxes — fills and rules carry the structure

One design rule now: borders only where state demands them. Panels lose
their boxes entirely (engraved mono title + rule separates sections);
option cards, storage rows, selects/inputs, the hw readout, the version
plate and the ghost buttons are all flat fills; active options glow with
an accent tint + LED; blocked rows and errors use a red tint + 2px inset
edge bar instead of a border. The badge chip is fill-only too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): quiet pass — every element earns its visual weight

- waveform becomes a whisper: 22px trace, 2px bars, ~half opacity — an
  ambient signature instead of a billboard
- storage readouts collapse to one mono line ('needs ~9 GB · 449 GB free');
  the LED meter now appears only when it carries information (install
  would consume >35% of free space, or the volume is blocked) — at 449 GB
  free a bar was a meaningless sliver
- Change… buttons go text-quiet (transparent until hover)
- custom-mirrors disclosure right-aligns under the region select it
  extends, instead of floating under the subtitle
- version plate moves to the footer next to the disk total — the masthead
  keeps only title, subtitle, and the two locale/region selects

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): platform-matrix awareness — distro+arch detection, ROCm gated to Linux, no Windows console flash

The install matrix is OS family × distro × arch × GPU vendor, and the
setup screen now both shows it and only offers choices valid for it:

- HardwareInfo gains os_name (distro PRETTY_NAME from /etc/os-release on
  Linux, macOS/Windows elsewhere) and arch (x86_64/aarch64) — the detected
  line reads 'CachyOS x86_64 · NVIDIA RTX 4070 · 32×CPU · 31 GB RAM',
  exactly what bug reports cite
- SetupState gains os; the ROCm option renders on Linux only (wheels
  don't exist elsewhere) and complete_setup clamps rocm→auto on
  non-Linux as the server-side backstop
- nvidia-smi probe gets CREATE_NO_WINDOW on Windows — no cmd flash on
  the first screen a user ever sees
- Apple Silicon → MPS, Intel mac → CPU, ARM Linux → CPU: all matrix
  cells resolve through the same base constructor

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): unify the whole first-run journey under the studio-console system

Setup → Installing → Model wizard now read as one continuous experience:
the same atmosphere, whisper waveform masthead, serif/mono type, LED
language and quiet fills across all three acts.

- Installing (BootstrapSplash): rebuilt in frs-* — segmented LED journey
  meter (completed steps + live byte progress), LED step rail (done=green,
  active=pulsing accent, pending=dim), engraved ACTIVITY panel with the
  quiet mono log (collapse/copy as text-quiet actions), failure act with
  red-tint error + hints + armed Retry. All logic untouched: stage poll,
  event subscription + backfill, dedupe, hints, region/language selects.
- Model wizard (SetupWizard): same masthead with the step rail as engraved
  mono LED steps top-right, welcome cards as option-card surfaces,
  preflight as LED check rows (pass/warn/fail), frs nav buttons with armed
  primaries, embedded Model Store / Engines / Dictation panels scroll
  inside the act. Old 556-line stylesheet replaced by ~60 lines of glue;
  BootstrapSplash.css reduced to a resolving stub.
- FirstRunSetup.css is now the journey's shared design system (step rails,
  log panel, banners, hints, wizard chrome, check rows appended).
- 2 new strings (Installing / Activity) translated across all 21 locales.

Validated end-to-end on this machine: setup screen → Start installation →
real venv bootstrap (~10 min) → backend healthy on 3900 → model wizard.

20/20 Rust tests · vite build · CJK guard green · installing act verified
via mocked-IPC screenshot at stage=installing_deps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): --setup re-entry flag + make the install-plan screen un-stealable

The setup stage is first-run-only by design (completed installs skip it),
but it must be reachable on demand and must actually win the mount when
engaged. Three fixes:

- 'omnivoice-studio --setup' parks the bootstrap in AwaitingSetup on any
  launch — checked before the attach-to-healthy-backend shortcut, so a
  running backend can't skip past it
- App routing: awaiting_setup now outranks everything (a live backend
  answering /setup/status used to route straight to the model wizard);
  the wizard additionally requires stage === 'ready' so it can't mount
  during the initial stage race
- useBootstrapStage: a transient IPC miss no longer permanently declares
  'ready' (which killed the poll loop and silently skipped the setup /
  progress screens) — it retries up to 5 ticks before conceding

Plus journey-wide titlebar clearance (content never sits under the GTK
headerbar / macOS traffic lights / Windows controls) and drag-region
mastheads on all three acts.

Verified: mocked-IPC harness with stage=awaiting_setup + a LIVE backend
answering /setup/status renders the setup screen, not the wizard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(setup): remove backdrop decoration — flat surface, state-only emphasis

The corner accent glows and SVG film grain rendered as visible banding /
noise artifacts on many panels — both gone; the journey now sits on a
clean flat chrome background. Also swept the remaining decorative bloom:
the active option card drops its glow shadow (flat accent tint + LED carry
the state), and the armed Start button loses its pulsing halo (the lit LED
already signals actionable). Remaining shadows are functional micro-detail
only: 6px LED glows, meter track inset, red edge bars.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): journey rail + verbosity diet — clean, smooth, elegant

The setup page is now visibly stage 1 of the install flow: a quiet
breadcrumb rail (SETUP → INSTALLING → MODELS & ENGINES) sits between the
waveform and the headline on both the setup and installing acts, LEDs
marking done/active/pending — one continuous story across the journey.

Verbosity halved without hiding information:
- option descriptions unfold (260ms ease) only on the selected card; the
  page shows exactly one explanation per group, collapsed cards keep the
  text as a tooltip
- storage rows drop their always-on caption (label + path + readout +
  Change… on one line; caption lives in the row tooltip)

The whole page now fits a laptop window without scrolling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): merge Models + Engines into one wizard act

Two tabs weren't necessary: models are the required gate, engines the
optional extras — now two stacked panels in a single 'Models & engines'
step (label reuses the journey-rail key, translated in 21 locales).
Wizard shrinks to 4 steps: Welcome → System check → Models & engines →
Dictation. Continue still gates on models_ready only; engines stay
optional. Welcome cards updated to the 3 remaining acts; static cards
keep their descriptions visible (the active-only fold is for radios).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(setup): wizard was skipped after first-run install — probe /setup/status on bootstrap ready

The models-needed probe started at mount with a ~30s retry ceiling. On a
first run, mount happens at the setup page — by the time the user reads
it and the multi-minute install finishes, the attempts were long burned,
so setupChecked landed as 'no wizard needed' and the studio rendered with
zero models on disk. The probe is now keyed on bootstrapStage and runs
when it hits 'ready' — the first moment a backend exists to answer.
Normal launches (backend up quickly) behave exactly as before.

Caught by running the full journey three times end-to-end: rounds 2–3
skipped Models & engines after install; with the fix the wizard mounts
with models_ready=false (Whisper large-v3 listed missing).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): drop the Welcome step — wizard opens on System check

The welcome act had nothing left to say: the journey rail names the
stages, the setup page already oriented the user, and the cards repeated
both. The wizard is now three steps — System check (auto-runs on mount) →
Models & engines → Try dictation — landing the user directly on live
preflight results instead of a page about the pages to come.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): true unified library — models + engines as ONE list

'Merge them' meant one list, not two panels stacked — fair criticism.
The wizard's Models & engines act is now a purpose-built WizardLibrary:
every installable is a row of the same grammar (LED · name · chip ·
size · action):

- required models lead (REQUIRED chip, Download action, live SSE
  progress bar + percent, green LED when installed) — they gate continue
- TTS engines follow (ENGINE chip): active engine glows accent,
  available ones offer one-click Use (selectEngine), heavy installs
  defer honestly to Settings ('install later in Settings' + reason
  tooltip)
- the optional-model tail folds behind 'Show N optional models'

The full management surface (search, HF token, deletes, sorting) stays
in Settings — a first run needs a checklist, not a store. 9 new strings
× 21 locales. Verified against the live backend via the browser harness:
required/installed/engine/active/Use/defer states all render in one list.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(diagnostics): local-first self-check, error journal, and bug-report pipeline (#296)

* feat(diagnostics): local self-check + scrubbed bug-report pipeline

Closes the gap between 'something broke' and 'a useful GitHub issue
exists' — entirely within the local-first constraint: the only outbound
path remains the user's own browser opening a prefilled issues/new URL.

Backend:
- core/scrub.py: privacy scrubber for anything leaving the machine —
  env-var secret values (*TOKEN*|*KEY*|*SECRET*|*PASSWORD*), credential
  shapes (hf_/ghp_/github_pat_/sk-), home dirs on all three OSes
- core/diagnose.py: 9-check self-check (device+GPU, ffmpeg, HF token,
  disk, data-dir writability, RAM, engine registry, hub reachability),
  pre-scrubbed, ASCII-safe output
- GET /system/diagnose + 'python main.py --diagnose' (exit 0/1)
- /system/info: hardware inventory (os_version, cpu_model, cpu_count,
  ram_total_gb, gpu_name, vram_total_gb, disk_free_gb), cached statics

Frontend:
- utils/bugReport.js: single source for the prefilled-URL builder —
  scrubText twin, hardware context capture, scrubbed error+stack embed,
  URL-length cap; ReportBugButton refactored onto it
- ErrorBoundary 'Report this bug' action with the error attached
- utils/errorToast.jsx toastErrorWithReport(); wired into export toasts
- Settings > About 'Run self-check' with per-check status badges

Tests: 27 pytest (scrub, diagnose) + 15 vitest (bugReport); existing
suites green; verified live (--diagnose, TestClient, vite build).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(diagnostics): error journal, diagnostic bundle, crash notice, global handlers

Second slice of the bug-tracking work — still zero outbound paths beyond
the user's own browser/file manager.

- core/error_journal.py: deduped ring of recent unhandled backend errors
  (fingerprint counts, error_class triage: GPU_OOM, HF_AUTH_FAILED,
  PYANNOTE_LICENSE_REQUIRED, DISK_FULL, FFMPEG_MISSING, NETWORK_ERROR),
  scrubbed, JSONL-persisted so the error that killed the last run survives
  restart. Wired into the global exception handler; 500 bodies now carry
  error_class; GET /system/errors/recent.
- core/diagnostic_bundle.py + POST /system/diagnostic-bundle + Settings >
  About 'Save diagnostic bundle': zip of self-check report, error journal,
  scrubbed log tails — drag onto a GitHub issue; bypasses the ~8k
  prefill-URL ceiling.
- crash-on-next-launch: /system/notifications flags a crash logged before
  this session started (size vs acked-size in prefs, mtime vs process
  start); POST /system/crash/ack; LogsFooter acks on action click.
- utils/globalErrorHandlers.js: uncaught errors + unhandled rejections get
  a throttled, noise-filtered 'Report this bug' toast.
- sidecar log parity fix: _tauri_log_candidates() now lists the Rust
  sidecar's backend.log/backend_err.log on Linux (XDG state dir) and
  Windows (LOCALAPPDATA) — sidecar crashes were only visible on macOS.

Tests: +19 pytest (journal, bundle); suite at 102 passed. Vitest 124
passed; vite build green. Live-verified: journal recorded and classified
a real HF 401 from the test run (HF_AUTH_FAILED, paths scrubbed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(diagnostics): breadcrumbs, deep self-check, report sweep, issue search

Final slice of the bug-tracking work.

- toastErrorWithReport adopted at the high-traffic failure sites: TTS
  generation, dub upload/ingest/transcribe, engine install, engines-matrix
  load, voice profile save/delete/test, batch enqueue/cancel/delete.
  Validation toasts and cancellations stay plain on purpose.
- utils/breadcrumbs.js: local-only ring of the last 20 action names
  (closed-set names only — never content or paths), embedded as a
  'Recent actions' section in the prefilled report. Instrumented: view
  changes, generate, dub pipeline, export, engine switch.
- deep self-check: /system/diagnose?deep=true and --diagnose --deep load
  the active engine and synthesize a short utterance (num_step=4) —
  catches 'installed but broken'. 180s time-box, skips during model load,
  scrubbed failure detail. Verified live: cold-loaded omnivoice and
  produced 2.2s of audio in 43.9s on CUDA.
- 'Search similar issues' action on the ErrorBoundary: scrubbed,
  noise-stripped GitHub issue search URL — dedupe before filing.
- bug_report.md template now points at the diagnostic bundle and the
  --diagnose CLI so manual reports arrive with the same evidence.

Tests: pytest 107 passed (4 new deep-check tests, CJK gate green);
vitest 218 passed (breadcrumbs + issue-search suites); vite build green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(diagnostics): self-diagnosis section in troubleshooting + README pointer

Settings > About self-check / --diagnose / --deep / diagnostic bundle are
now the documented first step before the per-error entries — and the
support team's first ask on every issue.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(setup): flush sticky action bar, global dbl-click maximize, open maximized

First-run polish on the studio-console journey:

- FirstRunSetup: fixed-footer / scrollable-middle layout — mast + decision grid
  live in a dedicated .frs__scroll region; the install action bar is the last
  flex item, so it sits flush at the window's bottom edge and nothing (e.g. an
  expanded compute-option description) can render beneath it on small windows.
- Double-click-to-maximize on the custom borderless titlebar now works on EVERY
  drag region (splash, first-run, wizard, main header) via one delegated
  listener in main.jsx, on all platforms; removed App.jsx's redundant inline
  handler so it doesn't double-toggle. Skips interactive controls in the bar.
- Window opens maximized to the available desktop size (tauri.conf.json).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(diagnostics): quiet Bandit on the journal hash and hub probe

The journal fingerprint is a dedup key, not a security boundary —
usedforsecurity=False. The hub reachability probe gets an explicit
https scheme guard on its constant URL so the urlopen sink is audited.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(setup): address PR #295 review findings — security, lifecycle, privacy, i18n

Security:
- setup.rs valid_mirror: reject plaintext http:// mirror URLs (MITM
  supply-chain path into UV_PYTHON_INSTALL_MIRROR / UV_INDEX_URL /
  HF_ENDPOINT); explicit http://localhost / 127.0.0.1 / [::1] exceptions
  only. Tests extended incl. loopback-lookalike hosts.
- setup.rs detect_hardware: AMD vendor ID alone no longer maps to
  kind="rocm" — a cheap ROCm userspace probe (/opt/rocm or rocminfo on
  PATH) gates it; bare AMD GPUs report kind="amd" so the UI offers ROCm
  without pre-selecting it ("matches this machine" only when verified).

Functional:
- lib.rs/setup.rs --setup re-entry: complete_setup now kills any backend
  still serving on the port before retry_bootstrap, so changed
  env/mirror/layout settings actually apply instead of re-attaching.
- setup.rs: nvidia-smi probe runs behind a 3 s timeout thread — a wedged
  driver degrades to CPU instead of hanging the first-run IPC.
- setup.rs: is_first_run is now a pure read; the existing-install
  migration write moved to migrate_existing_install_if_needed, invoked
  only from the bootstrap thread (get_setup_state no longer writes).
- setup.rs complete_setup: config save errors now abort setup and surface
  in the UI instead of bootstrapping into a stale on-disk layout.
- setup.rs complete_setup: logs default-vs-custom flags instead of the
  user's absolute env/data/models paths (privacy rule).
- scrub.py + bugReport.js: also redact forward-slash Windows homes
  (C:/Users/<name>, file:///C:/Users/...), ordered before the macOS
  pattern so "C:~" residue can't form. Tests added on both sides.
- bugReport.js: context fetches bounded by a 2.5 s AbortController
  timeout so report assembly degrades to partial context instead of
  hanging on a stalled backend.
- system.py: crash ack is now {size, mtime} (legacy size-only ack still
  honored) and /system/logs/clear drops the ack — truncation can no
  longer permanently suppress 'crash-last-session'.
- system.py: Linux Tauri-log probe honors XDG_DATA_HOME.
- setup.ts/WizardLibrary.jsx: SetupProgressEvent type now documents the
  full phase taxonomy actually emitted (per-file start/progress/done +
  install_*/delete_* lifecycle); reducer verified correct against the
  backend stream and annotated — a file-level 'done' must not clear the
  repo row.
- SetupWizard.jsx: step rail clamps to the highest unlocked step
  (preflight/models gates) — no more jumping straight to "Enter studio".

Polish:
- BootstrapSplash.jsx: Waveform heights wrapped in useMemo([bars]) like
  its siblings.
- BootstrapSplash.jsx: detectHints returns i18n keys (bootstrap.hint_*)
  rendered through t(); translated in all 21 locales.
- SetupWizard.jsx: step rail aria-label localized (setup.step_aria /
  setup.step_completed) in all 21 locales.
- FirstRunSetup.css: deprecated word-break: break-word → overflow-wrap:
  anywhere; reduced-motion override also stops the frs-hw-pulse LEDs
  (.frs-step.is-active LED + .swiz-lib__led--busy).

Deferred (design-level, follow-up PR): --setup re-entry round-tripping of
custom dirs/mirrors into the form (setup.rs), and worker-thread leak on
timed-out deep checks (diagnose.py).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(i18n): translate back-filled keys in all 20 locales, drop inline fallbacks

The reconciliation merge back-filled 16 new keys (about.self_check*,
about.*bundle*, dub.num_speakers_*, errors.*) with English text in
every non-English locale — CodeRabbit flagged 9 locales; fixed all 20.
Interpolation tokens preserved and asserted during the rewrite. Also
removed the two inline English fallback strings in App.jsx
(firstrun.first_sound_*) so copy lives only in locales/*.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: mergetest <test@local>
2026-06-11 12:15:28 +05:30
Palash DebnathandClaude Fable 5 7419986c8b fix(dictation): microphone permission — OS usage descriptions, WebView grant handler, actionable denied-state UI (#323) (#336)
On Windows 11 the dictation pill (Ctrl+Shift+Space) always reported
"Microphone access denied" even though OS-level mic permission was
granted (Voice Clone worked, backend transcribed fine). Root cause:
no WebView2 PermissionRequested handler was registered, so WebView2
fell back to its own permission UI — which the 300x64 transparent,
undecorated, deliberately-unfocused pill window can never host — and
getUserMedia() rejected with NotAllowedError.

Per-platform fixes:
- Windows (WebView2): register a PermissionRequested handler on both
  the main and widget webviews that allows microphone/camera requests
  in code, for the app's own origin only (tauri.localhost + dev
  loopback). The Windows privacy toggle still applies on top.
- Linux (WebKitGTK): the media-stream enable + permission auto-grant
  previously covered only the "main" window — the dictation widget is
  a separate WebView and was silently denied. Now applied to both.
- macOS: already correct — NSMicrophoneUsageDescription ships in
  src-tauri/Info.plist and wry grants media capture to the app origin;
  documented in the shared helper.

Frontend: getUserMedia failures are now mapped by error name
(utils/micError.js) instead of one blanket "access denied" toast —
permission denials get a per-OS "where to re-enable it" hint
(Windows hint now mentions the desktop-apps mic toggle), missing
devices and busy devices get their own messages, and the previously
hardcoded English toast in useRecording goes through i18n. New keys
added to all 21 locales.

Tests: vitest unit tests for the error mapping (19 cases) and a Rust
unit test for the WebView2 origin allow-list; Windows handler code
cross-checked against webview2-com 0.38.2 / windows-core 0.61.2.

Fixes #323

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:15:17 +05:30
Palash DebnathandClaude Fable 5 ea26893bfc fix(scripts): desktop-prod works from cmd/PowerShell via cross-platform launcher (#282) (#333)
`bun run desktop-prod` (and its :run/:upgrade/:pill/:run:pill variants)
invoked `bash scripts/desktop-prod.sh` directly. On Windows, cmd and
PowerShell have no `bash` on PATH unless Git Bash happens to be there,
so the documented from-source install path died with a cryptic spawn
failure before printing anything — the exact first step in issue #282's
repro.

Add scripts/desktop-prod.mjs, a tiny launcher (runs under bun or node):

- macOS/Linux: execs the bash script unchanged — zero behavior change.
- Windows: locates Git Bash via `where.exe bash`, well-known Git for
  Windows install paths, or derived from git.exe's location; explicitly
  skips C:\Windows\System32\bash.exe (the WSL launcher, which would run
  the script inside Linux and wipe/launch the wrong paths).
- No usable bash: prints an actionable error (install Git for Windows,
  use `bun run desktop`, or use the installer) instead of a spawn error.

All flags are forwarded untouched and the child's exit code is
propagated. scripts/desktop-prod.sh itself is unchanged, and
docs/install/windows.md now lists Git for Windows as a prerequisite
for from-source installs.

Refs #282

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:15:10 +05:30
Palash DebnathandClaude Fable 5 bd60559e3a chore(probe): standardized PR-report publisher with redaction + review gate (#334)
Turns the ad-hoc 'attach a probe trace to the PR' habit into one script:
redacts credentials/home-dirs/emails/IPs from the HTML report, prints a
markdown digest, prunes old local reports, and only uploads (secret gist +
PR comment) behind an explicit --post --yes after human browser review.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:15:01 +05:30
Palash DebnathandClaude Fable 5 78f5db71d7 fix(updater): preview channel offers the newest build across channels (#326) (#335)
Root cause, two layers:

1. tauri-plugin-updater's default comparator is plain semver
   (remote > current). Preview builds are published as X.Y.Z-N
   (e.g. 0.3.5-41 = main, 41 builds after the 0.3.5 tag), which semver
   treats as a *pre-release* of X.Y.Z — so it sorts BELOW stable X.Y.Z.
   Once stable 0.3.5 shipped, preview users were told "you already have
   the latest version" forever.

2. The endpoint list [preview, stable] is not a "best of both" — the
   plugin stops at the first manifest that parses and uses later
   endpoints only as network fallbacks, so a reachable preview manifest
   hid a newer stable release entirely.

Fix: for the preview channel, check BOTH manifests with a custom
version_comparator implementing cross-channel ordering (higher base
version wins; on equal base a suffixed preview build outranks the bare
stable it was built on; preview-vs-preview uses numeric-aware semver
pre-release comparison), then offer the newest candidate. A manifest
error is non-fatal while the other manifest answers. The stable channel
keeps the single endpoint and the plugin's default comparison —
default behavior unchanged on all platforms.

Adds 7 unit tests covering preview ahead of stable (the bug case),
stable passing preview, equal-base both directions, equal versions
(no ping-pong), numeric build-counter ordering, and base dominance.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:14:55 +05:30
Palash DebnathandClaude Fable 5 2ef42ee629 feat(design): free-text 'describe your voice' field maps to design parameters (#317) (#331)
Parity with the hosted omnivoice.app describe field, implemented fully
locally: a deterministic, ordered synonym-table mapper (no model, no
network, stdlib only) projects a natural-language description onto the
existing six-category design space (Gender/Age/Pitch/Style/EnglishAccent/
ChineseDialect). Every emitted token is validated at import time against
the engine taxonomy, so the mapper can never produce an instruct item the
engine validator would reject; Chinese token forms are derived from the
taxonomy, never hardcoded (the one functional pinyin->dialect mapping is
allowlisted in test_no_hardcoded_cjk.py with justification).

UI: a describe textarea in the Design tab fills the attribute picker live
(hand-tuning still possible afterwards); parts of the description the
taxonomy can't express are listed back to the user as 'ignored' instead
of failing silently. New i18n keys in all 21 locales.

Fixes #317

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:14:45 +05:30
48ae4dae1d fix(dub): re-dub honors transcript edits — fingerprints canonicalised, preview cache-busted, atomic mux (#281) (#329)
* fix(dub): re-dub honors transcript edits — fingerprints canonicalised, preview cache-busted, mux made atomic (#281)

Three symptoms, three causes:

1. Edited line, unchanged result: the dubbed preview-video URL was
   identical across re-dubs, so the WebView kept serving the previous
   dub. A generation nonce now cache-busts the preview after every
   completed generation.
2. Preview stuck loading forever: overlapping preview requests ran
   ffmpeg against the same output path and the mtime cache check saw
   the half-written file as valid. The mux now runs under a per-path
   lock, writes to a temp file, and os.replace()s into place.
3. One edit re-dubs all lines: server-side fingerprints were computed
   from pydantic-parsed segments (defaults filled in) but recomputed
   client-side from raw dicts (keys omitted), so every segment always
   looked stale and incremental degraded to a full re-dub. Values are
   now canonicalised on the backend and the frontend builds generation
   inputs through one shared helper (utils/segments.js) for both the
   generate request and the incremental plan.

Fixes #281

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Potential fix for pull request finding 'CodeQL / Uncontrolled data used in path expression'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(dub): realpath containment for job-derived preview paths (CodeQL)

Request-supplied job_id/lang flowed into the preview mux output path.
Both now pass a realpath containment guard against DUB_DIR (the file's
existing per-segment pattern) and lang is allowlist-validated before it
lands in a filename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dub): inline the containment guard — CodeQL can't track it through a helper

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-06-11 12:14:35 +05:30
Palash DebnathandClaude Fable 5 433f1ba617 fix(tts): /generate honors the selected TTS engine (#312) (#324)
* fix(tts): /generate honors the selected TTS engine (#312)

The /generate route always ran the OmniVoice model directly, ignoring both
the Settings engine selection and any per-request override. It now resolves
the active backend (env var > Settings selection > default), supports an
explicit `engine` form field (same pattern as /ws/tts and /v1/audio/speech),
reuses the per-process engine instance cache, keeps inline [pause Nms]
markers working on every engine, and honors applies_own_mastering so studio
engines skip the broadcast mastering chain. The OmniVoice default path is
byte-identical to the old behavior — existing API consumers see no change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(312): resolve modules at run time, drop lifespan client — fixes full-suite isolation

tests/backend/** runs before tests/test_*.py and pollutes sys.modules
(re-imports the services tree), so module-level imports bound at pytest
collection pointed at a stale services.tts_backend — registry patches
landed on a dict the routes no longer read ('Unknown TTS engine' in CI).
Modules are now resolved through sys.modules inside each test. The client
fixture also drops the module-scoped lifespan context manager that bound
event_bus queues to this module's loop (teardown 'Queue bound to a
different event loop') — plain function-scoped TestClient, the
test_api.py pattern.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:10:54 +05:30
Palash DebnathandClaude Fable 5 e8f1f5e057 fix(bootstrap): self-heal structurally broken venv instead of exiting 106 (#314) (#325)
A venv with no pyvenv.cfg (interrupted creation, half-deleted dir, or a
managed Python that was removed) made the backend exit 106 forever; the
only fix was manually deleting .venv. Bootstrap now (1) validates venv
structure before declaring it ready and (2) recognizes the broken-venv
death signature (exit 106 / 'No pyvenv.cfg file') after spawn — in both
cases it quarantines only the .venv itself (rename-aside if deletion
fails, never user data) and rebuilds through the normal setup path with
existing progress stages. Healing is attempted once per launch; a healthy
venv is never touched. The spawn+health-poll loop is extracted from
lib.rs and shared with the retry path.

Fixes #314

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:10:39 +05:30
Palash DebnathandClaude Fable 5 13a3794358 fix(design): stop button + single-playback manager for voice previews (#316) (#322)
Voice previews and synthesized outputs could overlap with no way to stop
them: playBlobAudio() fire-and-forgot a fresh Audio()/AudioContext per
call, and each component (Design demo grid, gallery, demo player) kept
its own uncoordinated audio handle.

- Add utils/playback.js: a global single-playback manager. claimPlayback()
  stops whatever was playing before registering the new playback, returns
  a release() for natural end, and exposes stopActivePlayback() plus a
  usePlaybackSource() hook for UI affordances.
- Register every preview/output path with the manager: playBlobAudio
  (Synthesize output, profile previews, dub segment previews),
  DemoPresetGrid cards, VoiceGallery previews (archetypes / community /
  imports), and the CloneDesignTab "Hear demo" player.
- Visible stop affordance: while a synthesized output is playing, the
  Design/Clone footer CTA becomes a "Stop playback" button (new i18n key
  clone.stop_playback in all 21 locales). Preview cards keep their
  existing play/pause toggle, now wired through the manager.
- Tests: unit suite for the playback manager (claim/stop/release/
  subscribe semantics) and two DemoPresetGrid regression tests for the
  single-playback invariant and the stop toggle.

Fixes #316

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 12:10:09 +05:30
Palash DebnathandClaude Opus 4.8 226aeaa81a style(icons): thinner HD icon strokes app-wide + themed native file inputs (#300)
Lucide ships stroke-width 2 on a 24px grid; at the app's 11-16px render
sizes that weight reads heavy. One global rule (svg.lucide) re-weights
every icon to 1.5 with geometricPrecision shape-rendering — crisper,
lighter, no call-site churn. Hand-rolled SVGs (logo mark, batch spinner)
don't carry the .lucide class and keep their bespoke weights; the one
explicit per-icon strokeWidth (archetype icons) is dropped so the global
weight governs everywhere.

Native <input type="file"> chips are now themed via
::file-selector-button mirroring .ui-btn--subtle (chrome tokens, pill
radius, hover states). All current file inputs hide behind themed labels,
but any visible one — future panels, the LAN/share web view — no longer
renders the OS-default grey button.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 01:34:59 +05:30
Palash DebnathandClaude Fable 5 9cc55ef75e feat(setup): flush action bar, global dbl-click maximize, open maximized (#318)
- First-run action bar is now a pinned flex sibling below a dedicated
  scroll region (.frs__scroll) — flush to the window's bottom edge, with
  nothing rendering beneath it; only the content above scrolls.
- Double-click-to-maximize is wired once in main.jsx, delegated across
  every data-tauri-drag-region (splash, first-run, wizard, main header)
  on all platforms, skipping interactive controls. Replaces the
  wizard-only handler in App.jsx.
- Main window opens maximized (tauri.conf.json).
- Setup wizard preflight checks flow into responsive columns on wide
  windows instead of one tall single column.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 01:34:55 +05:30
Palash DebnathandClaude Fable 5 bfc90e90f5 fix(bootstrap): sync venv deps on app upgrade — stale venv crashed on new imports (#307) (#319)
Upgraded installs replaced backend/ + omnivoice/ sources from the bundle
but never refreshed pyproject.toml/uv.lock or re-ran uv sync, so any
dependency added after the user's venv was created was missing at import
time — e.g. a venv predating scalar-fastapi (added May 4) died on
startup with ModuleNotFoundError once v0.3.5 code landed on it.

- bootstrap.rs: refresh pyproject.toml + uv.lock from the bundle whenever
  a healthy venv is reused; when the lockfile content changed, run
  `uv sync --frozen --no-dev` so newly added deps land. On sync failure
  (e.g. offline upgrade) keep the existing venv instead of bricking a
  previously-working install.
- bootstrap.rs: the repair path now refreshes manifests first (it used to
  sync against the stale lock from when the venv was created) and applies
  the restricted-network HTTP env tuning it was missing.
- backend/main.py: scalar_fastapi import is now guarded — it only powers
  /docs, so a venv without it must still boot; /docs returns 503 with an
  actionable message instead.

Closes #307

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 01:34:51 +05:30
Palash DebnathandClaude Fable 5 9312e434ef fix(asr): clone references transcribe via the ASR registry, not the broken transformers pipeline (#308) (#321)
Voice cloning without a transcript fell through to OmniVoice's built-in
load_asr_model() — a transformers pipeline() load of
whisper-large-v3-turbo that fails outright on transformers 5.3 — even
when whisperx / faster-whisper / mlx-whisper were installed and working.
The dub pipeline already used the registry; the /generate clone path
never did.

- services/asr_backend.py: new transcribe_reference() resolves the
  active registry backend (honoring auto-detect order and the
  OMNIVOICE_ASR_BACKEND override), extracts text from either result
  shape (top-level "text" or whisperx-style segments), and degrades to
  None on any failure so the model fallback behaves exactly as before.
  When the registry itself resolves to pytorch-whisper it defers to the
  model's lazy load instead of building a second pipeline.
- api/routers/generation.py: transcript-less references get transcribed
  in the GPU pool before inference.
- tests/test_transcribe_reference.py: covers both result shapes,
  failure degradation, and the pytorch-whisper deferral.

The remaining half of #308 — pytorch-whisper itself being incompatible
with transformers 5.3 when it truly is the last resort — is tracked in
the issue.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 01:34:47 +05:30
Palash DebnathandClaude Fable 5 d04c1fdd0d fix(dub): Timing strategy options never rendered — wrong prop name on Segmented (#313) (#320)
The Timing control passed `options=` to <Segmented>, whose prop is
`items=` (defaulting to []), so the toggle group rendered as a single
empty pill with nothing to click — users had no way to pick
Concise / Stretch Video / Strict slot. Broken since the control was
introduced; every other Segmented call site already uses `items=`.

Closes #313

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 01:28:08 +05:30
Tim Kaufmann 5d602c8871 fix(tts): let studio engines skip the broadcast mastering chain (#311)
* fix(tts): let studio engines skip the broadcast mastering chain

apply_mastering() (HighpassFilter + Compressor + 8% Reverb) is tuned for
OmniVoice's 24 kHz clone output. The OpenAI-compatible /v1/audio/speech
route (_run_tts) runs it on every engine, including VoxCPM2 — whose native
48 kHz output is already studio-grade. There the compressor pump and the
reverb tail are audible degradation rather than polish.

Add an opt-out class flag TTSBackend.applies_own_mastering (default False,
so all existing engines are unchanged) and set it True on VoxCPM2Backend.
_run_tts() skips apply_mastering() when the active backend declares it.
Loudness normalisation still runs for every engine (benign peak scale).

* fix(tts): also skip mastering on the streaming route for studio engines

tts_stream.py is the other route that runs the *active* TTS backend
(get_active_tts_backend), so it needs the same applies_own_mastering guard
as openai_compat._run_tts — otherwise VoxCPM2 output is still pumped/reverbed
when streamed. The remaining apply_mastering() call sites (generation.py,
batch.py, batched_tts.py, dub_generate.py) run the OmniVoice model directly
via get_model(), never the active backend, so VoxCPM2 cannot reach them.

* docs(tts): mark OmniVoice-only mastering sites with TODO(#312)

Per review: instead of always-False guards on routes that never run the
active backend, leave a pointer so the applies_own_mastering guard is added
exactly when those routes become engine-aware (issue #312).
2026-06-11 00:25:17 +05:30
suenandopenclawer 5ba8a5a8a0 fix(gguf): forward speech generation controls (#306)
Co-authored-by: openclawer <bdfzer8@gmail.com>
2026-06-11 00:23:29 +05:30
MUHAMED FAZAL PS e7f78bffef fix: disable tqdm on non-TTY to prevent OSError on Windows (#305)
* fix: disable tqdm on non-TTY to prevent OSError on Windows (#283)

When running as a Tauri backend (non-TTY stdout), tqdm tries to write
terminal control characters which fails with Errno 22 on Windows.

Set TQDM_DISABLE=1 when stdout is not a TTY during model loading.

* fix: guard sys.stdout against None and fix import ordering (#283)

- Add None check before calling isatty() to prevent AttributeError
- Fix import ordering (sys after re alphabetically)
2026-06-11 00:18:43 +05:30
Palash DebnathandClaude Opus 4.8 f3e403193e fix(dictation): macOS auto-paste — don't steal focus, write clipboard natively (#287) (#299)
Dictation via the global shortcut transcribed fine but the text never reached
the target app on macOS, due to two stacked bugs (diagnosed, patched, and
verified by @geektf in #287):

1. The ShortcutState::Pressed handler called win.set_focus(), making the
   widget frontmost — the simulated ⌘V from simulate_paste() landed in the
   widget instead of the app being dictated into. Skip set_focus() on macOS
   (same #[cfg(not(target_os = "macos"))] guard the other widget call sites
   already use).

2. With the widget unfocused, the WebView clipboard APIs
   (navigator.clipboard.writeText / execCommand('copy')) fail silently in
   WKWebView, so ⌘V pasted whatever was previously on the clipboard.
   simulate_paste now takes Option<String> and writes the transcript to the
   clipboard natively (arboard) before sending the keystroke — no window
   focus required. CaptureWidget passes the transcript; copyText() stays as
   best-effort for browser (non-Tauri) mode, and the optional param keeps
   any text-less call sites working.

cargo check clean (the unreachable_code warning in setup.rs is pre-existing
from #286); frontend node:test suite passes. End-to-end behavior verified by
the reporter on macOS 26 / M4 Pro with both patches applied.

Fixes #287

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 00:36:47 +05:30
Palash DebnathandClaude Opus 4.8 71cdc1553e fix(dub): video retry after URL ingest, responsive layout, icon-only toolbar (#304)
Three reported issues in the dubbing editor:

1. Dark video after YouTube ingest: the preview mounted while yt-dlp was
   still finalizing the media file — the first load failed (MediaError 2
   network / 4 non-media body) and the once-only error handler declared
   the source dead, leaving a black box until the project was reloaded.
   The error handler now retries with backoff (up to 6× over ~21s) before
   giving up; decode errors (3) stay terminal.

2. Responsive/resizable layout: min-width:0 on the split-grid columns
   (the classic shrink trap), settings-bar fields get real shrink room
   instead of locked min-widths, bulk selects flex, prep-bar overlays are
   viewport-bounded, and the segment table's fixed rails narrow at
   1100px and collapse speaker/gain entirely below 760px so the text
   column keeps usable width at any size.

3. Toolbar: Save / Reset / Export are icon-only with hover tooltips
   (+ aria-labels); Generate Dub keeps its label as the primary verb.
   Skeleton header matches.

Vitest 196/196 green.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 00:31:27 +05:30
Palash DebnathandClaude Opus 4.8 0bb026f6f8 feat(setup): optional Hugging Face token in the library act (#303)
The unified library dropped the inline HF-token field the old
ModelStoreTab embed used to provide — so onboarding produced installs
with no token, and users hit the 'speaker diarization disabled' wall on
their first multi-speaker dub. Restored as a quiet disclosure at the
bottom of the Models & engines act: password input → POST
/system/set-env HF_TOKEN (same durable persistence Settings uses),
saved/error states, Enter-to-save. Copy names the concrete benefit
(pyannote diarization) and the local-first promise (token stays on this
machine). 6 strings × 21 locales.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 22:23:55 +05:30
Palash DebnathandClaude Opus 4.8 e424f46656 test(setup): update DictationDemo asset-missing contract to #294 (#302)
The test asserted the component renders nothing when demo clips 404 —
the exact behavior #294 deliberately removed (it blanked the wizard's
Try-dictation act on every real install). New contract under test: the
script cards are asset-gated and disappear; the hotkey card (shortcut +
press-to-verify, zero assets needed) stays.

This was the single failure breaking CI on main since #294 merged
(34 files / 196 tests green with the fix).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 22:16:20 +05:30
Palash DebnathandClaude Opus 4.8 1171185c9d style(setup): stable scaffold — no layout shift anywhere in the journey (#301)
Fair criticism taken: vertically centering variable-height content meant
every act and step reflowed the page around its own center, and selecting
an option pushed everything below it. The journey now has one stable
scaffold — only the content region changes:

- deck is top-anchored (waveform opens the page right under the titlebar;
  the centering dead-zone is gone) and fills the viewport
- footer (serial plate, totals, armed action) is sticky at the bottom
  with a soft fade — never scrolls out of view, hugs the bottom when
  content is short
- variable text gets reserved space: masthead subtitles hold two lines;
  option descriptions move out of the cards into a fixed two-line caption
  slot per radio group (aria-live), so switching options swaps text in
  place with zero shift — cards themselves are title-only
- description tooltips retained on every card

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:55:01 +05:30
476 changed files with 59813 additions and 3678 deletions
+32
View File
@@ -0,0 +1,32 @@
# RTK - Rust Token Killer (Google Antigravity)
**Usage**: Token-optimized CLI proxy for shell commands.
## Rule
Always prefix shell commands with `rtk` to minimize token consumption.
Examples:
```bash
rtk git status
rtk cargo test
rtk ls src/
rtk grep "pattern" src/
rtk find "*.rs" .
rtk docker ps
rtk gh pr list
```
## Meta Commands
```bash
rtk gain # Show token savings
rtk gain --history # Command history with savings
rtk discover # Find missed RTK opportunities
rtk proxy <cmd> # Run raw (no filtering, for debugging)
```
## Why
RTK filters and compresses command output before it reaches the LLM context, saving 60-90% tokens on common operations. Always use `rtk <cmd>` instead of raw commands.
+119 -3
View File
@@ -7,12 +7,26 @@
language: "en-US"
early_access: false
# The review voice: a panel of senior domain experts, not a linter.
tone_instructions: >-
Review as a panel of principal engineers: ML inference, audio DSP, desktop
systems, product polish. Cite exact lines, name the failure mode, give the
concrete fix. No filler praise; raise nits only when they change a decision.
reviews:
# "chill" keeps the bot from blocking merges — it comments, it does not gate.
# Hard gating lives in CI (security.yml) and the constitution's human bar.
profile: chill
request_changes_workflow: false
high_level_summary: true
# Every walkthrough gets a visual: mermaid sequence diagrams for the
# mechanics, plus (via the summary instructions) an ASCII before/after
# sketch when the PR touches UI — so each PR is reviewable at a glance.
sequence_diagrams: true
high_level_summary_instructions: >-
If the PR changes UI (JSX/TSX/CSS/Tauri windows), include a compact ASCII
before/after sketch of the affected layout or component. If it changes
behavior, include a short mermaid flowchart of the new mechanism.
review_status: true
poem: false
@@ -39,7 +53,8 @@ reviews:
- "!**/*.onnx"
- "!tests/fixtures/**"
# Encode the project's hard constraints so the bot reviews against them.
# One expert lens per subsystem — encode what a passionate senior in each
# domain would actually check, beyond what linters and CI already gate.
path_instructions:
- path: "**/*.{py,rs,js,jsx,ts,tsx}"
instructions: >-
@@ -48,18 +63,119 @@ reviews:
model download, or an explicitly opt-in endpoint. Flag any code that
persists or logs values matching *TOKEN*/*KEY*/*SECRET* or absolute user
home paths (/Users/<name>/, C:\\Users\\<name>\\).
- path: "backend/services/**/*.py"
instructions: >-
Review as an ML-inference/audio engineer. Check: thread-safety of model
and cache state across the GPU worker pool; device/dtype assumptions
that break on one of CUDA/MPS/ROCm/CPU; VRAM lifecycle (load/unload,
leaks on the error path); sample-rate, channel-count and tensor-shape
assumptions at engine boundaries; blocking calls inside async paths;
model download/cache behavior when offline. Engine code must stay
backward-compatible with already-installed on-disk model state.
- path: "backend/**/*.py"
instructions: >-
Default features must behave identically on macOS, Windows and Linux.
Platform-specific implementation is allowed, but a divergent user-visible
default is a P0 bug — flag it and suggest an opt-in (Settings/env/flag).
Any DB schema change must go through an alembic migration with an upgrade
path; flag direct schema edits. Engine code must stay backward-compatible
with already-installed on-disk model state (no forced reinstall).
path; flag direct schema edits. The backend serves loopback HTTP: treat
every query/path/form param as hostile (path traversal, log injection,
CSRF from a browser tab), and never route user-chosen filesystem
destinations through HTTP — that authorization belongs in the Tauri
process.
- path: "frontend/src/**/*.{js,jsx,ts,tsx}"
instructions: >-
Review as a product-minded senior frontend engineer. Check: stale state
and races (async results landing after unmount or after newer requests);
every user-visible failure has an actionable, non-technical error
message; loading/disabled states during long operations. Every new
user-facing string must be an i18n t('...') key present in ALL 21
frontend/src/i18n/locales/*.json files — flag hardcoded UI strings and
keys missing from any locale.
- path: "frontend/src-tauri/**/*.rs"
instructions: >-
Review as a desktop-systems engineer. Check: every #[tauri::command] is
callable from the webview — validate inputs and scope filesystem/process
access accordingly; window and webview lifecycle on all three OSes;
child-process spawn/exit-code/stderr handling; no unwrap/expect on
user-controlled input; platform cfg blocks keep user-visible defaults
identical across macOS/Windows/Linux.
- path: "tests/**/*.py"
instructions: >-
Review as a test-infrastructure engineer. Check: the test would fail
before the fix and pass after (no tautologies); no sleeps as
synchronization; no module-level imports of app modules that go stale
under sys.modules pollution (resolve at run time); TestClient instances
are function-scoped and not lifespan-bound unless the test needs it;
new functional CJK is allowlisted in tests/test_no_hardcoded_cjk.py with
a justification.
- path: ".github/workflows/**"
instructions: >-
Pin actions to a major version tag at minimum. Flag any workflow that
grants write permissions it does not need.
# Non-gating pre-merge audits of the project's hard rules (warning mode —
# the human owner is the gate, these make the checklist visible per-PR).
pre_merge_checks:
title:
mode: warning
requirements: >-
Conventional-commit style with scope, e.g. "fix(dub): …", and the issue
reference in title or body. Never propose a version bump.
issue_assessment:
mode: warning
custom_checks:
- name: "Cross-platform default parity"
mode: warning
instructions: >-
If the PR changes any default-mode (out-of-the-box) behavior, verify
it behaves identically on macOS, Windows and Linux, or is moved behind
an explicit opt-in (Settings toggle, env var, or CLI flag). A
platform-divergent default is a P0 per CLAUDE.md — fail this check
and say which platform diverges.
- name: "i18n completeness (21 locales)"
mode: warning
instructions: >-
For every new or changed t('...') key in frontend code, verify the
key exists in all 21 files under frontend/src/i18n/locales/. List any
locale files missing the key. Also flag hardcoded user-facing strings
that bypass i18n entirely.
- name: "Local-first guarantee"
mode: warning
instructions: >-
Verify the PR adds no required cloud calls, accounts, API keys or
telemetry. Outbound traffic is only allowed to GitHub Issues (opt-in
bug reporting) and HuggingFace model downloads. The app must remain
fully functional offline and with reporting disabled.
- name: "Backward compatibility"
mode: warning
instructions: >-
Verify existing omnivoice_data/ (voices, projects, settings) and
already-installed engine model state keep working without manual
migration. Any DB schema change must ship an alembic migration with
an upgrade path. Flag anything that would force users to reinstall an
engine or re-download model weights.
finishing_touches:
docstrings:
enabled: true
unit_tests:
enabled: true
# Feed the bot the project constitution and docs, and let it accumulate
# learnings from review conversations ("@coderabbitai always/never …").
knowledge_base:
code_guidelines:
enabled: true
filePatterns:
- "CLAUDE.md"
- "docs/**/*.md"
learnings:
scope: auto
issues:
scope: auto
pull_requests:
scope: auto
chat:
auto_reply: true
-45
View File
@@ -1,45 +0,0 @@
---
name: 🐛 Bug Report
about: Report a bug to help us improve OmniVoice Studio
title: "[Bug] "
labels: ["bug", "triage"]
assignees: []
---
## Describe the bug
A clear and concise description of what the bug is.
## To reproduce
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '...'
3. See error
## Expected behavior
What you expected to happen.
## Screenshots / Logs
If applicable, add screenshots or paste relevant logs from **Settings → Logs**.
> **Tip:** **Settings → About → "Save diagnostic bundle"** produces a zip
> (self-check report, recent errors, scrubbed log tails) you can drag onto
> this issue — it answers most environment questions below automatically.
> Headless installs: `python backend/main.py --diagnose` prints the same
> self-check (`--deep` also test-loads the active engine).
## Environment
- **OS:** [e.g. macOS 15.2, Windows 11, Ubuntu 24.04]
- **Install method:** [Desktop app / Docker / From source]
- **Version:** [e.g. v0.2.7 — check Settings → About]
- **GPU:** [e.g. NVIDIA RTX 4090 / Apple M3 Pro / CPU only]
- **RAM:** [e.g. 16 GB]
- **Active TTS engine:** [e.g. omnivoice — check Settings → Engines]
## Additional context
Add any other context about the problem here.
+109
View File
@@ -0,0 +1,109 @@
name: 🐛 Bug report
description: Something works incorrectly or crashes (not a first-run/install problem — use the install template for those).
title: "[Bug] "
labels: ["bug", "triage"]
body:
- type: markdown
attributes:
value: |
Thanks for helping improve OmniVoice Studio! 🎙️
**Fastest path to a fix:** **Settings → About → "Save diagnostic bundle"** makes a
zip (self-check + recent errors + scrubbed log tails) — drag it onto this issue and
most of the environment questions below are answered automatically.
Headless: `python backend/main.py --diagnose` (add `--deep` to test-load the engine).
- type: checkboxes
id: preflight
attributes:
label: Before filing
options:
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and this isn't a duplicate.
required: true
- label: I'm on the latest release (or `main`) — older builds may already be fixed.
required: false
- type: textarea
id: what-happened
attributes:
label: What happened?
description: A clear description of the bug, including the exact error text / toast if any.
placeholder: "Voice cloning failed with '…' after I clicked Generate."
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to reproduce
value: |
1.
2.
3.
validations:
required: true
- type: textarea
id: expected
attributes:
label: What did you expect instead?
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS (Apple Silicon)
- macOS (Intel)
- Windows (x64)
- Linux (AppImage)
- Linux (.deb)
- Linux (other / from source)
- Docker
validations:
required: true
- type: dropdown
id: install
attributes:
label: How did you install it?
options:
- Desktop app (installer / AppImage)
- Docker image
- From source (uv sync)
validations:
required: true
- type: input
id: version
attributes:
label: Version
description: Settings → About (e.g. v0.3.5), or the Docker tag / git SHA.
placeholder: "v0.3.5"
validations:
required: true
- type: dropdown
id: device
attributes:
label: Compute device
options:
- NVIDIA GPU (CUDA)
- AMD GPU (ROCm)
- Apple Silicon (MPS)
- Intel GPU (XPU)
- CPU only
- Not sure
validations:
required: true
- type: input
id: engine
attributes:
label: Active TTS/ASR engine
description: Settings → Engines (e.g. omnivoice, cosyvoice, indextts2, whisperx).
placeholder: "omnivoice"
- type: textarea
id: logs
attributes:
label: Logs / diagnostic bundle
description: Drag the diagnostic bundle here, or paste relevant lines from **Settings → Logs**. Secrets are scrubbed automatically.
render: text
- type: textarea
id: extra
attributes:
label: Anything else?
description: Screenshots, the input that triggered it, RAM/VRAM, etc.
+11
View File
@@ -0,0 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: 💬 Discord — questions & quick help
url: https://discord.gg/bzQavDfVV9
about: Usage questions, setup help, and chat. Faster than an issue for "how do I…".
- name: 🗣️ GitHub Discussions
url: https://github.com/debpalash/OmniVoice-Studio/discussions
about: Ideas, show-and-tell, and open-ended Q&A that isn't a bug or a specific feature ask.
- name: 🔒 Security vulnerability
url: https://github.com/debpalash/OmniVoice-Studio/security/policy
about: Please report security issues privately — do NOT open a public issue.
-23
View File
@@ -1,23 +0,0 @@
---
name: ✨ Feature Request
about: Suggest an idea for OmniVoice Studio
title: "[Feature] "
labels: ["enhancement"]
assignees: []
---
## Is your feature request related to a problem?
A clear description of what the problem is. Ex. "I'm always frustrated when..."
## Describe the solution you'd like
A clear description of what you want to happen.
## Describe alternatives you've considered
Any alternative solutions or features you've considered.
## Additional context
Add any other context, mockups, or screenshots about the feature request here.
@@ -0,0 +1,50 @@
name: ✨ Feature request
description: Suggest an improvement or a new capability.
title: "[Feature] "
labels: ["enhancement"]
body:
- type: checkboxes
id: preflight
attributes:
label: Before filing
options:
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and [discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) for this idea.
required: true
- type: textarea
id: problem
attributes:
label: What problem does this solve?
description: The use case / friction this addresses ("When I … I can't …").
validations:
required: true
- type: textarea
id: solution
attributes:
label: Proposed solution
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
- type: dropdown
id: area
attributes:
label: Area
options:
- Voice cloning
- Voice design
- Video dubbing
- Real-time dictation
- Audiobook / Stories (long-form)
- TTS/ASR engines
- Install / setup / packaging
- Other
validations:
required: true
- type: markdown
attributes:
value: |
> OmniVoice is **local-first** — features must work fully offline with no accounts,
API keys, or cloud calls, and behave identically on macOS/Windows/Linux. Proposals
that fit those constraints are easiest to land.
@@ -0,0 +1,86 @@
name: 🧩 Install / first-run problem
description: The app won't install, set up, download models, or reach a first working output.
title: "[Install] "
labels: ["install", "triage"]
body:
- type: markdown
attributes:
value: |
A first-run that *just works* is the whole point — sorry it didn't. Let's fix it.
If the app launched far enough to open Settings, **Settings → About → "Save diagnostic
bundle"** captures most of this; otherwise the fields below are enough.
- type: dropdown
id: stage
attributes:
label: Where did it fail?
options:
- App won't launch / blank or broken window
- Python / uv environment bootstrap
- Model download (HuggingFace)
- Engine install (CosyVoice / IndexTTS / MLX / etc.)
- First synthesis / dub never completes
- Other
validations:
required: true
- type: textarea
id: error
attributes:
label: The error
description: The exact message, traceback, or what you see on screen.
render: text
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS (Apple Silicon)
- macOS (Intel)
- Windows (x64)
- Linux (AppImage)
- Linux (.deb)
- Linux (other / from source)
- Docker
validations:
required: true
- type: dropdown
id: install
attributes:
label: How are you installing it?
options:
- Desktop app (installer / AppImage)
- Docker image
- From source (uv sync)
validations:
required: true
- type: input
id: version
attributes:
label: Version
placeholder: "v0.3.5 (or installer build / git SHA)"
validations:
required: true
- type: dropdown
id: network
attributes:
label: Network conditions (model/dependency downloads)
description: Restricted networks are a known source of bootstrap failures (mirror fallback).
options:
- Normal / unrestricted
- Behind a corporate proxy / firewall
- Region with restricted access (e.g. China, Russia)
- Offline / air-gapped
- Not sure
validations:
required: true
- type: textarea
id: logs
attributes:
label: Logs / diagnostic bundle
description: Drag a diagnostic bundle, or paste the install/bootstrap log. Headless self-check — `python backend/main.py --diagnose`.
render: text
- type: textarea
id: tried
attributes:
label: What have you already tried?
+8 -13
View File
@@ -18,7 +18,7 @@
- [ ] 📝 Documentation
- [ ] 🧪 Tests
- [ ] 🔧 CI / Build
- [ ] 🚀 Release prep (RC or final)
- [ ] 🚀 Release prep
## Testing
@@ -33,17 +33,12 @@
- [ ] No local machine paths, logs, or personal env details in this PR
- [ ] Version files are in sync (if version bump): `pyproject.toml`, `package.json`, `tauri.conf.json`, `Cargo.toml`
- [ ] If this PR changes runtime behavior, the regression fixture at `tests/fixtures/omnivoice_data/` still loads green on the `smoke-matrix` CI job (macOS + Windows + Linux)
- [ ] If this is part of a release, I've read the "Release cadence" section below and confirmed this PR targets the right RC
## Release cadence (read once per RC)
## Release cadence
OmniVoice ships every minor on a **two-RC cadence**:
- `vX.Y.0-rc1` — cut from `main` once all GATE-* requirements pass; clean-VM exercise on 4 OSes (per `REL-01`)
- 48-hour soak (no new commits to release branch except fix-forward)
- `vX.Y.0` — promotion if rc1 is clean
If your PR touches install / bootstrap / CI, it MUST land before rc1 cut, not between rc1 and the promotion. During a soak, any merge needs explicit OK from the release captain.
## Screenshots
<!-- If applicable, add screenshots or recordings. -->
OmniVoice ships **continuous-to-main** — no release candidates, no soak windows.
Every merged PR is immediately part of the rolling preview (`main`, Docker
`:latest`, the desktop Preview channel). Versioned releases are tagged from
`main` when it's ready; `main` then bumps to the next patch automatically.
Users who want stability pin a release tag / Docker `:stable` / the desktop
Stable channel.
+8 -2
View File
@@ -92,7 +92,10 @@ jobs:
- name: Install frontend deps
working-directory: frontend
run: bun install
# --frozen-lockfile so a frontend/package.json change that forgets to
# regenerate the root bun.lock fails HERE (fast) instead of only in the
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
run: bun install --frozen-lockfile
# checkJs is true in tsconfig for IDE feedback, but 947 pre-existing
# JS errors remain. Override to false in CI so only .ts files block.
@@ -174,7 +177,10 @@ jobs:
- name: Install frontend deps
working-directory: frontend
run: bun install
# --frozen-lockfile so a frontend/package.json change that forgets to
# regenerate the root bun.lock fails HERE (fast) instead of only in the
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
run: bun install --frozen-lockfile
# tauri-build's setup hook reads tauri.conf.json's `frontendDist`
# ("../dist"), which only exists after a frontend build. Without this,
+62 -17
View File
@@ -5,14 +5,21 @@
# - push to main branch → :main, :sha- (rolling "edge" build)
# - workflow_dispatch → :sha- only (ad-hoc test build)
#
# Tag ↔ image mapping
# :latest — always the most recent versioned release (set on every v* tag push)
# :0.3.0 — exact version from the git tag
# Tag ↔ image mapping (versioning hard rule, owner-set 2026-06-11:
# :latest IS the preview channel; stable users pin :stable or a version tag)
# :latest — rolling preview: latest commit on main (always last release + 1 dev)
# :main — alias of the same rolling main build (kept for back-compat)
# :stable — most recent versioned release (set on every v* tag push)
# :0.3.6 — exact version from the git tag
# :0.3 — major.minor floating tag (updated on every patch within the minor)
# :main — latest commit on main; may be ahead of the last tagged release
# :sha-xxxx — specific commit SHA; produced by workflow_dispatch
#
# Images land at: ghcr.io/debpalash/omnivoice-studio
# Images land at: ghcr.io/debpalash/omnivoice-studio AND docker.io/palashdeb/omnivoice-studio
# (Docker Hub push gated on the DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets;
# if unset the build still pushes to GHCR.)
#
# On main pushes the Docker Hub repository overview is also synced from
# deploy/dockerhub-overview.md (source of truth for the hub.docker.com page).
#
# NOTE: the Docker image is the headless web-server build of OmniVoice (FastAPI
# backend + pre-built React frontend served over HTTP). The Tauri desktop
@@ -34,6 +41,7 @@ permissions:
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
DOCKERHUB_IMAGE: palashdeb/omnivoice-studio
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
jobs:
@@ -56,28 +64,43 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
# Docker Hub login — only when the secret is present, so forks / runs
# without the credential still publish to GHCR.
- name: Check Docker Hub credentials
id: dockerhub
run: echo "enabled=${{ secrets.DOCKERHUB_TOKEN != '' }}" >> "$GITHUB_OUTPUT"
- name: Log in to Docker Hub
if: steps.dockerhub.outputs.enabled == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
# Tag strategy (`:sha-<short>` is emitted on every trigger):
# v0.3.0 tag push → :0.3.0, :0.3, :latest, :sha-
# main branch push → :main, :sha-
# v0.3.6 tag push → :0.3.6, :0.3, :stable, :sha-
# main branch push → :latest, :main, :sha-
# workflow_dispatch → :sha- only
#
# Fix for stale :latest (issues #249, #251):
# The previous rule used `enable={{is_default_branch}}`, which evaluates
# to false on tag pushes (detached HEAD) — so :latest was never updated
# when a release tag was pushed. The version / :latest / :main rules are
# gated on `github.event_name == 'push'` so a manual workflow_dispatch can
# only ever produce a throwaway `:sha-` tag (never republish a mutable
# tag), and :latest additionally excludes prerelease tags (those contain a
# `-`, e.g. v1.0.0-rc.1) so a prerelease can't clobber :latest.
# All mutable-tag rules stay gated on `github.event_name == 'push'` so a
# manual workflow_dispatch can only ever produce a throwaway `:sha-` tag
# (the stale-:latest fix from #249/#251). :stable excludes prerelease
# tags (those contain a `-`) so a prerelease can't clobber it.
- name: Extract metadata (tags, labels)
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
# Same tag set applied to both registries. The Docker Hub line is
# blank when the secret is unset, so metadata-action emits GHCR-only
# tags in that case.
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
${{ steps.dockerhub.outputs.enabled == 'true' && env.DOCKERHUB_IMAGE || '' }}
tags: |
type=semver,pattern={{version}},enable=${{ github.event_name == 'push' }}
type=semver,pattern={{major}}.{{minor}},enable=${{ github.event_name == 'push' }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
type=raw,value=stable,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
type=raw,value=main,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
type=sha,prefix=sha-,format=short
@@ -91,3 +114,25 @@ jobs:
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Sync the Docker Hub repository overview from deploy/dockerhub-overview.md.
# Only on main pushes (the overview tracks the rolling preview) and only
# when Docker Hub creds are present, mirroring the push gating above.
#
# continue-on-error: the overview text is cosmetic, and the description
# PATCH 403s unless DOCKERHUB_TOKEN carries description-edit scope (many
# fine-grained Docker Hub tokens that can push still can't edit the
# description). The image build+push is what matters — a creds-scope
# mismatch on this cosmetic step must not fail the whole Docker run. To
# actually sync the overview, use a token with read/write (incl.
# description) scope, or the account password.
- name: Update Docker Hub description
if: steps.dockerhub.outputs.enabled == 'true' && github.event_name == 'push' && github.ref == 'refs/heads/main'
continue-on-error: true
uses: peter-evans/dockerhub-description@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
repository: ${{ env.DOCKERHUB_IMAGE }}
short-description: "Local ElevenLabs alternative: voice cloning, design & video dubbing in 646 languages. No API keys."
readme-filepath: ./deploy/dockerhub-overview.md
+97
View File
@@ -0,0 +1,97 @@
# Docs drift — daily inventory-vs-docs check with a single rolling issue.
#
# docs/features.yaml is the canonical inventory; scripts/check-docs-drift.py
# diffs it against README.md, docs/, and the engine registries. On drift the
# job updates (or creates) ONE issue labeled `docs-drift` in place — no issue
# spam — and closes it automatically when the check is clean again.
#
# Companion to the PR-gating validate-install-docs.py step in ci.yml.
# Spec: docs/competitive-analysis.md Spec 9a / parity program Wave 0.1.
# Rolling-issue pattern adapted from Patter (MIT).
name: docs-drift
on:
schedule:
# Daily 03:30 UTC — after most merges, before EU morning triage.
- cron: "30 3 * * *"
workflow_dispatch:
permissions:
contents: read
issues: write
jobs:
drift:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install checker deps
run: pip install "pyyaml>=6"
- name: Check inventory vs README/docs/registries
id: drift
continue-on-error: true
run: python scripts/check-docs-drift.py --output drift-report.md
- name: Update rolling docs-drift issue
uses: actions/github-script@v7
env:
DRIFT_OUTCOME: ${{ steps.drift.outcome }}
with:
script: |
const fs = require('fs');
const drifted = process.env.DRIFT_OUTCOME === 'failure';
const { owner, repo } = context.repo;
const label = 'docs-drift';
const open = await github.rest.issues.listForRepo({
owner, repo, state: 'open', labels: label, per_page: 5,
});
if (drifted) {
let body = '';
try {
body = fs.readFileSync('drift-report.md', 'utf8');
} catch {
body = '# Docs drift report\n\nThe checker failed before writing a report — see the workflow run logs.';
}
body += `\n\n---\n_Last checked by [run ${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})._\n`;
if (open.data.length > 0) {
await github.rest.issues.update({
owner, repo, issue_number: open.data[0].number, body,
});
core.info(`Updated rolling issue #${open.data[0].number}`);
} else {
const created = await github.rest.issues.create({
owner, repo,
title: 'docs-drift: feature inventory vs docs mismatch',
body,
labels: [label, 'documentation'],
});
core.info(`Created rolling issue #${created.data.number}`);
}
} else {
for (const issue of open.data) {
await github.rest.issues.createComment({
owner, repo, issue_number: issue.number,
body: 'Drift resolved — nightly check is clean again. Closing automatically.',
});
await github.rest.issues.update({
owner, repo, issue_number: issue.number, state: 'closed',
});
core.info(`Closed rolling issue #${issue.number}`);
}
}
- name: Surface drift as a failed run
if: steps.drift.outcome == 'failure'
run: |
echo "Docs drift detected — see the rolling docs-drift issue."
exit 1
+55
View File
@@ -0,0 +1,55 @@
# LLM-judge evals — semantic quality suites, NEVER a gate.
#
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
# judges never gate CI. This workflow is scheduled + manual only, the eval
# step is continue-on-error, and the JSON report is the deliverable
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
# the only gates.
#
# On the hosted runner there is no local LLM endpoint, so the run usually
# reports "skipped — no LLM backend configured"; the workflow exists so the
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
# (e.g. a self-hosted runner with Ollama).
name: evals
on:
schedule:
# Weekly, Sundays 04:00 UTC.
- cron: "0 4 * * 0"
workflow_dispatch:
permissions:
contents: read
jobs:
evals:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- uses: astral-sh/setup-uv@v3
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
- name: Install deps
run: uv sync
- name: Run eval suites (non-gating)
continue-on-error: true
env:
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
run: uv run python tests/evals/run_evals.py --output eval-report.json
- name: Upload report artifact
uses: actions/upload-artifact@v4
with:
name: eval-report
path: eval-report.json
if-no-files-found: warn
+110 -9
View File
@@ -133,9 +133,28 @@ jobs:
rust_target: aarch64-apple-darwin
bundles: "app,dmg,updater"
# macOS Intel dropped: Apple shipped the last Intel Mac in 2023 and
# Rosetta 2 runs the ARM build natively. macos-13 runner backlog
# was also blocking every release tag for ~10 min.
# macOS Intel (#279): reinstated. The earlier "Rosetta 2 runs the
# ARM build" rationale for dropping it was backwards — Rosetta only
# translates x86_64→arm64, so Intel Macs (supported through macOS
# Sequoia) simply cannot run the aarch64 bundle and had NO
# installable artifact. Runner: `macos-15-intel`, GitHub's
# designated migration target after macos-13 retired (Dec 2025);
# it's a standard (public-repo-free) image supported through
# August 2027 — the last x86_64 image Actions will offer. Building
# natively (not cross-compiling from the arm64 leg) keeps the
# per-TRIPLE uv/ffmpeg sidecar fetches, the DMG installer smoke,
# and the ad-hoc signing verification (scripts/
# verify-macos-signing.sh, PR #290) all exercising the real
# x86_64 artifact on real Intel hardware. The macos-13 queue
# backlog that motivated the original drop is contained by
# fail-fast:false — a slow Intel leg can delay the release run but
# can't fail the other targets.
- os: macos-15-intel
arch: x86_64-apple-darwin
label: "macOS Intel"
rust_target: x86_64-apple-darwin
bundles: "app,dmg,updater"
# Windows: force MSI bundling via --bundles. NSIS fails at makensis
# because our PyInstaller payload approaches its ~2 GB stub limit.
- os: windows-2022
@@ -291,7 +310,10 @@ jobs:
case "$TRIPLE" in
aarch64-apple-darwin|x86_64-apple-darwin)
# evermeet.cx ships each binary as a separate .zip containing
# a single x86_64 Mach-O executable (runs via Rosetta on arm64).
# a single x86_64 Mach-O executable — natively correct on the
# Intel leg, and runs via Rosetta 2 on the arm64 leg. Both
# darwin TRIPLEs therefore bundle the same payload; only the
# sidecar filename suffix differs.
for TOOL in ffmpeg ffprobe; do
if [ "$TOOL" = "ffmpeg" ]; then
URL="https://evermeet.cx/ffmpeg/getrelease/zip"
@@ -398,11 +420,12 @@ jobs:
# always reported the static 0.3.0 never looked "newer", so no update was
# ever delivered). Ephemeral, CI-only — never committed. Tauri reads the
# bundle + updater version from tauri.conf.json, so rewriting it here
# stamps the artifacts + latest.json. `0.3.0-preview.N` is a prerelease of
# the current target, so previews converge to stable when 0.3.0 ships
# (0.3.0 > 0.3.0-preview.N). NOTE: the Windows MSI ProductVersion strips
# the prerelease (→ 0.3.0), a wrinkle to verify for win preview→preview
# upgrades; mac/linux replace the bundle wholesale and are unaffected.
# stamps the artifacts + latest.json. Under the versioning hard rule
# (owner-set 2026-06-11) main is always last-release + 1, so BASE-N is a
# prerelease of the NEXT version and semver-sorts ABOVE the last stable
# (0.3.6-N > 0.3.5) — preview users naturally upgrade past stable, and
# the Windows MSI ProductVersion (which strips the prerelease → 0.3.6)
# is also correctly above the last stable.
- name: Stamp preview version
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
shell: bash
@@ -431,6 +454,12 @@ jobs:
# only on the opt-in stable path, leaving them ABSENT (not "") on
# preview/unsigned paths so Tauri's bundler skips cert import. A static
# env: here would always set them to "" and break the mac build.
# Unsigned paths still get a VALID ad-hoc seal from tauri.conf.json
# (bundle.macOS.signingIdentity = "-"), so a downloaded build shows the
# GUI-bypassable "unidentified developer" prompt (right-click → Open /
# Settings → "Open Anyway") instead of the un-bypassable "damaged"
# error. On the signed path APPLE_SIGNING_IDENTITY (env) overrides the
# "-" default; once notarized, Gatekeeper accepts it with no prompt.
# GH runners disable FUSE, so linuxdeploy's AppImage can't mount
# itself at bundle time. This env tells linuxdeploy to extract-and-run
# instead, which works without FUSE.
@@ -481,6 +510,35 @@ jobs:
echo "OK — bundle has shell + uv + backend resources"
hdiutil detach "$MOUNT" || true
# ── Signing / Gatekeeper / notarization verification ──────────────
# Runs codesign --verify, spctl (Gatekeeper), nested-binary, and
# stapler checks against the built .app (see docs/macos-signing-verification.md).
# STRICT (--require-signed) only on the opt-in signed stable path — same
# condition as "Configure Apple signing" above — so a failed or missing
# signature/notarization FAILS the job and STOPS the release instead of
# publishing an unsigned artifact. On every other (unsigned dev/preview)
# path it runs report-only and never breaks the build.
- name: Verify macOS signing
if: runner.os == 'macOS'
shell: bash
env:
STRICT: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && vars.MACOS_SIGNING_ENABLED == 'true') && '1' || '0' }}
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_PASSWORD: ${{ secrets.APPLE_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
run: |
set -uo pipefail
APP=$(find "frontend/src-tauri/target/${{ matrix.rust_target }}/release/bundle/macos" -maxdepth 1 -name '*.app' | head -1)
[ -n "$APP" ] || { echo "FAIL — no .app found to verify"; exit 1; }
MODE=""
if [ "$STRICT" = "1" ]; then
MODE="--require-signed"
echo "Signed stable release → STRICT verification (release stops on failure)."
else
echo "Unsigned dev/preview path → report-only verification."
fi
bash scripts/verify-macos-signing.sh "$APP" $MODE
- name: Installer smoke (Windows)
if: runner.os == 'Windows'
timeout-minutes: 5
@@ -637,3 +695,46 @@ jobs:
} > /tmp/preview-notes.md
gh release edit preview --repo "$REPO" --notes-file /tmp/preview-notes.md
echo "Applied auto-generated release notes + contributors to the preview release."
# ── Post-release version bump (versioning hard rule, owner-set 2026-06-11) ──
# main is always last-release + 1 patch. The moment a stable v* tag is
# released, bump the three version sources on main to the next patch so every
# PR and preview build identifies as the next version. Pushes directly to
# main with the workflow token (a metadata-only commit; CI runs on PRs).
version-bump:
if: github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-')
runs-on: ubuntu-22.04
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
ref: main
fetch-depth: 0
- name: Bump main to released version + 1 patch
shell: bash
run: |
set -euo pipefail
RELEASED="${GITHUB_REF_NAME#v}"
IFS=. read -r MAJ MIN PAT <<< "$RELEASED"
NEXT="$MAJ.$MIN.$((PAT + 1))"
CURRENT=$(jq -r .version frontend/src-tauri/tauri.conf.json)
if [ "$(printf '%s\n' "$NEXT" "$CURRENT" | sort -V | tail -1)" = "$CURRENT" ] && [ "$NEXT" != "$CURRENT" ]; then
echo "main is already at $CURRENT (>= $NEXT) — nothing to bump"; exit 0
fi
tmp=$(mktemp)
jq --arg v "$NEXT" '.version = $v' frontend/src-tauri/tauri.conf.json > "$tmp"
mv "$tmp" frontend/src-tauri/tauri.conf.json
# frontend/package.json drives __APP_VERSION__ (vite.config.js) — the
# first-run footer + every auto bug report. Keep it in lockstep too,
# set absolutely (jq) so any prior drift self-heals. (#248-sweep finding)
tmp=$(mktemp)
jq --arg v "$NEXT" '.version = $v' frontend/package.json > "$tmp"
mv "$tmp" frontend/package.json
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" frontend/src-tauri/Cargo.toml
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" pyproject.toml
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add frontend/src-tauri/tauri.conf.json frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml
git commit -m "chore(version): main -> $NEXT after $GITHUB_REF_NAME release"
git push origin main
+6 -2
View File
@@ -32,9 +32,13 @@ env:
permissions:
contents: read
# PR branches: a new push cancels the superseded scan (no wasted runners).
# main: every commit keeps its own group, so nothing is cancelled — a merge
# train used to leave a permanent red ✗ ("cancelled") on every intermediate
# commit in the history view even though nothing failed.
concurrency:
group: security-${{ github.ref }}
cancel-in-progress: true
group: security-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || 'branch' }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
# ── Secret scanning (gating) ─────────────────────────────────────────────
+9 -2
View File
@@ -1,10 +1,17 @@
# SPIKE-02: Adopt `ModelsLab/omnivoice-singing` as singing variant of the existing engine
**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
**Date:** 2026-05-18
**Status:** ⚠️ **SUPERSEDED (2026-06-14)** by [`specs/006-dubbing-singing-mode/`](../../specs/006-dubbing-singing-mode/spec.md)
**Date:** 2026-05-18 (superseded 2026-06-14)
**Decision-makers:** [maintainer]
**Related:** ROADMAP Phase 4; REQUIREMENTS SING-01..05; `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md`
> **Superseded:** This chose `ModelsLab/omnivoice-singing` for singing, but that
> model has **no melody (F0/MIDI) conditioning** — it sings its own melody and
> cannot follow the *source song* a dub must preserve. SoulX-Singer (arXiv
> 2602.07803, published after this decision) provides F0/MIDI conditioning and is
> selected in plan-06. This ADR stays valid only if reframed as an
> expressive-TTS styling toggle, not melody-matched dubbing.
## Context
`ModelsLab/omnivoice-singing` (HuggingFace, 1,053 downloads/month, verified 2026-05-18) is a finetune of `k2-fsa/OmniVoice` — same Apache-2.0 license, same Qwen3-0.6B backbone, same Higgs Audio v2 codec at 24 kHz mono, same `omnivoice` PyPI library (0.1.5, 2026-04-28) already shipping in OmniVoice Studio v0.2.7. Trained on additional singing + emotion-tagged data and activated by a `[singing]` text control tag at generation time.
@@ -0,0 +1,396 @@
---
phase: 260613-fdl
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- pyproject.toml
- backend/api/routers/setup/download.py
- backend/api/routers/setup/models.py
- backend/utils/hf_progress.py
- backend/utils/download_aggregator.py # NEW
- backend/services/segmented_download.py # NEW
- backend/api/routers/system.py
- frontend/src/pages/Settings.jsx
- frontend/src/api/setup.ts
- docs/downloading-models.md # NEW (docs-sync rule)
- tests/backend/setup/test_download_preflight.py # NEW
- tests/backend/services/test_segmented_download.py # NEW
autonomous: true
requirements:
# ── Wave 0 — Spike / gate ───────────────────────────────────────────────────
- FDL-00 # Classify all catalog repos Xet-backed vs legacy-LFS; the result sizes Wave 3
# ── Wave 1 — Maximize + guarantee the Xet fast path (default, no new deps) ───
- FDL-01 # Explicitly pin huggingface_hub>=1.7 + hf-xet in pyproject (today transitive/unpinned)
- FDL-02 # Drive snapshot_download with explicit max_workers + tqdm_class + endpoint (not implicit monkeypatch)
- FDL-03 # /system/info reports fast_download {xet_enabled, xet_version, high_performance}; logged at startup
- FDL-04 # Opt-in HF_XET_HIGH_PERFORMANCE + HDD sequential-write toggles via prefs (env wins)
# ── Wave 2 — Accurate downloaded/remaining + speed (the user-visible win) ────
- FDL-05 # dry_run preflight -> emit install_plan {total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
- FDL-06 # Backend aggregate tracker -> single 'aggregate' event {bytes_done, total_bytes, rate, eta, files_done/total}
- FDL-07 # Frontend overall bar: speed + downloaded/remaining + ETA from aggregate; per-file detail collapsible; cached-skip shown
# ── Wave 3 — Opt-in IDM-style accelerator for legacy-LFS repos ───────────────
- FDL-08 # Custom httpx segmented downloader: parallel Range GETs, resume, auth-safe redirect, etag/sha verify, cancel (default OFF)
- FDL-09 # Dispatch: accelerator ON + repo is LFS (not Xet) -> segmented path; else xet. Same aggregate progress + weight validation
# ── Wave 4 — Opt-in mirror path + docs ───────────────────────────────────────
- FDL-10 # Opt-in HF_ENDPOINT mirror setting (prefs); documented as classic-LFS fallback (no Xet); pairs with FDL-08
- FDL-11 # Cancel-in-flight endpoint + cooldown interplay (composes with MM2-06 bounded cooldowns)
- FDL-12 # docs/downloading-models.md (speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network) + README pointer
must_haves:
truths:
- "Xet is the default download backend and is provably engaged: /system/info reports fast_download.xet_enabled=true with the hf_xet version, and a Xet-backed repo downloads via parallel chunk range-gets (not single-stream LFS)."
- "Before any bytes flow, the UI shows an accurate denominator: total bytes to download, bytes already cached (skipped), and file count — sourced from snapshot_download(dry_run=True), not guessed from the first tqdm bar."
- "During a download the UI shows ONE overall progress bar with instantaneous speed (sampled over a window, not a single file's rate), bytes downloaded / bytes remaining, and ETA — accurate even while Xet fetches many chunks/files in parallel."
- "hf_transfer is NOT used or enabled anywhere (deprecated, breaks progress); the fast path is Xet only."
- "The custom segmented downloader is OPT-IN (default off), only engages for non-Xet/legacy-LFS repos, never forwards the HF Authorization header to the redirected CDN host, verifies the downloaded file against its expected size/etag before marking complete, resumes a partial .part file, and can be cancelled mid-flight."
- "Default download behavior is identical on macOS, Windows, Linux (Xet path, pure-Python). Every accelerator/mirror/high-perf knob is behind an explicit opt-in (Settings toggle or env var) per the cross-platform-parity strict rule — no bundled per-OS binary, no platform-divergent default."
- "No new on-disk model-state format; existing HF cache layout and already-installed models are untouched; the segmented downloader writes into the same HF cache blob/snapshot structure (or hands off to it) so a model it fetches is indistinguishable from one snapshot_download fetched."
- "uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py passes; existing download/install tests stay green."
- "pyproject pins huggingface_hub>=1.7 and hf-xet explicitly; uv.lock resolves with single versions (uv tree shows no duplicate huggingface_hub)."
artifacts:
- path: "backend/utils/download_aggregator.py"
provides: "Per-repo byte aggregator: sums bytes across parallel files/chunks, samples rate over a window, emits one 'aggregate' event"
contains: "class DownloadAggregator AND def snapshot"
- path: "backend/services/segmented_download.py"
provides: "Opt-in multi-connection Range downloader for legacy-LFS repos (auth-safe, resume, verify, cancel)"
contains: "async def segmented_download AND Range"
- path: "backend/api/routers/setup/download.py"
provides: "Driven snapshot_download (max_workers+tqdm_class+endpoint), dry_run preflight, dispatch to segmented path, cancel endpoint"
contains: "dry_run AND tqdm_class"
- path: "docs/downloading-models.md"
provides: "User docs for download speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network"
contains: "Xet"
key_links:
- from: "install_model (download.py:122)"
to: "snapshot_download(dry_run=True) preflight"
via: "compute total/cached/remaining before the real download; emit 'install_plan'"
pattern: "dry_run\\s*=\\s*True"
- from: "snapshot_download / segmented_download byte updates"
to: "DownloadAggregator -> single 'aggregate' SSE event"
via: "tqdm_class forwards bytes into the aggregator; segmented path calls aggregator.add() directly"
pattern: "aggregate"
- from: "dispatch in install_model"
to: "segmented_download vs snapshot_download"
via: "prefs accelerator toggle AND repo-is-LFS classification (FDL-00 helper)"
pattern: "segmented_download"
- from: "system_info (system.py:245)"
to: "fast_download status block"
via: "probe hf_xet import + version + HF_XET_HIGH_PERFORMANCE"
pattern: "fast_download"
---
<objective>
Make model downloads as fast as possible AND show accurate speed / downloaded / remaining / ETA.
**Framing (validated by research — see 260613-fdl-RESEARCH below):** HuggingFace's **hf-xet** backend ALREADY implements the "IDM/uGet technique" — content-defined chunking, parallel byte-range fetches with adaptive concurrency, dedup, and automatic resume — and does it auth-safely. It ships by default in modern `huggingface_hub` and `hf_xet` is already installed here (huggingface_hub 1.7.2). HF closed the multi-connection-downloader feature request as "solved by Xet." So we do NOT build a custom segmented downloader as the default path; that would be redundant and would violate the cross-platform-parity rule.
What's actually missing:
1. **We don't drive Xet well.** `install_model` calls `snapshot_download(**dl_kwargs)` with no `max_workers`, no `tqdm_class`, no `dry_run`, and no explicit dependency pin — progress rides on a global tqdm monkeypatch.
2. **No pre-flight total**, so "downloaded/remaining" has no denominator until files appear, and aggregate speed is summed frontend-side from per-file events (inaccurate under parallel fetch).
3. **Legacy non-Xet (LFS) repos get zero intra-file parallelism** — this is the one place a real IDM-style multi-connection fetch still helps, so we add it as an OPT-IN accelerator.
Five waves, in order (each independently shippable, continuous-to-main per v0.3.0 cadence):
- **Wave 0 — Spike/gate (FDL-00):** classify every catalog repo Xet vs LFS. Sizes Wave 3's value; if ~all repos are Xet-backed, Wave 3 is low-priority polish.
- **Wave 1 — Maximize + guarantee Xet (FDL-01..04):** pin deps, drive snapshot_download explicitly, surface fast-download status, opt-in high-perf/HDD knobs. No new deps, all platforms.
- **Wave 2 — Accurate progress (FDL-05..07):** dry_run preflight + backend aggregate tracker + overall UI bar (speed/remaining/ETA). The biggest user-visible win.
- **Wave 3 — Opt-in segmented accelerator (FDL-08..09):** custom httpx Range downloader for LFS repos. Default OFF, opt-in toggle.
- **Wave 4 — Mirror path + docs (FDL-10..12):** opt-in HF_ENDPOINT, cancel endpoint, docs-sync.
Out of scope / explicitly rejected (call out, do NOT do):
- **hf_transfer / HF_HUB_ENABLE_HF_TRANSFER** — deprecated, breaks progress callbacks. Never enable.
- **Bundling aria2c** — per-OS GPLv2 binary + parity burden; the custom httpx path covers the same need without a binary.
- **Making the segmented downloader the default** — redundant vs Xet, violates parity rule. Always opt-in.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@./CLAUDE.md
@.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-RESEARCH.md
# Files under edit (read before editing)
@backend/api/routers/setup/download.py
@backend/api/routers/setup/models.py
@backend/utils/hf_progress.py
@backend/api/routers/system.py
@frontend/src/pages/Settings.jsx
# Reference only — patterns, do NOT modify
@backend/core/prefs.py
@frontend/src/api/setup.ts
@frontend/src/api/hooks.ts
<interfaces>
<!-- Verified during planning against the live env (huggingface_hub 1.7.2, hf_xet installed). -->
huggingface_hub 1.7.2 snapshot_download params (confirmed via inspect):
repo_id, repo_type, revision, cache_dir, local_dir, library_name, library_version,
user_agent, etag_timeout, force_download, token, local_files_only,
allow_patterns, ignore_patterns, max_workers, tqdm_class, headers, endpoint, dry_run
- dry_run=True -> returns per-file info incl. size + cached/not-cached (use for FDL-05 preflight).
- tqdm_class=<cls> -> drives the AGGREGATE bar; Xet feeds bytes into it (this is the xet-aware progress hook).
- max_workers -> parallel FILES (default 8); orthogonal to Xet intra-file chunk parallelism.
- endpoint -> per-call HF endpoint override (FDL-10 mirror, instead of process-wide HF_ENDPOINT).
backend/utils/hf_progress.py (existing):
- Monkeypatches huggingface_hub.utils.tqdm.tqdm -> TrackedTqdm (install() at startup).
- register_listener/unregister_listener; emit(event); current_repo_id contextvar stamps events.
- TrackedTqdm.update()/display() emit per-file {filename, downloaded, total, pct, rate, phase} throttled ~0.3s.
- GAP: per-file only, no aggregate, no preflight total. Wave 2 adds the aggregator on top (keep TrackedTqdm; feed it).
backend/api/routers/setup/download.py (existing):
- install_model (line 122): snapshot_download(**dl_kwargs) inside asyncio.to_thread; 5-retry backoff; heartbeat;
_validate_snapshot_has_weights (line 55); _install_cooldowns (line 27, see MM2-06 for bounding).
- SSE feed: GET /setup/download-stream (line 80) forwards hf_progress events.
backend/core/prefs.py:
- resolve(key, *, env=None, default=None) (line 75) — env wins, then store, then default. Use for all new toggles.
Xet env knobs (research): HF_XET_HIGH_PERFORMANCE=1 (opt-in max throughput; needs RAM/bandwidth),
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 (HDD), HF_XET_NUM_CONCURRENT_RANGE_GETS (default 16),
HF_XET_DATA_PROGRESS_UPDATE_INTERVAL (200ms). hf_xet is 64-bit only.
</interfaces>
</context>
<tasks>
<!-- ════════════ WAVE 0 — SPIKE / GATE ════════════ -->
<task type="auto">
<name>Task 0 (FDL-00): Classify catalog repos Xet vs LFS</name>
<files>.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md</files>
<action>
For every repo in backend/config/models.yaml (25 entries), determine whether it's Xet-backed or legacy Git-LFS. Use huggingface_hub: `HfApi().repo_info(repo_id, files_metadata=True)` and inspect each LFS blob for xet info, OR call the model-info endpoint and check the `xetEnabled`/blob `xet` field. For gated/unavailable repos, record "unknown (gated/offline)".
Write 260613-fdl-SPIKE.md: a table repo_id | role | backend (xet|lfs|unknown) | size, plus a one-line GO/LOW-PRIORITY verdict for Wave 3:
- If the majority of *user-facing default* models (OmniVoice TTS, the default ASR) are Xet-backed -> Wave 3 is LOW priority (xet already fast); still build it for the LFS long tail.
- If many defaults are still LFS -> Wave 3 is HIGH priority.
This is read-only network classification — do not download anything (use repo_info, not snapshot_download).
</action>
<verify>
<automated>test -f .planning/quick/260613-fast-model-downloads/260613-fdl-SPIKE.md || test -f .planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md && echo "spike written"</automated>
</verify>
<done>SPIKE.md lists every catalog repo with its storage backend and a GO/LOW-PRIORITY verdict for Wave 3.</done>
</task>
<!-- ════════════ WAVE 1 — MAXIMIZE + GUARANTEE THE XET FAST PATH ════════════ -->
<task type="auto">
<name>Task 1 (FDL-01): Pin huggingface_hub + hf-xet explicitly</name>
<files>pyproject.toml</files>
<action>
Today huggingface_hub arrives transitively (1.7.2) and hf_xet is present but unpinned. Add explicit runtime pins so the fast path can never silently disappear on a resolve:
- huggingface_hub>=1.7 (keep compatible with transformers>=5.3.0 already in deps)
- hf-xet>=1.1 (the Xet backend; 64-bit only — fine for all OmniVoice targets)
Do NOT add hf_transfer. Run `uv sync` then `uv tree huggingface_hub` to confirm a single resolved version (no duplicate). If a transitive constraint conflicts, prefer the higher version and note it in the SUMMARY.
</action>
<verify>
<automated>grep -n "huggingface_hub\|hf-xet\|hf_xet\|hf-transfer\|hf_transfer" pyproject.toml</automated>
<automated>uv run python -c "import huggingface_hub,hf_xet; print('hub',huggingface_hub.__version__,'xet ok')"</automated>
</verify>
<done>pyproject pins huggingface_hub>=1.7 and hf-xet; no hf_transfer; uv resolves cleanly with one huggingface_hub.</done>
</task>
<task type="auto">
<name>Task 2 (FDL-02): Drive snapshot_download explicitly</name>
<files>backend/api/routers/setup/download.py</files>
<action>
In install_model's _do() (line ~148), build dl_kwargs with explicit, intentional args instead of the bare call:
- tqdm_class=<the TrackedTqdm class> so progress is deterministic and xet-aware rather than relying solely on the global monkeypatch. Expose TrackedTqdm from hf_progress (add a getter, e.g. hf_progress.tracked_tqdm_class()).
- max_workers: keep default 8 (don't crank — xet does intra-file parallelism; high max_workers multiplies buffer pressure). Make it prefs-overridable: prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8).
- endpoint=prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None) — wires FDL-10 mirror without process-wide env.
- Keep the existing 5-retry backoff, heartbeat, and _validate_snapshot_has_weights.
Do not remove the global monkeypatch (other libs — transformers/mlx_whisper — still rely on it); this task just makes the install path drive its own tqdm_class explicitly.
</action>
<verify>
<automated>grep -n "tqdm_class\|max_workers\|endpoint" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/ -k "download or install" -q 2>&amp;1 | tail -15</automated>
</verify>
<done>install_model drives snapshot_download with explicit tqdm_class + max_workers + endpoint; retry/validate intact; tests green.</done>
</task>
<task type="auto">
<name>Task 3 (FDL-03, FDL-04): fast_download status + opt-in xet knobs</name>
<files>backend/api/routers/system.py, backend/api/routers/setup/download.py</files>
<action>
- FDL-03: add a fast_download block to GET /system/info (system.py:245): {xet_enabled: bool, xet_version: str|None, high_performance: bool}. Probe by importing hf_xet (xet_enabled), reading its version, and reading the HF_XET_HIGH_PERFORMANCE env/pref. Must never throw (system_info is called on every Settings load). Log the same line once at startup ("fast download: Xet on (hf_xet X.Y), high_perf=...").
- FDL-04: opt-in knobs via prefs, applied at process/download setup (env wins):
high_performance = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
hdd_sequential = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
When set, export the corresponding HF_XET_* env before the snapshot/segmented download runs. Both default OFF (high-perf can hurt low-RAM machines — surface that as a tooltip in Wave 2 UI).
</action>
<verify>
<automated>curl -s http://127.0.0.1:3900/system/info | python3 -c "import json,sys; print(json.load(sys.stdin).get('fast_download'))" 2>/dev/null || grep -n "fast_download" backend/api/routers/system.py</automated>
</verify>
<done>/system/info reports fast_download truthfully; high-perf + HDD knobs resolve via prefs with env precedence, default off; startup logs Xet status.</done>
</task>
<!-- ════════════ WAVE 2 — ACCURATE DOWNLOADED/REMAINING + SPEED ════════════ -->
<task type="auto">
<name>Task 4 (FDL-05): dry_run preflight -> install_plan event</name>
<files>backend/api/routers/setup/download.py</files>
<action>
Before the real download in install_model, run snapshot_download(repo_id, dry_run=True, endpoint=...) on the worker thread. From the returned per-file info compute: total_bytes, cached_bytes (files already present), to_download_bytes, n_files, n_cached. Emit a new phase event:
{repo_id, phase:"install_plan", total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
This gives the UI an accurate denominator and a "M GB already cached, N GB to download" line BEFORE bytes flow. Wrap dry_run in try/except — if it fails (older/gated repo), emit install_plan with totals=None and fall back to today's behavior (denominator fills in as files appear). dry_run must respect the 'resolving' heartbeat (it can take a couple seconds).
</action>
<verify>
<automated>grep -n "dry_run\|install_plan\|to_download_bytes" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/backend/setup/test_download_preflight.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>An install emits install_plan with accurate total/cached/remaining before download; dry_run failure degrades gracefully to old behavior.</done>
</task>
<task type="auto">
<name>Task 5 (FDL-06): Backend aggregate progress tracker</name>
<files>backend/utils/download_aggregator.py, backend/utils/hf_progress.py</files>
<action>
New backend/utils/download_aggregator.py: a per-repo DownloadAggregator that owns the TRUTH for overall progress, so the frontend stops summing potentially-misrouted per-file events.
- Seeded by the install_plan totals (total_bytes, n_files).
- add(filename, bytes_delta) / set_file(filename, downloaded, total): track bytes per file; bytes_done = sum.
- Rate: sampled over a sliding window (e.g. last ~5-10s of (t, bytes_done) samples), not a single tqdm bar's rate. eta = remaining / rate.
- snapshot() -> {repo_id, bytes_done, total_bytes, rate, eta_seconds, files_done, files_total, phase}.
- Emits one throttled (~0.3-0.5s) phase:"aggregate" event via hf_progress.emit().
Wire it: hf_progress's TrackedTqdm._emit_progress already has per-file (filename, downloaded, total) — also feed those into the active repo's aggregator (look up by current_repo_id). The segmented downloader (Wave 3) calls aggregator.add() directly. Keep the per-file events too (UI detail view) — aggregate is additive, not a replacement.
</action>
<verify>
<automated>uv run python -c "from utils.download_aggregator import DownloadAggregator as A; a=A('r',total_bytes=100,files_total=2); a.set_file('f1',50,50); a.set_file('f2',25,50); s=a.snapshot(); print(s['bytes_done'], s['total_bytes'])"</automated>
</verify>
<done>DownloadAggregator sums bytes across parallel files, samples rate over a window, emits a single 'aggregate' event; fed by both tqdm and the segmented path.</done>
</task>
<task type="auto">
<name>Task 6 (FDL-07): Frontend overall progress bar</name>
<files>frontend/src/pages/Settings.jsx, frontend/src/api/setup.ts</files>
<action>
- setup.ts: extend SetupProgressEvent phase union with "install_plan" | "aggregate" and their fields (total_bytes, cached_bytes, to_download_bytes, n_files, n_cached, bytes_done, rate, eta_seconds, files_done, files_total).
- Settings.jsx ModelStoreTab: when an aggregate event arrives for a repo, render ONE overall progress row: a bar (bytes_done/total_bytes), instantaneous speed (format rate as MB/s), "X.X GB of Y.Y GB" downloaded/remaining, and ETA (mm:ss from eta_seconds). Seed the denominator from install_plan (show "M GB cached, N GB to download" before bytes flow). Keep the existing per-file rows as a collapsible "details" section instead of the primary display. Show a small "⚡ fast download" badge when /system/info fast_download.xet_enabled is true.
- Prefer the backend aggregate's rate/eta over the frontend's own per-file ETA computation (Settings.jsx ~614-631) — replace that local ETA math with the aggregate fields; keep a fallback if no aggregate event has arrived yet.
</action>
<verify>
<automated>cd frontend && bun run typecheck 2>&amp;1 | tail -15</automated>
<automated>grep -n "aggregate\|install_plan\|eta_seconds\|fast download" frontend/src/pages/Settings.jsx frontend/src/api/setup.ts</automated>
</verify>
<done>UI shows one overall bar with live speed + downloaded/remaining + ETA from the aggregate event; per-file detail collapsible; fast-download badge; typecheck passes.</done>
</task>
<!-- ════════════ WAVE 3 — OPT-IN IDM-STYLE SEGMENTED ACCELERATOR (LFS REPOS) ════════════ -->
<task type="auto">
<name>Task 7 (FDL-08): Custom httpx segmented downloader</name>
<files>backend/services/segmented_download.py, tests/backend/services/test_segmented_download.py</files>
<action>
New backend/services/segmented_download.py — an OPT-IN multi-connection Range downloader for ONE file (the IDM/uGet technique) used only for legacy-LFS repos where Xet gives no intra-file parallelism. httpx is already a dep.
Contract (async def segmented_download(url, dest, *, token, expected_size, expected_etag=None, num_connections=8, chunk_aggregator=None, cancel_event=None)):
1. HEAD (or GET Range: bytes=0-0) the resolve URL to learn size + Accept-Ranges + the redirect target. If server doesn't honor Range (Accept-Ranges != bytes) -> fall back to a single streamed GET (still works, just not parallel).
2. AUTH SAFETY (critical): send Authorization: Bearer <token> ONLY to the huggingface.co host. When the resolve URL 302-redirects to the CDN (cloudfront/etc.), do NOT forward Authorization to the CDN host — the presigned URL already carries auth. Follow redirects manually so you control header propagation per-host.
3. Split expected_size into num_connections ranges; download each with Range: bytes=start-end concurrently (asyncio + httpx.AsyncClient). Write to dest+".part" at the right offsets (preallocate, or per-range temp files then concat).
4. RESUME: if dest+".part" exists with a sidecar manifest of completed ranges, skip completed ranges.
5. CANCEL: check cancel_event between chunks; on cancel, leave the .part for resume and raise CancelledError.
6. VERIFY: after assembly, check size == expected_size and (if given) sha256/etag matches; only then atomically rename .part -> dest. On mismatch, raise (caller's retry/validate handles it).
7. PROGRESS: call chunk_aggregator.add(filename, bytes_delta) as ranges complete bytes (feeds DownloadAggregator).
Tests (use a local mock HTTP server / httpx MockTransport): honors Range + parallel assembly == single-GET bytes; falls back when Accept-Ranges absent; does NOT send Authorization to a different host on redirect; resumes from a partial .part; cancels and leaves resumable state; size/etag mismatch raises.
</action>
<verify>
<automated>uv run pytest tests/backend/services/test_segmented_download.py -q 2>&amp;1 | tail -20</automated>
<automated>grep -n "Authorization\|Range\|cancel_event\|expected_size" backend/services/segmented_download.py</automated>
</verify>
<done>segmented_download fetches a file via parallel ranges, is auth-safe across the CDN redirect, resumes, cancels, and verifies size/etag before commit; all tests pass.</done>
</task>
<task type="auto">
<name>Task 8 (FDL-09): Dispatch — accelerator for LFS repos only</name>
<files>backend/api/routers/setup/download.py, backend/api/routers/setup/models.py</files>
<action>
- models.py: add a small helper is_xet_backed(repo_id) -> bool|None (reuse FDL-00's classification approach; cache result). Used to decide the path.
- download.py install_model dispatch:
accelerator_on = prefs.resolve("segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False)
if accelerator_on and is_xet_backed(repo_id) is False:
-> resolve each LFS file's URL via hf_hub_url + HfApi file metadata, download via segmented_download into the HF cache layout (or download to a temp dir then place via the cache API so the result is a normal cache entry), feeding the same DownloadAggregator. Run _validate_snapshot_has_weights at the end.
else:
-> existing snapshot_download path (xet).
IMPORTANT: the segmented result MUST land in the same HF cache structure so /models install-state, delete, and is_cached() all keep working (truth: "indistinguishable from snapshot_download"). If matching the blob/snapshot symlink layout is too fiddly, the safe fallback is: segmented-download to a temp file, then hand the bytes to huggingface_hub so it finalizes the cache entry. Document the chosen approach in SUMMARY.
Default OFF -> zero behavior change unless the user opts in.
</action>
<verify>
<automated>grep -n "segmented_downloader\|is_xet_backed\|segmented_download" backend/api/routers/setup/download.py backend/api/routers/setup/models.py</automated>
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&amp;1 | tail -20</automated>
</verify>
<done>With the toggle ON, LFS repos download via the segmented path into the normal HF cache; Xet repos and the default (toggle OFF) use snapshot_download; install-state/delete unaffected.</done>
</task>
<!-- ════════════ WAVE 4 — MIRROR PATH + CANCEL + DOCS ════════════ -->
<task type="auto">
<name>Task 9 (FDL-10, FDL-11): Mirror opt-in + cancel endpoint</name>
<files>backend/api/routers/setup/download.py</files>
<action>
- FDL-10: the endpoint= wiring from Task 2 already reads prefs hf_endpoint. Surface it as a setting and document (Task 10) that a mirror routes through the CLASSIC LFS path (no Xet) — so it pairs naturally with the FDL-08 segmented accelerator for speed on mirrors. No process-wide HF_ENDPOINT mutation; per-call endpoint only.
- FDL-11: add POST /models/install/cancel {repo_id} that sets the repo's cancel_event (segmented path) and, for the snapshot path, best-effort marks the install cancelled (snapshot_download isn't trivially cancellable mid-file — at minimum stop retries and emit install_cancelled). Compose with MM2-06: on success OR cancel, clear the _install_cooldowns entry so a cancelled download isn't rate-limited. Emit phase:"install_cancelled".
</action>
<verify>
<automated>grep -n "install/cancel\|cancel_event\|install_cancelled\|hf_endpoint" backend/api/routers/setup/download.py</automated>
</verify>
<done>Per-call mirror endpoint wired (opt-in); cancel endpoint stops the segmented path and clears cooldown; emits install_cancelled.</done>
</task>
<task type="auto">
<name>Task 10 (FDL-12): Docs — downloading-models.md + README pointer</name>
<files>docs/downloading-models.md, README.md</files>
<action>
Per the docs-sync hard rule, document the user-facing surface introduced here:
- How fast downloads work (Xet on by default; what the ⚡ badge means; how to check via Settings/system info).
- Advanced toggles: high-performance mode (warn: needs RAM/bandwidth, can hurt low-RAM machines), HDD sequential-write, max workers, segmented accelerator (opt-in, for legacy-LFS repos), and the mirror/restricted-network HF_ENDPOINT setting (note: mirror = classic LFS, no Xet; pair with the accelerator).
- A short troubleshooting section (slow downloads, stuck at resolving, restricted networks/China).
Add a one-line pointer from README.md to docs/downloading-models.md. Do NOT enable any opt-in by default in docs examples.
</action>
<verify>
<automated>test -f docs/downloading-models.md && grep -n "Xet\|HF_ENDPOINT\|high-performance\|segmented" docs/downloading-models.md | head</automated>
<automated>grep -n "downloading-models" README.md</automated>
</verify>
<done>docs/downloading-models.md covers speed, status, all opt-in knobs, mirror/restricted-network, troubleshooting; README links it; no opt-in shown as default.</done>
</task>
</tasks>
<verification>
Gate per wave; full set before the last PR:
1. `uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py tests/ -k "download or install or model" -q` — green.
2. Live smoke (backend running): an install emits install_plan (accurate total/cached/remaining) THEN aggregate events with rising bytes_done + a non-zero rate + decreasing ETA; on completion bytes_done == total_bytes.
3. /system/info reports fast_download.xet_enabled=true with a version.
4. Auth-safety unit test proves Authorization is NOT sent to a non-huggingface.co host on redirect.
5. Default-off proof: with no opt-in set, an install uses snapshot_download (xet) — `OMNIVOICE_SEGMENTED_DOWNLOAD` unset means the segmented path is never taken.
6. `uv tree huggingface_hub` shows one version; no hf_transfer anywhere (`grep -ri hf_transfer backend/` is empty).
7. `cd frontend && bun run typecheck` passes.
8. Cross-platform parity: the default path (Xet, pure-Python) is identical on all 3 OSes; every accelerator/mirror/high-perf knob is opt-in (Settings/env). No bundled binary added.
</verification>
<success_criteria>
- Fast: Xet is pinned, engaged, and driven with explicit args; high-perf/HDD knobs available opt-in; legacy-LFS repos can use the opt-in segmented accelerator for real multi-connection speed.
- Accurate: UI shows pre-flight total/cached/remaining, then one overall bar with live speed + downloaded/remaining + ETA sourced from a backend aggregate (not frontend guesswork).
- Safe & compatible: no hf_transfer; segmented downloader is opt-in, auth-safe, resumable, verified, cancellable, and lands in the normal HF cache; default behavior identical on all 3 OSes; no new on-disk model state; existing installs untouched.
- All listed tests + typecheck pass; docs updated in the same PR (docs-sync rule).
</success_criteria>
<risks>
- **Segmented downloader auth leak (FDL-08) — highest risk.** Forwarding the HF Authorization header to the CDN host on redirect would leak the token. Mitigation: manual redirect handling, per-host header allow-list (Authorization only to huggingface.co), and a dedicated unit test asserting no Authorization on the CDN hop. This is a must-have truth, not optional.
- **Cache-layout mismatch (FDL-09).** If the segmented path writes files outside the HF cache blob/snapshot structure, /models install-state + delete + is_cached() break. Mitigation: prefer the temp-file-then-hand-to-huggingface_hub finalization approach over hand-rolling the symlink/blob layout; assert is_cached(repo_id) is true after a segmented install in a test.
- **dry_run cost/availability (FDL-05).** dry_run adds a metadata round-trip and may not exist for gated/older repos. Mitigation: try/except -> totals=None fallback to current fill-in-as-you-go behavior; keep the resolving heartbeat so the UI isn't blank during preflight.
- **Aggregate vs per-file double-count (FDL-06).** Feeding both tqdm per-file events and the aggregator risks the UI showing two competing numbers. Mitigation: aggregate is the single source of truth for the overall bar; per-file events only drive the collapsible detail view; the frontend's old per-file ETA math is removed (Task 6).
- **High-performance mode hurting low-RAM machines (FDL-04).** HF_XET_HIGH_PERFORMANCE can need ~tens of GB RAM. Mitigation: default OFF, opt-in only, tooltip warning in the UI.
- **Mirror + Xet confusion (FDL-10).** Users may expect Xet speed through a mirror; mirrors fall back to classic LFS. Mitigation: document explicitly; that's exactly why the segmented accelerator pairs with the mirror path.
- **Scope: do not let the segmented path become default.** It's opt-in for LFS repos only. Xet stays the default; making it default would regress dedup + violate the parity rule.
</risks>
<output>
Write 260613-fdl-SPIKE.md (Task 0) and 260613-fdl-SUMMARY.md when done. SUMMARY must record: the Xet-vs-LFS catalog breakdown and how it changed Wave 3 priority; the cache-finalization approach chosen for the segmented path (and the is_cached-after-segmented test result); the exact new SSE event shapes (install_plan, aggregate); which opt-in prefs keys + env vars were added; and the auth-safety test output. Note any "use judgment" decision an executor made.
Docs-sync (CLAUDE.md hard rule): docs/downloading-models.md + README pointer ship in the SAME PR as the user-facing toggles (Task 10). If the Settings UI gains the new toggles, the docs describing them land together.
</output>
@@ -0,0 +1,43 @@
# RESEARCH — Fast HuggingFace model downloads (2026)
**Date:** 2026-06-13 · **For:** 260613-fdl-PLAN.md
## Bottom line
As of mid-2026 the fast path is **hf-xet, on by default** in modern `huggingface_hub`. Xet is itself a chunk-level, content-defined, massively-parallel downloader with adaptive concurrency — it **is** the "IDM/uGet-style segmented download," done for you and dedup-aware. `hf_transfer` is **deprecated**. Rolling your own segmented downloader or bridging to aria2 is **not worth it as a default**; the only thing we must build is (a) better driving + progress UI and (b) an **opt-in** segmented path for the legacy-LFS long tail (repos Xet doesn't back).
Installed in this repo: `huggingface_hub 1.7.2`, `hf_xet` present. `snapshot_download` here supports `max_workers`, `tqdm_class`, `endpoint`, `dry_run` (confirmed via inspect).
## 1. hf-xet — USE (default, no action needed beyond pinning)
Content-defined chunks grouped into blocks ("xorbs") in a content-addressable store; download = send file SHA256 → get reconstruction metadata + presigned URLs → fetch needed xorb ranges **in parallel** → reassemble; already-present chunks skipped (dedup). Auto-used by `snapshot_download`/`hf_hub_download` for Xet-backed repos since huggingface_hub 0.32. 23× over Git-LFS, up to ~1 GB/s.
Knobs (defaults already tuned): `HF_XET_NUM_CONCURRENT_RANGE_GETS` (16), adaptive concurrency ON (max 64), `HF_XET_DATA_MAX_CONCURRENT_FILE_DOWNLOADS` (8), chunk cache disabled by default (better for pure download), `HF_XET_HIGH_PERFORMANCE=1` (opt-in max throughput, needs RAM/bandwidth), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` (HDD). **64-bit only.**
- https://huggingface.co/docs/huggingface_hub/en/guides/download
- https://huggingface.co/docs/hub/en/xet/using-xet-storage
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
## 2. hf_transfer — AVOID (deprecated)
`HF_HUB_ENABLE_HF_TRANSFER` flagged deprecated; Xet supersedes it. Historically **broke tqdm progress / had no callbacks** — directly conflicts with the accurate-progress goal. Successor for max throughput is `HF_XET_HIGH_PERFORMANCE=1`.
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
- https://github.com/huggingface/hf_transfer/issues/63
## 3. huggingface_hub native concurrency — USE defaults
`snapshot_download(max_workers=...)` = parallel FILES (default 8), orthogonal to Xet's intra-file chunk parallelism. For OmniVoice's 1few-large-file models the win is mostly Xet's intra-file parallelism; don't crank max_workers (multiplies buffer pressure). Resume is automatic via cache + ETag (no `resume_download` flag to manage).
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
## 4. Custom IDM-style Range downloader — AVOID as default, BUILD as opt-in for LFS
`/resolve/<rev>/<file>` 302-redirects to CDN (Cloudfront) which honors Range + parallel byte-ranges. Catch: follow redirect, **do NOT forward Authorization to the CDN host** (presigned URL carries auth), verify ETag/sha256, auth on first hop only. Redundant vs Xet for Xet-backed repos (HF closed issue #3232 as "use Xet"), **but genuinely helps non-Xet/legacy-LFS repos** which get no intra-file parallelism. → our Wave 3 opt-in.
- https://github.com/huggingface/huggingface_hub/issues/3232
## 5. aria2 — OPTIONAL, rejected for OmniVoice
`aria2c -x16 -s16 -c --header="Authorization: Bearer <token>"` is 35× on plain LFS, but: no dedup (worse than Xet for Xet repos), per-OS GPLv2 binary to package (parity burden — would have to be opt-in anyway), stdout/RPC progress scraping. The custom httpx path covers the same need with no binary. → not bundled.
- https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f
## 6. Mirrors / HF_ENDPOINT — OPTIONAL, region-gated, breaks Xet
`HF_ENDPOINT=https://hf-mirror.com` redirects Hub traffic (standard for China). **Xet CAS/presigned URLs point at HF infra → mirrors generally don't serve the Xet protocol → traffic falls back to classic LFS** (no dedup, no Xet parallelism). So mirror and Xet fast-path are mutually exclusive; the realistic China stack is mirror + LFS + (our opt-in) segmented accelerator. → our Wave 4 opt-in, per-call `endpoint=` not process-wide.
## 7. Progress / speed — USE `tqdm_class` (xet-aware) + `dry_run` preflight
Unlike hf_transfer, **Xet reports progress through the same tqdm interface**; huggingface_hub aggregates per-file/thread bytes into a shared bar and feeds the `tqdm_class` you pass. So `snapshot_download(tqdm_class=...)` yields reliable aggregate bytes/total/rate/ETA even under parallel fetch. `snapshot_download(dry_run=True)` returns per-file sizes + cached flags → use for "will download X of Y, N GB" preflight. Speed sampling tunable via `HF_XET_DATA_PROGRESS_UPDATE_INTERVAL` (200ms).
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
- https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/_snapshot_download.py
## Recommended architecture (→ plan)
Pin `huggingface_hub>=1.7` + `hf-xet`; let Xet be the default (it IS the IDM technique). Drive `snapshot_download(repo_id, tqdm_class=OmniVoiceProgress, max_workers=8, endpoint=<opt-in mirror>)`; `dry_run=True` first for total/remaining; aggregate bytes in a backend tracker → one overall bar (speed/remaining/ETA). Opt-in only: `HF_XET_HIGH_PERFORMANCE` (max speed), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (HDD), a custom httpx **segmented downloader for legacy-LFS repos**, and an `HF_ENDPOINT` mirror (classic-LFS fallback). Never enable hf_transfer; never bundle aria2; never make the segmented path the default.
@@ -0,0 +1,28 @@
# SPIKE — FDL-00: Catalog Xet vs LFS classification
**Date:** 2026-06-13 · **Method:** HF API `GET /api/models/{repo}?expand[]=xetEnabled` (authoritative).
## Result: 25 / 25 catalog repos are Xet-backed
| backend | count |
|---------|-------|
| xet | 25 |
| lfs | 0 |
| unknown | 0 |
Every repo in `backend/config/models.yaml` — including both first-run defaults (`k2-fsa/OmniVoice` TTS, `Systran/faster-whisper-large-v3` ASR) — returns `xetEnabled: true`. Full list: all entries under TTS / ASR / Diarisation (k2-fsa, Systran×5, mlx-community×9, openai, nvidia×2, UsefulSensors×2, pyannote, OpenMOSS, KittenML, deepdml).
## Detection caveat (important for the executor)
The installed client is **huggingface_hub 1.7.2**, whose `repo_info(..., files_metadata=True)` siblings expose only `blob_id, lfs, rfilename, size`**no `xet_file`, and no `xet_enabled` on the info object.** A first pass that inferred backend from siblings wrongly reported "0/25 xet, all LFS." Do **not** classify Xet status from `repo_info` siblings on this client version. The reliable signal is the Hub API `xetEnabled` expand field (used here) or `hf_xet` actually engaging at download time. Re-check after any `huggingface_hub` bump — newer versions surface `xet_enabled` directly.
## Verdict for Wave 3 (segmented accelerator): LOW priority
Because the entire current catalog is Xet-backed and `hf_xet` is installed, Xet already provides chunked parallel range-gets (the IDM/uGet behavior) for **every** model we ship. The custom segmented downloader (Wave 3) is therefore **not needed to speed up any current default model** — it remains valuable only for:
- the **mirror / restricted-network path** (Wave 4: `HF_ENDPOINT` falls back to classic LFS, no Xet), and
- any **future non-Xet repo** a user adds.
**Recommendation:** proceed with W1 (maximize/guarantee Xet) and W2 (accurate progress) as the real wins for today's catalog; keep W3 as opt-in, build it alongside W4's mirror path where it actually pays off. This matches the PLAN's original framing — confirmed, not changed.
## Consequence for W1/W2 framing
W1 "guarantee the Xet fast path" is correctly the primary lever: these repos download via Xet **only if** the client engages it (hf_xet installed ✓ + huggingface_hub recent ✓). The W2 live smoke test should confirm Xet is actually used (fast parallel aggregate progress on a real install), since `xetEnabled=true` is a Hub-side capability, not proof the client took the Xet path.
@@ -0,0 +1,49 @@
# SUMMARY — FDL Waves 02 (fast model downloads)
**Date:** 2026-06-13 · **Scope shipped:** W0 (spike), W1 (maximize Xet), W2 (accurate progress). W3/W4 deferred.
## What landed
**W0 — spike (FDL-00).** Classified all 25 `models.yaml` repos via the HF API `xetEnabled` field → **25/25 Xet-backed** (incl. both first-run defaults). See `260613-fdl-SPIKE.md`. Verdict: Wave 3 (segmented accelerator) is **LOW priority** — Xet already gives parallel chunked transfer for every shipped model. Detection caveat recorded: `repo_info` siblings on hf_hub **1.7.2** expose no xet metadata; classify via the `xetEnabled` API field, not siblings.
**W1 — maximize + guarantee Xet (FDL-01..04).**
- `pyproject.toml`: pinned `huggingface_hub>=1.7` + `hf-xet>=1.1` explicitly (was transitive/unpinned); no `hf_transfer`. Resolves to hf_hub 1.7.2 / hf-xet 1.4.2, single version.
- `download.py`: `install_model` now drives `snapshot_download` with explicit `tqdm_class` (our progress-emitting subclass), `max_workers` (prefs `download_max_workers`, default 8), and `endpoint` (prefs `hf_endpoint` — W4 hook). `apply_xet_env()` applies opt-in `HF_XET_HIGH_PERFORMANCE` + `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (both default OFF, env wins).
- `system.py`: `/system/info` now returns `fast_download {xet_enabled, xet_version, high_performance}`; logged once at startup.
**W2 — accurate downloaded/remaining + speed (FDL-05..07).**
- Preflight `snapshot_download(dry_run=True)``compute_plan()``install_plan` SSE event with `total_bytes / cached_bytes / to_download_bytes / n_files / n_cached` **before bytes flow**. Degrades to totals=None on gated/older repos.
- New `utils/download_aggregator.py`: one source of truth for overall progress. Fed by a byte-sink on the patched tqdm; distinguishes byte bars (unit 'B', keyed by bar id) from the "Fetching N files" count bar; emits one throttled `aggregate` event (bytes_done/total/windowed rate/eta/files).
- Frontend `Settings.jsx` + `setup.ts`: overall bar driven by the aggregate; bar % = `max(byte%, file%)`; shows cached-skip + files-progress; `⚡ fast download` badge from `/system/info`. i18n keys added to `en.json`.
## Rebase reconciliation (main disabled Xet)
Rebasing onto latest main surfaced that main now sets **`HF_HUB_DISABLE_XET=1`** (main.py) — a deliberate choice to force the classic LFS path because Xet's progress bypasses the tqdm hook (the exact limitation found here). Reconciled rather than fought:
- `fast_download` status now reports the **runtime truth**: `xet_installed` + `xet_active` (active = installed AND not disabled) + `xet_enabled` alias. Default `xet_active=false`; the ⚡ badge only shows when Xet actually runs. Startup log: `downloads: Xet disabled → legacy LFS …`.
- Docs rewritten: default backend is **legacy LFS for accurate progress**; Xet is opt-in via `HF_HUB_DISABLE_XET=0` (coarser progress). The hf-xet pin stays (harmless; ready for a future Xet progress hook).
- Net: W2's progress is the value either way; W1's "maximize Xet" is dormant by main's design, not removed.
## Decisions / "use judgment" notes
- **Xet progress limitation (verified by live smoke).** Under Xet + hf_hub 1.7.2 the per-file **byte** bars never advance `n` and never `close()` through our tqdm (Xet fetches chunks out-of-band). Only the **file-count bar** is live. So: mid-download the overall bar is **file-granular** (moves 0→N files), and `complete()` flushes `bytes_done` to the exact preflight total on success (verified: final `74420620/74420620`, files 4/4). True live byte-speed is only available on classic-LFS/mirror repos (W4). This is a real constraint, not a bug — documented here and worth surfacing in W4 docs.
- Per-file detail kept inline (existing single-line summary, now aggregate-sourced) rather than a new collapsible panel — limited risk; can revisit.
## Drive-by fix
- `download.py` imported no `os`, but `_validate_snapshot_has_weights` uses `os.walk` → latent `NameError` on every install. Added `import os`.
## Verification
- `tests/backend/setup/test_download_preflight.py` — 10 pass (compute_plan splits, aggregator byte/count routing, close-credit, windowed rate/eta, registry feed + finish noop).
- `pytest -k "download or install or model or engine or setup"` — 149 passed, 7 skipped, 0 failed.
- `frontend typecheck:ci` — exit 0.
- Live smoke (real install of `mlx-community/whisper-tiny-mlx`, then deleted): `install_plan` exact; aggregate files 0→1→4; final bytes==total; `/system/info` + startup log correct.
## W4 — mirror + cancel + docs (FDL-10..12, shipped)
- **Mirror (FDL-10):** `snapshot_download(endpoint=…)` honours prefs `hf_endpoint` / env `HF_ENDPOINT` on both preflight and download — per-call, no process-wide mutation. Documented as the classic-LFS (non-Xet) path that restores continuous byte-speed.
- **Cancel (FDL-11):** `POST /models/install/cancel {repo_id}` sets a cancel flag checked at each retry boundary → emits `install_cancelled`, clears the cooldown (cancel ≠ failure). Limitation: an in-flight single-file fetch isn't interruptible in hf_hub 1.7.2; cancel lands at the next retry boundary. Frontend treats `install_cancelled` as a terminator (clears row + refetch).
- **Docs (FDL-12):** `docs/downloading-models.md` (Xet fast path, progress semantics incl. the byte-speed limitation, opt-in tuning knobs, mirror/restricted-network, cancel, troubleshooting) + README pointer. Docs-sync rule satisfied in-PR.
## W3 — opt-in segmented accelerator (FDL-08/09, shipped)
Reprioritised from LOW to HIGH after the rebase: since main forces Xet off, the default path is single-stream legacy LFS, so a segmented downloader is the way to get **both** parallel speed and live byte progress.
- `services/segmented_download.py`: async multi-connection Range downloader for one file — parallel byte-ranges, resume (`.part` + manifest), per-segment short-read truncation guard, optional sha256/etag verify, cancel, single-stream fallback when the server won't range. **Auth-safe**: the HF `Authorization` header goes only to `huggingface.co`/`hf.co`; never forwarded to a CDN host on redirect (unit-tested).
- Dispatch (`download.py`): opt-in via prefs `segmented_downloader` / env `OMNIVOICE_SEGMENTED_DOWNLOAD` (default OFF). When on and Xet inactive, fetches each repo file into the HF cache mirroring `hf_hub_download` (blobs + snapshot symlinks + `refs/main`), feeding **real bytes** to the aggregator. Any failure falls back to `snapshot_download` — the accelerator can never break a correct install.
- Verified live (accelerator ON): real mid-download byte progress (1.5 KB → 71 MB, rate ramping to **16.6 MB/s**), final `bytes_done == total`, `/models` shows `installed: True`, delete frees the right bytes.
- Fixed a `complete()` double-count (was adding a full total on top of accumulated segmented bytes → 2×); now replaces byte bars so the sum is exactly total.
- Tests: `tests/backend/services/test_segmented_download.py` (7 cases) covering parallel range reassembly, single-stream fallback, the auth header reaching only the HF host (never a CDN), size/truncation rejection, cancellation, and byte-callback totals — plus an aggregator double-count regression.
@@ -0,0 +1,363 @@
---
phase: 260613-mm2
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- backend/services/tts_backend.py
- backend/services/model_manager.py
- backend/services/subprocess_backend.py
- backend/services/model_lifecycle.py # NEW
- backend/api/routers/system.py
- backend/api/routers/setup/download.py
- backend/api/routers/setup/models.py
- tests/test_engines.py
- tests/backend/services/test_model_lifecycle.py # NEW
- tests/backend/services/test_subprocess_reaper.py
autonomous: true
requirements:
# ── Tier 1 — Correctness (Wave 1) ──────────────────────────────────────────
- MM2-01 # Registry reuses one active instance and calls unload() on engine switch
- MM2-02 # Per-engine unload() overrides (in-process drop+free_vram; subprocess -> unload_sidecar)
- MM2-03 # /model/loaded + /model/unload report ASR honestly; no unloadable:False-but-loaded lies
# ── Tier 2 — Single lifecycle surface (Wave 2) ─────────────────────────────
- MM2-04 # model_lifecycle facade owns list_loaded/unload/unload_all/free_vram across all 3 worlds
- MM2-05 # Idle/timeout config unified through core.prefs.resolve (env still wins); no duplicated constants
# ── Tier 3 — Robustness & observability (Wave 3) ───────────────────────────
- MM2-06 # _install_cooldowns bounded (evict on success + TTL); no unbounded growth
- MM2-07 # Snapshot weight validation is per-role, not one 5 MB magic number
- MM2-08 # Subprocess sidecars self-report VRAM in pong; panel shows real MB, not 0
- MM2-09 # scan_cache_dir() -> disk-walk fallback logs WHY it fell back (WinError #117/#118)
must_haves:
truths:
- "Switching the active TTS backend in Settings releases the outgoing engine's VRAM before the new one loads — verified by asserting the outgoing instance's unload() was called exactly once on switch."
- "TTSBackend.unload() is overridden by OmniVoiceBackend (drops model ref + free_vram) and by every SubprocessBackend subclass (routes to unload_sidecar); all overrides are idempotent and safe before first generate()."
- "/model/loaded never reports a model as loaded with a misleading unloadable flag: the ASR row's unloadable reflects whether it can actually be released independently of the TTS lifecycle."
- "services.model_lifecycle is the single import surface for list_loaded()/unload(id)/unload_all()/free_vram(); system.py routers call it instead of re-enumerating models inline."
- "Idle timeouts for the in-process model and subprocess sidecars resolve through core.prefs.resolve(... env=...) so an env var still wins and the Settings store can override; no module duplicates IDLE_TIMEOUT_SECONDS by hand."
- "_install_cooldowns cannot grow without bound: entries are removed on successful install and stale entries are evicted by TTL."
- "A live subprocess sidecar reports a non-zero vram_mb in /model/loaded when it actually holds GPU memory (pong carries the figure); CPU-only sidecars report 0 truthfully."
- "When scan_cache_dir() raises and the code falls back to the on-disk walk, the reason is logged at WARNING with the exception type (the #117/#118 WinError-448 path is no longer silent)."
- "uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py passes."
- "No on-disk model state changes; no new runtime dependency added; behavior degrades gracefully (not errors) on MPS/CPU where VRAM APIs are sparse."
artifacts:
- path: "backend/services/tts_backend.py"
provides: "Active-instance reuse + unload-on-switch in get_active_tts_backend(); per-engine unload() overrides"
contains: "_active_instance AND (def unload)"
- path: "backend/services/model_lifecycle.py"
provides: "Facade owning list_loaded/unload/unload_all/free_vram across in-process + subprocess models"
contains: "def list_loaded AND def unload_all"
- path: "backend/api/routers/system.py"
provides: "Thin /model/loaded + /model/unload routers delegating to model_lifecycle"
contains: "model_lifecycle"
key_links:
- from: "get_active_tts_backend() (tts_backend.py:1235)"
to: "outgoing backend.unload()"
via: "module-level _active_instance compared against newly-resolved active_backend_id()"
pattern: "_active_instance"
- from: "system.py /model/loaded + /model/unload (system.py:129, 210)"
to: "model_lifecycle.list_loaded() / model_lifecycle.unload()"
via: "import services.model_lifecycle"
pattern: "model_lifecycle\\.(list_loaded|unload)"
- from: "subprocess sidecar pong reply (subprocess_backend.py:435-438)"
to: "list_live_sidecars() vram_mb field"
via: "ping reply carries allocated VRAM measured inside the sidecar process"
pattern: "vram_mb"
---
<objective>
Clean up OmniVoice's model-management subsystem ("v2"). Today load / unload / list / free-VRAM each behave differently across three worlds — the in-process model (`model_manager.py`), the TTS backend registry (`tts_backend.py`), and subprocess sidecars (`subprocess_backend.py`) — with no single lifecycle owner. This produces one real user-facing bug (VRAM leak on engine switch), inaccurate VRAM/unloadable reporting, an unbounded cooldown dict, and a silent cache fallback.
This is **cleanup + correctness, not a rewrite.** The Wave 13 idle-reaper and the SubprocessBackend primitive are sound and stay. The `TTSBackend.unload()` contract already exists as a documented default no-op (`tts_backend.py:149`) explicitly deferred to "Phase 2"; this plan *is* that Phase-2 follow-through — wire the registry to call it, override it per engine, and unify the surrounding surface.
Three tiers, executed in order (each independently shippable, continuous-to-main per the v0.3.0 cadence):
- **Wave 1 / Tier 1 — Correctness:** MM2-01..03. The VRAM leak on switch + honest unload reporting. Highest value; ship first.
- **Wave 2 / Tier 2 — Single lifecycle surface:** MM2-04..05. Extract `model_lifecycle` facade + unify idle/timeout config.
- **Wave 3 / Tier 3 — Robustness & observability:** MM2-06..09. Bounded cooldowns, per-role weight validation, sidecar VRAM self-report, cache-fallback logging.
Output: PRs on branches off `main` (one per wave is fine), each green on the listed pytest selection. No push until the orchestrator merges; tests added with each wave.
Out of scope (call out, do not touch): GPU-pool per-engine sizing (`model_manager.py:42`, `_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — those are performance, not cleanup, and carry regression risk against #278/#315.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@./CLAUDE.md
# Files under edit (read before editing)
@backend/services/tts_backend.py
@backend/services/model_manager.py
@backend/services/subprocess_backend.py
@backend/api/routers/system.py
@backend/api/routers/setup/download.py
@backend/api/routers/setup/models.py
# Reference only — establish patterns, do NOT modify
@backend/core/prefs.py
@tests/test_engines.py
@tests/backend/services/test_subprocess_reaper.py
<interfaces>
<!-- Verified during planning. Executor should use these directly. -->
backend/services/tts_backend.py
- class TTSBackend(ABC) (line 59); unload() default no-op (line 149) — contract already documented:
idempotent, synchronous, safe before first generate().
- OmniVoiceBackend.__init__(self, model=None) (line 174); self._model reuses model_manager singleton.
- _REGISTRY: dict[str, type[TTSBackend]] (line 1109, a _LazyRegistry).
- active_backend_id() (line 1228) -> prefs.resolve("tts_backend", env="OMNIVOICE_TTS_BACKEND", default="omnivoice").
- get_active_tts_backend(*, model=None) (line 1235) — builds a FRESH instance every call, no teardown. THE leak.
backend/core/prefs.py
- resolve(key: str, *, env: Optional[str] = None, default: Any = None) -> Any (line 75) — env wins, then store, then default.
backend/services/model_manager.py
- module global `model` (line 111); `_last_used`; free_vram() (line 678); idle_worker() (line 667).
- IDLE_TIMEOUT_SECONDS imported from core.config (line 33); duplicated as _IDLE_TIMEOUT_SECONDS (line 114). Collapse.
- offload_tts_for_asr() (line 701) / restore_tts_after_asr() — ad-hoc ASR<->TTS VRAM juggling; _diar_pipeline global.
backend/services/subprocess_backend.py
- protocol op set (line 74); SIDECAR_IDLE_TIMEOUT_S = env-only float (line 107) — move to prefs.resolve.
- list_live_sidecars() -> list[dict] (line 181); unload_sidecar(engine_id) (line 199); unload_all_sidecars() (line 205).
- health/ping: _send({"op":"ping"}) then expect {"op":"pong"} (lines 435-438). Add vram_mb to the pong here AND
in the sidecar entry-point that answers ping (search the sidecar worker for the "ping"->"pong" handler).
backend/api/routers/system.py
- GET /model/loaded (line 129) — ~80 lines of inline enumeration of TTS/ASR/diar/sidecars. Replace body with
model_lifecycle.list_loaded().
- POST /model/unload/{model_id} (line 210) — handles "tts" | "diarization" | "sidecar:<id>" | "sidecars".
Replace body with model_lifecycle.unload(model_id).
backend/api/routers/setup/download.py
- _install_cooldowns dict (line 27) — unbounded. _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB
(line 45) — single magic number across roles.
backend/api/routers/setup/models.py
- scan_cache_dir() with silent disk-walk fallback (~line 268-280) + _scan_cache_on_disk (line 177).
</interfaces>
</context>
<tasks>
<!-- ════════════ WAVE 1 / TIER 1 — CORRECTNESS ════════════ -->
<task type="auto">
<name>Task 1 (MM2-02): Per-engine unload() overrides</name>
<files>backend/services/tts_backend.py</files>
<action>
The base-class `unload()` no-op already exists (tts_backend.py:149) with a documented contract. Override it where it matters. Do this BEFORE Task 2 — the registry switch (Task 2) calls these.
- OmniVoiceBackend (line 162): override `unload(self)`. Drop the local model ref (`self._model = None`) and, because OmniVoice shares the singleton owned by model_manager, also release that: `import services.model_manager as mm; mm.model = None; mm.free_vram()`. Idempotent — guard on `mm.model is not None` before free_vram(). Safe before first generate() (no-op when nothing loaded).
- Every SubprocessBackend subclass: implement `unload(self)` on the SubprocessBackend base (subprocess_backend.py — the duck-typed `_is_subprocess_isolated` class) so all subclasses inherit it. It must call `unload_sidecar(self.id)` (force-shut this engine's sidecar; busy sidecars are skipped, never interrupted — existing semantics). Idempotent: unload_sidecar on a non-running engine returns 0, no raise.
- In-process non-OmniVoice engines that hold their own model (e.g. KittenTTS/VoxCPM2 keep refs in __init__): override unload() to drop the ref + best-effort empty_cache via the existing free_vram() helper if they used GPU. Where an engine genuinely holds nothing resident, leave the base no-op (and note it in the SUMMARY so the future CI gate knows it's intentional, not missed).
Honor the contract comment verbatim: idempotent, synchronous, safe pre-load.
</action>
<verify>
<automated>grep -n "def unload" backend/services/tts_backend.py backend/services/subprocess_backend.py</automated>
<automated>uv run python -c "from services.tts_backend import OmniVoiceBackend; b=OmniVoiceBackend(); b.unload(); b.unload(); print('idempotent ok')"</automated>
</verify>
<done>
- OmniVoiceBackend.unload() drops both self._model and mm.model and calls free_vram(), guarded for idempotency.
- SubprocessBackend.unload() routes to unload_sidecar(self.id); inherited by all subprocess engines.
- Calling unload() twice, and before any generate(), never raises.
</done>
</task>
<task type="auto">
<name>Task 2 (MM2-01): Registry reuses one active instance + unloads on switch</name>
<files>backend/services/tts_backend.py</files>
<action>
Fix the leak at get_active_tts_backend() (line 1235). Today it builds a fresh instance every call with no teardown of the prior engine — switching engines (or repeated synth) leaks VRAM until GC. This is the root cause behind the #278 comment thread.
- Add a module-level cache: `_active_instance: TTSBackend | None = None` and `_active_instance_id: str | None = None`.
- In get_active_tts_backend(): resolve `bid = active_backend_id()`. If `_active_instance is not None` and `_active_instance_id != bid`, call `_active_instance.unload()` (best-effort, wrap in try/except so a bad unload can't block the switch — log on failure) before discarding it.
- Build the new instance, store it as `_active_instance` + `_active_instance_id = bid`, return it.
- IMPORTANT subtlety: OmniVoiceBackend takes `model=`. When `model=` is passed (the caller already has a loaded model), do NOT cache that instance as the shared `_active_instance` blindly — it's a per-call view over the shared singleton. Keep current behavior for the `model=` path (return a fresh OmniVoiceBackend(model=model)) but still trigger unload() of a *different* outgoing engine first. Pick the simplest correct rule: the cache tracks the configured backend id; passing model= for the SAME id reuses, switching id always unloads the previous. Document the rule in a comment.
- Add a module-level `reset_active_backend()` helper that unloads + clears the cache, for app shutdown and tests.
</action>
<verify>
<automated>grep -n "_active_instance\|def reset_active_backend\|def get_active_tts_backend" backend/services/tts_backend.py</automated>
</verify>
<done>
- Switching backend id calls the outgoing instance's unload() exactly once before the new instance is built.
- A bad/raising unload() is caught + logged, never blocks the switch.
- reset_active_backend() exists and is idempotent.
- The model= fast-path for OmniVoice still works (no double-load).
</done>
</task>
<task type="auto">
<name>Task 3 (MM2-03): Honest /model/loaded + /model/unload for ASR</name>
<files>backend/api/routers/system.py</files>
<action>
The ASR row (system.py:166-175) is reported as unloadable:False, vram_mb:0 even when loaded on GPU, and /model/unload doesn't expose the offload-to-CPU path. Make reporting truthful WITHOUT changing the ASR<->TTS lifecycle coupling (that coupling is intentional — offload_tts_for_asr/restore_tts_after_asr).
- ASR row: keep unloadable reflecting reality. If ASR truly cannot be released independently of TTS, keep unloadable:False but add a `note` field ("released with TTS") so the UI explains it rather than showing a dead button. Do not invent a separate ASR unload that breaks the WhisperX large-v3 offload path.
- vram_mb: if ASR currently runs on CPU (device "cpu" in the row), 0 is correct — leave it but make the device value derive from where the pipe actually is, not a hardcoded "cpu".
- This task is intentionally small; the bigger restructure is Task 4 (facade). Land MM2-03 as the honest-reporting fix, then Task 4 moves the enumeration into the facade.
</action>
<verify>
<automated>uv run pytest tests/test_engines.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>
- No row reports loaded-but-with-a-misleading-unloadable flag; ASR carries an explanatory note when unloadable:False.
- Device field reflects the actual device of the ASR pipe.
</done>
</task>
<task type="auto">
<name>Task 4 (MM2-01..03 tests): Wave 1 regression tests</name>
<files>tests/test_engines.py</files>
<action>
Add tests proving the leak fix and the unload contract:
- test_switching_backend_unloads_previous: monkeypatch two fake backends into _REGISTRY, set active to A (get_active_tts_backend), switch prefs to B, assert A.unload() was called exactly once before B is returned.
- test_unload_is_idempotent_and_preload_safe: OmniVoiceBackend().unload() twice + before generate() never raises.
- test_reset_active_backend_clears_cache: after reset_active_backend(), the next get_active_tts_backend() builds fresh.
Reuse the existing fixture style in tests/test_engines.py (it already monkeypatches the registry / availability). Keep tests CPU-only (no real model load).
</action>
<verify>
<automated>uv run pytest tests/test_engines.py -q 2>&amp;1 | tail -20</automated>
</verify>
<done>All three new tests pass; existing test_engines.py tests still green.</done>
</task>
<!-- ════════════ WAVE 2 / TIER 2 — SINGLE LIFECYCLE SURFACE ════════════ -->
<task type="auto">
<name>Task 5 (MM2-04): Extract services/model_lifecycle.py facade</name>
<files>backend/services/model_lifecycle.py</files>
<action>
Create backend/services/model_lifecycle.py as the single owner of cross-world model lifecycle. It composes the existing pieces — it does NOT reimplement loading.
Public surface:
- list_loaded() -> list[dict]: returns the unified rows currently assembled inline in system.py:129-207 (TTS, ASR, diarization, subprocess sidecars). Move that logic here verbatim first, then improve (MM2-03 note field, MM2-08 sidecar vram once Task 8 lands).
- unload(model_id: str) -> dict: the dispatch currently inline in system.py:210-242 ("tts" | "diarization" | "sidecar:<id>" | "sidecars"). Move here; keep async-lock semantics for the in-process model (mm._model_lock).
- unload_all() -> dict: unload every releasable model (in-process TTS + diar + all sidecars). New convenience used by app shutdown.
- free_vram(): thin re-export of model_manager.free_vram() so callers have one import.
Keep the "never let sidecar enumeration break the panel" try/except guard.
</action>
<verify>
<automated>uv run python -c "import services.model_lifecycle as ml; print([f for f in ('list_loaded','unload','unload_all','free_vram') if hasattr(ml,f)])"</automated>
</verify>
<done>model_lifecycle exposes list_loaded/unload/unload_all/free_vram; logic moved out of system.py (not duplicated).</done>
</task>
<task type="auto">
<name>Task 6 (MM2-04): Thin system.py routers + facade tests</name>
<files>backend/api/routers/system.py, tests/backend/services/test_model_lifecycle.py</files>
<action>
- Replace the bodies of GET /model/loaded (line 129) and POST /model/unload/{model_id} (line 210) with calls to model_lifecycle.list_loaded() / model_lifecycle.unload(model_id). Preserve the exact response shapes (frontend hooks.ts useModelStatus/useFlushMemory + the flush dropdown depend on {models, count} and {unloaded, success, ...}). The 400 on unknown model_id stays.
- New tests/backend/services/test_model_lifecycle.py: list_loaded with nothing loaded returns {models:[], count:0}; unload("tts") when not loaded returns success:False reason:"not loaded"; unload("sidecars") with no sidecars returns count:0; unknown id raises/400 path. Mock model_manager + subprocess_backend so no real models load.
</action>
<verify>
<automated>uv run pytest tests/backend/services/test_model_lifecycle.py -q 2>&amp;1 | tail -20</automated>
<automated>grep -n "model_lifecycle" backend/api/routers/system.py</automated>
</verify>
<done>system.py routers are thin delegations; response shapes unchanged; new facade tests pass.</done>
</task>
<task type="auto">
<name>Task 7 (MM2-05): Unify idle/timeout config through prefs.resolve</name>
<files>backend/services/model_manager.py, backend/services/subprocess_backend.py</files>
<action>
- model_manager.py: remove the duplicated `_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS` (line 114). Resolve at use-site in idle_worker() via prefs: `prefs.resolve("idle_timeout_seconds", env="OMNIVOICE_IDLE_TIMEOUT_S", default=IDLE_TIMEOUT_SECONDS)`. Keep core.config.IDLE_TIMEOUT_SECONDS as the default source.
- subprocess_backend.py: replace the env-only `SIDECAR_IDLE_TIMEOUT_S` (line 107) read with prefs.resolve("sidecar_idle_timeout_seconds", env="OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S", default=300.0). Preserve "<=0 disables reaping" semantics and the existing reaper-start guard (line 222). Resolve lazily (function call), not at import, so a test/setting change takes effect — but keep a sensible cached default for the hot reaper loop.
- Both must keep env precedence (env wins over store) — that's exactly what prefs.resolve already does.
</action>
<verify>
<automated>grep -n "_IDLE_TIMEOUT_SECONDS\|prefs.resolve\|SIDECAR_IDLE_TIMEOUT" backend/services/model_manager.py backend/services/subprocess_backend.py</automated>
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&amp;1 | tail -20</automated>
</verify>
<done>
- No hand-duplicated IDLE_TIMEOUT constant; both timeouts resolve via prefs with env precedence.
- Reaper "<=0 disables" + busy-skip behavior unchanged; all 10+ reaper tests still pass.
</done>
</task>
<!-- ════════════ WAVE 3 / TIER 3 — ROBUSTNESS & OBSERVABILITY ════════════ -->
<task type="auto">
<name>Task 8 (MM2-08): Subprocess sidecars self-report VRAM in pong</name>
<files>backend/services/subprocess_backend.py</files>
<action>
Sidecar VRAM is reported as 0 (system.py:192-203 / list_live_sidecars) because the parent can't measure a child's GPU memory. Have the child measure itself.
- In the sidecar worker's ping handler (the code that answers {"op":"ping"} with {"op":"pong"} — find it in the sidecar entry-point module), include `vram_mb`: measure inside the child via torch.cuda.memory_allocated() (CUDA) or torch.mps.driver_allocated_memory() (MPS, guarded), else 0. Same degrade-gracefully pattern as system.py:147-156.
- Parent: in the health-check ping/pong path (subprocess_backend.py:435-438), capture reply["vram_mb"] and stash it on the sidecar record so list_live_sidecars() (line 181) can surface it. Refresh opportunistically on each successful ping; default to last-known or 0 if never measured.
- Keep the contract that enumeration never breaks the panel.
This is CUDA/MPS-aware and degrades to 0 on CPU — honoring cross-platform parity (default behavior identical; the number is just more accurate where the API exists).
</action>
<verify>
<automated>grep -n "vram_mb" backend/services/subprocess_backend.py</automated>
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>list_live_sidecars() exposes a vram_mb sourced from the child's own measurement; 0 only when truly CPU/unmeasured; reaper tests still green.</done>
</task>
<task type="auto">
<name>Task 9 (MM2-06, MM2-07): Bounded cooldowns + per-role weight validation</name>
<files>backend/api/routers/setup/download.py</files>
<action>
- MM2-06: _install_cooldowns (line 27) grows unbounded. On a successful install, delete the repo's cooldown entry. Add a TTL sweep: when reading/writing the dict, evict entries older than a fixed window (reuse the existing cooldown window constant; pick the larger of cooldown-window and e.g. 1h). Keep it simple — a dict + timestamps, swept on access. No new dep.
- MM2-07: _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB (line 45) is one magic number for all roles. Make the threshold per-role/per-extension: safetensors/bin/ckpt expect the existing floor; .onnx models (kittentts, supertonic, sherpa) can be legitimately smaller — set a lower, role-aware floor so a valid small ONNX model isn't flagged as truncated. Keep the #352 truncation-catch intent (catch a 0-byte / KB-sized partial), just stop false-positiving small-but-complete models.
</action>
<verify>
<automated>grep -n "_install_cooldowns\|_MIN_WEIGHT_BYTES\|def _validate_snapshot_has_weights" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&amp;1 | tail -20</automated>
</verify>
<done>Cooldown dict is bounded (evict-on-success + TTL sweep); weight validation floor varies by role/extension; #352 truncation still caught.</done>
</task>
<task type="auto">
<name>Task 10 (MM2-09): Log why scan_cache_dir() fell back to disk walk</name>
<files>backend/api/routers/setup/models.py</files>
<action>
The scan_cache_dir() -> _scan_cache_on_disk() fallback (~line 268-280, helper at line 177) silently swallows the exception — this is the #117/#118 Windows WinError-448 path. Wrap the fallback so it logs at WARNING with the exception type and a one-line reason ("scan_cache_dir failed (%s); falling back to on-disk walk of %s") before walking. Do not change the fallback behavior itself — just stop it being invisible in logs. Keep it from ever raising out (the panel must still render).
</action>
<verify>
<automated>grep -n "falling back\|logger.warning\|_scan_cache_on_disk\|scan_cache_dir" backend/api/routers/setup/models.py | head</automated>
</verify>
<done>The disk-walk fallback logs a WARNING naming the exception type; behavior otherwise unchanged; never raises out.</done>
</task>
</tasks>
<verification>
Full-suite gate after each wave (run the relevant subset per wave, full set before the last PR):
1. `uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py tests/test_model_manager_preload.py -q` — all green.
2. `uv run pytest tests/ -k "download or install or model or engine" -q` — green (Tier 3 touch points).
3. Response-shape guard: GET /model/loaded still returns {models, count}; POST /model/unload returns {unloaded, success, ...}; 400 on unknown id. (Covered by test_model_lifecycle.py.)
4. No new runtime dependency: `git diff pyproject.toml uv.lock` is empty.
5. Localization/CJK + redaction gates unaffected: `uv run pytest tests/test_no_hardcoded_cjk.py -q`.
</verification>
<success_criteria>
- Tier 1: switching the active backend releases the previous engine's VRAM (unload() called once on switch); contract overridden for OmniVoice + all subprocess engines; ASR reporting is honest. (MM2-01..03)
- Tier 2: services.model_lifecycle is the single lifecycle surface; system.py routers are thin delegations with unchanged response shapes; idle/timeout config flows through prefs.resolve with env precedence and no duplicated constants. (MM2-04..05)
- Tier 3: cooldown dict bounded; weight validation is per-role; sidecars self-report real VRAM; cache-fallback logs its reason. (MM2-06..09)
- All listed pytest selections pass; no on-disk model-state change; no new dep; cross-platform default behavior identical (VRAM numbers degrade gracefully on MPS/CPU).
</success_criteria>
<risks>
- **unload() correctness for the shared OmniVoice singleton (MM2-01/02):** OmniVoiceBackend shares model_manager's `model` global. unload() must release the shared singleton, but the idle_worker() + offload_tts_for_asr() paths also touch it. Risk: a switch during an in-flight ASR offload double-frees or races. Mitigation: take mm._model_lock around the shared release in unload(); guard on `mm.model is not None`; keep unload best-effort (try/except) so it can never wedge a switch. Add the idempotency test (Task 4).
- **Response-shape drift (MM2-04):** Moving /model/loaded + /model/unload bodies into the facade risks changing the JSON the frontend depends on (hooks.ts, flush dropdown). Mitigation: move verbatim first, assert shapes in test_model_lifecycle.py, only then layer MM2-03/08 improvements.
- **Sidecar protocol change (MM2-08):** Adding vram_mb to pong touches the parent/child wire format. Older sidecars (a long-running session mid-upgrade) won't send it. Mitigation: treat vram_mb as optional in the parent (`reply.get("vram_mb", <last-known or 0>)`); never require it; never break the existing pong==success check.
- **prefs.resolve at import time (MM2-05):** Resolving timeouts at import freezes them; the reaper loop reads SIDECAR_IDLE_TIMEOUT_S. Mitigation: resolve lazily inside the reaper tick / idle_worker tick (cheap) so a settings change takes effect, while keeping the import-time default for the start-guard.
- **Per-role weight floor (MM2-07):** Lowering the ONNX floor could let a genuinely-truncated ONNX through (#352 regression). Mitigation: keep a non-zero floor for every role (e.g. ONNX floor still >> a partial KB), key on extension, and keep the "largest file" heuristic — only the threshold becomes role-aware.
- **Scope creep into perf:** GPU-pool sizing and torch.compile are explicitly out of scope. If an executor is tempted, stop — those regress #278/#315.
</risks>
<output>
Write `.planning/quick/260613-mm2-clean-model-management-v2/260613-mm2-SUMMARY.md` when done (per wave or once at the end), documenting: which engines got real unload() overrides vs intentional no-ops (for the future CI gate), the exact response shapes preserved on the two endpoints, the per-role weight-validation thresholds chosen, and the pytest output for the verification selection. Note any decision an executor made where the plan said "use judgment."
Docs-sync check (CLAUDE.md hard rule): this is internal lifecycle cleanup with no user-facing install/Docker/versioning change, so no README/docs edit is expected. If MM2-05 surfaces the new idle-timeout settings keys in the Settings UI, add them to the relevant settings doc in the same PR.
</output>
@@ -0,0 +1,32 @@
# SUMMARY — model-management v2 cleanup (mm2)
**Date:** 2026-06-13 · **Scope:** all 3 tiers (MM2-01..09). Backend-only; no frontend, no on-disk model-state change, no new deps.
## Tier 1 — correctness
- **MM2-01 (VRAM leak on engine switch):** `get_active_tts_backend()` now caches one instance per configured backend id and calls the outgoing engine's `unload()` before switching. Added `reset_active_backend()` for shutdown/tests. The `model=` OmniVoice fast-path still returns a fresh view over the shared singleton (no double-load) but a switch *away from* another engine still releases it. `tts_backend.py`.
- **MM2-02 (per-engine unload()):** `OmniVoiceBackend.unload()` drops the local ref + the shared `model_manager.model` singleton + `free_vram()` (idempotent, preload-safe, best-effort — no async lock from the sync path). `SubprocessBackend.unload()` routes to `unload_sidecar(self.id)` (busy sidecars skipped) and is inherited by every subprocess engine.
- **MM2-03 (honest ASR row):** `/model/loaded` ASR row now reports the pipe's actual device and carries a `note: "released with the TTS model"` so the disabled unload button is explained rather than silent.
## Tier 2 — single lifecycle surface
- **MM2-04 (`services/model_lifecycle.py`):** new facade owns `list_loaded()` / `unload(id)` / `unload_all()` / `free_vram()` across in-process TTS+ASR, diarization, and sidecars. `system.py` `/model/loaded` + `/model/unload` are now thin delegations; **response shapes preserved exactly** (`{models,count}`, `{unloaded,success,...}`, 400 on unknown id) — frontend untouched.
- **MM2-05 (unified idle config):** removed the duplicated `_IDLE_TIMEOUT_SECONDS`; the in-process idle timeout and the sidecar idle timeout both resolve per-tick via `prefs.resolve(... env=...)` (env wins, settings can tune without restart). New keys: `idle_timeout_seconds` (`OMNIVOICE_IDLE_TIMEOUT_S`), `sidecar_idle_timeout_seconds` (`OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S`). `<=0` still disables sidecar reaping.
## Tier 3 — robustness & observability
- **MM2-06 (bounded cooldowns):** `_install_cooldowns` is swept (TTL 1h) on each install check and cleared on success — can no longer grow unbounded.
- **MM2-07 (per-role weight floor):** `_validate_snapshot_has_weights` uses per-extension floors (tensor formats keep 5 MB; `.onnx` floor 64 KB) **OR** the original ≥5 MB catch — strictly more lenient, so a small-but-complete ONNX model is no longer false-flagged as truncated while a 0/KB partial is still rejected (#352 intact).
- **MM2-08 (sidecar VRAM self-report):** the parent can't see a child's VRAM, so the GPU sidecar (`engines/indextts`) now reports `vram_mb` in its `pong` (CUDA/MPS-aware, 0 on CPU); the parent stashes the last-known figure and `list_live_sidecars()` surfaces it. CPU/absent sidecars honestly report 0.
- **MM2-09 (cache-fallback logging):** the `is_cached` `scan_cache_dir → on-disk` fallback now logs at WARNING with the exception type (was DEBUG/invisible) — the #117/#118 WinError-448 path is triagable from logs.
## Out of scope (as planned, not done)
GPU-pool per-engine sizing (`_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — perf, not cleanup; risk regressing #278/#315.
## Verification
- New `tests/test_mm2_lifecycle.py` — 15 tests (reuse/switch-unload/reset/idempotent-unload, facade list/unload/unknown/sidecars shapes + honest ASR, env-wins idle config, cooldown sweep, per-role weight floor ×3).
- Affected existing: `test_engines.py`, `test_subprocess_reaper.py`, `test_model_load_timeout.py`, `test_model_manager_preload.py` — green (no regressions).
- **Full suite: 1379 passed, 0 failed.** Live: facade endpoints return preserved shapes; engine switch calls the previous engine's `unload()` exactly once.
## Test placement note
MM2 tests live at top-level `tests/` (not `tests/backend/`) on purpose: adding files under `tests/backend/` reorders collection and can expose a pre-existing `sys.modules`-isolation leak in other backend fixtures (the issue debugged in the FDL PR). Top-level placement keeps `tests/backend/` order identical.
## Docs-sync
The new idle-timeout settings keys are internal env/prefs knobs with no UI surface, so no README/docs change is required by the docs-sync rule. If a future Settings panel exposes them, document there.
+12
View File
@@ -6,6 +6,18 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
The bundled TTS model package (`pyproject.toml`) is versioned independently.
## [Unreleased]
### Added
- **Portable personas (`.ovsvoice`).** Export any voice as a self-contained,
fully-local persona bundle — identity, optional reference clip, consent
attestation, SPDX license, and a watermarked preview — and import it back into
another OmniVoice install. A privacy toggle ships a **preview-only** bundle so
no raw recording of your voice has to travel. Verified-own-voice status can't
be forged by hand-editing a bundle (real recording + consent text + attestation
required). Legacy `.omnivoice` files still import. See
[docs/persona-format.md](docs/persona-format.md). (#29)
## [0.3.5] — 2026-06-03
### Fixed
+13 -3
View File
@@ -3,7 +3,7 @@
**OmniVoice Studio**
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. Today it's a v0.2.7 active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord.
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The latest stable release is **v0.3.5**; `main` rolls ahead at **v0.3.6** (latest release + 1 patch — see the Versioning rule below).
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
@@ -16,7 +16,7 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** v0.3.0 has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main. Tag `v0.3.0` once when the user calls "actually useful" — a qualitative bar, not a checklist. No `v0.3.0-rc1`. No phased release. No `v0.4` deferrals while v0.3.0 is open — every open issue and every open community PR gets absorbed into the v0.3.0 line or explicitly declined. Users follow `main` for previews; users wanting stable stay on `v0.2.7`. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release (currently **v0.3.5**). ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
<!-- GSD:project-end -->
<!-- GSD:stack-start source:research/STACK.md -->
@@ -190,10 +190,20 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
<!-- GSD:conventions-start source:CONVENTIONS.md -->
## Conventions
**Versioning (hard rule):** Everything ships on `v0.3.0`. Never mention, suggest, or label anything with a version bump — no v0.4, no RCs, no "defer to next version", no future-version labels — unless the user explicitly asks to bump. Zero unprompted version chatter.
**Versioning (hard rule, owner-set 2026-06-11):** main is always **latest release + 1 patch**. The moment `vX.Y.Z` is released, main's version files (`frontend/src-tauri/tauri.conf.json`, `frontend/src-tauri/Cargo.toml`, `pyproject.toml`, **and `frontend/package.json`** — keep all **four** in lockstep; `package.json` drives the runtime `__APP_VERSION__` via vite, shown in the first-run footer + every auto bug report, so a drift ships a build that misreports its own version — guarded by `tests/test_app_version.py::test_all_version_files_in_lockstep`) bump to `X.Y.(Z+1)`. Consequences:
- Every PR and preview build identifies as the **next** version. Preview builds stamp `X.Y.(Z+1)-N` (run number), which semver-sorts **above** the last stable `X.Y.Z` — the updater ordering is natural, no comparator tricks needed.
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. The post-release bump is automated by the `version-bump` job in release.yml; if it fails, do it manually in the same day.
- Docker: `ghcr.io/debpalash/omnivoice-studio:latest` = **main** (rolling preview); `:X.Y.Z` + `:X.Y` + `:stable` = tagged releases. `:latest` is the preview channel by design — stable users pin `:stable` or a version tag.
- Do not bump minor/major or invent RCs/codenames without the owner asking. No "defer to next version" labels — scope is absorbed or declined, never re-versioned.
**Docs-sync (hard rule, owner-set 2026-06-11):** any change that alters something these docs describe — README.md, CONTRIBUTING.md, SECURITY.md, SUPPORT.md, LICENSE, or `docs/**` (install flows, Docker tag semantics, platform support, versioning/release behavior, review process, supported versions) — must update those docs **in the same PR** as the change. If a doc impact is discovered after merge, the docs fix is the immediate next commit, not backlog. Stale docs are treated as bugs.
**Localization (hard rule):** No hardcoded non-English (CJK) **user-facing text** anywhere in the codebase except the translation layer (`frontend/src/i18n/`). All UI strings go through i18n (`t('...')` keys in `locales/*.json`); native language names live in `i18n/index.ts` (`LANGUAGES`). Functional CJK is allowed and tracked via the allowlist in `tests/test_no_hardcoded_cjk.py` — text-processing regexes, model/engine vocabulary & identifiers (e.g. CosyVoice speaker IDs), localized error matching, demo/eval data, and test fixtures. CI fails on any hardcoded CJK outside the allowlist; to add legitimate functional CJK, extend `_ALLOWED_FILES` there with a justification.
**Fix quality (hard rule, owner-set 2026-06-16):** Fix issues *properly* and future-maintenance-proof — don't stop at the symptom. Root-cause fully, fix the whole **class** of the bug (not just the one reported instance), add a fail-before/pass-after regression test, and harden against recurrence (e.g. if a lockfile drift only fails in Docker, also make CI catch it). Go the extra mile where it durably pays off. Be token-efficient about it — extra **effort**, not extra **verbosity**: no padding, no redundant re-checks, the smallest correct change that is also recurrence-proof. Don't be shy to spend the effort a proper fix needs; do be shy about wasting tokens.
**Keep main green (hard rule, owner-set 2026-06-16):** A merge must **never break `main`'s CI**. Before a change lands, verify the *full* CI matrix would pass — every workflow in `.github/workflows/` **and** `deploy/Dockerfile`, not only the checks you happened to run. Dependency / lockfile / config changes must be validated against **all** consumers. Specifically: `frontend/` is a bun **workspace monorepo** — the lockfile is the repo-root `bun.lock`, and `deploy/Dockerfile` runs `bun install --frozen-lockfile`, so any `frontend/package.json` change requires regenerating root `bun.lock` and confirming `bun install --frozen-lockfile` passes (plain `bun install` in `ci.yml` silently tolerates drift, so CI-green ≠ Docker-green). Likewise re-check CodeQL/Security on code changes and the Tauri `cargo` build on Rust/dep changes.
Other conventions not yet established. Will populate as patterns emerge during development.
<!-- GSD:conventions-end -->
+75 -1
View File
@@ -42,11 +42,22 @@ This starts both services:
### Desktop App (Tauri)
```bash
bun run desktop
bun run desktop # dev: hot-reload Tauri shell + backend
bun run desktop-prod # production: builds, bundles the backend, then launches
```
Both run `uv sync` first (so the Python backend env is set up) and start the
backend automatically — you do **not** start it separately. Use the exact script
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
If the app opens but stays on the **setup splash with no buttons**, the Python
backend didn't finish starting — the splash surfaces the stall reason, a log
panel, and a **Retry** button (and Settings → Logs → Backend has the full trace).
The most common from-source cause is `uv` or Python not being on your PATH.
---
## Project Structure
@@ -192,6 +203,69 @@ cd frontend/src-tauri && cargo check
---
---
## What code review looks like
Every PR is reviewed by two AI reviewers before a human looks at it:
- **CodeRabbit** posts a walkthrough (with a sequence diagram, and an ASCII
before/after sketch for UI changes), inline findings, and warning-mode
pre-merge checks against the project's hard rules.
- **Greptile** reviews with the same project rubrics and learns from 👍/👎
reactions on its comments — react to train it.
Both are advisory, not gating: CI and the maintainer's approval decide. Don't
be surprised by detailed bot comments minutes after you open a PR — address
what's right, push back (in a reply) on what's wrong.
**Commit & PR conventions:** conventional-commit style with a scope
(`fix(dub): …`, `feat(setup): …`) and link the issue (`Closes #N` / `Refs #N`)
in the title or body.
## Quality gates your PR must pass
- **Cross-platform parity (hard rule):** anything that ships in default mode
must behave identically on macOS, Windows, and Linux. Platform-specific
*implementation* is fine; platform-divergent *default behavior* is a P0.
Platform-only features go behind an explicit opt-in (Settings toggle, env
var, or CLI flag).
- **i18n — all 21 locales (hard rule):** every user-facing string goes through
`t('...')` and the key must exist in **all 21** files under
`frontend/src/i18n/locales/`. Translate; don't copy English into non-English
locales. CI fails on hardcoded CJK outside the allowlist in
`tests/test_no_hardcoded_cjk.py` (extend `_ALLOWED_FILES` with a
justification for legitimate functional CJK).
- **DB schema changes** go through an alembic migration with a tested upgrade
path — existing `omnivoice_data/` must keep working with no manual steps.
- **Engine back-compat:** already-installed engines (model weights on disk)
must not require reinstall or re-download.
- **Local-first:** no new outbound calls except GitHub Issues (opt-in
reporting) and HuggingFace model downloads. Never log or persist secrets or
absolute home paths.
- **Security posture:** the backend serves loopback HTTP — treat every
query/path/form parameter as hostile. User-chosen filesystem destinations
are authorized in the Tauri process (save dialog), never via HTTP params.
## Contribution licensing
OmniVoice Studio is **AGPL-3.0-only**, and the maintainer also offers a
**commercial license** (see [LICENSE](LICENSE)). By submitting a contribution
you agree that:
1. you have the right to submit it (your own work, or compatibly licensed);
2. it is licensed to the project under **AGPL-3.0**; and
3. you grant the project maintainer a perpetual, worldwide, non-exclusive
right to also distribute your contribution under the project's commercial
license terms.
This inbound grant is what keeps the dual-license model viable. If you can't
agree to (3) for a particular contribution, say so in the PR and we'll discuss
before merging. Adding a `Signed-off-by:` line (DCO) to your commits is
appreciated but not required.
---
## Need Help?
- **Stuck on setup?** Ask in [Discord #help](https://discord.gg/bzQavDfVV9)
+9 -6
View File
@@ -24,10 +24,11 @@
</p>
<p>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_aarch64.dmg"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_x64_en-US.msi"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.AppImage"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.deb"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<!-- Pre-built macOS bundle is Apple Silicon. Intel Macs: build from source (docs/install/macos.md); a pre-built Intel target is tracked in #279. -->
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
</p>
<p>
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy &amp; Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
@@ -123,7 +124,7 @@ Per-OS install guides — pick yours and follow it end-to-end:
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
- **Docker** — [docs/install/docker.md](docs/install/docker.md)
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
Stuck? Run the built-in self-check first — **Settings → About → "Run
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
@@ -136,7 +137,9 @@ bundle"** packages scrubbed logs + the self-check report for bug reports.
For Hugging Face token setup, see
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md). For
diarization-specific gating, see
[docs/features/diarization.md](docs/features/diarization.md).
[docs/features/diarization.md](docs/features/diarization.md). For download
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
options, see [docs/downloading-models.md](docs/downloading-models.md).
## Screenshots
+14 -4
View File
@@ -2,10 +2,20 @@
## Supported Versions
| Version | Supported |
|---------|--------------------|
| 0.2.x | ✅ Current release |
| < 0.2 | ❌ No longer supported |
| Version | Supported |
|---------|-----------|
| 0.3.x (latest release + `main` previews) | ✅ Current — all fixes land here |
| 0.2.7 | ⚠️ Legacy stable — security fixes only, upgrade recommended |
| < 0.2.7 | ❌ No longer supported |
## Model supply chain
OmniVoice supports models from **public, verifiable sources only** (Hugging
Face repos, official project releases). Privately sold or gated model files
are not supported: an archive from a private source can carry anything
(bundled executables, modified configs), and nobody else can verify or
reproduce it. Treat any privately distributed model file as an untrusted
download, and never run executables bundled with model archives.
## Reporting a Vulnerability
+36
View File
@@ -0,0 +1,36 @@
# Support
## Where to get help
| Channel | Best for |
|---|---|
| [Discord](https://discord.gg/bzQavDfVV9) — `#help` | Setup problems, quick questions, sharing results |
| [GitHub Issues](https://github.com/debpalash/OmniVoice-Studio/issues) | Bugs and feature requests — use the templates; attach the diagnostic bundle (Settings → About → "Save diagnostic bundle") |
| [GitHub Discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) | Design questions, ideas, show & tell |
| Security issues | **Never a public issue** — see [SECURITY.md](SECURITY.md) for private reporting |
## Model sources we support
OmniVoice is built on the idea that everything it runs is **open and available
to everyone**: free, public models with verifiable sources and licenses
(Hugging Face repos, official project releases), so the whole community can
use, test, and debug the same thing.
**We do not support privately sold, paywalled, or gated model files.** A model
delivered privately can't be verified, reproduced, or shared — it doesn't fit
the project's goals, and issues involving such models will be politely closed.
As a general safety rule, never run executables bundled inside any model
archive.
## Before filing a bug
1. Update to the latest release (or `main` if you follow previews) — fixes ship continuously.
2. Run the in-app self-check: **Settings → About → Run self-check**.
3. Search existing issues; add a 👍 + your details to an existing one rather than opening a duplicate.
## Response expectations
This is an open-source project maintained with the help of an automated triage
bot: issues are typically triaged within hours and every report gets a human-
approved response. Reproducible reports with a diagnostic bundle get fixed
fastest.
+32
View File
@@ -7,9 +7,13 @@ composed at the route or router level without surprises.
Currently exposed:
- `require_loopback`: 403 unless the request came from a loopback origin
(bypassed in explicit server mode — see `_server_mode`).
- `ws_remote_authorized`: whether a WebSocket handshake from a non-loopback
client carries the remote API key (Wave 2.3) — used by WS endpoints that
keep their own inline loopback guards.
"""
import os
import secrets
from fastapi import HTTPException, Request
@@ -71,3 +75,31 @@ def require_loopback(request: Request) -> None:
if _server_mode():
return
raise HTTPException(status_code=403, detail="loopback origin required")
def remote_api_key() -> str | None:
"""The remote-backend bearer key (Wave 2.3), or None when remote mode is
off. Read at call time so tests can monkeypatch the env."""
return os.environ.get("OMNIVOICE_API_KEY") or None
def ws_remote_authorized(websocket) -> bool:
"""Whether a WebSocket handshake presents the remote API key.
Browser WebSockets cannot set an Authorization header, so the key may
arrive as ``?api_key=`` or via the ``ov_key`` cookie that the bearer
middleware sets on the first authenticated HTTP request. Returns False
when remote mode is off — callers keep their loopback-only behavior.
"""
key = remote_api_key()
if not key:
return False
auth = websocket.headers.get("authorization", "")
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
if not supplied:
supplied = (
websocket.query_params.get("api_key")
or websocket.cookies.get("ov_key")
or ""
)
return secrets.compare_digest(supplied, key)
+712
View File
@@ -0,0 +1,712 @@
"""Audiobook creator endpoints (parity Wave 5).
``POST /audiobook/plan`` — pure preview: parse a chapter-delimited script
(Markdown ``# H1`` chapters, inline ``[voice:NAME]`` / ``[pause …]``) into the
chapter/span plan, no synthesis.
``POST /audiobook`` — the synth job: render each chapter through the active TTS
backend (reusing ``services.audiobook.synthesize_chapter`` + ``chunked_tts``),
then mux the chapter WAVs into a chapterized **m4b** (FFMETADATA1 chapters via
``build_m4b_cmd``). Progress streams as Server-Sent Events, mirroring the dub
pipeline. ffmpeg-gated — without ffmpeg the job reports an error event and
stops (the m4b is the only output format).
``GET /audiobook/jobs`` + ``POST /audiobook/resume/{job_id}`` — durable
crash-resume: an interrupted render persists its plan + params to a
``resume.json`` manifest in the job work dir, so it can be resumed later (the
content-addressed chapter cache makes finished chapters instant) even without
the original script. The resume UI affordance remains a follow-up.
epub/pdf ingest, ACX mastering shipped; the resume UI surface remains a follow-up.
"""
import asyncio
import json
import logging
import os
import re
import uuid
from fastapi import APIRouter, File, HTTPException, UploadFile
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from services.audiobook import (
parse_audiobook_script,
synthesize_chapter,
)
from services.longform_render import (
LOUDNESS_PRESETS,
build_concat_list,
build_ffmetadata,
build_render_cmd,
prune_cache_dir,
)
from services import longform_resume # pure (no torch) — durable resume manifest
logger = logging.getLogger("omnivoice.audiobook")
router = APIRouter()
# A cover filename as produced by /audiobook/cover: 12 hex chars + image ext.
# An exact-match allowlist is the strongest barrier (and the one CodeQL's
# path-injection query recognizes) — anything else is rejected outright.
_COVER_NAME_RE = re.compile(r"^[0-9a-f]{12}\.(?:jpg|jpeg|png)$")
def _safe_cover_path(cover_path: str | None) -> str | None:
"""Confine a user-supplied cover to the upload directory before it can flow
into ffmpeg.
Covers only ever come from ``/audiobook/cover``, which writes them to
``OUTPUTS_DIR/audiobook_covers`` with a generated name. We rebuild the path
from the basename alone (``os.path.basename`` strips any directory component
or ``..`` traversal) joined onto that fixed directory, so no caller-supplied
path — absolute or relative — can escape it. Returns the path only if the
file actually exists there, else None."""
if not cover_path:
return None
from core.config import OUTPUTS_DIR
name = os.path.basename(cover_path)
if not _COVER_NAME_RE.match(name):
return None # not a name the upload endpoint could have produced
cover_dir = os.path.realpath(os.path.join(OUTPUTS_DIR, "audiobook_covers"))
real = os.path.realpath(os.path.join(cover_dir, name))
# Containment check on the resolved path itself — it must live inside the
# covers dir. Belt-and-suspenders over the regex+basename above; the
# commonpath form is the path-injection barrier static analysis recognizes.
if os.path.commonpath([real, cover_dir]) != cover_dir:
return None
return real if os.path.isfile(real) else None
class AudiobookPlanRequest(BaseModel):
text: str
default_voice: str | None = None
@router.post("/audiobook/plan")
def audiobook_plan(req: AudiobookPlanRequest) -> dict:
"""Parse a script into a chapter/span plan (pure preview, no synthesis)."""
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
return plan.to_dict()
#: Cover size cap mirrors longform_render's guard (8 MB — a book cover, not a
#: payload). Kept in sync intentionally; the render builder re-validates too.
_COVER_MAX_BYTES = 8 * 1024 * 1024
#: Import upload cap — a generous ceiling for a .txt/.md/.epub manuscript that
#: still stops a memory-exhaustion upload (the whole file is read into RAM).
_IMPORT_MAX_BYTES = 64 * 1024 * 1024
#: Upper bound on chapters in a single /longform/render plan — far above any real
#: book, but stops a pathological request from allocating/holding the job forever.
_MAX_CHAPTERS = 10_000
@router.post("/audiobook/import")
async def audiobook_import(file: UploadFile = File(...)) -> dict:
"""Import a ``.txt``/``.md``/``.epub``/``.pdf`` into a chapter-delimited script.
EPUB is parsed in spine order (stdlib only, local); PDF text is extracted
with pypdf (pure-Python) then chapterized; plain text gets ``# `` headings
inserted ahead of obvious chapter-title lines. Returns the script text (for
the editor) + the resulting chapter count."""
from services.longform_import import (
chapterize_plaintext,
epub_to_chapter_script,
pdf_to_chapter_script,
)
name = (file.filename or "").lower()
data = await file.read()
if not data:
raise HTTPException(status_code=400, detail="empty file")
if len(data) > _IMPORT_MAX_BYTES:
raise HTTPException(status_code=400, detail="file too large (max 64 MB)")
if name.endswith(".epub"):
try:
script = epub_to_chapter_script(data)
except ValueError as e:
raise HTTPException(status_code=400, detail=f"couldn't parse EPUB: {e}")
elif name.endswith(".pdf"):
try:
script = pdf_to_chapter_script(data)
except ValueError as e:
raise HTTPException(status_code=400, detail=f"couldn't parse PDF: {e}")
else:
script = chapterize_plaintext(data.decode("utf-8", "ignore"))
if not script.strip():
raise HTTPException(status_code=400, detail="no text found in the file")
plan = parse_audiobook_script(script)
return {"text": script, "chapters": plan.chapter_count}
@router.post("/audiobook/cover")
async def audiobook_cover(cover: UploadFile = File(...)) -> dict:
"""Upload a cover image; returns a server-side ``path`` to pass back as
``cover_path`` in the synth request. Validated here (jpg/png + size cap) and
again at render time."""
from core.config import OUTPUTS_DIR
ext = os.path.splitext(cover.filename or "")[1].lower()
if ext not in (".jpg", ".jpeg", ".png"):
raise HTTPException(status_code=400, detail="cover must be a .jpg or .png")
data = await cover.read()
if not data or len(data) > _COVER_MAX_BYTES:
raise HTTPException(status_code=400, detail="cover must be between 1 byte and 8 MB")
cover_dir = os.path.join(OUTPUTS_DIR, "audiobook_covers")
os.makedirs(cover_dir, exist_ok=True)
path = os.path.join(cover_dir, f"{uuid.uuid4().hex[:12]}{ext}")
with open(path, "wb") as f:
f.write(data)
return {"path": path}
class AudiobookRequest(BaseModel):
text: str
default_voice: str | None = None # voice profile id; None = engine default
bitrate: str = "128k"
format: str = "m4b" # "m4b" | "mp3"
loudness: str | None = None # None/"off" | "acx" | "podcast" (opt-in)
cover_path: str | None = None # server-side path to a jpg/png cover
# Global tags embedded in the output: {title, author, narrator, year,
# genre, description}. Player-visible (Apple Books / Audible read these).
metadata: dict | None = None
# Optional pronunciation lexicon {word: respelling} applied before synthesis.
lexicon: dict | None = None
def _resolve_voice(profile_id: str | None) -> dict:
"""Map a voice-profile id to (ref_audio, ref_text, instruct, seed).
Compact form of the resolver in generation.py — covers locked, design and
clone profiles. Returns all-None for the engine default (no profile).
"""
out = {"ref_audio": None, "ref_text": None, "instruct": None, "seed": None}
if not profile_id:
return out
from core.config import VOICES_DIR
from core.db import db_conn
with db_conn() as conn:
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if not row:
return out
try:
kind = row["kind"] or "clone"
except (KeyError, IndexError):
kind = "clone"
if row["is_locked"] and row["locked_audio_path"]:
out["ref_audio"] = os.path.join(VOICES_DIR, row["locked_audio_path"])
out["ref_text"] = row["ref_text"]
out["instruct"] = row["instruct"]
elif kind == "design":
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
out["ref_text"] = row["ref_text"] if out["ref_audio"] else None
out["instruct"] = row["instruct"]
else:
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
out["ref_text"] = row["ref_text"]
out["instruct"] = row["instruct"]
try:
if row["seed"] is not None:
out["seed"] = row["seed"]
except (KeyError, IndexError):
pass
return out
def _build_synth(default_voice: str | None) -> dict:
"""Describe how to synthesize for the active TTS engine.
Returns a dict with ``mode``, ``resolve`` (voice-id → resolved refs, cached
per id) and ``engine_id``. For OmniVoice it also carries the async
``get_model``; other engines carry a ready ``synth`` + ``sample_rate``.
:func:`_prepare_synth` turns this into a uniform ``(synth, sr, resolve,
engine_id)`` once the (async) model is in hand.
"""
from services.tts_backend import OmniVoiceBackend, active_backend_id, get_backend_class
cache: dict = {}
def resolve(voice_id):
key = voice_id or default_voice
if key not in cache:
cache[key] = _resolve_voice(key)
return cache[key]
engine_id = active_backend_id()
cls = get_backend_class(engine_id)
if cls is OmniVoiceBackend:
from services.model_manager import get_model
return {"mode": "omnivoice", "resolve": resolve,
"engine_id": engine_id, "get_model": get_model}
backend = cls()
def synth(text, voice_id, speed=None):
v = resolve(voice_id)
return backend.generate(
text, language=None, ref_audio=v["ref_audio"],
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
speed=float(speed) if speed else 1.0,
)
return {"mode": "generic", "resolve": resolve, "engine_id": engine_id,
"synth": synth, "sample_rate": backend.sample_rate}
async def _prepare_synth(default_voice: str | None):
"""Resolve :func:`_build_synth` into ``(synth, sample_rate, resolve,
engine_id)`` — awaiting the OmniVoice model load when needed. Shared by the
full job and the per-chapter preview."""
info = _build_synth(default_voice)
resolve, engine_id = info["resolve"], info["engine_id"]
if info["mode"] == "omnivoice":
model = await info["get_model"]()
sr = getattr(model, "sampling_rate", 24000)
def synth(text, voice_id, speed=None):
v = resolve(voice_id)
return model.generate(
text=text, language=None, ref_audio=v["ref_audio"],
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
speed=float(speed) if speed else 1.0,
)[0]
return synth, sr, resolve, engine_id
return info["synth"], info["sample_rate"], resolve, engine_id
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None):
"""Render one chapter, content-addressed so a re-run reuses it (resume).
Returns ``(wav_path, duration_s, was_cached)``. The WAV lives at
``cache_dir/<key>.wav`` where ``key`` is :func:`chapter_cache_key` over the
chapter's spans + sample rate + engine + each voice's resolved signature
(+ the lexicon, so a lexicon edit re-renders), so an unchanged chapter is
never re-synthesized. Runs in the GPU-pool executor.
"""
import json
import wave
from services.audio_io import atomic_save_wav
from services.longform_render import chapter_cache_key
from services.pronunciation import normalize_lexicon
spans_tuples = [(s.voice_id, s.text, s.pause_ms_after, getattr(s, "speed", None))
for s in chapter.spans]
sig: dict = {}
for s in chapter.spans:
k = s.voice_id or ""
if k not in sig:
v = resolve(s.voice_id)
sig[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
if lexicon:
# Fold the lexicon into the cache key so editing pronunciations
# invalidates cached chapters (reserved key can't collide with a voice id).
sig["\x00lexicon"] = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
key = chapter_cache_key(spans_tuples, sample_rate=sr, engine_id=engine_id, voice_sig=sig)
wav_path = os.path.join(cache_dir, f"{key}.wav")
if os.path.exists(wav_path):
try:
with wave.open(wav_path, "rb") as w:
dur = w.getnframes() / float(w.getframerate() or sr)
return wav_path, dur, True
except Exception:
pass # corrupt cache entry — fall through and re-render
audio, dur = synthesize_chapter(chapter.spans, synth, sr, lexicon=lexicon)
atomic_save_wav(wav_path, audio, sr)
return wav_path, dur, False
class AudiobookPreviewRequest(BaseModel):
text: str
chapter_index: int = 0
default_voice: str | None = None
lexicon: dict | None = None
@router.post("/audiobook/preview")
async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
"""Render a single chapter so the user can audition it before the full run.
Reuses the same content-addressed cache as the job, so a preview warms the
cache (the later full render reuses it) and a re-preview is instant.
"""
from core.config import OUTPUTS_DIR
from services.model_manager import _gpu_pool
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
if not plan.chapters:
raise HTTPException(status_code=400, detail="no chapters parsed from the script")
n = len(plan.chapters)
if not (0 <= req.chapter_index < n):
raise HTTPException(status_code=400, detail=f"chapter_index out of range (0..{n - 1})")
chapter = plan.chapters[req.chapter_index]
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
os.makedirs(cache_dir, exist_ok=True)
synth, sr, resolve, engine_id = await _prepare_synth(req.default_voice)
loop = asyncio.get_running_loop()
wav_path, dur, was_cached = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
req.lexicon,
)
return {
"output": os.path.relpath(wav_path, OUTPUTS_DIR), # served via /audio
"duration_s": round(dur, 2),
"cached": was_cached,
"title": chapter.title,
}
async def _render_longform_sse(
plan,
*,
default_voice: str | None,
fmt: str = "m4b",
bitrate: str = "128k",
loudness: str | None = None,
cover_path: str | None = None,
metadata: dict | None = None,
lexicon: dict | None = None,
job_type: str = "audiobook",
job_id: str | None = None,
resume: bool = False,
):
"""Shared chapterized-render SSE generator for Audiobook *and* Stories.
Takes a ready ``plan`` (``.chapters`` → ``.title`` + ``.spans``) — Audiobook
parses it from a script, Stories compiles it from cast/lines — and renders
each chapter (content-addressed cache → resume), isolating per-chapter
failures, then muxes the successful chapters into a tagged file. This is the
convergence point: one renderer, two front doors.
"""
from core.config import OUTPUTS_DIR
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
from services.model_manager import _gpu_pool
# Resume reuses the original job_id (continuing the same job row + cached
# chapters); a fresh render generates a new one. The id may arrive from the
# /resume/{job_id} path param, so strip it to a safe token (no path
# separators, no CR/LF) before it ever reaches a filesystem path or a log
# line — CodeQL py/path-injection + py/log-injection. Empty after the strip
# → a fresh id.
job_id = re.sub(r"[^A-Za-z0-9_-]", "", job_id or "")[:64] or uuid.uuid4().hex[:16]
try:
from core import job_store
if not resume:
job_store.create(job_id, type=job_type)
job_store.mark_running(job_id)
except Exception:
job_store = None # job history is best-effort; never block synthesis
# Persist a durable resume manifest (plan + params) so an interrupted render
# can be resumed later even without the original script. Best-effort.
try:
title = (metadata or {}).get("title") or (plan.chapters[0].title if plan.chapters else "")
longform_resume.write_manifest(longform_resume.build_manifest(
job_id=job_id, job_type=job_type, title=title,
plan_chapters=[
{"title": c.title, "spans": [s.to_dict() for s in c.spans]}
for c in plan.chapters
],
params={
"default_voice": default_voice, "fmt": fmt, "bitrate": bitrate,
"loudness": loudness, "cover_path": cover_path,
"metadata": metadata, "lexicon": lexicon,
},
))
except Exception: # resume durability is an enhancement; never block the render
logger.debug("[%s] resume manifest write skipped", job_id, exc_info=True)
def _emit(payload: dict) -> str:
if job_store is not None:
try:
job_store.append_event(job_id, json.dumps(payload))
except Exception:
pass # best-effort job history; never block the stream
return f"data: {json.dumps(payload)}\n\n"
if not plan.chapters:
yield _emit({"type": "error", "error": "nothing to render (no chapters)"})
return
ffmpeg = find_ffmpeg()
if not ffmpeg:
yield _emit({"type": "error", "error": "ffmpeg not available; the output needs it"})
return
# Confined work dir (job_id is already token-sanitized above; work_dir adds
# the basename + realpath barrier so CodeQL sees a clean path).
work = longform_resume.work_dir(job_type, job_id)
if work is None:
yield _emit({"type": "error", "error": "invalid job id"})
return
os.makedirs(work, exist_ok=True)
# Chapter WAVs are content-addressed in a shared cache so a re-run (after a
# failure or interruption) reuses what already rendered — only the
# missing/changed chapters synthesize again (resume). Shared across both
# front doors: an identical chapter renders once.
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache")
os.makedirs(cache_dir, exist_ok=True)
prune_cache_dir(cache_dir) # bound disk before this job adds its chapters
loop = asyncio.get_running_loop()
try:
synth, sr, resolve, engine_id = await _prepare_synth(default_voice)
total = len(plan.chapters)
chapter_files: list[str] = []
chapters_meta: list[tuple[str, int]] = []
cached_n = 0
failed: list[int] = []
yield _emit({"type": "started", "job_id": job_id, "chapters": total})
for i, chapter in enumerate(plan.chapters):
try:
wav_path, dur, was_cached = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached,
chapter, synth, sr, engine_id, resolve, cache_dir, lexicon,
)
except Exception: # isolate a bad chapter — keep going
logger.warning("[%s] chapter %d (%s) failed to render",
job_id, i, chapter.title, exc_info=True)
failed.append(i)
yield _emit({"type": "chapter_error", "index": i, "total": total,
"title": chapter.title, "error": "chapter failed to render"})
continue
chapter_files.append(wav_path)
chapters_meta.append((chapter.title, int(round(dur * 1000))))
cached_n += 1 if was_cached else 0
yield _emit({"type": "chapter", "index": i, "total": total,
"title": chapter.title, "duration_s": round(dur, 2),
"cached": was_cached})
if not chapter_files:
yield _emit({"type": "error", "error": "all chapters failed to render"})
return
yield _emit({"type": "assembling"})
meta_path = os.path.join(work, "chapters.ffmeta")
with open(meta_path, "w", encoding="utf-8") as f:
f.write(build_ffmetadata(chapters_meta, global_meta=metadata))
concat_path = os.path.join(work, "concat.txt")
with open(concat_path, "w", encoding="utf-8") as f:
f.write(build_concat_list(chapter_files))
ext = "mp3" if (fmt or "").lower() == "mp3" else "m4b"
out_name = f"{job_type}_{job_id}.{ext}"
out_path = os.path.join(OUTPUTS_DIR, out_name)
# Two-pass loudness master (#28): for a known preset, measure the
# concatenated program first, then feed the measured values back into the
# single mux encode. `measured is None` (skip OR any failure) → the mux
# falls back to single-pass. Gated identically to the pure builders
# (.lower(), no strip), so off/None/unknown/whitespace skip cleanly.
measured = None
norm = (loudness or "").lower()
if norm in LOUDNESS_PRESETS:
yield _emit({"type": "mastering", "preset": norm})
from services.loudness import measure_loudness
measured = await measure_loudness(ffmpeg, concat_path, norm, job_id=job_id)
await run_ffmpeg(
build_render_cmd(
ffmpeg, concat_path, meta_path, out_path,
fmt=ext, bitrate=bitrate, cover_path=_safe_cover_path(cover_path),
loudness=loudness, measured=measured,
),
job_id=job_id,
)
if job_store is not None:
try:
job_store.mark_done(job_id)
except Exception:
pass # best-effort job history
# The render finished — drop the resume manifest so this job is no longer
# offered for resume.
longform_resume.clear_manifest(job_type, job_id)
total_s = sum(d for _, d in chapters_meta) / 1000.0
done = {"type": "done", "output": out_name,
"chapters": len(chapter_files), "duration_s": round(total_s, 2),
"cached_chapters": cached_n, "failed_chapters": failed}
# Loudness verdict only when a preset was requested — off/None paths keep
# the exact legacy `done` shape (additive, old clients unaffected).
if norm in LOUDNESS_PRESETS:
p = LOUDNESS_PRESETS[norm]
done["loudness"] = {
"preset": norm, "target_i": p.i, "target_tp": p.tp,
"two_pass": measured is not None,
"measured_i": measured.input_i if measured else None,
}
yield _emit(done)
except Exception as e: # surface, don't 500 the stream
logger.exception("[%s] longform render failed", job_id)
if job_store is not None:
try:
job_store.mark_failed(job_id, str(e))
except Exception:
pass # best-effort job history
# Generic message only — don't leak the stack/exception text to the client.
yield _emit({"type": "error", "error": "render failed (see backend log)"})
@router.post("/audiobook")
async def audiobook_synthesize(req: AudiobookRequest):
"""Synthesize a chapterized audiobook from a script, streaming SSE progress."""
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
return StreamingResponse(
_render_longform_sse(
plan, default_voice=req.default_voice, fmt=req.format, bitrate=req.bitrate,
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
lexicon=req.lexicon, job_type="audiobook",
),
media_type="text/event-stream",
)
# ── Shared longform render: Stories (and any future front door) post a plan ──
class LongformSpan(BaseModel):
voice_id: str | None = None
text: str
pause_ms_after: int = 0
speed: float | None = None
class LongformChapter(BaseModel):
title: str = ""
spans: list[LongformSpan] = []
class LongformRenderRequest(BaseModel):
chapters: list[LongformChapter] = []
default_voice: str | None = None
bitrate: str = "128k"
format: str = "m4b"
loudness: str | None = None
cover_path: str | None = None
metadata: dict | None = None
lexicon: dict | None = None
@router.post("/longform/render")
async def longform_render(req: LongformRenderRequest):
"""Render a pre-built chapter/span plan (the Stories Editor's compiled
cast+lines) through the shared chapterized renderer — same resume, loudness,
cover, metadata, and output formats as the Audiobook job."""
from services.audiobook import AudiobookPlan, Chapter, Span
if len(req.chapters) > _MAX_CHAPTERS:
raise HTTPException(status_code=422, detail=f"too many chapters (max {_MAX_CHAPTERS})")
chapters = []
for i, c in enumerate(req.chapters):
# Keep a span if it has text to speak OR a pause to render (pause-only
# spans carry inter-line silence with empty text).
spans = [Span(voice_id=s.voice_id, text=(s.text or "").strip(),
pause_ms_after=max(0, int(s.pause_ms_after)), speed=s.speed)
for s in c.spans if ((s.text and s.text.strip()) or s.pause_ms_after > 0)]
if spans:
chapters.append(Chapter(title=c.title or f"Chapter {i + 1}", spans=spans))
plan = AudiobookPlan(chapters=chapters)
return StreamingResponse(
_render_longform_sse(
plan, default_voice=req.default_voice, fmt=req.format, bitrate=req.bitrate,
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
lexicon=req.lexicon, job_type="story",
),
media_type="text/event-stream",
)
# ── Durable resume: interrupted longform renders ────────────────────────────
def _chapters_done(job_id: str) -> int:
"""Count chapters that finished rendering, from the job's persisted events.
Best-effort (0 if unavailable) — used only to show resume progress."""
try:
from core import job_store
n = 0
for ev in job_store.events_since(job_id, 0, limit=100_000):
try:
if json.loads(ev["payload"]).get("type") == "chapter":
n += 1
except (ValueError, KeyError, TypeError):
continue
return n
except Exception:
return 0
@router.get("/audiobook/jobs")
def list_resumable_jobs() -> dict:
"""List interrupted longform renders that can be resumed — a work dir that
still holds a resume manifest (a job left mid-render by a crash/quit). The
ids come from scanning the filesystem, so the UI can offer one-click resume."""
from core import job_store
out = []
for e in longform_resume.scan_resumable():
jid = e["job_id"]
manifest = longform_resume.load_manifest_file(e["manifest_path"]) or {}
job = job_store.get(jid) or {}
out.append({
"job_id": jid,
"type": e["job_type"],
"status": job.get("status", "interrupted"),
"title": manifest.get("title", ""),
"total_chapters": manifest.get("total_chapters", 0),
"chapters_done": _chapters_done(jid),
"created_at": job.get("created_at"),
})
return {"jobs": out}
@router.post("/audiobook/resume/{job_id}")
async def resume_longform(job_id: str):
"""Resume an interrupted longform render from its persisted manifest. The
already-rendered chapters are content-addressed in the shared cache, so they
return instantly — only the unrendered chapters synthesize again. Streams the
same SSE event shape as the original render, under the original job_id."""
from services.audiobook import AudiobookPlan, Chapter, Span
# Find the requested job among the trusted filesystem scan (every path there
# is os.listdir-sourced, never request input) and read its manifest via the
# scan's own trusted path — the request job_id is used ONLY to *select* an
# entry, never to build a path. No request-controlled value reaches a file
# operation (CodeQL py/path-injection-safe).
entry = next((e for e in longform_resume.scan_resumable()
if e["job_id"] == job_id), None)
if entry is None:
raise HTTPException(status_code=404, detail="No resumable job for that id")
manifest = longform_resume.load_manifest_file(entry["manifest_path"])
if not manifest:
raise HTTPException(status_code=404, detail="No resume manifest for that job")
chapters = [
Chapter(title=c.get("title", ""),
spans=[Span(**s) for s in c.get("spans", [])])
for c in manifest["plan"]
]
plan = AudiobookPlan(chapters=chapters)
p = manifest.get("params", {})
# Retire the interrupted job's manifest (trusted scan path) so it stops
# showing as resumable once we've kicked off the fresh-id resume.
longform_resume.discard_manifest_file(entry["manifest_path"])
# Resume under a FRESH job id (job_id=None → a server uuid in the renderer).
# The chapter cache is content-addressed (keyed by chapter content, not the
# job id), so the already-rendered chapters still hit instantly — only the
# unrendered ones synthesize. Using a fresh id means the request's job_id
# never names a work dir / output file (defence-in-depth path-injection).
return StreamingResponse(
_render_longform_sse(
plan, default_voice=p.get("default_voice"),
fmt=p.get("fmt", "m4b"), bitrate=p.get("bitrate", "128k"),
loudness=p.get("loudness"), cover_path=p.get("cover_path"),
metadata=p.get("metadata"), lexicon=p.get("lexicon"),
job_type=entry["job_type"],
),
media_type="text/event-stream",
)
+3
View File
@@ -298,6 +298,9 @@ async def _run_batch_pipeline(job_id: str, job: dict):
denoise=True, postprocess_output=True,
)
audio_out = audios[0]
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(
audio_out,
sample_rate=sr,
+38 -2
View File
@@ -25,12 +25,19 @@ router = APIRouter()
logger = logging.getLogger("omnivoice.capture")
def _truthy(value: Optional[str]) -> bool:
"""Parse a multipart form flag. Treats '1'/'true'/'yes'/'on'/'auto'
(any case) as on; everything else including None as off."""
return (value or "").strip().lower() in {"1", "true", "yes", "on", "auto"}
@router.post("/transcribe")
async def transcribe_audio(
audio: UploadFile = File(...),
language: Optional[str] = Form(None),
model: Optional[str] = Form(None),
mode: Optional[str] = Form(None),
refine: Optional[str] = Form(None),
):
"""Transcribe an audio file to text.
@@ -40,10 +47,19 @@ async def transcribe_audio(
model: Whisper model size (legacy; ignored in dual-mode architecture).
mode: 'fast' (default) uses MLX Turbo for speed; 'accurate' uses
WhisperX with forced alignment for word-level timing.
refine: Opt-in local-LLM cleanup of the final text (disfluencies,
self-corrections, punctuation) same pipeline the live
dictation socket uses. Off by default so MCP/CLI callers don't
pay LLM latency unless they ask; honours the user's
Settings Dictation-refinement config and silently passes
through when no LLM backend is configured. The raw ``text``
is always returned; ``refined_text`` is added only when the
LLM actually changed something.
Returns:
{
"text": "full transcription",
"refined_text": "cleaned text", # only when refine=true changed it
"segments": [ {"start": 0.0, "end": 1.5, "text": "..."}, ... ],
"language": "en",
"duration_s": 4.2,
@@ -91,6 +107,11 @@ async def transcribe_audio(
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
# Wave 1.1: strip Whisper hallucination loops from the final text.
# Segments keep the raw recognition so their timings stay truthful.
from services.refinement import collapse_repetitive_artifacts
full_text = collapse_repetitive_artifacts(full_text)
# Calculate audio duration from segments if available
duration = 0.0
if segments:
@@ -98,12 +119,24 @@ async def transcribe_audio(
detected_lang = result.get("language", language or "unknown")
# Opt-in Wave 2.1 refinement, mirroring the live-dictation socket
# (capture_ws). Off-thread (it's a network call, not GPU); never
# raises — maybe_refine swallows failures and a missing LLM into a
# None pass-through, so the raw text always stands.
refined_text = None
if _truthy(refine) and full_text:
from services.refinement import maybe_refine
refined = await asyncio.to_thread(maybe_refine, full_text)
if refined and refined != full_text:
refined_text = refined
logger.info(
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s",
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
engine_id, elapsed, duration, "accurate" if use_accurate else "fast",
refined_text is not None,
)
return {
response = {
"text": full_text,
"segments": [
{
@@ -118,6 +151,9 @@ async def transcribe_audio(
"transcription_time_s": elapsed,
"engine": engine_id,
}
if refined_text is not None:
response["refined_text"] = refined_text
return response
finally:
try:
os.unlink(tmp.name)
+109 -8
View File
@@ -8,6 +8,12 @@ live dictation feedback.
Protocol:
Client sends binary audio frames (16-bit PCM or WebM/Opus blobs)
Server sends JSON messages:
Opt-in AEC mode (``?aec=1[&sr=16000]``, parity Action 8b): for dictating
while the app plays audio. Frames must be raw int16 mono PCM, each tagged
with a 1-byte prefix 0x00 = microphone, 0x01 = playback reference. The
server runs an NLMS echo canceller, cleaning the mic against the reference
before transcription. Without the param the protocol is unchanged.
{"type": "partial", "text": "Hello wor..."} interim result
{"type": "final", "text": "Hello world.", committed result
"segments": [...], "language": "en",
@@ -25,7 +31,7 @@ import time
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
from api.dependencies import _LOOPBACK_HOSTS
from api.dependencies import _LOOPBACK_HOSTS, ws_remote_authorized
router = APIRouter()
logger = logging.getLogger("omnivoice.capture_ws")
@@ -44,6 +50,56 @@ MIN_BUFFER_BYTES = 64000 # ~2s of 16-bit mono 16kHz — needs enough WebM frame
# to transcribe whatever the user recorded, even short utterances.
MIN_FINAL_BUFFER_BYTES = 4000 # ~125ms of 16-bit mono 16kHz
# ── Dictate-over-playback AEC (parity Action 8b, opt-in) ──────────────────
# Activated by the ``?aec=1`` query param. When OFF (the default), the
# protocol and behaviour are byte-for-byte unchanged. When ON, the client
# streams raw int16 mono PCM frames tagged with a 1-byte type prefix so the
# server can tell mic audio from the playback reference it must cancel:
_AEC_NEAR = 0x00 # microphone frame (clean it, then buffer for ASR)
_AEC_FAR = 0x01 # playback reference frame (feed the echo model only)
def _demux_aec_frame(data: bytes) -> tuple[str, bytes]:
"""Split a prefixed AEC binary frame into ``(kind, pcm)``.
``kind`` is ``"near"`` (mic) or ``"far"`` (playback reference). An empty
or prefix-only frame yields an empty payload. Unknown prefixes are treated
as ``"near"`` so a malformed tag degrades to plain dictation rather than
dropping audio.
"""
if not data:
return "near", b""
kind = "far" if data[0] == _AEC_FAR else "near"
return kind, data[1:]
def _pcm16_to_wav(pcm: bytes, sample_rate: int) -> str | None:
"""Write raw int16 mono PCM to a temp WAV via stdlib ``wave`` (no ffmpeg).
Used on the AEC path, where frames are already decoded PCM the cleaned
samples have no container, so the ffmpeg-sniffing ``_chunks_to_wav`` would
misdetect them. Returns the temp path, or ``None`` for a too-short buffer.
"""
if not pcm or len(pcm) < 100:
return None
import wave
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
tmp.close()
try:
with wave.open(tmp.name, "wb") as wf:
wf.setnchannels(1)
wf.setsampwidth(2) # int16
wf.setframerate(sample_rate)
wf.writeframes(pcm)
return tmp.name
except Exception as e:
logger.debug("PCM->WAV failed: %s", e)
try:
os.unlink(tmp.name)
except OSError:
pass
return None
@router.websocket("/ws/transcribe")
async def ws_transcribe(websocket: WebSocket):
@@ -53,13 +109,33 @@ async def ws_transcribe(websocket: WebSocket):
# WebSocket dependency injection differs across FastAPI versions, so we
# inline the check before accept(). Without it, any local process could
# stream the user's microphone over this endpoint.
# Wave 2.3 (remote backend): a non-loopback client that presents the
# OMNIVOICE_API_KEY bearer is the thin-client dictation case — the mic
# lives on the user's machine, the GPU here — and is allowed through.
host = websocket.client.host if websocket.client else None
if host not in _LOOPBACK_HOSTS:
if host not in _LOOPBACK_HOSTS and not ws_remote_authorized(websocket):
await websocket.close(code=1008, reason="loopback origin required")
return
await websocket.accept()
# Opt-in dictate-over-playback AEC (parity Action 8b). Default OFF →
# identical legacy behaviour. When on, frames are 1-byte-tagged raw PCM
# and the cleaned mic stream is muxed via stdlib wave (not ffmpeg).
aec = None
pcm_sr: int | None = None
if websocket.query_params.get("aec") in ("1", "true", "on"):
try:
pcm_sr = int(websocket.query_params.get("sr", "16000"))
from services.aec import NlmsEchoCanceller
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
logger.info("AEC enabled for dictation session (sr=%d)", pcm_sr)
except Exception as e:
# Bad sr or import failure → fall back to plain dictation.
logger.warning("AEC requested but disabled: %s", e)
aec = None
pcm_sr = None
audio_chunks: list[bytes] = []
total_bytes = 0
last_audio_time = time.monotonic()
@@ -97,6 +173,16 @@ async def ws_transcribe(websocket: WebSocket):
# Empty binary frame also acts as EOF — connection stays open.
running = False
break
if aec is not None:
# Tagged PCM: route the playback reference into the echo
# model and clean the mic before it reaches the buffer.
kind, payload = _demux_aec_frame(data)
if kind == "far":
aec.push_far_end(payload)
continue
if not payload:
continue
data = aec.process_near_end(payload)
audio_chunks.append(data)
total_bytes += len(data)
last_audio_time = time.monotonic()
@@ -143,7 +229,7 @@ async def ws_transcribe(websocket: WebSocket):
# Transcribe current buffer
try:
text = await _transcribe_buffer(audio_chunks[:])
text = await _transcribe_buffer(audio_chunks[:], pcm_sr=pcm_sr)
if text and text != partial_text:
partial_text = text
await _safe_send({
@@ -173,7 +259,16 @@ async def ws_transcribe(websocket: WebSocket):
# Final transcription on complete buffer — skip if client already gone.
if total_bytes > MIN_FINAL_BUFFER_BYTES:
try:
result = await _transcribe_buffer_full(audio_chunks)
result = await _transcribe_buffer_full(audio_chunks, pcm_sr=pcm_sr)
# Wave 2.1: optional local-LLM refinement of the final text.
# Off-thread (network call, not GPU); pass-through on any
# failure or when no LLM backend is configured. The raw text
# always ships too — clients paste refined_text ?? text.
if result.get("text"):
from services.refinement import maybe_refine
refined = await asyncio.to_thread(maybe_refine, result["text"])
if refined and refined != result["text"]:
result["refined_text"] = refined
if not await _safe_send({"type": "final", **result}):
logger.debug("Skipped final send — client already disconnected")
except Exception as e:
@@ -197,10 +292,10 @@ async def ws_transcribe(websocket: WebSocket):
pass
async def _transcribe_buffer(chunks: list[bytes]) -> str:
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
"""Quick partial transcription of the current audio buffer."""
tmp = _chunks_to_wav(chunks)
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
if tmp is None:
return ""
@@ -223,9 +318,9 @@ async def _transcribe_buffer(chunks: list[bytes]) -> str:
pass
async def _transcribe_buffer_full(chunks: list[bytes]) -> dict:
async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict:
"""Full transcription with timing info for the final result."""
tmp = _chunks_to_wav(chunks)
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
if tmp is None:
return {"text": "", "segments": [], "language": "unknown",
"duration_s": 0, "transcription_time_s": 0, "engine": "none"}
@@ -245,6 +340,12 @@ async def _transcribe_buffer_full(chunks: list[bytes]) -> dict:
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
# Wave 1.1: strip Whisper hallucination loops from the final
# text (the string that gets auto-pasted). Segments keep the
# raw recognition so their timings stay truthful.
from services.refinement import collapse_repetitive_artifacts
full_text = collapse_repetitive_artifacts(full_text)
duration = max((s.get("end", 0) for s in segments), default=0.0)
return {
+9 -3
View File
@@ -258,13 +258,19 @@ async def community_use(item_id: str, name: Optional[str] = Query(None)):
raise HTTPException(status_code=503, detail=f"Couldn't add this voice right now. Error: {e}")
try:
# A community "preset" is a synthetic designed voice (rendered from an
# instruct string) → kind='design'; a "voice" carries a real reference
# clip → kind='clone'. Setting kind makes the persona-gallery
# synthetic-only gating work (§R3) instead of defaulting all imports to
# 'clone'.
kind = "design" if item["type"] == "preset" else "clone"
with db_conn() as conn:
conn.execute(
"INSERT INTO voice_profiles "
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at, kind) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, profile_name, audio_filename, ref_text, instruct,
item.get("language", "Auto"), None, item["id"], time.time()),
item.get("language", "Auto"), None, item["id"], time.time(), kind),
)
except Exception:
with __import__("contextlib").suppress(OSError):
+35
View File
@@ -0,0 +1,35 @@
"""Voice-design "describe your voice" API (issue #317).
Maps a free-text voice description onto the existing design parameter space
via the deterministic keyword mapper in ``core.describe_voice``. Pure CPU +
stdlib no model, no network so it imports and responds instantly in any
environment, including test/CI without model weights.
"""
from __future__ import annotations
from fastapi import APIRouter
from pydantic import BaseModel, Field
from core.describe_voice import parse_description
router = APIRouter()
class DescribeRequest(BaseModel):
description: str = Field(default="", max_length=2000)
@router.post("/design/describe")
def describe_voice(req: DescribeRequest) -> dict:
"""Parse a free-text description into design attrs + a validator-safe instruct.
Response shape::
{
"attrs": {"Gender": "female", "Age": "elderly", ... or "Auto"},
"instruct": "female, elderly, low pitch, british accent",
"matched": [{"category": "Age", "token": "elderly", "phrase": "elderly"}, ...],
"unmatched": ["slightly raspy"]
}
"""
return parse_description(req.description)
+72 -5
View File
@@ -25,6 +25,7 @@ from services.segmentation import (
assign_speakers_heuristic,
clean_up_segments,
)
from services.onset_align import snap_segment_starts
from services import dub_pipeline
router = APIRouter()
@@ -366,7 +367,11 @@ _prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local
@router.get("/dub/transcribe-stream/{job_id}")
async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None):
async def dub_transcribe_stream(
job_id: str,
num_speakers: Optional[int] = None,
per_segment_refs: bool = True,
):
"""Stream per-chunk segments via SSE, then emit diarized final pass.
Pre-flight checks (missing job, missing audio, ASR not loaded) are emitted
@@ -548,6 +553,15 @@ async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None)
detected_lang = part["language"]
asr_speaker_turns.extend(part.get("speaker_turns") or [])
chunk_segs = segment_transcript(part, duration=t1, scene_cuts=scene_cuts)
# #280: Whisper often stretches a segment's start back over
# leading music/silence (classic case: speech begins at 0:03,
# transcript says 0.0 → the dub plays 3 s early). Snap starts
# forward to the actual speech onset. `audio_np` is the same
# track ASR ran on — vocals.wav when Demucs succeeded.
try:
snap_segment_starts(chunk_segs, audio_np, sr)
except Exception as e:
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
chunk_segs = assign_speakers_heuristic(chunk_segs)
for s in chunk_segs:
s["id"] = f"s{next_seg_id:05x}"
@@ -759,16 +773,61 @@ async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None)
clones = done.pop().result()
break
yield _sse_event("ping", {})
if clones:
job["speaker_clones"] = clones
# Default each segment's profile_id to its speaker's auto-clone,
# but only if the user hasn't already assigned something.
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
# own reference from the vocals so the dub of each line matches the
# prosody of its source line. Short lines fall back to the
# per-speaker clone below. Default on; the user can force
# per-speaker by disabling it (job["per_segment_refs"]).
seg_clones = {}
job["per_segment_refs"] = per_segment_refs
if per_segment_refs:
try:
from services.speaker_clone import extract_segment_refs
seg_ids_for_clone = [s.get("id", i) for i, s in enumerate(final_segs)]
seg_clones = await loop.run_in_executor(
_cpu_pool, lambda: extract_segment_refs(
vocals_for_clone, final_segs,
os.path.dirname(vocals_for_clone),
seg_ids=seg_ids_for_clone,
),
)
if seg_clones:
job["segment_clones"] = seg_clones
except Exception as e:
logger.warning("per-segment clone refs skipped: %s", e)
if clones or seg_clones:
if clones:
job["speaker_clones"] = clones
# Default each segment's profile_id to its detected speaker's
# auto-clone — but only if the user hasn't already assigned
# something. (#486)
#
# We prefer the UI-visible `auto:{speaker}` id over the
# per-segment `auto-seg:{id}` id even when a per-segment ref
# exists, because the dub editor's Voice dropdown only renders
# `auto:` options ("From Video → Speaker N"). An `auto-seg:`
# value matches no <option>, so the row silently read
# "Default" while the speaker was actually bound — exactly the
# reported bug. The per-segment ref is NOT lost: dub_generate's
# `auto:` branch transparently prefers this segment's own
# per-segment ref (job["segment_clones"][seg_id]) when present,
# so a row shown as "Speaker 1" still clones from its own line
# when that line is long enough.
for s in final_segs:
if s.get("profile_id"):
continue
spk = s.get("speaker_id") or "Speaker 1"
if spk in clones:
s["profile_id"] = auto_profile_id(spk)
continue
# No per-speaker clone for this speaker (too little usable
# audio overall) but this single line was long enough for
# its own ref — fall back to the per-segment id. The editor
# can't render it, but generation still clones correctly.
sid = str(s.get("id", ""))
if sid and sid in seg_clones:
s["profile_id"] = f"auto-seg:{sid}"
except Exception as e:
logger.warning("speaker_clone extraction skipped: %s", e)
@@ -862,6 +921,14 @@ async def dub_transcribe(job_id: str):
scene_cuts = job.get("scene_cuts") or []
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
# #280: snap segment starts forward to the actual speech onset so the
# dub doesn't begin seconds before the original speaker does.
try:
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
snap_segment_starts(segments, audio_for_onset, onset_sr)
except Exception as e:
logger.warning("onset alignment skipped: %s", e)
diar_pipe = get_diarization_pipeline()
if diar_pipe:
try:
+647 -99
View File
@@ -1,5 +1,7 @@
import os
import io
import re
import json
import time
import uuid
import asyncio
@@ -12,6 +14,13 @@ from core.config import DUB_DIR, dub_seg_path
from core.tasks import task_manager
from api.routers.dub_core import _get_job
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
from services.video_retime import (
DRIFT_TOLERANCE_S,
RetimeError,
build_chunk_filter_graph,
expand_retime_chunks,
prepare_smart_fit_video,
)
router = APIRouter()
logger = logging.getLogger("omnivoice.api")
@@ -22,6 +31,9 @@ def _unique_stamp() -> str:
return f"{time.strftime('%Y%m%dT%H%M%S')}-{uuid.uuid4().hex[:8]}"
_SAFE_LANG = re.compile(r"^[A-Za-z0-9_-]{1,32}$")
def _native_save(source: str, destination: str, display_name: str, media_type: str):
"""Copy a generated export file to a user-chosen destination and return JSON."""
import shutil
@@ -158,15 +170,23 @@ async def dub_list_tracks(job_id: str):
return {"tracks": job.get("dubbed_tracks", {})}
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool) -> str | None:
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
fitted_segments: "list[dict] | None" = None) -> str | None:
"""Build a temp SRT from job segments for use with ffmpeg's subtitles filter.
Returned path is already ffmpeg-filter-safe (plain ASCII basename under exports_dir).
Returns None if there are no segments to render.
``fitted_segments`` (Smart Fit): {id, start, end} cue records on the
fitted timeline when provided, cue times come from there instead of
the original ``job["segments"]`` timings, so burned subs track the
retimed video / fitted audio rather than the source timeline.
"""
segments = job.get("segments", [])
if not segments:
return None
if fitted_segments:
segments = _apply_fitted_times(segments, fitted_segments)
lines = []
for i, seg in enumerate(segments):
lines.append(str(i + 1))
@@ -214,41 +234,14 @@ def _build_video_stretch_filter_graph(
if not plan:
return "", in_label or f"[{video_input_idx}:v]"
chunks: list[tuple[float, float, float]] = [] # (a, b, ratio)
cursor = 0.0
for entry in plan:
a = float(entry["orig_start"])
b = float(entry["orig_end"])
if a > cursor + 1e-3:
chunks.append((cursor, a, 1.0)) # gap or pre-roll at native rate
ratio = float(entry["stretch_ratio"])
if b > a:
chunks.append((a, b, ratio))
cursor = max(cursor, b)
if orig_dur > cursor + 1e-3:
chunks.append((cursor, orig_dur, 1.0)) # tail at native rate
chunks = [(a, b, r) for (a, b, r) in chunks if b > a]
# Chunk expansion + graph emission live in services.video_retime now so
# the Smart Fit batched pipeline shares the exact same boundary math.
# With default options the emitted graph is byte-identical to the
# original inline implementation.
chunks = expand_retime_chunks(plan, orig_dur)
if not chunks:
return "", in_label or f"[{video_input_idx}:v]"
src = in_label or f"[{video_input_idx}:v]"
parts: list[str] = []
labels: list[str] = []
# `split` lets us tap the same source stream once per chunk without re-
# decoding. setpts={ratio}*PTS slows down (ratio > 1) or speeds up
# (ratio < 1) each chunk; PTS-STARTPTS first to normalise the timestamp
# base after the trim.
split_labels = [f"[vsplit{idx}]" for idx in range(len(chunks))]
parts.append(f"{src}split={len(chunks)}{''.join(split_labels)}")
for idx, ((a, b, ratio), split_lbl) in enumerate(zip(chunks, split_labels)):
out_label = f"[vstr{idx}]"
labels.append(out_label)
parts.append(
f"{split_lbl}trim=start={a:.4f}:end={b:.4f},"
f"setpts=PTS-STARTPTS,setpts={ratio:.6f}*PTS{out_label}"
)
parts.append("".join(labels) + f"concat=n={len(chunks)}:v=1:a=0[vstretched]")
return ";".join(parts), "[vstretched]"
return build_chunk_filter_graph(chunks, in_label or f"[{video_input_idx}:v]")
def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
@@ -264,6 +257,76 @@ def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
return entry
def _video_retime_plan_for(job: dict, lang_code: str) -> "tuple[str, dict] | None":
"""Resolve the video retime plan for ``lang_code`` across both keyspaces.
Returns ``(kind, entry)`` where kind is ``"stretch_video"`` (legacy
Mode B plans resolution byte-identical to ``_video_stretch_plan_for``)
or ``"smart_fit"`` (Phase A ``job["fit_plans"]`` entries, gated on the
track actually having been generated under smart_fit so a stale plan
from an earlier run can't retime a track re-generated under another
strategy). ``None`` when neither applies.
"""
legacy = _video_stretch_plan_for(job, lang_code)
if legacy is not None:
return "stretch_video", legacy
entry = (job.get("fit_plans") or {}).get(lang_code)
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
if entry and entry.get("plan") and track.get("timing_strategy") == "smart_fit":
return "smart_fit", entry
return None
def _fitted_segments_for(job: dict, lang_code: "str | None") -> "list[dict] | None":
"""Fitted-timeline subtitle cues ({id, start, end}) for a Smart Fit
track, or None. Same staleness gate as ``_video_retime_plan_for``."""
if not lang_code:
return None
entry = (job.get("fit_plans") or {}).get(lang_code)
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
if not entry or track.get("timing_strategy") != "smart_fit":
return None
fitted = entry.get("fitted_segments")
return fitted or None
def _apply_fitted_times(segments: list[dict], fitted: list[dict]) -> list[dict]:
"""Overlay fitted cue times onto subtitle segments (copies; non-destructive).
Matches by segment ``id``; when the fitted record carries no ids at all
(defensive), falls back to positional pairing. Segments without a match
keep their original timings.
"""
by_id = {str(f["id"]): f for f in fitted if f.get("id") is not None}
out: list[dict] = []
for i, seg in enumerate(segments):
cue = None
if seg.get("id") is not None:
cue = by_id.get(str(seg["id"]))
if cue is None and not by_id and i < len(fitted):
cue = fitted[i]
if cue is None:
out.append(seg)
continue
patched = dict(seg)
patched["start"] = float(cue["start"])
patched["end"] = float(cue["end"])
out.append(patched)
return out
def _burn_subs_allowed(retime_kind: "str | None") -> bool:
"""Subtitle burn-in combined with video retime.
Allowed for ``smart_fit`` (fitted cue records exist, and the burn pass
runs AFTER the retime graph so cues land on the retimed timeline) and
for plain exports. Still rejected for legacy ``stretch_video``, which
has no fitted-cue record cues would burn at original timestamps onto
a re-timed video and drift.
"""
return retime_kind != "stretch_video"
#: Audio export formats → ffmpeg codec args. Unknown formats fall back to
#: AAC/m4a so a bad request can never produce a broken command.
_AUDIO_FORMAT_CODECS: dict[str, list[str]] = {
@@ -312,6 +375,11 @@ async def dub_download(
dual: bool = Query(False, description="When burn_subs=1, render translated on top of italicised original."),
out_format: str = Query("m4a", description="Audio-only jobs (#119): output container — wav, m4a, mp3, or flac. Ignored for video jobs."),
):
# Strict allowlist on the path param BEFORE it reaches any filesystem
# path or ffmpeg argv (export dir, retime work path, slice paths). Real
# job ids are short uuid slices — alnum/hyphen/underscore only.
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
raise HTTPException(status_code=400, detail="Invalid job id")
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -390,18 +458,24 @@ async def dub_download(
)
# Determine whether this export should drive video through a per-segment
# stretch graph (Mode B). Stretch is keyed off the default_track's plan
# because the video can only physically follow one timeline at a time.
# If multiple dub tracks are included and they were generated under
# stretch_video, only the default_track is visually in sync — other
# tracks share the same (stretched) video. Single-track export is the
# supported common case.
stretch_entry = _video_stretch_plan_for(job, default_track) if default_track and default_track != "original" else None
# Subtitle burn under stretch_video would render cues at the original
# timestamps onto a re-timed video — they'd drift. Skip the burn pass
# in that combo and log; the user can still export the SRT/VTT
# separately and the new-layout timing lives there.
if stretch_entry and burn_subs:
# retime (legacy stretch_video Mode B, or Smart Fit). Retime is keyed off
# the default_track's plan because the video can only physically follow
# one timeline at a time. If multiple dub tracks are included, only the
# default_track is visually in sync — other tracks share the same
# (retimed) video. Single-track export is the supported common case.
retime_kind: "str | None" = None
retime_entry: "dict | None" = None
if default_track and default_track != "original":
_retime = _video_retime_plan_for(job, default_track)
if _retime:
retime_kind, retime_entry = _retime
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
# Subtitle burn under legacy stretch_video would render cues at the
# original timestamps onto a re-timed video — they'd drift (no fitted-cue
# record exists for that mode). Skip the burn pass in that combo and log;
# the user can still export the SRT/VTT separately. Smart Fit DOES carry
# fitted cues, so burn+retime is allowed there (burn runs post-retime).
if not _burn_subs_allowed(retime_kind) and burn_subs:
logger.warning(
"stretch_video + burn_subs is not supported in one pass; "
"skipping subtitle burn for job %s. Export the SRT/VTT separately.",
@@ -409,11 +483,71 @@ async def dub_download(
)
burn_subs = False
sub_path = _write_burn_srt(job, exports_dir, stamp, dual) if burn_subs else None
# Smart Fit: cue times come from the fitted timeline — that's where the
# dubbed audio actually sits, whether or not the video retime succeeds.
fitted_segments = _fitted_segments_for(job, default_track) if default_track and default_track != "original" else None
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments) if burn_subs else None
# ── Smart Fit video retime (two-tier) ─────────────────────────────────
# Tier 1 (≤48 chunks): single filter_complex graph inlined into the mux
# command below. Tier 2: batched slice renders joined by the concat
# demuxer into an intermediate file, muxed as an extra input. Failures
# fall back to an un-retimed export with a structured warning rather
# than failing the whole download.
retime_decision = None
retime_warning: "dict | None" = None
smart_track_dur = 0.0
if retime_kind == "smart_fit" and retime_entry:
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
smart_track_dur = float(
retime_entry.get("total_duration")
or (filtered_tracks.get(default_track) or {}).get("duration")
or 0.0
)
# A fresh export is a fresh user intent — clear any sticky abort flag
# from a previous /dub/abort so it can't kill this run's first batch.
job.pop("aborted", None)
# realpath-normalised + containment-checked inline at the sink (the
# file's established pattern — CodeQL does not track the guard
# through a helper's return value).
_base = os.path.realpath(DUB_DIR)
retime_work_path = os.path.realpath(
os.path.join(exports_dir, f"retimed_{stamp}.mp4")
)
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid export path")
try:
retime_decision = await prepare_smart_fit_video(
job_id=job_id,
ffmpeg=ffmpeg,
video_path=video_path,
plan=retime_entry["plan"],
orig_dur=smart_orig_dur,
track_dur=smart_track_dur,
work_path=retime_work_path,
abort_check=lambda: bool(job.get("aborted")),
)
except Exception as e:
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
raise HTTPException(status_code=409, detail="Export aborted")
from core.failure import build_failure
retime_warning = build_failure(e, stage="video-retime", include_diagnostic=False)
job["last_export_warning"] = {"type": "video_retime_fallback", **retime_warning}
logger.error(
"Smart Fit video retime failed for job %s — exporting "
"without per-segment retime: %s",
job_id.replace("\n", " ").replace("\r", " "), e,
)
cmd = [ffmpeg, "-i", video_path]
input_idx = 1
retimed_idx = None
if retime_decision is not None and retime_decision.mode == "file":
cmd += ["-i", retime_decision.file_path]
retimed_idx = input_idx
input_idx += 1
bg_audio = job.get("no_vocals_path") if preserve_bg else None
bg_idx = None
if bg_audio and os.path.exists(bg_audio) and filtered_tracks:
@@ -429,9 +563,37 @@ async def dub_download(
filter_parts: list[str] = []
video_map = "0:v:0"
video_reencode = False
if retime_decision is not None:
if retime_decision.mode == "filter":
filter_parts.append(retime_decision.graph)
video_map = retime_decision.label
video_reencode = True
else:
video_map = f"{retimed_idx}:v:0"
# Residual drift after the batched render (fps rounding): video
# shorter than the fitted track → freeze the last frame out to
# the track length. Rare — the predicted tail pad inside the
# render usually lands within tolerance.
residual = smart_track_dur - retime_decision.video_dur
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
filter_parts.append(
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
)
video_map = "[vtpad]"
video_reencode = True
if sub_path:
esc = _ffmpeg_filter_escape(sub_path)
filter_parts.append(f"[0:v]subtitles='{esc}'[vsub]")
# Burn AFTER any retime so cues (already on the fitted timeline for
# Smart Fit) land on the retimed video. Without retime this reduces
# to the legacy `[0:v]subtitles=…[vsub]` graph.
if video_map.startswith("["):
sub_src = video_map
elif retimed_idx is not None:
sub_src = f"[{retimed_idx}:v]"
else:
sub_src = "[0:v]"
filter_parts.append(f"{sub_src}subtitles='{esc}'[vsub]")
video_map = "[vsub]"
if stretch_entry:
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
@@ -447,10 +609,32 @@ async def dub_download(
if include_original:
cmd += ["-map", "0:a:0"]
# Smart Fit drift absorption, audio side: when the retimed video runs
# longer than the fitted track (its tail passes through at 1.0× beyond
# the last cue, or encoder rounding), pad the dub-track chain with
# silence out to the video length so players don't end audio early.
apad_dur = 0.0
if (
retime_decision is not None
and smart_track_dur
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
):
apad_dur = retime_decision.video_dur
if bg_idx is not None:
for i, t in enumerate(tracks_to_process):
out_label = f"[aout{i}]"
filter_parts.append(f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2{out_label}")
chain = f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
if apad_dur:
chain += f",apad=whole_dur={apad_dur:.4f}"
filter_parts.append(chain + out_label)
t["out_label"] = out_label
for t in tracks_to_process:
cmd += ["-map", t["out_label"]]
elif apad_dur:
for i, t in enumerate(tracks_to_process):
out_label = f"[aout{i}]"
filter_parts.append(f"[{t['idx']}:a]apad=whole_dur={apad_dur:.4f}{out_label}")
t["out_label"] = out_label
for t in tracks_to_process:
cmd += ["-map", t["out_label"]]
@@ -461,10 +645,11 @@ async def dub_download(
if filter_parts:
cmd += ["-filter_complex", ";".join(filter_parts)]
# Burning subs or per-segment video stretch both force a real video
# re-encode; stream-copy is only viable when nothing touches the video
# filter chain.
if sub_path or stretch_entry:
# Burning subs or per-segment video retime both force a real video
# re-encode; stream-copy is viable when nothing touches the video
# filter chain — including the batched Smart Fit path, whose retimed
# intermediate is already encoded with these exact settings.
if sub_path or stretch_entry or video_reencode:
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
else:
cmd += ["-c:v", "copy"]
@@ -497,17 +682,17 @@ async def dub_download(
break
cmd += [f"-disposition:a:{target_idx}", "default"]
# In stretch_video mode the video and audio durations should match
# within sub-frame precision, but `-shortest` can still cut off the
# trailing frame; let ffmpeg keep both streams. Otherwise keep the
# legacy `-shortest` so a slightly-overrunning track doesn't extend
# the mux past the video.
if not stretch_entry:
# When retiming (legacy stretch_video or Smart Fit) the video and audio
# durations should match within sub-frame precision, but `-shortest`
# can still cut off the trailing frame; let ffmpeg keep both streams.
# Otherwise keep the legacy `-shortest` so a slightly-overrunning track
# doesn't extend the mux past the video.
if not stretch_entry and retime_decision is None:
cmd += ["-shortest"]
cmd += [output_path, "-y"]
try:
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0, job_id=job_id)
if rc != 0:
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
except asyncio.TimeoutError:
@@ -519,6 +704,14 @@ async def dub_download(
status_code=500,
detail=f"ffmpeg failed to combine video + dubbed audio: {e}. Verify ffmpeg is installed (`ffmpeg -version`), and check that every dubbed track file exists in the job folder.",
)
finally:
# The batched retime intermediate is a full re-encoded video — never
# leave it behind (success or failure; it's stamp-unique, no reuse).
if retime_decision is not None and retime_decision.mode == "file":
try:
os.remove(retime_decision.file_path)
except OSError as e:
logger.debug("cleanup remove failed: %s", e)
if not os.path.exists(output_path) or os.path.getsize(output_path) == 0:
raise HTTPException(status_code=500, detail="ffmpeg mux produced no output file")
@@ -528,12 +721,22 @@ async def dub_download(
safe_name = ''.join(c for c in base_name if c.isalnum() or c in '-_ ').strip() or 'output'
dl_name = f"dubbed_{safe_name}_{stamp}.mp4"
# Structured warning surface for the Smart Fit fallback ladder: header is
# a fixed ASCII token (FileResponse headers must be latin-1 safe); the
# full build_failure payload is persisted on the job for the UI to read.
extra_headers = {}
if retime_warning is not None:
extra_headers["X-Dub-Export-Warning"] = "video-retime-fallback"
if save_path:
return _native_save(output_path, save_path, dl_name, media_type="video/mp4")
result = _native_save(output_path, save_path, dl_name, media_type="video/mp4")
if retime_warning is not None:
result["warning"] = {"type": "video_retime_fallback", **retime_warning}
return result
return FileResponse(
output_path, media_type="video/mp4",
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
headers={"Content-Disposition": f'attachment; filename="{dl_name}"', **extra_headers},
)
@@ -568,6 +771,21 @@ async def dub_get_media(job_id: str):
ext = os.path.splitext(video_path)[1].lower()
return FileResponse(video_path, media_type=_MEDIA_TYPES.get(ext, "video/mp4"))
# One mux at a time per preview file. Without this, two overlapping requests
# (e.g. the <video> element remounting right after a re-dub) both ran ffmpeg
# against the same output path, and the mtime check below saw the half-written
# file as a valid cache — serving a truncated MP4 that left the player stuck
# loading forever (#281).
_preview_mux_locks: dict[str, asyncio.Lock] = {}
def _preview_lock(path: str) -> asyncio.Lock:
lock = _preview_mux_locks.get(path)
if lock is None:
lock = _preview_mux_locks.setdefault(path, asyncio.Lock())
return lock
@router.get("/dub/preview-video/{job_id}")
async def dub_preview_video(
job_id: str,
@@ -579,6 +797,11 @@ async def dub_preview_video(
Caches per lang+preserve_bg combination under exports/preview_{lang}_{bg}.mp4.
Cache is invalidated when the underlying dubbed track mtime is newer than the cache.
"""
# Strict allowlist on the path param BEFORE it reaches any filesystem
# path or ffmpeg argv (exports dir, preview/retime work paths) — same
# boundary check as dub_download.
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
raise HTTPException(status_code=400, detail="Invalid job id")
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -599,23 +822,92 @@ async def dub_preview_video(
bg_audio = job.get("no_vocals_path") if preserve_bg else None
has_bg = bool(bg_audio and os.path.exists(bg_audio))
exports_dir = os.path.join(DUB_DIR, job_id, "exports")
if not _SAFE_LANG.match(lang):
raise HTTPException(status_code=400, detail="Invalid lang")
# realpath-normalised + containment-checked inline BEFORE any filesystem
# access so the guard dominates every sink (the file's established
# pattern — see dub_preview_segment; CodeQL does not track the guard
# through a helper's return value).
_base = os.path.realpath(DUB_DIR)
exports_dir = os.path.realpath(os.path.join(_base, job_id, "exports"))
if not exports_dir.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid job id")
os.makedirs(exports_dir, exist_ok=True)
bg_suffix = "bg" if (preserve_bg and has_bg) else "nobg"
preview_path = os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
preview_path = os.path.realpath(
os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
)
if not preview_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid path")
track_mtime = os.path.getmtime(track_path)
cache_ok = (
os.path.exists(preview_path)
and os.path.getsize(preview_path) > 0
and os.path.getmtime(preview_path) >= track_mtime
)
if not cache_ok:
def _cache_ok() -> bool:
return (
os.path.exists(preview_path)
and os.path.getsize(preview_path) > 0
and os.path.getmtime(preview_path) >= track_mtime
)
async def _mux_preview():
# Mux into a temp file and os.replace() into place so a concurrent
# reader never sees a partially-written preview (#281: video stuck
# loading forever after a re-dub).
mux_path = preview_path + ".tmp.mp4"
ffmpeg = find_ffmpeg()
stretch_entry = _video_stretch_plan_for(job, lang)
# Resolve the same retime plan the download path uses (legacy
# stretch_video or Smart Fit) so the in-app preview matches export.
retime = _video_retime_plan_for(job, lang)
retime_kind, retime_entry = retime if retime else (None, None)
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
retime_decision = None
smart_track_dur = 0.0
if retime_kind == "smart_fit" and retime_entry:
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
smart_track_dur = float(
retime_entry.get("total_duration") or track_info.get("duration") or 0.0
)
job.pop("aborted", None) # fresh user intent — clear sticky abort
# realpath-normalised + containment-checked inline at the sink
# (same pattern as preview_path above — _base is the realpath
# of DUB_DIR from the top of this endpoint).
retime_work_path = os.path.realpath(os.path.join(
exports_dir, f"preview_retimed_{lang}_{bg_suffix}.tmp.mp4",
))
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid export path")
try:
retime_decision = await prepare_smart_fit_video(
job_id=job_id,
ffmpeg=ffmpeg,
video_path=video_path,
plan=retime_entry["plan"],
orig_dur=smart_orig_dur,
track_dur=smart_track_dur,
work_path=retime_work_path,
abort_check=lambda: bool(job.get("aborted")),
)
except Exception as e:
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
raise HTTPException(status_code=409, detail="Preview aborted")
# Preview is best-effort: fall back to the un-retimed video
# rather than a black player. The export path surfaces the
# structured warning; here we just log.
retime_decision = None
logger.error(
"Smart Fit preview retime failed for job %s — previewing "
"without per-segment retime: %s",
job_id.replace("\n", " ").replace("\r", " "), e,
)
cmd = [ffmpeg, "-i", video_path]
input_idx = 1
retimed_idx = None
if retime_decision is not None and retime_decision.mode == "file":
cmd += ["-i", retime_decision.file_path]
retimed_idx = input_idx
input_idx += 1
if preserve_bg and has_bg:
cmd += ["-i", bg_audio]
bg_idx = input_idx
@@ -625,12 +917,13 @@ async def dub_preview_video(
cmd += ["-i", track_path]
track_idx = input_idx
# Build filter graph. In stretch_video mode we splice the source
# video into per-segment chunks, setpts each to match the dub audio
# Build filter graph. Under a retime plan we splice the source video
# into per-segment chunks, setpts each to match the dub audio
# layout, and concat them — so audio plays at natural rate and the
# visuals follow. Otherwise we stream-copy video for speed.
filter_parts: list[str] = []
video_map = "0:v:0"
video_reencode = False
if stretch_entry:
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
graph, vlabel = _build_video_stretch_filter_graph(
@@ -639,49 +932,175 @@ async def dub_preview_video(
if graph:
filter_parts.append(graph)
video_map = vlabel
elif retime_decision is not None:
if retime_decision.mode == "filter":
filter_parts.append(retime_decision.graph)
video_map = retime_decision.label
video_reencode = True
else:
video_map = f"{retimed_idx}:v:0"
residual = smart_track_dur - retime_decision.video_dur
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
filter_parts.append(
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
)
video_map = "[vtpad]"
video_reencode = True
# Smart Fit drift absorption (audio): silence-pad the dub chain out
# to the retimed video length so the preview doesn't end audio early.
apad_dur = 0.0
if (
retime_decision is not None
and smart_track_dur
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
):
apad_dur = retime_decision.video_dur
audio_map = f"{track_idx}:a:0"
if bg_idx is not None:
filter_parts.append(
f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2[aout]"
)
chain = f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
if apad_dur:
chain += f",apad=whole_dur={apad_dur:.4f}"
filter_parts.append(chain + "[aout]")
audio_map = "[aout]"
elif apad_dur:
filter_parts.append(f"[{track_idx}:a]apad=whole_dur={apad_dur:.4f}[aout]")
audio_map = "[aout]"
cmd += ["-map", video_map]
if bg_idx is not None:
cmd += ["-map", "[aout]"]
else:
cmd += ["-map", f"{track_idx}:a:0"]
cmd += ["-map", audio_map]
if filter_parts:
cmd += ["-filter_complex", ";".join(filter_parts)]
# Stretch path needs a real encode; stream-copy otherwise.
if stretch_entry:
# Retime path needs a real encode; stream-copy otherwise (the batched
# Smart Fit intermediate is already encoded — copy unless tpad'ed).
if stretch_entry or video_reencode:
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
else:
cmd += ["-c:v", "copy"]
cmd += ["-c:a", "aac", "-b:a", "192k"]
# `-shortest` would cut the stretched video at the (slightly different)
# `-shortest` would cut the retimed video at the (slightly different)
# audio length and lose the trailing frame; only use it on the copy path.
if not stretch_entry:
if not stretch_entry and retime_decision is None:
cmd += ["-shortest"]
cmd += [preview_path, "-y"]
cmd += [mux_path, "-y"]
def _discard_tmp():
try:
os.remove(mux_path)
except OSError as e:
logger.debug("cleanup remove failed: %s", e)
try:
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0)
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0, job_id=job_id)
if rc != 0:
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
if not os.path.exists(mux_path) or os.path.getsize(mux_path) == 0:
raise Exception("preview mux produced empty file")
except asyncio.TimeoutError:
_discard_tmp()
raise HTTPException(status_code=504, detail="preview mux timed out")
except HTTPException:
_discard_tmp()
raise
except Exception as e:
_discard_tmp()
raise HTTPException(
status_code=500,
detail=f"ffmpeg failed to build the preview stream: {str(e)[:300]}. This usually means the source video can't be re-encoded on the fly — try downloading the MP4 instead.",
)
finally:
if retime_decision is not None and retime_decision.mode == "file":
try:
os.remove(retime_decision.file_path)
except OSError as e:
# Best-effort scratch cleanup — never fail the export.
logger.debug("retime intermediate cleanup failed: %s", e)
if not os.path.exists(preview_path) or os.path.getsize(preview_path) == 0:
raise HTTPException(status_code=500, detail="preview mux produced empty file")
os.replace(mux_path, preview_path)
return FileResponse(preview_path, media_type="video/mp4")
async with _preview_lock(preview_path):
if not _cache_ok():
await _mux_preview()
# no-store: the URL is stable across re-dubs, so any HTTP-level caching
# in the WebView would keep showing the previous dub after a re-generate
# (#281: "edits don't change the result").
return FileResponse(
preview_path,
media_type="video/mp4",
headers={"Cache-Control": "no-store"},
)
def _compute_onsets_sync(src_path: str) -> list[float]:
"""Blocking part of onset analysis — runs in a worker thread."""
import soundfile as sf
from services.onset_align import detect_speech_onsets
audio, sr = sf.read(src_path, dtype="float32")
return detect_speech_onsets(audio, sr)
@router.get("/dub/onsets/{job_id}")
async def dub_get_onsets(job_id: str):
"""Speech-onset times for the timeline editor's snap-to-onset ticks (#280).
Prefers the Demucs-isolated vocals track (clean speech energy); falls
back to the mixed audio. Computed once per job and cached as
``onsets.json`` in the job directory; recomputed if the source audio is
newer than the cache (e.g. re-ingest into the same job dir).
"""
import json
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
vocals = job.get("vocals_path")
mix = job.get("audio_path")
if vocals and os.path.exists(vocals):
src_path, source = vocals, "vocals"
elif mix and os.path.exists(mix):
src_path, source = mix, "mix"
else:
raise HTTPException(status_code=404, detail="No audio track available for onset analysis")
# Containment inlined (not via _safe_job_path): CodeQL can't track the
# sanitizer through a helper's return — the file's established idiom.
base = os.path.realpath(DUB_DIR)
cache_path = os.path.realpath(os.path.join(base, job_id, "onsets.json"))
if not cache_path.startswith(base + os.sep):
raise HTTPException(status_code=400, detail="Invalid job id")
try:
if (
os.path.exists(cache_path)
and os.path.getmtime(cache_path) >= os.path.getmtime(src_path)
):
with open(cache_path, "r", encoding="utf-8") as f:
cached = json.load(f)
if isinstance(cached, dict) and isinstance(cached.get("onsets"), list):
return cached
except (OSError, ValueError):
pass # unreadable/corrupt cache → recompute below
try:
onsets = await asyncio.to_thread(_compute_onsets_sync, src_path)
except Exception as e:
raise HTTPException(
status_code=500,
detail=f"Onset analysis failed: {str(e)[:200]}",
)
payload = {"onsets": onsets, "source": source}
try:
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
tmp_path = cache_path + ".tmp"
with open(tmp_path, "w", encoding="utf-8") as f:
json.dump(payload, f)
os.replace(tmp_path, cache_path)
except OSError as e:
logger.warning("onsets cache write failed for %s: %s", job_id, e)
return payload
@router.get("/dub/thumb/{job_id}")
@@ -729,6 +1148,94 @@ async def dub_preview_segment(job_id: str, segment_index: int):
return FileResponse(seg_path, media_type="audio/wav")
# ── Second-pass ASR QC (Wave 3.3 / Spec 5) ───────────────────────────────────
@router.post("/dub/qc/{job_id}")
async def dub_qc_pass(job_id: str, lang: str = Query(None), drift_threshold: float = Query(0.5)):
"""Re-recognize the dubbed audio and flag lines whose recognized text
drifts from the target text. Opt-in, never fatal: the dub is untouched
this only annotates segments with a per-line drift score and a measured
start/end, surfaced as "verify this line" markers feeding incremental
re-dub. The generated text stays authoritative (design delta from
pyvideotrans, which overwrites subtitles)."""
from services import dub_qc
from services.dub_pipeline import put_job, save_job
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
tracks = job.get("dubbed_tracks", {})
if lang and lang in tracks:
wav_path = tracks[lang]["path"]
elif tracks:
wav_path = list(tracks.values())[0]["path"]
else:
raise HTTPException(status_code=400, detail="No dubbed audio track generated yet")
if not os.path.exists(wav_path):
raise HTTPException(status_code=404, detail="Dubbed audio file not found")
segments = job.get("segments") or []
if not segments:
raise HTTPException(status_code=400, detail="Job has no segments")
def _recognize():
from services.asr_backend import get_active_asr_backend
backend = get_active_asr_backend()
result = backend.transcribe(wav_path, word_timestamps=False)
return result.get("segments", []), backend.id
try:
from services.model_manager import _get_gpu_pool
loop = asyncio.get_running_loop()
recognized, engine_id = await loop.run_in_executor(_get_gpu_pool(), _recognize)
except Exception as e:
logger.exception("dub QC ASR pass failed for %s", job_id)
raise HTTPException(status_code=500, detail=f"QC transcription failed: {e}")
seg_ids = job.get("seg_order") or [s.get("id", i) for i, s in enumerate(segments)]
scored = dub_qc.score_dub(segments, recognized, drift_threshold=drift_threshold, seg_ids=seg_ids)
# Annotate each segment (non-destructive — content text untouched).
by_id = {q.seg_id: q for q in scored}
for i, s in enumerate(segments):
sid = str(seg_ids[i]) if i < len(seg_ids) else str(s.get("id", i))
q = by_id.get(sid)
if q is None:
continue
s["qc_drift"] = q.drift
s["qc_flagged"] = q.flagged
s["qc_recognized"] = q.recognized_text
if q.new_start is not None:
s["qc_measured_start"] = q.new_start
s["qc_measured_end"] = q.new_end
put_job(job_id, job)
save_job(job_id, job)
flagged = [q for q in scored if q.flagged]
payload = json.dumps({"event": "qc_done", "engine": engine_id,
"flagged": len(flagged), "total": len(scored)})
try:
from core import job_store
job_store.append_event(job_id, f"data: {payload}\n\n")
except Exception as e:
# QC event fan-out is best-effort; the scores are already in the response.
logger.debug("QC event append failed: %s", e)
return {
"engine": engine_id,
"total": len(scored),
"flagged_count": len(flagged),
"drift_threshold": drift_threshold,
"segments": [
{"seg_id": q.seg_id, "drift": q.drift, "flagged": q.flagged,
"recognized_text": q.recognized_text,
"measured_start": q.new_start, "measured_end": q.new_end}
for q in scored
],
}
@router.get("/dub/download-audio/{job_id}")
@router.get("/dub/download-audio/{job_id}/{filename}")
async def dub_download_audio(job_id: str, lang: str = Query(None), preserve_bg: bool = Query(True), save_path: str = Query("")):
@@ -806,9 +1313,31 @@ def _pick_subtitle_text(seg: dict, dual: bool) -> str:
return f"{translated}\n<i>{original}</i>"
# Subtitles deliberately have no ?save_path= variant: they're small text
# bodies, so the Tauri side fetches them raw and writes the file itself via
# the save_text_file command — the OS save dialog is the write authorization
# (#309). The frontend's JSON-envelope save flow stays for binary exports.
def _fitted_cue_times(job: dict, lang: str | None) -> list | None:
"""Per-segment (start, end) on the fitted timeline when this job used
stretch_video; None to use the original segment times. (Wave 3.1.)"""
tracks = job.get("dubbed_tracks", {})
lc = lang if (lang and lang in tracks) else (next(iter(tracks), None))
entry = _video_stretch_plan_for(job, lc) if lc else None
if not entry:
return None
from services.fitted_subtitles import fitted_cues
return fitted_cues(job.get("segments", []), entry["plan"])
@router.get("/dub/srt/{job_id}")
@router.get("/dub/srt/{job_id}/{filename}")
async def dub_export_srt(job_id: str, dual: bool = False):
async def dub_export_srt(
job_id: str,
dual: bool = False,
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -817,22 +1346,31 @@ async def dub_export_srt(job_id: str, dual: bool = False):
if not segments:
raise HTTPException(status_code=400, detail="No transcript segments available")
# Subtitles must follow the audio the viewer hears, per timing strategy:
# Smart Fit (phase B) overlays the fitted segment times directly;
# stretch_video (Wave 3.1) regenerates cue times from the stretch plan.
# Neither applies → original times.
fitted = _fitted_segments_for(job, lang)
if fitted:
segments = _apply_fitted_times(segments, fitted)
cues = None if fitted else _fitted_cue_times(job, lang)
srt_lines = []
for i, seg in enumerate(segments):
start_ts = _format_srt_time(seg["start"])
end_ts = _format_srt_time(seg["end"])
s, e = cues[i] if cues else (seg["start"], seg["end"])
srt_lines.append(f"{i + 1}")
srt_lines.append(f"{start_ts} --> {end_ts}")
srt_lines.append(f"{_format_srt_time(s)} --> {_format_srt_time(e)}")
srt_lines.append(_pick_subtitle_text(seg, dual))
srt_lines.append("")
srt_content = "\n".join(srt_lines)
base_name = os.path.splitext(job.get('filename', 'video'))[0]
suffix = "_dual" if dual else ""
dl_name = f"subtitles_{base_name}{suffix}.srt"
return Response(
content=srt_content,
media_type="text/plain",
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.srt"'},
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
)
def _format_vtt_time(seconds):
@@ -844,7 +1382,11 @@ def _format_vtt_time(seconds):
@router.get("/dub/vtt/{job_id}")
@router.get("/dub/vtt/{job_id}/{filename}")
async def dub_export_vtt(job_id: str, dual: bool = False):
async def dub_export_vtt(
job_id: str,
dual: bool = False,
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -853,22 +1395,28 @@ async def dub_export_vtt(job_id: str, dual: bool = False):
if not segments:
raise HTTPException(status_code=400, detail="No transcript segments available")
# Same strategy-aware cue timing as /dub/srt (see comment there).
fitted = _fitted_segments_for(job, lang)
if fitted:
segments = _apply_fitted_times(segments, fitted)
cues = None if fitted else _fitted_cue_times(job, lang)
vtt_lines = ["WEBVTT", ""]
for i, seg in enumerate(segments):
start_ts = _format_vtt_time(seg["start"])
end_ts = _format_vtt_time(seg["end"])
s, e = cues[i] if cues else (seg["start"], seg["end"])
vtt_lines.append(str(i + 1))
vtt_lines.append(f"{start_ts} --> {end_ts}")
vtt_lines.append(f"{_format_vtt_time(s)} --> {_format_vtt_time(e)}")
vtt_lines.append(_pick_subtitle_text(seg, dual))
vtt_lines.append("")
vtt_content = "\n".join(vtt_lines)
base_name = os.path.splitext(job.get('filename', 'video'))[0]
suffix = "_dual" if dual else ""
dl_name = f"subtitles_{base_name}{suffix}.vtt"
return Response(
content=vtt_content,
media_type="text/vtt",
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.vtt"'},
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
)
+239 -101
View File
@@ -3,7 +3,6 @@ import json
import logging
import time
import asyncio
import numpy as np
import torch
import torchaudio
from fastapi import APIRouter, HTTPException
@@ -15,9 +14,18 @@ from schemas.requests import DubRequest
from services.model_manager import get_model, _gpu_pool
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
from services.audio_io import atomic_save_wav, _safe_torchaudio_save
from services.ffmpeg_utils import find_ffmpeg, spawn_subprocess
from services.ffmpeg_utils import (
find_ffmpeg,
spawn_subprocess,
# Moved to ffmpeg_utils so the Smart Fit export pipeline (Phase B) can
# reuse them; re-imported here so `dub_generate._atempo_chain` /
# `_pitch_preserving_stretch` keep working for existing importers.
_atempo_chain,
_pitch_preserving_stretch,
)
from services.rvc import apply_rvc, is_enabled as rvc_is_enabled
from services.incremental import segment_fingerprint
from services.incremental import segment_fingerprint, fit_fingerprint
from services.fit_planner import FitParams, plan_fit
from services.watermark import embed_watermark
from api.routers.dub_core import _get_job, _save_job
@@ -39,75 +47,44 @@ GAP_OVERFLOW_MAX_S = 0.25
GAP_OVERFLOW_BUFFER_S = 0.05
def _atempo_chain(ratio: float) -> str:
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
def _sync_job_segments(job: dict, req: DubRequest) -> None:
"""Persist the segments this dub was actually generated from back onto the job.
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
multiple stages multiplies the effective ratio while keeping each
individual stage inside the well-behaved range. Pitch is preserved
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
slows down.
The editor only sends the (translated / user-edited) segment text in the
generate request; the job itself kept the original-language ASR transcript.
SRT/VTT export and ffmpeg subtitle burn-in read `job["segments"]`, so they
rendered the source language instead of the dub the user just heard (#309).
Merge strategy: rebuild `job["segments"]` from the request, carrying over
per-segment metadata (speaker_id, id, ) from the existing job segment
matched by stable id (fallback: index). `text_original` always keeps the
source-language text so dual-subtitle layouts can still stack it under the
translation.
"""
stages: list[str] = []
remaining = ratio
while remaining > 2.0:
stages.append("atempo=2.0")
remaining /= 2.0
while remaining < 0.5:
stages.append("atempo=0.5")
remaining /= 0.5
stages.append(f"atempo={remaining:.6f}")
return ",".join(stages)
async def _pitch_preserving_stretch(
wav: torch.Tensor, target_samples: int, sr: int,
) -> torch.Tensor:
"""Time-stretch a (1, samples) tensor to `target_samples` while
preserving pitch, by piping the audio through `ffmpeg atempo`.
Async so it never blocks the event loop: it's awaited from the `_stream`
generator, and each ffmpeg call is ~50-100 ms a synchronous
``subprocess.run`` here froze health-checks / SSE / every concurrent
request for the whole multi-segment job.
Returns a (1, target_samples) tensor on the same device as input.
Raises RuntimeError when ffmpeg fails callers should fall back to
naive linear interpolation, accepting the pitch shift, to ensure the
output isn't silent.
"""
wl = int(wav.shape[-1])
if target_samples <= 0 or wl == target_samples:
return wav
ratio = wl / target_samples
filter_str = _atempo_chain(ratio)
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per segment,
# run off the event loop so concurrent requests stay responsive.
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
proc = await spawn_subprocess(
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
"-af", filter_str,
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await proc.communicate(input=arr.tobytes())
if proc.returncode != 0 or not stdout:
raise RuntimeError(
(stderr.decode(errors="replace") or "atempo failed")[:200]
)
out_arr = np.frombuffer(stdout, dtype=np.float32)
# atempo rarely lands exactly on the integer sample count, so
# pad/trim to the requested slot length.
if len(out_arr) < target_samples:
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
out_arr = np.concatenate([out_arr, pad])
elif len(out_arr) > target_samples:
out_arr = out_arr[:target_samples]
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
if not req.segments:
return
existing = [s for s in (job.get("segments") or []) if isinstance(s, dict)]
by_id = {str(s["id"]): s for s in existing if s.get("id") is not None}
seg_ids = req.segment_ids or []
merged: list[dict] = []
for i, seg in enumerate(req.segments):
seg_id = seg_ids[i] if i < len(seg_ids) else None
prev = by_id.get(str(seg_id)) if seg_id is not None else None
if prev is None and i < len(existing):
prev = existing[i]
row = dict(prev) if prev else {}
if seg_id is not None:
# The request id is authoritative — seg_order and the per-segment
# WAV manifest are keyed by it.
row["id"] = seg_id
# Source-language text survives the overwrite so dual-subtitle export
# keeps working; never let the translation clobber it.
row["text_original"] = row.get("text_original") or row.get("text") or ""
row["start"] = seg.start
row["end"] = seg.end
row["text"] = seg.text
merged.append(row)
job["segments"] = merged
router = APIRouter()
@@ -134,6 +111,15 @@ async def dub_generate(job_id: str, req: DubRequest):
# `seg_i.wav` on disk and slot into the final mix unchanged.
regen_only = set(req.regen_only or []) if req.regen_only is not None else None
seg_ids = req.segment_ids or []
strategy = (req.timing_strategy or "concise").lower()
# Strategy-transition guard: smart_fit re-mixes the *natural-rate*
# per-segment WAVs from disk. If the previous run used strict_slot,
# the on-disk WAVs are slot-squeezed ("slotted") — reusing them would
# double-compress. Force one full regen; afterwards seg_wav_kind is
# "natural" and partial regen / fit-only re-mix (regen_only=[]) work.
# Jobs predating this field have unknown kind → also regen once.
if strategy == "smart_fit" and regen_only is not None and job.get("seg_wav_kind") != "natural":
regen_only = None
# Manifest: stable segment id per current index. Per-segment WAVs are
# named by stable id (dub_seg_path) so regen reuses the right audio after
# reorder; index-keyed readers (preview/export) resolve via this manifest.
@@ -184,13 +170,17 @@ async def dub_generate(job_id: str, req: DubRequest):
if cached_sr != _model.sampling_rate:
import torchaudio.functional as AF
cached_wav = AF.resample(cached_wav, cached_sr, _model.sampling_rate)
# Pad/trim to slot.
target_samples = int(seg_duration * _model.sampling_rate)
current_samples = cached_wav.shape[-1]
if target_samples > current_samples:
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
elif current_samples > target_samples:
cached_wav = cached_wav[..., :target_samples]
# Pad/trim to slot — except smart_fit, whose mix
# loop needs the natural-rate length to compute the
# audio/video split (the seg_wav_kind guard above
# guarantees these cached WAVs are natural-rate).
if strategy != "smart_fit":
target_samples = int(seg_duration * _model.sampling_rate)
current_samples = cached_wav.shape[-1]
if target_samples > current_samples:
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
elif current_samples > target_samples:
cached_wav = cached_wav[..., :target_samples]
all_segment_wavs.append((seg.start, seg.end, cached_wav, _model.sampling_rate))
sync_scores.append(getattr(seg, 'sync_ratio', None) or 1.0)
_t_cache += time.perf_counter() - _t_cache_0
@@ -214,18 +204,42 @@ async def dub_generate(job_id: str, req: DubRequest):
# (see services/speaker_clone.py) live at job["speaker_clones"]
# keyed by speaker_id. We use the `auto:` prefix so they can't
# collide with persistent voice_profiles.id values.
if profile_id and profile_id.startswith("auto:"):
key = profile_id[len("auto:"):]
clones = job.get("speaker_clones") or {}
# Match by the safe-name key first, fall back to speaker_id.
auto = None
for spk, info in clones.items():
if spk.lower().replace(" ", "_") == key or spk == key:
auto = info
break
if auto:
ref_audio = auto.get("ref_audio")
ref_text = auto.get("ref_text")
# Wave 3.2: a per-segment clone ref (cut from this line's own
# source audio) takes precedence over the per-speaker clone.
if profile_id and profile_id.startswith("auto-seg:"):
sid = profile_id[len("auto-seg:"):]
info = (job.get("segment_clones") or {}).get(sid)
if info:
ref_audio = info.get("ref_audio")
ref_text = info.get("ref_text")
profile_id = None # prevent the voice_profiles lookup below
elif profile_id and profile_id.startswith("auto:"):
# #486: an `auto:{speaker}` binding still prefers THIS
# segment's own per-segment ref when one exists (cut from
# this line's source audio → matches its prosody), falling
# back to the per-speaker clone otherwise. This keeps the
# Wave 3.2 per-segment-ref quality win while letting every
# segment carry the UI-visible `auto:` id the dub editor's
# Voice dropdown can actually render ("From Video →
# Speaker N"). `seg_id` is closed over from the per-segment
# loop below.
seg_ref = (job.get("segment_clones") or {}).get(str(seg_id))
if seg_ref:
ref_audio = seg_ref.get("ref_audio")
ref_text = seg_ref.get("ref_text")
else:
key = profile_id[len("auto:"):]
clones = job.get("speaker_clones") or {}
# Match by the safe-name key first, fall back to speaker_id.
auto = None
for spk, info in clones.items():
if spk.lower().replace(" ", "_") == key or spk == key:
auto = info
break
if auto:
ref_audio = auto.get("ref_audio")
ref_text = auto.get("ref_text")
profile_id = None # prevent the voice_profiles lookup below
if profile_id:
@@ -265,6 +279,9 @@ async def dub_generate(job_id: str, req: DubRequest):
if seg_effect_preset == "raw":
return audio_out
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
effect_chain = get_effect_chain(seg_effect_preset)
if effect_chain:
@@ -310,6 +327,9 @@ async def dub_generate(job_id: str, req: DubRequest):
if seg_effect_preset == "raw":
return audio_out
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
effect_chain = get_effect_chain(seg_effect_preset)
if effect_chain:
@@ -373,13 +393,13 @@ async def dub_generate(job_id: str, req: DubRequest):
_t_tts_0 = time.perf_counter()
seg_effect_preset = getattr(seg, "effect_preset", None) or "broadcast"
# In concise / stretch_video modes we pass dur_s=None so the
# TTS model speaks at its natural rate for this text length —
# the whole point of the new timing strategies is to never
# squeeze the speech to fit. strict_slot keeps the legacy
# behaviour where dur_s is the slot hint.
_strategy = (req.timing_strategy or "concise").lower()
_dur_for_tts = seg_duration if _strategy == "strict_slot" else None
# In concise / stretch_video / smart_fit modes we pass
# dur_s=None so the TTS model speaks at its natural rate for
# this text length — the whole point of the new timing
# strategies is to never squeeze the speech to fit at
# synthesis time. strict_slot keeps the legacy behaviour
# where dur_s is the slot hint.
_dur_for_tts = seg_duration if strategy == "strict_slot" else None
audio_tensor = await loop.run_in_executor(
_gpu_pool, _gen,
@@ -396,7 +416,7 @@ async def dub_generate(job_id: str, req: DubRequest):
target_samples = int(seg_duration * _model.sampling_rate)
current_samples = audio_tensor.shape[-1]
if _strategy == "strict_slot":
if strategy == "strict_slot":
# Legacy: pad short audio + trim long audio so the mix
# loop receives slot-sized buffers. The atempo squeeze
# in the mix loop never fires here because we already
@@ -406,9 +426,10 @@ async def dub_generate(job_id: str, req: DubRequest):
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, pad_amount))
elif current_samples > target_samples:
audio_tensor = audio_tensor[..., :target_samples]
# concise / stretch_video: keep audio at its natural length.
# The mix loop decides per-mode whether to trim, slip, or
# stretch the video to accommodate it.
# concise / stretch_video / smart_fit: keep audio at its
# natural length. The mix loop decides per-mode whether to
# trim, slip, stretch the video, or split audio/video
# retiming (smart_fit) to accommodate it.
generated_dur = audio_tensor.shape[-1] / _model.sampling_rate
sync_ratio = round(generated_dur / max(seg_duration, 0.01), 3)
@@ -487,7 +508,6 @@ async def dub_generate(job_id: str, req: DubRequest):
_t_diskw = time.perf_counter() - _t_diskw_0
sr = _model.sampling_rate
strategy = (req.timing_strategy or "concise").lower()
slot_fit = (req.slot_fit or "time_stretch").lower()
overflow_budget_s = max(0.0, float(req.overflow_budget_s or 0.0))
@@ -502,6 +522,7 @@ async def dub_generate(job_id: str, req: DubRequest):
# the matching per-segment setpts filter chain on the source video.
new_layout: list[tuple[float, float]] = []
video_stretch_plan: list[dict] = []
fit_plan = None # smart_fit only — services.fit_planner.FitPlan
orig_total_dur = float(job.get("duration") or 0.0)
if strategy == "stretch_video":
@@ -535,9 +556,43 @@ async def dub_generate(job_id: str, req: DubRequest):
cursor += max(0.0, orig_total_dur - last_orig_end)
new_total_dur = max(cursor, orig_total_dur)
total_samples = int(new_total_dur * sr)
elif strategy == "smart_fit":
# Smart Fit: plan the audio-rate / video-ratio split per segment
# from the natural-rate WAV lengths. Pure planning — the mix
# loop below applies the audio side; the video side ships as
# fit_plans[lang] for the (Phase B) export pipeline.
_fo = req.fit_options
_fit_defaults = FitParams()
fit_params = FitParams(
max_audio_only_rate=float(getattr(_fo, "max_audio_only_rate", None) or _fit_defaults.max_audio_only_rate),
audio_rate_cap=float(getattr(_fo, "audio_rate_cap", None) or _fit_defaults.audio_rate_cap),
video_slow_cap=float(getattr(_fo, "video_slow_cap", None) or _fit_defaults.video_slow_cap),
gap_guard_s=float(_fo.gap_guard_s) if _fo is not None and _fo.gap_guard_s is not None else _fit_defaults.gap_guard_s,
allow_video_retime=bool(_fo.allow_video_retime) if _fo is not None and _fo.allow_video_retime is not None else _fit_defaults.allow_video_retime,
)
_seg_order = job.get("seg_order") or []
fit_plan = plan_fit(
[
{
"id": _seg_order[i] if i < len(_seg_order) else f"seg_{i}",
"start": s,
"end": e,
}
for i, (s, e, _w, _) in enumerate(all_segment_wavs)
],
[w.shape[-1] / sr for (_s, _e, w, _) in all_segment_wavs],
orig_total_dur,
fit_params,
)
total_samples = int(fit_plan.total_duration * sr)
else:
total_samples = int(orig_total_dur * sr)
# smart_fit: cue times for the fitted timeline, computed from the
# ACTUAL stretched/trimmed sample positions in the mix loop below —
# not from the plan — so subtitles land exactly on the audio.
fitted_cues: list[dict] = []
full_audio = torch.zeros(1, total_samples)
for i, (start, end, wav, _) in enumerate(all_segment_wavs):
@@ -560,6 +615,55 @@ async def dub_generate(job_id: str, req: DubRequest):
"stretch_ratio": round(natural_dur / max(orig_dur, 1e-3), 3),
})
elif strategy == "smart_fit":
# Smart Fit: apply the planner's audio_rate via the same
# pitch-preserving atempo pipe strict_slot uses, place the
# result at the planned new_start, and hard-trim whatever
# the caps couldn't absorb. The video side (video_ratio per
# chunk) is persisted below for the export pipeline.
sf = fit_plan.segments[i]
place_at = sf.new_start
if sf.audio_rate > 1.0 + 1e-6 and wl > 0:
target = max(1, int(round(wl / sf.audio_rate)))
try:
adjusted = await _pitch_preserving_stretch(
adjusted, target, sr,
)
except Exception as e:
logger.warning(
"atempo stretch failed for seg %d (%.2f×), "
"falling back to linear interp: %s",
i, sf.audio_rate, e,
)
adjusted = torch.nn.functional.interpolate(
adjusted.unsqueeze(0),
size=target,
mode='linear',
align_corners=False,
).squeeze(0)
wl = adjusted.shape[-1]
# Residual overflow → hard-trim to the segment's new video
# slot (fade below keeps the cut pop-free).
new_slot_samples = int(max(0.0, sf.new_end - sf.new_start) * sr)
if new_slot_samples > 0 and wl > new_slot_samples:
adjusted = adjusted[..., :new_slot_samples]
wl = adjusted.shape[-1]
# Truthful per-segment verdict for the UI badge.
entry = {"status": sf.status}
if sf.audio_rate > 1.0 + 1e-6:
entry["audio_rate"] = round(sf.audio_rate, 3)
if sf.video_ratio > 1.0 + 1e-6:
entry["video_ratio"] = round(sf.video_ratio, 3)
if sf.overflow_s > 0:
entry["overflow_s"] = round(sf.overflow_s, 3)
fit_status.append(entry)
# Cue times from the ACTUAL stretched sample positions.
fitted_cues.append({
"id": sf.seg_id,
"start": round(place_at, 4),
"end": round(place_at + wl / sr, 4),
})
elif strategy == "concise":
# Mode A: never compress. Allow the audio to extend into the
# silent gap before the next seg (existing heuristic) plus
@@ -681,6 +785,9 @@ async def dub_generate(job_id: str, req: DubRequest):
job["language"] = req.language
job["language_code"] = lang_code
job["timing_strategy"] = strategy
# Keep job segments in lock-step with what was just rendered so
# subtitle export / burn-in use the translated text (#309).
_sync_job_segments(job, req)
if strategy == "stretch_video":
stretch_plans = job.setdefault("video_stretch_plans", {})
stretch_plans[lang_code] = {
@@ -688,6 +795,34 @@ async def dub_generate(job_id: str, req: DubRequest):
"total_duration": round(track_dur, 4),
"orig_duration": round(orig_total_dur, 4),
}
elif strategy == "smart_fit" and fit_plan is not None:
# video_stretch_plans stays untouched — smart_fit persists its
# own keyspace so a job can carry both without clobbering.
_fit_params_payload = {
"timing_strategy": strategy,
"max_audio_only_rate": fit_params.max_audio_only_rate,
"audio_rate_cap": fit_params.audio_rate_cap,
"video_slow_cap": fit_params.video_slow_cap,
"gap_guard_s": fit_params.gap_guard_s,
"allow_video_retime": fit_params.allow_video_retime,
}
fit_fp = fit_fingerprint(_fit_params_payload)
job.setdefault("fit_plans", {})[lang_code] = {
# Same dict shape _build_video_stretch_filter_graph consumes.
"plan": fit_plan.video_plan,
# Cue times from actual stretched sample positions — for
# subtitle export on the fitted timeline.
"fitted_segments": fitted_cues,
"total_duration": round(track_dur, 4),
"orig_duration": round(orig_total_dur, 4),
"params": _fit_params_payload,
"fit_fp": fit_fp,
}
job["dubbed_tracks"][lang_code]["fit_fp"] = fit_fp
# Record what kind of per-segment WAVs are on disk so a later
# smart_fit run knows whether partial regen / fit-only re-mix can
# reuse them ("natural") or must regen once ("slotted").
job["seg_wav_kind"] = "slotted" if strategy == "strict_slot" else "natural"
_save_job(job_id, job)
_t_total = time.perf_counter() - _t_start
@@ -784,6 +919,9 @@ async def preview_segment(job_id: str, req: SegmentPreviewRequest):
postprocess_output=True,
)
audio_out = audios[0]
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(
audio_out,
sample_rate=getattr(_model, "sampling_rate", 24000),
+124 -7
View File
@@ -2,6 +2,7 @@ import os
import time
import asyncio
import logging
from typing import Optional
from fastapi import APIRouter
from fastapi.responses import JSONResponse
@@ -45,6 +46,70 @@ LANG_NAMES = {
"id": "Indonesian", "uk": "Ukrainian",
}
# Regional dialect hints (#280 item 2). Maps a BCP-47 dialect code to the
# instruction injected into LLM translation prompts so the output uses that
# region's vocabulary and grammar (the reporter's example: choosing Argentina
# should yield "Vos sos muy listo", not the Peninsular "Tú eres muy listo").
# Only LLM-backed paths can honor these — provider="openai" and the
# quality="cinematic" refine pass. Keep entries short: they ride on every
# per-segment prompt, so verbosity = wall time.
DIALECT_HINTS = {
# Spanish
"es-ES": "European Spanish (Spain): use tú/vosotros forms and Peninsular vocabulary.",
"es-MX": "Mexican Spanish: use tú/ustedes forms and Mexican vocabulary.",
"es-AR": "Rioplatense Spanish (Argentina): use voseo — 'vos' with its verb forms (e.g. 'vos sos', 'tenés') and 'ustedes'; prefer Argentinian vocabulary.",
"es-CO": "Colombian Spanish: use tú/usted as natural in Colombia and Colombian vocabulary.",
"es-CL": "Chilean Spanish: use Chilean vocabulary and expressions.",
# Portuguese
"pt-BR": "Brazilian Portuguese: use 'você' forms, Brazilian vocabulary and spelling.",
"pt-PT": "European Portuguese: use European vocabulary, spelling, and 'tu' where natural.",
# English
"en-US": "American English: use US spelling and vocabulary.",
"en-GB": "British English: use UK spelling and vocabulary.",
"en-AU": "Australian English: use Australian spelling and vocabulary.",
"en-IN": "Indian English: use Indian English vocabulary and conventions.",
# French
"fr-FR": "Metropolitan French (France): use standard French vocabulary.",
"fr-CA": "Canadian French (Québec): use Québécois vocabulary and expressions.",
"fr-BE": "Belgian French: use Belgian vocabulary (e.g. septante, nonante).",
# German
"de-DE": "Standard German (Germany): use Federal German vocabulary.",
"de-AT": "Austrian German: use Austrian vocabulary (e.g. Jänner, Erdapfel).",
"de-CH": "Swiss Standard German: use Swiss vocabulary and 'ss' instead of 'ß'.",
# Arabic
"ar-EG": "Egyptian Arabic: use Egyptian colloquial vocabulary where natural for dubbing.",
"ar-SA": "Gulf/Saudi Arabic flavor: prefer vocabulary natural to the Gulf region.",
"ar-MA": "Moroccan Arabic (Darija) flavor: prefer vocabulary natural to Morocco.",
# Dutch
"nl-NL": "Netherlands Dutch: use vocabulary standard in the Netherlands.",
"nl-BE": "Belgian Dutch (Flemish): use Flemish vocabulary and expressions.",
}
def dialect_clause(dialect: Optional[str]) -> str:
"""Prompt fragment for a requested dialect, or '' when unset/unknown.
Unknown-but-plausible codes (e.g. "es-PE") still get a generic regional
clause so users aren't limited to the curated list.
"""
if not dialect or not str(dialect).strip():
return ""
code = str(dialect).strip()
hint = DIALECT_HINTS.get(code)
if hint:
return f" Target dialect — {hint}"
# Generic fallback for any lang-REGION shaped code we don't curate.
if "-" in code:
lang, _, region = code.partition("-")
lang_name = LANG_NAMES.get(lang, lang)
if region:
return (
f" Use the vocabulary, grammar, and expressions of {lang_name} "
f"as spoken in the region '{region}'."
)
return ""
# Per-language script enforcement. Maps language code → required Unicode
# block(s) the translation must contain. Used as a sanity gate after the
# LLM responds: if the output contains <50% characters from the expected
@@ -92,15 +157,49 @@ _nllb_tokenizer = None
_nllb_device = None
def _dialect_flags(req, applied: bool) -> dict:
"""Response fields describing whether the requested dialect was honored.
Empty dict when no dialect was requested, so existing response shapes
stay byte-identical for callers that never send one.
"""
if not getattr(req, "dialect", None):
return {}
return {"dialect": req.dialect, "dialect_applied": bool(applied)}
def _guess_lang_from_text(segments) -> str | None:
"""Best-effort source language from segment text, by script.
Used only as a last resort when neither the request nor the job carries a
detected language. Without this, the bare "en" fallback below forces
en -> en on non-English audio (e.g. Korean), which has no Argos package and
fails every segment even though ASR detected the language correctly.
"""
text = " ".join((getattr(s, "text", "") or "") for s in (segments or [])[:8])
has = lambda lo, hi: any(lo <= ord(c) <= hi for c in text)
if has(0x3040, 0x30FF):
return "ja" # Hiragana/Katakana — check before CJK (Japanese uses Kanji too)
if has(0xAC00, 0xD7A3) or has(0x1100, 0x11FF):
return "ko" # Hangul
if has(0x4E00, 0x9FFF):
return "zh" # CJK ideographs
if has(0x0400, 0x04FF):
return "ru" # Cyrillic
if has(0x0600, 0x06FF):
return "ar" # Arabic
return None
def _resolve_source_lang(req: TranslateRequest) -> str:
"""Pick source language: explicit request > job.source_lang > 'en' fallback."""
"""Pick source language: explicit request > job.source_lang > text guess > 'en'."""
if getattr(req, "source_lang", None):
return req.source_lang
if getattr(req, "job_id", None):
job = _get_job(req.job_id)
if job and job.get("source_lang"):
return job["source_lang"]
return "en"
return _guess_lang_from_text(getattr(req, "segments", None)) or "en"
def _unload_nllb():
@@ -203,7 +302,8 @@ async def dub_translate(req: TranslateRequest):
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
_unload_nllb()
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
**_dialect_flags(req, applied=False)}
# OpenAI / Ollama Local LLM Translation
if provider == "openai":
@@ -236,10 +336,16 @@ async def dub_translate(req: TranslateRequest):
f"only — do not use Latin/Roman letters, do not "
f"transliterate, do not output any other language."
)
# #280 item 2 — regional dialect/vocabulary. Only applied when
# the dialect belongs to the target language (a leftover
# "es-AR" must not contaminate a French translation).
dia_clause = ""
if req.dialect and str(req.dialect).lower().startswith(str(tgt_code).lower()[:2]):
dia_clause = dialect_clause(req.dialect)
return (
f"You are a professional dubbing translator. "
f"Translate the user's text from {src_name} into "
f"{tgt_name}.{script_clause} "
f"{tgt_name}.{script_clause}{dia_clause} "
f"Reply ONLY with the translated {tgt_name} text, do not "
f"add quotes, notes, headers, explanations, or commentary."
)
@@ -299,7 +405,8 @@ async def dub_translate(req: TranslateRequest):
tasks = [loop.run_in_executor(_cpu_pool, _translate_llm, seg) for seg in req.segments]
translated = await asyncio.gather(*tasks)
translated.sort(key=lambda x: str(x["id"]))
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
**_dialect_flags(req, applied=True)}
# Offline Argos Translate
if provider == "argos" or provider == "libretranslate":
@@ -353,7 +460,8 @@ async def dub_translate(req: TranslateRequest):
return results
translated = await loop.run_in_executor(_cpu_pool, _translate_argos)
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
**_dialect_flags(req, applied=False)}
# Legacy / API Deep_Translator logic.
# Preflight the optional `deep_translator` dep once so we fail with a
@@ -462,7 +570,8 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
except Exception as e:
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang, "quality_used": "fast"}
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
"quality_used": "fast", **_dialect_flags(req, applied=False)}
if quality != "cinematic":
return base
@@ -492,12 +601,19 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
if not pairs:
return base
# #280 item 2: thread the regional-dialect hint into the reflect/adapt
# prompts. Guard against a stale dialect from another language.
dialect_hint = ""
if req.dialect and str(req.dialect).lower().startswith(str(req.target_lang).lower()[:2]):
dialect_hint = dialect_clause(req.dialect)
refined = await cinematic_refine_many(
pairs,
source_lang=src_lang,
target_lang=req.target_lang,
glossary=req.glossary,
directions=directions,
dialect_hint=dialect_hint,
executor=_cpu_pool,
)
refined_by_id = {r["id"]: r for r in refined}
@@ -560,4 +676,5 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
"target_lang": req.target_lang,
"source_lang": src_lang,
"quality_used": "cinematic",
**_dialect_flags(req, applied=bool(dialect_hint)),
}
+35 -6
View File
@@ -266,11 +266,26 @@ class SelectEngineRequest(BaseModel):
backend_id: str
@router.post("/engines/select")
class SelectEngineResponse(BaseModel):
family: str
active: str
env_override: bool
# Routing verdict for the selected engine on THIS host (#21). Always present
# so the UI can show a confirm/warning toast on a cpu_fallback pick without
# branching on key presence; defaults match a legacy/degraded row.
routing_status: str = "cpu_only"
effective_device: str = "cpu"
routing_reason: str | None = None
@router.post("/engines/select", response_model=SelectEngineResponse)
def select_engine(req: SelectEngineRequest):
"""Persist a family's engine pick to prefs.json. Refuses unknown backends
+ refuses backends whose deps aren't installed (so the UI can't silently
brick a pipeline by picking an unavailable engine)."""
"""Persist a family's engine pick to prefs.json. Refuses unknown backends,
backends whose deps aren't installed, AND backends that cannot run on THIS
host's hardware (routing_status == "unavailable") — so the UI can't silently
brick a pipeline by picking an engine that needs a GPU this machine lacks.
A `cpu_fallback` pick is allowed (it runs, just slower) only a hard
`unavailable` is blocked. LLM is never routing-gated (its status is "n/a")."""
family = _FAMILIES.get(req.family)
if not family:
raise HTTPException(400, f"Unknown family: {req.family}. Expected one of tts/asr/llm.")
@@ -278,12 +293,26 @@ def select_engine(req: SelectEngineRequest):
available = {b["id"]: b for b in module.list_backends()}
if req.backend_id not in available:
raise HTTPException(400, f"Unknown {req.family} backend: {req.backend_id!r}")
if not available[req.backend_id]["available"]:
reason = available[req.backend_id].get("reason") or "unavailable"
entry = available[req.backend_id]
if not entry["available"]:
reason = entry.get("reason") or "unavailable"
raise HTTPException(400, f"Backend {req.backend_id} not ready: {reason}")
# Host-routing gate (no silent CPU fallback). `.get` is defensive so an
# older/legacy payload without routing keys still selects cleanly.
if entry.get("routing_status") == "unavailable":
why = entry.get("routing_reason") or "requires a GPU this host doesn't have"
raise HTTPException(
400,
f"Backend {req.backend_id} can't run on this machine: {why}. "
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
)
prefs.set_(pref_key, req.backend_id)
return {
"family": req.family,
"active": module.active_backend_id(),
"env_override": bool(__import__("os").environ.get(f"OMNIVOICE_{req.family.upper()}_BACKEND")),
# Echo the routing verdict so the UI can warn on a cpu_fallback pick.
"routing_status": entry.get("routing_status", "cpu_only"),
"effective_device": entry.get("effective_device", "cpu"),
"routing_reason": entry.get("routing_reason"),
}
+344 -64
View File
@@ -59,13 +59,87 @@ def _render_with_pauses(gen_span, segments, sample_rate):
parts.append(torch.zeros(*shape, dtype=ref.dtype, device=ref.device))
return torch.cat(parts, dim=-1)
def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering=False):
"""Shared post-DSP for /generate: preset validation → mastering →
effect chain loudness normalization.
``skip_mastering`` honors a backend's ``applies_own_mastering`` flag
(issue #312): studio engines (e.g. VoxCPM2's native 48 kHz output)
opt out of the broadcast Compressor + Reverb chain that's tuned for
OmniVoice's 24 kHz clone output. Loudness normalization still runs —
it's a benign peak scale. Mirrors ``_run_tts`` in openai_compat.py.
"""
from services.audio_dsp import (
EFFECT_PRESETS, apply_mastering, normalize_audio,
apply_effects_chain, get_effect_chain,
)
preset = effect_preset or "broadcast"
if preset not in EFFECT_PRESETS:
raise ValueError(
f"Unknown effect preset: {preset!r}. "
f"Valid: {list(EFFECT_PRESETS.keys())}"
)
if preset == "raw":
# Raw: skip all DSP — return raw model output
return audio_out
if not skip_mastering:
audio_out = apply_mastering(audio_out, sample_rate=sample_rate)
chain = get_effect_chain(preset)
if chain:
audio_out = apply_effects_chain(
audio_out, sample_rate=sample_rate, chain=chain,
)
return normalize_audio(audio_out, target_dBFS=-2.0)
def _oom_friendly_reraise(e):
"""Best-effort cache flush + the user-facing OOM hint shared by both
inference paths."""
import gc
import torch
gc.collect()
if torch.backends.mps.is_available():
torch.mps.empty_cache()
elif torch.cuda.is_available():
torch.cuda.empty_cache()
# #278: don't mislabel a torch.compile/Triton/Inductor crash as an
# out-of-memory condition. (model_manager's generate wrapper already
# retries these eagerly; this only triggers if that retry also died.)
from services.model_manager import _is_compile_runtime_failure
if _is_compile_runtime_failure(e):
raise RuntimeError(
f"TTS engine hit a torch.compile/Triton error (not out of memory). "
f"Disable torch.compile in Settings → Performance, use the Flush "
f"button to reload the model, then regenerate. Underlying error: {e}"
) from e
# #437: a Permission-denied / exec failure (e.g. a bundled engine binary
# that lost its +x bit) is NOT an OOM — don't send the user to the Flush
# button; tell them what's actually wrong.
es = str(e)
if isinstance(e, PermissionError) or "Permission denied" in es or "Errno 13" in es:
raise RuntimeError(
f"A required engine binary couldn't be executed (permission denied). "
f"This usually means a bundled binary lost its execute bit — reinstall, "
f"or run `chmod +x` on the engine binary named in the error. "
f"Underlying error: {e}"
) from e
raise RuntimeError(
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
)
def _run_inference(
model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset="broadcast",
max_chunk_chars=None, crossfade_ms=None,
):
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
import torch
try:
if used_seed is not None:
@@ -99,53 +173,124 @@ def _run_inference(
)[0]
audio_out = _render_with_pauses(_gen_span, segments, sr)
else:
audios = model.generate(
text=text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=duration,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
# Wave 1.2: long text is split at sentence boundaries and the
# per-chunk audio crossfaded — removes the length ceiling. Short
# text takes the single-shot path below unchanged. [pause] inputs
# keep the dedicated stitcher above (spans are already short).
from services.chunked_tts import (
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
concatenate_audio_chunks, split_text_into_chunks,
)
audio_out = audios[0]
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
text_chunks = split_text_into_chunks(text, _max_chars)
if len(text_chunks) > 1:
parts = []
for i, chunk_text in enumerate(text_chunks):
# Vary the seed per chunk (deterministically) to avoid
# correlated RNG artifacts across chunk boundaries.
if used_seed is not None:
torch.manual_seed(used_seed + i)
parts.append(model.generate(
text=chunk_text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=None,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
)[0])
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
else:
audios = model.generate(
text=text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=duration,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
)
audio_out = audios[0]
# Apply DSP effect preset
_effect_preset = effect_preset or "broadcast"
# Apply DSP effect preset. The OmniVoice model never masters its own
# output, so mastering always runs here (unchanged behavior).
return _apply_effect_chain(audio_out, sr, effect_preset)
# Validate preset ID
from services.audio_dsp import EFFECT_PRESETS
if _effect_preset not in EFFECT_PRESETS:
raise ValueError(
f"Unknown effect preset: {_effect_preset!r}. "
f"Valid: {list(EFFECT_PRESETS.keys())}"
)
if _effect_preset == "raw":
# Raw: skip all DSP — return raw model output
return audio_out
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
_chain = get_effect_chain(_effect_preset)
if _chain:
mastered_audio = apply_effects_chain(
mastered_audio, sample_rate=sr, chain=_chain,
)
return normalize_audio(mastered_audio, target_dBFS=-2.0)
except ValueError as e:
# Don't wrap validation errors in OOM message
raise e
except Exception as e:
import gc
gc.collect()
if torch.backends.mps.is_available():
torch.mps.empty_cache()
elif torch.cuda.is_available():
torch.cuda.empty_cache()
raise RuntimeError(
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
_oom_friendly_reraise(e)
def _run_backend_inference(
backend, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, denoise, postprocess_output,
used_seed, effect_preset="broadcast",
max_chunk_chars=None, crossfade_ms=None,
):
"""Engine-aware twin of :func:`_run_inference` (issue #312).
Runs the request through a pluggable ``TTSBackend`` adapter instead of the
OmniVoice model directly. The adapter protocol is narrower than the
OmniVoice-native surface engine-specific extras (``t_shift``,
``layer_penalty_factor``, ) only exist on the native path, which is why
OmniVoice itself still goes through ``_run_inference``.
"""
import torch
try:
if used_seed is not None:
torch.manual_seed(used_seed)
if language and language.lower() == "auto":
language = None
gen_kwargs = dict(
language=language, ref_audio=ref_audio_path, ref_text=ref_text,
instruct=instruct, num_step=num_step, guidance_scale=guidance_scale,
speed=speed, denoise=denoise, postprocess_output=postprocess_output,
)
sr = backend.sample_rate
# Inline [pause Nms] markers (issue #276) work for every engine — the
# silence stitching is model-free.
from omnivoice.utils.text import parse_pause_markers
segments = parse_pause_markers(text)
has_pause = len(segments) > 1 or (segments and segments[0][1] > 0)
if has_pause:
def _gen_span(span_text):
# Per-span duration is left to the engine; an explicit overall
# `duration` can't be meaningfully split across spans.
return backend.generate(span_text, duration=None, **gen_kwargs)
audio_out = _render_with_pauses(_gen_span, segments, sr)
else:
# Wave 1.2: sentence-boundary chunking for long text (see
# _run_inference for the rationale; behavior is identical here).
from services.chunked_tts import (
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
concatenate_audio_chunks, split_text_into_chunks,
)
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
text_chunks = split_text_into_chunks(text, _max_chars)
if len(text_chunks) > 1:
parts = []
for i, chunk_text in enumerate(text_chunks):
if used_seed is not None:
torch.manual_seed(used_seed + i)
parts.append(backend.generate(chunk_text, duration=None, **gen_kwargs))
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
else:
audio_out = backend.generate(text, duration=duration, **gen_kwargs)
return _apply_effect_chain(
audio_out, sr, effect_preset,
skip_mastering=getattr(backend, "applies_own_mastering", False),
)
except ValueError as e:
# Don't wrap validation errors in OOM message
raise e
except Exception as e:
_oom_friendly_reraise(e)
@router.post("/generate")
@@ -168,19 +313,88 @@ async def generate_speech(
profile_id: Optional[str] = Form(None),
seed: Optional[int] = Form(None),
effect_preset: str = Form("broadcast"),
engine: Optional[str] = Form(None),
# Wave 1.2 — unlimited-length generation: long text is split at sentence
# boundaries and crossfaded. 0 disables chunking (whole text to engine).
max_chunk_chars: int = Form(800, ge=0),
crossfade_ms: int = Form(50, ge=0, le=1000),
):
_model = await get_model()
# ── Engine resolution (issue #312) ──────────────────────────────────────
# The request runs on the engine selected in Settings (POST /engines/select,
# env var OMNIVOICE_TTS_BACKEND wins), or an explicit per-request `engine`
# override — same pattern as /ws/tts's `engine` field and /v1/audio/speech's
# `model`. Omitting both keeps the historical default (OmniVoice), so
# existing API consumers see no change.
from services.tts_backend import (
OmniVoiceBackend, _mask_hf_tokens, active_backend_id, get_backend_class,
)
engine_id = engine or active_backend_id()
try:
backend_cls = get_backend_class(engine_id)
except ValueError:
raise HTTPException(
status_code=400,
detail=(
f"Unknown TTS engine: {engine_id!r}. "
"See GET /engines/tts for the list of valid engine ids."
),
)
_model = None
_backend = None
if backend_cls is OmniVoiceBackend:
# OmniVoice keeps its native path: it carries the full advanced
# parameter surface (t_shift, layer/position/class controls) that the
# generic adapter protocol doesn't. Byte-identical to the old behavior.
_model = await get_model()
else:
try:
ok, msg = backend_cls.is_available()
except Exception as exc:
ok, msg = False, f"{type(exc).__name__}: {exc}"
if not ok:
raise HTTPException(
status_code=400,
detail=f"TTS engine '{engine_id}' is not available: {_mask_hf_tokens(msg)}",
)
# Reuse the per-process instance cache shared with the engine
# health-check route so weights load once, not per request.
from api.routers.engines import _get_engine_instance
_backend = _get_engine_instance(backend_cls)
# ── Routing gate (#21 — no silent CPU fallback). Computed ONCE per request
# (host caps are constant; the per-request engine= override bypasses the
# /engines/select gate, so this is the only place it's enforced for synth).
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
_routing = resolve_routing(getattr(backend_cls, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
# The engine needs an accelerator this host lacks and has no CPU path.
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
ref_audio_path = None
cleanup_ref = False
used_seed = seed
resolved_profile_id = None
history_mode = None # profile.kind when a profile drives; else inferred at insert
if profile_id:
with db_conn() as conn:
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if row:
resolved_profile_id = profile_id
# `kind` is authoritative (0005): 'design' profiles condition on
# their deterministic rendered sample + instruct; 'clone' on the
# user's reference. Lock always wins (it pins a specific take).
# Rows from pre-0004 DBs mid-upgrade may lack the column → fall
# back to the legacy is_locked/instruct inference.
try:
profile_kind = row["kind"] or "clone"
except (KeyError, IndexError):
profile_kind = "design" if (row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]) else "clone"
history_mode = profile_kind
if row["is_locked"] and row["locked_audio_path"]:
ref_audio_path = os.path.join(VOICES_DIR, row["locked_audio_path"])
if not ref_text:
@@ -189,7 +403,19 @@ async def generate_speech(
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
used_seed = row["seed"]
elif row["instruct"] and not row["is_locked"]:
elif profile_kind == "design":
# Rendered sample (if present) carries the voice identity;
# instruct alone is the fallback for legacy archetype rows.
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
if ref_audio_path and not ref_text and row["ref_text"]:
ref_text = row["ref_text"]
if not instruct:
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
used_seed = row["seed"]
elif row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]:
# Legacy design-shaped row (pre-0004 archetype materialization
# failure path): instruct-only conditioning.
if not instruct:
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
@@ -213,16 +439,41 @@ async def generate_speech(
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
# #308: a transcript-less reference is transcribed with the active ASR
# backend (whisperx / faster-whisper / mlx-whisper) instead of the model's
# built-in transformers pipeline, which cannot load whisper-large-v3-turbo
# on transformers 5.3. On failure ref_text stays None and the model's
# fallback behaves exactly as before.
if ref_audio_path and not ref_text:
from services.asr_backend import transcribe_reference
ref_text = await asyncio.get_running_loop().run_in_executor(
_gpu_pool, transcribe_reference, ref_audio_path
)
start_time = time.time()
try:
loop = asyncio.get_running_loop()
audio_tensor = await loop.run_in_executor(
_gpu_pool, _run_inference,
_model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset,
)
if _backend is not None:
audio_tensor = await loop.run_in_executor(
_gpu_pool, _run_backend_inference,
_backend, text, language, ref_audio_path, ref_text, instruct,
duration, num_step, guidance_scale, speed, denoise,
postprocess_output, used_seed, effect_preset,
max_chunk_chars, crossfade_ms,
)
# Read after generation: engines with lazy model loading report
# their real rate only once weights are up.
sample_rate = _backend.sample_rate
else:
audio_tensor = await loop.run_in_executor(
_gpu_pool, _run_inference,
_model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset,
max_chunk_chars, crossfade_ms,
)
sample_rate = _model.sampling_rate
# Invisible AudioSeal provenance watermark on the final audio. Embedding
# was previously only wired into the dub pipeline (dub_generate.py), so
# plain TTS came out unmarked despite the setting being on. embed_watermark
@@ -231,28 +482,28 @@ async def generate_speech(
# generation.
from services.watermark import embed_watermark
audio_tensor = await loop.run_in_executor(
_gpu_pool, embed_watermark, audio_tensor, _model.sampling_rate
_gpu_pool, embed_watermark, audio_tensor, sample_rate
)
gen_time = round(time.time() - start_time, 2)
audio_id = str(uuid.uuid4())[:8]
audio_filename = f"{audio_id}.wav"
audio_path = os.path.join(OUTPUTS_DIR, audio_filename)
_safe_torchaudio_save(audio_path, audio_tensor, _model.sampling_rate)
_safe_torchaudio_save(audio_path, audio_tensor, sample_rate)
audio_dur = round(audio_tensor.shape[-1] / _model.sampling_rate, 2)
audio_dur = round(audio_tensor.shape[-1] / sample_rate, 2)
with db_conn() as conn:
conn.execute(
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
(audio_id, text[:200], "clone" if ref_audio_path else "design",
(audio_id, text[:200], history_mode or ("clone" if ref_audio_path else "design"),
language or "Auto", instruct or "", resolved_profile_id,
audio_filename, audio_dur, gen_time, used_seed, time.time())
)
event_bus.emit("generation_history", {"action": "created", "id": audio_id})
buffer = io.BytesIO()
_safe_torchaudio_save(buffer, audio_tensor, _model.sampling_rate, format="wav")
_safe_torchaudio_save(buffer, audio_tensor, sample_rate, format="wav")
buffer.seek(0)
wav_bytes = buffer.read()
@@ -261,17 +512,26 @@ async def generate_speech(
for i in range(0, len(wav_bytes), chunk_size):
yield wav_bytes[i:i + chunk_size]
_resp_headers = {
"X-Audio-Id": audio_id,
"X-Gen-Time": str(gen_time),
"X-Audio-Path": audio_filename,
"X-Seed": str(used_seed) if used_seed is not None else "",
"X-Audio-Duration": str(audio_dur),
"Content-Length": str(len(wav_bytes)),
}
# Routing notice (#21): cpu_fallback or accelerated-with-caveat only;
# the WAV body is binary so the header channel is the carrier.
if _routing_notice:
from services.engine_routing import header_safe_reason
_resp_headers["X-OmniVoice-Routing"] = _routing_notice[0]
_hr = header_safe_reason(_routing_notice[1])
if _hr:
_resp_headers["X-OmniVoice-Routing-Reason"] = _hr
return StreamingResponse(
_stream_wav(),
media_type="audio/wav",
headers={
"X-Audio-Id": audio_id,
"X-Gen-Time": str(gen_time),
"X-Audio-Path": audio_filename,
"X-Seed": str(used_seed) if used_seed is not None else "",
"X-Audio-Duration": str(audio_dur),
"Content-Length": str(len(wav_bytes)),
}
headers=_resp_headers,
)
except HTTPException:
raise
@@ -308,9 +568,29 @@ def _safe_output_path(name):
@router.get("/history")
def list_history():
"""Newest 50 generations whose audio still exists on disk.
Rows whose WAV was deleted out-of-band (cleared outputs dir, manual
cleanup) used to come back anyway and render dead players that 404 on
every fetch; prune them here so the UI never sees them again."""
with db_conn() as conn:
rows = conn.execute("SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50").fetchall()
return [dict(r) for r in rows]
rows = conn.execute(
"SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50"
).fetchall()
alive, stale_ids = [], []
for r in rows:
p = _safe_output_path(r["audio_path"]) if r["audio_path"] else None
if r["audio_path"] and (not p or not os.path.exists(p)):
stale_ids.append(r["id"])
else:
alive.append(dict(r))
if stale_ids:
conn.executemany(
"DELETE FROM generation_history WHERE id=?",
[(i,) for i in stale_ids],
)
logger.info("pruned %d stale history rows (audio file gone)", len(stale_ids))
return alive
@router.delete("/history")
def clear_history():
+158
View File
@@ -0,0 +1,158 @@
"""Longform Job Library (PR 7).
``GET /longform/jobs`` list finished Audiobook + Story renders so the user can
re-download them from the Projects view. The render itself (the m4b/mp3) already
landed in ``OUTPUTS_DIR`` and is served at ``/audio/<output>``; here we just
recover, from each finished job's persisted SSE tail, the output filename plus
the chapter count and duration the ``done`` event carried.
Pure recovery, no synthesis. Defensive by construction: a job whose ``done``
event is missing or unparseable is skipped, never surfaced and never a 500.
The work lives in :func:`build_longform_library`, a pure function over the
job-store callables, so it's unit-testable without importing ``main`` (and the
torch graph behind it).
"""
from __future__ import annotations
import json
import logging
from typing import Callable, Optional
from fastapi import APIRouter, Query
logger = logging.getLogger("omnivoice.longform_jobs")
router = APIRouter()
#: Job types this library surfaces. Both flow through the shared longform
#: renderer (``_render_longform_sse``) and emit the same ``done`` event shape.
_LONGFORM_TYPES = ("audiobook", "story")
def _done_payload_from_events(events: list[dict]) -> Optional[dict]:
"""Recover the final ``{"type": "done", ...}`` payload from a job's SSE tail.
Each row's ``payload`` is the JSON the renderer stored via
``job_store.append_event(job_id, json.dumps(payload))``. We scan newest-first
and return the first parseable ``done`` event. Anything malformed is skipped
this never raises.
"""
for ev in reversed(events):
raw = ev.get("payload") if isinstance(ev, dict) else None
if not raw or not isinstance(raw, str):
continue
try:
obj = json.loads(raw)
except (ValueError, TypeError):
continue
if isinstance(obj, dict) and obj.get("type") == "done":
return obj
return None
def _coerce_int(value, default: int = 0) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def _coerce_float(value, default: float = 0.0) -> float:
try:
return float(value)
except (TypeError, ValueError):
return default
def build_longform_library(
list_jobs: Callable[..., list[dict]],
events_since: Callable[..., list[dict]],
*,
limit: int = 50,
) -> list[dict]:
"""Build the newest-first list of finished longform renders.
Pure over the two job-store callables so tests can pass them directly:
* ``list_jobs(status="done", limit=...)`` all done jobs, newest-first.
* ``events_since(job_id)`` that job's persisted SSE events.
Returns ``[{job_id, type, title?, output, duration_s, chapters,
created_at}]``. Jobs that aren't a longform type, or whose ``done`` event /
output filename can't be recovered, are silently skipped — the library only
ever lists things the user can actually re-download.
"""
limit = max(1, min(_coerce_int(limit, 50), 500))
try:
# Over-fetch: non-longform done jobs (dub, etc.) get filtered out below,
# so ask for more rows than the caller's limit to still fill the page.
rows = list_jobs(status="done", limit=limit * 4)
except Exception:
logger.warning("longform library: list_jobs failed", exc_info=True)
return []
out: list[dict] = []
for row in rows or []:
if len(out) >= limit:
break
try:
job_type = row.get("type")
job_id = row.get("id")
if job_type not in _LONGFORM_TYPES or not job_id:
continue
try:
events = events_since(job_id)
except Exception:
logger.warning("longform library: events_since failed for %s",
job_id, exc_info=True)
continue
done = _done_payload_from_events(events or [])
if not done:
continue
output = done.get("output")
if not output or not isinstance(output, str):
continue # nothing to re-download → not worth listing
item = {
"job_id": job_id,
"type": job_type,
"output": output,
"duration_s": round(_coerce_float(done.get("duration_s")), 2),
"chapters": _coerce_int(done.get("chapters")),
"created_at": row.get("created_at"),
}
# Title is optional — prefer the done event, fall back to job meta.
title = done.get("title")
if not title:
meta_raw = row.get("meta_json")
if isinstance(meta_raw, str) and meta_raw:
try:
meta = json.loads(meta_raw)
if isinstance(meta, dict):
title = meta.get("title")
except (ValueError, TypeError):
title = None
if title:
item["title"] = title
out.append(item)
except Exception:
# Per-row isolation: one bad row never sinks the whole list.
logger.warning("longform library: skipping unparseable job row",
exc_info=True)
continue
return out
@router.get("/longform/jobs")
def longform_jobs(limit: int = Query(50, ge=1, le=500)) -> dict:
"""Finished Audiobook + Story renders, newest-first, ready to re-download.
Each item's ``output`` is served at ``/audio/<output>``. Never 500s — on any
backend hiccup it returns an empty list rather than an error.
"""
from core import job_store
jobs = build_longform_library(
job_store.list_jobs, job_store.events_since, limit=limit,
)
return {"jobs": jobs}
+40 -28
View File
@@ -58,6 +58,31 @@ MAX_BUNDLE_BYTES = 100 * 1024 * 1024
# ── Export ──────────────────────────────────────────────────────────────────
def _bundle_metadata(profile: dict, **extra) -> dict:
"""Common .omnivoice metadata for export + publish.
Captures ``kind`` and ``vd_states`` so a *designed* persona survives the
bundle round-trip as a design (not silently demoted to a clone) required
for the synthetic-only gating of the persona gallery (§R3). Old bundles
without these keys import as ``kind='clone'`` (backward-compatible).
"""
meta = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"kind": profile.get("kind") or "clone",
"vd_states": profile.get("vd_states"),
"is_locked": bool(profile.get("is_locked")),
"omnivoice_version": APP_VERSION,
}
meta.update(extra)
return meta
@router.post("/export/{profile_id}")
def export_profile(profile_id: str):
"""Export a voice profile as a downloadable .omnivoice bundle (ZIP)."""
@@ -75,19 +100,9 @@ def export_profile(profile_id: str):
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
# Metadata
metadata = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"is_locked": bool(profile.get("is_locked")),
"created_at": profile.get("created_at"),
"exported_at": time.time(),
"omnivoice_version": APP_VERSION,
}
metadata = _bundle_metadata(
profile, created_at=profile.get("created_at"), exported_at=time.time(),
)
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
# Reference audio
@@ -191,8 +206,9 @@ async def import_profile(
conn.execute(
"""INSERT INTO voice_profiles
(id, name, ref_audio_path, ref_text, instruct, language,
seed, personality, is_locked, locked_audio_path, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
seed, personality, is_locked, locked_audio_path, created_at,
kind, vd_states)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
profile_id,
metadata.get("profile_name", "Imported Voice"),
@@ -205,6 +221,10 @@ async def import_profile(
1 if is_locked else 0,
locked_audio_filename or "",
time.time(),
# Preserve the design/clone distinction across the round-trip;
# old bundles without these keys import as a clone.
metadata.get("kind") or "clone",
metadata.get("vd_states"),
),
)
@@ -253,19 +273,11 @@ def publish_to_marketplace(
# Build the bundle
with zipfile.ZipFile(str(bundle_path), "w", zipfile.ZIP_DEFLATED) as zf:
metadata = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"is_locked": bool(profile.get("is_locked")),
"tags": [t.strip() for t in tags.split(",") if t.strip()],
"published_at": time.time(),
"omnivoice_version": APP_VERSION,
}
metadata = _bundle_metadata(
profile,
tags=[t.strip() for t in tags.split(",") if t.strip()],
published_at=time.time(),
)
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
ref_path = profile.get("ref_audio_path")
+53
View File
@@ -0,0 +1,53 @@
"""REST CRUD for per-agent MCP voice bindings (Wave 2.2 / Spec 2).
Loopback-gated the Settings UI manages bindings here. The MCP tools
themselves resolve voices via ``services.mcp_bindings.resolve_voice``.
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from api.dependencies import require_loopback
from services import mcp_bindings
router = APIRouter(
prefix="/api/mcp",
tags=["mcp"],
dependencies=[Depends(require_loopback)],
)
class _BindingBody(BaseModel):
client_id: str = Field(..., min_length=1, max_length=128)
label: str | None = None
profile_id: str | None = None
default_engine: str | None = None
@router.get("/bindings")
def list_bindings():
"""All per-agent voice bindings, most-recently-seen first."""
return mcp_bindings.list_bindings()
@router.put("/bindings")
def upsert_binding(body: _BindingBody):
"""Create or update the binding for an MCP client id."""
try:
return mcp_bindings.upsert_binding(
body.client_id,
label=body.label,
profile_id=body.profile_id,
default_engine=body.default_engine,
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/bindings/{client_id}")
def delete_binding(client_id: str):
if not mcp_bindings.delete_binding(client_id):
raise HTTPException(status_code=404, detail="No binding for that client id")
return {"deleted": client_id}
+66 -5
View File
@@ -82,6 +82,33 @@ class SpeechRequest(BaseModel):
"E.g. 'young female, warm tone, slight British accent'.",
)
instruct: Optional[str] = Field(default=None, description="Style instruction for the TTS engine.")
duration: Optional[float] = Field(
default=None,
gt=0,
description="OmniVoice extension: target output duration in seconds.",
)
seed: Optional[int] = Field(
default=None,
description="OmniVoice extension: deterministic sampling seed.",
)
denoise: bool = Field(
default=True,
description="OmniVoice extension: prepend denoise control when supported.",
)
preprocess_prompt: bool = Field(
default=True,
description="OmniVoice extension: trim/preprocess reference prompt when supported.",
)
chunk_duration: Optional[float] = Field(
default=None,
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk duration.",
)
chunk_threshold: Optional[float] = Field(
default=None,
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
)
class TranscriptionResponse(BaseModel):
@@ -209,7 +236,14 @@ def _run_tts(backend, text: str, kw: dict):
from services.audio_dsp import apply_mastering, normalize_audio
wav = backend.generate(text, **kw)
sr = backend.sample_rate
wav = apply_mastering(wav, sample_rate=sr)
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
# That chain's Compressor + 8% Reverb is tuned for OmniVoice's 24 kHz clone
# output; applied to a studio engine it adds an audible level pump and a
# reverb tail that degrade the very output we want clean. Loudness
# normalisation still runs — it's a benign peak scale, not dynamics.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr)
wav = normalize_audio(wav, target_dBFS=-2.0)
return wav, sr
@@ -219,10 +253,28 @@ async def create_speech(req: SpeechRequest):
"""Generate audio from text. Compatible with OpenAI's POST /v1/audio/speech."""
backend = _resolve_engine(req.model)
# Routing gate (#21 — no silent CPU fallback), identical to REST /generate.
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
_routing = resolve_routing(getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
# Build kwargs for the backend's generate() method
kw: dict = {
"speed": req.speed,
"denoise": req.denoise,
"preprocess_prompt": req.preprocess_prompt,
}
if req.duration is not None:
kw["duration"] = req.duration
if req.seed is not None:
kw["seed"] = req.seed
if req.chunk_duration is not None:
kw["chunk_duration"] = req.chunk_duration
if req.chunk_threshold is not None:
kw["chunk_threshold"] = req.chunk_threshold
if req.language:
kw["language"] = req.language
if req.instruct:
@@ -251,6 +303,8 @@ async def create_speech(req: SpeechRequest):
kw["ref_text"] = row["ref_text"]
if row["instruct"] and not req.instruct:
kw["instruct"] = row["instruct"]
if req.seed is None and row["seed"] is not None:
kw["seed"] = row["seed"]
else:
# Not a profile ID — forward as engine preset name
kw["voice"] = voice
@@ -267,13 +321,20 @@ async def create_speech(req: SpeechRequest):
audio_bytes, mime_type, ext = _encode_audio(wav, sr, req.response_format)
_headers = {
"Content-Length": str(len(audio_bytes)),
"Content-Disposition": f'inline; filename="speech.{ext}"',
}
if _routing_notice:
from services.engine_routing import header_safe_reason
_headers["X-OmniVoice-Routing"] = _routing_notice[0]
_hr = header_safe_reason(_routing_notice[1])
if _hr:
_headers["X-OmniVoice-Routing-Reason"] = _hr
return StreamingResponse(
io.BytesIO(audio_bytes),
media_type=mime_type,
headers={
"Content-Length": str(len(audio_bytes)),
"Content-Disposition": f'inline; filename="speech.{ext}"',
},
headers=_headers,
)
+327
View File
@@ -0,0 +1,327 @@
"""HTTP layer for the `.ovsvoice` portable persona format (#29 / parity §R3 G1).
Thin router over `services.persona_bundle`:
POST /personas/export/{profile_id} stream a downloadable .ovsvoice
POST /personas/import create a profile from a bundle
POST /personas/inspect read a bundle's manifest, no writes
Mirrors the legacy `.omnivoice` endpoints (`marketplace.py`) and reuses the
same path-confinement (`_voices_path`) + consent floor. `.ovsvoice` is additive;
`.omnivoice` import stays a compatible legacy reader.
"""
from __future__ import annotations
import asyncio
import functools
import logging
import os
import time
import uuid
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
from fastapi.responses import StreamingResponse
from core import event_bus
from core.config import VOICES_DIR # noqa: F401 — re-exported for tests/monkeypatch
from core.db import db_conn
from core.version import APP_VERSION
from services import persona_bundle as pb
router = APIRouter()
logger = logging.getLogger("omnivoice.personas")
def _safe_name(name: str, profile_id: str) -> str:
"""Sanitised download filename stem (marketplace idiom); empty → persona_<id>."""
cleaned = "".join(
c if c.isalnum() or c in "-_ " else "" for c in (name or "")
).strip().replace(" ", "_")[:40]
return cleaned or f"persona_{profile_id}"
# ── Export ────────────────────────────────────────────────────────────────
@router.post("/personas/export/{profile_id}")
async def export_persona(
profile_id: str,
license_spdx: str = Query(pb.DEFAULT_LICENSE),
tags: str = Query(""),
include_reference: bool = Query(True),
):
"""Build + stream a `.ovsvoice` bundle for a profile."""
with db_conn() as conn:
row = conn.execute(
"SELECT * FROM voice_profiles WHERE id = ?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Voice profile not found")
profile = dict(row)
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
try:
loop = asyncio.get_running_loop()
content = await loop.run_in_executor(
None,
functools.partial(
pb.build_persona_bundle,
profile,
license_spdx=license_spdx,
tags=tag_list,
include_reference=include_reference,
engine_id=os.environ.get("OMNIVOICE_MODEL", ""),
omnivoice_version=APP_VERSION,
),
)
except pb.NoPreviewSource:
raise HTTPException(
status_code=503,
detail="This profile has no readable reference or locked audio to "
"build a preview from — re-create or re-import it.",
)
except Exception:
logger.exception("persona export failed for %s", profile_id)
raise HTTPException(
status_code=503,
detail="Could not build the persona bundle — see Settings → Logs.",
)
filename = f"{_safe_name(profile.get('name'), profile_id)}.ovsvoice"
from io import BytesIO
return StreamingResponse(
BytesIO(content),
media_type="application/zip",
headers={
"Content-Disposition": f'attachment; filename="{filename}"',
"Content-Length": str(len(content)),
},
)
# ── Import ────────────────────────────────────────────────────────────────
def _voices_dest(filename: str) -> str:
"""Resolve an output filename inside VOICES_DIR; 400 on escape (belt+braces —
the name is always server-generated `{profile_id}`)."""
from api.routers.profiles import _voices_path
path = _voices_path(filename)
if path is None:
raise HTTPException(status_code=400, detail="Invalid profile id")
return path
def _consent_verified(parsed: pb.ParsedPersona, consent_path: str | None) -> bool:
"""B12-B16: trust verified-own-voice ONLY with a real recording (≥ floor) AND
non-empty consent_text AND a consent.json present. The manifest flag alone
can't forge it."""
if not parsed.consent or not consent_path:
return False
if os.path.getsize(consent_path) < pb._MIN_CONSENT_AUDIO_BYTES:
return False
return bool((parsed.consent.get("consent_text") or "").strip())
@router.post("/personas/import")
async def import_persona(file: UploadFile = File(...)):
"""Create a new voice profile from a `.ovsvoice` (or legacy `.omnivoice`) bundle."""
name = (file.filename or "").lower()
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
content = await file.read()
try:
parsed = pb.parse_persona_bundle(content)
except pb.BundleError as e:
raise HTTPException(status_code=e.status, detail=e.detail)
persona = parsed.manifest.get("persona") or {}
written: list[str] = []
def _gen_id() -> str:
return str(uuid.uuid4())[:8]
profile_id = _gen_id()
try:
# ── Audio members → server-named files (never the member name). ──
ref_filename = None
locked_filename = None
if "ref_audio" in parsed.members:
ref_filename = f"{profile_id}{parsed.member_ext('ref_audio')}"
dest = _voices_dest(ref_filename)
parsed.extract_member("ref_audio", dest); written.append(dest)
if "locked_audio" in parsed.members:
locked_filename = f"{profile_id}_locked{parsed.member_ext('locked_audio')}"
dest = _voices_dest(locked_filename)
parsed.extract_member("locked_audio", dest); written.append(dest)
# Preview-only bundle (A12/B8): use the preview as the usable ref clip.
if ref_filename is None and locked_filename is None and "preview" in parsed.members:
ref_filename = f"{profile_id}{parsed.member_ext('preview')}"
dest = _voices_dest(ref_filename)
parsed.extract_member("preview", dest); written.append(dest)
if ref_filename is None and locked_filename is None:
raise HTTPException(status_code=400, detail="bundle has no usable audio")
# ── Consent recording (optional) ──
consent_filename = None
consent_path = None
if "consent_audio" in parsed.members:
consent_filename = f"{profile_id}_consent{parsed.member_ext('consent_audio')}"
consent_path = _voices_dest(consent_filename)
parsed.extract_member("consent_audio", consent_path); written.append(consent_path)
verified = _consent_verified(parsed, consent_path)
consent_text = ((parsed.consent or {}).get("consent_text") or "").strip()
recorded_at = None
if verified:
try:
recorded_at = float(parsed.consent.get("recorded_at"))
except (TypeError, ValueError):
recorded_at = time.time()
is_locked = bool(persona.get("is_locked") and locked_filename)
ref_for_db = ref_filename or locked_filename # at least one is set
def _insert(pid: str):
with db_conn() as conn:
conn.execute(
"""INSERT INTO voice_profiles
(id, name, ref_audio_path, ref_text, instruct, language,
seed, personality, is_locked, locked_audio_path, created_at,
kind, vd_states,
verified_own_voice, consent_text, consent_audio_path, consent_recorded_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
pid,
persona.get("name") or "Imported Voice",
ref_for_db,
persona.get("ref_text", ""),
persona.get("instruct", ""),
persona.get("language", "Auto"),
persona.get("seed"),
persona.get("personality", ""),
1 if is_locked else 0,
locked_filename or "",
time.time(),
persona.get("kind") or "clone",
persona.get("vd_states"),
1 if verified else 0,
# Keep the attestation text so the user can re-attest locally,
# even when imported unverified.
consent_text,
consent_filename if verified else "",
recorded_at if verified else None,
),
)
import sqlite3
try:
_insert(profile_id)
except sqlite3.IntegrityError:
profile_id = _gen_id() # one retry on id collision (B20)
# rename the on-disk files to the new id so they still match the row
written = _rename_for_new_id(written, profile_id)
ref_for_db = _retarget(ref_for_db, profile_id)
locked_filename = _retarget(locked_filename, profile_id)
consent_filename = _retarget(consent_filename, profile_id)
_insert(profile_id)
except HTTPException:
_cleanup(written)
raise
except Exception:
_cleanup(written)
logger.exception("persona import failed")
raise HTTPException(status_code=500, detail="Import failed; no files were kept.")
event_bus.emit("profiles", {"action": "created", "id": profile_id})
logger.info("Imported persona %r as %s (verified=%s)", persona.get("name"), profile_id, verified)
return {
"success": True,
"profile_id": profile_id,
"name": persona.get("name") or "Imported Voice",
"kind": persona.get("kind") or "clone",
"verified_own_voice": verified,
"preview_only": parsed.preview_only,
"license_spdx": parsed.license_spdx,
"watermarked_preview": parsed.watermarked_preview,
"source_bundle": file.filename,
"schema_version_ahead": parsed.schema_version_ahead,
}
def _cleanup(paths: list[str]) -> None:
for p in paths:
try:
if p and os.path.exists(p):
os.remove(p)
except OSError:
pass
def _rename_for_new_id(written: list[str], new_id: str) -> list[str]:
"""After an id-collision retry, rename each written file to carry the new id
(filenames are `{old_id}`; swap the leading 8-char stem)."""
out = []
for p in written:
d, base = os.path.split(p)
# base looks like {id}{ext} | {id}_locked{ext} | {id}_consent{ext}
new_base = new_id + base[8:]
new_path = os.path.join(d, new_base)
try:
os.replace(p, new_path)
out.append(new_path)
except OSError:
out.append(p)
return out
def _retarget(filename: str | None, new_id: str) -> str | None:
return new_id + filename[8:] if filename else filename
# ── Inspect (no-write preview) ──────────────────────────────────────────────
@router.post("/personas/inspect")
async def inspect_persona(file: UploadFile = File(...)):
"""Read a bundle's manifest + consent summary WITHOUT writing any file or row."""
name = (file.filename or "").lower()
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
content = await file.read()
try:
parsed = pb.parse_persona_bundle(content)
except pb.BundleError as e:
raise HTTPException(status_code=e.status, detail=e.detail)
persona = parsed.manifest.get("persona") or {}
consent_summary = None
if parsed.consent:
has_recording = "consent_audio" in parsed.members
consent_summary = {
"verified_claimed": bool(parsed.consent.get("verified_own_voice")),
"method": parsed.consent.get("method", ""),
"has_recording": has_recording,
# would_verify mirrors import's gate, minus the byte-floor check
# (inspect never extracts to measure size — advisory only).
"would_verify": has_recording and bool((parsed.consent.get("consent_text") or "").strip()),
}
return {
"format": "omnivoice-legacy" if parsed.is_legacy else pb.OVSVOICE_FORMAT,
"schema_version": parsed.manifest.get("schema_version", pb.OVSVOICE_SCHEMA_VERSION),
"name": persona.get("name") or "Imported Voice",
"kind": persona.get("kind") or "clone",
"language": persona.get("language", "Auto"),
"personality": persona.get("personality", ""),
"is_locked": bool(persona.get("is_locked")),
"license_spdx": parsed.license_spdx,
"tags": parsed.manifest.get("tags") or [],
"preview_only": parsed.preview_only,
"watermarked_preview": parsed.watermarked_preview,
"consent": consent_summary,
"schema_version_ahead": parsed.schema_version_ahead,
}
+277 -19
View File
@@ -1,4 +1,5 @@
import os
import re
import uuid
import time
import shutil
@@ -34,29 +35,103 @@ def list_profiles():
rows = conn.execute("SELECT * FROM voice_profiles ORDER BY created_at DESC").fetchall()
return [dict(r) for r in rows]
_DESIGN_SEED = 42 # deterministic sample render, same as archetype previews
@router.post("/profiles")
async def create_profile(
name: str = Form(...),
ref_audio: UploadFile = File(...),
ref_audio: Optional[UploadFile] = File(None),
ref_text: str = Form(""),
instruct: str = Form(""),
language: str = Form("Auto"),
seed: Optional[int] = Form(None),
personality: str = Form(""),
kind: str = Form("clone"),
vd_states: Optional[str] = Form(None),
):
profile_id = str(uuid.uuid4())[:8]
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
audio_filename = f"{profile_id}{ext}"
audio_path = os.path.join(VOICES_DIR, audio_filename)
"""Create a voice profile (spec: docs/specs/voice-studio-unification.md §5).
with open(audio_path, "wb") as f:
f.write(await ref_audio.read())
kind='clone' requires `ref_audio` (the user's reference recording).
kind='design' requires `vd_states` (JSON of category picks); the server
renders a deterministic sample WAV (seed 42, same path as
archetype materialization) and stores it as the profile's
reference so the voice identity is stable across runs.
"""
if kind not in ("clone", "design"):
raise HTTPException(status_code=422, detail="kind must be 'clone' or 'design'")
if kind == "clone" and ref_audio is None:
raise HTTPException(status_code=422, detail="clone profiles require ref_audio")
if kind == "design":
if not (vd_states or "").strip():
raise HTTPException(status_code=422, detail="design profiles require vd_states")
import json as _json
try:
parsed = _json.loads(vd_states)
if not isinstance(parsed, dict):
raise ValueError("not an object")
except ValueError:
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
# An all-Auto design (every category left on "Auto") yields an empty
# instruct — that's still a valid, saveable voice: synthesis falls back
# to neutral instruct-only conditioning (see generation.py design path).
# Don't gate save on a non-empty instruct.
profile_id = str(uuid.uuid4())[:8]
if kind == "clone":
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
audio_filename = f"{profile_id}{ext}"
audio_path = os.path.join(VOICES_DIR, audio_filename)
with open(audio_path, "wb") as f:
f.write(await ref_audio.read())
used_seed = seed
else:
# Saving a design profile is a pure persistence operation — it must not
# depend on a loaded TTS model (issue #476: on a fresh model-less Docker
# image the render forced a full model load + inference that 503'd, so
# the save failed). We try the deterministic identity sample opportunist-
# ically through the one shared TTS path (archetypes' renderer, never a
# second inference code path); if the engine isn't ready it's rendered
# lazily on first preview/use. The row carries vd_states + instruct, so
# the voice is fully usable without the sample (synthesis falls back to
# instruct-only conditioning — see generation.py's design path).
from pathlib import Path
from api.routers.archetypes import _render_archetype_wav
audio_filename = f"{profile_id}.wav"
audio_path = os.path.join(VOICES_DIR, audio_filename)
try:
await _render_archetype_wav(
{
"language": language,
"sample_script": ref_text, # optional custom sample line
"instruct": instruct,
},
Path(audio_path),
)
except Exception:
# Engine unavailable / OOM / inference failure — defer the sample.
# Store the row with no ref_audio_path; the identity sample is
# rendered on first preview or use. Never let this block the save.
import logging
logging.getLogger("omnivoice.profiles").info(
"Design profile %s saved with sample pending — "
"voice engine not ready; will render on first use", profile_id,
)
if os.path.exists(audio_path): # partial/blank render: don't keep it
with __import__("contextlib").suppress(OSError):
os.remove(audio_path)
audio_filename = None
used_seed = seed if seed is not None else _DESIGN_SEED
try:
with db_conn() as conn:
conn.execute(
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, name, audio_filename, ref_text, instruct, language, seed, personality, time.time())
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, "
"language, seed, personality, kind, vd_states, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, name, audio_filename, ref_text, instruct, language,
used_seed, personality, kind, vd_states, time.time())
)
except Exception:
# Clean up orphaned audio file if DB insert fails
@@ -64,7 +139,7 @@ async def create_profile(
os.remove(audio_path)
raise
event_bus.emit("profiles", {"action": "created", "id": profile_id})
return {"id": profile_id, "name": name}
return {"id": profile_id, "name": name, "kind": kind}
@router.get("/profiles/{profile_id}")
def get_profile(profile_id: str):
@@ -163,20 +238,96 @@ def get_profile_usage(profile_id: str):
}
# profile_id is a request path param and the audio filename derives from it, so
# constrain it to the generated-id charset (no separators / `..` possible) before
# any path use, and read only a *direct child* of VOICES_DIR — os.path.basename()
# strips any directory component (a path-injection / CWE-22 barrier).
_PROFILE_ID_RE = re.compile(r"[A-Za-z0-9_-]{1,64}")
@router.get("/profiles/{profile_id}/audio")
def get_profile_audio(profile_id: str):
async def get_profile_audio(profile_id: str):
if not _PROFILE_ID_RE.fullmatch(profile_id or ""):
return Response("Profile not found", status_code=404)
with db_conn() as conn:
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
row = conn.execute(
"SELECT ref_audio_path, locked_audio_path, kind, instruct, language, ref_text "
"FROM voice_profiles WHERE id=?",
(profile_id,),
).fetchone()
if not row:
return Response("Profile not found", status_code=404)
audio_file = row["locked_audio_path"] or row["ref_audio_path"]
if not audio_file:
return Response("No audio available", status_code=404)
audio_path = os.path.join(VOICES_DIR, audio_file)
if not os.path.exists(audio_path):
# A design profile saved before the engine was ready (issue #476) has no
# identity sample yet. Render it lazily now — the deterministic seed-42
# sample is reproducible, so a deferred render matches a save-time one.
rendered = await _materialize_design_sample(profile_id, row)
if rendered is None:
return Response("No audio available", status_code=404)
audio_file = rendered
# CWE-22: resolve the DB-stored filename strictly inside VOICES_DIR via the
# shared guard — _voices_path() applies the os.path.basename() barrier plus
# symlink-resolved containment (same path the consent endpoint trusts).
audio_path = _voices_path(str(audio_file))
if audio_path is None or not os.path.exists(audio_path):
return Response("Audio file missing", status_code=404)
return FileResponse(audio_path, media_type="audio/wav")
async def _materialize_design_sample(profile_id: str, row) -> Optional[str]:
"""Render a design profile's pending identity sample on first request.
Returns the stored filename on success, or None if this isn't a renderable
design row. Raises HTTPException(503) with a precise "model not ready"
message if the engine is genuinely unavailable saving never depends on
this, but a user who explicitly asks for the sample gets a clear signal.
"""
try:
kind = row["kind"]
except (KeyError, IndexError):
kind = "clone"
if kind != "design":
return None
from pathlib import Path
from api.routers.archetypes import _render_archetype_wav
audio_filename = f"{profile_id}.wav"
# CWE-22: resolve under VOICES_DIR via the shared basename + containment
# guard before rendering (rejects any escape).
audio_path = _voices_path(audio_filename)
if audio_path is None:
raise HTTPException(status_code=400, detail="invalid profile identifier")
try:
await _render_archetype_wav(
{
"language": row["language"] or "Auto",
"sample_script": row["ref_text"] or "",
"instruct": row["instruct"] or "",
},
Path(audio_path),
)
except Exception as e:
with __import__("contextlib").suppress(OSError):
if os.path.exists(audio_path):
os.remove(audio_path)
raise HTTPException(
status_code=503,
detail=(
"The voice engine isn't ready yet, so this designed voice's "
"preview sample can't be rendered. Finish setup / download a "
f"model, then try again. ({e})"
),
)
with db_conn() as conn:
conn.execute(
"UPDATE voice_profiles SET ref_audio_path=? WHERE id=?",
(audio_filename, profile_id),
)
return audio_filename
@router.post("/profiles/{profile_id}/lock")
async def lock_profile(
profile_id: str,
@@ -234,15 +385,122 @@ async def unlock_profile(profile_id: str):
event_bus.emit("profiles", {"action": "unlocked", "id": profile_id})
return {"unlocked": True, "profile_id": profile_id}
# ── Consent lock (parity program Wave 0.2) ─────────────────────────────────
#
# A profile becomes "verified own voice" when its owner records themselves
# reading a consent statement. The recording is provenance, not a voiceprint
# check — agentic features and gallery sharing gate on the flag; plain local
# synthesis never does. Spec: docs/competitive-analysis.md Action 22.
_MIN_CONSENT_AUDIO_BYTES = 1000 # same floor as the frontend recorder
# Upload filename extension whitelist — anything else falls back to .wav so a
# crafted filename can never influence the on-disk path (py/path-injection).
_CONSENT_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
def _voices_path(filename: str) -> Optional[str]:
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR.
Rejects anything that isn't a bare filename or that escapes the voices
directory after symlink resolution. Returns None instead of raising so
cleanup paths can simply skip bad values.
"""
if not filename or os.path.basename(filename) != filename:
return None
root = os.path.realpath(VOICES_DIR)
path = os.path.realpath(os.path.join(root, filename))
if not path.startswith(root + os.sep):
return None
return path
@router.post("/profiles/{profile_id}/consent")
async def record_consent(
profile_id: str,
consent_audio: UploadFile = File(...),
consent_text: str = Form(...),
):
if not consent_text.strip():
raise HTTPException(status_code=422, detail="consent_text must not be empty")
data = await consent_audio.read()
if len(data) < _MIN_CONSENT_AUDIO_BYTES:
raise HTTPException(status_code=422, detail="consent recording is too short")
with db_conn() as conn:
row = conn.execute(
"SELECT id, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Profile not found")
ext = os.path.splitext(consent_audio.filename or "")[1]
if not _CONSENT_EXT_RE.match(ext):
ext = ".wav"
audio_filename = f"{profile_id}_consent{ext}"
audio_path = _voices_path(audio_filename)
if audio_path is None: # profile_id is server-generated; this is belt+braces
raise HTTPException(status_code=400, detail="Invalid profile id")
with open(audio_path, "wb") as f:
f.write(data)
# A re-record may change the extension; drop the superseded file.
old = row["consent_audio_path"]
if old and old != audio_filename:
old_path = _voices_path(old)
if old_path and os.path.exists(old_path):
os.remove(old_path)
recorded_at = time.time()
try:
with db_conn() as conn:
conn.execute(
"UPDATE voice_profiles SET verified_own_voice=1, consent_text=?, "
"consent_audio_path=?, consent_recorded_at=? WHERE id=?",
(consent_text.strip(), audio_filename, recorded_at, profile_id),
)
except Exception:
if os.path.exists(audio_path):
os.remove(audio_path)
raise
event_bus.emit("profiles", {"action": "consent_recorded", "id": profile_id})
return {
"id": profile_id,
"verified_own_voice": True,
"consent_recorded_at": recorded_at,
}
@router.delete("/profiles/{profile_id}/consent")
def revoke_consent(profile_id: str):
with db_conn() as conn:
row = conn.execute(
"SELECT consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Profile not found")
conn.execute(
"UPDATE voice_profiles SET verified_own_voice=0, consent_text='', "
"consent_audio_path='', consent_recorded_at=NULL WHERE id=?",
(profile_id,),
)
if row["consent_audio_path"]:
path = _voices_path(row["consent_audio_path"])
if path and os.path.exists(path):
os.remove(path)
event_bus.emit("profiles", {"action": "consent_revoked", "id": profile_id})
return {"id": profile_id, "verified_own_voice": False}
@router.delete("/profiles/{profile_id}")
def delete_profile(profile_id: str):
with db_conn() as conn:
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
row = conn.execute("SELECT ref_audio_path, locked_audio_path, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if row:
for col in ["ref_audio_path", "locked_audio_path"]:
for col in ["ref_audio_path", "locked_audio_path", "consent_audio_path"]:
if row[col]:
path = os.path.join(VOICES_DIR, row[col])
if os.path.exists(path):
path = _voices_path(row[col])
if path and os.path.exists(path):
os.remove(path)
# Prevent FOREIGN KEY constraint failure
conn.execute("UPDATE generation_history SET profile_id = NULL WHERE profile_id=?", (profile_id,))
+25
View File
@@ -2,6 +2,7 @@ import uuid
import time
import json
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from core.db import db_conn
from core import event_bus
@@ -9,6 +10,10 @@ from schemas.requests import ProjectSaveRequest
router = APIRouter()
class ProjectRenameRequest(BaseModel):
name: str
@router.get("/projects")
async def list_projects():
with db_conn() as conn:
@@ -59,6 +64,26 @@ async def update_project(project_id: str, req: ProjectSaveRequest):
event_bus.emit("projects", {"action": "updated", "id": project_id})
return {"id": project_id, "name": req.name, "updated_at": now}
@router.patch("/projects/{project_id}")
async def rename_project(project_id: str, req: ProjectRenameRequest):
"""Lightweight rename — updates only the project name (and updated_at),
without re-serialising the whole state blob like PUT does."""
name = req.name.strip()
if not name:
raise HTTPException(status_code=400, detail="Project name cannot be empty")
now = time.time()
with db_conn() as conn:
row = conn.execute("SELECT id FROM studio_projects WHERE id=?", (project_id,)).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Project not found")
conn.execute(
"UPDATE studio_projects SET name=?, updated_at=? WHERE id=?",
(name, now, project_id),
)
event_bus.emit("projects", {"action": "renamed", "id": project_id})
return {"id": project_id, "name": name, "updated_at": now}
@router.delete("/projects/{project_id}")
async def delete_project(project_id: str):
with db_conn() as conn:
+168
View File
@@ -123,6 +123,117 @@ def set_torch_compile_disabled(body: _TorchCompileBody):
return _torch_compile_state()
# ── Dictation refinement (parity program Wave 2.1 / Spec 3 phase 2) ───────
class _RefinementBody(BaseModel):
auto: bool | None = None
smart_cleanup: bool | None = None
self_correction: bool | None = None
preserve_technical: bool | None = None
def _refinement_state():
from services.refinement import get_refinement_config
from services.llm_backend import get_active_llm_backend
cfg = get_refinement_config()
# The UI shows whether refinement can actually run (needs an LLM).
cfg["llm_ready"] = get_active_llm_backend().id != "off"
return cfg
@router.get("/dictation-refinement")
def get_dictation_refinement():
"""Current refinement config + whether an LLM backend is configured."""
return _refinement_state()
@router.put("/dictation-refinement")
def set_dictation_refinement(body: _RefinementBody):
from services.refinement import set_refinement_config
try:
set_refinement_config({k: v for k, v in body.model_dump().items() if v is not None})
except Exception:
logger.exception("set_dictation_refinement failed")
raise HTTPException(status_code=500, detail="Failed to persist setting")
return _refinement_state()
# ── LLM endpoint (parity program Wave 2.4 / §R2 rung 4) ───────────────────
# Focused configuration for the OpenAI-compatible LLM endpoint that powers
# cinematic translate, glossary auto-extract, and dictation refinement.
# Persistence rides the existing TRANSLATE_BASE_URL / TRANSLATE_API_KEY /
# TRANSLATE_MODEL env vars (already in system.py PERSISTENT_KEYS, restored
# at startup) so the resolution path in llm_backend/translator is unchanged.
class _LLMEndpointBody(BaseModel):
base_url: str | None = None
model: str | None = None
api_key: str | None = None # None = leave unchanged; "" = clear
def _mask(secret: str | None) -> str | None:
if not secret:
return None
return f"{secret[-4:]}" if len(secret) > 4 else "set"
def _llm_endpoint_state():
from services.llm_backend import OpenAICompatBackend
ok, reason = OpenAICompatBackend.is_available()
return {
"base_url": os.environ.get("TRANSLATE_BASE_URL", ""),
"model": os.environ.get("TRANSLATE_MODEL", ""),
"api_key_masked": _mask(
os.environ.get("TRANSLATE_API_KEY") or os.environ.get("OPENAI_API_KEY")
),
"available": ok,
"reason": None if ok else reason,
}
@router.get("/llm-endpoint")
def get_llm_endpoint():
"""Current OpenAI-compatible LLM endpoint config + live availability."""
return _llm_endpoint_state()
@router.put("/llm-endpoint")
def set_llm_endpoint(body: _LLMEndpointBody):
"""Persist base URL / model / API key for the OpenAI-compatible endpoint.
Reuses the env-var persistence path (prefs.json, restored at startup):
base_url -> TRANSLATE_BASE_URL, model -> TRANSLATE_MODEL,
api_key -> TRANSLATE_API_KEY. A None field is left unchanged; an empty
string clears it. Ollama ignores the key; vLLM / LM Studio require it.
"""
from core.prefs import set_ as prefs_set, delete as prefs_delete
mapping = {
"TRANSLATE_BASE_URL": body.base_url,
"TRANSLATE_MODEL": body.model,
"TRANSLATE_API_KEY": body.api_key,
}
for env_key, val in mapping.items():
if val is None:
continue # untouched
val = val.strip()
if val:
os.environ[env_key] = val
prefs_set(f"env.{env_key}", val)
else:
os.environ.pop(env_key, None)
prefs_delete(f"env.{env_key}")
# get_active_llm_backend() builds a fresh backend (and its OpenAI client
# reads env at construction) on every call, so there's no singleton to
# invalidate — the next translate/refine picks up the new values.
return _llm_endpoint_state()
# ── License acceptance (Phase 3 Plan 03-01 / TTS-05) ──────────────────────
# Frontend ``SupertonicLicenseDialog`` flips the engine-license bit via this
# endpoint. The handler is loopback-gated (router-level dep) and the
@@ -284,3 +395,60 @@ def set_models_dir(body: _ModelsDirBody):
user_env.set_user_env(_MODELS_DIR_ENV, path)
return {"configured": path, "effective": _effective_models_dir(), "restart_required": True}
# ── HF mirror endpoint (parity program Wave 4.3 / §R4 c) ──────────────────
# Restricted-network users (e.g. behind the Great Firewall) need to point
# huggingface_hub at a mirror. HF reads HF_ENDPOINT at import time, so a
# change takes effect on the next backend start — persisted to the durable
# per-user env so it survives Tauri/Finder launches that don't inherit a
# shell. Loopback-gated via the router dep.
_HF_ENDPOINT_ENV = "HF_ENDPOINT"
# A few well-known mirrors, surfaced as quick-picks in the UI. hf-mirror.com
# is the community mirror most-used in China; the official endpoint clears it.
_HF_MIRROR_PRESETS = [
{"label": "Hugging Face (official)", "url": ""},
{"label": "hf-mirror.com (community, China)", "url": "https://hf-mirror.com"},
]
class _HFMirrorBody(BaseModel):
url: str = Field("", description="HF_ENDPOINT URL; empty string clears it (official endpoint)")
@router.get("/hf-mirror")
def get_hf_mirror():
from core import user_env
configured = user_env.get_user_env(_HF_ENDPOINT_ENV) or ""
return {
# The value that will apply after restart (persisted), and what's
# live in this process (env may differ until then).
"configured": configured,
"effective": os.environ.get(_HF_ENDPOINT_ENV, ""),
"presets": _HF_MIRROR_PRESETS,
}
@router.put("/hf-mirror")
def set_hf_mirror(body: _HFMirrorBody):
from core import user_env
url = (body.url or "").strip().rstrip("/")
if url and not url.startswith(("http://", "https://")):
raise HTTPException(status_code=400, detail="Mirror URL must start with http(s)://")
try:
if url:
user_env.set_user_env(_HF_ENDPOINT_ENV, url)
os.environ[_HF_ENDPOINT_ENV] = url # best-effort for new downloads this session
else:
user_env.unset_user_env(_HF_ENDPOINT_ENV)
os.environ.pop(_HF_ENDPOINT_ENV, None)
except Exception:
logger.exception("set_hf_mirror failed")
raise HTTPException(status_code=500, detail="Failed to persist mirror setting")
# HF endpoint is read at import time by huggingface_hub, so the override
# is only guaranteed once the backend restarts.
return {"configured": url, "restart_required": True, "presets": _HF_MIRROR_PRESETS}
+343 -2
View File
@@ -11,13 +11,16 @@ from __future__ import annotations
import asyncio
import json
import logging
import os
import sys
from fastapi import APIRouter, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from core import prefs
from utils import hf_progress
from utils import download_aggregator
from .models import KNOWN_MODELS, invalidate_cache
logger = logging.getLogger("omnivoice.setup.download")
@@ -26,6 +29,177 @@ router = APIRouter()
# Cooldown: prevent rapid re-install after a failure. Maps repo_id → last_fail_time.
_install_cooldowns: dict[str, float] = {}
_COOLDOWN_SECS = 60.0
# Evict cooldown entries older than this so the dict can't grow unbounded across
# a long-lived process (MM2-06). Anything past the cooldown window is dead state.
_COOLDOWN_TTL_SECS = 3600.0
def _sweep_cooldowns(now: float) -> None:
"""Drop cooldown entries older than the TTL (MM2-06). Keeps the dict bounded
without this it accumulated one entry per ever-failed repo forever."""
stale = [k for k, t in _install_cooldowns.items() if (now - t) > _COOLDOWN_TTL_SECS]
for k in stale:
_install_cooldowns.pop(k, None)
# Repo_ids the user asked to cancel (FDL-11). Checked between retry attempts.
# Note: a single in-flight snapshot_download/Xet fetch is not interruptible
# mid-file in hf_hub 1.7.2 — cancel stops further retries, marks the row
# cancelled, and clears the cooldown so a cancel isn't rate-limited.
_cancelled: set[str] = set()
def _download_max_workers() -> int:
"""Parallel-FILES worker count for snapshot_download (FDL-02). Default 8 —
don't crank it: Xet already parallelises *within* each file via concurrent
byte-range gets, so a high count just multiplies buffer pressure. Override
via prefs / OMNIVOICE_DOWNLOAD_MAX_WORKERS for power users."""
raw = prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8)
try:
return max(1, int(raw))
except (TypeError, ValueError):
return 8
def _download_endpoint() -> "str | None":
"""Optional HF endpoint override (FDL-10 mirror path, opt-in). Returned as a
per-call ``endpoint=`` rather than a process-wide HF_ENDPOINT mutation. A
mirror routes through the classic LFS path (no Xet) documented in
docs/downloading-models.md."""
ep = prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None)
return ep or None
def apply_xet_env() -> None:
"""Apply opt-in Xet tuning knobs to the environment before a download
(FDL-04). Both default OFF; env wins over the prefs store. high-performance
can *hurt* low-RAM machines (needs lots of RAM/bandwidth); HDD-sequential
avoids parallel-write thrash on spinning disks. Idempotent."""
import os as _os
high_perf = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
if _truthy(high_perf):
_os.environ["HF_XET_HIGH_PERFORMANCE"] = "1"
hdd_seq = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
if _truthy(hdd_seq):
_os.environ["HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY"] = "1"
def _truthy(v) -> bool:
if isinstance(v, bool):
return v
return str(v).strip().lower() in {"1", "true", "yes", "on"}
class _InstallCancelled(Exception):
"""Raised inside the install worker when the user cancels (FDL-11)."""
def compute_plan(plan_files) -> dict:
"""Summarise a snapshot_download(dry_run=True) result into the install_plan
payload (FDL-05): total bytes, bytes already cached (skipped), bytes that
will actually download, and file counts. ``will_download`` defaults to
``not is_cached`` for forward-compat with older DryRunFileInfo shapes."""
total = sum(int(getattr(f, "file_size", 0) or 0) for f in plan_files)
cached = sum(
int(getattr(f, "file_size", 0) or 0)
for f in plan_files if getattr(f, "is_cached", False)
)
will = [
f for f in plan_files
if getattr(f, "will_download", not getattr(f, "is_cached", False))
]
to_dl = sum(int(getattr(f, "file_size", 0) or 0) for f in will)
n_files = len(plan_files)
n_cached = sum(1 for f in plan_files if getattr(f, "is_cached", False))
return {
"total_bytes": total,
"cached_bytes": cached,
"to_download_bytes": to_dl,
"n_files": n_files,
"n_cached": n_cached,
}
def _segmented_enabled() -> bool:
"""Opt-in IDM-style accelerator (FDL-09), default OFF. Most useful when Xet
is inactive (the app's default): the legacy-LFS path is single-stream, so
this restores parallel speed AND gives real live byte progress."""
return _truthy(prefs.resolve(
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False,
))
def _xet_active() -> bool:
"""True only when hf_xet is installed AND not disabled. The app sets
HF_HUB_DISABLE_XET=1 by default, so this is normally False which is when
the segmented accelerator pays off."""
import importlib.util
if importlib.util.find_spec("hf_xet") is None:
return False
return os.environ.get("HF_HUB_DISABLE_XET", "").strip().lower() not in {"1", "true", "yes", "on"}
def _repo_cancelled(repo_id: str) -> bool:
return repo_id in _cancelled
def _segmented_snapshot(repo_id: str, *, endpoint: "str | None") -> str:
"""Fetch every file of a repo via the segmented downloader into the HF
cache, mirroring hf_hub_download's blob+snapshot+refs layout so the result
is indistinguishable from snapshot_download (FDL-09) keeping /models
install-state, is_cached, and delete working. Feeds real bytes to the
aggregator. Raises on any error; the caller falls back to snapshot_download.
"""
import asyncio as _asyncio
from huggingface_hub import HfApi, constants as _C
from huggingface_hub.file_download import (
hf_hub_url, get_hf_file_metadata, repo_folder_name, _create_symlink,
)
from services.segmented_download import segmented_download
from services.token_resolver import resolve as _resolve_token
token = _resolve_token()
api = HfApi(endpoint=endpoint, token=token)
info = api.repo_info(repo_id, repo_type="model")
commit = info.sha
files = [s.rfilename for s in (info.siblings or [])]
if not commit or not files:
raise RuntimeError("repo_info returned no commit/siblings")
repo_dir = os.path.join(_C.HF_HUB_CACHE, repo_folder_name(repo_id=repo_id, repo_type="model"))
blobs_dir = os.path.join(repo_dir, "blobs")
snap_dir = os.path.join(repo_dir, "snapshots", commit)
refs_dir = os.path.join(repo_dir, "refs")
for d in (blobs_dir, snap_dir, refs_dir):
os.makedirs(d, exist_ok=True)
for rel in files:
if _repo_cancelled(repo_id):
raise _InstallCancelled()
url = hf_hub_url(repo_id, rel, endpoint=endpoint, revision=commit)
meta = get_hf_file_metadata(url, token=token)
etag = (meta.etag or "").strip('"')
if not etag:
raise RuntimeError(f"no etag for {rel}")
blob_path = os.path.join(blobs_dir, etag)
pointer = os.path.join(snap_dir, rel)
os.makedirs(os.path.dirname(pointer), exist_ok=True)
if not os.path.exists(blob_path):
_asyncio.run(segmented_download(
meta.location or url, blob_path,
token=token, expected_size=meta.size, expected_etag=etag,
on_bytes=lambda d, k=rel: download_aggregator.add_bytes(repo_id, k, d),
cancel_check=lambda: _repo_cancelled(repo_id),
))
if not os.path.lexists(pointer):
_create_symlink(blob_path, pointer, new_blob=True)
# refs/main → commit so scan_cache_dir maps the revision correctly.
try:
with open(os.path.join(refs_dir, "main"), "w") as f:
f.write(commit)
except OSError:
pass
return snap_dir
# ── SSE Download Stream ───────────────────────────────────────────────────
@@ -42,6 +216,66 @@ def _safe_put(queue: asyncio.Queue, event) -> None:
pass
# Minimum size for "this snapshot actually contains model weights". An
# interrupted snapshot_download can leave config/tokenizer files but no
# weights; the install then looks complete and synthesis later fails with
# "does not appear to have a file named pytorch_model.bin or
# model.safetensors" (#352). 5 MB clears every weight format we ship
# (safetensors/bin shards, onnx, pt, gguf) without false-positiving on
# config-only aux repos.
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024
# Per-role weight-file floors (MM2-07). A valid model has at least one
# recognized weight file at or above its extension's floor. ONNX graphs are
# legitimately small (a complete model can be well under 5 MB), so a single
# 5 MB rule false-positives on them as "truncated" (#352 over-trigger); give
# .onnx a lower floor while still rejecting a 0/KB partial. Tensor formats keep
# the original 5 MB floor.
_WEIGHT_FLOORS = {
".safetensors": _MIN_WEIGHT_BYTES,
".bin": _MIN_WEIGHT_BYTES,
".ckpt": _MIN_WEIGHT_BYTES,
".pt": _MIN_WEIGHT_BYTES,
".pth": _MIN_WEIGHT_BYTES,
".gguf": _MIN_WEIGHT_BYTES,
".onnx": 64 * 1024, # a real ONNX graph is ≥ tens of KB; a truncated one is bytes
}
def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
"""Raise OSError when a finished snapshot has no plausible weight file —
surfaces the truncated-download class (#352) at install time, where the
retry loop and the UI's re-download path can deal with it, instead of at
first synthesis with an opaque transformers error.
A snapshot is valid if it contains a recognized weight file meeting its
per-extension floor (MM2-07) OR any file the global 5 MB floor (the
original lenient catch kept so this is never stricter than before)."""
try:
biggest = 0
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
for f in files:
try:
size = os.path.getsize(os.path.join(root, f))
except OSError:
continue
biggest = max(biggest, size)
ext = os.path.splitext(f)[1].lower()
floor = _WEIGHT_FLOORS.get(ext)
if floor is not None and size >= floor:
return # a recognized weight file of plausible size
if size >= _MIN_WEIGHT_BYTES:
return # original lenient catch (non-standard weight names)
except OSError:
return # can't inspect — don't block the install on the checker itself
raise OSError(
f"{repo_id}: download finished but no model weights were found in the "
"snapshot (largest file "
f"{biggest} bytes). The download was likely interrupted — delete the "
"model in Settings → Models and install it again."
)
@router.get("/setup/download-stream")
async def setup_download_stream():
"""SSE: forward every HuggingFace download tqdm update as a JSON event."""
@@ -98,6 +332,7 @@ async def install_model(req: InstallModelRequest):
)
# Cooldown guard — don't retry if the same model just failed.
import time as _time_check
_sweep_cooldowns(_time_check.time()) # bound the dict (MM2-06)
last_fail = _install_cooldowns.get(req.repo_id)
if last_fail and (_time_check.time() - last_fail) < _COOLDOWN_SECS:
remaining = int(_COOLDOWN_SECS - (_time_check.time() - last_fail))
@@ -112,6 +347,7 @@ async def install_model(req: InstallModelRequest):
def _do():
token = hf_progress.current_repo_id.set(req.repo_id)
_cancelled.discard(req.repo_id) # clear any stale cancel from a prior run
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
@@ -125,7 +361,22 @@ async def install_model(req: InstallModelRequest):
LocalEntryNotFoundError,
)
logger.info("model install starting: %s", req.repo_id)
dl_kwargs: dict = {"repo_id": req.repo_id}
# Apply opt-in Xet tuning knobs (high-perf / HDD) before downloading.
apply_xet_env()
# Drive snapshot_download explicitly (FDL-02): pass our progress-
# emitting tqdm subclass so progress is deterministic + Xet-aware
# (Xet feeds bytes into whatever tqdm_class is supplied), bound the
# parallel-files worker count, and honour an optional mirror endpoint.
dl_kwargs: dict = {
"repo_id": req.repo_id,
"max_workers": _download_max_workers(),
}
_tqdm_cls = hf_progress.tracked_tqdm_class()
if _tqdm_cls is not None:
dl_kwargs["tqdm_class"] = _tqdm_cls
_endpoint = _download_endpoint()
if _endpoint:
dl_kwargs["endpoint"] = _endpoint
if sys.platform == "win32":
dl_kwargs["local_dir_use_symlinks"] = False
@@ -153,12 +404,70 @@ async def install_model(req: InstallModelRequest):
hb = threading.Thread(target=_heartbeat, daemon=True)
hb.start()
# Pre-flight (FDL-05): a dry-run resolve gives the UI an accurate
# denominator — total bytes, bytes already cached (skipped), and the
# bytes that will actually download — BEFORE any byte flows. Seeds
# the overall aggregator so its bar/ETA are correct from the first
# event. Degrades gracefully (totals=None) on older/gated repos.
_preflight_kwargs = {"repo_id": req.repo_id, "dry_run": True}
if _endpoint:
_preflight_kwargs["endpoint"] = _endpoint
try:
_plan = snapshot_download(**_preflight_kwargs)
_summary = compute_plan(_plan)
download_aggregator.start(
req.repo_id,
total_bytes=_summary["to_download_bytes"],
files_total=max(0, _summary["n_files"] - _summary["n_cached"]),
)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"phase": "install_plan",
**_summary,
})
except Exception as _pf_err:
# No preflight (older/gated repo, mirror without dry-run, etc.):
# fall back to today's fill-in-as-files-appear behaviour.
logger.info("model install %s: preflight unavailable (%s)", req.repo_id, _pf_err)
download_aggregator.start(req.repo_id)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"phase": "install_plan",
"total_bytes": None,
"cached_bytes": None,
"to_download_bytes": None,
"n_files": None,
"n_cached": None,
})
_max_attempts = 5
_attempt = 0
while True:
if req.repo_id in _cancelled:
raise _InstallCancelled()
_attempt += 1
try:
snapshot_download(**dl_kwargs)
# Opt-in segmented accelerator (FDL-09): parallel byte-range
# fetch with real live progress, for the legacy-LFS path.
# Any failure falls through to snapshot_download — the
# accelerator can never compromise a correct install.
_snapshot_path = None
if _attempt == 1 and _segmented_enabled() and not _xet_active():
try:
_snapshot_path = _segmented_snapshot(req.repo_id, endpoint=_endpoint)
except _InstallCancelled:
raise
except Exception as _seg_err:
logger.info(
"segmented download for %s failed (%s); falling back to snapshot_download",
req.repo_id, _seg_err,
)
_snapshot_path = None
if _snapshot_path is None:
_snapshot_path = snapshot_download(**dl_kwargs)
_validate_snapshot_has_weights(req.repo_id, _snapshot_path)
break
except (HfHubHTTPError, LocalEntryNotFoundError, OSError) as net_err:
if _attempt >= _max_attempts:
@@ -179,6 +488,10 @@ async def install_model(req: InstallModelRequest):
_t.sleep(_backoff)
# Stop heartbeat once download completes
_resolving.set()
# Flush the overall bar to 100% with the true byte total (FDL-06):
# under Xet the per-file byte bars don't surface completion, so the
# aggregator can sit below 100% even though every file landed.
download_aggregator.complete(req.repo_id)
logger.info("model install done: %s", req.repo_id)
hf_progress.emit({
"repo_id": req.repo_id,
@@ -186,7 +499,19 @@ async def install_model(req: InstallModelRequest):
"downloaded": 0, "total": 0, "pct": 1.0,
"phase": "install_done",
})
_install_cooldowns.pop(req.repo_id, None) # success clears any cooldown (MM2-06)
invalidate_cache()
except _InstallCancelled:
_resolving.set()
logger.info("model install cancelled: %s", req.repo_id)
# A cancel is user intent, not a failure — don't set a cooldown.
_install_cooldowns.pop(req.repo_id, None)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_cancelled",
})
except Exception as e:
_resolving.set()
logger.info("model install failed for %s: %s", req.repo_id, e)
@@ -200,12 +525,28 @@ async def install_model(req: InstallModelRequest):
"error": str(e),
})
finally:
_cancelled.discard(req.repo_id)
download_aggregator.finish(req.repo_id)
hf_progress.current_repo_id.reset(token)
loop.create_task(asyncio.to_thread(_do))
return {"status": "install_started", "repo_id": req.repo_id}
@router.post("/models/install/cancel")
async def cancel_install(req: InstallModelRequest):
"""Request cancellation of an in-flight install (FDL-11).
Best-effort: stops further retry attempts and marks the row cancelled. A
single in-flight snapshot_download/Xet fetch isn't interruptible mid-file
in hf_hub 1.7.2, so an already-streaming file finishes; the cancel takes
effect at the next retry boundary. Clears the cooldown so the user can
immediately restart."""
_cancelled.add(req.repo_id)
_install_cooldowns.pop(req.repo_id, None)
return {"cancelling": req.repo_id}
# ── Delete ─────────────────────────────────────────────────────────────────
@router.delete("/models/{repo_id:path}")
+6 -2
View File
@@ -220,8 +220,12 @@ def is_cached(repo_id: str) -> bool:
except Exception as e:
# scan_cache_dir can raise on Windows (WinError 448 'untrusted mount
# point'); fall back to a direct disk check so a cached model isn't
# mistaken for missing and re-downloaded in a loop (#117/#118).
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
# mistaken for missing and re-downloaded in a loop (#117/#118). Logged
# at WARNING with the exception type (MM2-09) so this fallback isn't
# invisible when triaging a Windows cache report — it previously logged
# at DEBUG and never showed at the default level.
logger.warning("is_cached: scan_cache_dir failed (%s: %s); using on-disk fallback for %s",
type(e).__name__, e, repo_id)
return _is_cached_on_disk(repo_id)
+44
View File
@@ -373,6 +373,46 @@ def preflight():
"status": gpu_status, "detail": gpu_detail, "fix": gpu_fix,
})
# ── GPU routing for the ACTIVE TTS engine (#21 — no silent CPU fallback).
# Distinct from the hardware "gpu" check above: this asks "will the engine
# the user actually selected use that GPU on this host?" Built from the same
# canonical probe + resolver the Engine Compatibility Matrix uses.
try:
from services.tts_backend import gpu_routing_verdict
gpu_routing = gpu_routing_verdict()
except Exception as exc: # never break preflight on a routing hiccup
logger.warning("preflight gpu_routing failed: %s", exc)
gpu_routing = None
if gpu_routing:
_rs = gpu_routing.get("routing_status")
_eng = gpu_routing.get("engine") or "active engine"
_dev = gpu_routing.get("effective_device") or "?"
_why = gpu_routing.get("routing_reason")
if _rs == "accelerated" and not _why:
r_status, r_detail, r_fix = "pass", f"{_eng}{_dev} (accelerated)", None
elif _rs == "accelerated": # driver/arch caveat
r_status, r_detail, r_fix = "warn", f"{_eng}{_dev}: {_why}", (
"GPU selected but may fail at kernel launch — update drivers / "
"reinstall torch for this GPU architecture.")
elif _rs == "cpu_fallback":
r_status, r_detail, r_fix = "warn", (
f"{_eng} runs on CPU here: {_why or 'no GPU path for this host'}"), (
"Pick an engine that supports this host's GPU for a speedup, or "
"continue on CPU (slower).")
elif _rs == "cpu_only":
r_status, r_detail, r_fix = "pass", f"{_eng} → cpu (no accelerator on this host)", None
elif _rs == "unavailable":
r_status, r_detail, r_fix = "fail", (
f"{_eng} can't run on this host: {_why or 'needs a GPU this machine lacks'}"), (
"Select an engine with a CPU path in Settings → Engines.")
else: # "none" / unknown
r_status, r_detail, r_fix = "warn", "No active TTS engine resolved for routing.", (
"Pick an engine in Settings → Engines.")
checks.append({
"id": "gpu_routing", "label": "Active engine routing",
"status": r_status, "detail": r_detail, "fix": r_fix,
})
# ── Network
net_ok = _probe_network()
checks.append({
@@ -400,9 +440,13 @@ def preflight():
"gpu_available": gpu["available"],
"gpu_driver": gpu["driver"],
"gpu_device_name": gpu["device_name"],
# Canonical probe (distinguishes ROCm from CUDA):
"gpu_family": (gpu_routing or {}).get("host_family", "cpu"),
"vram_gb": (gpu_routing or {}).get("vram_gb", 0.0),
"ram_gb": round(ram, 1),
"disk_free_gb": round(free, 1),
},
"gpu_routing": gpu_routing,
}
+120 -85
View File
@@ -107,6 +107,52 @@ def _ui_port() -> int:
return 3901
def _fast_download_status() -> dict:
"""Report the download-acceleration state for the Settings UI (FDL-03).
Reports the *runtime* truth, not just whether hf_xet is importable. The app
currently sets ``HF_HUB_DISABLE_XET=1`` by default (main.py) Xet's chunked
transfer is fast but its progress bypasses our tqdm patch, so the legacy-LFS
path is forced to keep accurate byte progress. So:
* ``xet_installed`` hf_xet present
* ``xet_active`` installed AND not disabled via HF_HUB_DISABLE_XET
* ``xet_enabled`` alias of xet_active (what the UI badge keys off)
Must never throw: /system/info is called on every Settings load.
"""
installed = False
version = None
try:
import hf_xet # noqa: F401
installed = True
try:
from importlib.metadata import version as _ver
version = _ver("hf-xet")
except Exception:
version = None
except Exception:
installed = False
disabled = str(os.environ.get("HF_HUB_DISABLE_XET", "")).strip().lower() in {"1", "true", "yes", "on"}
active = installed and not disabled
try:
from core import prefs
high_perf = prefs.resolve(
"xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False
)
high_perf = high_perf if isinstance(high_perf, bool) else \
str(high_perf).strip().lower() in {"1", "true", "yes", "on"}
except Exception:
high_perf = False
return {
"xet_installed": installed,
"xet_active": active,
"xet_enabled": active, # UI badge: only true when Xet actually runs
"xet_version": version,
"high_performance": bool(high_perf),
}
def _has_hf_token() -> bool:
# Phase 1 AUTH-01..06 cascade. Delegates to the 3-source resolver
# (App → Env → HF-CLI) instead of reading env/HF-CLI directly. This
@@ -128,88 +174,23 @@ def model_status():
@router.get("/model/loaded")
def loaded_models():
"""Return details about all currently loaded models for the flush dropdown.
Returns a list of models with name, type, device, and estimated VRAM usage.
"""
import services.model_manager as mm
models = []
# 1. TTS model (OmniVoice)
if mm.model is not None:
device = "unknown"
vram_mb = 0
try:
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, 'parameters') else get_best_device()
except Exception:
device = get_best_device()
try:
torch = mm._lazy_torch()
if torch.cuda.is_available():
vram_mb = torch.cuda.memory_allocated() / (1024 ** 2)
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
driver = getattr(torch.mps, "driver_allocated_memory", None)
if driver:
vram_mb = driver() / (1024 ** 2)
except Exception:
pass
models.append({
"id": "tts",
"name": "OmniVoice TTS",
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
"device": device,
"vram_mb": round(vram_mb, 1),
"unloadable": True,
})
# 2. ASR model (WhisperX)
if mm.model is not None and hasattr(mm.model, '_asr_pipe') and mm.model._asr_pipe is not None:
models.append({
"id": "asr",
"name": "WhisperX ASR",
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"device": "cpu",
"vram_mb": 0,
"unloadable": False, # tied to TTS model lifecycle
})
# 3. Diarization pipeline
if mm._diar_pipeline is not None:
models.append({
"id": "diarization",
"name": "Pyannote Diarization",
"checkpoint": "pyannote/speaker-diarization-3.1",
"device": get_best_device(),
"vram_mb": 0,
"unloadable": True,
})
return {"models": models, "count": len(models)}
"""List all currently loaded models for the flush dropdown (MM2-04).
Thin delegation to the model_lifecycle facade shape unchanged:
``{models, count}``."""
from services import model_lifecycle
return model_lifecycle.list_loaded()
@router.post("/model/unload/{model_id}")
async def unload_model(model_id: str):
"""Unload a specific model by ID."""
import services.model_manager as mm
if model_id == "tts":
async with mm._model_lock:
if mm.model is not None:
mm.model = None
mm.free_vram()
return {"unloaded": "tts", "success": True}
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
elif model_id == "diarization":
if mm._diar_pipeline is not None:
mm._diar_pipeline = None
mm.free_vram()
return {"unloaded": "diarization", "success": True}
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
else:
raise HTTPException(status_code=400, detail=f"Unknown model id: {model_id}")
"""Unload a specific model by id (MM2-04). Delegates to model_lifecycle;
an unknown id maps to HTTP 400. ``tts`` | ``diarization`` |
``sidecar:<id>`` | ``sidecars``."""
from services import model_lifecycle
try:
return await model_lifecycle.unload(model_id)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.get("/system/info", response_model=SystemInfoResponse)
@@ -231,6 +212,7 @@ def system_info():
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
"has_hf_token": _has_hf_token(),
"fast_download": _fast_download_status(),
"device": get_best_device(),
"python": sys.version.split()[0],
"platform": sys.platform,
@@ -322,9 +304,10 @@ def _tauri_log_candidates():
os.path.join(home, "Library/Logs/OmniVoice/backend_err.log"),
]
if sys.platform.startswith("linux"):
data_dir = os.environ.get("XDG_DATA_HOME") or os.path.join(home, ".local/share")
state_dir = os.environ.get("XDG_STATE_HOME") or os.path.join(home, ".local/state")
return [
os.path.join(home, ".local/share", bid, "logs", "tauri.log"),
os.path.join(data_dir, bid, "logs", "tauri.log"),
os.path.join(home, ".config", bid, "logs", "tauri.log"),
os.path.join(state_dir, "OmniVoice", "backend.log"),
os.path.join(state_dir, "OmniVoice", "backend_err.log"),
@@ -464,6 +447,14 @@ async def clear_system_logs():
status_code=500,
detail=f"Could not clear log at {p}: {e}. The file may be open in another process or read-only — close tailing tools and retry.",
)
if cleared_any:
# The crash log just shrank to zero — drop any stored ack so a stale
# byte count can't suppress the next 'crash-last-session' notice.
for key in ("crash_log_acked", "crash_log_acked_size"):
try:
prefs_delete(key)
except Exception:
pass
return {"cleared": cleared_any}
@@ -565,6 +556,21 @@ async def flush_memory(unload_model: bool = False):
# ── Actionable notifications ──────────────────────────────────────────────
_GPU_ARCH_WARNING: "list[str | None]" = [] # [-1] = computed result
def _gpu_arch_warning_cached() -> "str | None":
"""check_device_compatibility() once per process (it lazy-imports torch —
too heavy for the 30s notifications poll)."""
if not _GPU_ARCH_WARNING:
try:
from services.model_manager import check_device_compatibility
compatible, warning = check_device_compatibility()
_GPU_ARCH_WARNING.append(None if compatible else warning)
except Exception:
_GPU_ARCH_WARNING.append(None)
return _GPU_ARCH_WARNING[-1]
@router.get("/system/notifications")
def system_notifications():
@@ -596,6 +602,20 @@ def system_notifications():
},
})
# 1b. GPU compute capability unsupported by this torch build (#284) —
# the model "runs" but emits pure noise, the worst silent failure mode
# (RTX 50-series Blackwell sm_120 on pre-cu128 wheels). The loader logs
# this, but a log line never reached the affected users — surface it in
# the panel. Checked once per process: it lazy-imports torch.
gpu_warn = _gpu_arch_warning_cached()
if gpu_warn:
notes.append({
"id": "gpu-arch-unsupported",
"level": "error",
"title": "GPU not supported by this PyTorch build",
"message": gpu_warn + " Until then, output will be noise/garbage.",
})
# 2. Missing ffmpeg
ffmpeg_ok = False
try:
@@ -686,18 +706,33 @@ def _crashed_last_session() -> bool:
if not os.path.exists(CRASH_LOG_PATH):
return False
size = os.path.getsize(CRASH_LOG_PATH)
acked = int(prefs_get("crash_log_acked_size", 0) or 0)
if size <= acked:
if size == 0:
return False
return os.path.getmtime(CRASH_LOG_PATH) < _PROCESS_START_TS
mtime = os.path.getmtime(CRASH_LOG_PATH)
# Composite ack (size + mtime): a bare byte count goes stale after the log
# is truncated — the next crash log can stay smaller than the old acked
# size forever, silently suppressing 'crash-last-session'. The ack only
# holds while it still covers the file's current state.
ack = prefs_get("crash_log_acked")
if isinstance(ack, dict):
if float(ack.get("mtime", 0) or 0) >= mtime and int(ack.get("size", 0) or 0) >= size:
return False
else:
# Legacy size-only ack from older builds.
if size <= int(prefs_get("crash_log_acked_size", 0) or 0):
return False
return mtime < _PROCESS_START_TS
@router.post("/system/crash/ack")
async def ack_crash():
"""Mark the current crash log as seen — dismisses the
'crash-last-session' notification until the log grows again."""
size = os.path.getsize(CRASH_LOG_PATH) if os.path.exists(CRASH_LOG_PATH) else 0
prefs_set("crash_log_acked_size", size)
'crash-last-session' notification until the log changes again."""
size = mtime = 0
if os.path.exists(CRASH_LOG_PATH):
size = os.path.getsize(CRASH_LOG_PATH)
mtime = os.path.getmtime(CRASH_LOG_PATH)
prefs_set("crash_log_acked", {"size": size, "mtime": mtime})
return {"acked_size": size}
+81 -32
View File
@@ -98,6 +98,30 @@ async def ws_tts(websocket: WebSocket):
model = await get_model()
backend = get_active_tts_backend(model=model)
# ── Routing gate (#21 — no silent CPU fallback). WebSockets have
# no response headers, so this uses frames: an error frame +
# close on `unavailable`, a one-time `routing` frame on
# cpu_fallback / accelerated-with-caveat (before any audio).
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
from core.scrub import scrub_text
_routing = resolve_routing(
getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
await websocket.send_json({
"type": "error",
"detail": scrub_text(_routing["routing_reason"])
or "engine cannot run on this host",
})
continue # don't stream; wait for the next request
_notice = routing_notice(_routing)
if _notice:
await websocket.send_json({
"type": "routing",
"status": _notice[0],
"reason": scrub_text(_notice[1]) if _notice[1] else None,
})
# Build generation kwargs
kw: dict = {"speed": data.get("speed", 1.0)}
if data.get("language"):
@@ -144,50 +168,75 @@ async def ws_tts(websocket: WebSocket):
except Exception:
kw["voice"] = voice
# Wave 1.4: split the request into sentences so the first
# sentence's audio streams while later sentences are still
# synthesizing — this is the time-to-first-audio win. The
# chunker handles abbreviations/acronyms/decimals and CJK /
# non-Latin terminators; single-sentence requests behave
# exactly like the old single-shot path.
from services.sentence_chunker import SentenceChunker
_chunker = SentenceChunker(language=(data.get("language") or "en"))
sentences = _chunker.push(text)
sentences.extend(_chunker.flush())
if not sentences:
sentences = [text]
# Run generation in the GPU pool
from services.model_manager import _gpu_pool
loop = asyncio.get_running_loop()
def _generate():
def _generate(sentence_text):
from services.audio_dsp import apply_mastering, normalize_audio
wav = backend.generate(text, **kw)
wav = backend.generate(sentence_text, **kw)
sr_actual = backend.sample_rate
wav = apply_mastering(wav, sample_rate=sr_actual)
# Like _run_tts in openai_compat: studio engines (VoxCPM2)
# opt out of the broadcast mastering chain. This is the
# other route that runs the active backend, so it needs the
# same guard. Loudness normalisation still runs.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr_actual)
wav = normalize_audio(wav, target_dBFS=-2.0)
return wav, sr_actual
wav_tensor, sr = await loop.run_in_executor(_gpu_pool, _generate)
# Send metadata after generation so sample_rate is real
await websocket.send_json({
"type": "start",
"sample_rate": sr,
"channels": 1,
"format": "pcm16",
"engine": backend.id,
})
# Stream PCM16 chunks over the WebSocket
import torch
# Convert to 16-bit PCM
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
if pcm.ndim == 2:
pcm = pcm[0] # mono
pcm_bytes = pcm.numpy().tobytes()
total_samples = 0
sr = backend.sample_rate
started = False
total_samples = len(pcm)
sent_samples = 0
chunk_bytes = CHUNK_SAMPLES * 2 # 2 bytes per int16 sample
for sentence in sentences:
wav_tensor, sr = await loop.run_in_executor(
_gpu_pool, _generate, sentence
)
while sent_samples < total_samples:
end = min(sent_samples + CHUNK_SAMPLES, total_samples)
start_byte = sent_samples * 2
end_byte = end * 2
chunk = pcm_bytes[start_byte:end_byte]
await websocket.send_bytes(chunk)
sent_samples = end
# Yield to event loop between chunks for responsiveness
await asyncio.sleep(0)
if not started:
# Send metadata after the first generation so
# sample_rate is real (lazy-loading engines report
# their true rate only once weights are up).
await websocket.send_json({
"type": "start",
"sample_rate": sr,
"channels": 1,
"format": "pcm16",
"engine": backend.id,
})
started = True
# Convert to 16-bit PCM and stream
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
if pcm.ndim == 2:
pcm = pcm[0] # mono
pcm_bytes = pcm.numpy().tobytes()
n_samples = len(pcm)
sent_samples = 0
while sent_samples < n_samples:
end = min(sent_samples + CHUNK_SAMPLES, n_samples)
chunk = pcm_bytes[sent_samples * 2: end * 2]
await websocket.send_bytes(chunk)
sent_samples = end
# Yield to event loop between chunks for responsiveness
await asyncio.sleep(0)
total_samples += n_samples
gen_time = round(time.perf_counter() - t0, 3)
duration = round(total_samples / sr, 3)
+27
View File
@@ -34,6 +34,8 @@ class SystemInfoResponse(BaseModel):
asr_model: str = "unknown"
translate_provider: str = "unknown"
has_hf_token: bool = False
# Xet fast-download backend state (FDL-03): {xet_enabled, xet_version, high_performance}
fast_download: dict | None = None
device: str = "cpu"
python: str = ""
platform: str = ""
@@ -127,16 +129,41 @@ class DeviceInfo(BaseModel):
gpu_available: bool = False
gpu_driver: str | None = None
gpu_device_name: str | None = None
# From the canonical device probe (core.device_caps) — distinguishes ROCm
# from CUDA, unlike the legacy nvidia-smi-based gpu_vendor/gpu_backend.
gpu_family: str = "cpu"
vram_gb: float = 0.0
ram_gb: float = 0.0
disk_free_gb: float = 0.0
class GpuRouting(BaseModel):
"""Routing verdict for the active TTS engine on THIS host (#21).
Distinct from the per-engine `routing_*` keys in `/engines`: this is the
single verdict for the *currently-selected* engine, surfaced in preflight +
diagnose so the user hears about a CPU fallback / unavailable GPU before a
slow or failed synth no silent CPU fallback.
"""
model_config = ConfigDict(extra="allow")
engine: str | None = None # active TTS engine id
effective_device: str | None = None # device it will actually use here
routing_status: str | None = None # accelerated|cpu_fallback|cpu_only|unavailable|none
routing_reason: str | None = None # scrubbed; null when none
host_family: str = "cpu" # detect_host_caps().family
vram_gb: float = 0.0
class PreflightResponse(BaseModel):
"""GET /setup/preflight"""
ok: bool
has_warnings: bool = False
checks: list[PreflightCheck] = Field(default_factory=list)
device: DeviceInfo
# Explicit field (PreflightResponse has no extra="allow") so the verdict
# survives serialization instead of being silently dropped.
gpu_routing: GpuRouting | None = None
class InstallModelRequest(BaseModel):
+19
View File
@@ -49,6 +49,12 @@ _BASE_SCHEMA = """
personality TEXT DEFAULT '',
description TEXT DEFAULT '',
is_demo INTEGER DEFAULT 0,
verified_own_voice INTEGER DEFAULT 0,
consent_text TEXT DEFAULT '',
consent_audio_path TEXT DEFAULT '',
consent_recorded_at REAL DEFAULT NULL,
kind TEXT DEFAULT 'clone',
vd_states TEXT DEFAULT NULL,
created_at REAL
);
CREATE TABLE IF NOT EXISTS generation_history (
@@ -138,6 +144,19 @@ _BASE_SCHEMA = """
value TEXT NOT NULL,
updated_at REAL NOT NULL
);
-- Wave 2.2: per-agent MCP voice bindings. An MCP client (Claude Code,
-- Cursor, ) identified by the X-OmniVoice-Client-Id header it sends is
-- bound to a default voice profile / engine. Fresh installs create it
-- here; v0.3.x upgrades get it via alembic 0004.
CREATE TABLE IF NOT EXISTS mcp_client_bindings (
client_id TEXT PRIMARY KEY,
label TEXT NOT NULL DEFAULT '',
profile_id TEXT,
default_engine TEXT,
last_seen_at REAL,
created_at REAL
);
"""
# Only tables/columns this module is allowed to ALTER. Prevents SQL injection via
+336
View File
@@ -0,0 +1,336 @@
"""Free-text voice-description → voice-design parameter mapper (issue #317).
Parity with the hosted omnivoice.app "Describe your voice" field, implemented
fully locally: a deterministic keyword/phrase mapper that projects a natural-
language description (e.g. ``"a warm elderly British storyteller, slightly
raspy"``) onto the **existing** voice-design parameter space — the same six
categories the Design tab's attribute picker drives (Gender / Age / Pitch /
Style / EnglishAccent / ChineseDialect).
Design notes
============
* **No model, no network.** This is an ordered synonym-table matcher, not an
LLM call it runs identically on macOS/Windows/Linux with zero deps beyond
the stdlib, preserving the local-first guarantee.
* **Single source of truth.** Every canonical token this module can emit is
validated at import time against the engine taxonomy in
``omnivoice/utils/voice_design.py`` (loaded via ``core.archetypes``), so the
mapper can never produce an instruct item the engine validator would reject
(the issue-#89 / #115 crash modes). The Chinese translations of each token
(e.g. ````/``中年``) are *derived* from that taxonomy, never hardcoded.
* **Ordered rules, first match wins.** Within a category, rules are checked in
a hand-ordered list so more specific phrases outrank generic ones
("young child" child, not young adult; "very deep" very low pitch, not
low pitch). Within one rule, the earliest occurrence in the text is reported
as the matched phrase. Deterministic by construction.
* **Graceful degradation.** Anything the taxonomy can't express (timbre words
like "raspy", role words like "storyteller") is returned in ``unmatched`` so
the UI can tell the user exactly which parts were ignored instead of failing
silently (issue #317's validation-feedback note). A description with no
matches at all yields all-``Auto`` attrs and an empty instruct.
Localization note (CLAUDE.md): the only hardcoded CJK here is
``DIALECT_PINYIN`` a functional pinyin Chinese-dialect-token mapping
(model vocabulary, like ``frontend/src/utils/constants.js``). Registered in
``tests/test_no_hardcoded_cjk.py``'s allowlist with this justification.
"""
from __future__ import annotations
import re
# Reuse the taxonomy already loaded (stdlib-only, by file path) by the
# archetype engine — same single source of truth, one loader to maintain.
from core.archetypes import _VD
_EN_TO_ZH = _VD._INSTRUCT_EN_TO_ZH # {"male": "男", ...}
_ZH_RE = _VD._ZH_RE
_VALID = _VD._INSTRUCT_ALL_VALID # every token the engine accepts
_DIALECTS = set(_VD._INSTRUCT_CATEGORIES[5]) # the 12 Chinese dialect tokens
# Category names match the frontend's CATEGORIES keys (utils/constants.js) and
# the archetype ``attrs`` shape, so the response drops straight into vdStates.
CATEGORY_ORDER = ("Gender", "Age", "Pitch", "Style", "EnglishAccent", "ChineseDialect")
# ── Pinyin / romanized names → Chinese-dialect tokens (functional vocabulary) ─
DIALECT_PINYIN = {
"henan": "河南话",
"shaanxi": "陕西话",
"sichuan": "四川话",
"szechuan": "四川话",
"guizhou": "贵州话",
"yunnan": "云南话",
"guilin": "桂林话",
"jinan": "济南话",
"shijiazhuang": "石家庄话",
"gansu": "甘肃话",
"ningxia": "宁夏话",
"qingdao": "青岛话",
"dongbei": "东北话",
"northeastern chinese": "东北话",
}
# ── Synonym tables ────────────────────────────────────────────────────────────
# Per category: ordered list of (canonical_token, [phrases]). First rule with
# any hit wins the category, so specific phrases must precede generic ones.
# Each canonical token's Chinese translation from the taxonomy is appended
# automatically at compile time (so "中年" maps to "middle-aged", etc.).
_GENDER_RULES = [
("female", [
"female", "woman", "women", "lady", "ladies", "girl", "girls",
"feminine", "gal", "grandma", "grandmother", "granny", "mother",
"mom", "mum", "aunt", "auntie", "queen", "princess", "actress",
"she", "her",
]),
("male", [
"male", "man", "men", "guy", "guys", "boy", "boys", "masculine",
"gentleman", "gentlemen", "dude", "grandpa", "grandfather", "father",
"dad", "uncle", "king", "prince", "actor", "he", "him", "his",
]),
]
# Order is load-bearing: "child" precedes "young adult" so "young child" →
# child; "middle-aged" precedes "elderly" so elderly's bare "aged" synonym
# can't fire inside the hyphenated "middle-aged" (hyphen is a \b boundary);
# "elderly" precedes "young adult" so grandparent words don't fall through.
_AGE_RULES = [
("child", [
"child", "children", "kid", "kiddo", "toddler", "little boy",
"little girl", "young boy", "young girl", "small child", "childlike",
]),
("teenager", ["teenager", "teen", "teenage", "adolescent"]),
("middle-aged", [
"middle-aged", "middle aged", "middle age", "midlife", "forties",
"fifties", "sixties", "mature",
]),
("elderly", [
"elderly", "old man", "old woman", "old lady", "older man",
"older woman", "elder", "senior", "aged", "grandpa", "grandfather",
"grandma", "grandmother", "granny", "retired", "seventies",
"eighties", "nineties", "old",
]),
("young adult", [
"young adult", "young woman", "young man", "young lady", "youthful",
"twenties", "thirties", "college", "young",
]),
]
# "very …" rules precede their plain counterparts so "very deep" doesn't stop
# at "deep". Bare "low"/"high" only count next to a voice word (pitch/voice/
# tone/register) to avoid false hits like "high quality" or "low effort".
_PITCH_RULES = [
("very low pitch", [
"very low pitch", "very low-pitched", "very low pitched",
"very low voice", "very low tone", "very deep", "extremely deep",
"extremely low", "ultra deep", "booming",
]),
("very high pitch", [
"very high pitch", "very high-pitched", "very high pitched",
"very high voice", "very high tone", "extremely high", "squeaky",
"shrill", "falsetto", "chipmunk",
]),
("low pitch", [
"low pitch", "low-pitched", "low pitched", "low voice", "low tone",
"low register", "deep", "deeper", "bass", "baritone", "husky",
]),
("high pitch", [
"high pitch", "high-pitched", "high pitched", "high voice",
"high tone", "high register", "soprano",
]),
("moderate pitch", [
"moderate pitch", "medium pitch", "medium-pitched", "medium pitched",
"mid-range", "midrange", "average pitch", "moderate",
]),
]
_STYLE_RULES = [
("whisper", [
"whisper", "whispering", "whispered", "whispery", "hushed",
"breathy", "soft-spoken", "soft spoken",
]),
]
# Bare "english" means the language, so only the explicit "english accent"
# phrase maps to british. "chinese" maps to the chinese *accent* (English
# speech with a Chinese accent); actual dialect words live in DIALECT_PINYIN.
_ACCENT_RULES = [
("american accent", [
"american", "america", "usa", "us accent", "midwestern",
"californian", "new york",
]),
("british accent", [
"british", "britain", "english accent", "england", "uk accent",
"london", "cockney", "posh", "received pronunciation",
]),
("australian accent", ["australian", "australia", "aussie"]),
("canadian accent", ["canadian", "canada"]),
("indian accent", ["indian", "india"]),
("chinese accent", ["chinese accent", "chinese-accented", "chinese"]),
("korean accent", ["korean", "korea"]),
("japanese accent", ["japanese", "japan"]),
("portuguese accent", ["portuguese", "portugal", "brazilian", "brazil"]),
("russian accent", ["russian", "russia"]),
]
_DIALECT_RULES = [
(token, [pinyin for pinyin, tok in DIALECT_PINYIN.items() if tok == token])
for token in sorted(_DIALECTS)
]
_RULES = {
"Gender": _GENDER_RULES,
"Age": _AGE_RULES,
"Pitch": _PITCH_RULES,
"Style": _STYLE_RULES,
"EnglishAccent": _ACCENT_RULES,
"ChineseDialect": _DIALECT_RULES,
}
# Import-time guard: every canonical token must be in the engine taxonomy, so
# a taxonomy rename upstream fails loudly here instead of at synthesis time.
for _cat_rules in _RULES.values():
for _token, _ in _cat_rules:
assert _token in _VALID, f"describe_voice token not in taxonomy: {_token!r}"
for _tok in DIALECT_PINYIN.values():
assert _tok in _DIALECTS, f"DIALECT_PINYIN value not a taxonomy dialect: {_tok!r}"
# ── Pattern compilation ───────────────────────────────────────────────────────
def _compile_phrase(phrase: str) -> re.Pattern:
"""Compile a synonym phrase to a regex.
Latin phrases get word boundaries (so "male" never fires inside "female",
"old" never inside "bold") and flexible separators (space or hyphen, so
"middle aged" also matches "middle-aged"). CJK phrases match as plain
substrings word boundaries are meaningless without spaces.
"""
if _ZH_RE.search(phrase):
return re.compile(re.escape(phrase))
parts = [re.escape(p) for p in re.split(r"[ -]+", phrase) if p]
return re.compile(r"\b" + r"[\s\-]+".join(parts) + r"\b")
def _compiled_rules():
out = {}
for cat, rules in _RULES.items():
compiled = []
for token, phrases in rules:
pats = list(phrases)
# Derive the Chinese form of each canonical token from the
# taxonomy (e.g. "middle-aged" → "中年") — never hardcoded here.
zh = _EN_TO_ZH.get(token)
if zh:
pats.append(zh)
if token not in pats:
pats.append(token) # the canonical token always matches itself
compiled.append((token, [_compile_phrase(p) for p in pats]))
out[cat] = compiled
return out
_COMPILED = _compiled_rules()
# "<N> year(s) old / <N>-year-old / <N> yo" → an age bracket. Runs before the
# keyword rules so the trailing "old" never misfires as elderly.
_AGE_NUM = re.compile(
r"\b(\d{1,3})(?:[\s\-]*(?:years?|yrs?|yr)[\s\-]*old|[\s\-]*(?:yo|y/o))\b"
)
def _age_token_for(years: int) -> str:
if years <= 12:
return "child"
if years <= 19:
return "teenager"
if years <= 39:
return "young adult"
if years <= 64:
return "middle-aged"
return "elderly"
def _normalize(description: str) -> str:
text = (description or "").lower()
text = text.replace("", "'").replace("", "'")
text = text.replace("", '"').replace("", '"')
return re.sub(r"[ \t]+", " ", text)
def _match_category(category: str, text: str):
"""Return (token, match) for the first rule with a hit, else None.
Rule order decides the winning token; within the winning rule the earliest
occurrence in the text is reported as the matched phrase.
"""
if category == "Age":
m = _AGE_NUM.search(text)
if m:
return _age_token_for(int(m.group(1))), m
for token, patterns in _COMPILED[category]:
best = None
for pat in patterns:
m = pat.search(text)
if m is not None and (best is None or m.start() < best.start()):
best = m
if best is not None:
return token, best
return None
# Fragment splitter for the "unmatched" report: clause separators (incl. the
# CJK comma/ideographic stop, which CJK descriptions use instead of ASCII).
_FRAGMENT = re.compile(r"[^,;.!?()\n,。;!?、]+")
_HAS_CONTENT = re.compile(r"[\w一-鿿]")
def parse_description(description: str) -> dict:
"""Map a free-text voice description onto the design parameter space.
Returns a dict with:
* ``attrs`` full category token map (``"Auto"`` where nothing
matched); same shape as the Design tab's ``vdStates``.
* ``instruct`` validator-safe instruct string built from the matched
tokens, in canonical category order (may be ``""``).
* ``matched`` list of ``{category, token, phrase}`` for transparency.
* ``unmatched`` clause fragments that contributed no attribute, so the
UI can show what was ignored instead of failing silently.
"""
text = _normalize(description)
attrs = {cat: "Auto" for cat in CATEGORY_ORDER}
matched = []
spans = []
for category in CATEGORY_ORDER:
hit = _match_category(category, text)
if hit is None:
continue
token, m = hit
attrs[category] = token
matched.append({"category": category, "token": token, "phrase": m.group(0)})
spans.append((m.start(), m.end()))
# Accents are English-only and dialects Chinese-only in the engine
# taxonomy; a dialect voice speaks Chinese, so an accent token alongside
# it is contradictory (the issue-#114 conflict class). Dialect wins.
if attrs["ChineseDialect"] != "Auto" and attrs["EnglishAccent"] != "Auto":
dropped = attrs["EnglishAccent"]
attrs["EnglishAccent"] = "Auto"
matched = [m for m in matched if not (m["category"] == "EnglishAccent" and m["token"] == dropped)]
instruct = ", ".join(attrs[c] for c in CATEGORY_ORDER if attrs[c] != "Auto")
unmatched = []
for frag in _FRAGMENT.finditer(text):
if not _HAS_CONTENT.search(frag.group(0)):
continue
lo, hi = frag.start(), frag.end()
if any(s < hi and e > lo for s, e in spans):
continue
unmatched.append(frag.group(0).strip())
return {
"attrs": attrs,
"instruct": instruct,
"matched": matched,
"unmatched": unmatched,
}
+279
View File
@@ -0,0 +1,279 @@
"""Canonical host compute-capability probe — the single source of truth for
"what can this machine actually accelerate on."
Every routing decision (the engine compatibility matrix, ``/setup/preflight``,
``/system/diagnose``, and the synth-time no-silent-fallback gating) reads from
``detect_host_caps()`` so the probe and the model loader can never disagree.
Design contract (load-bearing):
- **Never raises** to a caller. A broken torch / driver crash degrades to a
cached CPU-only ``probe_ok=False`` result; every endpoint stays responsive
(local-first: the app must work with no GPU and even with a broken torch).
- **No network call** driver/sysctl reads only, no tensor allocation, so it
stays kernel-free on cold start.
- **No new regex** on any driver/device string (CodeQL py/polynomial-redos):
the only string parse is the ``int(driver.split(".")[0])`` shape reused
from the wizard, and arch comparison is plain list membership.
- Distinguishes **ROCm from CUDA** (unlike the gguf ``hardware_probe``):
ROCm-on-HIP presents through ``torch.cuda`` but is reported ``family="rocm"``.
The ``get_best_device()`` loader (``services.model_manager``) delegates its
*family* decision here while keeping its own DirectML branch and the ROCm
``HSA_OVERRIDE_GFX_VERSION`` env side-effect the probe **reads**, the loader
**writes**. (The gguf ``hardware_probe.detect_capabilities()`` rebase onto this
module is a deliberate follow-up: it has its own torch-mocked test suite and a
VRAM-driven quant table that is unaffected by the family rename, so it is kept
out of this backend-only slice.)
"""
from __future__ import annotations
import functools
import platform as _platform
import sys
from dataclasses import dataclass
from typing import Literal
DeviceFamily = Literal["cuda", "rocm", "mps", "xpu", "cpu"]
# Stable substring stamped onto notes that represent a real kernel-launch risk
# (arch/driver mismatch) — as opposed to advisory notes (multi-GPU, VRAM query
# failed, DirectML present). ``engine_routing`` keys the "accelerated, but…"
# caveat off this marker so advisory notes never downgrade an accelerated badge.
KERNEL_RISK_MARKER = "may fail at kernel launch"
# Substring marking a DirectML-present (Windows GPU) host. The probe reports
# such hosts as ``family="cpu"`` (DirectML is not a torch device family); the
# router reads this marker to explain the neutral badge instead of "no GPU".
DIRECTML_MARKER = "DirectML device present"
# NOTE: the NVIDIA driver-version check (min R555 for the bundled CUDA runtime)
# is intentionally NOT done here — it requires shelling to ``nvidia-smi``, which
# would put a subprocess on the cold-start probe path. That check stays in
# ``wizard._detect_gpu`` (preflight), which already runs it. The probe only
# emits the torch-visible SM-arch caveat (cheap, metadata-only).
@dataclass(frozen=True)
class HostCaps:
"""Snapshot of the host's accelerator capability. Immutable + cached."""
family: DeviceFamily
"""Best available accelerator family, else ``"cpu"``."""
available_families: tuple[DeviceFamily, ...]
"""Everything usable; **always includes** ``"cpu"`` (invariant)."""
device_name: str = ""
"""Device 0's name, e.g. ``"NVIDIA RTX 4090"`` / ``"Apple Silicon (MPS)"``."""
vram_gb: float = 0.0
"""CUDA/ROCm total VRAM in GB; MPS = system RAM / 2; 0 for cpu/xpu."""
driver: str | None = None
"""Raw ROCm HIP version string (``torch.version.hip``) or ``None``. The
NVIDIA driver-version check is owned by ``wizard._detect_gpu`` (it already
shells to ``nvidia-smi``); the probe stays subprocess-free."""
notes: tuple[str, ...] = ()
"""Author-controlled English advisories (never user input). Empty on a
clean accelerated host."""
probe_ok: bool = True
"""``False`` only when torch could not be imported (degraded CPU-only)."""
def _probe() -> HostCaps:
"""Run the probe once. Enumerates every failure branch from the spec's
degradation contract; never raises."""
try:
import torch
except Exception:
return HostCaps(
family="cpu",
available_families=("cpu",),
notes=("torch not importable; treating host as CPU-only",),
probe_ok=False,
)
notes: list[str] = []
# Probe EVERY accelerator independently into this list (don't short-circuit
# after the first hit) so `available_families` is honest on hybrid hosts
# (e.g. an NVIDIA GPU + an Intel iGPU exposed via IPEX). The preferred
# `family` is chosen by priority at the end.
detected: list[DeviceFamily] = []
device_name = ""
vram_gb = 0.0
driver: str | None = None
# ── CUDA / ROCm (both present through torch.cuda) ────────────────────
cuda_ok = False
try:
cuda_ok = bool(torch.cuda.is_available())
except Exception as exc: # broken CUDA init (forked process / driver crash)
notes.append(f"CUDA init raised: {type(exc).__name__}")
if cuda_ok:
try:
count = int(torch.cuda.device_count())
except Exception:
count = 0
if count == 0:
notes.append("CUDA reports available but device_count==0")
else:
is_rocm = getattr(torch.version, "hip", None) is not None
detected.append("rocm" if is_rocm else "cuda")
if is_rocm:
driver = getattr(torch.version, "hip", None)
if count > 1:
notes.append(f"{count} GPUs detected; routing reflects device 0")
try:
device_name = torch.cuda.get_device_name(0)
except Exception:
device_name = ""
try:
_free, total = torch.cuda.mem_get_info()
vram_gb = float(total) / (1024 ** 3)
except Exception:
notes.append("VRAM query failed")
# SM-arch mismatch (mirrors model_manager.check_device_compatibility).
try:
major, minor = torch.cuda.get_device_capability(0)
arch_list = getattr(torch.cuda, "_get_arch_list", lambda: [])()
if arch_list:
sm_tag = f"sm_{major}{minor}"
compute_tag = f"compute_{major}{minor}"
if sm_tag not in arch_list and compute_tag not in arch_list:
notes.append(
f"{device_name or 'GPU'} ({sm_tag}) not in this torch "
f"build's archs ({', '.join(arch_list)}) — "
f"{KERNEL_RISK_MARKER}"
)
except Exception:
# Arch metadata unavailable on this torch build — skip the check
# (treated as compatible, exactly as check_device_compatibility).
pass
# ── Intel XPU via IPEX ───────────────────────────────────────────────
try:
import intel_extension_for_pytorch # noqa: F401
if hasattr(torch, "xpu") and torch.xpu.is_available():
detected.append("xpu")
if not device_name:
try:
device_name = torch.xpu.get_device_name(0)
except Exception:
# XPU present but unnamed — family classification still holds.
pass
notes.append("XPU VRAM not queried (unreliable across IPEX versions)")
except Exception:
# IPEX absent or XPU probe failed — no XPU on this host.
pass
# ── Apple Silicon MPS ────────────────────────────────────────────────
try:
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
detected.append("mps")
if not device_name:
device_name = "Apple Silicon (MPS)"
if not vram_gb:
try:
import psutil
vram_gb = float(psutil.virtual_memory().total) / (1024 ** 3) / 2
except Exception:
notes.append("psutil unavailable; MPS VRAM unknown")
except Exception:
# MPS probe raised on a non-Apple/old torch — treat as no MPS.
pass
# ── DirectML — Windows GPU, NOT a torch device family ────────────────
try:
import torch_directml
if torch_directml.device_count() > 0:
notes.append(
f"{DIRECTML_MARKER} (Windows GPU); torch-family probe treats "
f"as non-accelerated"
)
except Exception:
# torch_directml absent (the common case) — no DirectML on this host.
pass
# Preferred family by priority; cpu when nothing accelerated was detected.
family: DeviceFamily = "cpu"
for pref in ("cuda", "rocm", "xpu", "mps"):
if pref in detected:
family = pref # type: ignore[assignment]
break
# available_families: every detected accelerator + cpu, deduped, cpu last.
available: tuple[DeviceFamily, ...] = tuple(dict.fromkeys([*detected, "cpu"]))
return HostCaps(
family=family,
available_families=available,
device_name=device_name,
vram_gb=vram_gb,
driver=driver,
notes=tuple(notes),
probe_ok=True,
)
@functools.lru_cache(maxsize=1)
def detect_host_caps() -> HostCaps:
"""Cached per-process host capabilities. Never raises, makes no network
call, kernel-free on cold start. Host compute capability does not change at
runtime in any supported desktop flow (no GPU hot-plug; switching the active
engine does not re-probe routing is recomputed from these same caps), so
a single probe per process is correct. ``probe_ok=False`` is cached too."""
return _probe()
def refresh() -> HostCaps:
"""Clear the cache and re-probe. **TEST-ONLY** — nothing in the running app
calls this (host caps are immutable per process)."""
detect_host_caps.cache_clear()
return detect_host_caps()
def mlx_supported() -> tuple[bool, str]:
"""``(ok, reason)``. ``ok=True`` **only** on Apple Silicon
(``sys.platform == "darwin"`` and ``platform.machine() == "arm64"``) with
torch MPS available the shared gate for MLX-Audio / MLX-Whisper (#390).
Gates on exact-string equality (no regex no CodeQL surface). On any
non-Apple host it returns ``False`` **before** any package import, so a
stray ``mlx_*`` wheel on Linux/Windows never reports available.
"""
if sys.platform != "darwin" or _platform.machine() != "arm64":
if sys.platform == "darwin":
return (False, "MLX requires Apple Silicon; this Mac is Intel")
return (
False,
f"MLX requires Apple Silicon; this host is "
f"{sys.platform}/{_platform.machine()}",
)
try:
import torch
except Exception:
return (False, "torch not importable; cannot confirm MPS")
try:
if torch.backends.mps.is_available():
return (True, "")
except Exception:
# MPS query raised — fall through to the conservative unavailable path.
pass
return (
False,
"Apple Silicon detected but torch MPS unavailable; "
"reinstall torch with MPS support",
)
__all__ = [
"DeviceFamily",
"HostCaps",
"detect_host_caps",
"refresh",
"mlx_supported",
"KERNEL_RISK_MARKER",
"DIRECTML_MARKER",
]
+47 -1
View File
@@ -4,7 +4,7 @@ One pass over everything a working install needs: Python, compute device,
ffmpeg, HF token, disk, data-dir permissions, RAM, TTS engines, and (when
requested) network reachability of the HuggingFace hub. Surfaced two ways:
- ``GET /system/diagnose`` (Settings > About "Run self-check")
- ``GET /system/diagnose`` (Settings > About -> "Run self-check")
- ``python main.py --diagnose`` for headless installs / issue triage
Every ``detail``/``hint`` string is passed through ``core.scrub`` before it
@@ -194,6 +194,49 @@ def _check_engines() -> dict:
return _check("engines", "TTS engines", OK, detail)
def _check_gpu_routing() -> dict:
"""Routing verdict for the active TTS engine on THIS host (#21).
Surfaces a CPU fallback / unavailable-GPU *before* a slow or failed synth
the no-silent-fallback contract. `cpu_only` on a no-GPU machine is the
expected normal state and stays OK (never noise-warns)."""
try:
from services.tts_backend import gpu_routing_verdict
v = gpu_routing_verdict()
except Exception as e:
return _check("gpu_routing", "GPU routing", WARN, f"could not resolve: {e}")
status = v.get("routing_status")
engine = v.get("engine") or "active engine"
dev = v.get("effective_device") or "?"
reason = v.get("routing_reason")
host = v.get("host_family", "cpu")
if status == "accelerated":
if reason: # driver/arch caveat — accelerated but at risk
return _check("gpu_routing", "GPU routing", WARN,
f"{engine} -> {dev}: {reason}",
"The GPU is selected but may fail at kernel launch — "
"update drivers / reinstall torch for this GPU arch.")
return _check("gpu_routing", "GPU routing", OK, f"{engine} -> {dev} (accelerated)")
if status == "cpu_fallback":
return _check("gpu_routing", "GPU routing", WARN,
f"{engine} runs on CPU: {reason or 'no GPU path for this host'}",
"Pick an engine that supports this host's GPU for a big speedup, "
"or continue on CPU (slower).")
if status == "cpu_only":
return _check("gpu_routing", "GPU routing", OK,
f"{engine} -> cpu (no accelerator on this host)")
if status == "unavailable":
return _check("gpu_routing", "GPU routing", FAIL,
f"{engine} can't run on this host: {reason or f'needs a GPU; host is {host}'}",
"Select an engine with a CPU path in Settings -> Engines.")
# status == "none" / unknown — no active engine resolved.
return _check("gpu_routing", "GPU routing", WARN,
"No active TTS engine resolved for routing.",
"Pick an engine in Settings -> Engines.")
_DEEP_TIMEOUT_S = 180
@@ -264,6 +307,8 @@ def _check_network() -> dict:
# all model downloads need to get started. urllib honors HTTP(S)_PROXY.
import urllib.request
import urllib.error
if not _HUB_URL.startswith("https://"): # constant today; guard the sink anyway
raise ValueError(f"hub URL must be https, got {_HUB_URL!r}")
req = urllib.request.Request(_HUB_URL, method="HEAD")
try:
with urllib.request.urlopen(req, timeout=_HUB_TIMEOUT_S):
@@ -296,6 +341,7 @@ def run_diagnostics(include_network: bool = True, deep: bool = False) -> dict:
_check_data_dir(),
_check_ram(),
_check_engines(),
_check_gpu_routing(),
]
if include_network:
checks.append(_check_network())
+2 -1
View File
@@ -111,7 +111,8 @@ def classify_exception(exc: BaseException, trace: str = "") -> str:
def _fingerprint(error_class: str, exc: BaseException) -> str:
import hashlib
raw = f"{error_class}|{type(exc).__name__}|{scrub_text(str(exc))[:200]}"
return hashlib.sha1(raw.encode("utf-8", "replace")).hexdigest()[:16]
# Dedup key for the journal, not a security boundary.
return hashlib.sha1(raw.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]
def _persist_locked() -> None:
+1 -1
View File
@@ -32,7 +32,7 @@ _REDACTED_VALUE = "***REDACTED***"
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
# taxonomy; the docs URL itself stays owned by error_docs_map.
_HINTS: dict[str, str] = {
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
"PKG_RESOURCES_MISSING": "Run `uv pip install --reinstall 'setuptools>=75,<80'` in the backend venv (a plain install is skipped when setuptools' metadata is present but its pkg_resources files were removed by antivirus). Restart after.",
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
+4 -1
View File
@@ -42,9 +42,12 @@ _TOKEN_PATTERNS = (
# pattern-wise (not just this machine's $HOME) so paths quoted from a
# user's pasted log on another OS get cleaned too.
_HOME_PATTERNS = (
# Windows-with-forward-slashes must run BEFORE the bare macOS shape, or
# `/Users/<name>` inside `C:/Users/<name>` gets eaten first, leaving `C:~`.
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+"), # Windows, forward slashes (file URLs, normalized traces)
re.compile(r"/Users/[^/\s\"']+"), # macOS
re.compile(r"/home/[^/\s\"']+"), # Linux
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows, backslashes
)
# Values shorter than this are too entropy-poor to be real secrets and too
+25
View File
@@ -82,3 +82,28 @@ def unset_user_env(key: str, path: Optional[str] = None) -> None:
prefix = f"{key}="
lines = [ln for ln in _read_lines(path) if not ln.startswith(prefix)]
_write_lines(path, lines)
def load_into_environ(path: Optional[str] = None) -> bool:
"""Load the durable per-user env file into ``os.environ``, **overriding**
any value a launcher already injected. Returns True if a file was loaded.
This file is the in-app Settings source of truth. The desktop launcher
(Tauri) injects defaults like ``OMNIVOICE_CACHE_DIR`` (and ``HF_ENDPOINT``)
from its *own* config into the backend's environment *before* startup, so
loading this file with ``override=False`` meant a models directory the user
changed in Settings was silently ignored on every launch the effective
location stayed on the old one no matter how many restarts (#480). Both keys
this file can hold are the user's explicit Settings choice and should beat
the launcher's default, so we override. Restores this file's documented
"values written here take effect on the next backend launch" contract.
"""
path = path or os.environ.get("OMNIVOICE_ENV_FILE") or USER_ENV_PATH
if not os.path.isfile(path):
return False
try:
import dotenv
except ImportError:
return False
dotenv.load_dotenv(path, override=True)
return True
+120
View File
@@ -0,0 +1,120 @@
"""Crash-isolated faster-whisper ASR sidecar (Wave 4.2 / Spec 7).
Runs faster-whisper in a child process so a CTranslate2 GPU-teardown segfault
becomes a failed job, not a dead backend. Speaks the SubprocessBackend wire
protocol (length-prefixed JSON over stdin/stdout):
on start {"op":"ready","engine":"faster-whisper-isolated"}
{"op":"ping"} {"op":"pong"}
{"op":"transcribe","audio_path":...,"word_timestamps":bool}
{"op":"segments","result":{"segments":[...],"language":...}}
{"op":"shutdown"} exit 0
error {"op":"error","message":...}
Runs under the PARENT venv (faster-whisper is already a dependency) only the
process boundary is new. torch/CTranslate2 import lazily inside transcribe so
the ready handshake fits the spawn timeout.
"""
from __future__ import annotations
import json
import os
import struct
import sys
import traceback
MAX_FRAME_BYTES = 64 * 1024 * 1024
_model = None
def _send(stream, obj):
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
def _get_model():
global _model
if _model is None:
from faster_whisper import WhisperModel
name = os.environ.get("ASR_MODEL_FW", "large-v3")
try:
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
except Exception:
device = "cpu"
compute = "float16" if device == "cuda" else "int8"
_model = WhisperModel(name, device=device, compute_type=compute)
return _model
def _transcribe(audio_path, word_timestamps):
model = _get_model()
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps)
out = []
for s in segments:
seg = {"start": float(s.start), "end": float(s.end), "text": s.text}
if word_timestamps and getattr(s, "words", None):
seg["words"] = [
{"word": w.word, "start": float(w.start), "end": float(w.end),
"probability": float(getattr(w, "probability", 0.0))}
for w in s.words
]
out.append(seg)
return {
"segments": out,
"text": " ".join(s["text"].strip() for s in out).strip(),
"language": getattr(info, "language", "unknown"),
}
def main() -> int:
stdin, stdout = sys.stdin.buffer, sys.stdout.buffer
_send(stdout, {"op": "ready", "engine": "faster-whisper-isolated"})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {"op": "error", "stage": "recv", "message": f"{type(exc).__name__}: {exc}"})
return 1
if msg is None:
return 0
op = msg.get("op")
try:
if op == "ping":
_send(stdout, {"op": "pong"})
elif op == "transcribe":
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)))
_send(stdout, {"op": "segments", "result": result})
elif op == "shutdown":
return 0
else:
_send(stdout, {"op": "error", "stage": "dispatch", "message": f"unknown op: {op!r}"})
except Exception as exc:
_send(stdout, {
"op": "error", "stage": "handler",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 0
if __name__ == "__main__":
sys.exit(main())
+19
View File
@@ -101,6 +101,12 @@ def main() -> int:
return 0
op = msg.get("op")
# Wave 4.2: deterministic "crash mid-transcription" hook — exit BEFORE
# sending any reply so the parent's blocking recv sees a dead pipe
# (reply=None). The crash-after-one hook below replies first, so it
# can't deterministically exercise the no-reply path.
if op == "transcribe" and os.environ.get("OMNIVOICE_ECHO_CRASH_NO_REPLY") == "1":
os._exit(1)
try:
if op == "ping":
_send(stdout, {"op": "pong"})
@@ -113,6 +119,19 @@ def main() -> int:
"sample_rate": sr,
"n_samples": n_samples,
})
elif op == "transcribe":
# Wave 4.2: echo ASR op — a canned segments result so the
# SubprocessASRBackend round-trip + respawn path is testable
# without a real ASR engine.
_send(stdout, {
"op": "segments",
"result": {
"segments": [{"start": 0.0, "end": 1.0,
"text": f"echo:{msg.get('audio_path', '')}"}],
"text": f"echo:{msg.get('audio_path', '')}",
"language": "en",
},
})
elif op == "shutdown":
return 0
elif op == "probe_env" and test_mode:
+5
View File
@@ -79,6 +79,11 @@ class IndexTTS2Backend(SubprocessBackend):
display_name = "IndexTTS2 (emotion control, duration control, zero-shot)"
supports_voice_design = False # requires ref audio for timbre
_DEFAULT_SAMPLE_RATE = 24000
# Explicit so IndexTTS2 stops advertising the inherited CPU-only default:
# the sidecar runs the IndexTTS PyTorch model on CUDA when present, else
# CPU. ROCm left unclaimed (the sidecar's own venv would need a ROCm torch);
# a ROCm host honestly resolves to cpu_fallback.
gpu_compat = ("cuda", "cpu")
@classmethod
def is_available(cls) -> tuple[bool, str]:
+19 -1
View File
@@ -74,6 +74,24 @@ import traceback
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
MAX_FRAME_BYTES = 64 * 1024 * 1024
def _measure_vram_mb() -> float:
"""This sidecar's own GPU memory in MB, for the loaded-models panel
(MM2-08). The parent can't see a child's VRAM, so we self-report it in the
pong. Degrades to 0 on CPU / when torch isn't loaded yet — never raises."""
try:
import torch # already a dep inside the indextts venv
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
mps = getattr(torch.backends, "mps", None)
if mps is not None and mps.is_available():
drv = getattr(torch.mps, "driver_allocated_memory", None)
if drv:
return round(drv() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
# Sample rate IndexTTS-2 emits natively. Advertised in the ready frame so
# the parent doesn't have to import IndexTTS just to learn the rate.
INDEXTTS_SAMPLE_RATE = 24000
@@ -282,7 +300,7 @@ def main() -> int:
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong"})
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
+67 -6
View File
@@ -353,6 +353,21 @@ def _make_backend_class():
f"This clears the quarantine on the .app and its "
f"bundled binaries. See docs/install/macos.md."
)
# Execute bit (issue #437). A `git clone` / zip extract on POSIX
# can drop +x, which only surfaces at spawn time as
# "[Errno 13] Permission denied" — and the generic synth handler
# then mislabels it as out-of-memory. Self-heal here, AFTER the
# SHA check has confirmed this is the right file (so we never
# chmod a foreign binary). No-op on Windows.
if os.name == "posix" and not os.access(bin_path, os.X_OK):
try:
bin_path.chmod(bin_path.stat().st_mode | 0o111)
except OSError:
return False, (
f"GGUF binary {bin_path.name} isn't executable and "
f"couldn't be made so — run `chmod +x {bin_path}` "
f"and retry."
)
return True, "ready"
except Exception as exc:
return False, f"{type(exc).__name__}: {exc}"
@@ -493,6 +508,12 @@ def _make_backend_class():
* ``ref_audio`` (str/Path) speaker reference WAV for cloning.
* ``ref_text`` (str) transcript of ``ref_audio``.
* ``language`` (str) ISO code or omnivoice-tts lang label.
* ``instruct`` (str) style instruction.
* ``duration`` (float) target duration in seconds.
* ``seed`` (int) deterministic sampling seed.
* ``denoise`` (bool) omit denoise token when false.
* ``preprocess_prompt`` (bool) skip prompt preprocessing when false.
* ``chunk_duration`` / ``chunk_threshold`` (float) binary long-form controls.
"""
import soundfile as sf # local import keeps module import cheap
import torch
@@ -503,15 +524,32 @@ def _make_backend_class():
fd, out_str = tempfile.mkstemp(prefix="omnivoice-gguf-", suffix=".wav")
os.close(fd)
out_path = Path(out_str)
ref_text_path: Optional[Path] = None
try:
ref_text = kw.get("ref_text")
if kw.get("ref_audio") and ref_text:
text_fd, text_str = tempfile.mkstemp(
prefix="omnivoice-gguf-ref-", suffix=".txt"
)
os.close(text_fd)
ref_text_path = Path(text_str)
ref_text_path.write_text(str(ref_text), encoding="utf-8")
argv = self._build_argv(
base=base_path,
tokenizer=tok_path,
out_path=out_path,
ref_audio=kw.get("ref_audio"),
ref_text=kw.get("ref_text"),
ref_text=str(ref_text_path) if ref_text_path else None,
language=kw.get("language"),
instruct=kw.get("instruct"),
duration=kw.get("duration"),
seed=kw.get("seed"),
denoise=kw.get("denoise", True),
preprocess_prompt=kw.get("preprocess_prompt", True),
chunk_duration=kw.get("chunk_duration"),
chunk_threshold=kw.get("chunk_threshold"),
)
self._run_subprocess(argv, stdin_text=text)
wav, sr = sf.read(str(out_path))
@@ -520,6 +558,11 @@ def _make_backend_class():
out_path.unlink()
except OSError:
pass
if ref_text_path is not None:
try:
ref_text_path.unlink()
except OSError:
pass
# soundfile returns (n,) for mono or (n, c) for multichannel.
# OmniVoice/Higgs Audio v2 is mono → (n,). Wrap to (1, n).
@@ -544,6 +587,13 @@ def _make_backend_class():
ref_audio: Optional[str],
ref_text: Optional[str],
language: Optional[str],
instruct: Optional[str] = None,
duration: Optional[float] = None,
seed: Optional[int] = None,
denoise: bool = True,
preprocess_prompt: bool = True,
chunk_duration: Optional[float] = None,
chunk_threshold: Optional[float] = None,
) -> list[str]:
"""Compose argv from typed Path objects only (T-04-02)."""
argv: list[str] = [
@@ -555,6 +605,20 @@ def _make_backend_class():
lang = _iso_to_omnivoice_lang(language)
if lang:
argv += ["--lang", lang]
if instruct:
argv += ["--instruct", str(instruct)]
if duration is not None:
argv += ["--duration", str(float(duration))]
if seed is not None:
argv += ["--seed", str(int(seed))]
if denoise is False:
argv += ["--no-denoise"]
if preprocess_prompt is False:
argv += ["--no-preprocess-prompt"]
if chunk_duration is not None:
argv += ["--chunk-duration", str(float(chunk_duration))]
if chunk_threshold is not None:
argv += ["--chunk-threshold", str(float(chunk_threshold))]
if ref_audio:
# Two-stage validation (defense in depth):
# (a) Reject anything outside the project's voices /
@@ -588,11 +652,8 @@ def _make_backend_class():
)
argv += ["--ref-wav", str(ref_path)]
if ref_text:
# ref_text is free-form text; pass via stdin would
# collide with the synthesis prompt, so the only safe
# channel is argv. The binary treats this as a quoted
# string at the OS layer (Popen escapes argv per
# platform); we don't pre-escape.
# The C++ runtime expects a transcript file path.
# generate() creates this file in the system temp dir.
argv += ["--ref-text", str(ref_text)]
return argv
+162 -7
View File
@@ -43,11 +43,13 @@ try:
_project_env = os.path.join(os.path.dirname(_backend_dir), ".env")
if os.path.isfile(_project_env):
dotenv.load_dotenv(_project_env, override=False)
# Also load the durable per-user config so env vars set once survive
# Tauri/Finder launches that don't inherit a shell environment.
_user_env = os.path.expanduser("~/.config/omnivoice/env")
if os.path.isfile(_user_env):
dotenv.load_dotenv(_user_env, override=False)
# Load the durable per-user config (the in-app Settings source of truth) so
# env vars set once survive Tauri/Finder launches that don't inherit a shell
# environment. This OVERRIDES launcher-injected defaults: the desktop app
# injects a stale OMNIVOICE_CACHE_DIR from its own config before startup, so
# without override a models dir changed in Settings was ignored forever (#480).
from core.user_env import load_into_environ as _load_user_env
_load_user_env()
except ImportError:
pass
@@ -284,7 +286,12 @@ from fastapi.responses import JSONResponse, RedirectResponse, Response
from fastapi.staticfiles import StaticFiles
from fastapi.middleware.cors import CORSMiddleware
from starlette.datastructures import MutableHeaders
from scalar_fastapi import get_scalar_api_reference
# Docs-only dependency: a venv created before scalar-fastapi entered the
# dependency set must still boot the backend (#307) — /docs degrades instead.
try:
from scalar_fastapi import get_scalar_api_reference
except ImportError:
get_scalar_api_reference = None
import traceback
_crash_log_lock = threading.Lock()
@@ -313,6 +320,7 @@ from api.routers import (
setup,
gallery,
archetypes,
describe_voice,
community,
batch,
watermark,
@@ -322,7 +330,10 @@ from api.routers import (
openai_compat,
tts_stream,
marketplace,
personas,
sonitranslate,
audiobook,
longform_jobs,
settings as settings_router, # Phase 1 AUTH-03: HF token save/clear/state
)
from utils import hf_progress
@@ -332,6 +343,30 @@ from utils import hf_progress
# the patched class, not the original.
hf_progress.install()
# Wire the overall download aggregator's byte sink onto the patched tqdm so
# parallel per-file updates feed one accurate overall bar (FDL-06).
try:
from utils import download_aggregator
download_aggregator.install()
except Exception:
pass
# Log the download-acceleration state once at startup (FDL-03) so a slow
# download report can be triaged from the logs without reproducing. Note: the
# app sets HF_HUB_DISABLE_XET=1 above by default (legacy LFS for byte progress),
# so xet_active is normally False even though hf_xet is installed.
try:
from api.routers.system import _fast_download_status as _fd_status
_fd = _fd_status()
_xet_ver = f" {_fd['xet_version']}" if _fd.get("xet_version") else ""
logging.getLogger("omnivoice.model").info(
"downloads: Xet %s (hf_xet%s installed=%s), high_perf=%s",
"ACTIVE" if _fd["xet_active"] else "disabled → legacy LFS",
_xet_ver, _fd["xet_installed"], _fd["high_performance"],
)
except Exception:
pass
def _env_flag(name: str, default: bool = False) -> bool:
value = os.environ.get(name)
@@ -422,7 +457,23 @@ async def lifespan(app: FastAPI):
capture_preload_task = asyncio.create_task(_preload_capture_asr())
else:
logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
yield
# ── MCP session manager (Wave 2.2) ────────────────────────────────────
# FastMCP's Streamable-HTTP transport needs its session manager running
# for the lifetime of the app. It's created lazily by streamable_http_app()
# (called in mount_mcp below), so we stack its `run()` context into ours
# via AsyncExitStack rather than replacing this lifespan. Best-effort: a
# missing/broken MCP layer must never stop the rest of the backend.
from contextlib import AsyncExitStack
async with AsyncExitStack() as _mcp_stack:
_sm = getattr(app.state, "mcp_session_manager", None)
if _sm is not None:
try:
await _mcp_stack.enter_async_context(_sm.run())
logger.info("MCP server mounted at /mcp")
except Exception as e:
logger.warning("MCP session manager failed to start: %s", e)
yield
# ── Graceful shutdown (SIGTERM from Tauri, Ctrl+C, etc.) ────────────
logger.info("Shutdown: cleaning up…")
idle_task.cancel()
@@ -471,6 +522,14 @@ app = FastAPI(
@app.get("/docs", include_in_schema=False)
async def scalar_docs():
"""Interactive API documentation powered by Scalar."""
if get_scalar_api_reference is None:
return JSONResponse(
status_code=503,
content={
"detail": "API docs unavailable: scalar-fastapi is not installed "
"in the backend environment (#307)."
},
)
return get_scalar_api_reference(
openapi_url=app.openapi_url,
title=app.title,
@@ -579,6 +638,68 @@ class NetworkAccessMiddleware:
return await self.app(scope, receive, send)
class BearerKeyMiddleware:
"""When OMNIVOICE_API_KEY is set, non-loopback clients must present it on
every HTTP + WebSocket request: ``Authorization: Bearer <key>``,
``?api_key=<key>`` (browser WebSockets cannot set headers), or the
``ov_key`` cookie (set on the first successful HTTP auth). Loopback
always bypasses the desktop default is unchanged and the SPA shell
paths stay reachable so a remote UI can load and show what's wrong.
Inert when the env var is unset (the default). Pure ASGI for the same
no-buffering reason as NetworkAccessMiddleware above. Plain-HTTP caveat
is documented in docs/remote-gpu.md: the key is sniffable outside a
WireGuard (Tailscale) or TLS (tailscale serve) transport.
"""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] not in ("http", "websocket"):
return await self.app(scope, receive, send)
key = os.environ.get("OMNIVOICE_API_KEY") or ""
if not key:
return await self.app(scope, receive, send)
client = scope["client"][0] if scope.get("client") else None
if client in _LOOPBACK_CLIENTS:
return await self.app(scope, receive, send)
path = scope.get("path", "")
if scope["type"] == "http" and (
path in _SHELL_PATHS or path.startswith("/assets/") or path.startswith("/favicon")
):
return await self.app(scope, receive, send)
from starlette.requests import HTTPConnection
conn = HTTPConnection(scope)
auth = conn.headers.get("authorization", "")
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
if not supplied:
supplied = conn.query_params.get("api_key") or conn.cookies.get("ov_key") or ""
if not secrets.compare_digest(supplied, key):
if scope["type"] == "websocket":
# Reject the handshake; 1008 = policy violation.
await receive() # consume websocket.connect
await send({"type": "websocket.close", "code": 1008})
return
resp = JSONResponse({"detail": "API key required"}, status_code=401)
return await resp(scope, receive, send)
if scope["type"] == "http" and conn.cookies.get("ov_key") != key:
async def send_with_cookie(message):
if message["type"] == "http.response.start":
headers = MutableHeaders(scope=message)
headers.append(
"set-cookie", f"ov_key={key}; Path=/; SameSite=Lax"
)
await send(message)
return await self.app(scope, receive, send_with_cookie)
return await self.app(scope, receive, send)
# UI dev-server port — single-sourced from OMNIVOICE_UI_PORT so a user who
# moves the Vite dev server off 3901 still gets a matching CORS allow-list.
def _ui_port() -> int:
@@ -610,6 +731,15 @@ app.add_middleware(
# applied even to the 401 PIN-required responses). Inert unless a PIN is set.
app.add_middleware(NetworkAccessMiddleware)
# Remote-backend bearer gate (parity program Wave 2.3 / §R2). Inert unless
# OMNIVOICE_API_KEY is set. Distinct from the PIN gate above: the PIN guards
# casual LAN-share guests for one session; the API key is the durable
# credential for running this backend remotely (Tailscale / Docker GPU box).
# Covers WebSockets too — the PIN gate never did, because every WS endpoint
# carried its own loopback guard; remote mode is exactly the case where a
# keyed non-loopback client must reach them.
app.add_middleware(BearerKeyMiddleware)
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
@@ -652,6 +782,7 @@ app.include_router(stories.router)
app.include_router(setup.router)
app.include_router(gallery.router)
app.include_router(archetypes.router)
app.include_router(describe_voice.router) # issue #317: free-text voice design
app.include_router(community.router)
app.include_router(batch.router)
app.include_router(watermark.router)
@@ -661,8 +792,32 @@ app.include_router(capture_ws.router)
app.include_router(openai_compat.router)
app.include_router(tts_stream.router)
app.include_router(marketplace.router)
app.include_router(personas.router)
app.include_router(sonitranslate.router)
app.include_router(audiobook.router)
app.include_router(longform_jobs.router)
app.include_router(settings_router.router) # Phase 1 AUTH-03 endpoints
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
app.include_router(_mcp_bindings_router.router) # Wave 2.2 per-agent voice bindings
# ── Mount the MCP server (Wave 2.2) ───────────────────────────────────────
# FastMCP's Streamable-HTTP app is sub-mounted at /mcp; its session manager is
# stashed on app.state for the lifespan above to run. Opt-out via
# OMNIVOICE_MCP_DISABLE=1; best-effort so a missing mcp package or a build
# without it never breaks startup.
if os.environ.get("OMNIVOICE_MCP_DISABLE", "").strip().lower() not in ("1", "true", "yes", "on"):
try:
from mcp_server import create_mcp_server
_mcp = create_mcp_server()
_mcp_app = _mcp.streamable_http_app()
app.state.mcp_session_manager = _mcp.session_manager
app.mount("/mcp", _mcp_app)
logging.getLogger("omnivoice.api").info("MCP app mounted at /mcp")
except Exception as _mcp_err: # noqa: BLE001
logging.getLogger("omnivoice.api").info(
"MCP server not mounted (%s); /mcp disabled.", _mcp_err
)
frontend_path = os.path.join(os.path.dirname(__file__), "..", "frontend", "dist")
if os.path.exists(frontend_path):
+64 -1
View File
@@ -53,6 +53,14 @@ def create_mcp_server():
"voice design, and video dubbing in 646 languages."
),
)
# Serve the Streamable-HTTP transport at the app root so mounting the whole
# app at "/mcp" on the main FastAPI yields the endpoint at "/mcp". FastMCP's
# default path is "/mcp", which would double-prefix to "/mcp/mcp" when
# sub-mounted. Harmless for the standalone CLI run() path.
try:
mcp.settings.streamable_http_path = "/"
except Exception:
pass
# ── Helpers ─────────────────────────────────────────────────────────
@@ -75,6 +83,21 @@ def create_mcp_server():
# ── Tools ───────────────────────────────────────────────────────────
def _current_client_id() -> str | None:
"""The X-OmniVoice-Client-Id of the calling MCP client, if any.
FastMCP exposes the HTTP request via its request context on the
Streamable-HTTP transport; stdio clients (and any version where the
accessor differs) simply resolve to None and fall back to the
global default voice."""
try:
req = mcp.get_context().request_context.request
if req is not None:
return req.headers.get("x-omnivoice-client-id")
except Exception:
pass
return None
@mcp.tool()
async def generate_speech(
text: str,
@@ -89,7 +112,8 @@ def create_mcp_server():
Args:
text: The text to synthesize into speech.
language: Target language (ISO code or 'Auto'). 646 languages supported.
profile_id: ID of a saved voice profile to clone. Omit for voice design mode.
profile_id: ID of a saved voice profile to clone. Omit to use this
agent's bound voice (Settings → MCP), else the global default.
instruct: Style instruction (e.g. 'whisper', 'excited', 'narrator').
speed: Speech speed multiplier (0.52.0, default 1.0).
steps: Diffusion steps (8=fast/draft, 16=balanced, 32=quality).
@@ -98,6 +122,17 @@ def create_mcp_server():
JSON with audio_id, generation_time, audio_duration, and
base64-encoded WAV data.
"""
# Per-agent voice binding (Wave 2.2): explicit arg wins; otherwise
# resolve this client's bound profile, then the global default.
client_id = _current_client_id()
try:
from services import mcp_bindings
resolved = mcp_bindings.resolve_voice(client_id, profile_id)
profile_id = resolved.get("profile_id")
mcp_bindings.touch_last_seen(client_id) if client_id else None
except Exception:
pass # binding layer unavailable — use whatever was passed
form = {
"text": text,
"language": language,
@@ -159,6 +194,34 @@ def create_mcp_server():
'],"note":"Pass any ISO 639 code or set language=Auto for detection."}'
)
@mcp.tool()
async def transcribe(audio_base64: str, language: str | None = None) -> str:
"""Transcribe spoken audio to text.
Args:
audio_base64: Base64-encoded audio bytes (wav/mp3/webm/m4a).
language: Optional language hint; omit for auto-detect.
Returns:
JSON with the recognized text, language, and duration.
"""
try:
raw = base64.b64decode(audio_base64, validate=True)
except Exception:
return '{"error":"audio_base64 is not valid base64"}'
# 200 MB cap — same spirit as voicebox's transcribe gate. Keeps a
# buggy/hostile agent from posting an unbounded blob.
if len(raw) > 200 * 1024 * 1024:
return '{"error":"audio exceeds 200 MB limit"}'
data = {}
if language:
data["language"] = language
r = await _api_post_form(
"/transcribe", data=data,
files={"audio": ("audio.wav", raw, "application/octet-stream")},
)
return str(r.json())
@mcp.tool()
async def check_health() -> str:
"""Check if the OmniVoice backend is running and what GPU device is active."""
+1
View File
@@ -0,0 +1 @@
"""omnivoice-mcp — stdio MCP shim for clients that only speak stdio."""
+176
View File
@@ -0,0 +1,176 @@
"""omnivoice-mcp — stdio ↔ Streamable-HTTP MCP proxy (Wave 2.2).
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
Copyright (c) voicebox contributors.
Some MCP clients only speak stdio. They spawn this binary; we pipe each
JSON-RPC message to ``http://127.0.0.1:<port>/mcp/`` (the FastMCP app mounted
on the running OmniVoice backend) and stream the server's response back.
Environment variables:
OMNIVOICE_PORT backend port (default 3900).
OMNIVOICE_HOST host (default 127.0.0.1).
OMNIVOICE_CLIENT_ID forwarded as X-OmniVoice-Client-Id on every request
(drives per-agent voice binding).
Stdout is JSON-RPC only. Diagnostics go to stderr.
Exit 0 on clean EOF, 1 on transport error, 2 if the backend never answers.
Usage in an MCP client config (stdio):
command: python
args: ["-m", "backend.mcp_shim"]
env: { OMNIVOICE_CLIENT_ID: "claude-code" }
"""
from __future__ import annotations
import asyncio
import json
import os
import sys
from typing import Any
import httpx
CLIENT_ID_HEADER = "X-OmniVoice-Client-Id"
SESSION_HEADER = "mcp-session-id"
HEALTH_TIMEOUT_S = 30.0
DEFAULT_PORT = 3900
def _err(msg: str) -> None:
print(f"omnivoice-mcp: {msg}", file=sys.stderr, flush=True)
def _base_url() -> tuple[str, str]:
host = os.environ.get("OMNIVOICE_HOST", "127.0.0.1")
port = int(os.environ.get("OMNIVOICE_PORT", str(DEFAULT_PORT)))
return f"http://{host}:{port}/mcp/", f"http://{host}:{port}/health"
async def _wait_for_backend(client: httpx.AsyncClient, health_url: str) -> bool:
loop = asyncio.get_running_loop()
deadline = loop.time() + HEALTH_TIMEOUT_S
while loop.time() < deadline:
try:
r = await client.get(health_url, timeout=2.0)
if r.status_code == 200:
return True
except Exception:
pass
await asyncio.sleep(0.5)
return False
async def _read_stdin_line() -> str | None:
loop = asyncio.get_running_loop()
line = await loop.run_in_executor(None, sys.stdin.readline)
return line or None
def _write_stdout(obj: Any) -> None:
sys.stdout.write(json.dumps(obj, separators=(",", ":")))
sys.stdout.write("\n")
sys.stdout.flush()
async def _handle_request(
client: httpx.AsyncClient,
url: str,
raw: str,
headers: dict[str, str],
session_id: list[str | None],
) -> None:
try:
message = json.loads(raw)
except json.JSONDecodeError as exc:
_err(f"invalid JSON on stdin: {exc}")
return
req_headers = {
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
**headers,
}
if session_id[0]:
req_headers[SESSION_HEADER] = session_id[0]
is_notification = isinstance(message, dict) and "id" not in message
async with client.stream("POST", url, headers=req_headers, content=raw.encode("utf-8")) as response:
if session_id[0] is None:
sid = response.headers.get(SESSION_HEADER)
if sid:
session_id[0] = sid
if response.status_code == 202:
return # notification acknowledged
if response.status_code >= 400:
body = await response.aread()
_err(f"server {response.status_code}: {body.decode('utf-8', errors='replace')[:400]}")
if is_notification:
return
_write_stdout({
"jsonrpc": "2.0",
"id": message.get("id"),
"error": {"code": -32000, "message": f"OmniVoice MCP proxy got HTTP {response.status_code}"},
})
return
ctype = response.headers.get("content-type", "")
if "text/event-stream" in ctype:
async for line in response.aiter_lines():
if line.startswith("data:"):
payload = line[5:].strip()
if not payload:
continue
try:
_write_stdout(json.loads(payload))
except json.JSONDecodeError:
_err(f"malformed SSE payload: {payload[:200]}")
else:
body = await response.aread()
try:
_write_stdout(json.loads(body))
except json.JSONDecodeError:
_err(f"non-JSON response ({ctype}): {body.decode('utf-8', errors='replace')[:200]}")
async def _run() -> int:
url, health_url = _base_url()
forward_headers: dict[str, str] = {}
client_id = os.environ.get("OMNIVOICE_CLIENT_ID")
if client_id:
forward_headers[CLIENT_ID_HEADER] = client_id
session_id: list[str | None] = [None]
async with httpx.AsyncClient(timeout=httpx.Timeout(300.0)) as client:
if not await _wait_for_backend(client, health_url):
_err(f"timed out waiting for OmniVoice at {health_url} — is the app running?")
return 2
try:
while True:
line = await _read_stdin_line()
if line is None:
return 0
line = line.strip()
if not line:
continue
await _handle_request(client, url, line, forward_headers, session_id)
except (KeyboardInterrupt, SystemExit):
return 0
except Exception as exc:
_err(f"proxy failed: {exc!r}")
return 1
def main() -> int:
try:
return asyncio.run(_run())
except KeyboardInterrupt:
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,58 @@
"""Parity program Wave 0.2: consent-locked voice profiles
Revision ID: 0003_voice_profile_consent
Revises: 0002_voice_profile_demo_fields
Create Date: 2026-06-12 00:00:00.000000
Adds four additive columns to ``voice_profiles`` backing the
``verified_own_voice`` consent lock (docs/competitive-analysis.md Action 22 /
parity program Wave 0.2). A profile becomes "verified" when its owner records
a spoken consent statement; agentic features and gallery sharing will require
the flag plain local synthesis never does.
* ``verified_own_voice INTEGER DEFAULT 0`` the consent lock itself.
* ``consent_text TEXT DEFAULT ''`` the statement that was read aloud.
* ``consent_audio_path TEXT DEFAULT ''`` filename of the recorded
statement in VOICES_DIR (kept as provenance, deletable via revoke).
* ``consent_recorded_at REAL DEFAULT NULL`` UNIX timestamp.
Behavior mirrors 0002: ``_has_column`` PRAGMA guards make upgrade a no-op on
fresh installs (where _BASE_SCHEMA already has the columns), satisfying the
"Backward-compatible project data" constraint; downgrade drops the columns
(SQLite >= 3.35).
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0003_voice_profile_consent"
down_revision: Union[str, None] = "0002_voice_profile_demo_fields"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_COLUMNS = (
("verified_own_voice", sa.Column("verified_own_voice", sa.Integer(), nullable=False, server_default="0")),
("consent_text", sa.Column("consent_text", sa.Text(), nullable=False, server_default="")),
("consent_audio_path", sa.Column("consent_audio_path", sa.Text(), nullable=False, server_default="")),
("consent_recorded_at", sa.Column("consent_recorded_at", sa.Float(), nullable=True)),
)
def _has_column(table: str, column: str) -> bool:
bind = op.get_bind()
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
return any(r[1] == column for r in rows)
def upgrade() -> None:
for name, column in _COLUMNS:
if not _has_column("voice_profiles", name):
op.add_column("voice_profiles", column)
def downgrade() -> None:
for name, _ in reversed(_COLUMNS):
if _has_column("voice_profiles", name):
op.drop_column("voice_profiles", name)
@@ -0,0 +1,60 @@
"""Parity program Wave 2.2: per-agent MCP voice bindings
Revision ID: 0004_mcp_client_bindings
Revises: 0003_voice_profile_consent
Create Date: 2026-06-12 00:00:00.000000
Adds the ``mcp_client_bindings`` table backing per-agent voice binding
(docs/competitive-analysis.md Spec 2): each MCP client (identified by the
``X-OmniVoice-Client-Id`` header it sends) can be bound to a default voice
profile / engine, so "Claude Code speaks in Morgan, Cursor in Scarlett".
* ``client_id`` TEXT PRIMARY KEY the agent's stable id.
* ``label`` TEXT human label shown in Settings.
* ``profile_id`` TEXT voice profile to speak in (nullable FK-by-convention).
* ``default_engine`` TEXT engine override (nullable).
* ``last_seen_at`` REAL updated when the client calls a tool.
* ``created_at`` REAL.
Additive + idempotent (guarded by sqlite_master), matching 0002/0003, so
re-running on a fresh-install DB where _BASE_SCHEMA already created it is a
no-op (Backward-compatible project data constraint).
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0004_mcp_client_bindings"
down_revision: Union[str, None] = "0003_voice_profile_consent"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _has_table(name: str) -> bool:
bind = op.get_bind()
row = bind.execute(
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
{"n": name},
).fetchone()
return row is not None
def upgrade() -> None:
if _has_table("mcp_client_bindings"):
return
op.create_table(
"mcp_client_bindings",
sa.Column("client_id", sa.Text(), primary_key=True),
sa.Column("label", sa.Text(), nullable=False, server_default=""),
sa.Column("profile_id", sa.Text(), nullable=True),
sa.Column("default_engine", sa.Text(), nullable=True),
sa.Column("last_seen_at", sa.Float(), nullable=True),
sa.Column("created_at", sa.Float(), nullable=True),
)
def downgrade() -> None:
if _has_table("mcp_client_bindings"):
op.drop_table("mcp_client_bindings")
@@ -0,0 +1,65 @@
"""Voice Studio unification: profile `kind` discriminator + design params
Revision ID: 0005_unified_profiles
Revises: 0004_mcp_client_bindings
Create Date: 2026-06-13 00:00:00.000000
Adds two additive columns to ``voice_profiles`` so a *designed* voice
(category sliders + instruct, no user reference audio) is a first-class
profile rather than a transient UI state
(docs/specs/voice-studio-unification.md §3):
* ``kind TEXT DEFAULT 'clone'`` ``'clone'`` (user reference audio) or
``'design'`` (rendered sample + stored design params). Replaces the
brittle is_locked/instruct inference in /generate.
* ``vd_states TEXT DEFAULT NULL`` JSON of the design category picks
(Gender/Age/Pitch/Style/accent/dialect) so selecting a design profile
can restore the sliders for re-editing.
Backfill: every existing row becomes ``kind='clone'`` all of them carry a
real or rendered ``ref_audio_path`` today (archetype materialization
included), so the default is semantically true and no audio is re-rendered.
Behavior mirrors 0002/0003: ``_has_column`` PRAGMA guards make upgrade a
no-op on fresh installs (where _BASE_SCHEMA already has the columns),
satisfying the "Backward-compatible project data" constraint; downgrade
drops the columns (SQLite >= 3.35).
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0005_unified_profiles"
down_revision: Union[str, None] = "0004_mcp_client_bindings"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _has_column(table: str, column: str) -> bool:
bind = op.get_bind()
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
return any(r[1] == column for r in rows)
def upgrade() -> None:
if not _has_column("voice_profiles", "kind"):
op.add_column(
"voice_profiles",
sa.Column("kind", sa.Text(), nullable=False, server_default="clone"),
)
# server_default covers new rows; make existing rows explicit too.
op.execute("UPDATE voice_profiles SET kind='clone' WHERE kind IS NULL OR kind=''")
if not _has_column("voice_profiles", "vd_states"):
op.add_column(
"voice_profiles",
sa.Column("vd_states", sa.Text(), nullable=True),
)
def downgrade() -> None:
if _has_column("voice_profiles", "vd_states"):
op.drop_column("voice_profiles", "vd_states")
if _has_column("voice_profiles", "kind"):
op.drop_column("voice_profiles", "kind")
+42 -1
View File
@@ -25,6 +25,11 @@ class DubSegment(BaseModel):
speed: Optional[float] = None
gain: Optional[float] = None # Per-segment volume (0.0 - 2.0, default 1.0)
target_lang: Optional[str] = None # Per-segment language override (ISO code)
# Phase 4.2 free-form directorial note ("urgent, whispered…"). The client
# has always sent this; without the field pydantic silently dropped it,
# so directions never reached TTS and never entered the regen
# fingerprint (#281).
direction: Optional[str] = None
effect_preset: str = "broadcast" # NEW: DSP preset id (default: broadcast)
@field_validator("effect_preset")
@@ -37,6 +42,19 @@ class DubSegment(BaseModel):
)
return v
class FitOptions(BaseModel):
"""Optional knob overrides for the `smart_fit` timing strategy.
All fields default to None the server fills in the canonical
defaults (services.fit_planner.FitParams) so old clients and sparse
payloads behave identically to a fully-populated default payload.
"""
max_audio_only_rate: Optional[float] = None # default 1.2
audio_rate_cap: Optional[float] = None # default 1.5
video_slow_cap: Optional[float] = None # default 2.0
gap_guard_s: Optional[float] = None # default 0.05
allow_video_retime: Optional[bool] = None # default True
class DubRequest(BaseModel):
segments: List[DubSegment]
language: str = "Auto"
@@ -75,18 +93,35 @@ class DubRequest(BaseModel):
# each segment's video portion is stretched (via
# ffmpeg setpts) to fit the natural-rate dub audio.
# Audio plays at 1.0×; total video duration grows.
# "smart_fit" — dub-length fitting v2: split the burden between a
# mild pitch-preserving audio speed-up (≤1.2× alone,
# ≤1.5× in hybrid) and a mild per-segment video
# slow-down (≤2.0×), per services/fit_planner.py.
# Residual overflow is trimmed and surfaced.
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
# when audio > slot). Kept for back-compat.
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot"]] = "concise"
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot", "smart_fit"]] = "concise"
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
# gap absorption + this much extra time has been consumed.
overflow_budget_s: Optional[float] = 0.0
# Knob overrides for `smart_fit` (ignored by other strategies). Omitted
# fields default server-side to fit_planner.FitParams values.
fit_options: Optional[FitOptions] = None
class TranslateSegment(BaseModel):
id: str
text: str
target_lang: Optional[str] = None
# Free-form delivery direction ("urgent, whispering") — feeds the
# cinematic reflect/adapt prompts. The frontend has sent this since
# Phase 4.2 but pydantic silently dropped it as an undeclared extra,
# so the per-segment direction hint never reached the LLM.
direction: Optional[str] = None
# Available time slot (end - start, seconds) for rate-ratio prediction
# and the cinematic slot-fit pass. Same silent-drop fix as `direction`.
slot_seconds: Optional[float] = None
class TranslateRequest(BaseModel):
segments: List[TranslateSegment]
@@ -96,6 +131,12 @@ class TranslateRequest(BaseModel):
job_id: Optional[str] = None # Dub job id, used to resolve detected source_lang
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect → adapt)
glossary: Optional[List[dict]] = None # [{"source": "...", "target": "...", "note": "..."}]
# Optional regional dialect (BCP-47, e.g. "es-AR", "pt-BR") — #280 item 2.
# Applied by LLM-backed paths (provider="openai" or quality="cinematic"):
# the prompt asks for that region's vocabulary/grammar (e.g. Argentinian
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
# Google) can't honor it; the response then carries dialect_applied=false.
dialect: Optional[str] = None
class DubIngestUrlRequest(BaseModel):
url: str
+281
View File
@@ -0,0 +1,281 @@
"""Acoustic echo cancellation for dictate-over-playback (parity Action 8b).
When the user dictates while OmniVoice is *playing* audio (a TTS preview, a
dub render, a video), the loudspeaker signal leaks back into the microphone.
The streaming ASR on ``/ws/transcribe`` then transcribes that bleed as if it
were speech the "it typed back what the app just said" symptom. A browser's
``getUserMedia({echoCancellation:true})`` would help, but its quality and even
its availability differ per platform/webview, which would make a *default*
feature behave differently on macOS/Windows/Linux against the project's
cross-platform-parity rule. A server-side canceller behaves identically
everywhere, so it is the local-first, platform-neutral choice.
This is an NLMS (normalised least-mean-squares) time-domain adaptive filter
with a Geigel double-talk detector. It is a clean-room-grade *port* of
Patter's ``getpatter/audio/aec.py`` (MIT) — see docs/competitive-analysis.md,
Action 8. It is NOT production-grade DSP (WebRTC AEC3 / Speex AEC are); it is
a dependency-free, good-enough canceller that removes the steady-state echo
the ASR would otherwise hallucinate on.
Wiring (one instance per ``/ws/transcribe`` session NOT thread-safe)::
aec = NlmsEchoCanceller(sample_rate=16000)
# Far-end: every PCM chunk the client is about to play through speakers.
aec.push_far_end(playback_pcm_bytes)
# Near-end: the mic PCM, cleaned before it reaches the ASR buffer.
cleaned = aec.process_near_end(mic_pcm_bytes)
"""
from __future__ import annotations
import logging
import time
from typing import Final
import numpy as np
logger = logging.getLogger("omnivoice.aec")
_DEFAULT_FILTER_TAPS: Final[int] = 512
"""Adaptive-filter length in samples. 512 taps @ 16 kHz = 32 ms, covering a
typical near-field laptop/desktop echo path. Longer tails (large rooms) can
pass ``filter_taps=1024``+ at proportionally more CPU per frame; 512 converges
in ~0.5 s with the warm-up ramp and is the sweet spot for dictation."""
_DEFAULT_STEP_SIZE: Final[float] = 0.1
"""Steady-state NLMS step size. Larger = faster channel tracking but less
stable; 0.1 is the textbook value for narrowband voice."""
_DEFAULT_WARMUP_STEP_SIZE: Final[float] = 0.5
"""Aggressive step used during the warm-up window so the filter reaches a
usable echo estimate within ~0.5 s instead of several seconds. The Geigel
double-talk detector still gates updates, so the bigger step does not learn
the user's own voice as echo."""
_DEFAULT_WARMUP_SECONDS: Final[float] = 0.5
"""Length of the warm-up window. After this many seconds of processed
near-end audio the step decays from ``warmup_step_size`` to ``step_size``."""
_DEFAULT_LEAKAGE: Final[float] = 0.9999
"""Per-iteration weight leakage (slightly < 1) so the filter slowly forgets
stale taps when the echo path drifts (the user moves the mic)."""
_DOUBLE_TALK_RHO: Final[float] = 0.6
"""Geigel double-talk threshold. When ``max(|near|) > rho * max(|far|)`` the
near-end carries energy the far-end cannot explain (the user is talking)
freeze adaptation so the filter does not model the user's voice as echo."""
_FAR_END_BUFFER_SECONDS: Final[float] = 0.5
"""How much past far-end (playback) audio to retain. The echo arrives at the
mic tens of ms after playback; the filter needs that much look-back to align.
500 ms is generous headroom."""
class NlmsEchoCanceller:
"""Time-domain NLMS adaptive filter with Geigel double-talk detection.
Operates on narrowband mono PCM at 16 kHz (the rate the dictation path
resamples to) or 8 kHz. Not thread-safe each ``/ws/transcribe`` session
owns its own instance.
"""
# Far-end staleness window (seconds): once the most recent far-end push
# is older than this, ``process_near_end`` passes the mic through instead
# of cancelling against a frozen reference (which would superimpose the
# same stale ~50 ms waveform on every mic frame as an audible buzz).
_FAR_STALE_S: float = 0.25
def __init__(
self,
sample_rate: int = 16000,
*,
filter_taps: int = _DEFAULT_FILTER_TAPS,
step_size: float = _DEFAULT_STEP_SIZE,
warmup_step_size: float = _DEFAULT_WARMUP_STEP_SIZE,
warmup_seconds: float = _DEFAULT_WARMUP_SECONDS,
leakage: float = _DEFAULT_LEAKAGE,
double_talk_rho: float = _DOUBLE_TALK_RHO,
) -> None:
if sample_rate not in (8000, 16000):
raise ValueError(
"NlmsEchoCanceller supports 8000 Hz or 16000 Hz only; "
f"got {sample_rate}."
)
if filter_taps < 64:
raise ValueError(
f"filter_taps must be >= 64 to model a meaningful echo path; "
f"got {filter_taps}."
)
if not 0 < step_size <= 1:
raise ValueError(f"step_size must be in (0, 1]; got {step_size}.")
if not 0 < warmup_step_size <= 1:
raise ValueError(
f"warmup_step_size must be in (0, 1]; got {warmup_step_size}."
)
if warmup_seconds < 0:
raise ValueError(f"warmup_seconds must be >= 0; got {warmup_seconds}.")
if not 0 < leakage <= 1:
raise ValueError(f"leakage must be in (0, 1]; got {leakage}.")
self._sample_rate = sample_rate
self._taps = filter_taps
self._step = float(step_size)
self._warmup_step = float(warmup_step_size)
self._warmup_samples = int(warmup_seconds * sample_rate)
self._leakage = float(leakage)
self._rho = float(double_talk_rho)
# Counts near-end samples processed so the step can taper from
# warmup_step to step over the first warmup_samples. Counted from the
# first process_near_end call so the window aligns with playback start.
self._processed_samples: int = 0
self._last_far_push_monotonic: float | None = None
# Filter coefficients (zeros — adapts to the channel within ~0.52 s).
self._w = np.zeros(filter_taps, dtype=np.float32)
# Far-end ring buffer holding >= filter_taps samples of playback
# history, with headroom so push/process can interleave freely.
max_buf_samples = max(
filter_taps * 2,
int(sample_rate * _FAR_END_BUFFER_SECONDS),
)
self._far_buf = np.zeros(max_buf_samples, dtype=np.float32)
self._far_write_idx = 0 # next write position (head)
self._far_filled = 0 # samples written so far (capped at len(far_buf))
# Diagnostics only — never read in the hot path.
self.frames_processed: int = 0
self.double_talk_frames: int = 0
# ── Public API ──────────────────────────────────────────────────────────
def push_far_end(self, pcm_bytes: bytes) -> None:
"""Append far-end (playback) audio to the reference ring buffer.
Accepts raw int16 little-endian mono PCM at the configured rate.
"""
if not pcm_bytes:
return
self._last_far_push_monotonic = time.monotonic()
samples = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
n = samples.shape[0]
buf_len = self._far_buf.shape[0]
if n >= buf_len:
# More than the buffer holds — keep only the newest buf_len.
self._far_buf[:] = samples[-buf_len:]
self._far_write_idx = 0
self._far_filled = buf_len
return
end = self._far_write_idx + n
if end <= buf_len:
self._far_buf[self._far_write_idx:end] = samples
else:
head = buf_len - self._far_write_idx
self._far_buf[self._far_write_idx:] = samples[:head]
self._far_buf[: n - head] = samples[head:]
self._far_write_idx = (self._far_write_idx + n) % buf_len
self._far_filled = min(self._far_filled + n, buf_len)
def process_near_end(self, pcm_bytes: bytes) -> bytes:
"""Subtract the estimated echo from the near-end (mic) signal.
Returns int16 little-endian mono PCM with the estimated echo removed.
Passes the frame through unchanged when there is nothing worth
cancelling: no playback has been primed, or the far-end reference is
stale (the app went silent).
"""
if not pcm_bytes:
return pcm_bytes
# Not enough far-end history to fill the filter window yet — passing
# through avoids emitting garbage on the first frames.
if self._far_filled < self._taps:
return pcm_bytes
# Far-end reference is stale (app stopped playing): the ring only
# advances on push_far_end, so the "most recent" window is frozen at
# the tail of the last playback. Convolving against it would buzz.
last_push = self._last_far_push_monotonic
if last_push is None or (time.monotonic() - last_push) > self._FAR_STALE_S:
return pcm_bytes
near = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
cleaned = self._block_nlms(near)
out = np.clip(cleaned * 32768.0, -32768.0, 32767.0).astype(np.int16)
self.frames_processed += 1
return out.tobytes()
def reset(self) -> None:
"""Clear filter coefficients and far-end history (e.g. on a new turn)."""
self._w.fill(0)
self._far_buf.fill(0)
self._far_write_idx = 0
self._far_filled = 0
self._processed_samples = 0
self._last_far_push_monotonic = None
self.frames_processed = 0
self.double_talk_frames = 0
# ── Internals ───────────────────────────────────────────────────────────
def _far_window(self, length: int) -> np.ndarray:
"""Most recent ``length`` far-end samples, oldest first / newest last."""
buf_len = self._far_buf.shape[0]
if length > self._far_filled:
length = self._far_filled
end = self._far_write_idx # newest sample is at (end - 1) mod buf_len
if end >= length:
return self._far_buf[end - length: end]
head = self._far_buf[buf_len - (length - end):]
tail = self._far_buf[:end]
return np.concatenate((head, tail))
def _block_nlms(self, near: np.ndarray) -> np.ndarray:
"""Sample-by-sample NLMS over one frame of near-end samples.
Classical NLMS depends on the weights adapted at the previous sample,
so the inner loop is sequential. Each sample is O(taps); numpy keeps a
320-sample / 512-tap frame well under a millisecond on commodity CPUs.
"""
taps = self._taps
far_window = self._far_window(taps + near.shape[0] - 1)
if far_window.shape[0] < taps + near.shape[0] - 1:
# Still warming up — left-pad with zeros so indices line up.
pad = np.zeros(
taps + near.shape[0] - 1 - far_window.shape[0], dtype=np.float32
)
far_window = np.concatenate((pad, far_window))
# Geigel double-talk detector (frame-wise).
far_max = float(np.max(np.abs(far_window))) if far_window.size else 0.0
near_max = float(np.max(np.abs(near)))
# Freeze adaptation when the far reference is effectively silent
# (<= -60 dBFS): adapting against a fade-out tail with near-zero norm
# blows the weights up against user speech when playback resumes.
if far_max <= 1e-3:
return near
double_talk = near_max > self._rho * far_max
if double_talk:
self.double_talk_frames += 1
out = np.empty_like(near)
w = self._w
leakage = self._leakage
# Constant step within the frame keeps the inner loop branch-free.
if self._processed_samples < self._warmup_samples:
step = self._warmup_step
else:
step = self._step
for i in range(near.shape[0]):
x = far_window[i: i + taps]
y_est = float(np.dot(w, x))
e = float(near[i] - y_est)
out[i] = e
if not double_talk:
# NLMS update with leakage. +1e-6 guards divide-by-zero.
norm = float(np.dot(x, x)) + 1e-6
w *= leakage
w += (step * e / norm) * x
self._processed_samples += near.shape[0]
return out
+232 -11
View File
@@ -31,12 +31,74 @@ from abc import ABC, abstractmethod
logger = logging.getLogger("omnivoice.asr")
def _decode_audio_16k_mono(audio_path: str):
"""Decode `audio_path` to a 16 kHz mono float32 waveform using OmniVoice's
*validated* ffmpeg, instead of whisperx.load_audio's bare ``"ffmpeg"`` PATH
lookup.
whisperx (and openai-whisper) shell out to a literal ``"ffmpeg"`` resolved
against the OS PATH. On Windows that resolves to whatever the system finds
first a WindowsApps alias stub or a corrupt/wrong-arch download which
passes `which` but explodes at spawn with ``[WinError 193] %1 is not a valid
Win32 application``. whisperx only catches `CalledProcessError`, so the
spawn-time `OSError` escapes and the dub/batch path reports the opaque
"Transcription produced no segments" (#479). ``find_ffmpeg()`` probes each
candidate with ``-version`` and returns a runnable binary (the bundled
imageio-ffmpeg / Tauri sidecar) or None, so we can raise an actionable
error. This also fixes the imageio case a PATH-prepend can't: its binary is
named ``ffmpeg-<plat>-vN.exe``, not ``ffmpeg``, so bare lookup never finds
it. Mirrors whisperx.audio.load_audio's command exactly (16 kHz, mono, s16le).
"""
import subprocess
import numpy as np
from services.ffmpeg_utils import find_ffmpeg
ffmpeg = find_ffmpeg()
if not ffmpeg:
raise RuntimeError(
"Cannot transcribe: ffmpeg is missing or not runnable. Install "
"ffmpeg (or let OmniVoice's bundled binary download), then retry. "
"On Windows a '[WinError 193]' here means the ffmpeg binary is "
"corrupt or the wrong architecture — reinstall it or clear the "
"imageio-ffmpeg cache."
)
cmd = [
ffmpeg, "-nostdin", "-threads", "0", "-i", audio_path,
"-f", "s16le", "-ac", "1", "-acodec", "pcm_s16le", "-ar", "16000", "-",
]
try:
out = subprocess.run(cmd, capture_output=True, check=True).stdout
except OSError as e:
# Belt-and-suspenders: find_ffmpeg() already -version-validated this
# binary, so a WinError 193 here is unexpected — surface it clearly
# rather than letting it become "no segments".
raise RuntimeError(
f"ffmpeg at {ffmpeg!r} could not be executed ({e}). Reinstall "
"ffmpeg or clear the imageio-ffmpeg cache."
) from e
except subprocess.CalledProcessError as e:
stderr = (e.stderr or b"").decode(errors="replace")[:500]
raise RuntimeError(f"Failed to decode audio for transcription: {stderr}") from e
return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0
# ── Protocol ────────────────────────────────────────────────────────────────
class ASRBackend(ABC):
id: str = "base"
display_name: str = "Base ASR"
# Accelerator families this backend can use, in preference order; always
# includes a fallback. Subset of {cuda, rocm, mps, xpu, cpu}. Mirrors the
# TTSBackend.gpu_compat contract so engine_routing.resolve_routing() can
# surface the effective device per host (no silent CPU fallback). The
# conservative default is CPU-only; subclasses declare what they really run
# on. (ROCm is intentionally NOT claimed yet for any ASR engine — see the
# per-engine notes; an unverified `rocm` claim would route ROCm hosts to a
# broken GPU path, strictly worse than the honest `cpu_fallback`.)
gpu_compat: tuple[str, ...] = ("cpu",)
@classmethod
@abstractmethod
@@ -61,6 +123,10 @@ class ASRBackend(ABC):
class WhisperXBackend(ASRBackend):
id = "whisperx"
display_name = "WhisperX (faster-whisper + wav2vec2 forced alignment)"
# CTranslate2 backend: CUDA fp16 or CPU int8 (see _pick_device). ROCm not
# claimed — CTranslate2 has no upstream HIP build, so a ROCm host honestly
# gets cpu_fallback rather than a false GPU promise.
gpu_compat = ("cuda", "cpu")
def __init__(self):
self._model_name = os.environ.get("ASR_MODEL_WHISPERX", "large-v3")
@@ -313,10 +379,13 @@ class WhisperXBackend(ASRBackend):
return None
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
import whisperx
import whisperx # used for whisperx.align() below
self._ensure_asr()
logger.info("whisperx transcribing %s (word_timestamps=%s)", audio_path, word_timestamps)
audio = whisperx.load_audio(audio_path)
# Decode via OmniVoice's validated ffmpeg, NOT whisperx.load_audio's bare
# "ffmpeg" PATH lookup which yields [WinError 193] -> "no segments" on
# Windows (#479). Same 16 kHz mono s16le array whisperx expects.
audio = _decode_audio_16k_mono(audio_path)
try:
result = self._asr.transcribe(audio)
except IndexError:
@@ -382,6 +451,8 @@ class WhisperXBackend(ASRBackend):
class FasterWhisperBackend(ASRBackend):
id = "faster-whisper"
display_name = "Faster-Whisper (CTranslate2 — Linux/Windows/macOS)"
# CTranslate2: CUDA or CPU (no upstream ROCm/HIP build — see WhisperX note).
gpu_compat = ("cuda", "cpu")
def __init__(self):
# Defaulting to the CTranslate2-converted large-v3 repo. Matches
@@ -497,6 +568,7 @@ _MLX_MODEL_TURBO = "mlx-community/whisper-large-v3-turbo"
class MLXWhisperBackend(ASRBackend):
id = "mlx-whisper"
display_name = "MLX Whisper (Apple Silicon CoreML)"
gpu_compat = ("mps", "cpu")
def __init__(self, model_name: str | None = None):
self._model_name = model_name or os.environ.get(
@@ -505,14 +577,23 @@ class MLXWhisperBackend(ASRBackend):
@classmethod
def is_available(cls) -> tuple[bool, str]:
# #390: shared platform gate FIRST — one rule for MLX-Audio + MLX-Whisper.
# Returns False on Linux/Windows/mac-Intel before any package import, so
# a stray mlx-whisper wheel never reports available or advertises `mps`.
from core.device_caps import mlx_supported
ok, why = mlx_supported()
if not ok:
return False, why
try:
import torch
if not (hasattr(torch.backends, "mps") and torch.backends.mps.is_available()):
return False, "Apple Silicon (MPS) not available."
import mlx_whisper # noqa: F401
return True, "ready"
except ImportError as e:
return False, f"mlx-whisper not installed: {e}"
# Catch OSError/RuntimeError too, not just ImportError: in a
# PyInstaller bundle mlx's native dylib/metallib can fail to load
# even when the package imports, raising OSError/RuntimeError. We must
# report unavailable (so the picker falls back) rather than crash the
# registry scan (Wave 4.4).
except (ImportError, OSError, RuntimeError) as e:
return False, f"mlx-whisper unavailable: {e}"
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
import mlx_whisper
@@ -561,6 +642,9 @@ class MLXWhisperBackend(ASRBackend):
class PyTorchWhisperBackend(ASRBackend):
id = "pytorch-whisper"
display_name = "PyTorch Whisper (CUDA / CPU via transformers pipeline)"
# Pure transformers pipeline → runs wherever torch does (CUDA, MPS, CPU).
# ROCm-via-HIP would also work but is left unclaimed pending verification.
gpu_compat = ("cuda", "mps", "cpu")
def __init__(self, asr_pipe=None):
# Reuses the `_asr_pipe` attached to the TTS model when available.
@@ -633,6 +717,11 @@ class NeMoASRBackend(ASRBackend):
Requires NVIDIA GPU.
"""
id = "nemo-parakeet"
# CUDA-only: is_available() hard-fails without a GPU ("Parakeet TDT requires
# NVIDIA GPU (CUDA)"), so declaring a CPU path would be a false claim. On a
# CPU host this correctly resolves to routing_status="unavailable", matching
# is_available()=False (the matrix suppresses the routing badge there).
gpu_compat = ("cuda",)
display_name = "Parakeet TDT (NVIDIA NeMo — English SOTA)"
def __init__(self):
@@ -741,6 +830,7 @@ class MoonshineASRBackend(ASRBackend):
Great for live capture and CPU-only environments.
"""
id = "moonshine"
gpu_compat = ("cpu",) # edge/CPU-optimized by design
display_name = "Moonshine (edge-optimized, ONNX)"
def __init__(self):
@@ -885,6 +975,7 @@ class FunASRBackend(ASRBackend):
#182); WhisperX remains the cross-platform default.
"""
id = "funasr"
gpu_compat = ("cuda", "cpu") # FunASR: CUDA or CPU
display_name = "FunASR (SenseVoice — 50+ languages, all-in-one)"
def __init__(self):
@@ -931,7 +1022,47 @@ class FunASRBackend(ASRBackend):
pass
_REGISTRY: dict[str, type[ASRBackend]] = {
def _isolated_faster_whisper():
"""Lazy import so the subprocess_asr → subprocess_backend chain isn't
pulled in at registry definition time."""
from services.subprocess_asr import IsolatedFasterWhisperBackend
return IsolatedFasterWhisperBackend
class _LazyASRRegistry(dict):
"""Registry with one lazily-resolved entry (Wave 4.2). Mirrors the TTS
registry's lazy pattern so listing/selecting the crash-isolated ASR
backend doesn't import the subprocess stack unless it's used."""
_LAZY = {"faster-whisper-isolated": _isolated_faster_whisper}
def __contains__(self, key):
return dict.__contains__(self, key) or key in self._LAZY
def __getitem__(self, key):
if dict.__contains__(self, key):
return dict.__getitem__(self, key)
if key in self._LAZY:
cls = self._LAZY[key]()
self[key] = cls
return cls
raise KeyError(key)
def __iter__(self):
seen = set()
for k in dict.__iter__(self):
seen.add(k)
yield k
for k in self._LAZY:
if k not in seen:
yield k
def items(self):
for k in self:
yield k, self[k]
_REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
"whisperx": WhisperXBackend,
"faster-whisper": FasterWhisperBackend,
"mlx-whisper": MLXWhisperBackend,
@@ -939,18 +1070,70 @@ _REGISTRY: dict[str, type[ASRBackend]] = {
"nemo-parakeet": NeMoASRBackend,
"moonshine": MoonshineASRBackend,
"funasr": FunASRBackend,
# "faster-whisper-isolated": resolved lazily (crash-isolated subprocess).
})
# Short install hints surfaced as tooltips on the Settings → Engines UI
# (parity with tts_backend._INSTALL_HINTS).
_INSTALL_HINTS: dict[str, str] = {
"whisperx": "pip install whisperx (CTranslate2 + wav2vec2 alignment; CUDA or CPU)",
"faster-whisper": "pip install faster-whisper (CTranslate2; cross-platform, CUDA or CPU)",
"mlx-whisper": "pip install mlx-whisper (Apple Silicon only)",
"pytorch-whisper": "Bundled with transformers — no extra install (CUDA/MPS/CPU)",
"nemo-parakeet": "pip install nemo_toolkit[asr] (NVIDIA Parakeet; CUDA or CPU)",
"moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)",
"funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)",
}
# Most-recent failure per backend, so a transient probe error survives between
# Settings refreshes (parity with tts_backend._LAST_ERRORS).
_LAST_ERRORS: dict[str, str] = {}
def list_backends() -> list[dict]:
out = []
"""Enumerate every ASR backend with the **same 11-key shape as TTS** so the
Engine Compatibility Matrix renders all families uniformly.
Per-entry: id, display_name, available, reason (scrubbed), install_hint,
last_error, isolation_mode, gpu_compat, effective_device, routing_status,
routing_reason. A backend whose ``is_available()`` raises is reported
``available: false`` (never a 500), exactly like TTS.
"""
from core.device_caps import detect_host_caps
from core.scrub import scrub_text
from services.engine_routing import routing_fields
caps = detect_host_caps()
out: list[dict] = []
for bid, cls in _REGISTRY.items():
ok, msg = cls.is_available()
try:
ok, msg = cls.is_available()
except Exception as exc:
ok = False
msg = f"{type(exc).__name__}: {exc}"
logger.warning(
"asr list_backends: %s.is_available() raised — degrading "
"gracefully so the picker still renders: %s", bid, msg,
)
if ok:
_LAST_ERRORS.pop(bid, None)
else:
_LAST_ERRORS[bid] = scrub_text(msg)
isolation = "subprocess" if getattr(cls, "_is_subprocess_isolated", False) else "in-process"
gpu_compat = getattr(cls, "gpu_compat", ("cpu",))
out.append({
"id": bid,
"display_name": cls.display_name,
"available": ok,
"reason": None if ok else msg,
# ASR previously emitted `reason` UNMASKED — scrub it now (closes a
# pre-existing token-leak gap, matching TTS's redaction guarantee).
"reason": None if ok else scrub_text(msg),
"install_hint": _INSTALL_HINTS.get(bid),
"last_error": _LAST_ERRORS.get(bid),
"isolation_mode": isolation,
"gpu_compat": list(gpu_compat),
**routing_fields(gpu_compat, caps),
})
return out
@@ -1016,6 +1199,44 @@ def get_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
return _REGISTRY[bid]()
def transcribe_reference(audio_path: str) -> str | None:
"""Transcribe a voice-clone reference clip with the active ASR backend.
Voice cloning without a user-supplied transcript used to fall through to
``OmniVoice.load_asr_model()`` a transformers ``pipeline()`` load of
whisper-large-v3-turbo that fails outright on transformers 5.3 (#308),
even when whisperx / faster-whisper / mlx-whisper are installed and
working. Route the reference transcript through the registry instead, so
the model-attached pipeline is only reached when it is genuinely the last
resort. Returns ``None`` on any failure callers pass ``ref_text=None``
through and the model's built-in fallback still gets its chance.
"""
try:
backend = get_active_asr_backend()
except Exception as e: # noqa: BLE001 — never let ASR break generation
logger.warning("transcribe_reference: no ASR backend available (%s)", e)
return None
if isinstance(backend, PyTorchWhisperBackend):
# The registry fell through to the model-attached pipeline; let the
# model load it lazily rather than constructing a second copy here.
return None
try:
result = backend.transcribe(audio_path, word_timestamps=False)
except Exception as e: # noqa: BLE001 — degrade to the model fallback
logger.warning(
"transcribe_reference: %s failed (%s) — deferring to the model's "
"built-in ASR fallback",
backend.id, e,
)
return None
result = result or {}
text = result.get("text") or " ".join(
(seg.get("text") or "").strip() for seg in result.get("segments", [])
)
text = (text or "").strip()
return text or None
_capture_backend: ASRBackend | None = None
+170
View File
@@ -0,0 +1,170 @@
"""Audiobook creator — chapterized long-form narration (parity Wave 5).
Turns a chapter-delimited script into a chapterized audiobook. This module is
the engine-agnostic core:
* ``parse_audiobook_script`` pure parser: Markdown ``# H1`` headings become
chapters; inline ``[voice:NAME]`` switches the narrator; ``[pause ]`` is
delegated to the existing :func:`omnivoice.utils.text.parse_pause_markers`
so audiobooks and single-shot synthesis share one pause dialect.
* ``synthesize_chapter`` orchestration: renders a chapter's spans through an
injected ``synth(text, voice_id) -> tensor`` callable (reusing the
``chunked_tts`` splitter + crossfade), stitching the inter-span silences.
Injecting the synth keeps this unit-testable with a stub backend (no torch
model, no GPU).
* ``build_chapter_ffmetadata`` / ``build_m4b_cmd`` pure builders for the
ffmpeg chapterized-m4b mux (FFMETADATA1 ``[CHAPTER]`` blocks + concat-demux
argv). The actual ffmpeg run lives in the (impure) caller.
Scope (first cut): plain chapter-delimited text/Markdown input. epub/pdf
ingestion, the streaming synth job + UI are deferred follow-ups.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Callable, Optional
@dataclass
class Span:
"""One contiguous run of text in a single voice, plus trailing silence.
``speed`` (when set) is the per-span rate passed to the engine Stories'
per-line speed slider rides through here so the shared server render honours
it the way the old client export did.
"""
voice_id: Optional[str]
text: str
pause_ms_after: int = 0
speed: Optional[float] = None
def to_dict(self) -> dict:
return {"voice_id": self.voice_id, "text": self.text,
"pause_ms_after": self.pause_ms_after, "speed": self.speed}
@dataclass
class Chapter:
title: str
spans: list[Span] = field(default_factory=list)
@property
def char_count(self) -> int:
return sum(len(s.text) for s in self.spans)
def to_dict(self) -> dict:
return {"title": self.title, "char_count": self.char_count,
"spans": [s.to_dict() for s in self.spans]}
@dataclass
class AudiobookPlan:
chapters: list[Chapter] = field(default_factory=list)
@property
def char_count(self) -> int:
return sum(c.char_count for c in self.chapters)
def to_dict(self) -> dict:
return {
"chapters": [c.to_dict() for c in self.chapters],
"chapter_count": len(self.chapters),
"char_count": self.char_count,
}
def parse_audiobook_script(text: str, *, default_voice: Optional[str] = None) -> AudiobookPlan:
"""Parse a chapter-delimited script into an :class:`AudiobookPlan`.
Thin wrapper over the canonical :func:`services.longform_parser.
parse_script_to_spans` (the single grammar source of truth, #27); wraps its
span dicts in the ``Span``/``Chapter``/``AudiobookPlan`` dataclasses so the
four router call sites and ``.to_dict()`` shape are unchanged.
"""
from services.longform_parser import parse_script_to_spans
chapters = [
Chapter(title=c["title"], spans=[Span(**s) for s in c["spans"]])
for c in parse_script_to_spans(text, default_voice=default_voice)
]
return AudiobookPlan(chapters=chapters)
def synthesize_chapter(
spans: list[Span],
synth: Callable[[str, Optional[str], Optional[float]], "object"],
sample_rate: int,
*,
crossfade_ms: int = 50,
lexicon: Optional[dict] = None,
):
"""Render a chapter's spans to one waveform via an injected ``synth``.
``synth(text, voice_id, speed)`` returns a 1-D float32 audio tensor for a
span of text in the given voice (``speed`` may be ``None`` for the engine
default). Long spans are split with the ``chunked_tts`` splitter and
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
given) respells each span's text before chunking so the engine pronounces
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
Returns ``(audio_tensor, duration_seconds)``. torch + chunked_tts are
imported lazily so this module stays import-light for the pure parser path.
"""
import torch
from services.chunked_tts import concatenate_audio_chunks, split_text_into_chunks
from services.pronunciation import apply_lexicon
parts: list = []
for span in spans:
if span.text:
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
if len(rendered) == 1:
parts.append(rendered[0])
elif rendered:
parts.append(concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms))
if span.pause_ms_after > 0:
n = int(sample_rate * span.pause_ms_after / 1000.0)
if n > 0:
parts.append(torch.zeros(n, dtype=torch.float32))
if not parts:
return torch.zeros(0, dtype=torch.float32), 0.0
# Hard-concat spans + silences (crossfading silence would bleed the gap).
audio = parts[0] if len(parts) == 1 else concatenate_audio_chunks(parts, sample_rate, crossfade_ms=0)
return audio, audio.shape[-1] / float(sample_rate)
# ── ffmpeg / metadata builders ──────────────────────────────────────────────
#
# These now live in the shared ``longform_render`` core (Stories + Audiobook
# converge on one mux). The thin wrappers below preserve the original
# audiobook-only call sites/signatures; new callers should use
# ``longform_render`` directly to reach global metadata, cover art, loudness,
# and mp3 output.
from services.longform_render import ( # noqa: E402
build_concat_list,
build_ffmetadata,
build_render_cmd,
)
def build_chapter_ffmetadata(chapters: list[tuple[str, int]]) -> str:
"""Backward-compatible alias: chapters-only FFMETADATA (no global tags)."""
return build_ffmetadata(chapters)
def build_m4b_cmd(
ffmpeg: str,
concat_list_path: str,
metadata_path: str,
out_path: str,
*,
bitrate: str = "128k",
) -> list[str]:
"""Backward-compatible alias: a chapterized faststart m4b, no cover/loudness."""
return build_render_cmd(
ffmpeg, concat_list_path, metadata_path, out_path,
fmt="m4b", bitrate=bitrate,
)
+3
View File
@@ -151,6 +151,9 @@ async def generate_segments_batched(
# Raw: skip all DSP — return raw model output
return audio_out
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(audio_out, sample_rate=sr)
effect_chain = get_effect_chain(seg_effect_preset)
if effect_chain:
+174
View File
@@ -0,0 +1,174 @@
"""Chunked TTS generation utilities (Wave 1.2 — unlimited-length generation).
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
Copyright (c) voicebox contributors. The concatenation half is reworked for
torch tensors (our inference helpers pass raw model output possibly
multi-channel to the effect chain), and the sample rate comes from the
engine's declared rate rather than the first chunk (fixes a latent upstream
bug where a mid-run rate change was silently ignored).
Splits long text into sentence-boundary chunks and joins the per-chunk audio
with a short crossfade. Pure functions the generation loop itself lives in
``api/routers/generation.py`` next to the existing ``[pause]`` span stitcher,
so this module stays unit-testable without a model.
Short text (<= max_chunk_chars) never reaches this module's concat path; the
callers keep their unchanged single-shot fast path.
"""
from __future__ import annotations
import logging
import re
from typing import List
logger = logging.getLogger("omnivoice.chunked_tts")
# Default chunk size in characters. 0 disables chunking entirely.
DEFAULT_MAX_CHUNK_CHARS = 800
# Default crossfade between chunks. 0 = hard cut.
DEFAULT_CROSSFADE_MS = 50
# Common abbreviations that should NOT be treated as sentence endings.
# Lowercase for case-insensitive matching.
_ABBREVIATIONS = frozenset({
"mr", "mrs", "ms", "dr", "prof", "sr", "jr", "st", "ave", "blvd",
"inc", "ltd", "corp", "dept", "est", "approx", "vs", "etc",
"e.g", "i.e", "a.m", "p.m", "u.s", "u.s.a", "u.k",
})
# Inline bracket tags (paralinguistic tags like [laugh]; our own
# [pause 300ms] markers). The splitter must never cut inside one.
_BRACKET_TAG_RE = re.compile(r"\[[^\]]*\]")
def split_text_into_chunks(text: str, max_chars: int = DEFAULT_MAX_CHUNK_CHARS) -> List[str]:
"""Split *text* at natural boundaries into chunks of at most *max_chars*.
Priority: sentence-end (``.!?`` not after an abbreviation/decimal and not
inside brackets, plus fullwidth equivalents) -> clause boundary
(``;:,`` / em dash) -> whitespace -> hard cut that avoids splitting a
``[tag]``.
"""
text = text.strip()
if not text:
return []
if max_chars <= 0 or len(text) <= max_chars:
return [text]
chunks: List[str] = []
remaining = text
while remaining:
remaining = remaining.lstrip()
if not remaining:
break
if len(remaining) <= max_chars:
chunks.append(remaining)
break
segment = remaining[:max_chars]
split_pos = _find_last_sentence_end(segment)
if split_pos == -1:
split_pos = _find_last_clause_boundary(segment)
if split_pos == -1:
split_pos = segment.rfind(" ")
if split_pos == -1:
split_pos = _safe_hard_cut(segment, max_chars)
chunk = remaining[: split_pos + 1].strip()
if chunk:
chunks.append(chunk)
remaining = remaining[split_pos + 1:]
return chunks
def _find_last_sentence_end(text: str) -> int:
"""Index of the last sentence-ending punctuation, or -1.
Skips periods after common abbreviations and decimals, anything inside
a bracket tag, and also recognizes fullwidth sentence punctuation
(ideographic full stop / fullwidth ! and ?) for no-space scripts.
"""
best = -1
for m in re.finditer(r"[.!?](?:\s|$)", text):
pos = m.start()
if text[pos] == ".":
word_start = pos - 1
while word_start >= 0 and text[word_start].isalpha():
word_start -= 1
word = text[word_start + 1: pos].lower()
if word in _ABBREVIATIONS:
continue
if word_start >= 0 and text[word_start].isdigit():
continue
if _inside_bracket_tag(text, pos):
continue
best = pos
# Fullwidth sentence enders (ideographic full stop, fullwidth !, ?)
# written as escapes to keep the repo's no-literal-CJK gate clean.
for m in re.finditer("[\u3002\uff01\uff1f]", text):
if m.start() > best:
best = m.start()
return best
def _find_last_clause_boundary(text: str) -> int:
best = -1
for m in re.finditer(r"[;:,—](?:\s|$)", text):
if _inside_bracket_tag(text, m.start()):
continue
best = m.start()
return best
def _inside_bracket_tag(text: str, pos: int) -> bool:
for m in _BRACKET_TAG_RE.finditer(text):
if m.start() < pos < m.end():
return True
return False
def _safe_hard_cut(segment: str, max_chars: int) -> int:
cut = max_chars - 1
for m in _BRACKET_TAG_RE.finditer(segment):
if m.start() < cut < m.end():
return m.start() - 1 if m.start() > 0 else cut
return cut
def concatenate_audio_chunks(chunks: list, sample_rate: int,
crossfade_ms: int = DEFAULT_CROSSFADE_MS):
"""Join per-chunk waveforms with a linear crossfade on the sample axis.
``chunks`` are torch tensors as returned by the engine (1-D, or N-D with
samples on the last axis matching what ``_render_with_pauses`` handles).
Crossfade overlap is clamped to the shorter neighbor; ``crossfade_ms=0``
is a hard concat.
"""
import torch
chunks = [c for c in chunks if c is not None and c.shape[-1] > 0]
if not chunks:
return torch.zeros(1, dtype=torch.float32)
if len(chunks) == 1:
return chunks[0]
crossfade_samples = int(sample_rate * crossfade_ms / 1000)
result = chunks[0]
for chunk in chunks[1:]:
chunk = chunk.to(device=result.device, dtype=result.dtype)
overlap = min(crossfade_samples, result.shape[-1], chunk.shape[-1])
if overlap > 0:
fade_out = torch.linspace(1.0, 0.0, overlap, dtype=result.dtype, device=result.device)
fade_in = torch.linspace(0.0, 1.0, overlap, dtype=result.dtype, device=result.device)
blended = result[..., -overlap:] * fade_out + chunk[..., :overlap] * fade_in
result = torch.cat([result[..., :-overlap], blended, chunk[..., overlap:]], dim=-1)
else:
result = torch.cat([result, chunk], dim=-1)
return result
+19 -40
View File
@@ -13,7 +13,8 @@ What's here
* **Content-hash cache lookup** `compute_file_hash`, `find_cached_job`.
* **Safe path resolution** `safe_job_dir`.
* **Process lifecycle** ffmpeg/demucs subprocess tracking + `kill_job_procs`
so `POST /dub/abort/{id}` can tear down in-flight work.
so `POST /dub/abort/{id}` can tear down in-flight work (implemented in
`services.proc_registry`, re-exported here for compatibility).
* **SSE helpers** `sse_event`, `prep_event`.
What stays in the router
@@ -44,6 +45,17 @@ from core.config import DUB_DIR
from fastapi import HTTPException
from services.ffmpeg_utils import find_ffmpeg, find_ffprobe, _get_semaphore, _spawn_with_retry
from services.model_manager import get_best_device
# Process lifecycle moved to its own leaf module so ffmpeg_utils can import
# it at module top (no dub_pipeline ↔ ffmpeg_utils cycle). Re-exported here —
# dub_core and tests still alias these names through this module.
from services.proc_registry import ( # noqa: F401 — re-exports
_active_procs,
_active_procs_lock,
has_active_procs,
kill_job_procs,
register_proc,
unregister_proc,
)
from core.db import db_conn
from core import event_bus
from core import failure
@@ -56,8 +68,6 @@ logger = logging.getLogger("omnivoice.dub_pipeline")
_dub_jobs: dict[str, dict] = {}
_dub_jobs_lock = threading.Lock()
_active_procs: dict[str, list] = {}
_active_procs_lock = threading.Lock()
_DUB_DIR_REAL = os.path.realpath(DUB_DIR)
_HASH_BUF_SIZE = 1 << 18 # 256 KB chunks for hashing
@@ -138,42 +148,8 @@ def find_cached_job(content_hash: str, exclude_job_id: str) -> Optional[dict]:
# ── Process lifecycle ───────────────────────────────────────────────────────
def register_proc(job_id: str, proc) -> None:
"""Track an in-flight subprocess so /dub/abort can kill it."""
with _active_procs_lock:
_active_procs.setdefault(job_id, []).append(proc)
def unregister_proc(job_id: str, proc) -> None:
with _active_procs_lock:
lst = _active_procs.get(job_id)
if lst and proc in lst:
lst.remove(proc)
if lst is not None and not lst:
_active_procs.pop(job_id, None)
def kill_job_procs(job_id: str) -> None:
"""Kill every subprocess still running under a given job id. Idempotent."""
with _active_procs_lock:
procs = list(_active_procs.get(job_id, []))
for proc in procs:
try:
if proc.returncode is None:
proc.kill()
except ProcessLookupError:
pass
except Exception as e:
logger.warning("Failed to kill subprocess for %s: %s", job_id, e)
with _active_procs_lock:
_active_procs.pop(job_id, None)
def has_active_procs(job_id: str) -> bool:
with _active_procs_lock:
return bool(_active_procs.get(job_id))
# register_proc / unregister_proc / kill_job_procs / has_active_procs live in
# services.proc_registry (imported + re-exported above).
# ── Job state (in-memory + SQLite fallback) ────────────────────────────────
@@ -195,7 +171,10 @@ def get_job(job_id: str) -> Optional[dict]:
_dub_jobs[job_id] = job
return job
except json.JSONDecodeError as e:
logger.error("Failed to decode dub_history.job_data for %s: %s", job_id, e)
# job_id arrives from request paths — strip newlines so a crafted
# id can't forge extra log lines (py/log-injection).
safe_id = str(job_id).replace("\r", "").replace("\n", "")
logger.error("Failed to decode dub_history.job_data for %s: %s", safe_id, e)
return None
+118
View File
@@ -0,0 +1,118 @@
"""Second-pass ASR quality control for dubs (Wave 3.3 / Spec 5).
After a dub is generated, re-recognize the synthetic audio and compare what
the ASR *heard* against what we asked the TTS to *say*. Where the two drift
apart, the line is flagged for the user to verify turning subtitle timing
and pronunciation from "trusted math" into "measured truth", and doubling as
an automatic dub-quality check.
Design delta from pyvideotrans (whose second pass lets recognized text
*replace* the subtitles wholesale): we keep the GENERATED text authoritative
for content and use the second pass for *measurement* timing + a drift
score that feeds the incremental re-dub loop, never silently overwriting the
translation.
Pure functions here (no ASR, no I/O) so the scoring is unit-testable; the
pipeline stage that runs the ASR pass lives in the dub router.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
def _tokens(text: str) -> list[str]:
"""Lowercase word tokens, punctuation stripped — the unit drift is scored
in. Script-agnostic: for no-space scripts each character is a token, which
still gives a sensible edit-distance ratio."""
text = (text or "").lower().strip()
if not text:
return []
words = re.findall(r"\w+", text, flags=re.UNICODE)
return words or list(text.replace(" ", ""))
def _edit_distance(a: list[str], b: list[str]) -> int:
"""Levenshtein distance between two token lists (iterative, O(len(a)*len(b))
time, O(len(b)) space)."""
if not a:
return len(b)
if not b:
return len(a)
prev = list(range(len(b) + 1))
for i, ta in enumerate(a, 1):
cur = [i]
for j, tb in enumerate(b, 1):
cost = 0 if ta == tb else 1
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + cost))
prev = cur
return prev[-1]
def word_error_rate(reference: str, hypothesis: str) -> float:
"""Normalized token edit distance in [0.0, 1.0+].
0.0 = the ASR heard exactly the target text. ~1.0 = entirely different.
Can exceed 1.0 when the hypothesis is much longer than the reference
(insertions); callers clamp/threshold as needed. An empty reference with a
non-empty hypothesis scores 1.0 (everything is an insertion)."""
ref = _tokens(reference)
hyp = _tokens(hypothesis)
if not ref and not hyp:
return 0.0
if not ref:
return 1.0
return _edit_distance(ref, hyp) / len(ref)
@dataclass
class SegmentQC:
seg_id: str
target_text: str
recognized_text: str
drift: float # word_error_rate(target, recognized)
flagged: bool # drift >= threshold
new_start: float | None # measured onset from the dubbed-audio recognition
new_end: float | None
def _overlap(a0: float, a1: float, b0: float, b1: float) -> float:
return max(0.0, min(a1, b1) - max(a0, b0))
def score_dub(
dub_segments: list[dict],
recognized: list[dict],
*,
drift_threshold: float = 0.5,
seg_ids: list | None = None,
) -> list[SegmentQC]:
"""Match the second-pass recognition to the dub segments and score drift.
``dub_segments`` are the segments we generated (each {start, end, text});
``recognized`` are the ASR result segments on the dubbed audio (each
{start, end, text}). Each dub segment is matched to the recognized
segment(s) it overlaps in time; their text is concatenated as the
hypothesis and scored against the dub segment's ``text``. The recognized
span's bounds become the measured start/end (subtitle-timing truth).
"""
results: list[SegmentQC] = []
for i, seg in enumerate(dub_segments):
sid = str(seg_ids[i]) if (seg_ids and i < len(seg_ids)) else str(seg.get("id", i))
s0, s1 = float(seg.get("start", 0.0)), float(seg.get("end", 0.0))
hits = [r for r in recognized if _overlap(s0, s1, float(r.get("start", 0.0)), float(r.get("end", 0.0))) > 0]
hyp = " ".join((r.get("text") or "").strip() for r in hits).strip()
drift = word_error_rate(seg.get("text", ""), hyp)
new_start = min((float(r.get("start", 0.0)) for r in hits), default=None)
new_end = max((float(r.get("end", 0.0)) for r in hits), default=None)
results.append(SegmentQC(
seg_id=sid,
target_text=(seg.get("text") or "").strip(),
recognized_text=hyp,
drift=round(drift, 3),
flagged=drift >= drift_threshold,
new_start=new_start,
new_end=new_end,
))
return results
+97 -1
View File
@@ -24,6 +24,78 @@ logger = logging.getLogger("omnivoice.engine_env")
_TORCH_COMPILE_KEY = "perf.torch_compile_disabled"
# #278: explicit opt-in override — set to 1/true to attempt torch.compile even
# when the GPU's compute capability is not in this PyTorch build's arch list
# (e.g. a brand-new architecture running through PTX forward-compat).
_FORCE_COMPILE_ENV = "OMNIVOICE_FORCE_TORCH_COMPILE"
# #278: set (with a reason) the first time torch.compile — or *running* the
# compiled model — fails at runtime in this process. Once set, every later
# load in the same session goes straight to eager instead of re-tripping the
# same Dynamo/Inductor/Triton failure.
_compile_runtime_failure: Optional[str] = None
def mark_compile_runtime_failure(reason: str) -> None:
"""Record that torch.compile (or compiled execution) failed at runtime.
Called by ``services.model_manager`` when compilation raises, or when a
generation through the compiled model dies inside the Dynamo / Inductor /
Triton stack (#278). Disables compile for the rest of the process — eager
mode from here on; the next app restart probes again.
"""
global _compile_runtime_failure
_compile_runtime_failure = reason or "unknown torch.compile runtime failure"
logger.warning(
"torch.compile disabled for this session after a runtime failure: %s",
_compile_runtime_failure,
)
def _force_compile_requested() -> bool:
value = os.environ.get(_FORCE_COMPILE_ENV, "")
return value.strip().lower() in {"1", "true", "yes", "on"}
def _cuda_arch_supported_for_compile() -> "tuple[bool, str]":
"""Check the GPU's compute capability against this torch build's arch list.
New GPU architectures (e.g. Blackwell sm_120, issue #278) routinely break
torch.compile/Triton before upstream support lands: the eager model runs
via PTX forward-compat, but Inductor/Triton kernel compilation targets the
new arch directly and fails mid-generation. If the device's ``sm_XY`` tag
is absent from ``torch.cuda.get_arch_list()`` we treat compile as
unsupported and use eager.
Returns ``(supported, reason)``. Fails open any probe error returns
``(True, "")`` so a weird torch build never silently loses the
optimization (the runtime fallback in model_manager still protects
generation).
"""
try:
import torch
if not torch.cuda.is_available():
return True, ""
major, minor = torch.cuda.get_device_capability(0)
arch_list = list(getattr(torch.cuda, "get_arch_list", lambda: [])() or [])
if not arch_list:
return True, ""
sm_tag = f"sm_{major}{minor}"
if sm_tag in arch_list or f"compute_{major}{minor}" in arch_list:
return True, ""
try:
device_name = torch.cuda.get_device_name(0)
except Exception:
device_name = "GPU"
return False, (
f"{device_name} (compute capability {major}.{minor} / {sm_tag}) is not "
f"in this PyTorch build's supported arch list ({', '.join(arch_list)})"
)
except Exception:
logger.debug("CUDA arch probe for torch.compile failed; assuming supported", exc_info=True)
return True, ""
def should_torch_compile(device: str) -> bool:
"""Decide whether to apply ``torch.compile`` to an in-process model.
@@ -34,9 +106,14 @@ def should_torch_compile(device: str) -> bool:
- device == "cuda" (compile only helps the CUDA path here),
- Triton importable (``find_spec`` the cross-platform gate that closes
#65; no Windows wheel ⇒ skip ⇒ eager),
- the user has NOT set the ``perf.torch_compile_disabled`` escape hatch.
- the user has NOT set the ``perf.torch_compile_disabled`` escape hatch,
- compile has NOT already failed at runtime in this process (#278),
- the GPU's compute capability is in this torch build's arch list (#278)
overridable via ``OMNIVOICE_FORCE_TORCH_COMPILE=1``.
Returns False ( eager mode) on any of those, logging the reason at INFO.
torch.compile is an optimization, never a requirement generation must
always work without it.
"""
if device != "cuda":
return False
@@ -51,6 +128,25 @@ def should_torch_compile(device: str) -> bool:
return False
except Exception:
logger.exception("should_torch_compile: settings read failed; proceeding")
if _compile_runtime_failure is not None:
logger.info(
"torch.compile skipped: failed earlier this session (%s) — using eager mode.",
_compile_runtime_failure,
)
return False
supported, reason = _cuda_arch_supported_for_compile()
if not supported:
if _force_compile_requested():
logger.warning(
"torch.compile forced via %s=1 despite: %s", _FORCE_COMPILE_ENV, reason,
)
return True
logger.info(
"torch.compile skipped: %s — using eager mode. "
"(Set %s=1 to attempt compile anyway.)",
reason, _FORCE_COMPILE_ENV,
)
return False
return True
+156
View File
@@ -0,0 +1,156 @@
"""Pure, host-aware routing resolver — maps an engine's declared ``gpu_compat``
against the cached host capabilities to "where will this engine *actually* run
on this machine, and is that a problem the user should hear about?"
No model load, no probe (the caller passes the cached ``HostCaps``), no I/O.
Deterministic and byte-identical for a given ``(gpu_compat, HostCaps)`` across
macOS/Windows/Linux that cross-OS determinism is the whole point of the
no-silent-fallback contract.
Reason strings are author-controlled English (interpolating only family/device
names) but are **still** scrubbed by the caller (``core.scrub.scrub_text``)
before serialization, because an interpolated ``device_name`` or probe note can
carry a home path.
"""
from __future__ import annotations
from typing import Literal, TypedDict
from core.device_caps import (
DIRECTML_MARKER,
KERNEL_RISK_MARKER,
HostCaps,
)
RoutingStatus = Literal["accelerated", "cpu_fallback", "cpu_only", "unavailable", "n/a"]
class RoutingResult(TypedDict):
effective_device: str # a DeviceFamily value or "cpu"
routing_status: RoutingStatus # resolve_routing never emits "n/a" (LLM-only)
routing_reason: str | None # raw, pre-scrub
def _caveat(caps: HostCaps) -> str | None:
"""A kernel-risk caveat string for an otherwise-accelerated host, or None.
Advisory notes (multi-GPU, VRAM-query-failed, DirectML) never qualify."""
for note in caps.notes:
if KERNEL_RISK_MARKER in note:
return f"{caps.family.upper()} selected, but: {note}"
return None
def resolve_routing(gpu_compat: tuple[str, ...], caps: HostCaps) -> RoutingResult:
"""Resolve the effective device + status for an engine on this host.
Rules are evaluated in order; the first match wins (see spec §2)."""
targets = tuple(gpu_compat or ())
fam = caps.family
# 1. Empty compat — reserved for LLM (which never calls this). Defensive.
if not targets:
return {
"effective_device": "cpu",
"routing_status": "cpu_only",
"routing_reason": "engine declares no compute targets",
}
# 2. Host accelerator is one the engine supports → accelerated.
if fam != "cpu" and fam in targets:
return {
"effective_device": fam,
"routing_status": "accelerated",
"routing_reason": _caveat(caps),
}
# 3. Host has an accelerator the engine lacks, but engine supports cpu
# → the no-silent-fallback signal.
if fam != "cpu" and "cpu" in targets:
if fam == "rocm" and "cuda" in targets and "rocm" not in targets:
reason = "declares CUDA only; ROCm not in its compat set"
else:
reason = f"engine has no {fam.upper()} path; running on CPU"
return {
"effective_device": "cpu",
"routing_status": "cpu_fallback",
"routing_reason": reason,
}
# 4. Genuine CPU-only host (or DirectML, which the probe reports as cpu)
# and engine supports cpu → benign; must not warn or block.
if fam == "cpu" and "cpu" in targets:
reason = None
for note in caps.notes:
if DIRECTML_MARKER in note:
reason = (
"DirectML GPU present; engine routes via torch CPU path "
"(DirectML acceleration not wired into routing)"
)
break
return {
"effective_device": "cpu",
"routing_status": "cpu_only",
"routing_reason": reason,
}
# 5. Engine needs an accelerator this host lacks and has no cpu path.
first = targets[0]
return {
"effective_device": first,
"routing_status": "unavailable",
"routing_reason": f"requires {', '.join(targets)}; this host has {fam}",
}
def routing_notice(result: RoutingResult) -> tuple[str, str | None] | None:
"""`(status, reason)` when a synth-time notice SHOULD be surfaced to the
user, else `None`. Surfaced for `cpu_fallback` (always) and for
`accelerated` ONLY when it carries a driver/arch caveat reason everything
else (`cpu_only`, clean `accelerated`, `n/a`) is benign and stays silent."""
st = result["routing_status"]
if st == "cpu_fallback" or (st == "accelerated" and result["routing_reason"]):
return (st, result["routing_reason"])
return None
def header_safe_reason(reason: str | None) -> str | None:
"""A routing reason made safe for an HTTP header value: scrubbed, then
ASCII-sanitized (headers are latin-1; a non-ASCII device name would 500 the
response otherwise), **control characters stripped** (a CR/LF could split
the header / inject a new one), and length-capped at 256. Returns None for
an empty reason. No regex `.encode`/membership only (CodeQL-clean)."""
if not reason:
return None
from core.scrub import scrub_text
ascii_only = scrub_text(reason).encode("ascii", "ignore").decode("ascii")
# Drop ASCII control chars (0x00-0x1F + DEL 0x7F) — incl. CR/LF, so the
# value can never break out of its header line.
cleaned = "".join(c for c in ascii_only if 0x20 <= ord(c) < 0x7F)
return cleaned[:256] or None
def routing_fields(gpu_compat: tuple[str, ...], caps: HostCaps) -> dict:
"""The three serialization-ready routing keys for a ``list_backends`` entry.
Resolves routing and applies the redaction contract: ``routing_reason`` is
scrubbed via ``core.scrub.scrub_text`` only when truthy, so a ``None`` reason
serializes as JSON ``null`` (NOT ``""`` ``scrub_text(None)`` would coerce
to ``""``). Used by tts/asr ``list_backends`` so the scrub rule lives in one
place. (LLM emits its own literal ``network``/``n/a``/``null`` fields and
does NOT call this.)
"""
from core.scrub import scrub_text
r = resolve_routing(tuple(gpu_compat or ()), caps)
reason = r["routing_reason"]
return {
"effective_device": r["effective_device"],
"routing_status": r["routing_status"],
"routing_reason": scrub_text(reason) if reason else None,
}
__all__ = [
"RoutingStatus", "RoutingResult", "resolve_routing", "routing_fields",
"routing_notice", "header_safe_reason",
]
+203 -10
View File
@@ -5,6 +5,10 @@ import os
import shutil
import subprocess
# Leaf module (stdlib-only) — safe to import at module top, unlike
# services.dub_pipeline which imports this module and would cycle.
from services.proc_registry import register_proc, unregister_proc
logger = logging.getLogger("omnivoice.api")
# Cap concurrent ffmpeg jobs so macOS posix_spawn can't hit EAGAIN under load.
@@ -19,6 +23,36 @@ def _get_semaphore() -> asyncio.Semaphore:
return _FFMPEG_SEMAPHORE
# Candidate paths that exist but won't run (validated once per process).
# Windows users hit this as `[WinError 193] %1 is not a valid Win32
# application` (#360/#361/#362): a corrupt/wrong-arch imageio-ffmpeg
# download or a WindowsApps alias stub passes `os.path.isfile` / `which`
# but explodes at spawn. Probe each candidate with `-version` and fall
# through to the next source instead of returning a time bomb.
_BINARY_OK: dict[str, bool] = {}
def _binary_runs(path: str) -> bool:
cached = _BINARY_OK.get(path)
if cached is not None:
return cached
try:
subprocess.run(
[path, "-version"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
timeout=10, check=False,
)
ok = True
except (OSError, subprocess.TimeoutExpired, subprocess.SubprocessError) as e:
logger.warning(
"Rejecting non-runnable ffmpeg/ffprobe candidate %s: %s",
os.path.basename(str(path)), e,
)
ok = False
_BINARY_OK[path] = ok
return ok
def find_ffmpeg():
"""Locate an ffmpeg binary.
@@ -33,15 +67,15 @@ def find_ffmpeg():
env_path = os.environ.get("FFMPEG_PATH")
if env_path:
resolved = shutil.which(env_path)
if resolved:
if resolved and _binary_runs(resolved):
return resolved
# 2. imageio-ffmpeg bundled static binary
try:
import imageio_ffmpeg
candidate = imageio_ffmpeg.get_ffmpeg_exe()
if candidate and os.path.isfile(candidate):
if candidate and os.path.isfile(candidate) and _binary_runs(candidate):
return candidate
logger.debug("imageio_ffmpeg binary not found at %s", candidate)
logger.debug("imageio_ffmpeg binary not usable at %s", candidate)
except Exception as e:
logger.debug("imageio_ffmpeg unavailable: %s", e)
# 3. Well-known system paths + PATH lookup
@@ -54,9 +88,10 @@ def find_ffmpeg():
"ffmpeg",
]
for path in common:
if shutil.which(path):
return path
logger.warning("ffmpeg not found in env, imageio, or system PATH")
resolved = shutil.which(path)
if resolved and _binary_runs(resolved):
return resolved
logger.warning("ffmpeg not found (or not runnable) in env, imageio, or system PATH")
return None
@@ -80,14 +115,14 @@ def resolve_ffprobe() -> str | None:
continue
# The env var may carry either an absolute path to a file OR a bare
# command name (legacy). Accept both shapes — file first.
if os.path.isfile(path):
if os.path.isfile(path) and _binary_runs(path):
return path
resolved = shutil.which(path)
if resolved:
if resolved and _binary_runs(resolved):
return resolved
system_probe = shutil.which("ffprobe")
if system_probe:
if system_probe and _binary_runs(system_probe):
return system_probe
return None
@@ -206,16 +241,168 @@ async def _spawn_with_retry(cmd, **kwargs):
raise last_err if last_err else RuntimeError("spawn failed")
async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True):
def _atempo_chain(ratio: float) -> str:
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
multiple stages multiplies the effective ratio while keeping each
individual stage inside the well-behaved range. Pitch is preserved
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
slows down.
"""
stages: list[str] = []
remaining = ratio
while remaining > 2.0:
stages.append("atempo=2.0")
remaining /= 2.0
while remaining < 0.5:
stages.append("atempo=0.5")
remaining /= 0.5
stages.append(f"atempo={remaining:.6f}")
return ",".join(stages)
async def _pitch_preserving_stretch(wav, target_samples: int, sr: int):
"""Time-stretch a (1, samples) tensor to `target_samples` while
preserving pitch, by piping the audio through `ffmpeg atempo`.
Async so it never blocks the event loop: it's awaited from the dub
generate `_stream` generator, and each ffmpeg call is ~50-100 ms a
synchronous ``subprocess.run`` here froze health-checks / SSE / every
concurrent request for the whole multi-segment job.
Returns a (1, target_samples) tensor on the same device as input.
Raises RuntimeError when ffmpeg fails callers should fall back to
naive linear interpolation, accepting the pitch shift, to ensure the
output isn't silent.
"""
# Lazy imports keep this module importable in torch-free contexts
# (setup scripts, smoke probes) — only the stretch path needs them.
import numpy as np
import torch
wl = int(wav.shape[-1])
if target_samples <= 0 or wl == target_samples:
return wav
ratio = wl / target_samples
filter_str = _atempo_chain(ratio)
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per segment,
# run off the event loop so concurrent requests stay responsive.
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
proc = await spawn_subprocess(
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
"-af", filter_str,
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await proc.communicate(input=arr.tobytes())
if proc.returncode != 0 or not stdout:
raise RuntimeError(
(stderr.decode(errors="replace") or "atempo failed")[:200]
)
out_arr = np.frombuffer(stdout, dtype=np.float32)
# atempo rarely lands exactly on the integer sample count, so
# pad/trim to the requested slot length.
if len(out_arr) < target_samples:
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
out_arr = np.concatenate([out_arr, pad])
elif len(out_arr) > target_samples:
out_arr = out_arr[:target_samples]
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
async def probe_duration(path: str) -> float | None:
"""Return a media file's duration in seconds via ffprobe, or None.
Used by the Smart Fit pipeline to sanity-check source/track lengths
without loading the media. Never raises probing is best-effort.
"""
ffprobe = find_ffprobe()
if not ffprobe or not os.path.isfile(path):
return None
try:
proc = await spawn_subprocess(
ffprobe, "-v", "error",
"-show_entries", "format=duration",
"-of", "default=noprint_wrappers=1:nokey=1",
path,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await proc.communicate()
if proc.returncode != 0:
return None
return float(stdout.decode().strip())
except Exception as e:
logger.debug("probe_duration failed for %s: %s", os.path.basename(str(path)), e)
return None
async def probe_frame_rates(path: str) -> "tuple[str, str] | None":
"""Return (r_frame_rate, avg_frame_rate) strings for the first video
stream (e.g. ``("30000/1001", "2997/100")``), or None on any failure.
A mismatch between the two is the practical VFR signature used by the
Smart Fit retime pipeline to decide whether to normalise with ``fps=``
before trim/setpts. Never raises probing is best-effort.
"""
ffprobe = find_ffprobe()
if not ffprobe or not os.path.isfile(path):
return None
try:
proc = await spawn_subprocess(
ffprobe, "-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=r_frame_rate,avg_frame_rate",
"-of", "csv=p=0",
path,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await proc.communicate()
if proc.returncode != 0:
return None
parts = stdout.decode().strip().split(",")
if len(parts) < 2:
return None
return parts[0].strip(), parts[1].strip()
except Exception as e:
logger.debug("probe_frame_rates failed for %s: %s", os.path.basename(str(path)), e)
return None
async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True,
job_id: "str | None" = None):
"""Run an ffmpeg subprocess with concurrency cap, timeout, and proper cleanup.
Returns (returncode, stdout_bytes, stderr_bytes). Raises asyncio.TimeoutError
on hard timeout (after killing + reaping the process).
``job_id`` (optional) registers the process with the dub pipeline's
process tracker (``services.proc_registry``) so ``/dub/abort`` can kill
long export encodes (used by the Smart Fit batched retime).
Path-injection note: every filesystem path placed in ``cmd`` by callers
is realpath-normalised and containment-checked against its workspace
root (e.g. DUB_DIR) at the call site before the argv is assembled
see api.routers.dub_export and services.video_retime.
"""
stdout = asyncio.subprocess.PIPE if capture else asyncio.subprocess.DEVNULL
stderr = asyncio.subprocess.PIPE
async with _get_semaphore():
proc = await _spawn_with_retry(cmd, stdout=stdout, stderr=stderr)
if job_id:
try:
register_proc(job_id, proc)
except Exception as e:
# Newline-strip the id inline — it can originate from a path
# param, and the log stream must stay one-event-per-line.
logger.debug("register_proc failed for %s: %s",
job_id.replace("\n", " ").replace("\r", " "), e)
try:
try:
out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout)
@@ -231,6 +418,12 @@ async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True):
raise
return proc.returncode, out, err
finally:
if job_id:
try:
unregister_proc(job_id, proc)
except Exception as e:
logger.debug("unregister_proc failed for %s: %s",
job_id.replace("\n", " ").replace("\r", " "), e)
# Guarantee reaping — prevents zombie pileup under timeouts or errors.
if proc.returncode is None:
try:
+211
View File
@@ -0,0 +1,211 @@
"""Smart Fit planner — dub-length fitting v2, Phase A.
Pure planning functions for the ``smart_fit`` timing strategy: given the
original segment timeline and the *natural-rate* duration of each dubbed
segment's TTS audio, decide per segment how to reconcile the two by
splitting the burden between a mild pitch-preserving audio speed-up and a
mild per-segment video slow-down.
Clean-room note: this is a reimplementation from a *published description*
of the audio-speedup + video-slowdown fitting approach (see
docs/competitive-analysis.md, "Dub-length fitting"). No GPL source was
consulted.
Algorithm per segment (defaults in :class:`FitParams`):
1. **Slack absorption** the usable slot extends past the segment's
original end into the silent gap before the next segment, keeping a
small ``gap_guard_s`` clear of the next onset (the last segment may run
to the end of the video). ``need = natural_dur / slot``.
2. ``need <= 1.0`` fits as-is; nothing to do.
3. ``1.0 < need <= max_audio_only_rate`` audio-only speed-up at exactly
``need`` (imperceptible up to ~1.2×).
4. ``need > max_audio_only_rate`` geometric 50/50 split:
``audio_rate = min(sqrt(need), audio_rate_cap)`` and
``video_ratio = min(need / audio_rate, video_slow_cap)``. Whatever the
caps can't absorb becomes ``overflow_s`` (trimmed at mix time).
5. ``allow_video_retime=False`` audio-only mode: rate capped at the
legacy ``MAX_AUDIO_RATE_HARD`` (1.8, matching dub_generate's
MAX_STRETCH_RATIO guard rail), residual overflows.
6. **Timeline cursor** mirrors the existing ``stretch_video`` layout
loop: pre-roll and inter-segment gaps pass through at 1.0×; each
segment's video chunk ``[start, effective_end]`` occupies
``slot * video_ratio`` on the new timeline.
This module is deliberately I/O-free and torch-free so it can be unit- and
golden-tested without a model, ffmpeg, or an event loop.
"""
from __future__ import annotations
import math
from dataclasses import dataclass, field
# Hard ceiling for audio-only compression when video retiming is disabled.
# Matches dub_generate.MAX_STRETCH_RATIO — above ~1.8× speech becomes a
# garbled stream no DSP can rescue.
MAX_AUDIO_RATE_HARD = 1.8
_EPS = 1e-9
@dataclass(frozen=True)
class FitParams:
"""Tunable knobs for the Smart Fit planner.
All defaults are deliberately conservative: 1.2× audio-only is
imperceptible to most listeners; 1.5× audio is the intelligibility
cap; 2.0× video slow-down is the limit before motion looks syrupy.
"""
max_audio_only_rate: float = 1.2
audio_rate_cap: float = 1.5
video_slow_cap: float = 2.0
gap_guard_s: float = 0.05
allow_video_retime: bool = True
@dataclass
class SegmentFit:
"""Planner verdict for one segment."""
index: int
seg_id: str
audio_rate: float # ≥ 1.0 — pitch-preserving speed-up applied to TTS audio
video_ratio: float # ≥ 1.0 — setpts slow-down applied to the video chunk
new_start: float # placement on the fitted (possibly longer) timeline
new_end: float # end of the video chunk on the fitted timeline
orig_start: float
orig_end: float
effective_end: float # orig_end + absorbed slack (≤ next start gap guard)
status: str # "fits" | "audio_stretched" | "hybrid" | "overflow_trimmed"
overflow_s: float # seconds of (stretched) audio that still don't fit
@dataclass
class FitPlan:
"""Full plan for one dub track."""
segments: list[SegmentFit] = field(default_factory=list)
# EXACT dict shape consumed by dub_export._build_video_stretch_filter_graph.
video_plan: list[dict] = field(default_factory=list)
total_duration: float = 0.0
orig_duration: float = 0.0
params: FitParams = field(default_factory=FitParams)
@property
def needs_video_retime(self) -> bool:
return any(s.video_ratio > 1.0 + 1e-6 for s in self.segments)
def _fit_one(need: float, params: FitParams) -> tuple[float, float, str]:
"""Resolve one segment's need ratio into (audio_rate, video_ratio, status)."""
if need <= 1.0 + _EPS:
return 1.0, 1.0, "fits"
if need <= params.max_audio_only_rate + _EPS:
return need, 1.0, "audio_stretched"
if not params.allow_video_retime:
audio_rate = min(need, MAX_AUDIO_RATE_HARD)
status = "audio_stretched" if audio_rate >= need - _EPS else "overflow_trimmed"
return audio_rate, 1.0, status
# Geometric 50/50 split: equal perceptual burden on audio and video.
audio_rate = min(math.sqrt(need), params.audio_rate_cap)
video_ratio = min(need / audio_rate, params.video_slow_cap)
if audio_rate * video_ratio >= need - _EPS:
return audio_rate, video_ratio, "hybrid"
return audio_rate, video_ratio, "overflow_trimmed"
def plan_fit(
segments: list[dict],
natural_durs_s: list[float],
total_dur_s: float,
params: FitParams | None = None,
) -> FitPlan:
"""Plan the Smart Fit layout for a dub track.
``segments``: original-timeline segments in chronological order, each a
dict with ``id``, ``start``, ``end`` (seconds). ``natural_durs_s``: the
natural-rate TTS audio duration for each segment (parallel list).
``total_dur_s``: original video duration (0/unknown tolerated the last
segment then gets no tail slack).
Pure function: no I/O, no torch, deterministic.
"""
params = params or FitParams()
n = len(segments)
if len(natural_durs_s) != n:
raise ValueError(
f"segments ({n}) and natural_durs_s ({len(natural_durs_s)}) must be parallel"
)
plan = FitPlan(params=params, orig_duration=round(float(total_dur_s), 4))
if n == 0:
plan.total_duration = round(max(0.0, float(total_dur_s)), 4)
return plan
cursor = 0.0
for i, seg in enumerate(segments):
start = float(seg["start"])
end = float(seg["end"])
natural = max(0.0, float(natural_durs_s[i]))
# (a) Slack absorption. Extend-only: the slot never shrinks below
# the original [start, end] even when segments are back-to-back.
if i + 1 < n:
next_start = float(segments[i + 1]["start"])
effective_end = max(end, next_start - params.gap_guard_s)
# Never bleed past the next segment's onset (overlapping or
# near-touching source segments).
effective_end = min(max(effective_end, start), max(next_start, end))
else:
effective_end = max(end, float(total_dur_s)) if total_dur_s > 0 else end
slot = max(effective_end - start, 1e-3)
need = natural / slot if natural > 0 else 0.0
audio_rate, video_ratio, status = _fit_one(need, params)
# (f) Timeline cursor — mirror the stretch_video layout loop:
# pre-roll and gaps at 1.0×, the segment's video chunk
# [start, effective_end] occupies slot × video_ratio.
if i == 0:
cursor = start # pre-roll preserved at native rate
new_start = cursor
new_end = new_start + slot * video_ratio
cursor = new_end
if i + 1 < n:
# Unretimed sliver between this chunk and the next chunk's
# start (the gap guard, or more if extend-only clamped).
cursor += max(0.0, float(segments[i + 1]["start"]) - effective_end)
# Residual overflow after both knobs: stretched audio length vs the
# segment's new video slot.
stretched = natural / audio_rate if audio_rate > 0 else natural
overflow_s = max(0.0, stretched - slot * video_ratio)
if overflow_s <= 1e-6:
overflow_s = 0.0
elif status != "overflow_trimmed":
status = "overflow_trimmed"
plan.segments.append(SegmentFit(
index=i,
seg_id=str(seg.get("id", f"seg_{i}")),
audio_rate=round(audio_rate, 6),
video_ratio=round(video_ratio, 6),
new_start=round(new_start, 4),
new_end=round(new_end, 4),
orig_start=round(start, 4),
orig_end=round(end, 4),
effective_end=round(effective_end, 4),
status=status,
overflow_s=round(overflow_s, 4),
))
plan.video_plan.append({
"orig_start": round(start, 4),
"orig_end": round(effective_end, 4),
"new_start": round(new_start, 4),
"new_end": round(new_end, 4),
"stretch_ratio": round(video_ratio, 4),
})
# Tail (anything after the last segment's effective end) at 1.0×.
last_eff = float(plan.segments[-1].effective_end)
cursor += max(0.0, float(total_dur_s) - last_eff)
plan.total_duration = round(max(cursor, float(total_dur_s)), 4)
return plan
+65
View File
@@ -0,0 +1,65 @@
"""Map subtitle cues onto the Smart-Fit timeline (Wave 3.1 / Spec 1).
When a dub uses ``stretch_video`` mode, the video is re-timed per segment so
the dubbed audio fits (see fit_planner + the export stretch filter). The
dubbed audio therefore plays at *fitted* positions, not the original
timestamps. A subtitle file exported with the original times would drift
against the dubbed video so we regenerate the cue timeline from the same
plan the video stretch uses ("subtitles track actual dub placement", the
last piece of Spec 1).
Pure functions no I/O so the remapping is unit-testable. The plan is the
persisted ``video_stretch_plan`` list of
``{orig_start, orig_end, new_start, new_end, stretch_ratio}`` chunks.
"""
from __future__ import annotations
def map_time_to_fitted(t: float, plan: list[dict]) -> float:
"""Map a time on the original timeline to its position on the fitted one.
Finds the plan chunk whose original span contains ``t`` and interpolates
linearly into that chunk's fitted span (a chunk's stretch is uniform).
Before the first chunk maps 1:1; after the last chunk the trailing offset
is carried at 1:1 (the planner runs gaps/tail at rate 1.0). Empty plan
identity.
"""
if not plan:
return t
for chunk in plan:
o0 = float(chunk.get("orig_start", 0.0))
o1 = float(chunk.get("orig_end", 0.0))
n0 = float(chunk.get("new_start", o0))
n1 = float(chunk.get("new_end", o1))
if t < o0:
# In a gap before this chunk — carry the offset at 1:1 from the
# previous chunk's fitted end (or from 0 for the very first).
return n0 - (o0 - t)
if o0 <= t <= o1:
span = o1 - o0
if span <= 0:
return n0
return n0 + (t - o0) / span * (n1 - n0)
# Past the last chunk: 1:1 tail from its fitted end.
last = plan[-1]
return float(last.get("new_end", 0.0)) + (t - float(last.get("orig_end", 0.0)))
def fitted_cues(segments: list[dict], plan: list[dict]) -> list[tuple[float, float]]:
"""Return ``[(start, end), ...]`` for each segment on the fitted timeline.
Monotonicity guard: a cue's end is never before its start, and successive
starts never go backwards (rounding across chunk seams can't produce a
non-monotone SRT).
"""
out: list[tuple[float, float]] = []
prev_end = 0.0
for seg in segments:
s = map_time_to_fitted(float(seg.get("start", 0.0)), plan)
e = map_time_to_fitted(float(seg.get("end", 0.0)), plan)
s = max(s, prev_end if out else 0.0)
e = max(e, s)
out.append((s, e))
prev_end = e
return out
+81 -3
View File
@@ -22,6 +22,32 @@ import json
_GEN_INPUT_FIELDS = ("text", "target_lang", "profile_id", "instruct", "speed", "direction", "effect_preset")
# Pydantic fills `effect_preset` with this default when the client omits it,
# while the client-side recompute (/tools/incremental) sends nothing. Both
# representations must hash identically or every segment looks "stale" after
# every generate and incremental re-dub degrades to a full re-dub (#281).
_DEFAULT_EFFECT_PRESET = "broadcast"
def _canon_value(field: str, value):
"""Normalise one generation-input value so that the server-side view
(pydantic-parsed `DubSegment`, defaults filled in) and the client-side
view (raw segment dict, unset keys omitted) produce the same hash.
- missing / None / "" all mean "default" for string fields
- `effect_preset` default is "broadcast" (pydantic fills it server-side)
- numbers are coerced to float so `speed: 1` (JS) == `speed: 1.0` (pydantic)
"""
if field == "effect_preset":
return value or _DEFAULT_EFFECT_PRESET
if value is None or value == "":
return ""
if isinstance(value, bool):
return value
if isinstance(value, (int, float)):
return float(value)
return value
def segment_fingerprint(seg: dict) -> str:
"""Deterministic hash of the inputs that actually affect TTS output.
@@ -31,11 +57,63 @@ def segment_fingerprint(seg: dict) -> str:
badge don't trigger regen, which is what we want.
Currently includes: text, target_lang, profile_id, instruct, speed,
direction, effect_preset.
direction, effect_preset. Values are canonicalised (see `_canon_value`)
so a fingerprint computed from the generate request (server defaults
filled in) matches one recomputed later from the client's raw segment
state the root cause of #281's "1 edit re-dubs all N lines".
"""
payload = {k: (seg.get(k) if seg.get(k) is not None else "") for k in _GEN_INPUT_FIELDS}
payload = {k: _canon_value(k, seg.get(k)) for k in _GEN_INPUT_FIELDS}
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
return hashlib.sha1(blob.encode("utf-8")).hexdigest()[:16]
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
# ── Smart Fit (dub-length fitting v2) fingerprints ─────────────────────────
#
# Fitting parameters stay OUT of segment_fingerprint on purpose: changing a
# fit knob (caps, gap guard, strategy) must trigger a RE-MIX of the already
# rendered natural-rate WAVs (generate with regen_only=[]), never a re-TTS.
# A separate per-track fingerprint tracks the fit configuration; a dubbed
# track is stale iff its fit_fp differs OR any segment hash differs.
_FIT_PARAM_FIELDS = (
"timing_strategy",
"max_audio_only_rate",
"audio_rate_cap",
"video_slow_cap",
"gap_guard_s",
"allow_video_retime",
)
# Server-side defaults (must mirror services.fit_planner.FitParams). Filled
# in for omitted keys so a fingerprint computed from a fully-populated
# server view matches one recomputed from a sparse client payload — the
# same #281 regression class segment_fingerprint already guards against.
_FIT_PARAM_DEFAULTS = {
"timing_strategy": "smart_fit",
"max_audio_only_rate": 1.2,
"audio_rate_cap": 1.5,
"video_slow_cap": 2.0,
"gap_guard_s": 0.05,
"allow_video_retime": True,
}
def fit_fingerprint(params: dict) -> str:
"""Deterministic hash of the fit configuration for one dub track.
Canonicalised with the same `_canon_value` rules as segment
fingerprints (int vs float, None/"" vs omitted), plus default-filling
so `{}` and `{"audio_rate_cap": 1.5}` hash identically.
"""
params = params or {}
payload = {}
for k in _FIT_PARAM_FIELDS:
v = params.get(k)
if v is None or v == "":
v = _FIT_PARAM_DEFAULTS[k]
payload[k] = _canon_value(k, v)
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
def plan_incremental(
+67 -7
View File
@@ -37,6 +37,12 @@ logger = logging.getLogger("omnivoice.llm")
class LLMBackend(ABC):
id: str = "base"
display_name: str = "Base LLM"
# LLM backends call out over the network (OpenAI/Ollama/LM Studio) or are a
# no-op — none run a model on the user's GPU. So `gpu_compat` is empty and
# list_backends() labels the family `effective_device:"network"` /
# routing_status:"n/a" rather than asserting a false GPU claim. Routing is
# never gated for LLM (see engines.select_engine + diagnose).
gpu_compat: tuple[str, ...] = ()
@classmethod
@abstractmethod
@@ -106,6 +112,22 @@ class OpenAICompatBackend(LLMBackend):
return self._client
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
return self.chat_messages(
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
timeout=timeout,
)
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None) -> str:
"""One-shot completion over a full message list.
Additive surface for callers that need structured few-shot turns
(dictation refinement, Wave 2.1) small local models pattern-match
and echo inline examples, so examples must arrive as prior chat
turns, not inside the system prompt.
"""
if timeout is None:
try:
timeout = float(os.environ.get("OMNIVOICE_LLM_TIMEOUT", "45"))
@@ -114,10 +136,7 @@ class OpenAICompatBackend(LLMBackend):
res = self._get_client().chat.completions.create(
model=self.model_name,
timeout=timeout,
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
messages=messages,
)
return (res.choices[0].message.content or "").strip()
@@ -143,6 +162,9 @@ class OffBackend(LLMBackend):
"to use features that need one (Cinematic translate, glossary auto-extract)."
)
def chat_messages(self, **kw) -> str:
return self.chat(**kw)
_REGISTRY: dict[str, type[LLMBackend]] = {
"openai-compat": OpenAICompatBackend,
@@ -150,15 +172,53 @@ _REGISTRY: dict[str, type[LLMBackend]] = {
}
# Most-recent failure per backend (parity with tts/asr list_backends).
_LAST_ERRORS: dict[str, str] = {}
_INSTALL_HINTS: dict[str, str] = {
"openai-compat": "Set TRANSLATE_BASE_URL (+ TRANSLATE_API_KEY) — OpenAI, "
"Ollama (http://localhost:11434/v1), or any compatible host.",
}
def list_backends() -> list[dict]:
out = []
"""Same 11-key shape as tts/asr so the matrix renders families uniformly.
LLM is NOT a GPU family: every entry carries literal
``effective_device:"network"`` / ``routing_status:"n/a"`` /
``routing_reason:null`` (NOT via resolve_routing that would be a false
GPU claim). ``effective_device:"network"`` is a label, not a probe: nothing
here touches the network (local-first).
"""
from core.scrub import scrub_text
out: list[dict] = []
for bid, cls in _REGISTRY.items():
ok, msg = cls.is_available()
try:
ok, msg = cls.is_available()
except Exception as exc:
ok = False
msg = f"{type(exc).__name__}: {exc}"
logger.warning(
"llm list_backends: %s.is_available() raised — degrading "
"gracefully so the picker still renders: %s", bid, msg,
)
if ok:
_LAST_ERRORS.pop(bid, None)
else:
_LAST_ERRORS[bid] = scrub_text(msg)
out.append({
"id": bid,
"display_name": cls.display_name,
"available": ok,
"reason": None if ok else msg,
"reason": None if ok else scrub_text(msg),
"install_hint": _INSTALL_HINTS.get(bid),
"last_error": _LAST_ERRORS.get(bid),
"isolation_mode": "in-process",
"gpu_compat": list(getattr(cls, "gpu_compat", ())),
"effective_device": "network",
"routing_status": "n/a",
"routing_reason": None,
})
return out
+251
View File
@@ -0,0 +1,251 @@
"""Import plain text / EPUB into the chapter-delimited script the audiobook
parser understands.
Both helpers are pure (bytes/str in, script-str out) so they're unit-tested
without a server. EPUB parsing is **stdlib only** (zipfile + ElementTree +
html.parser) no new dependency, no network, consistent with the local-first
guarantee. The output is the same ``# Heading`` + body grammar
:func:`services.audiobook.parse_audiobook_script` already consumes, so import is
just a front door onto the existing pipeline.
"""
from __future__ import annotations
import io
import posixpath
import re
import zipfile
from html.parser import HTMLParser
from xml.etree import ElementTree as ET
# A line that *starts* with a chapter keyword and is short enough to be a title
# (not a sentence that happens to begin with "Chapter"). Anchored, no ambiguous
# quantifiers → ReDoS-safe and applied per-line (short input) anyway.
_CH_RE = re.compile(r"^(?:chapter|part|book|prologue|epilogue|section)\b", re.IGNORECASE)
# Already-present Markdown H1 — if the text has any, we leave it untouched.
_H1_RE = re.compile(r"^[ \t]*#[ \t]+\S", re.MULTILINE)
_CHAPTER_TITLE_MAX = 60
# Zip-bomb / OOM guards for EPUB ingestion: per-entry and cumulative caps on
# *uncompressed* bytes read from the archive.
_EPUB_MAX_ENTRY_BYTES = 25 * 1024 * 1024
_EPUB_MAX_TOTAL_BYTES = 300 * 1024 * 1024
def chapterize_plaintext(text: str) -> str:
"""Insert ``# `` headings ahead of obvious chapter-title lines.
No-op if the text already has Markdown H1 headings (the user has structured
it). Otherwise short standalone lines beginning with a chapter keyword
(``Chapter 3``, ``Prologue`` ) become headings; everything else is left
verbatim. Text with no detectable breaks falls through as a single chapter.
"""
text = text or ""
if _H1_RE.search(text):
return text
out = []
for line in text.split("\n"):
s = line.strip()
if s and len(s) <= _CHAPTER_TITLE_MAX and _CH_RE.match(s):
out.append(f"# {s}")
else:
out.append(line)
return "\n".join(out)
class _TextExtractor(HTMLParser):
"""Collect visible text from XHTML, dropping script/style and collapsing
whitespace. First <h1>/<h2>/<title> seen is kept as the chapter title."""
_SKIP = {"script", "style", "head"}
_BREAK = {"p", "br", "div", "h1", "h2", "h3", "li", "tr"}
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self._parts: list[str] = []
self._skip_depth = 0
self._in_title = False
self.title = ""
def handle_starttag(self, tag, attrs):
if tag in self._SKIP:
self._skip_depth += 1
if tag in ("h1", "h2", "title") and not self.title:
self._in_title = True
if tag in self._BREAK:
self._parts.append("\n")
def handle_endtag(self, tag):
if tag in self._SKIP and self._skip_depth:
self._skip_depth -= 1
if tag in ("h1", "h2", "title"):
self._in_title = False
def handle_data(self, data):
if self._skip_depth:
return
if self._in_title:
# The first heading becomes the chapter's `# Title` (metadata, not
# narrated) — capture it but keep it out of the body. Later headings
# (title already set) fall through and are narrated as subheadings.
if not self.title:
self.title = data.strip()
return
self._parts.append(data)
def text(self) -> str:
raw = "".join(self._parts)
# Collapse runs of blank lines / trailing spaces into tidy paragraphs.
lines = [ln.strip() for ln in raw.split("\n")]
out: list[str] = []
for ln in lines:
if ln or (out and out[-1]):
out.append(ln)
return "\n".join(out).strip()
def _html_to_title_body(xhtml: str) -> tuple[str, str]:
p = _TextExtractor()
try:
p.feed(xhtml)
except Exception:
pass
return p.title, p.text()
_OPF_NS = {"opf": "http://www.idpf.org/2007/opf", "c": "urn:oasis:names:tc:opendocument:xmlns:container"}
def _opf_path(zf: zipfile.ZipFile) -> str:
container = zf.read("META-INF/container.xml")
# The EPUB is a local file the user chose to import (not a remote/untrusted
# surface); stdlib ElementTree doesn't expand external entities by default.
root = ET.fromstring(container) # nosec B314
rootfile = root.find(".//c:rootfiles/c:rootfile", _OPF_NS)
if rootfile is None or not rootfile.get("full-path"):
raise ValueError("EPUB container.xml has no rootfile")
return rootfile.get("full-path")
def epub_to_chapter_script(
data: bytes,
*,
max_entry_bytes: int = _EPUB_MAX_ENTRY_BYTES,
max_total_bytes: int = _EPUB_MAX_TOTAL_BYTES,
) -> str:
"""Convert EPUB bytes into a ``# Chapter`` / body script in spine order.
Reads the OPF manifest + spine (the publisher's reading order), extracts
each document's title + visible text, and emits one ``# Title`` block per
document with renderable text. ``max_entry_bytes`` / ``max_total_bytes``
bound the *uncompressed* bytes read (zip-bomb guard). Raises ``ValueError``
on a malformed EPUB.
"""
try:
zf = zipfile.ZipFile(io.BytesIO(data))
except zipfile.BadZipFile as e:
raise ValueError(f"not a valid EPUB (zip) file: {e}") from e
opf_path = _opf_path(zf)
opf = ET.fromstring(zf.read(opf_path)) # nosec B314 — local user EPUB; see _opf_path
base = posixpath.dirname(opf_path)
manifest: dict[str, str] = {}
for item in opf.findall(".//opf:manifest/opf:item", _OPF_NS):
iid, href = item.get("id"), item.get("href")
if iid and href:
manifest[iid] = href
blocks: list[str] = []
names = set(zf.namelist())
total = 0 # cumulative uncompressed bytes read — zip-bomb guard
for ref in opf.findall(".//opf:spine/opf:itemref", _OPF_NS):
href = manifest.get(ref.get("idref") or "")
if not href:
continue
full = posixpath.normpath(posixpath.join(base, href)) if base else href
if full not in names:
continue
# Bound decompression: skip an absurdly large entry, and stop once the
# cumulative uncompressed size crosses the ceiling (defends against a
# zip bomb / a maliciously huge chapter exhausting memory).
try:
info = zf.getinfo(full)
except KeyError:
continue
if info.file_size > max_entry_bytes:
continue
if total + info.file_size > max_total_bytes:
break
try:
raw = zf.read(full)
except KeyError:
continue
total += len(raw)
title, body = _html_to_title_body(raw.decode("utf-8", "ignore"))
if not body.strip():
continue # nav docs, empty pages
title = title or f"Chapter {len(blocks) + 1}"
blocks.append(f"# {title}\n\n{body}")
if not blocks:
raise ValueError("no readable chapters found in the EPUB")
return "\n\n".join(blocks)
# Page-count ceiling for PDF ingestion — a defence against a pathological
# document tying up the worker. 5000 pages comfortably covers any real book.
_PDF_MAX_PAGES = 5000
def pdf_to_chapter_script(data: bytes, *, max_pages: int = _PDF_MAX_PAGES) -> str:
"""Convert PDF bytes into a ``# Chapter`` / body script.
Extracts the embedded text layer page-by-page (in page order), joins it,
and runs it through :func:`chapterize_plaintext` so ``Chapter N`` /
``Prologue`` lines become headings same grammar EPUB and plaintext emit.
Unlike EPUB this needs a real parser (``pypdf``, pure-Python, no native
deps identical on every platform).
Limitations surfaced as ``ValueError`` (the route maps these to a 400 with
the message, so the user gets actionable feedback rather than a silent
empty import):
* **Scanned / image-only PDFs** have no text layer there's nothing to
extract without OCR, so we raise rather than return an empty script.
* **Password-protected PDFs** that don't open with an empty password can't
be read.
"""
from pypdf import PdfReader
from pypdf.errors import PdfReadError
try:
reader = PdfReader(io.BytesIO(data))
except (PdfReadError, OSError, ValueError) as e:
raise ValueError(f"not a valid PDF file: {e}") from e
if reader.is_encrypted:
# Many PDFs are encrypted with an empty user password (owner-locked but
# freely readable). Try that; a real password we can't supply.
try:
if reader.decrypt("") == 0: # 0 == wrong password
raise ValueError("PDF is password-protected")
except (NotImplementedError, PdfReadError) as e:
raise ValueError(f"can't read this encrypted PDF: {e}") from e
pages = reader.pages
if len(pages) > max_pages:
raise ValueError(f"PDF has too many pages (max {max_pages})")
parts: list[str] = []
for page in pages:
try:
text = page.extract_text() or ""
except Exception: # noqa: BLE001 — one bad page shouldn't kill the import
continue
if text.strip():
parts.append(text)
if not parts:
raise ValueError(
"no extractable text — this looks like a scanned or image-only PDF")
return chapterize_plaintext("\n\n".join(parts))
+143
View File
@@ -0,0 +1,143 @@
"""Canonical longform marker parser (#27) — the single source of grammar truth.
The longform marker dialect (``# heading``, ``[voice:NAME]``, ``[pause …]``,
``[slow]/[fast]/[emphasis]/[spell]``) was parsed by three independent code
paths that disagreed (client/server/regex-level). This module is the one
canonical Python parser; ``frontend/src/utils/longformParser.js`` is its
mechanically-mirrored JS twin, and ``tests/fixtures/longform_parser_cases.json``
is the shared golden corpus asserted byte-for-byte against both.
Pure textplan, import-light (no torch). Grammar precedence (outerinner):
# chapter → [voice:] → [pause] → SSML-lite → [spell]
It reuses the existing pause dialect (``omnivoice.utils.text.parse_pause_markers``)
and SSML-lite (``services.ssml_lite``) verbatim so those modules stay the single
home of their sub-grammars.
"""
from __future__ import annotations
import re
from typing import Optional
from omnivoice.utils.text import parse_pause_markers
# A Markdown H1 (``# Title``) starts a new chapter. Deeper headings (``##``…)
# stay in the body as ordinary text. The title capture starts with ``\S`` (a
# non-space) so the leading ``[ \t]+`` and the title's ``.*`` can't both match
# the same whitespace run — that overlap is what makes ``[ \t]+(.+)``
# polynomial-time on adversarial tabs (ReDoS). Moved verbatim from
# audiobook.py (already CodeQL-cleared). Stripped in code.
_HEADING_RE = re.compile(r"^[ \t]*#[ \t]+(\S.*)$", re.MULTILINE)
# ``[voice:NAME]`` switches the active narrator. The content class excludes BOTH
# brackets (``[^\]\[]``) so nested ``[voice:`` prefixes can't create overlapping
# match attempts across ``finditer`` (the ReDoS source). A voice name never
# contains a bracket; the value is stripped in code. Empty → default voice.
_VOICE_RE = re.compile(r"\[voice:([^\]\[]*)\]")
def _normalize(text: Optional[str]) -> str:
"""Coerce None→'' and normalize CRLF/CR→LF so ``$`` (re.MULTILINE) and span
text never carry a stray ``\\r`` on Windows-authored scripts a
cross-platform default-behaviour divergence the JS twin mirrors exactly."""
if not text:
return ""
return text.replace("\r\n", "\n").replace("\r", "\n")
def _parse_chapter_body(
body: str,
*,
default_voice: Optional[str] = None,
default_speed: Optional[float] = None,
) -> list[dict]:
"""Voice→pause→SSML layering for ONE chapter body (no chapter split).
Returns a list of span dicts ``{voice_id, text, pause_ms_after, speed}``.
A ``#`` inside ``body`` is NOT treated as a heading here — that is the
caller's (chapter-split) concern. The JS twin (``parseChapterBody``) is what
``storyToSpans`` calls per spoken track."""
spans: list[dict] = []
cur_voice = default_voice
runs: list[tuple[Optional[str], str]] = []
last = 0
for m in _VOICE_RE.finditer(body):
if m.start() > last:
runs.append((cur_voice, body[last:m.start()]))
cur_voice = (m.group(1).strip() or default_voice)
last = m.end()
runs.append((cur_voice, body[last:]))
from services.ssml_lite import parse_ssml_lite, spell_out
for voice, run_text in runs:
for span_text, pause_ms in parse_pause_markers(run_text):
t = span_text.strip()
if not t and pause_ms == 0:
continue # pure whitespace between markers — nothing to render
rendered: list[tuple[str, Optional[float]]] = []
for seg in (parse_ssml_lite(t) if t else []):
st = (spell_out(seg["text"]) if seg["spell"] else seg["text"]).strip()
if st:
# Inline SSML speed overrides the per-line default; a plain
# segment inherits default_speed.
sp = seg["speed"] if seg["speed"] is not None else default_speed
rendered.append((st, sp))
if not rendered:
# Only-markers / empty text but a real pause → carry the silence.
if pause_ms > 0:
spans.append({"voice_id": voice, "text": "",
"pause_ms_after": pause_ms, "speed": None})
continue
for j, (st, sp) in enumerate(rendered):
spans.append({
"voice_id": voice, "text": st,
"pause_ms_after": pause_ms if j == len(rendered) - 1 else 0,
"speed": sp,
})
return spans
def parse_script_to_spans(
text: Optional[str],
*,
default_voice: Optional[str] = None,
default_speed: Optional[float] = None,
) -> list[dict]:
"""Parse a chapter-delimited script into ``[{"title", "spans": [...]}, …]``.
span dict == ``{"voice_id": str|None, "text": str, "pause_ms_after": int,
"speed": float|None}`` (key order matches ``Span.to_dict()``).
Contract:
* None / "" / whitespace-only input ``[]``.
* CRLF/CR normalized to LF at entry (cross-platform parity).
* H1 (``# <non-space>…``) opens a chapter; ``##``…``######`` and ``# ``
(no ``\\S`` title) are body.
* Each chapter body resets the active voice to ``default_voice``.
* A span is dropped iff its text is empty AND pause_ms_after == 0.
* Chapters with no surviving spans are dropped; untitled bodies are
numbered ``Chapter {kept_so_far + 1}`` (post-drop numbering).
"""
text = _normalize(text)
matches = list(_HEADING_RE.finditer(text))
if not matches:
raw = [(None, text)]
else:
raw = []
intro = text[:matches[0].start()]
if intro.strip():
raw.append((None, intro))
for i, m in enumerate(matches):
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
raw.append((m.group(1).strip(), text[m.end():end]))
chapters: list[dict] = []
for title, body in raw:
spans = _parse_chapter_body(body, default_voice=default_voice,
default_speed=default_speed)
if not spans:
continue
chapters.append({"title": title or f"Chapter {len(chapters) + 1}",
"spans": spans})
return chapters
+389
View File
@@ -0,0 +1,389 @@
"""Shared long-form render core (Stories + Audiobook convergence).
Both the Audiobook tab and the Stories Editor produce the *same* artifact: a
chapter-marked audio file built from chapter WAVs. This module owns the pure,
engine-agnostic ffmpeg/metadata builders for that mux so neither feature has to
reimplement it:
* ``build_ffmetadata`` FFMETADATA1 doc: an optional ``[global]`` tag block
(title / author / narrator / year / genre / description) followed by one
``[CHAPTER]`` per (title, duration_ms).
* ``build_concat_list`` ffmpeg concat-demuxer list of chapter WAVs.
* ``build_loudnorm_filter`` an ``-af loudnorm=`` string for an ACX /
podcast loudness preset (off by default opt-in, so the default-behavior
stays platform-identical).
* ``validate_cover_image`` guard a cover path (type + size) before it
reaches ffmpeg.
* ``build_render_cmd`` pure argv for the mux: chapter WAVs + FFMETADATA
(+ optional cover art, loudness filter), output as ``m4b`` or ``mp3``.
* ``chapter_cache_key`` deterministic content hash so a re-run reuses
already-rendered chapters (resume) and re-renders only what changed.
Every function here is pure (string/argv in, string/argv out) so it's unit
tested without ffmpeg, torch, or a GPU. The impure ffmpeg run lives in the
caller (the audiobook router today; the stories job tomorrow).
"""
from __future__ import annotations
import hashlib
import json
import math
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, Optional
_BITRATE_RE = re.compile(r"^\d{2,3}k$")
#: Default ceiling for the content-addressed chapter cache. Above this, the
#: oldest cached chapter WAVs are evicted (LRU by mtime). Override via
#: OMNIVOICE_LONGFORM_CACHE_MAX_GB.
_CACHE_MAX_BYTES = int(float(os.environ.get("OMNIVOICE_LONGFORM_CACHE_MAX_GB", "2")) * 1024 ** 3)
_COVER_EXTS = {".jpg", ".jpeg", ".png"}
_COVER_MAX_BYTES = 8 * 1024 * 1024 # 8 MB — a book cover, not a payload
#: Our metadata field → FFMETADATA tag key. Order is stable for deterministic
#: output (tested). ``author`` maps to ``artist`` and ``narrator`` to
#: ``composer`` — the tags audiobook players (Apple Books, Audible) read for
#: those roles.
_GLOBAL_TAG_KEYS: list[tuple[str, str]] = [
("title", "title"),
("author", "artist"),
("album", "album"),
("narrator", "composer"),
("year", "date"),
("genre", "genre"),
("description", "comment"),
]
def _escape_meta(value: str) -> str:
"""Escape an FFMETADATA value (``=``, ``;``, ``#``, ``\\``, newline)."""
return re.sub(r"([=;#\\\n])", r"\\\1", value or "")
def prune_cache_dir(cache_dir: str, max_bytes: int = _CACHE_MAX_BYTES) -> tuple[int, int]:
"""Evict the oldest files in ``cache_dir`` until the total size is within
``max_bytes`` (LRU by mtime). The content-addressed chapter cache otherwise
grows without bound uncompressed WAVs accumulate across every render.
Best-effort: returns ``(remaining_bytes, removed_count)`` and never raises
(a missing dir / unstattable file is just skipped). Call it *before* writing
a job's chapters so the fresh ones are never the eviction target.
"""
try:
names = os.listdir(cache_dir)
except OSError:
return (0, 0)
entries: list[tuple[float, int, str]] = []
total = 0
for name in names:
p = os.path.join(cache_dir, name)
try:
if not os.path.isfile(p):
continue
size = os.path.getsize(p)
mtime = os.path.getmtime(p)
except OSError:
continue
entries.append((mtime, size, p))
total += size
if total <= max_bytes:
return (total, 0)
entries.sort() # oldest first
removed = 0
for _mtime, size, p in entries:
if total <= max_bytes:
break
try:
os.remove(p)
total -= size
removed += 1
except OSError:
continue
return (total, removed)
# ── Chapter cache key (resume) ──────────────────────────────────────────────
def chapter_cache_key(
spans: Iterable[tuple],
*,
sample_rate: int,
engine_id: str,
voice_sig: Optional[dict] = None,
) -> str:
"""Deterministic content hash for a chapter's rendered audio.
``spans`` is an ordered list of ``(voice_id, text, pause_ms_after[, speed])``
(speed optional, defaults to None). Same inputs same key reuse the
cached chapter WAV on a re-run (resume); any change (text, voice, order,
pauses, speed, sample rate, engine, or a voice's resolved signature) → new
key re-render. ``voice_sig`` maps each voice id to a stable signature
string (e.g. ``ref_audio|instruct|seed``) so editing the underlying profile
also invalidates the cache.
"""
payload = {
"sr": int(sample_rate),
"engine": engine_id or "",
"spans": [[s[0], s[1], int(s[2]), (s[3] if len(s) > 3 else None)] for s in spans],
"voices": {k: voice_sig[k] for k in sorted(voice_sig)} if voice_sig else {},
}
raw = json.dumps(payload, sort_keys=True, ensure_ascii=False)
# Content-addressing only — not a security digest. usedforsecurity=False
# keeps bandit's B324 (weak-hash) check quiet.
return hashlib.sha1(raw.encode("utf-8"), usedforsecurity=False).hexdigest()[:20]
# ── Loudness normalization ──────────────────────────────────────────────────
@dataclass(frozen=True)
class LoudnessPreset:
"""A loudnorm target. ``i`` = integrated LUFS, ``tp`` = true-peak ceiling
(dBTP), ``lra`` = loudness range."""
key: str
i: float
tp: float
lra: float
#: ``acx`` targets Audible/ACX submission (≈ -19 LUFS integrated, ≤ -3 dBTP
#: peak — inside ACX's -23…-18 dB RMS / -3 dB peak window). ``podcast`` targets
#: the -16 LUFS streaming norm.
LOUDNESS_PRESETS: dict[str, LoudnessPreset] = {
"acx": LoudnessPreset("acx", -19.0, -3.0, 11.0),
"podcast": LoudnessPreset("podcast", -16.0, -1.5, 11.0),
}
def build_loudnorm_filter(preset: Optional[str]) -> Optional[str]:
"""Return an ``-af`` loudnorm filter string for ``preset``, or ``None`` for
off / unknown (single-pass; two-pass measureapply is a runner enhancement).
"""
if not preset:
return None
p = LOUDNESS_PRESETS.get(preset.lower())
if p is None: # "off", "none", or anything unrecognized → no filter
return None
return f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}"
@dataclass(frozen=True)
class MeasuredLoudness:
"""The five loudnorm measure-pass values (FFmpeg JSON keys), all finite
floats. Fed back into the second (apply) pass as ``measured_*`` + ``offset``."""
input_i: float
input_tp: float
input_lra: float
input_thresh: float
target_offset: float
def build_loudnorm_measure_filter(preset: Optional[str]) -> Optional[str]:
"""First-pass loudnorm filter (``print_format=json``) for ``preset``, or
``None`` for off/unknown mirrors :func:`build_loudnorm_filter`'s lookup
(no whitespace stripping) so the same values count as 'no filter'."""
if not preset:
return None
p = LOUDNESS_PRESETS.get(preset.lower())
if p is None:
return None
return f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}:print_format=json"
def parse_loudnorm_measure(stderr_text: Optional[str]) -> Optional[MeasuredLoudness]:
"""Extract the loudnorm measure JSON from ffmpeg stderr → MeasuredLoudness,
or ``None`` on ANY failure (caller falls back to single-pass). FFmpeg prints
the JSON object amid other non-JSON lines (and possibly a config dump block),
so we take the LAST balanced ``{...}`` via a linear brace-depth scan no
regex (CodeQL-safe), O(n), no backtracking then json.loads + coerce/validate
the five required keys to finite floats."""
if not stderr_text:
return None
# Find the last balanced top-level {...} block via a single linear scan.
start = -1
depth = 0
block = None
for i, ch in enumerate(stderr_text):
if ch == "{":
if depth == 0:
start = i
depth += 1
elif ch == "}":
if depth > 0:
depth -= 1
if depth == 0 and start != -1:
block = stderr_text[start:i + 1] # keep scanning → last wins
if block is None:
return None
try:
obj = json.loads(block)
except (json.JSONDecodeError, ValueError):
return None
if not isinstance(obj, dict):
return None
keys = ("input_i", "input_tp", "input_lra", "input_thresh", "target_offset")
vals = {}
for k in keys:
if k not in obj:
return None
try:
v = float(obj[k])
except (TypeError, ValueError):
return None
if not math.isfinite(v): # rejects "-inf"/"inf"/"nan" (silent clip)
return None
vals[k] = v
return MeasuredLoudness(**vals)
def build_loudnorm_apply_filter(
preset: Optional[str], measured: Optional["MeasuredLoudness"],
) -> Optional[str]:
"""Second-pass (apply) loudnorm filter feeding the measured values back in.
``None`` for off/unknown preset OR when ``measured`` is None (so a caller
that forgot to branch never emits ``measured_I=None``)."""
if not preset or measured is None:
return None
p = LOUDNESS_PRESETS.get(preset.lower())
if p is None:
return None
return (
f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}"
f":measured_I={measured.input_i}:measured_TP={measured.input_tp}"
f":measured_LRA={measured.input_lra}:measured_thresh={measured.input_thresh}"
f":offset={measured.target_offset}:linear=true:print_format=summary"
)
def build_loudnorm_measure_cmd(ffmpeg: str, concat_list_path: str, filt: str) -> list[str]:
"""Pure argv for the measure pass: decode the concat list, run the
print_format=json loudnorm filter, discard audio to the portable null muxer.
Input segment is byte-identical to build_render_cmd so measured == muxed."""
return [
ffmpeg, "-y", "-hide_banner", "-loglevel", "info",
"-f", "concat", "-safe", "0", "-i", str(concat_list_path),
"-af", filt, "-f", "null", "-",
]
# ── FFMETADATA ──────────────────────────────────────────────────────────────
def build_ffmetadata(
chapters: Iterable[tuple[str, int]],
global_meta: Optional[dict] = None,
) -> str:
"""Build an FFMETADATA1 doc: optional global tags + one ``[CHAPTER]`` per
``(title, duration_ms)``. START/END are cumulative millisecond offsets.
"""
lines = [";FFMETADATA1"]
if global_meta:
for field_key, meta_key in _GLOBAL_TAG_KEYS:
val = global_meta.get(field_key)
if val is not None and str(val).strip():
lines.append(f"{meta_key}={_escape_meta(str(val).strip())}")
start = 0
for title, dur_ms in chapters:
end = start + max(0, int(dur_ms))
lines += [
"[CHAPTER]",
"TIMEBASE=1/1000",
f"START={start}",
f"END={end}",
f"title={_escape_meta(title)}",
]
start = end
return "\n".join(lines) + "\n"
def build_concat_list(wav_paths: Iterable[str]) -> str:
"""Build an ffmpeg concat-demuxer list. Single quotes in paths are escaped
the ffmpeg way (``'`` → ``'\\''``) so paths can't break the list or inject
arguments."""
lines = []
for p in wav_paths:
safe = str(p).replace("'", "'\\''")
lines.append(f"file '{safe}'")
return "\n".join(lines) + "\n"
# ── Cover art ───────────────────────────────────────────────────────────────
def validate_cover_image(path: Optional[str]) -> bool:
"""True if ``path`` is a readable jpg/png within the size cap. Anything
dubious (missing, wrong type, too big, unreadable) False, and the caller
simply omits the cover rather than failing the render."""
if not path:
return False
try:
p = Path(path)
return (
p.is_file()
and p.suffix.lower() in _COVER_EXTS
and 0 < p.stat().st_size <= _COVER_MAX_BYTES
)
except OSError:
return False
# ── Render command ──────────────────────────────────────────────────────────
def build_render_cmd(
ffmpeg: str,
concat_list_path: str,
metadata_path: str,
out_path: str,
*,
fmt: str = "m4b",
bitrate: str = "128k",
cover_path: Optional[str] = None,
loudness: Optional[str] = None,
measured: Optional[MeasuredLoudness] = None,
) -> list[str]:
"""Pure argv for muxing chapter WAVs + FFMETADATA into a tagged,
chapter-marked audio file.
Inputs: 0 = concat-demuxer list of chapter WAVs, 1 = FFMETADATA (chapters +
global tags), 2 = cover image (only when present + valid). ``fmt`` is
``m4b`` (AAC in mp4, faststart) or ``mp3`` (libmp3lame). A loudness preset
adds an ``-af loudnorm`` pass; an invalid/oversized cover is silently
dropped (see :func:`validate_cover_image`).
"""
if not _BITRATE_RE.match(bitrate or ""):
bitrate = "128k"
is_mp3 = (fmt or "").lower() == "mp3"
# Cover art is embedded for M4B only. The MP3 muxer rejects an
# ``attached_pic`` video stream via ``-c:v copy`` (produces a corrupt file
# across ffmpeg versions), and a reliable cross-version ID3 APIC path is
# finicky — so for MP3 we skip the cover rather than ship a broken file.
# M4B is the cover-bearing audiobook format anyway.
embed_cover = validate_cover_image(cover_path) and not is_mp3
cmd = [
ffmpeg, "-y", "-hide_banner", "-loglevel", "error",
"-f", "concat", "-safe", "0", "-i", str(concat_list_path),
"-i", str(metadata_path),
]
if embed_cover:
cmd += ["-i", str(cover_path)]
cmd += ["-map", "0:a", "-map_metadata", "1"]
if embed_cover:
cmd += ["-map", "2:v", "-disposition:v", "attached_pic"]
# Two-pass apply when measured values are present; else single-pass. Both
# return None for a non-preset loudness, so the `if filt:` guard below
# gives an off-render no -af (byte-identical to today).
filt = build_loudnorm_apply_filter(loudness, measured) if measured is not None else build_loudnorm_filter(loudness)
if filt:
cmd += ["-af", filt]
if is_mp3:
cmd += ["-c:a", "libmp3lame", "-b:a", bitrate, "-f", "mp3", str(out_path)]
else: # m4b — AAC in an mp4 container
cmd += ["-c:a", "aac", "-b:a", bitrate]
if embed_cover:
cmd += ["-c:v", "copy"]
cmd += ["-movflags", "+faststart", "-f", "mp4", str(out_path)]
return cmd
+195
View File
@@ -0,0 +1,195 @@
"""Durable resume for longform (audiobook / story) renders.
Chapter WAVs are already content-addressed in a shared cache, so re-rendering an
identical plan reuses what finished the synthesis-level resume. The missing
piece this module adds is **durability of the *plan itself***: a render that's
interrupted (crash, app quit, power loss) leaves a `resume.json` manifest in the
job's work dir holding the compiled plan + render params, so the job can be
resumed later *without the user still having the original script* which matters
for Stories, whose plan is compiled from cast+lines and can't be retyped.
Pure file/JSON I/O (no torch, no model) so it's unit-testable. The router wires
it into the SSE renderer (write on start, clear on done) and exposes
``GET /audiobook/jobs`` (resumable) + ``POST /audiobook/resume/{job_id}``.
"""
from __future__ import annotations
import json
import os
import re
from typing import Optional
MANIFEST_VERSION = 1
_MANIFEST_NAME = "resume.json"
# Longform front doors that produce a resumable work dir (job_type → dir prefix).
RESUMABLE_TYPES = ("audiobook", "story")
# A job id is server-generated (uuid4 hex) — confine to a strict token so a
# request-supplied id (the /audiobook/resume/{job_id} path param) can never
# carry a path separator, `..`, NUL, or anything that escapes OUTPUTS_DIR
# (CodeQL py/path-injection). Anchored, single bounded quantifier → ReDoS-safe.
_SAFE_ID_RE = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
# Exact-match allowlist on the WHOLE work-dir name (the value actually joined
# onto OUTPUTS_DIR). Validating the joined string itself — not just job_id — is
# the barrier CodeQL's path-injection query recognizes (same shape as
# audiobook._safe_cover_path's _COVER_NAME_RE). Mirrors RESUMABLE_TYPES.
_SAFE_SEG_RE = re.compile(r"^(?:audiobook|story)_[A-Za-z0-9_-]{1,64}$")
def work_dir(job_type: str, job_id: str) -> Optional[str]:
"""The per-job work directory ``OUTPUTS_DIR/<job_type>_<job_id>``, resolved
strictly inside OUTPUTS_DIR. Returns None for an unknown ``job_type``, an
id that isn't a bare safe token, or any path that escapes OUTPUTS_DIR — so a
crafted ``job_id`` can never reach a foreign path (py/path-injection-safe)."""
if job_type not in RESUMABLE_TYPES or not _SAFE_ID_RE.match(job_id or ""):
return None
# os.path.basename strips any directory component (the sanitizer CodeQL's
# path-injection query recognizes — same as audiobook._safe_cover_path), and
# the exact-match allowlist on the result is a second barrier: the joined
# value is provably a single bare dir name of the expected shape.
seg = os.path.basename(f"{job_type}_{job_id}")
if not _SAFE_SEG_RE.match(seg):
return None
from core.config import OUTPUTS_DIR
root = os.path.realpath(OUTPUTS_DIR)
path = os.path.realpath(os.path.join(root, seg))
# commonpath containment — the form static analysis recognizes (belt over
# the regex). Raises ValueError on mixed drives (Windows) → reject.
try:
if os.path.commonpath([path, root]) != root:
return None
except ValueError:
return None
return path
def manifest_path(job_type: str, job_id: str) -> Optional[str]:
d = work_dir(job_type, job_id)
return os.path.join(d, _MANIFEST_NAME) if d else None
def build_manifest(
*,
job_id: str,
job_type: str,
plan_chapters: list[dict],
params: dict,
title: str = "",
) -> dict:
"""Assemble the manifest dict. ``plan_chapters`` is the canonical span-plan
(``[{title, spans:[{voice_id,text,pause_ms_after,speed}]}]``); ``params`` is
the render kwargs (default_voice / fmt / bitrate / loudness / cover_path /
metadata / lexicon). Pure no I/O."""
return {
"version": MANIFEST_VERSION,
"job_id": job_id,
"job_type": job_type,
"title": title or "",
"total_chapters": len(plan_chapters),
"params": params,
"plan": plan_chapters,
}
def write_manifest(manifest: dict) -> Optional[str]:
"""Persist the manifest to the job work dir. Best-effort — resume is an
enhancement, never block the render returns the path or None on failure /
unsafe id."""
path = manifest_path(manifest.get("job_type", ""), manifest.get("job_id", ""))
if path is None:
return None
try:
os.makedirs(os.path.dirname(path), exist_ok=True)
tmp = path + ".tmp"
with open(tmp, "w", encoding="utf-8") as f:
json.dump(manifest, f, ensure_ascii=False)
os.replace(tmp, path) # atomic — a half-written manifest never resumes
return path
except OSError:
return None
def read_manifest(job_type: str, job_id: str) -> Optional[dict]:
"""Load a job's resume manifest, or None if absent/unreadable/foreign-shape/
unsafe id."""
path = manifest_path(job_type, job_id)
if path is None:
return None
try:
with open(path, encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError):
return None
if not isinstance(data, dict) or data.get("version") != MANIFEST_VERSION:
return None
if not isinstance(data.get("plan"), list):
return None
return data
def clear_manifest(job_type: str, job_id: str) -> None:
"""Remove the manifest once a job completes (no resume needed). Best-effort."""
path = manifest_path(job_type, job_id)
if path is None:
return
try:
os.remove(path)
except OSError:
return # best-effort; already gone or unwritable
def has_manifest(job_type: str, job_id: str) -> bool:
path = manifest_path(job_type, job_id)
return bool(path) and os.path.isfile(path)
def scan_resumable() -> list[dict]:
"""Enumerate resumable jobs by scanning OUTPUTS_DIR for ``<type>_<id>`` work
dirs that hold a ``resume.json``.
Returns ``[{"job_type", "job_id", "manifest_path"}, ]`` where **every path
component is sourced from os.listdir** (the trusted filesystem), never from
request input. The ``manifest_path`` is built here from the listed dir name,
so callers can read it directly without re-deriving a path from a
request-supplied id (CodeQL py/path-injection-safe no tainted value ever
reaches a file operation)."""
from core.config import OUTPUTS_DIR
out: list[dict] = []
try:
root = os.path.realpath(OUTPUTS_DIR)
names = os.listdir(root)
except OSError:
return out
for name in names:
for jt in RESUMABLE_TYPES:
prefix = f"{jt}_"
mpath = os.path.join(root, name, _MANIFEST_NAME)
if name.startswith(prefix) and os.path.isfile(mpath):
out.append({"job_type": jt, "job_id": name[len(prefix):],
"manifest_path": mpath})
return out
def discard_manifest_file(path: str) -> None:
"""Remove a manifest by a path obtained via :func:`scan_resumable` (trusted,
os.listdir-derived). Best-effort used to retire an interrupted job once
it's been resumed under a fresh id."""
try:
os.remove(path)
except OSError:
return
def load_manifest_file(path: str) -> Optional[dict]:
"""Read + validate a manifest from a path obtained via :func:`scan_resumable`
(a trusted, os.listdir-derived path NOT a request-derived one). Returns
None on missing/unreadable/foreign-shape."""
try:
with open(path, encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError):
return None
if not isinstance(data, dict) or data.get("version") != MANIFEST_VERSION:
return None
if not isinstance(data.get("plan"), list):
return None
return data
+68
View File
@@ -0,0 +1,68 @@
"""Two-pass loudnorm measure orchestrator (#28).
The impure half of the two-pass ACX/podcast master: run ffmpeg's measure pass
over the concatenated chapters and parse the printed loudnorm JSON. The pure
builders/parser live in :mod:`services.longform_render`; this only drives ffmpeg.
Contract: **never raises.** Every failure (skip / non-zero rc / timeout / spawn
error / empty or unparseable stderr / silent program) is caught, logged at
WARNING, and converted to ``None`` so the caller falls back to single-pass. A
slow or broken measure must degrade the master, never abort the render.
"""
from __future__ import annotations
import logging
from typing import Optional
from services.longform_render import (
MeasuredLoudness,
build_loudnorm_measure_cmd,
build_loudnorm_measure_filter,
parse_loudnorm_measure,
)
logger = logging.getLogger("omnivoice.loudness")
async def measure_loudness(
ffmpeg: str,
concat_list_path: str,
preset: str,
*,
job_id: str,
) -> Optional[MeasuredLoudness]:
"""Measure the concatenated program's loudness for ``preset`` (acx/podcast),
or ``None`` for off/unknown or on ANY failure ( single-pass fallback).
Only the ffmpeg rc and a short static message are logged never the raw
stderr (it can carry the concat path under OUTPUTS_DIR), keeping the log
local-first / path-safe.
"""
filt = build_loudnorm_measure_filter(preset)
if filt is None:
return None # off / unknown — a normal skip, not an error (no log)
cmd = build_loudnorm_measure_cmd(ffmpeg, concat_list_path, filt)
from services.ffmpeg_utils import run_ffmpeg # lazy → patchable at source
try:
# asyncio.TimeoutError is a subclass of Exception (Py≥3.11) — caught
# here so a slow measure degrades to single-pass instead of killing the
# whole render via the caller's outer except.
rc, _out, err = await run_ffmpeg(cmd, capture=True, job_id=job_id)
except Exception as exc:
logger.warning("loudness measure pass did not run (%s) — single-pass fallback",
type(exc).__name__)
return None
if rc != 0:
logger.warning("loudness measure pass exited rc=%s — single-pass fallback", rc)
return None
try:
stderr_text = err.decode("utf-8", "replace") if isinstance(err, (bytes, bytearray)) else (err or "")
except Exception:
return None
measured = parse_loudnorm_measure(stderr_text)
if measured is None:
logger.warning("loudness measure output unparseable — single-pass fallback")
return measured
__all__ = ["measure_loudness"]
+126
View File
@@ -0,0 +1,126 @@
"""Per-agent MCP voice bindings (Wave 2.2 / Spec 2).
An MCP client identifies itself with the ``X-OmniVoice-Client-Id`` header.
Each client can be bound to a default voice profile + engine so different
agents speak in different voices ("Claude Code in Morgan, Cursor in
Scarlett"). Pure data layer over the ``mcp_client_bindings`` table — the
FastMCP tools call :func:`resolve_voice`; the Settings UI calls the CRUD
helpers via the REST router.
"""
from __future__ import annotations
import time
from typing import Optional
from core.db import db_conn
def list_bindings() -> list[dict]:
with db_conn() as conn:
# SQLite sorts NULL as smallest, so DESC naturally puts never-seen
# bindings after recently-active ones.
rows = conn.execute(
"SELECT * FROM mcp_client_bindings ORDER BY last_seen_at DESC, created_at DESC"
).fetchall()
return [dict(r) for r in rows]
def get_binding(client_id: str) -> Optional[dict]:
with db_conn() as conn:
row = conn.execute(
"SELECT * FROM mcp_client_bindings WHERE client_id=?", (client_id,)
).fetchone()
return dict(row) if row else None
def upsert_binding(
client_id: str,
*,
label: Optional[str] = None,
profile_id: Optional[str] = None,
default_engine: Optional[str] = None,
) -> dict:
"""Create or update a binding. Fields left as None on an existing row are
preserved; on a new row they default to empty/null."""
if not client_id or not client_id.strip():
raise ValueError("client_id must be non-empty")
cid = client_id.strip()
existing = get_binding(cid)
now = time.time()
if existing:
merged = {
"label": existing["label"] if label is None else label,
"profile_id": existing["profile_id"] if profile_id is None else (profile_id or None),
"default_engine": existing["default_engine"] if default_engine is None else (default_engine or None),
}
with db_conn() as conn:
conn.execute(
"UPDATE mcp_client_bindings SET label=?, profile_id=?, default_engine=? WHERE client_id=?",
(merged["label"], merged["profile_id"], merged["default_engine"], cid),
)
else:
with db_conn() as conn:
conn.execute(
"INSERT INTO mcp_client_bindings "
"(client_id, label, profile_id, default_engine, last_seen_at, created_at) "
"VALUES (?, ?, ?, ?, NULL, ?)",
(cid, label or "", profile_id or None, default_engine or None, now),
)
return get_binding(cid)
def delete_binding(client_id: str) -> bool:
with db_conn() as conn:
cur = conn.execute("DELETE FROM mcp_client_bindings WHERE client_id=?", (client_id,))
return cur.rowcount > 0
def touch_last_seen(client_id: str) -> None:
"""Best-effort 'last heard from this agent' stamp. Never raises — it's
telemetry for the Settings list, not load-bearing."""
if not client_id:
return
try:
with db_conn() as conn:
conn.execute(
"UPDATE mcp_client_bindings SET last_seen_at=? WHERE client_id=?",
(time.time(), client_id),
)
except Exception:
pass
def _global_default_profile() -> Optional[str]:
"""The fallback voice when a client has no binding. Reads the same
pref the Settings 'default playback voice' would set; None if unset."""
try:
from core import prefs
return prefs.get("mcp_default_profile_id") or None
except Exception:
return None
def resolve_voice(client_id: Optional[str], explicit_profile_id: Optional[str]) -> dict:
"""Resolve which voice an MCP speak call should use.
Precedence (Spec 2): explicit tool arg the client's binding →
the global default nothing (caller decides / errors with a hint).
Returns ``{profile_id, default_engine, source}`` where ``source`` is one
of ``explicit`` | ``binding`` | ``global`` | ``none`` for diagnostics.
"""
if explicit_profile_id:
return {"profile_id": explicit_profile_id, "default_engine": None, "source": "explicit"}
if client_id:
binding = get_binding(client_id)
if binding and binding.get("profile_id"):
return {
"profile_id": binding["profile_id"],
"default_engine": binding.get("default_engine"),
"source": "binding",
}
g = _global_default_profile()
if g:
return {"profile_id": g, "default_engine": None, "source": "global"}
return {"profile_id": None, "default_engine": None, "source": "none"}
+158
View File
@@ -0,0 +1,158 @@
"""Single lifecycle surface for loaded models (MM2-04).
Before this, ``GET /model/loaded`` and ``POST /model/unload`` each hand-rolled
enumeration/dispatch across three worlds the in-process TTS+ASR model
(``model_manager``), the diarization pipeline, and subprocess sidecars
(``subprocess_backend``). This module owns that logic so the routers are thin
delegations and there's one place to reason about model lifecycle.
Response shapes are preserved exactly the frontend (hooks.ts model status +
the flush dropdown) depends on ``{models, count}`` and
``{unloaded, success, ...}``.
"""
from __future__ import annotations
import os
from typing import Optional
import services.model_manager as mm
from services.model_manager import get_best_device
def _tts_vram_mb() -> float:
"""Best-effort allocated VRAM for the in-process model. Accurate on CUDA,
sparse on MPS, 0 elsewhere degrade gracefully, never raise."""
try:
torch = mm._lazy_torch()
if torch.cuda.is_available():
return torch.cuda.memory_allocated() / (1024 ** 2)
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
driver = getattr(torch.mps, "driver_allocated_memory", None)
if driver:
return driver() / (1024 ** 2)
except Exception:
pass
return 0.0
def _asr_device() -> str:
"""Where the ASR pipe actually lives, rather than a hardcoded 'cpu'."""
pipe = getattr(mm.model, "_asr_pipe", None)
for attr in ("device",):
dev = getattr(pipe, attr, None)
if dev is not None:
return str(dev)
return "cpu"
def list_loaded() -> dict:
"""Enumerate every currently-loaded model. Shape: ``{"models": [...],
"count": n}`` with per-model id/name/checkpoint/device/vram_mb/unloadable
(+ optional ``note``)."""
models: list[dict] = []
# 1. In-process TTS model (OmniVoice)
if mm.model is not None:
try:
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, "parameters") else get_best_device()
except Exception:
device = get_best_device()
models.append({
"id": "tts",
"name": "OmniVoice TTS",
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
"device": device,
"vram_mb": round(_tts_vram_mb(), 1),
"unloadable": True,
})
# 2. ASR (WhisperX) — co-loaded with and released alongside the TTS model.
# Honest reporting (MM2-03): the device is read from the pipe, and the
# dead "unload" button is explained by a note rather than left silent.
if mm.model is not None and getattr(mm.model, "_asr_pipe", None) is not None:
models.append({
"id": "asr",
"name": "WhisperX ASR",
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"device": _asr_device(),
"vram_mb": 0,
"unloadable": False,
"note": "released with the TTS model",
})
# 3. Diarization pipeline
if mm._diar_pipeline is not None:
models.append({
"id": "diarization",
"name": "Pyannote Diarization",
"checkpoint": "pyannote/speaker-diarization-3.1",
"device": get_best_device(),
"vram_mb": 0,
"unloadable": True,
})
# 4. Subprocess engine sidecars — each holds a process (and on GPU, VRAM)
# until idle-reaped. VRAM is reported by the child itself when available
# (MM2-08); 0 means CPU-only or not-yet-measured. Enumeration must never
# break the panel.
try:
from services.subprocess_backend import list_live_sidecars
for s in list_live_sidecars():
models.append({
"id": f"sidecar:{s['id']}",
"name": f"{s['id']} (sidecar)",
"checkpoint": s["id"],
"device": get_best_device(),
"vram_mb": round(float(s.get("vram_mb") or 0), 1),
"unloadable": True,
})
except Exception:
pass
return {"models": models, "count": len(models)}
async def unload(model_id: str) -> dict:
"""Unload one model by id. Preserves the original per-id response shapes.
``tts`` | ``diarization`` | ``sidecar:<id>`` | ``sidecars``. Raises
ValueError for an unknown id (router maps to HTTP 400)."""
if model_id == "sidecars" or model_id.startswith("sidecar:"):
from services.subprocess_backend import unload_all_sidecars, unload_sidecar
n = unload_all_sidecars() if model_id == "sidecars" else unload_sidecar(model_id.split(":", 1)[1])
return {"unloaded": model_id, "success": n > 0, "count": n,
**({} if n > 0 else {"reason": "not running or busy"})}
if model_id == "tts":
async with mm._model_lock:
if mm.model is not None:
mm.model = None
mm.free_vram()
return {"unloaded": "tts", "success": True}
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
if model_id == "diarization":
if mm._diar_pipeline is not None:
mm._diar_pipeline = None
mm.free_vram()
return {"unloaded": "diarization", "success": True}
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
raise ValueError(f"Unknown model id: {model_id}")
async def unload_all() -> dict:
"""Release every releasable model — in-process TTS + diarization + all
sidecars. Convenience for app shutdown / a global flush."""
results = {}
for mid in ("tts", "diarization", "sidecars"):
try:
results[mid] = await unload(mid)
except Exception as exc: # noqa: BLE001
results[mid] = {"unloaded": mid, "success": False, "reason": str(exc)}
return {"unloaded_all": True, "results": results}
def free_vram() -> None:
"""One import surface for callers that just want to drop GPU caches."""
mm.free_vram()
+275 -23
View File
@@ -2,6 +2,7 @@ import os
import time
import asyncio
import logging
import threading
from concurrent.futures import ThreadPoolExecutor
# ── Lazy imports ─────────────────────────────────────────────────────
@@ -110,7 +111,8 @@ def __getattr__(name: str):
model = None # type: ignore
_model_lock = asyncio.Lock()
_last_used = time.time()
_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS
# Idle timeout is resolved per-tick in _resolve_idle_timeout() (MM2-05) from
# prefs/env/core.config — no module-level duplicate of IDLE_TIMEOUT_SECONDS.
# ── Loading sub-stage tracker ────────────────────────────────────────
# Updated by _load_model_sync() so get_model_status() can report
@@ -195,12 +197,23 @@ def get_best_device():
"""Detect the best available compute device.
Priority: CUDA/ROCm > Intel XPU > DirectML > MPS > CPU
"""
torch = _lazy_torch()
# ── NVIDIA CUDA or AMD ROCm ──────────────────────────────────────
# ROCm-enabled PyTorch reports through torch.cuda, so this covers both.
if torch.cuda.is_available():
The *family* decision delegates to ``core.device_caps.detect_host_caps()``
(the single source of truth) so the probe and this loader can never
disagree. This function keeps the side-effects the probe deliberately
avoids: the ROCm ``HSA_OVERRIDE_GFX_VERSION`` env override and the
DirectML device-string return (DirectML is not a torch device family, so
the probe reports it as ``cpu`` we still resolve the real device string
here for Windows DirectML users). The string contract is unchanged:
``"cuda"`` / ``"xpu"`` / a DirectML device string / ``"mps"`` / ``"cpu"``.
"""
from core.device_caps import detect_host_caps
torch = _lazy_torch()
family = detect_host_caps().family
# ── NVIDIA CUDA or AMD ROCm (both present through torch.cuda) ─────
if family in ("cuda", "rocm"):
_configure_rocm_if_needed(torch)
compatible, warning = check_device_compatibility()
if not compatible:
@@ -208,15 +221,23 @@ def get_best_device():
return "cuda"
# ── Intel Arc / discrete GPU via IPEX ────────────────────────────
try:
import intel_extension_for_pytorch # noqa: F401
if hasattr(torch, "xpu") and torch.xpu.is_available():
if family == "xpu":
try:
logger.info("Using Intel XPU device: %s", torch.xpu.get_device_name(0))
return "xpu"
except ImportError:
pass
except Exception:
logger.info("Using Intel XPU device")
return "xpu"
# ── DirectML — universal Windows GPU (AMD, Intel, NVIDIA fallback)
# ── Apple Silicon MPS ────────────────────────────────────────────
# Checked BEFORE DirectML to mirror the probe's family-priority order
# (cuda > rocm > xpu > mps; DirectML is not a torch family) so the loader
# and detect_host_caps() never disagree on a host that somehow exposes both.
if family == "mps":
return "mps"
# ── DirectML — universal Windows GPU (probe reports this as "cpu") ─
# Reached only when no torch family was detected (family == "cpu"), which is
# exactly the DirectML case — the probe classifies DirectML hosts as cpu.
try:
import torch_directml
if torch_directml.device_count() > 0:
@@ -225,12 +246,184 @@ def get_best_device():
except ImportError:
pass
# ── Apple Silicon MPS ────────────────────────────────────────────
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
return "mps"
return "cpu"
_COMPILE_ERR_MODULE_PREFIXES = ("torch._dynamo", "torch._inductor", "torch.fx", "triton")
_COMPILE_ERR_TB_MARKERS = ("/_dynamo/", "/_inductor/", "/triton/", "torch/fx/")
_COMPILE_ERR_MSG_MARKERS = (
"dynamo", "inductor", "triton", "cudagraph",
"symbolically trace", "torch.compile", "fx graph",
)
def _is_compile_runtime_failure(exc: BaseException) -> bool:
"""True when an exception originates in the torch.compile stack (Dynamo /
Inductor / Triton / FX / CUDA-graph trees) rather than in the model itself.
#278: on GPU architectures Triton doesn't support yet (e.g. Blackwell
sm_120), the compiled model dies mid-generation with errors like
"Detected that you are using FX to symbolically trace a dynamo-optimized
function" or an AssertionError out of torch/_inductor/cudagraph_trees.py.
Walks the exception chain and checks (a) the exception type's module,
(b) the message, (c) the traceback file paths the cudagraph case is a
bare AssertionError, so the traceback check is load-bearing.
"""
import traceback as _tb
seen: set[int] = set()
cur: BaseException | None = exc
while cur is not None and id(cur) not in seen:
seen.add(id(cur))
mod = type(cur).__module__ or ""
if mod.startswith(_COMPILE_ERR_MODULE_PREFIXES):
return True
msg = str(cur).lower()
if any(marker in msg for marker in _COMPILE_ERR_MSG_MARKERS):
return True
try:
for frame in _tb.extract_tb(cur.__traceback__):
filename = (frame.filename or "").replace("\\", "/")
if any(marker in filename for marker in _COMPILE_ERR_TB_MARKERS):
return True
except Exception as traceback_scan_error:
logging.debug(
"Skipping traceback marker scan while classifying compile runtime failure: %s",
traceback_scan_error,
)
# Follow the chain, honoring `raise ... from None` (the eager-retry
# path suppresses the original compile error so a genuine eager
# failure isn't misclassified as a compile failure).
if cur.__cause__ is not None:
cur = cur.__cause__
elif not cur.__suppress_context__:
cur = cur.__context__
else:
cur = None
return False
def _install_compile_fallback(_model) -> None:
"""Wrap ``model.generate`` so a torch.compile failure at inference time
falls back to the eager (uncompiled) model instead of failing the
generation (#278).
All TTS paths (generate, archetype previews, dub, stream, batch) funnel
through ``model.generate``, so this is the single choke point. On a
compile-stack failure we: log a clear warning, restore the eager module
(``OptimizedModule._orig_mod``), disable compile for the rest of the
session via ``engine_env.mark_compile_runtime_failure``, reset dynamo
state, and retry the call once eagerly. Non-compile errors (real OOM,
validation, ) propagate unchanged fully backward compatible for users
whose torch.compile works.
"""
orig_generate = _model.generate
def _generate_with_compile_fallback(*args, **kwargs):
try:
return orig_generate(*args, **kwargs)
except Exception as exc:
compiled = getattr(_model, "llm", None)
eager = getattr(compiled, "_orig_mod", None)
if eager is None or not _is_compile_runtime_failure(exc):
raise
logger.warning(
"torch.compile runtime failure during generation (%s: %s) — "
"falling back to the eager model and disabling torch.compile "
"for this session. Generation is being retried without it.",
type(exc).__name__, exc,
)
from services import engine_env
engine_env.mark_compile_runtime_failure(f"{type(exc).__name__}: {exc}")
_model.llm = eager
try:
torch = _lazy_torch()
torch._dynamo.reset()
except Exception as reset_exc:
logger.debug(
"Non-fatal: failed to reset torch._dynamo state after compile failure (%s: %s). "
"Continuing with eager fallback.",
type(reset_exc).__name__,
reset_exc,
)
try:
return orig_generate(*args, **kwargs)
except Exception as eager_exc:
# `from None` so a genuine eager failure (e.g. a real OOM)
# isn't chained to — and misclassified as — the compile error.
raise eager_exc from None
_model.generate = _generate_with_compile_fallback
# ── #315: thread affinity for cudagraph-compiled models ─────────────────────
# `torch.compile(mode="reduce-overhead")` captures CUDA graphs, and captured
# graph state is **thread-local** (torch/_inductor/cudagraph_trees keys its
# tree manager off the capturing thread). The `_gpu_pool` runs up to
# `_GPU_WORKER_CAP` threads, so render #1 captures the graph on worker A and a
# later render dispatched to worker B replays against mismatched cudagraph
# state — silently corrupting the audio (static / slowed playback, no
# exception, so the #278 eager fallback never fires). Fix: every call into a
# cudagraph-compiled model executes on ONE dedicated thread; uncompiled
# models (CPU / MPS / Windows-no-Triton / compile-disabled) keep the full pool.
_TORCH_COMPILE_MODE = "reduce-overhead"
# Compile modes that enable CUDA graphs under the hood — these need the
# single-thread affinity below. "default" / "max-autotune-no-cudagraphs"
# would not.
_CUDAGRAPH_COMPILE_MODES = frozenset({"reduce-overhead", "max-autotune"})
_compiled_inference_executor: "ThreadPoolExecutor | None" = None
_compiled_inference_thread_ident: "int | None" = None
def _get_compiled_inference_executor() -> ThreadPoolExecutor:
"""The single-thread executor that owns ALL inference on a compiled model.
Created lazily the first time a model is compiled with a cudagraph mode;
reused across model reloads (idle unload reload keeps the same thread,
which is fine a fresh compile simply captures its graphs there too).
The worker is spun up eagerly so its thread ident is known for the
re-entrancy guard in `_install_compile_thread_affinity`.
"""
global _compiled_inference_executor, _compiled_inference_thread_ident
if _compiled_inference_executor is None:
_compiled_inference_executor = ThreadPoolExecutor(
max_workers=1, thread_name_prefix="compiled-infer",
)
_compiled_inference_thread_ident = _compiled_inference_executor.submit(
threading.get_ident
).result()
return _compiled_inference_executor
def _install_compile_thread_affinity(_model) -> None:
"""Pin every ``model.generate`` call to the dedicated compile thread (#315).
Wraps ``model.generate`` (the single choke point all TTS paths funnel
through generate, archetype previews, dub, stream, batch) so the call
body always runs on `_get_compiled_inference_executor()`'s one thread.
That makes the thread that *captures* the CUDA graph on the first render
and the thread that *replays* it on every later render the same thread,
deterministically, regardless of which `_gpu_pool` worker dispatched it.
Installed AFTER `_install_compile_fallback`, so the call-time order is:
caller thread hop to the dedicated thread eager-fallback wrapper
real generate (the #278 classification/retry also runs on the dedicated
thread, with native tracebacks). The hop is a no-op when already on the
dedicated thread a 1-worker executor submitting to itself would
deadlock, so the re-entrancy guard is load-bearing.
"""
executor = _get_compiled_inference_executor()
inner_generate = _model.generate
def _generate_on_compile_thread(*args, **kwargs):
if threading.get_ident() == _compiled_inference_thread_ident:
return inner_generate(*args, **kwargs)
return executor.submit(inner_generate, *args, **kwargs).result()
_model.generate = _generate_on_compile_thread
def _set_loading(sub_stage: str, detail: str = "", error: str | None = None, progress: float | None = None):
"""Update the loading detail dict atomically."""
_loading_detail["sub_stage"] = sub_stage
@@ -290,9 +483,23 @@ def _load_model_sync():
logger.info("Preloading PyTorch Whisper with TTS model.")
else:
logger.info("Skipping PyTorch Whisper preload; ASR will load on demand.")
_model = OmniVoice.from_pretrained(
checkpoint, device_map=device, dtype=torch.float16, load_asr=preload_asr,
)
try:
_model = OmniVoice.from_pretrained(
checkpoint, device_map=device, dtype=torch.float16, load_asr=preload_asr,
)
except OSError as e:
# #352: a truncated HF cache surfaces here as "does not appear to
# have a file named pytorch_model.bin or model.safetensors".
# Translate to an actionable message instead of the raw
# transformers error.
if "does not appear to have a file named" in str(e):
raise RuntimeError(
f"The TTS model cache for {checkpoint} is incomplete "
"(weights missing — usually an interrupted download). "
"Open Settings → Models, delete the OmniVoice TTS model, "
"and install it again."
) from e
raise
try:
# plan-02 (#65): gate on Triton availability (+ user setting), not
@@ -303,8 +510,38 @@ def _load_model_sync():
if should_torch_compile(device):
_set_loading("compiling", "Compiling model (torch.compile)…")
_model.llm = torch.compile(_model.llm, mode="reduce-overhead")
logger.info("torch.compile applied.")
try:
_model.llm = torch.compile(_model.llm, mode=_TORCH_COMPILE_MODE)
except Exception as compile_exc:
# #278: compile is an optimization, never a point of
# failure — keep the eager model and remember the failure
# so later loads this session skip compile up front.
from services.engine_env import mark_compile_runtime_failure
mark_compile_runtime_failure(f"{type(compile_exc).__name__}: {compile_exc}")
logger.warning(
"torch.compile failed (%s) — continuing with the eager model.",
compile_exc,
)
else:
# Compilation is lazy: Dynamo/Inductor/Triton can still
# blow up on the first *forward* (e.g. unsupported new GPU
# archs, #278). Wrap generate so that falls back to eager
# instead of failing the generation.
_install_compile_fallback(_model)
if _TORCH_COMPILE_MODE in _CUDAGRAPH_COMPILE_MODES:
# #315: reduce-overhead uses CUDA graphs, whose
# captured state is thread-local. Pin all inference to
# one dedicated thread so a later render dispatched to
# a different _gpu_pool worker can't replay a graph it
# didn't capture (static / slowed audio from the 2nd
# render onward).
_install_compile_thread_affinity(_model)
logger.info(
"torch.compile mode %r uses CUDA graphs — compiled-model "
"inference pinned to a single dedicated thread (#315).",
_TORCH_COMPILE_MODE,
)
logger.info("torch.compile applied.")
except Exception as e:
logger.info("torch.compile skipped: %s", e)
@@ -443,13 +680,28 @@ def get_model_status():
result["error"] = err
return result
def _resolve_idle_timeout() -> float:
"""In-process model idle timeout in seconds (MM2-05): prefs store → env →
core.config default, env winning. Resolved per-tick so a settings change
takes effect without a restart."""
try:
from core import prefs
return float(prefs.resolve(
"idle_timeout_seconds",
env="OMNIVOICE_IDLE_TIMEOUT_S",
default=IDLE_TIMEOUT_SECONDS,
))
except (TypeError, ValueError, ImportError):
return float(IDLE_TIMEOUT_SECONDS)
async def idle_worker():
global model
torch = _lazy_torch()
while True:
await asyncio.sleep(30)
async with _model_lock:
if model is not None and time.time() - _last_used > _IDLE_TIMEOUT_SECONDS:
if model is not None and time.time() - _last_used > _resolve_idle_timeout():
logger.info("Idle timeout reached. Unloading OmniVoice model to free VRAM.")
model = None
free_vram()
+187
View File
@@ -0,0 +1,187 @@
"""
Speech-onset alignment for transcript segments (issue #280, item 1).
Whisper-family ASR models are prone to stretching a segment's *start* back
over leading non-speech (intro music, room tone, silence). The classic
symptom from the issue report: the speaker starts talking at 0:020:03,
but the first transcript segment says ``start=0.0`` so the dubbed line
plays the moment the video begins and everything feels desynchronised.
``snap_segment_starts`` post-processes segments against the actual audio
(ideally the Demucs-isolated vocals track, which the dub pipeline already
produces): for each segment it scans the waveform inside ``[start, end]``
for the first frame whose RMS rises above an adaptive threshold and moves
``start`` forward to just before that onset.
Design constraints:
* **Forward-only.** A segment start is never moved earlier that could
collide with the previous speaker. We only trim leading non-speech.
* **Conservative.** Shifts below ``min_shift_s`` are ignored (word-level
timestamps are usually within ~100 ms already); a minimum segment
duration is always preserved; segments whose window looks silent
(no frame above the absolute floor) are left untouched.
* **Pure NumPy.** No model, no platform-specific code identical
behaviour on macOS / Windows / Linux, trivially unit-testable.
"""
from __future__ import annotations
import logging
from typing import Sequence
import numpy as np
logger = logging.getLogger("omnivoice.onset_align")
# Analysis frame for RMS energy. 20 ms is fine-grained enough to localise
# a syllable onset while staying cheap (a 10-min track is ~30k frames).
FRAME_S = 0.02
# Keep this much audio before the detected onset so plosives/breaths that
# sit just under the threshold aren't clipped off.
PRE_ROLL_S = 0.05
# Shifts smaller than this are noise — word-level ASR timestamps are
# usually accurate to ~0.1 s, so don't churn segment data for less.
MIN_SHIFT_S = 0.15
# Never shrink a segment below this duration when shifting its start.
MIN_SEG_DUR_S = 0.30
# A frame must exceed `RELATIVE_THRESHOLD × peak RMS of the window` to
# count as speech onset…
RELATIVE_THRESHOLD = 0.10
# …and the window's peak RMS must exceed this absolute floor, otherwise
# the whole window is treated as silence and left alone (we'd only be
# snapping to noise).
ABS_RMS_FLOOR = 1e-3
def _frame_rms(x: np.ndarray, frame_len: int) -> np.ndarray:
"""RMS per non-overlapping frame; the ragged tail frame is dropped."""
n = (len(x) // frame_len) * frame_len
if n <= 0:
return np.zeros(0, dtype=np.float32)
frames = x[:n].reshape(-1, frame_len).astype(np.float64, copy=False)
return np.sqrt((frames * frames).mean(axis=1)).astype(np.float32)
def detect_speech_onset(
audio: np.ndarray,
sr: int,
start_s: float,
end_s: float,
) -> float | None:
"""Return the absolute time (s) of the first speech-like frame inside
``[start_s, end_s]``, or ``None`` when the window is empty / silent.
"""
if sr <= 0 or end_s <= start_s:
return None
i0 = max(0, int(start_s * sr))
i1 = min(len(audio), int(end_s * sr))
if i1 <= i0:
return None
window = audio[i0:i1]
frame_len = max(1, int(FRAME_S * sr))
rms = _frame_rms(window, frame_len)
if rms.size == 0:
return None
peak = float(rms.max())
if peak < ABS_RMS_FLOOR:
return None # whole window is effectively silent
threshold = max(RELATIVE_THRESHOLD * peak, ABS_RMS_FLOOR)
above = np.nonzero(rms >= threshold)[0]
if above.size == 0:
return None
return start_s + float(above[0]) * (frame_len / sr)
# Hysteresis for full-track onset listing: after a frame crosses the
# threshold, the energy must stay *below* it for at least this long before
# the next rise counts as a new onset. Stops syllable-internal dips from
# spamming the timeline with ticks.
MIN_ONSET_GAP_S = 0.15
def detect_speech_onsets(audio: np.ndarray, sr: int) -> list[float]:
"""Return the times (s) of every speech-like onset across the whole track.
Powers the timeline editor's snap-to-onset ticks (issue #280, item 3):
frame RMS over the full track, single adaptive threshold
``max(RELATIVE_THRESHOLD × peak, ABS_RMS_FLOOR)``, and hysteresis a
new onset registers only when the energy rises above the threshold
after at least ``MIN_ONSET_GAP_S`` below it.
Pure NumPy, identical behaviour on every platform. Returns ``[]`` for
empty/silent audio.
"""
if sr <= 0 or audio is None or len(audio) == 0:
return []
if audio.ndim > 1:
audio = audio.mean(axis=1)
frame_len = max(1, int(FRAME_S * sr))
rms = _frame_rms(audio, frame_len)
if rms.size == 0:
return []
peak = float(rms.max())
if peak < ABS_RMS_FLOOR:
return [] # whole track is effectively silent
threshold = max(RELATIVE_THRESHOLD * peak, ABS_RMS_FLOOR)
gap_frames = max(1, int(round(MIN_ONSET_GAP_S / FRAME_S)))
frame_s = frame_len / sr
onsets: list[float] = []
below_run = gap_frames # armed, so speech at t=0 still counts
for i, v in enumerate(rms):
if v >= threshold:
if below_run >= gap_frames:
onsets.append(round(i * frame_s, 3))
below_run = 0
else:
below_run += 1
return onsets
def snap_segment_starts(
segments: Sequence[dict],
audio: np.ndarray,
sr: int,
*,
min_shift_s: float = MIN_SHIFT_S,
) -> int:
"""Snap each segment's ``start`` forward to the actual speech onset.
Mutates the segment dicts in place (the shape the dub pipeline passes
around). Returns the number of segments adjusted.
``audio`` should be mono float; the Demucs vocals track gives the best
signal but the mixed track still beats nothing.
"""
if sr <= 0 or audio is None or len(audio) == 0:
return 0
if audio.ndim > 1:
audio = audio.mean(axis=1)
adjusted = 0
for seg in segments:
try:
start = float(seg.get("start", 0.0))
end = float(seg.get("end", 0.0))
except (TypeError, ValueError):
continue
if end - start < MIN_SEG_DUR_S + min_shift_s:
continue # too short for a meaningful shift
onset = detect_speech_onset(audio, sr, start, end)
if onset is None:
continue
new_start = max(start, onset - PRE_ROLL_S)
shift = new_start - start
if shift < min_shift_s:
continue
# Preserve a minimum playable duration.
new_start = min(new_start, end - MIN_SEG_DUR_S)
if new_start - start < min_shift_s:
continue
seg["start"] = round(new_start, 3)
adjusted += 1
if adjusted:
logger.info("onset-align: snapped %d/%d segment start(s) to speech onset",
adjusted, len(segments))
return adjusted
+436
View File
@@ -0,0 +1,436 @@
"""`.ovsvoice` persona-bundle format (#29 / parity §R3 G1).
A portable ZIP that packages a voice profile's identity + an optional reference
clip + a consent attestation + an SPDX license tag + a watermarked preview.
This module owns the **pure, model-free** core: the format constants, SPDX
normalization, and the manifest/consent builders. The audio preview + ZIP
pack/unpack (which lazily import torchaudio/watermark) layer on top of these.
"""
from __future__ import annotations
import io
import json
import os
import re
import time
import zipfile
from dataclasses import dataclass
from typing import Optional
# ── Format constants ─────────────────────────────────────────────────────────
OVSVOICE_FORMAT = "ovsvoice"
OVSVOICE_SCHEMA_VERSION = 1
MAX_BUNDLE_BYTES = 100 * 1024 * 1024 # 100 MB (mirrors marketplace cap)
_MIN_CONSENT_AUDIO_BYTES = 1000 # the consent-recording floor
DEFAULT_LICENSE = "LicenseRef-OmniVoice-Personal"
PREVIEW_MAX_SECONDS = 8.0 # preview length cap (A6)
PREVIEW_SAMPLE_RATE = 24_000 # preview rate; mono, 16-bit PCM (A8)
# Audio member prefixes the importer recognises. The ZIP member NAME is never
# used to build an output path (zip-slip safe, B10) — only its extension, and
# only after the linear allowlist below.
_AUDIO_MEMBER_PREFIXES = ("ref_audio", "locked_audio", "consent_audio", "preview")
# Reused verbatim from profiles.py:306 — single linear quantifier, no ReDoS.
_MEMBER_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
# Membership allowlist for SPDX validation — a fixed-string set + the
# ``LicenseRef-`` prefix. NO regex over the (user-supplied) SPDX string, so this
# carries no CodeQL py/polynomial-redos surface.
_SPDX_ALLOWLIST: frozenset[str] = frozenset({
"CC0-1.0", "CC-BY-4.0", "CC-BY-SA-4.0", "CC-BY-NC-4.0", "CC-BY-NC-SA-4.0",
"CC-BY-ND-4.0", "MIT", "Apache-2.0", "LicenseRef-OmniVoice-Personal",
})
class BundleError(Exception):
"""A bundle build/parse failure carrying the HTTP status the router maps to."""
def __init__(self, status: int, detail: str):
super().__init__(detail)
self.status = status
self.detail = detail
class NoPreviewSource(Exception):
"""No readable source clip exists to build a preview from (A2/A3/A4/A5/A12).
The router maps this to HTTP 503."""
def _safe_member_ext(member_name: str) -> str:
"""The extension for a ZIP member, allowlisted to ``^\\.[A-Za-z0-9]{1,8}$``
(else ``.wav``). Used ONLY to choose the output extension never the path
(B11). Linear regex, no ReDoS."""
ext = os.path.splitext(member_name)[1]
return ext if _MEMBER_EXT_RE.match(ext) else ".wav"
def normalize_spdx(spdx: Optional[str]) -> str:
"""Return a safe SPDX id: the value if it's allowlisted or a ``LicenseRef-``
custom id, else :data:`DEFAULT_LICENSE`. Never raises, never 400s a junk
id (incl. shell-injection attempts) normalizes to the default. Membership /
fixed-prefix only no regex (CodeQL-clean)."""
if not spdx or not isinstance(spdx, str):
return DEFAULT_LICENSE
s = spdx.strip()
if s in _SPDX_ALLOWLIST or s.startswith("LicenseRef-"):
return s
return DEFAULT_LICENSE
def build_manifest(
profile: dict,
*,
license_spdx: str,
tags: list[str],
engine_id: str = "",
custom_license_text: Optional[str] = None,
preview: Optional[dict] = None,
members: Optional[dict] = None,
omnivoice_version: str = "",
) -> dict:
"""Build the ``manifest.json`` object for a profile row. Mirrors the legacy
``_bundle_metadata`` persona fields, adds the format discriminator, license
(normalized never raises on a bad id, A19), tags, preview + members blocks.
Pure: no I/O, no model."""
return {
"format": OVSVOICE_FORMAT,
"schema_version": OVSVOICE_SCHEMA_VERSION,
"omnivoice_version": omnivoice_version or "",
"exported_at": time.time(),
"persona": {
"name": profile.get("name") or "",
"kind": profile.get("kind") or "clone",
"language": profile.get("language") or "Auto",
"personality": profile.get("personality") or "",
"instruct": profile.get("instruct") or "",
"ref_text": profile.get("ref_text") or "",
"seed": profile.get("seed"), # int or None (A16)
"is_locked": bool(profile.get("is_locked")),
"vd_states": profile.get("vd_states"), # JSON string or None (A15) — never re-parsed
},
"engine": {"id": engine_id or "", "design_params": None},
"license": {"spdx": normalize_spdx(license_spdx), "custom_text": custom_license_text or None},
"tags": list(tags or []),
"preview": preview, # set by the audio step; None for legacy/no-preview
"members": members or {"ref_audio": None, "locked_audio": None, "consent_audio": None},
}
def build_consent_json(profile: dict, *, has_recording: bool) -> Optional[dict]:
"""The optional ``consent.json`` for a profile, or None when there's nothing
to attest. A ``design`` persona attests as designed-synthetic by definition;
a verified clone attests as a self-recorded statement. Import treats these
fields as ADVISORY real verification needs the actual consent_audio member
(see the import rules), so this can't forge verified-own-voice."""
kind = profile.get("kind") or "clone"
consent_text = (profile.get("consent_text") or "").strip()
verified = bool(profile.get("verified_own_voice"))
if kind == "design":
method = "designed-synthetic"
verified = True
elif verified or consent_text or has_recording:
method = "self-recorded-statement"
else:
return None # nothing to attest
recorded_at = profile.get("consent_recorded_at")
try:
recorded_at = float(recorded_at)
except (TypeError, ValueError):
recorded_at = time.time()
return {
"verified_own_voice": verified,
"method": method,
"consent_text": consent_text,
"recorded_at": recorded_at,
"has_recording": bool(has_recording),
}
def _legacy_metadata(profile: dict, omnivoice_version: str) -> dict:
"""A ``metadata.json`` payload shaped like marketplace ``_bundle_metadata`` so
an OLDER OmniVoice (which only reads metadata.json) can still import the ref
audio from a ``.ovsvoice`` bundle."""
return {
"bundle_version": 1,
"profile_name": profile.get("name") or "",
"ref_text": profile.get("ref_text") or "",
"instruct": profile.get("instruct") or "",
"language": profile.get("language") or "Auto",
"personality": profile.get("personality") or "",
"seed": profile.get("seed"),
"kind": profile.get("kind") or "clone",
"vd_states": profile.get("vd_states"),
"is_locked": bool(profile.get("is_locked")),
"omnivoice_version": omnivoice_version or "",
}
def _resolve_voice_file(filename: Optional[str]) -> Optional[str]:
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR, returning
an absolute path only if the file actually exists. None on missing/escape
mirrors profiles._voices_path (basename + realpath confinement, E1)."""
if not filename or os.path.basename(filename) != filename:
return None
from core.config import VOICES_DIR
root = os.path.realpath(VOICES_DIR)
path = os.path.realpath(os.path.join(root, filename))
if not path.startswith(root + os.sep):
return None
return path if os.path.isfile(path) else None
def _generate_preview(profile: dict, embed_fn) -> tuple[bytes, bool, float]:
"""Load the profile's source clip, downmix→mono, resample→24 kHz, trim ≤8 s,
watermark (forced), and return ``(wav_bytes, watermarked, duration_s)``.
Source precedence is locked-over-ref (profiles.py:230). Raises
:class:`NoPreviewSource` when neither clip is readable (A2-A5). All heavy
imports (torch/torchaudio/watermark) are lazy so the module stays model-free
at collection time (avoids the known local torch/Triton segfault)."""
import torch # noqa: F401 (torchaudio needs it loaded)
import torchaudio
from services.audio_io import _safe_torchaudio_save
from services.watermark import _check_available
candidates = [profile.get("locked_audio_path"), profile.get("ref_audio_path")]
wav = None
for name in candidates:
path = _resolve_voice_file(name)
if not path:
continue
try:
waveform, sr = torchaudio.load(path)
except Exception: # noqa: BLE001 — try the next candidate (A4)
continue
if waveform.numel() == 0: # empty/zero-length (A5)
continue
if waveform.shape[0] > 1: # downmix to mono (A7)
waveform = waveform.mean(dim=0, keepdim=True)
if sr != PREVIEW_SAMPLE_RATE: # resample (A8)
waveform = torchaudio.functional.resample(waveform, sr, PREVIEW_SAMPLE_RATE)
cap = int(PREVIEW_SAMPLE_RATE * PREVIEW_MAX_SECONDS)
waveform = waveform[:, :cap] # trim, shorter used whole (A6)
wav = waveform
break
if wav is None or wav.numel() == 0:
raise NoPreviewSource("no readable reference or locked audio for a preview")
# Forced watermark — bypasses the user pref but still no-ops without AudioSeal.
fn = embed_fn or _default_embed
wav = fn(wav, PREVIEW_SAMPLE_RATE)
watermarked = bool(_check_available()) # best-effort honesty (A11)
duration_s = round(wav.shape[-1] / PREVIEW_SAMPLE_RATE, 3)
buf = io.BytesIO()
_safe_torchaudio_save(buf, wav, PREVIEW_SAMPLE_RATE, format="wav", bits_per_sample=16)
return buf.getvalue(), watermarked, duration_s
def _default_embed(wav, sample_rate):
"""Default preview watermarker: services.watermark.embed_watermark(force=True)."""
from services.watermark import embed_watermark
return embed_watermark(wav, sample_rate, force=True)
def build_persona_bundle(
profile: dict,
*,
license_spdx: str = DEFAULT_LICENSE,
tags: Optional[list[str]] = None,
custom_license_text: Optional[str] = None,
include_reference: bool = True,
engine_id: str = "",
omnivoice_version: str = "",
embed_fn=None,
) -> bytes:
"""Assemble a ``.ovsvoice`` ZIP in memory and return its bytes.
Always writes a watermarked ``preview.wav`` + ``manifest.json`` +
(legacy-shaped) ``metadata.json``. Writes ``consent.json`` when there's
something to attest, the raw ``ref_audio``/``locked_audio`` members unless
``include_reference=False`` (privacy / preview-only, A12), and
``consent_audio`` when a recording exists. Raises :class:`NoPreviewSource`
(router 503) when no source clip is readable."""
preview_bytes, watermarked, duration_s = _generate_preview(profile, embed_fn)
members: dict = {"ref_audio": None, "locked_audio": None, "consent_audio": None}
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
if include_reference:
ref_path = _resolve_voice_file(profile.get("ref_audio_path"))
if ref_path:
name = f"ref_audio{os.path.splitext(ref_path)[1] or '.wav'}"
zf.write(ref_path, name)
members["ref_audio"] = name
locked_path = _resolve_voice_file(profile.get("locked_audio_path"))
if locked_path:
name = f"locked_audio{os.path.splitext(locked_path)[1] or '.wav'}"
zf.write(locked_path, name)
members["locked_audio"] = name
# Consent recording travels only when it exists and clears the floor.
consent_path = _resolve_voice_file(profile.get("consent_audio_path"))
has_recording = False
if consent_path and os.path.getsize(consent_path) >= _MIN_CONSENT_AUDIO_BYTES:
name = f"consent_audio{os.path.splitext(consent_path)[1] or '.wav'}"
zf.write(consent_path, name)
members["consent_audio"] = name
has_recording = True
zf.writestr("preview.wav", preview_bytes)
preview_block = {
"file": "preview.wav", "watermarked": watermarked,
"duration_s": duration_s, "sample_rate": PREVIEW_SAMPLE_RATE,
}
manifest = build_manifest(
profile, license_spdx=license_spdx, tags=tags or [],
engine_id=engine_id, custom_license_text=custom_license_text,
preview=preview_block, members=members,
omnivoice_version=omnivoice_version,
)
zf.writestr("manifest.json", json.dumps(manifest, ensure_ascii=False, indent=2))
zf.writestr("metadata.json",
json.dumps(_legacy_metadata(profile, omnivoice_version),
ensure_ascii=False, indent=2))
consent = build_consent_json(profile, has_recording=has_recording)
if consent is not None:
zf.writestr("consent.json", json.dumps(consent, ensure_ascii=False, indent=2))
return buf.getvalue()
@dataclass
class ParsedPersona:
manifest: dict # parsed manifest.json OR synthesized from metadata.json
consent: Optional[dict] # parsed consent.json, or None
is_legacy: bool # only metadata.json was found (B6/B23)
schema_version_ahead: bool # manifest.schema_version > OVSVOICE_SCHEMA_VERSION (B7)
license_spdx: str # normalized (B21)
preview_only: bool # only preview.wav, no ref/locked member (A12/B8)
members: dict # {prefix: member_name} for audio members present
watermarked_preview: bool # manifest.preview.watermarked (False for legacy)
_zip: zipfile.ZipFile # open handle; router extracts via extract_member()
def member_ext(self, prefix: str) -> str:
name = self.members.get(prefix)
return _safe_member_ext(name) if name else ".wav"
def extract_member(self, prefix: str, dest_path: str) -> bool:
"""Stream the audio member named by ``prefix`` to ``dest_path`` (a path
the CALLER derived from a server-generated id never from the member
name). Returns False when the member is absent. Last-wins on dup (B9)."""
name = self.members.get(prefix)
if not name:
return False
import shutil
with self._zip.open(name) as src, open(dest_path, "wb") as dst:
shutil.copyfileobj(src, dst)
return True
def parse_persona_bundle(content: bytes) -> ParsedPersona:
"""Validate the ZIP and read manifest/consent WITHOUT touching the DB or
writing files. Raises :class:`BundleError` (400|413) for B1-B11. The caller
must use the returned ``ParsedPersona`` while the process holds ``content``
(the open ZIP reads from the in-memory bytes)."""
if len(content) > MAX_BUNDLE_BYTES:
raise BundleError(413, f"Bundle too large. Max is {MAX_BUNDLE_BYTES} bytes.")
try:
zf = zipfile.ZipFile(io.BytesIO(content))
except zipfile.BadZipFile:
raise BundleError(400, "not a valid ZIP bundle")
names = [n for n in zf.namelist() if not n.endswith("/")]
# Manifest selection: prefer manifest.json, fall back to legacy metadata.json.
manifest: dict = {}
is_legacy = False
if "manifest.json" in names:
try:
manifest = json.loads(zf.read("manifest.json"))
except (ValueError, UnicodeDecodeError):
raise BundleError(400, "manifest is not valid JSON")
if not isinstance(manifest, dict):
raise BundleError(400, "manifest is not valid JSON")
# A bundle whose format is neither ovsvoice nor absent → still read
# leniently (B6); we only branch on schema_version below.
elif "metadata.json" in names:
is_legacy = True
try:
legacy = json.loads(zf.read("metadata.json"))
except (ValueError, UnicodeDecodeError):
raise BundleError(400, "manifest is not valid JSON")
if not isinstance(legacy, dict):
raise BundleError(400, "manifest is not valid JSON")
manifest = {
"format": "omnivoice-legacy",
"schema_version": OVSVOICE_SCHEMA_VERSION,
"persona": {
"name": legacy.get("profile_name") or legacy.get("name") or "Imported Voice",
"kind": legacy.get("kind") or "clone",
"language": legacy.get("language") or "Auto",
"personality": legacy.get("personality") or "",
"instruct": legacy.get("instruct") or "",
"ref_text": legacy.get("ref_text") or "",
"seed": legacy.get("seed"),
"is_locked": bool(legacy.get("is_locked")),
"vd_states": legacy.get("vd_states"),
},
"license": {"spdx": DEFAULT_LICENSE, "custom_text": None},
"tags": [],
"preview": None,
"members": {},
}
else:
raise BundleError(400, "bundle is missing a manifest")
# Audio members by prefix (last-wins on duplicates, B9). The member NAME is
# retained only to read bytes + pick an extension — never to build a path.
members: dict = {}
for name in names:
for prefix in _AUDIO_MEMBER_PREFIXES:
if os.path.basename(name).startswith(prefix):
members[prefix] = name
has_audio = any(p in members for p in ("ref_audio", "locked_audio", "preview"))
if not has_audio:
raise BundleError(400, "bundle has no audio member")
consent = None
if "consent.json" in names:
try:
parsed = json.loads(zf.read("consent.json"))
if isinstance(parsed, dict):
consent = parsed
except (ValueError, UnicodeDecodeError):
consent = None # advisory only — a bad consent.json never 400s
schema_version = manifest.get("schema_version", OVSVOICE_SCHEMA_VERSION)
try:
ahead = int(schema_version) > OVSVOICE_SCHEMA_VERSION
except (TypeError, ValueError):
ahead = False
preview_block = manifest.get("preview") or {}
watermarked_preview = bool(preview_block.get("watermarked")) if isinstance(preview_block, dict) else False
license_spdx = normalize_spdx((manifest.get("license") or {}).get("spdx"))
preview_only = ("preview" in members
and "ref_audio" not in members and "locked_audio" not in members)
return ParsedPersona(
manifest=manifest, consent=consent, is_legacy=is_legacy,
schema_version_ahead=ahead, license_spdx=license_spdx,
preview_only=preview_only, members=members,
watermarked_preview=watermarked_preview, _zip=zf,
)
__all__ = [
"OVSVOICE_FORMAT", "OVSVOICE_SCHEMA_VERSION", "MAX_BUNDLE_BYTES",
"PREVIEW_MAX_SECONDS", "PREVIEW_SAMPLE_RATE", "DEFAULT_LICENSE",
"BundleError", "NoPreviewSource", "ParsedPersona",
"normalize_spdx", "build_manifest", "build_consent_json",
"build_persona_bundle", "parse_persona_bundle",
]

Some files were not shown because too many files have changed in this diff Show More