Compare commits

..
70 Commits
Author SHA1 Message Date
55c40be307 release: freeze v0.3.16 — version bump, lockfiles, changelog (#1057)
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 00:56:18 +05:30
7f59d5f8fe fix(models): self-heal HF-cache snapshots with broken file links, retry load once (#1056)
A first-run breaker: all blobs download fine, but the snapshots/<rev>/
entries are dangling symlinks (0 KB) — os.path.isfile() is False on a
dangling link, so transformers reports the weights missing even though
the bytes are on disk, and the existing resume repair can't fix it.

New services/hf_cache_repair.py deletes exactly the broken snapshot
entries (dangling symlinks + zero-byte weight/config stand-ins; never
blobs, never resolving entries) and restores them via snapshot_download,
verifying afterwards — if the restore recreates broken links (hub's
memoized symlink probe passing while real links come out broken), it
forces hub into copy-mode and repairs once more with real files.
model_manager retries the load exactly once per repo per process
(rung 0 of the cache-recovery ladder); dead-end errors now name the
exact models--<org>--<name> folder to delete. failure.py classifies the
class as MODEL_CACHE_CORRUPT so the user-facing error and auto bug
report explain the automatic repair.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 00:35:10 +05:30
d3ec4ed371 fix(tts): voxcpm2 — version floor, reference-clip prep, trailing-silence guard (#1055)
Three hardenings of the voxcpm2 engine path, all backward-compatible and
platform-identical:

- Version floor: every install hint now says pip install "voxcpm>=2.0.3"
  (2.0.3 fixed an Apple-Silicon/MPS audio-quality bug). Floor only — an
  already-installed older version stays available and working; it just
  surfaces an actionable upgrade hint in the is_available reason and a
  load-time warning.
- Reference-clip prep: the voxcpm package no longer trims reference audio
  itself, so raw user clips reached the model unconditioned. The clone path
  now trims leading/trailing near-silence (-50 dBFS floor, 50 ms edge pad)
  and caps the reference at 30 s. Fail-open (any prep problem falls back to
  the raw clip) and a strict no-op for short clean clips.
- Trailing-silence guard: generated output is trimmed to the last voiced
  sample + ~0.3 s natural tail via the new audio_dsp.trim_trailing_silence.
  Silence-trim only, no content analysis; a no-op on outputs without a
  silent tail and on all-silent (dead) renders.

22 new fake-module tests in tests/test_voxcpm2_guardrails.py; existing
engine/hint tests strengthened to guard the floor.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 00:21:54 +05:30
a99f1fdff9 feat(tts): normalization covers the OpenAI-compat API, streaming, and batch paths (#1054)
The engine-agnostic text-normalization pre-pass now runs at the three
remaining text→engine choke points, applied exactly once per request:
/v1/audio/speech (req.language), /ws/tts (whole text, before the sentence
chunker fans it out), and the batch queue's per-segment _gen (target
language) — matching the /generate, dub, and audiobook wiring. Route-level
tests pin exactly-once (spy) + toggle-off-raw for each path.

Also fixes a pre-existing /ws/tts bug the new test exposed: any request
omitting emo_alpha hit a KeyError and got an error frame instead of audio.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 00:05:04 +05:30
236c727cd4 docs(changelog): unreleased entries for #1048-#1052 (#1053)
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 19:06:44 +05:30
efc99be337 feat(studio): generation takes — star, replay, and restore past takes; capped history retention (#1052)
Every generate already recorded a generation_history row; now that history is
usable: a takes rail in the workspace history lists recent takes with star/
unstar, replay, and one-click restore as the active output. Alembic migration
0009 adds the starred column (the startup schema self-heal covers pre-
migration DBs), a retention cap (setting, default 200) prunes the oldest
UNstarred rows — starred takes are never pruned — and history WAVs are only
deleted when no other row references them.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 18:54:14 +05:30
5ce9d0e51d feat(dub): predict segment fit before synthesis — tight/impossible badges + opt-in shorter rewrites (#1051)
New pure planning layer (services/duration_planner.py) runs after translation,
before TTS: estimates each translated line's natural speech duration (self-
calibrating from the job's already-synthesized segments, static per-language
rates as cold-start fallback) and classifies it fits/tight/impossible against
slot + capped gap borrow, with thresholds derived from fit_planner's own caps
so "impossible" means "would be trimmed". Verdicts ride the /dub/translate
response and badge the segment table; an opt-in (default OFF) LLM pass attaches
one-click shorter-rewrite suggestions for impossible lines. Never blocks
generation — informs before GPU time is burned.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 18:40:16 +05:30
7dbb95fa15 feat(dub): LLM translations keep terms consistent and sound spoken — auto-glossary brief + reflect pass (#1050)
One up-front LLM pass over the full transcript extracts a theme summary +
terminology map, merges it under the user's manual glossary (user entries
always win), caches it on the dub job per target language (job_data blob, no
schema change), and injects the brief into every per-segment prompt. A new
reflect pass then critiques each segment's direct translation for wordiness /
stiff register and rewrites it as natural spoken dialogue — any failure or
divergence silently keeps the direct translation. Both stages have Dub-tab
toggles (default ON for the LLM engine, persisted; MT engines unaffected),
with i18n strings across all 21 locales and docs updated.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 18:26:07 +05:30
60b29b4006 feat(tts): numbers, times, and abbreviations are spoken correctly in every engine (#1049)
New conservative, idempotent pre-TTS normalization pass
(services/text_normalization.py): strips zero-width/control junk, caps
pathological repeat runs, expands digits/times/ordinals/currency via
num2words (29 locales) and per-language abbreviation maps (EN/DE/ES/FR).
Wired once at each text-to-engine choke point — /generate, dub segments
(+ preview), and longform chapters — BEFORE the pronunciation dictionary
so user respellings stay the final say. Pref-gated
(text_normalization_enabled, default ON) with OMNIVOICE_TEXT_NORMALIZATION
env override; num2words promoted to a direct dependency.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 18:13:08 +05:30
04d6d0cb7a feat(longform): segment-level render cache — edit one sentence, re-render one segment (#1048)
Adds a content-addressed segment cache (segment_cache_key + SegmentCache,
cache_dir/segments/) under the existing chapter cache: a changed chapter now
reuses every untouched span's WAV and synthesizes only the edited/missing
ones, and an interrupted chapter render resumes from the segments that already
finished (each persists the moment it renders). The chapter key derivation is
unchanged so on-disk caches from released versions keep hitting, a fully-
unchanged chapter never touches segment files, and prune_cache_dir now walks
both layers so one byte cap bounds the whole cache. Chapter SSE events gain
additive segments/cached_segments counts.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 17:40:15 +05:30
2695ef97ae docs(research): adjacent-projects notes — voicebox, RTVC, VoxCPM, ebook2audiobook, VideoLingo (#1047)
* docs(research): adjacent-projects read — RTVC, VoxCPM upstream, ebook2audiobook, VideoLingo

Owner-requested comparative research tied to the current maturity map:
voxcpm2 upstream sync items (>=2.0.3 MPS fix, ref-trim removal in 2.0.1,
trailing-audio guard), audiobook per-sentence cache playbook, dub
translation reflect-loop + glossary, RTVC migration positioning.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(research): add voicebox (jamiepine) — the direct competitor read

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:56:53 +05:30
800207ddb5 fix(watermark): bound AudioSeal memory — long audio embeds/detects in 30s chunks (#1045) (#1046)
A multi-minute generation pushed the whole waveform through the AudioSeal
generator in one call; its activation memory grows linearly with length, and
a reporter's 16 GB Windows box failed a single ~2.2 GB CPU allocation mid-
generate (DefaultCPUAllocator: not enough memory). Embedding and detection
now slice audio into ~30 s chunks (sub-second tails fold into the previous
chunk), so peak memory is flat regardless of audio length. Detection keeps
the best-confidence chunk, which also stops whole-file averaging from
diluting spliced audio.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 15:26:40 +05:30
2d073cfaec fix(clone): ⊕ Insert popover opens below the script input instead of climbing out of the viewport (#1043)
The popover was hard-anchored bottom-[60px] — always growing upward
from the textarea. ScriptPanel's only mount (CloneDesignTab) puts that
input at the very top of the panel, so the tag list (max-h 280px,
including the CMU phoneme chips visible in the owner's screenshot)
extended past the viewport top, unreachable and unscrollable. Anchored
top-[calc(100%+6px)] instead: below the input, where the panel's
topmost placement guarantees room in its one mount.

Regression test locks the placement (top-anchored, bottom-[60px]
banned).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 07:11:14 +05:30
b1e83658f0 feat(ui): global audio mini-player — waveform, seek, and time for every playback that had none (#1042)
playBlobAudio-path audio (generate auto-play, profile/segment previews,
story lines, gallery voices, Projects renders) played through a bare
Audio()/AudioContext with no on-screen player; the #1032 stop pill was a
stop-only band-aid with a fixed-overlay overlap quirk at 1440x900.

- playback.js: claimTrackedPlayback extends the single-playback manager
  with label + seek/pause/resume transport and a timeupdate-driven track
  snapshot (currentTime/duration/paused/peaks); claimPlayback stays as
  the thin wrapper, single-playback invariant unchanged.
- media.js: every playBlobAudio path registers tracked - element paths
  get real seek/timeupdate, the Tauri Web Audio path gets offset-based
  seek + suspend/resume pause, and peaks are computed once from the
  blob/decoded buffer already in hand (never refetched). onDone(reason)
  lets callers chain (stories) or reset card state (gallery).
- GlobalAudioPlayer.jsx: persistent bottom bar (only for source
  'output' — exact pill exclusion semantics) with peaks canvas,
  click/drag/keyboard seek, play/pause, elapsed/total, label, stop.
- Layout: the bar is a real grid row (row 3) above the LogsFooter,
  mirroring the footer's in-flow fix — content physically ends at its
  top edge, so the pill's overlay-overlap class cannot recur; fixed
  overlays anchored above the footer also clear --audio-dock-height.
  Verified headless (Chromium 1440x900 + 1000x700, isolated vite, all
  :3900 traffic intercepted): bar meets footer edge-to-edge, clears the
  nav rail, seek/pause/stop drive the owner callbacks.
- Callers pass labels: "Generated audio" (useTTS/first-sound), profile
  name / segment text (useProfiles), story line (StoriesEditor), voice
  name (VoiceGallery/CommunityZone/ImportsZone), render title
  (Projects). VoiceGallery drops its bespoke copy of the Tauri playback
  detour; StoriesEditor line previews now actually play under WebKit
  (blob: media URLs never worked there) and are stoppable mid-chain.
- PlaybackStopPill.jsx + its test deleted; intent migrated into
  GlobalAudioPlayer.test.jsx (appears on output/hidden when idle/stop
  works/excluded sources) plus transport coverage; playback.test.js
  covers the tracked API; playBlobAudioTracked.test.js covers the
  media wiring incl. onDone reasons; logsFooterInFlow.test.js now
  guards both bars' grid rows and the overlay anchor calc.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 06:07:44 +05:30
29b6f30f2b release: freeze v0.3.15 — version bump, lockfiles, changelog (#1041)
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 04:59:59 +05:30
72e979f1d9 fix(tts): model-load time stops eating the generate timeout budget (#1039)
* fix(tts): model-load time stops eating the generate timeout budget (#1033, #1037)

The generate guard (OMNIVOICE_GENERATE_TIMEOUT_S, 300s) wrapped the
adapter's lazy _ensure_loaded() — weight download included — together
with the synthesis. A cold first request burned the whole window on
the download and died with the VRAM-guidance 503; #1014's T4
verification measured it (0% GPU util for the full 300s), and #1033 +
#1037 match the signature.

New public TTSBackend.ensure_ready() (dispatches to the adapter's
_ensure_loaded when present) runs FIRST under the model-load budget
(OMNIVOICE_MODEL_LOAD_TIMEOUT, 1200s) in both /generate's adapter path
and /v1/audio/speech — the same load/generate split get_model()
already gave the native engine. Warm engines no-op. A load exceeding
its own budget 503s with load-specific text pointing at Settings →
Models, never the misleading 'too heavy for compute' guidance.

Tests: end-to-end class test (load slower than a tiny generate budget
but inside the load budget → succeeds; fail-before verified), the
stalled-load error path, and the base-hook dispatch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* changelog entry for the load-budget split (#1033, #1037)

* catch the builtin TimeoutError base — reload-proof class identity (CI-only miss)

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 04:35:17 +05:30
6c6207a132 docs: link the community Colab notebook (#1038) (#1040)
@shakib30 built and tested a working Colab notebook for the project
and offered it upstream. Linking it from the README (community-
maintained, credited) makes the no-local-GPU path discoverable without
taking on notebook maintenance in-repo.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 04:19:54 +05:30
0effe485f4 fix(studio): restore Clear History, stoppable auto-play preview, + cached ref transcripts (#1032) (#1036)
Three-part fix for the v0.3.5-comparison report:

1. Perf: since v0.3.6 (#308), a clone reference without a stored
   transcript triggered a FULL ASR model load + transcribe on every
   /generate — get_active_asr_backend() builds a fresh whisper backend
   per call. Measured live: 92.7s wall vs 14.9s of actual TTS. Now the
   first auto-transcript is persisted onto the (unlocked, clone-kind)
   profile row, and transcribe_reference caches results by audio
   content hash (bounded LRU, no model/VRAM held), so the cost is paid
   once per clip, not per request. User-typed transcripts are never
   overwritten; locked/design profiles are excluded from the persist.

2. Clear History: the workspace UX overhaul (#374) moved history into
   the right-side WorkspaceHistory panels and dropped the old Sidebar's
   clear-all control (the Sidebar is now hidden in every mode). Both
   the Voice and Dub panels get a scoped Clear History button wired to
   the existing DELETE /history and /dub/history endpoints, with the
   same confirm dialog the Sidebar used.

3. Auto-play: the finished-render playback (playBlobAudio) has no
   on-screen player and the only stop lived in the Voice ActionBar's
   CTA morph — unstoppable from the Dub workspace, profile pages, or
   after navigating away. A global PlaybackStopPill now appears for any
   'output' playback on every page. The existing Settings → Appearance
   "Auto-play preview" pref (#667) now also gates the generate path,
   as its label always promised (default ON — no behavior change).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 18:43:33 +05:30
32407bc781 fix(api): /v1/audio/speech honors num_step + guidance_scale instead of silently dropping them (#1014) (#1035)
A contributor's measured Tesla T4 verification (PR #1014) caught that
POST /v1/audio/speech accepted num_step/guidance_scale in the JSON
body with a 200 OK and discarded both (pydantic's default
extra=ignore) — API callers could never reach the model's documented
quality preset (num_step=32) through the OpenAI-compatible surface,
while the native /generate exposes both as form fields.

Both are now declared as validated optional extensions (num_step 1-128,
guidance_scale 0-20) and passed through to the engine's generate()
kwargs — omitted means absent (engines that don't accept the kwargs
never see a stray None), exactly like the existing duration/seed
extensions.

Tests: passthrough reaches the engine kwargs; omitted stays absent.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 17:49:25 +05:30
df870e9ed3 docs(agents): add verified Tesla T4 (16GB) inference notes (#1014)
* docs: add AGENTS.md with verified Tesla T4 (16GB) inference notes

Documents two things found while verifying inference on a real T4:
1. Cold-cache first /v1/audio/speech call can hit the 300s
   OMNIVOICE_GENERATE_TIMEOUT_S because the checkpoint download happens
   inside that budget — workaround via existing POST /models/install or
   raising the timeout, no code change needed.
2. The OpenAI-compatible endpoint silently ignores num_step/guidance_scale
   (schema doesn't declare them) — use native /generate for those.

Also documents the T4 acceleration checklist (dtype/attention/int8/CUDA
graphs) and measured VRAM (peak 2.05GB). No code changes.

* fix(docs): make /models/install workaround command actually executable

Addresses Greptile review: the instruction omitted the required
repo_id body field (InstallModelRequest rejects an empty body).

* fix(docs): correct port in /models/install example (3900, not 8000)

The app serves on port 3900 (confirmed: /health returns 200 there,
connection refused on 8000). Verified the exact corrected curl command
returns 200 {"status":"install_started",...}.

* move T4 notes to docs/hardware-notes-tesla-t4.md — AGENTS.md is the auto-loaded agent-instructions filename

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 17:30:05 +05:30
stronghamjjiandstronghamjji 323892b36c fix(clone): bound the ref-text re-transcribe like every other ASR dispatch (#730) (#1031)
The re-transcribe added for the (ref_audio, ref_text) mismatch fix is
dispatched with a bare run_in_executor(_gpu_pool, ...), and refine_ref_text
calls asr_backend.transcribe() directly. Its try/except catches a raised
error but not a *hang* — a wedged whisperx/CTranslate2 transcribe (#730)
holds the GPU-pool worker forever. On a <=10 GB card the pool is 1 worker,
so that starves every later GPU job into the misleading "can't reach the
local backend", and there's no ping on the await so the EventSource drops.

Route both refine dispatches (per-speaker and per-segment) through the same
run_transcribe_guarded the rest of dub_core.py already uses (the chunk loop
and the whole-file "Dub" transcribe). On timeout it resets the pool and
raises ASRTimeoutError; keep the original clones, matching refine_ref_text's
own "failure is a strict no-op" fallback.

Adds a repro test: refine_ref_texts dispatched raw is unbounded on a hang;
through the guard it times out and falls back to the original ref_text.

Co-authored-by: stronghamjji <289942360+stronghamjji@users.noreply.github.com>
2026-07-09 17:10:23 +05:30
36ee06c7cc feat(skills): installable Agent Skills — npx skills add debpalash/omnivoice-studio (#1034)
Two skills in the standard skills/<name>/SKILL.md layout (vercel-labs/
skills CLI; listed on skills.sh via install telemetry):

- omnivoice — teaches any agent (Claude Code, Cursor, Codex, …) to
  speak and transcribe through the user's LOCAL install via the
  OpenAI-compatible API at localhost:3900: health preflight, TTS with
  cloned-voice-profile discovery via /v1/audio/voices, STT with
  srt/vtt subtitle formats, and the local-first rule (never silently
  fall back to a cloud API).
- oss-maintainer — the maintainer methodology this repo is actually
  run with, distilled from real sessions: absorbed-or-declined queue
  discipline, check-the-PR-queue-before-implementing, root-cause →
  fix-the-class → regression-test, structural merge gates with
  flaky-vs-real judgment, the release protocol, and
  thank-contributors-specifically.

Every endpoint/flag in the omnivoice skill verified against
backend/api/routers/openai_compat.py and the README's API section.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 17:06:14 +05:30
9bfd8f9a13 fix(update): app updates stop uninstalling user-added engines — drift sync goes --inexact (#1029) (#1030)
Every app update whose uv.lock changed ran `uv sync --frozen` to
reconcile the venv (#307 drift path) — and uv sync's exact mode
UNINSTALLS every package not in the lockfile. That silently deleted
user-pip-installed optional engines (voxcpm, kittentts — packages the
app's own Settings → Engines hints tell users to install into this
venv) on every single update. Reported as "VoxCPM2 is automatically
uninstalled after updating Studio."

Fix: the routine drift sync now carries --inexact — locked deps are
still installed/upgraded exactly per the lockfile, but extras the user
added on purpose are left alone. Deliberate asymmetry: the venv-REPAIR
sync stays exact, because repair runs when the venv is broken and a
user-installed extra is a plausible cause — healing must restore the
known-good locked state. First-run syncs are untouched (a fresh venv
has no extras; exact == inexact there).

Both sync arg sets are now named constants with contract tests pinning
the asymmetry, so neither side can silently regress.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 10:06:23 +05:30
cb6c72b409 docs(faq): honest ElevenLabs comparison — where each wins, and why dub quality varies (community question) (#1028)
Asked directly on Discord ('how it compares to something like 11 labs
in quality?'). The old answer ('yes, comparable for most use cases')
oversold — the honest version names where ElevenLabs still wins
(out-of-the-box English polish/consistency) and where OmniVoice is
genuinely competitive (cloning from clean references, 646 languages,
structural advantages), plus the dubbing-specific truth another
same-day report surfaced: a dub is a chain, and incoherent output
usually traces to transcription quality on the user's source audio —
with the check-the-original-text-first debugging step that actually
helps.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 09:13:11 +05:30
a721c1fdcf release: freeze v0.3.14 — version bump, lockfiles, changelog (#1027)
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 09:12:24 +05:30
80f10289fe feat(asr): ASR engines get the same Settings picker TTS has (env var still wins) (#1026)
Settings → Engines now stacks one pinned Engine Compatibility Matrix per
family (TTS, ASR, LLM) instead of a single TTS-titled table with the other
families tucked behind a low-discoverability tab. The backend select/prefs
path (family="asr" → prefs.asr_backend, env > prefs > auto-detect) already
worked but was unexercised and undocumented — it's now locked by API and
resolution-order tests, and README + the openai-compat-asr doc stop
promising a picker that didn't exist / denying one that now does.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 04:43:16 +05:30
a0ad314736 docs: CLAUDE.md refresh — de-rot versions, compress shipped stack research, replace the dead GSD gate (#1025)
Three classes of staleness that actively misled agent sessions:

- The Project section hardcoded "latest stable v0.3.5 / main at v0.3.6"
  — eight releases behind. Now points at the sources of truth
  (frontend/package.json, the Releases page) and documents the current
  AUTO_VERSION_BUMP-off holding behavior instead of a version literal
  that rots every release.
- ~165 lines of May-2026 stack research for five capabilities that have
  ALL since shipped (HF-token panel, prefilled-URL bug reporting, uv
  mirror fallback, Supertonic-3, in-repo docs). Compressed to the
  durable don'ts it established (no telemetry endpoints, no app-side
  GitHub tokens, no setx, no MkDocs, no hf_transfer) plus a pointer to
  prefer what's already pinned.
- The GSD Workflow Enforcement gate referenced /gsd-quick//gsd-debug/
  /gsd-execute-phase skills that exist nowhere in this environment; the
  owner explicitly chose direct edits over restoring them (2026-07-08).
  It cost a real mid-task detour when a subagent correctly refused to
  work under an unsatisfiable rule. Replaced with the owner decision
  and the working conventions that actually bind (merge gating,
  check-the-PR-queue-first).

244 → 83 lines. GSD section markers preserved so the generating tool
can still find its blocks.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 04:25:14 +05:30
1808a373a1 fix(linux): AppRun workaround detection reads the BUNDLED WebKitGTK version, not the host's (#961 follow-up) (#1024)
The launcher decided whether to export WEBKIT_DISABLE_COMPOSITING_MODE
by asking the host's pkg-config — but LD_LIBRARY_PATH makes the
BUNDLED libwebkit2gtk the one that actually runs, so on any machine
where the two diverge the detection read the wrong number. This was
the second bug identified during #961's investigation (the reporter
built from source, so their dev packages answered pkg-config with a
healthy version while the shipped bundle ran an older lib) and was
explicitly deferred in #1007 as not-safely-fixable at runtime.

The fix makes it knowable by construction instead: inject-apprun.sh
runs at bundle time ON the build host whose libwebkit2gtk gets
bundled, so it stamps that version into .bundled-webkitgtk-version
inside the AppDir. AppRun reads the stamp first and only falls back to
host pkg-config for bundles predating it. Empty/unreadable stamp fails
safe (workaround on), same philosophy as the missing-pkg-config path.

Tests: 3 new cases in AppRun.test.sh — marker-beats-host in both
directions (broken-marker/healthy-host and the #961 inversion,
healthy-marker/broken-host) plus empty-marker fail-safe. Also wires
AppRun.test.sh into pytest (tests/test_apprun_launcher.py) — it was
previously run by NO CI job, so the launcher could regress silently.

Also documents Windows install-to-another-drive behavior in
docs/install/windows.md (#938): local drives work via the wizard's
directory picker, mapped network drives are a Windows Installer
limitation, and the data directory moves independently of the app.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 03:53:46 +05:30
67789fb31c release: freeze v0.3.13 — version bump, lockfiles, changelog (#1023)
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 02:52:25 +05:30
da4bef8e42 docs: correct troubleshooting §16 — mic bug was a missing entitlement, not an upstream limitation; changelog for #1016/#1020/#1021 (#1022)
troubleshooting.md §16 claimed the macOS microphone-permission bug was
an unresolved upstream Tauri/wry limitation with no available fix.
That was wrong: @MahdiHedhli read the wry/tauri sources more carefully
and found the real cause — Tauri's Hardened Runtime default blocks mic
hardware access without com.apple.security.device.audio-input in the
bundle's entitlements, which also explains why TCC never listed the
app. Their fix (#1016) is merged; §16 now documents the real mechanism,
credits the correction, and keeps the record-elsewhere workaround for
users on ≤0.3.12 builds.

Also brings CHANGELOG [Unreleased] current for the three merges that
lacked entries: #1016 (mic fix), #1020 (shutdown wait 3s→20s), #1021
(CI flaky-trio root cause + guard).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 02:27:00 +05:30
0076d0067e test(ci): root-cause and neutralize the flaky trio — a leaked torch fp16 default dtype (#1021)
test_effects_chain / test_generation_audio_guard / test_persona_bundle
failed intermittently on CI (never locally) with identical signatures
across three unrelated PRs today (#1002, #1019, #1016) — costing a
full CI cycle per occurrence and repeatedly muddying merge decisions.

Root cause, confirmed by local reproduction: a leaked
torch.set_default_dtype(torch.float16) from some earlier test in the
CI-Linux ordering. The smoking gun was test_generation_audio_guard's
observed 0.0999755859375 — exactly float16(0.1), i.e.
torch.tensor([0.1, …]) built under a leaked fp16 default. Reproducing
with a simulated polluter locally produced the trio's exact failures:
Pedalboard refuses fp16 audio outright ("only supports 32-bit and
64-bit floating point") and silently returns unmodified audio for
every preset, so test_effects_chain's preset outputs compare
identical; and the fp16 tensor value breaks the sanitize approx-check.

Fix: an autouse conftest guard (same philosophy as the existing
LLM-state isolation guard, #878) that checks torch's default dtype
after every test, resets any leak to float32, and emits a UserWarning
naming the offending test's nodeid — so the actual CI-only polluter
identifies itself in the next CI log instead of being chased blind.
Regression test: a deliberate-leak pair proving reset-between-tests.

Fail-before/pass-after verified: with the guard stashed, a simulated
polluter + the trio reproduced 2/3 failures locally with the exact CI
signatures; with the guard active, 73/73 pass and the warning names
the polluter.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 02:19:07 +05:30
Mahdi Hedhli faf34348c8 fix(macos): add microphone/camera entitlements so TCC ever sees a request (#1016)
Root cause of #1013 (macOS: "Microphone access denied" but OmniVoice never
appears in Privacy & Security → Microphone to enable it):

Tauri's macOS bundle config defaults `hardenedRuntime` to true, and Hardened
Runtime blocks camera/microphone hardware access unless the matching
entitlement is present — regardless of Info.plist's NSMicrophoneUsageDescription
(that only supplies the *prompt text*, it isn't itself the grant) and
regardless of wry's own WKUIDelegate already granting the request at the
WebKit/JS layer (WryWebViewUIDelegate::request_media_capture_permission
unconditionally calls WKPermissionDecision::Grant — confirmed by reading
wry 0.55.1's source; that part was never the problem). With Hardened Runtime
on and zero entitlements, TCC never registers a request at all, which is
exactly the reported symptom: nothing to enable because the OS never saw a
legitimately-entitled process ask. This also explains the workaround in
#1013 and its comments (launching the raw binary from Terminal works, but
as Terminal's identity, not the app's) — Terminal is a properly entitled,
hardened-runtime process; the ad-hoc/unentitled app binary isn't.

Adds src-tauri/entitlements.plist (com.apple.security.device.audio-input,
plus com.apple.security.device.camera matching the forward-looking
NSCameraUsageDescription already in Info.plist) and wires it in via
tauri.conf.json's bundle.macOS.entitlements. Also corrects the stale
"nothing to do here" module comment in lib.rs that documented the
incomplete assumption this bug falsified.

Verified: built a debug .app (`tauri build --debug --bundles app`) and
diffed `codesign -dv --entitlements -` before/after this change — the
entitlements dictionary goes from absent to containing exactly the two
keys added here, alongside the runtime (Hardened Runtime) flag that was
already on. `cargo test` — 60 passed, 0 failed.
2026-07-09 02:15:28 +05:30
69ce697ee5 fix(backend): shutdown wait bound 3s→20s — post-merge review finding on #1002; absorb #1015's design-path test (#1020)
Greptile's review of the merged #1002 flagged a real residual gap: a
cold transformers import alone can exceed the 3s shutdown wait, and
cancelling the asyncio task doesn't stop the underlying OS thread —
so quitting during an unusually slow preload could still let shutdown
report "done" while that thread was alive, the exact #1000 class with
lower odds. Python cannot forcibly kill a running thread, so no finite
bound eliminates this outright; 20s shrinks the window from "any
preload" to "an unusually slow cold-import," the practical ceiling
before a long shutdown becomes its own complaint. New source-level
contract test pins the production bound at ≥15s so a future edit
can't quietly shrink it back without deliberate consideration.

Also absorbs the one test case from community PR #1015 (superseded by
the earlier-merged #1017, which duplicated it — my fault for not
checking the PR queue) that the merged version lacked: the
design/instruct path with no ref kwargs at all stays untouched by the
ref_text forwarding fix.

Co-authored-by: mergetest <test@local>
Co-authored-by: MahdiHedhli <noreply@github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 02:10:06 +05:30
93a3cb260a fix(ui): dub editor play button no longer sticks disabled; remove donate heart from nav rail (#1019)
WaveformTimeline's play button (disabled={!ready}) stayed permanently
disabled whenever the initial WaveSurfer decode failed and the
component fell back to loading pre-computed peaks. The waveform still
rendered fine from those peaks (nothing looked visibly broken), but
`ready` was only ever flipped by the 'ready' event re-firing on that
recovery load — which this component's own error-handling never
actually confirmed, just assumed. Each of the three fallback ws.load()
calls now explicitly confirms readiness once it settles (via .then()/
.catch(), or the existing synchronous-throw catch), instead of hoping
the event fires again.

Regression test: WaveformTimeline.readyFallback.test.js — a
source-level contract guard (driving a real decode-failure/recovery
sequence through jsdom is brittle, same house pattern as the sibling
WaveformTimeline.unlock.test.js) asserting every fallback load in the
error handler is followed by an explicit setReady(true).

Also removes the "Support OmniVoice" heart button from NavRail — the
donate page stays reachable from Settings' footer and the Contact page.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 02:09:54 +05:30
33379890ad fix(voice): free-text instruct now filtered before every generate/save call (#1010) (#1018)
buildDesignInstruct() already keeps Studio's design/clone generate
calls (useTTS.js) from round-tripping a 400 "Unsupported instruct
items" — it filters free-text against the active engine's supported
vocabulary client-side, with a toast instead of a failed request. Three
other call sites built their own instruct string directly and skipped
it entirely:

- handleSegmentPreview (Dub tab's per-segment preview) — instruct comes
  straight from segment/preset data; a preset's raw attrs merged with a
  free-text style field can carry phrases outside the vocabulary.
- handleSaveProfile / handleSaveHistoryAsProfile — both always create a
  kind='clone' profile; the backend only sanitizes instruct on save for
  kind='design' (see profiles.py's heal_design_instruct branch), so a
  clone profile could silently persist an unusable instruct and then
  400 every single time it's later used to generate.

All three now filter through the same buildDesignInstruct({}, instruct)
call useTTS.js's own clone path already uses, with the same
unsupported/duplicate-item toasts.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 01:47:06 +05:30
7f8a42ce51 fix(tts): mlx-audio CSM cloning drops ref_text, breaking every clone attempt (#1012, #1013) (#1017)
MLXAudioBackend.generate() reads voice/ref_audio/language/speed from
its kwargs but never extracted ref_text — it was built, then silently
never passed through to self._model.generate(). CSM (sesame.py) only
builds its cloning context when BOTH ref_audio AND ref_text are
present; with ref_text missing, the context list stays empty and
indexing into it raises "IndexError: list index out of range" deep
inside mlx-audio, instead of the clone ever being attempted. Voice
cloning on the CSM engine could never have worked as shipped.

generation.py already threads ref_text all the way through — even
auto-transcribing it via the GPU pool when the caller supplies
ref_audio without one (~line 780) — so the value was always available
in kwargs; it just never survived the crossing into this specific
backend.

Reported with the precise root cause and a working fix (community
member independently diagnosed and patched it locally, confirmed
working on MPS/0.3.12). Two-line fix: extract ref_text and pass it
through when both ref_audio and ref_text are present (guards against
passing an orphaned ref_text with no accompanying audio to engines
that don't expect it).

Tests: tests/test_engines.py — ref_text is passed through when paired
with ref_audio, omitted when ref_audio is absent.

Also documents the second bug from the same report (#1013): macOS
microphone permission never prompts, so OmniVoice never appears in
System Settings to grant access. Root-caused to an unresolved upstream
Tauri/WebKit limitation (WKWebView's requestMediaCapturePermissionFor
delegate — wry#1195, tauri#11951, fix wry#1196 still open/unmerged, no
released version to bump to) — not something fixable here without an
unverified native Rust/WKWebView hack this session has no way to test.
Documented in docs/install/troubleshooting.md with the confirmed
workaround (record elsewhere, upload the file).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 01:34:18 +05:30
4de3c824d0 fix(gallery): surface the real error instead of a generic guess (#1009)
Found while triaging Discord: a community member (lehoangan227) hit
"Could not create that voice — the engine may be loading" trying to
use an archetype from the Gallery. That message is hardcoded and
shown for ANY failure — the actual cause (a 500, a validation error,
anything) is caught and discarded.

api/client.js's ApiError already builds a clean, user-facing message
for every failure mode (HTTP status + backend detail, a network
failure, or a detected backend crash) — this codebase's own
established convention elsewhere is to interpolate that message via
`{{message}}` (see BatchQueue.jsx, Settings.jsx, ToolsPage.jsx). The
Gallery's own catch blocks just weren't following it.

Fixed the whole class across VoiceGallery.jsx (use/preview),
CommunityZone.jsx (add-to-voices, whose catch clause didn't even bind
the error), and ImportsZone.jsx (search/upload/save/delete/trim —
handleDelete previously failed completely silently, no message at
all). All now interpolate the real error message, matching the
gallery.download_failed key that already did this correctly a few
lines away.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 18:45:34 +05:30
11fbbd3f6d fix(dub): cross-language dub no longer speaks source-language reference text verbatim (#1004) (#1008)
extract_speaker_clones/extract_segment_refs pair each audio slice (cut
at ASR segment timestamps) with that segment's own `text` field, on
the assumption the two agree. They routinely don't — Whisper (and
friends) frequently drift on segment boundaries: a trailing word
audible in [start, end] but missing from text, or vice versa. When the
(ref_audio, ref_text) pair disagrees, zero-shot TTS prompt-priming
breaks down and the clone can emit the mismatched reference text
itself instead of the target-language line it was asked to speak —
reported with an exceptionally clear root-cause diagnosis and a
working A/B repro (matched pair: clean on the first try; mismatched
pair: wrong language 6/6 times).

Fix (as proposed in the report): re-transcribe each written reference
clip via the already-loaded, already-warm active ASR backend and use
that transcript as ref_text — this guarantees the pair matches by
construction, independent of whether the original segment text was
ever right. Falls back to the original text on any re-transcribe
failure or empty result — never a regression from current behavior,
only ever a fix.

New services.speaker_clone.refine_ref_text (single clip, unit-testable
against a duck-typed fake ASR backend) and refine_ref_texts (batch —
one executor round-trip per whole clones/seg_clones dict rather than
one per reference). Wired into dub_core.py's two clone-extraction call
sites, routed through _gpu_pool to match the established convention
for ASR-backend calls (the model is mid-lifecycle: TTS is offloaded,
ASR is loaded and exclusive, right where the existing per-chunk
transcribe calls already run on this same pool).

Tests: tests/test_speaker_clone_purity.py — 6 new cases covering the
mismatch-correction path, ASR-failure fallback, empty-transcript
fallback, no-backend no-op, and batch behavior (one failing entry
doesn't affect the others). Full backend suite: 2412 passed, 0 failed.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 18:18:15 +05:30
f546f04c8e fix(ci): bump Linux release runner to ubuntu-24.04, fixing stale bundled WebKitGTK (#961) (#1007)
The AppImage bundles whatever libwebkit2gtk-4.1-dev the build runner's
apt repos resolve at build time (see the "Linux system deps" step) —
AppRun's LD_LIBRARY_PATH then makes that bundled copy take priority
over the host's system WebKitGTK at runtime. ubuntu-22.04's version
was stale relative to what current distros (Ubuntu 24.04+, Fedora 44)
ship, which is why a from-source build (linking straight against the
host's healthy system library) worked fine on the exact machine where
the shipped AppImage white-screened — the released binary was running
an older, buggier WebKitGTK under the hood regardless of the host.

Bumped the Linux release matrix entry to ubuntu-24.04, and ci.yml's
Tauri shell-check job to match (its own comment already says "Mirror
release.yml" — now it actually does, so a green PR check accurately
predicts the release build will also succeed).

Raises the AppImage's glibc floor from 2.35 to 2.39 (Ubuntu 24.04+) —
README's system-requirements table corrected from the now-false
"Ubuntu 20.04+" claim. No reports of anyone on a pre-2022 distro.

This does not fix the AppRun launcher's separate, related bug (its
WebKitGTK-version auto-detection reads the *system's* pkg-config
version, not the version actually bundled and running) — that would
need a reliable way to read the bundled .so's version from within the
AppImage, which isn't straightforward (WebKitGTK's soname doesn't map
1:1 to its release version) and isn't verifiable without a real Linux
build environment to test against. Left as a known, separate gap.

Cannot be verified from here on a real Ubuntu 26.04 machine — shipped
on the strength of the root-cause diagnosis, pending the reporter's
confirmation.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 17:21:50 +05:30
270b3b1c4c fix(backend): quitting mid-preload no longer reports a clean shutdown while a GPU-pool thread is still importing (#1000) (#1002)
A user-pasted backend log revealed the real cause behind a class of
'can't reach backend' reports: three rapid restart cycles, each ending
with 'Shutdown: done.' immediately followed by a 'Model loading failed:
Could not import module AutoFeatureExtractor' error. That error text is
transformers' own generic lazy-import wrapper (import_utils.py's
_LazyModule.__getattr__), not a real dependency problem — pyproject.toml
already pins transformers/torch/torchaudio/soundfile/librosa as core,
non-optional deps, and the same venv loaded the model successfully 90
seconds later in the same log.

Root cause: preload_task and capture_preload_task were created at
startup but never referenced in the lifespan shutdown block — idle_task
and worker_task got cancelled-and-awaited, the preload tasks were simply
abandoned. Cancelling an asyncio task awaiting run_in_executor() can't
stop the underlying OS thread once it's inside blocking import/load
work, so 'Shutdown: done.' logged while a GPU-pool thread was still
mid-Version: ImageMagick 7.1.2-25 Q16-HDRI aarch64 037e46295:20260604 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/license/
Features: Cipher DPC HDRI Modules
Delegates (built-in): bzlib freetype heic jng jpeg lcms ltdl lzma png tiff webp xml zlib zstd
Compiler: clang (21.0.0)
Usage: import [options ...] [ file ]

Image Settings:
  -adjoin              join images into a single multi-image file
  -border              include window border in the output image
  -channel type        apply option to select image channels
  -colorspace type     alternate image colorspace
  -comment string      annotate image with comment
  -compress type       type of pixel compression when writing the image
  -define format:option
                       define one or more image format options
  -density geometry    horizontal and vertical density of the image
  -depth value         image depth
  -descend             obtain image by descending window hierarchy
  -display server      X server to contact
  -dispose method      layer disposal method
  -dither method       apply error diffusion to image
  -delay value         display the next image after pausing
  -encipher filename   convert plain pixels to cipher pixels
  -endian type         endianness (MSB or LSB) of the image
  -encoding type       text encoding type
  -filter type         use this filter when resizing an image
  -format "string"     output formatted image characteristics
  -frame               include window manager frame
  -gravity direction   which direction to gravitate towards
  -identify            identify the format and characteristics of the image
  -interlace type      None, Line, Plane, or Partition
  -interpolate method  pixel color interpolation method
  -label string        assign a label to an image
  -limit type value    Area, Disk, Map, or Memory resource limit
  -monitor             monitor progress
  -page geometry       size and location of an image canvas
  -pause seconds       seconds delay between snapshots
  -pointsize value     font point size
  -quality value       JPEG/MIFF/PNG compression level
  -quiet               suppress all warning messages
  -regard-warnings     pay attention to warning messages
  -repage geometry     size and location of an image canvas
  -respect-parentheses settings remain in effect until parenthesis boundary
  -sampling-factor geometry
                       horizontal and vertical sampling factor
  -scene value         image scene number
  -screen              select image from root window
  -seed value          seed a new sequence of pseudo-random numbers
  -set property value  set an image property
  -silent              operate silently, i.e. don't ring any bells
  -snaps value         number of screen snapshots
  -support factor      resize support: > 1.0 is blurry, < 1.0 is sharp
  -synchronize         synchronize image to storage device
  -taint               declare the image as modified
  -transparent-color color
                       transparent color
  -treedepth value     color tree depth
  -verbose             print detailed information about the image
  -virtual-pixel method
                       Constant, Edge, Mirror, or Tile
  -window id           select window with this id or name
                       root selects whole screen

Image Operators:
  -annotate geometry text
                       annotate the image with text
  -colors value        preferred number of colors in the image
  -crop geometry       preferred size and location of the cropped image
  -encipher filename   convert plain pixels to cipher pixels
  -extent geometry     set the image size
  -geometry geometry   preferred size or location of the image
  -help                print program options
  -monochrome          transform image to black and white
  -negate              replace every pixel with its complementary color
  -quantize colorspace reduce colors in this colorspace
  -resize geometry     resize the image
  -rotate degrees      apply Paeth rotation to the image
  -strip               strip image of all profiles and comments
  -thumbnail geometry  create a thumbnail of the image
  -transparent color   make this color transparent within the image
  -trim                trim image edges
  -type type           image type

Miscellaneous Options:
  -debug events        display copious debugging information
  -help                print program options
  -list type           print a list of supported option arguments
  -log format          format of debugging information
  -version             print version information

By default, 'file' is written in the MIFF image format.  To
specify a particular image format, precede the filename with an image
format name and a colon (i.e. ps:image) or specify the image type as
the filename suffix (i.e. image.ps).  Specify 'file' as '-' for
standard input or output., and interpreter finalization tore down module
state under it — producing exactly this misleading error.

Fix: extract the existing cancel+bounded-await pattern into
_cancel_and_await_tasks() and apply it to all four background tasks, not
just two. An early-stage load (still importing, not yet mid weight-
download) now gets a real chance to finish before shutdown proceeds; a
load genuinely deep in blocking work still times out at the same 3s
bound, and _reset_gpu_pool() abandons it same as before. Also: both
error handlers around this path logged only str(exc), discarding
__cause__ — added exc_info so a future incident (even one this fix
doesn't fully prevent) surfaces the real underlying error instead of the
misleading generic wrapper text.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 17:09:18 +05:30
5a7d9cc05c feat(asr): generic OpenAI-compatible transcription backend (#877) (#1003)
First slice of the community's two-track proposal for #877: a generic
OpenAI-compatible ASR backend that works TODAY, without waiting on
transformers to ship a direct Qwen3-ASR integration (tracked separately,
still blocked upstream). Points OmniVoice's transcription at any server
exposing POST /v1/audio/transcriptions — a self-hosted Qwen3-ASR/
FunASR/SenseVoice server, or OpenAI's own API.

- New OpenAICompatASRBackend (backend/services/asr_backend.py): a pure
  network client, no local model, no install. Prefers
  response_format=verbose_json for real per-segment timestamps,
  degrades to plain text (matching MoonshineASRBackend's shape) when a
  minimal server rejects that format. Never leaks a raw SDK/httpx
  exception to the caller (#977 convention) — wraps network/auth
  failures in a clean, actionable RuntimeError naming the server.
- Settings persist via the same encrypted-secret convention as
  services/llm_providers.py (settings_store.set_secret for the API key
  — Fernet-encrypted, never a .env row, never echoed back; get_text/
  set_text for base_url/model). New GET/PUT /api/settings/
  asr-openai-compat, loopback-gated like every other settings route.
- Frontend: a small settings panel (Settings → Models) mirroring
  HFMirrorPanel's exact structure. No ASR engine picker exists yet for
  ANY ASR backend (only TTS has one) — activating this engine still
  needs OMNIVOICE_ASR_BACKEND=openai-compat-asr; documented plainly
  rather than pretending otherwise.
- README's ASR Engines table (9 → 10 engines) and docs/features.yaml's
  drift-checker inventory updated; the '9 engines, all fully local'
  claim corrected since this one genuinely isn't.
- docs/engines/openai-compatible-asr.md: setup steps + an explicit
  privacy note (unlike every other ASR engine, audio leaves the
  machine to whatever server is configured).

Regression tests: tests/test_asr_openai_compat_877.py (12 tests) —
is_available() gating, verbose_json + plain-text response adaptation,
network-failure error hygiene, SDK retry disabling, and the settings
endpoints' persist/mask/clear-vs-unchanged semantics.

Fixed two real full-suite-only failures found during verification (not
brushed aside): the API route inventory snapshot needed regenerating
for the two new routes, and this file's own tests had a module-
staleness bug — a collection-time settings_store import went stale
relative to a test-time-fresh fixture when another test elsewhere in
the ~2400-test suite reimports the module — fixed by making
settings_store itself a fixture resolved at test-run time, same
lesson already applied to tests/test_mm2_lifecycle.py earlier this
session.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 08:33:03 +05:30
5871ec9d28 release: freeze v0.3.12 — version bump, lockfiles, changelog (#1001)
19-issue GitHub sweep: 16 PRs merged since v0.3.11, most fixing reports
filed same-day or in the days prior. Highlights: MLX-Audio's 6 other
curated models are finally selectable (was silently stuck on Kokoro
regardless of what was downloaded), first-run no longer dead-ends behind
restricted networks or corporate TLS-inspecting proxies, dubbing/batch
TTS honor your active engine selection, and a run of sharp community
diagnoses (ROCm wheel index, Windows dictation focus-steal, a genuine
frontend crash regression) got fixed largely because reporters did the
hard diagnostic work themselves.

Full backend suite: 2390 passed, 0 failed. Full frontend suite: 918
passed, 0 failed. Version lockstep (tests/test_app_version.py): 6/6
passed. Docker frozen-lockfile parity (bun install --frozen-lockfile):
clean, no drift.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 07:28:41 +05:30
58b023ff16 fix(diagnostics): TTS generate timeout message tells you to Flush/Unload (#939) (#999)
The guard itself (#851/#896) is working as designed — this is a message
gap, not a logic bug. The GPU-branch timeout guidance explained VRAM
contention but never mentioned the Flush/Unload action that actually
resolves it, even though: (a) that action already exists (POST
/system/flush-memory, wired to the header's Flush button), and (b) the
sibling ASR-timeout guard's message already recommends it verbatim
(asr_backend.py's _CUDA_VRAM_BUDGET_GB guidance). The maintainer ended up
manually explaining 'Settings → Models → Flush caches / Unload' in an
issue thread reply — information the error message should have carried
itself.

String-only change, no control-flow touched, mirrors the exact precedent
of #896 (a guidance-only change to this same function).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 05:18:00 +05:30
b61d17dd61 fix(diagnostics): filter Tauri's benign IPC-fallback warning from frontend log capture (#975) (#998)
On some Windows configurations, Tauri's custom-protocol IPC probe fails
once at startup and Tauri logs a console.warn before silently — and
successfully — falling back to postMessage + WebSocket. Fully functional,
happens at most once per launch, and not a bug in our code (confirmed:
this is Tauri's own internal fallback mechanism, structurally intentional
across its recent 2.11.x releases, not something being actively patched
upstream — so not bumping the framework speculatively for this).

It IS real noise though: as a captured console.warn it spuriously flips
the Settings > Logs footer's Frontend pill to "1 warning" on every
affected Windows launch. Filtered at the capture source (consoleBuffer.js)
rather than the display layer, so it never enters the ring buffer or a
copied diagnostic dump either — narrowly scoped to this one known message
prefix, not a general warning-suppression mechanism.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 05:17:12 +05:30
93aab6dadc docs(linux): mention yt-dlp as an optional prerequisite (#973) (#997)
The preflight system check already warns in-app when yt-dlp is missing
(Voice Gallery/Dub YouTube downloads fail without it), but the install
docs never mentioned it — a user has to hit the in-app warning first
instead of seeing it up front alongside the other optional prereqs.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 04:42:00 +05:30
7098be1c8b fix(dictation): Windows pill no longer steals foreground focus (#982) (#996)
Root-caused and verified against the actual code (community diagnosis in
#982 was accurate): showing the dictation pill — an always-on-top top-level
WebviewWindow — grants it Win32 foreground activation by default on
Windows, which macOS doesn't do for a shown window. Nothing marked the pill
non-activating, so on Windows the synthesized Ctrl+V from `simulate_paste`
landed back in the pill instead of the app being dictated into. Code review
also found a second, more direct culprit at the same call sites: the
global-shortcut Pressed handler's `win.set_focus()` was only skipped on
macOS (`#[cfg(not(target_os = "macos"))]`), so Windows explicitly focused
the pill on top of the implicit Win32 activation; the tray "dictate" menu
handler called `win.show(); win.set_focus();` unconditionally on every
platform, including Windows.

This is the Windows counterpart of #287 (macOS auto-paste — don't steal
focus): same class of bug, same "pill must stay unfocused so the paste
lands in the target app" intent (already stated in the `grant_webview_
media_permissions` module comment), different OS mechanism.

Fix, mirroring #287's platform-cfg pattern:
  1. WS_EX_NOACTIVATE applied to the pill's HWND once, right after creation
     (`mark_pill_noactivate`), via the `windows` crate pinned to the exact
     0.61.x tauri itself already resolves to — Cargo.lock confirms this
     unifies to the same `windows 0.61.3` already in the graph, so
     `WebviewWindow::hwnd()`'s HWND and our Win32 calls share one type and
     no new crate version was added.
  2. `ShowWindow(SW_SHOWNOACTIVATE)` (`show_pill_noactivate`) in place of
     `.show()` at the two dictation-trigger call sites (global shortcut +
     tray "Start Dictation"), since `.show()` alone still raced the style
     bit on some paths.
  3. The explicit `set_focus()` calls at those same two sites are now
     skipped on Windows too, the same way they already were on macOS.

macOS and Linux are untouched: the macOS cfg branch is unchanged, and the
Linux branch of the `set_focus()` guard still runs exactly as before.

The pill's auto-dismiss (`scheduleDismiss`/`dismiss` in CaptureWidget.jsx)
was checked and is a plain unconditional setTimeout chain — it is not
gated on any native focus-loss/paste-completion signal, so there's no
independent bug to fix there. The "never dismisses" symptom is a
consequence of the focus-steal, not a separate stall: once the pill wrongly
held foreground for the whole session, hiding it later left Windows'
foreground state inconsistent. With the pill never taking focus, the
target app stays foreground throughout and there's nothing to reconcile.

Win32 window-activation syscalls (`#[cfg(target_os = "windows")]`) can't
run under `cargo test`/`cargo build` on this non-Windows CI runner, so the
new `pill_noactivate_tests` module tests the pure flag math
(`with_noactivate_style`) instead — platform-agnostic, runs everywhere,
verified passing here. The actual HWND-touching code is logic-reviewed but
UNVERIFIED on real Windows; the reporter offered to test a patched build,
which is the recommended next step before this ships in a release.

`cargo build` and `cargo test --lib` both pass (60/60 tests, including the
3 new ones); the pre-existing `setup.rs` unreachable_code warning (#286) is
unrelated.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 04:41:25 +05:30
8520b84b68 fix(clone): voice-design panel no longer crashes on a partial vd_states shape (#983) (#995)
Crash: DesignMethodPanel's optLabel() called val.replace(...) on an
undefined category value — a regression from f678e33, which swapped a
safe plain template literal for an i18n lookup that assumes vdStates[key]
is always a string. Both occurrences (the label kicker and the chip
list) are now guarded, falling back to 'Auto' the same way the rest of
the component treats an unset category.

Root cause: vdStates could actually go partial in real usage. Selecting
a design profile (useProfiles.js) or restoring legacy localStorage state
(useAppData.js) applied the backend/stored vd_states object as-is, with
no check that all 6 CATEGORIES keys were present — so an older client,
hand-edited payload, or partial API write reproduced the crash on
selection. Both call sites now run the restored object through
mergeDescribedAttrs() (voiceInstruct.js), the existing completion helper
already used for the "describe your voice" path, which fills any
missing/unknown category with 'Auto'. useAppData.js also gained the
typeof === 'object' guard useProfiles.js already had.

Closes the class at the source: POST /profiles now completes vd_states
against CATEGORY_ORDER (core/describe_voice.py, the same list the
frontend's CATEGORIES mirrors) before persisting, so a design profile
can never be *saved* with an incomplete shape regardless of which
client wrote it — updated two existing tests whose fixtures asserted
the old (partial) persisted shape.

Regression tests: DesignMethodPanel render test with a partial vdStates
input, a mergeDescribedAttrs unit test for the exact partial shape from
the issue, and a backend test asserting POST /profiles fills all 6 keys.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 04:34:14 +05:30
549fa4009f feat(engines): expose MLX-Audio's curated model picker (#981) (#994)
mlx-audio multiplexes 7+ curated models (Kokoro, CSM, Qwen3-TTS, Dia,
Chatterbox, MeloTTS, OuteTTS) behind a single "mlx-audio" backend id, but
MLXAudioBackend resolved its active model ONLY from the
OMNIVOICE_MLX_AUDIO_MODEL env var — invisible to Settings and unreachable
without restarting the packaged app with that var set. A user who
downloaded e.g. Llama-OuteTTS via Settings → Models had no way anywhere
in the UI or API to actually load it; the backend silently kept using
Kokoro.

Fix:
- MLXAudioBackend.__init__ now resolves its model via
  prefs.resolve("mlx_audio_model_id", env=..., default=...), mirroring
  active_backend_id()'s env > prefs > default order exactly.
- get_active_tts_backend()'s switch-detection now also tracks the
  resolved mlx-audio model key, so a model-only change (same backend id)
  invalidates the cached instance and reconstructs it — no app restart
  needed to pick up a different curated model.
- POST /engines/select gained an optional model_id field; for
  family=tts/backend_id=mlx-audio it validates against
  MLXAudioBackend.CURATED_MODELS (or a raw HF repo id, matching the
  class's existing tolerance) and persists it via prefs.
- GET /engines now includes a curated_models roster + active_model_id on
  the mlx-audio entry only.
- Settings → Engines renders a small model dropdown on the mlx-audio row,
  pre-selected to the active model, wired through selectEngine's new
  optional modelId argument.

Regression coverage: prefs resolution + env override, cache invalidation
on model-only switch, /engines/select 400s on an unknown model id and
persists a valid one, curated_models present only on mlx-audio, and a
new EngineCompatibilityMatrix vitest suite for the dropdown.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 04:32:24 +05:30
dfe2bd1bc3 fix(install): classify SSL handshake failures + trust the OS cert store (#976) (#992)
Windows users behind a corporate/antivirus TLS-inspecting proxy got a raw
`[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE]` on every model install — the TCP
connection reaches the server fine, but the handshake fails because the
OS trusts the proxy's re-signed root CA and Python's bundled certifi CA
list doesn't. A genuinely different failure mode from #984 (that was
TCP-level unreachability to a blocked host, before any TLS negotiation).

- backend/core/failure.py: new SSL_HANDSHAKE_FAILURE classification
  (handshake/cert-verify-failed/sslv3_alert/sslcertverificationerror
  substring markers) with an actionable hint, added to
  _CONTEXT_FREE_HINT_CLASSES so append_hint() (already called by
  setup/download.py's install worker) surfaces it without further wiring.
- backend/main.py: truststore.inject_into_ssl() at module level, before
  any huggingface_hub/requests/httpx network I/O — patches ssl.SSLContext
  to verify against the OS trust store instead of only certifi's bundled
  CA list. Not platform-gated (correctness improvement everywhere);
  wrapped in try/except so it never blocks startup.
- pyproject.toml/uv.lock: truststore>=0.9 — pure Python, MIT, PyPA-
  maintained, zero transitive deps, same class of fix as socksio.

Verified: uv lock --check + uv sync --frozen clean (lockfile diff is
just the one new package); main.py imports cleanly; full backend suite
passes; no hiddenimports entry needed (main.py is PyInstaller's direct
entry script per backend.spec, so a top-level import traces normally —
unlike socksio's case, which was httpx's internal lazy import).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 04:29:21 +05:30
324d417d27 fix(engines): mlx-audio no longer crashes on unsupported languages, error messages never leak raw exception internals (#977) (#993)
Root cause: MLXAudioBackend.generate() blindly truncated the full
language display name to two characters (language[:2].lower()),
assuming an ISO code — 'Dutch' -> 'du', which crashed Kokoro's vendored
pipeline's internal assertion (assert lang_code in LANG_CODES, (lang_code,
LANG_CODES)) for any language whose first two letters didn't coincidentally
match one of Kokoro's single-letter codes. The raw AssertionError's
tuple-containing-a-dict args then leaked straight into the user-facing
500 message via two stacked f"...{e}" formatters in generation.py.

- resolve_kokoro_lang_code() resolves against the AUTHORITATIVE
  ALIASES/LANG_CODES table read from the installed mlx_audio package
  (never a hardcoded guess), and only applies when Kokoro is the actual
  active curated model — other curated models (CSM, Dia, Qwen3-TTS,
  OuteTTS, ...) either ignore the kwarg or expect a different format, so
  Kokoro's strict validation doesn't wrongly reject them. Unsupported
  languages now raise a clear ValueError naming what Kokoro supports,
  which generation.py already converts to a clean 400.
- _safe_exc_text() hardens both generic exception formatters in
  generation.py: if any element of an exception's .args is a container
  (dict/list/tuple/set), never interpolate str(e) raw — name the
  exception type and point at the log instead. Protects every current
  and future engine's generate() from leaking a raw container repr, not
  just this one Kokoro assertion.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 04:29:01 +05:30
637020b82b fix(engines): nemo-parakeet install hint stops recommending a shared-venv-breaking pip install (#974) (#991)
The Engines page told users to run `pip install nemo_toolkit[asr]` for
the NeMo Parakeet ASR engine. nemo_toolkit[asr]==2.7.3 hard-pins
transformers>=4.57,<4.58, which is unsatisfiable alongside OmniVoice's
own transformers>=5.3 requirement (needed by
omnivoice/models/omnivoice.py for HiggsAudioV2TokenizerModel). A user
who followed the hint ended up with a backend that wouldn't start
(ImportError: cannot import name 'HiggsAudioV2TokenizerModel').

_INSTALL_HINTS["nemo-parakeet"] in backend/services/asr_backend.py now
states plainly that installing into the shared venv will break the
backend, names the transformers conflict, and tells users to use a
separate/dedicated Python environment instead — without implying a
safe one-line fix or an isolated-venv env var exists (unlike
dots-tts/moss-tts-v15/confucius4-tts, nemo-parakeet has no isolated
venv option yet; that's a separate, larger follow-up).

Also adds one sentence to docs/install/troubleshooting.md's existing
"engine venv clash" section (#11) pointing at the same class of issue
on the ASR side, and a regression test asserting the hint never again
contains the literal bare `pip install nemo_toolkit[asr]` string.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 04:26:12 +05:30
dab6456581 docs(linux): stop advertising a .deb package that isn't published (#990)
README's Quickstart badges linked a 'Download Debian .deb' button
straight to the releases page — but .deb bundling was deliberately
dropped from release.yml (tauri-cli bug, 'Failed to create control
scripts') and no release has ever shipped one. A community member
investigating #961 confirmed this by checking the actual release
assets. Users clicking that badge got a broken promise, not a package.

Removed the badge; docs/install/linux.md's '## Install (.deb)' section
now honestly states it's unavailable pending a tauri-cli fix, points to
the AppImage as the supported path, and keeps the historical pre-v0.3
.deb upgrade note (ffprobe conflict) since that's still relevant to
existing installs.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 04:25:40 +05:30
015d29cafb test(voice-design): lock in RTL/non-Latin script handling for the instruct field (#980) (#989)
Issue #980 reported a raw 400 for Hebrew text ('שמואל') typed into the
Clone tab's Style field. Investigated: this is the same failure class
as #612 (Vietnamese free-text) and was already fixed when #612 landed
in commit 10b9d69 (first released v0.3.8) — buildDesignInstruct() drops
any unsupported free-text client-side before it ever reaches the
backend's validator, regardless of script. The reporter was on v0.3.7,
which predates that fix.

No behavior change needed — only a regression test, since the existing
Vietnamese test case covered Latin-script-with-diacritics but nothing
exercised a right-to-left / non-Latin script specifically.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 04:25:24 +05:30
1a03c59f82 fix(install): AMD ROCm torch reinstall targets rocm6.4, not rocm6.2 (#988)
Community-diagnosed (issue #972, Kaihui-AMD): pyproject.toml pins
torch==2.8.0, but the rocm6.2 wheel index only ever published up to
2.5.1 — the reinstall silently failed to resolve and fell back to the
default CUDA build, which runs on CPU on an AMD GPU. The failure was
correctly logged (bootstrap.rs's emit_log warning), just never actioned
because the index itself couldn't succeed. rocm6.4 carries a matching
torch==2.8.0 build.

Docs updated with the corrected index plus a repo.radeon.com find-links
path for users who want a driver-matched ROCm 7.2.x build the PyTorch
index doesn't carry (OMNIVOICE_TORCH_INDEX only accepts a PEP 503 index,
not find-links, so that's documented as a manual step).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 04:25:00 +05:30
8281b7c798 fix(engines): dub and batch TTS honor the active-engine selection (#987)
* fix(engines): dub and batch TTS honor the active-engine selection — with a real capability gate, not a silent OmniVoice fallback

Dub generation and batch TTS hardcoded services.model_manager.get_model()
(OmniVoice) regardless of the engine picked in Settings → Engines. A user
selecting VoxCPM2 (or any other engine) still got OmniVoice output with no
error — the silent fallback IS the bug class, not just the one report.

Root-caused and fixed for the whole class:
  - New `TTSBackend.supports_cloning` capability flag (default True) marks
    engines that can only offer fixed preset voices — kittentts,
    supertonic3, sherpa-onnx set it False. MLXAudioBackend exposes it as an
    instance property (Kokoro doesn't clone, CSM does) since the adapter
    multiplexes multiple models with different capabilities.
  - `cloning_capable_engine_ids()` and a shared `resolve_generation_backend()`
    helper in services/tts_backend.py centralize engine resolution
    (id → is_available() → routing gate → optional cloning gate), mirroring
    generation.py's /generate resolution instead of inventing a third
    parallel mechanism. Both routers now standardize on the existing
    get_active_tts_backend() cache (unload-on-switch already handled).
  - dub_generate.py's two TTS-generate call sites (main run + OOM retry) and
    the /dub/preview-segment route resolve once, up front, with
    require_cloning=True — dub's ref_audio is populated for essentially
    every real job, so an engine that can't clone fails the whole job with
    one actionable message instead of mis-cloning per segment.
  - batch.py resolves once per job, require_cloning only when voice_id is
    pinned — an unpinned batch job runs fine on any engine.
  - Applied the three pre-existing TODO(#312) comments: mastering now skips
    via `applies_own_mastering` for both pipelines, matching generation.py.

Regression tests cover the capability-id list, the fail-fast gate (proving
no OmniVoice fallback), the success path on a selected non-OmniVoice
engine, batch's pinned-vs-unpinned voice_id behavior, and the mastering
skip for both pipelines. Three existing dub tests that mocked get_model()
directly were updated to mock the new resolver instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(engines): exclude model-dependent adapters from cloning_capable_engine_ids()

getattr(cls, "supports_cloning", True) at the CLASS level returns a
property descriptor object (always truthy) when the flag is an instance
@property, not a plain attribute — MLXAudioBackend uses exactly this
pattern because its cloning capability depends on which of its 7+ curated
models is loaded (only CSM clones; Kokoro etc. don't). Without this fix,
the dub/batch capability-gate error message would always recommend
'switch to mlx-audio' even when the user's configured MLX model can't
clone, sending them in a circle back to the same error.

isinstance(value, bool) distinguishes a resolved boolean from a
descriptor object, so mlx-audio is excluded from the suggestion list
until its actual per-instance capability can be checked (already handled
correctly by resolve_generation_backend()'s per-call instance check).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): engine-aware dub/batch entry (#987)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 03:42:14 +05:30
efe57ee49d fix(ui): loaded-models panel says when a resident model is not the active engine (#985)
* fix(ui): loaded-models panel says when a resident model is not the active engine (#TBD)

A switched-away TTS model stays resident in VRAM until Unload/Flush or the
idle evictor, so users who picked another engine (e.g. voxcpm2) saw
"OmniVoice TTS - cuda:0 - 1937 MB" in the LOADED MODELS flyout and concluded
synthesis was still routing to OmniVoice. It wasn't - the panel just gave no
hint that resident != active.

/model/loaded entries for TTS-family models (in-process OmniVoice +
subprocess sidecars) now carry engine_id + is_active_engine, computed against
active_backend_id(); attribution failure degrades to the old shape
(is_active_engine: null) and non-TTS entries (ASR, diarization) are left
unannotated. The flyout renders a muted "not active - safe to unload" tag
(i18n: header.model_not_active, en + zh-CN) on inactive entries; Unload/Flush
behavior is unchanged. Regression tests cover both attribution states, the
ASR non-label, and the degradation path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): loaded-models active-engine hint entry (#985)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(mm2): string-target monkeypatch for active_backend_id — immune to sys.modules reimports

The two attribution tests patched the collection-time module alias; other
suites pop+reimport services.* modules mid-run, so in full-suite order the
patch landed on a stale module object while _active_tts_id late-imported the
fresh one (CI-only failure). String targets resolve at patch time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 02:41:47 +05:30
7f77f4d7bf fix(audio): remove hidden reverb from the mastering pre-stage — reverb is preset-declared only (#986)
* fix(audio): remove hidden reverb from the mastering pre-stage — reverb is preset-declared only (#TBD)

Field report (Discord): baked-in echo/reverb on some voices. apply_mastering()
hardcoded a Reverb that ran on every non-raw synthesis before the user's
preset chain — broadcast shipped reverb it never declared, podcast broke its
"no reverb" promise, cinematic/warm got doubled reverb.

The mastering pre-stage is now data-driven (MASTERING_CHAIN: highpass +
compressor, same params as before) and reverb-free; cinematic/warm keep their
user-chosen reverb. Regression tests pin the contract, incl. a burst-then-
silence echo-tail check and pedalboard-missing passthrough.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): hidden mastering reverb entry (#986)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 02:26:19 +05:30
087309259b fix(setup): first-run network check is mirror-aware and never hard-blocks (#984)
* fix(setup): first-run network check is mirror-aware and never hard-blocks

Field report (Discord, China): the Launchpad preflight probed hardcoded
huggingface.co:443 and any failure disabled Continue outright — users behind
the GFW were stuck on the very first screen, before Settings (and its
HF mirror quick-pick) was even reachable.

- The probe now targets the HF endpoint actually in effect (HF_ENDPOINT /
  hf_endpoint pref via configured_hf_mirror), with the real port.
- An unreachable endpoint is a WARNING, not a blocker: local-first — cached
  models work offline, and downloads surface their own actionable errors.
- When huggingface.co is blocked but hf-mirror.com answers, the fix text says
  exactly that, and the wizard shows an inline mirror quick-pick (presets +
  custom URL) that applies via PUT /hf-mirror — effective immediately for
  downloads — then re-checks.
- Docs updated (downloading-models, install troubleshooting); regression
  tests cover warn-not-fail, mirror-host probing, and the mirror suggestion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): open [Unreleased] with the preflight mirror fix (#984)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 02:11:18 +05:30
453db55f12 release: freeze v0.3.11 — version bump, lockfiles, changelog (#970)
package.json + three mirrors -> 0.3.11 in lockstep; Cargo.lock/uv.lock/
bun.lock regenerated; CHANGELOG [Unreleased] -> [0.3.11] — 2026-07-05
with the multi-language-release headline; nine entries since v0.3.10.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 22:28:43 +05:30
620321a9cc feat(diagnostics): backend crashes become self-documenting — exit code + stderr tail surfaced and attached to bug reports (#969)
* feat(diagnostics): backend crashes become self-documenting — exit code + stderr tail surfaced and attached to bug reports

When the backend PROCESS died (native CUDA abort, OOM kill, DLL crash) the
user saw only "Can't reach the local OmniVoice backend" and the evidence
died with the process — every #941-class report needed a logs-please
round-trip nobody answers. The v0.3.9 guard fixed HANGS; this fixes the
class of invisible DEATHS:

- Rust (crash.rs): every unexpected child exit — detected by the startup
  health poll and the post-Ready supervisor — writes a rotating (last 3)
  JSON crash marker next to the backend logs: ts, exit code/signal,
  backend version, uptime, ~40-line stderr tail. Intentional shutdowns
  never forensicate: app-quit raises the quitting flag first (now also on
  macOS Cmd+Q via ExitRequested), and retry/clean-retry kills set a
  BACKEND_KILL_INTENDED flag cleared when the fresh child is tracked.
- Tauri commands get_last_backend_crash / acknowledge_backend_crash;
  ack is a persisted watermark, never a delete — bug reports still get
  the evidence after the user viewed it.
- Crash-loop escalation: the supervisor budget goes 5-in-60s → 3-in-10min
  so slow crash loops stop respawning and land on the Failed screen with
  the last exit code + stderr tail.
- Frontend: apiFetch's transport-failure path swaps the vague message for
  "the backend crashed (exit code X) N s ago…" when an unacknowledged
  marker exists, and BackendCrashNotice (banner + details dialog,
  i18n'd, ack-on-view) surfaces it even with no request in flight.
- Bug-report prefill gains a "Last backend crash" section (exit code +
  home-path-scrubbed stderr tail via the existing scrubText), so the next
  report arrives WITH the evidence.

Tests: cargo --lib 57 pass (marker rotation write-4-keep-3, ack
semantics, store IO, ExitStatus decomposition, 3-in-10min policy);
vitest 909 pass incl. crash-notice branch, client crash-message branch,
bug-report enrichment; legacy node:test 41 pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add backend crash forensics under [Unreleased] (#969)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 21:36:47 +05:30
738d45f1c7 fix(ui): timeline box colors pre-blended in JS — visible on any WebView2 (#963) (#968)
* fix(ui): timeline box colors pre-blended in JS — visible on any WebView2, color-mix dependency removed (#963)

#951 moved the segment-box palette to `color-mix(in srgb, tint 45%,
var(--chrome-bg))` strings applied as inline styles. WebView2/Chromium
< 111 has no color-mix, so the CSSOM rejects the whole `background`
assignment — and since .seg-track__box declares no background of its
own, the boxes rendered fully transparent on pinned/enterprise WebView2
runtimes (the Windows installer never enforces a minimum runtime).

Fix the class, not the instance: no engine-dependent CSS may reach this
lane's inline styles. The 0.45·tint + 0.55·bg blend now happens in JS —
timeline.js keeps the tints as numeric [r,g,b], reads --chrome-bg off
the document root (fallback #0f1011), and emits literal `rgb(r, g, b)`
strings every engine parses. Pixel-identical to what color-mix painted.
Theme-awareness is preserved by re-blending when [data-theme] changes
on <html> (the seam App.jsx switches themes through), observed via
MutationObserver; SegmentTrack subscribes with useSyncExternalStore so
mounted boxes recolor live.

Guards updated: palette entries must match plain opaque rgb() (no
color-mix/var()/alpha), the default-theme blend is asserted against
independently computed literals, theme-change re-blend and rgb()/
garbage --chrome-bg parsing are covered, and SegmentTrack's rendered
inline background is asserted to be a literal rgb() — fails on any
reintroduction of engine-dependent CSS in this lane.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add WebView2 box-color fix under [Unreleased] (#968)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 21:28:19 +05:30
d959aae41b fix(dub): dialogue starts stop snapping to footsteps — sustained-energy onsets, bounded snap (#963) (#967)
* fix(dub): dialogue starts stop snapping to footsteps — sustained-energy onsets, bounded snap distance (#963)

Field report #963 (point 3): dubbed speakers start seconds early or late.
The reporter's own theory was right on the money — 'when a noise is heard
(a sigh or footsteps), it's interpreted as the start of the conversation.'
The #280 onset snapper took the FIRST 20 ms frame above an adaptive RMS
threshold as the speech onset, so any transient qualified; it also had no
snap-distance bound (a wrong onset could move a start by the whole segment
minus 0.3 s) and ran even when Demucs had failed and the 'vocals' track was
really the raw mix, where every ambient sound is a candidate.

Three layered guards, all pure NumPy (no new deps):

- Sustained energy: an onset only counts when >=160 ms of the following
  300 ms stays above the threshold. Footsteps/door thuds light up one or
  two frames and die; syllables keep the energy up.
- Bounded snap distance: shifts beyond 1.5 s are only trusted when the
  skipped span is (near-)silent — that is exactly the genuine #280
  whisper start-stretch on the vocals track (Demucs removed the music,
  leaving real silence), so long trims over silence still work in full.
  Long jumps over audible content (e.g. quiet speech under the relative
  threshold) are refused instead of playing the dub seconds late; an
  isolated transient in the span (<10% audible frames) doesn't block it.
- Source-aware: snapping now runs only on the separated vocals track.
  dub_core detects the Demucs fallback (vocals_path == audio_path, see
  dub_pipeline) at both call sites and passes separated_vocals=False on
  mixed audio, disabling snapping — whisper's own timestamps beat a
  confidently wrong snap when music/ambience is sustained energy too.

Tests (tests/test_onset_align.py, fail-before/pass-after): transient burst
rejected at detect- and snap-level, transient-only window yields no onset,
long jump over audible content refused, bounded shift over audible lead
still allowed, >1.5 s trim over true silence still snaps (#280 regression
guard), mixed-audio mode is a no-op. 28 pass in the file; full dub-adjacent
suites green.

Credit: theory and repro description by the #963 reporter.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add onset-snap robustness under [Unreleased] (#967)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 21:15:14 +05:30
de99bc3bd5 fix(net): SOCKS-proxy users can synthesize again — ship socksio, cache-first model resolution (#959) (#966)
* fix(net): SOCKS-proxy users can synthesize again — ship socksio, cache-first model resolution, degrade LLM clients (#959)

Under ALL_PROXY/HTTPS_PROXY=socks5:// without socksio installed, httpx
raises ImportError AT CLIENT CONSTRUCTION ("Using SOCKS proxy, but the
'socksio' package is not installed"). huggingface_hub's get_session()
builds exactly that client inside snapshot_download, so POST /generate
500'd with the bare message even for a fully installed model, and
preload_model's model_info probe hit the same error and silently
skipped warm-up. Latent since v0.3.5 — #947's fresh-process engine
spawning unmasked it in v0.3.10 by handing the user's proxy env
directly to a clean backend process.

Three layers, so the class (any session-construction failure) is dead,
not just the reported instance:

* Ship SOCKS support: socksio>=1.0 in [project] dependencies (pure
  Python, MIT, zero transitive deps) AND in backend.spec hiddenimports
  — httpx imports it lazily in try/except, so PyInstaller's tracer
  misses it and the frozen installers would stay broken without the
  explicit entry. uv.lock regenerated; `uv lock --check` and
  `uv sync --frozen` (the Docker/release bootstrap semantics) verified.

* Cache-first model resolution: from_pretrained's snapshot resolution
  extracted into _resolve_snapshot_dir() — local dir, else
  snapshot_download(local_files_only=True) (a complete cache resolves
  with NO HTTP session constructed), else the original network path.
  preload_model's failed network probe now falls back to a cache-only
  check and warms up anyway instead of silently skipping (honest log
  either way).

* Class guards: resolve_skill_client wraps OpenAI() construction —
  env-shaped construction failures degrade to the existing "LLM
  unavailable" contract instead of 500ing the calling feature; and
  core.failure learns SOCKS_PROXY_SUPPORT_MISSING with an actionable
  hint, appended on the raw-string surfaces (global 500 handler,
  model-install SSE) via the new append_hint().

Fail-before/pass-after verified by reverting the fix: 11 of the 12 new
tests fail pre-fix (the remaining one is the unchanged network-fallback
contract). 165 tests green across the touched suites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add SOCKS-proxy resilience under [Unreleased] (#966)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 20:00:08 +05:30
c47633a409 fix(settings): a saved LLM provider survives restart — explicit save activates, stale TRANSLATE_* prefs stop hijacking (#963) (#965)
* fix(settings): a saved LLM provider survives restart — explicit save activates, stale TRANSLATE_* prefs stop hijacking (#963)

"Ollama works until I restart OmniVoice" had three stacked causes:

1. Only "Save & use for translation" persisted the selection. Plain
   "Save" and "Test" sent make_active:false, and on restart
   active_provider_id() deliberately excludes local providers
   (Ollama/LM Studio) from auto-select — so a saved-and-tested Ollama
   was never resolved active again. The PUT handler now also claims the
   active slot on an explicit save when the user has never chosen a
   provider (new llm_providers.stored_active_provider_id(): the stored
   row only — no env pin, no legacy fallback, no auto-detect). An
   explicit prior choice is never stolen; an unconfigured provider
   can't claim the slot; make_active:true still flips.

2. Users of the retired (≤v0.3.7) Translation-LLM panel had
   env.TRANSLATE_* rows in prefs.json, re-imported into os.environ
   every launch — and a live TRANSLATE_BASE_URL resolves the active
   provider to "custom" ahead of auto-select on every restart. New
   startup migration (llm_providers.migrate_legacy_translate_prefs,
   run in main.py BEFORE the prefs→env import) moves those values into
   the custom provider's own settings-store rows (only where the store
   has no value yet) and deletes the prefs rows. Real process env vars
   are never touched; a failed store write keeps the prefs row and
   retries next boot. The legacy endpoint keeps working — via the
   store, without hijacking the active slot.

3. The panel read as "done" after a green Test even when another
   provider stayed active. It now shows a notice after save/Test when
   the edited provider is not the effective active one (suppressed
   while LLM_DEFAULT_PROVIDER pins the choice — the env banner already
   covers that).

Tests (fail-before): 7 new backend tests fail on the old code
(save-activates, never-steals, migration semantics, env untouched,
end-to-end ollama-beats-legacy-env) and the new panel test fails
without the notice; all pass after. Full LLM/settings suites, frontend
vitest (890), typecheck:ci, oxlint, oxfmt and vite build are green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add LLM-provider persistence fix under [Unreleased] (#965)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 19:47:54 +05:30
4aa9abe22a docs+scripts: install fixes — desktop-prod tauri resolution, Ubuntu white-screen guidance, honest GPU/prereq docs (#960 #961 #962) (#964)
* docs+scripts: install fixes — desktop-prod tauri resolution, Ubuntu white-screen guidance, honest GPU/prereq docs (#960 #961 #962)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add the install-fixes batch under [Unreleased] (#964)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 19:34:01 +05:30
d90cfde1bb feat(dub): per-language translations + per-track caches — switching languages stops destroying work (P1) (#958)
* feat(dub): per-language translations + per-track caches — switching languages stops destroying work (P1)

Multi-language dubbing translated per language (#957) but stored everything
in single-slot state, so tracks silently destroyed each other's work:

P1.2 — per-language translation storage (additive):
- Frontend keeps every translation in s.translations[langCode] alongside the
  legacy s.text slot (still = the shown language). Translate All writes both;
  the new store action switchDubLangCode swaps text through the map on a
  user-driven language switch (non-destructive; restore paths keep the plain
  setter); manual edits / restore-original update the current language's
  entry; merge joins per-language texts, split drops them. Rides project
  save/load inside dubSegments — legacy projects behave exactly as before.
- Backend mirrors it as job["segments_i18n"] = {lang: {segKey: text}}
  (segKey = stable id, index for id-less legacy rows), written by
  _sync_job_segments; job["segments"] stays byte-identical for every existing
  consumer. /dub/srt|vtt?lang= and subtitle burn-in now emit THAT language's
  text when present — ExportModal's "all dubs" batch stops producing N
  identical files. Legacy jobs without the field fall back to today's output.

P1.3 — per-track WAV cache + fingerprints:
- Per-segment WAVs are language-keyed (seg_{lang}_{id}.wav). The partial-regen
  read path falls back to legacy seg_{id}.wav ONLY while the job has no
  other-language track — single-language jobs keep their whole on-disk cache;
  multi-track jobs stop splicing the last-generated language into the current
  track. Read-only endpoints (segment preview, clips zip) gained ?lang= with
  the permissive legacy fallback they always had.
- Fingerprints include the track language (segment_fingerprint(track_lang=…),
  /tools/incremental lang=…) and live in job["seg_hashes_by_lang"]; the flat
  job["seg_hashes"] stays as the current track's mirror so the done event,
  history restore and older frontends read it unchanged. A legacy flat map is
  attributed to the job's last-generated language (dropped when unknown) and
  reads stale once — the safe direction. seg_wav_kind is per-track too.
- The frontend stores fingerprints per language and judges "Regen N changed"
  against the ACTIVE track; project save/load and dub-history restore carry
  all tracks' hashes (segHashesByLang / seg_hashes_by_lang, additive).

Tests: fail-before regression coverage — two-track regen never splices the
other language's audio (sample-level assert on the mixed track), legacy
single-track cache reuse + multi-track gate, per-lang seg_hashes with flat
mirror + migration semantics, /dub/srt|vtt?lang= emitting different text per
track with legacy fallbacks, per-lang burn-in, /tools/incremental lang
scoping, and 14 frontend tests for translations round-trips, per-track
fingerprints and legacy-project behaviour. Full backend + frontend suites,
typecheck, lint and format:check green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add per-language storage + per-track caches under [Unreleased] (#958)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 03:05:40 +05:30
bd92729623 feat(dub): multi-language generate translates each language first + picks persist (P1) (#957)
* feat(dub): multi-language generate translates each language first + picks persist with the project

P1.1 — the "Generate N dubs" loop never translated: the backend synthesizes
segment text verbatim, so every multi-language track rendered the same words
and at most one was actually in its language. The loop now runs
translate → generate per pick:

- handleTranslateAll(langOverride?) accepts an optional ISO-code override
  (no-arg Translate All behavior unchanged; a click-event first arg is
  guarded). It resolves true only when a translation actually landed, and
  both it and handleDubGenerate snapshot segments from the store at call
  time — the click-time closures went stale the moment the previous pick's
  translate pass rewrote the segments.
- A pick whose translate fails (request error or all-segments-errored) is
  SKIPPED — never a wrong-language track — the batch continues, and the
  skipped languages are reported in a final toast.
- The redundant first translate is skipped only when pick 1 targets the
  language the editor text is already translated into; every later pick
  always translates.
- Honest progress: the pill shows "Translating → {lang} (i/N)…" before each
  generate, and the header CTA is inert while translating so a re-click
  can't start a second batch (belt: a ref guard in the loop).

P1.4 — multiLangMode/multiLangs move from DubTab component state into the
dub store slice and ride the project save/load payload (exportTracks too).
Additive and back-compat: legacy payloads default to off/empty and leave the
in-session exportTracks untouched (utils/projectState.js).

Tests (fail-before verified: 9 failures on the pre-fix code):
- handleTranslateAll override targets + return semantics + call-time
  segment snapshot (dubTranslateAllOverride.test.jsx)
- per-language translate-before-generate call order, skip-on-failure with
  continuation + skip-report toast, first-pick skip heuristic, unchanged
  single-language path (dubMultiLangGenerate.test.jsx)
- slice defaults/setters/reset, payload round-trip, legacy-payload defaults,
  App.jsx wiring guards (dubMultiLangPersist.test.js)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add multi-lang auto-translate under [Unreleased] (#957)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 01:49:41 +05:30
mergetestandClaude Fable 5 afe819f498 fix(ci): oxfmt the two #956 test files (unbreak main format check)
#956 merged with a red Tests gate — my merge script ran unconditionally
instead of aborting on the gate value; the failure was oxfmt-only on the
two new test files. Whitespace-only fix, tests re-verified green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 01:23:03 +05:30
6b91205036 fix(dub): completed tracks always show their tabs + history keeps its language (P0) (#956)
* fix(dub): completed tracks always show their tabs + history keeps its language (P0)

Root cause chain: the track switcher's visibility expression required
dubLangCode !== 'und' and ended in a tautology (dubTracks?.length > 0 ||
!!dubTracks), so it was effectively keyed to the language dropdown, not
the persisted tracks. History restore always handed the frontend 'und'
because the dub_history language/language_code COLUMNS froze at the
ingest-time "" — the save_job UPSERT never updated them after generation
set them on the job dict (only the job_data JSON carried the real value).
Net effect: a restored project with finished tracks showed no track tabs
until the user re-picked a language.

- DubTab: hasDubbedTrack = done && dubTracks.length > 0 (tracks only;
  also stops the tautology from showing a trackless switcher).
- DubTab auto-jump: membership-guarded — the preview only jumps to a
  language that has a track, else tracks[0]. Kills the preview-404 class
  (restores falling back to 'en' with tracks ['bn'] pointed the player
  at /dub/preview-video?lang=en).
- dub_pipeline.save_job UPSERT: language/language_code now update when
  non-empty (same CASE guard as content_hash), so new saves heal the
  frozen columns and empty re-saves can't clobber them back.
- App.restoreDubHistory: falls back to job_data's language/language_code
  so EXISTING rows in users' DBs restore correctly with no migration.
- P0.2 polish: track pills get duration + timing-strategy tooltips,
  hydrated lazily and failure-silently from the existing
  GET /dub/tracks/{job_id} via new api/dub.dubListTracks; all new
  strings through i18n (en.json).

Tests (fail-before/pass-after): DubTab-level visibility + auto-jump
membership-guard tests (3 of 4 fail pre-fix), pill-tooltip hydration
tests, and save_job language heal/no-clobber tests (heal fails pre-fix).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): open [Unreleased] with the dub track-tabs fix (#956)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 01:21:23 +05:30
1dfa032be4 feat(dub): project title first in the editor header, pipeline stages below (#955)
Header reordered per owner: row 1 = title (+ duration/segments) with the
action buttons, row 2 = the Upload→Export pipeline spine directly beneath
with a tight 2px gap (was: stepper and title side-by-side on one row).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 01:13:15 +05:30
228 changed files with 19027 additions and 1296 deletions
+5 -2
View File
@@ -118,9 +118,12 @@ jobs:
working-directory: frontend
run: bun run format:check
# `bun run test` (frontend/package.json), not `bunx vitest` — bunx
# resolves by npm package name and can miss workspace-hoisted bins,
# then falls back to fetching from npm (#962 class).
- name: Run Vitest (frontend)
working-directory: frontend
run: bunx vitest run
run: bun run test
# Legacy node:test runner for tests/frontend/*.test.mjs
- name: Run frontend node:test (legacy)
@@ -146,7 +149,7 @@ jobs:
- os: windows-2022
label: Windows
rust_target: x86_64-pc-windows-msvc
- os: ubuntu-22.04
- os: ubuntu-24.04
label: Linux
rust_target: x86_64-unknown-linux-gnu
runs-on: ${{ matrix.os }}
+13 -2
View File
@@ -213,13 +213,24 @@ jobs:
bundles: "msi,updater"
# Linux: ship .AppImage only. AppImage is universal (no distro
# package-manager dep), runs on any glibc-2.31+ host, and is the
# package-manager dep), runs on any glibc-2.39+ host, and is the
# Linux auto-update target. The .deb target was dropped: tauri-bundler
# fails it with "Failed to create control scripts: No such file or
# directory" (no custom deb config of ours is at fault) — revisit on a
# tauri-cli bump. FUSE unavailability on GH runners is handled via
# APPIMAGE_EXTRACT_AND_RUN=1.
- os: ubuntu-22.04
#
# Bumped from ubuntu-22.04 → ubuntu-24.04 (#961): the AppImage
# bundles whatever `libwebkit2gtk-4.1-dev` the build runner's apt
# repos resolve (see the "Linux system deps" step below) — 22.04's
# was meaningfully stale relative to what current Ubuntu/Fedora
# ship, and AppRun's LD_LIBRARY_PATH makes that bundled, stale copy
# take priority over a healthy system WebKitGTK at runtime. Raises
# the AppImage's glibc floor from 2.35 to 2.39 — pre-2022 distros
# (Ubuntu <22.04, Debian <12) lose support; no report of anyone on
# something that old has come in, and the project's own install
# docs already assume Debian 12 / Ubuntu 22.04+.
- os: ubuntu-24.04
arch: x86_64-unknown-linux-gnu
label: "Linux x64"
rust_target: x86_64-unknown-linux-gnu
@@ -0,0 +1,205 @@
# Adjacent open-source projects — research notes (2026-07-10)
Owner-requested research on five neighboring projects, read against OmniVoice
Studio's current feature-maturity map. Each section ends with what we should
take from it. Priorities are consolidated at the bottom.
| Project | Stars | License | Status | Why it matters to us |
|---|---|---|---|---|
| [Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning) | ~60k | MIT | Retired (models frozen 2019, maintainer quit 2020) | Positioning/SEO opportunity, cautionary tales |
| [VoxCPM](https://github.com/OpenBMB/VoxCPM) | ~33k | Apache-2.0 | Very active (VoxCPM2, Apr 2026) | **Upstream of our `voxcpm2` engine** — sync items below |
| [ebook2audiobook](https://github.com/DrewThomasson/ebook2audiobook) | ~19.5k | Apache-2.0 (default XTTS weights are CPML non-commercial) | Very active, weekly releases | The playbook for our weakest shipped surface (audiobook) |
| [VideoLingo](https://github.com/Huanshere/VideoLingo) | ~17.7k | Apache-2.0 | Active, bursty | Dub-pipeline techniques (translation loop, timeline fit) |
| [voicebox](https://github.com/jamiepine/voicebox) | ~40.2k | MIT | Very active, post-viral triage debt | **Direct competitor** — same stack, same pitch, 10x the audience |
## 1. Real-Time-Voice-Cloning — the retired ancestor
The 2019 SV2TTS implementation ("clone a voice in 5 seconds") that created the
DIY voice-cloning category. Explicitly retired: the maintainer said in 2020 he
won't develop it again; the README now calls itself old and redirects users to
Chatterbox. Models are frozen 2019 checkpoints — 16 kHz, English-only, weak
similarity, Tacotron+WaveRNN. Community PRs keep the install alive (uv
one-command install landed Sept 2025), but ~163 open issues are mostly "how do
I make it sound good" — the answer is: you can't.
**Integrating it as an engine: no.** Strictly worse than everything we ship,
plus PyQt/legacy baggage.
**Take:**
- 60k stars of traffic reads a README that says "go elsewhere," and the
redirect target is a model repo, not a product. An honest
"Real-Time-Voice-Cloning alternative" comparison page is cheap, truthful,
and lands exactly our pitch (local, free, modern quality, 646 languages,
actual installer).
- Its headline copy discipline ("Clone a voice in 5 seconds, generate
arbitrary speech in real-time") is better than ours; our 3-second-reference
claim deserves the same outcome-first, time-boxed phrasing.
- Its failure modes validate our Core Value: out-of-band model links rotted
for years; a research toolbox without packaging drowned in install issues.
## 2. VoxCPM — upstream of our `voxcpm2` engine
Tokenizer-free TTS on a MiniCPM-4 backbone. Current model is **VoxCPM2**
(Apr 2026): 2B params, 30 languages + 9 Chinese dialects, 48 kHz, ~8 GB VRAM,
RTF ~0.30 (0.13 with Nano-vLLM). Latest tag v2.0.3 (May 2026); main has
unreleased seed support and timestamp alignment. Apache-2.0, healthy cadence,
~868k monthly HF downloads.
**Sync items for our integration** (we install `voxcpm` unpinned):
1. **Floor the install at `voxcpm>=2.0.3`** — it carries the MPS
audio-quality fix (low-precision dtypes promoted to float32 on Apple
Silicon). Directly relevant to our default-platform-parity rule.
2. **v2.0.1 removed reference-audio auto-trim** — if we hand raw user clips
to cloning, we now own trim/normalize. Verify our clone path; cloning
quality may have silently regressed when upstream released 2.0.1.
3. **Trailing-audio guard**: end-of-audio gibberish/hallucination is a known
open upstream bug (#352). A trailing-silence/garbage trim on our side is
cheap insurance.
4. **Later, when tagged**: seed support (reproducible generation — currently
buggy upstream, #351) and timestamp alignment (useful for dub sync);
`generate_streaming()` is a candidate for `tts_stream.py`.
5. **Risk**: unpinned dependency + active upstream = next release lands
silently in fresh installs. Consider pinning a tested range.
## 3. ebook2audiobook — the audiobook playbook
Any-format ebook (epub/pdf/docx/even scanned images via OCR) → Calibre
normalize to EPUB → TOC/spine chapters ("blocks") → per-language sentence
split → per-sentence TTS → chapterized m4b with metadata/cover. Gradio UI +
headless CLI + Docker for every accelerator. Engine roster is 2023-era Coqui
(XTTSv2 default, Bark, Piper, MMS…), with voice-conversion post-processing to
fake cloning on non-cloning engines. 19.5k stars, near-weekly releases, only
4 open issues.
This is the mature version of exactly the surface where we're weakest: our
audiobook/stories feature is a thin UI over per-chapter render caching, with
no server-side ebook parsing and no per-segment regeneration.
**Take (prioritized):**
1. **Per-sentence render cache + content-hashed blocks + missing-file
resume.** Every sentence is its own file; restart re-renders only what's
missing; editing a block invalidates only that block. This closes our
biggest audiobook gap (per-chapter cache, no crash resume) and is the same
span-level model spec 03 already calls for — dub's `incremental.py`
pattern, extended to longform.
2. **Normalize-to-EPUB ingestion** (Calibre `ebook-convert`) instead of
building N format parsers; blocks carry keep/drop flags for front matter.
3. **Engine-agnostic text-normalization pre-pass**: per-language abbreviation
maps, num2words, roman numerals, and a non-text character filter that
kills TTS hallucination triggers. Benefits every engine we ship, not just
audiobooks.
4. **Chapterized m4b output** (ffmpeg FFMETADATA chapters, cover art, VTT
sidecar) — small work, high perceived value.
5. **Inline voice/pause tags** for multi-voice narration — our cloning
quality makes this worth more to us than it is to them.
**Where we already win:** native desktop UX, modern engine quality
(CosyVoice3/IndexTTS2/VoxCPM2 vs 2023 Coqui), real zero-shot cloning without
VC hacks, no Calibre-wall install, and a commercially-clean default engine
(their default XTTS weights are CPML non-commercial).
## 4. VideoLingo — dub-pipeline techniques
"Netflix-quality subtitles + dubbing" as a 14-stage Streamlit pipeline:
yt-dlp → WhisperX word-level ASR → spaCy + LLM two-candidate semantic split →
summarize-first terminology glossary → 3-step TranslateReflectAdapt →
length-constrained subtitles → duration-aware dub-chunk planning →
per-chunk reference audio → TTS → merge. Its recommended path is
cloud-heavy (API LLM/TTS, optionally API ASR); fully-local is possible but
fragile. Single-speaker only — it explicitly gave up on diarized multi-voice
dubbing. Apache-2.0, ~17.7k stars, bursty maintenance, install pain on
Windows/CUDA.
**Take (prioritized):**
1. **TranslateReflectAdapt** — add a reflection/critique pass to our
per-segment translation prompt. Prompt-level change, meaningful quality
win on idiomatic output.
2. **Summarize-first glossary** — extract theme + terminology once per video,
inject into every segment's translation. Fixes term drift on long videos.
3. **Duration-aware chunk planning** — estimate TTS duration *before*
generating; classify each line ok / needs-speedup / impossible; borrow
inter-subtitle gap time and merge adjacent segments before resorting to
atempo; for impossible lines, LLM-trim filler from the dub text instead of
chipmunking. Our smart-fit handles the tail of this; their pre-planning
avoids generating doomed audio at all.
4. **Two-candidate split prompt** — generate two `[br]` segmentations, have
the LLM pick, instead of accepting the first.
**Where we already win:** fully local by design, per-segment regeneration +
directorial AI (they have coarse folder-state resume, no per-segment redo),
cross-platform installers, cloning stable across languages. Their
single-speaker ceiling is our opening if diarized multi-voice dubbing ever
ships.
## 5. voicebox — the direct competitor
Jamie Pine's (Spacedrive founder) "open-source AI voice studio. Clone,
dictate, create." — architecturally a near-twin: **Tauri + React/TS +
FastAPI/Python + SQLite**, MIT, local-first, explicitly pitched as
ElevenLabs-out + WisprFlow-in replacement. Launched Jan 29, 2026; the launch
post did ~17M views on X, and it sits at **~40.2k stars** with ~10 community
contributors and heavy AI co-authorship. Latest tagged release v0.5.0
(Apr 2026); main is active but untagged for ~10 weeks, with **434 open
issues / 105 open PRs** — a polished happy path with thin edges.
Engines: Qwen3-TTS 0.6B/1.7B (flagship cloner), Qwen CustomVoice, LuxTTS,
Chatterbox Multilingual (23 langs) + Turbo, HumeAI TADA, Kokoro. Features
where they lead: global-hotkey dictation overlay with LLM transcript cleanup
(macOS-verified), Pedalboard post-FX chain, generation versioning/starring,
multi-track Stories editor, **MCP per-client voice bindings** ("Claude Code
speaks in your cloned voice") used as a viral wedge, DirectML/Intel-Arc
coverage, and an agent-facing CONTRIBUTING pattern that farms drive-by
contributions.
Two strategic facts:
- **They are adding accounts.** "Log in with browser" auth for a
`voicebox.sh` cloud tier merged July 5 (their PR #812). Open-core with a
paid cloud is visibly forming — which cuts against the pitch that won them
their audience.
- **Press already flagged their missing consent/misuse policy** — we ship
watermarking by default and consent attestation in `.ovsvoice`.
**Where we're ahead:** 646 languages vs 23, video dubbing (they have none),
voice design from text descriptions (roadmap item for them, shipped for us),
engine breadth (CosyVoice3/VoxCPM2/IndexTTS2/GPT-SoVITS/sherpa-onnx), and
backward-compat/release discipline.
**Take:**
1. **Positioning: own "no accounts, ever."** Their cloud login is our
opening — state the local-first guarantee in the README as a permanent
commitment, next to the 646-language and dubbing advantages they can't
match today.
2. **Tell the MCP agent-voice story loudly.** We already ship an MCP server
and Agent Skills; per-client voice bindings + a speak-in-your-voice demo
was their single best growth hook and costs us mostly marketing effort.
3. **Generation versioning/starring and post-FX presets** — cheap,
high-perceived-value Studio features worth absorbing.
4. **Watch their triage debt** (434 open issues): our absorb-or-decline
queue discipline is a real contributor-trust differentiator — keep it.
## Consolidated priorities
Ordered by (user impact on already-shipped surfaces) × (effort):
1. **voxcpm2 upstream sync** (§2 items 13): version floor, ref-clip trim
audit, trailing-audio guard. Small, protects an engine users already run.
2. **Dub translation quality loop** (§4 items 12): reflect pass + glossary.
Prompt-level, no new deps, lifts the flagship dubbing feature.
3. **Audiobook maturity via per-sentence cache + resume** (§3 item 1): the
established pattern for the feature the maturity survey ranked weakest —
and it's the same architecture spec 03 already prescribes.
4. **Text-normalization pre-pass** (§3 item 3): engine-agnostic hallucination
reduction; pairs with the pronunciation dictionary we already shipped.
5. **Duration-aware dub planning** (§4 item 3) and **chapterized m4b export**
(§3 item 4): next tier, both self-contained.
6. **Competitive positioning vs voicebox** (§5 items 12): own "no accounts,
ever" while they onboard a cloud tier, and tell the MCP agent-voice story
we already technically ship.
7. **RTVC comparison/migration page** (§1): marketing, not engineering;
cheap and honest.
*Method note: compiled from five parallel research passes over the repos'
READMEs, releases, issues, and (for ebook2audiobook) source; figures as of
2026-07-10.*
+139
View File
@@ -6,6 +6,145 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
The bundled TTS model package (`pyproject.toml`) is versioned independently.
## [0.3.16] — 2026-07-11
The quality release. Three long-standing frictions got structural fixes: **regenerating no longer destroys good takes** (a takes rail with starring and restore), **audiobooks stop redoing finished work** (per-sentence caching — edit one line, re-render one line; crashes resume where they stopped), and **dub translations stay consistent and fit their timeline** (auto-glossary + a naturalness pass, plus fit prediction before any GPU time is spent). Under the hood, every text path now speaks numbers, times, and abbreviations correctly, the VoxCPM2 engine gained upstream-alignment guards, and a Windows first-run breaker — model downloads completing but the cache ending up with broken file links — now self-heals automatically. Thank you @dmnobunaga for the razor-sharp diagnosis on that last one.
### Fixed
- **Windows: model downloads that finished but wouldn't load now repair themselves.** On machines without Developer Mode, the model cache could end up with all its multi-gigabyte files downloaded but the snapshot's file links broken — and the app reported a misleading "does not appear to have a file named model.safetensors". The app now detects the broken links on load failure, restores just the missing pieces (reusing everything already downloaded, and falling back to real file copies where links can't be trusted), and retries once; if repair is impossible, the error finally names the actual cache folder to delete. Root-caused in the wild by @dmnobunaga — thank you. (#1056)
- **VoxCPM2: cloning reference clips are now conditioned, and outputs lose their silent tails.** Reference audio used to reach the model completely raw; it now gets edge-silence trimming and a 30-second cap (fail-open — short clean clips pass through untouched), and generated audio gets a trailing-silence trim. The install hint also moved to `voxcpm>=2.0.3`, which carries an important Apple-Silicon audio-quality fix — older installs keep working and see an upgrade hint in the logs. (#1055)
- **Streaming TTS requests without an `emo_alpha` field no longer crash.** A minimal `/ws/tts` request hit a `KeyError` and returned an error frame instead of audio — found while giving that route its first tests. (#1054)
- **Long generations no longer risk a multi-gigabyte memory spike while being watermarked.** The invisible watermark (on by default) pushed the entire waveform through AudioSeal in a single call, and its memory use grows with audio length — a multi-minute generation demanded a single ~2 GB allocation, enough to fail outright on a 16 GB machine already holding a model ("DefaultCPUAllocator: not enough memory"). Watermark embedding — and the Verify-audio detector, which had the same flaw with uploaded files — now processes audio in ~30-second chunks, so peak memory stays flat no matter how long the audio is. Detection also got sharper for spliced files: it now reports the strongest chunk instead of a whole-file average. (#1045)
- **The ⊕ Insert token list no longer climbs out of the viewport.** In the voice-clone script panel, the insert popover (expression tags, CMU phoneme chips) always opened *upward* from the textarea — and since that input sits at the very top of the panel, the list disappeared past the top of the window with no way to see or scroll it. It now opens below the input, where there's always room. (owner-reported)
### Added
- **Edit one sentence, re-render one sentence.** Audiobook and Stories renders now cache every synthesized sentence individually (content-addressed, under the existing chapter cache): fixing a single line in a chapter reuses all the untouched audio, and an interrupted render — crash, quit, power loss — resumes from the sentences that already finished instead of redoing the whole chapter. One byte cap bounds both cache layers, and chapter caches from released versions keep working. (#1048)
- **Numbers, times, and abbreviations are spoken correctly in every engine.** A conservative normalization pass now runs before TTS everywhere (Studio, dubbing, audiobooks): "3:30" is read as a time, "2" as "two" (29 languages), "Dr." as "Doctor" — while stray control characters and markup remnants that trigger engine hallucinations are stripped. Deliberately cautious: when a rewrite could be wrong, the text is left alone, and your pronunciation-dictionary entries always have the final say. Toggleable (`text_normalization_enabled`, default on). The OpenAI-compatible API, streaming TTS, and the batch queue run the same pass, so every door into the engines speaks text identically. (#1049, #1054)
- **Dub translations stay consistent and sound natural (LLM engine).** Before translating, one pass over the whole transcript builds a terminology glossary (your manual glossary entries always win) that rides along on every segment, so names and terms stop drifting mid-video. After each segment's direct translation, an optional reflect pass critiques and rewrites stiff lines into natural spoken dialogue — any failure silently keeps the direct translation. Both toggleable in the Dub tab; the reflect toggle states its 3-calls-per-segment cost. (#1050)
- **The Dub tab now predicts which lines won't fit — before wasting GPU time on them.** After translation, each segment gets a duration estimate (self-calibrating to your engine and language from the segments already rendered) and a "Tight fit" or "Won't fit +Ns" badge when the dubbed audio can't match the timeline even with speed-up. An opt-in "Suggest shorter lines" option asks the LLM for a meaning-preserving shorter rewrite you can apply per segment — never applied automatically. (#1051)
- **Generation takes: star the good ones, restore any of them.** Regenerating no longer means losing the previous result — recent takes appear in the workspace history with replay, star/unstar, and one-click restore as the active output. History is now capped (Settings → Storage, default 200 takes): the oldest unstarred takes are pruned, starred ones are kept forever, and an audio file is only deleted when nothing else references it. (#1052)
- **A persistent mini-player for all the audio that used to play "invisibly".** Generated output, voice-profile and dub-segment previews, story lines, Gallery voices, and Projects renders all played through a bare audio pipe — no waveform, no seek, no time, and (until v0.3.15's stop pill) no way to stop them. A slim player bar now docks above the Logs footer whenever such audio plays, on every page: live waveform (decoded once from the audio already in memory — nothing is re-fetched), click/drag/keyboard seek, play/pause, elapsed/total time, what's-playing label, and a stop button. It replaces the stop-only pill, and because it's part of the app's layout rather than a floating overlay, the pill's "covers the Production Overrides row at 1440×900" overlap class can't come back. Stories line previews also route through it — which makes them stoppable *and* fixes them being silent on the macOS/Linux desktop builds (their old playback path used blob: URLs, which WebKit refuses to play). (no issue — owner request following #1032's stop-pill band-aid)
## [0.3.15] — 2026-07-10
The cold-start release. Three "why is this broken on my machine" mysteries got solved at their roots: **first generations stop dying at 300 seconds** (the timeout was counting the model download as generation time — @moduvoice measured it on a Tesla T4: 0% GPU for the full window), **updates stop deleting engines you installed yourself** (the updater's dependency sync removed anything not in the app's lockfile — including things our own UI told you to install), and **the "slower than v0.3.5" regression is found and fixed** (clone profiles without a transcript were silently re-running a full Whisper transcription on every single generate). Also: Clear History is back, auto-played audio is finally stoppable, @stronghamjji hardened the dub pipeline against wedged transcribes, and @shakib30's community Colab notebook is now the linked no-GPU path. Thank you all.
### Added
- **Agent Skills: `npx skills add debpalash/omnivoice-studio`.** Two installable [skills](https://skills.sh) now ship in the repo — `omnivoice` teaches any AI agent (Claude Code, Cursor, Codex, …) to speak and transcribe through your local install via the OpenAI-compatible API, including your cloned voices; `oss-maintainer` packages the maintainer methodology this project is run with.
### Fixed
- **A fresh install's first generation no longer dies at 300 seconds while the model is still downloading.** The generate timeout was one clock around everything — including the engine's lazy multi-GB weight download on a cold start — so first requests burned the whole budget on the download (0% GPU the entire time, as a contributor's Tesla T4 verification measured) and failed with a misleading "too heavy for the available compute" error. Model loading now runs first under its own, much larger budget; the generate clock starts only once the engine is warm. A genuinely stalled download gets a new error that says so and points at Settings → Models. (#1033, #1037, evidence from #1014)
- **The OpenAI-compatible speech endpoint stops silently discarding quality settings.** `POST /v1/audio/speech` accepted `num_step` and `guidance_scale` in the request body with a 200 OK — and dropped them without a word, so API callers couldn't reach the model's documented quality preset (`num_step: 32`). Both are now declared, validated, and passed through to the engine, matching the native `/generate` endpoint. Caught by a contributor's measured Tesla T4 verification pass. (#1014)
- **Updating no longer uninstalls engines you added yourself.** Optional engines installed with pip into the app's environment (VoxCPM2, KittenTTS — exactly what Settings → Engines' own install hints say to do) were silently deleted by every app update, because the update's dependency sync removed anything not in the app's lockfile. Routine updates now leave your additions alone; the repair path ("Clean & Retry") still restores the exact known-good state, since a broken environment is sometimes *caused* by an extra package. (#1029)
- **Voice cloning stops re-transcribing the same reference clip on every generate.** Since v0.3.6, a profile saved without a transcript (the default) triggered a full ASR model load *plus* a transcription of the reference on every single synthesis — the "TTS got much slower than v0.3.5, same settings" regression. The first auto-transcription is now saved onto the profile, and repeated ad-hoc uploads of the same clip reuse a content-keyed transcript cache — so the cost is paid once, not per request. A transcript you typed yourself is never overwritten. (#1032)
- **The Clear History button is back.** The workspace redesign moved generation history into the right-side panels but dropped the old sidebar's clear-all control, leaving one-by-one deletion as the only way to empty a long history. Both the Voice and Dub history panels now have a Clear History button (with a confirmation), scoped to that workspace's history. (#1032)
- **The audio that auto-plays after a render can finally be stopped anywhere.** The finished-render playback has no on-screen player, and the only stop control lived in the Voice workspace's action bar — audio started from the Dub workspace, a profile preview, or after navigating away simply played to the end. A stop button now appears above the status area whenever such playback is active, on every page. The existing Settings → Appearance "Auto-play preview" toggle now also governs this playback, as its description always promised. (#1032)
## [0.3.14] — 2026-07-09
A fast follow to v0.3.13: **every engine family now has a visible picker.** Settings → Engines showed only a TTS table, with the ASR and LLM pickers hidden behind a low-discoverability tab — so the 10 transcription engines (including the new OpenAI-compatible backend) looked unswitchable without env vars. Now all three families get their own table. Also in: the Linux AppImage's white-screen auto-workaround now checks the WebKitGTK it actually ships (not whatever your system reports), and installing to a different drive on Windows is properly documented.
### Added
- **ASR engines get the same Settings picker TTS has.** Settings → Engines now shows a visible picker table per family — TTS, ASR, and LLM — instead of a single TTS-titled table with the other families tucked behind a tab (README even promised a Settings ASR picker that didn't exist). The OpenAI-compatible backend and the 9 local ASR engines become selectable with one click, no env vars needed; an explicit `OMNIVOICE_ASR_BACKEND` still wins over the Settings pick, so pinned setups behave exactly as before. (no issue — UX gap found during #877)
### Fixed
- **The Linux AppImage's white-screen auto-workaround now checks the right WebKitGTK.** The launcher decided whether to apply the compositing workaround by asking the *system's* `pkg-config` — but the version that actually runs is the *bundled* one, which the AppImage prioritizes. On any machine where the two diverge (e.g. building from source with newer dev packages installed), the detection read the wrong number and could skip a workaround the running library needed. The build now stamps the bundled version into the AppImage at package time, and the launcher reads that stamp — correct by construction. The launcher's shell tests also now run in CI, which they previously never did. (#961 follow-up)
### Docs
- **Windows: installing to a different drive is documented** — the wizard's directory picker works for any local drive; mapped network drives are a Windows Installer limitation (not installable-to by design); and the big data (models/voices) moves independently via Settings → Storage or Portable mode. (#938)
## [0.3.13] — 2026-07-09
The community-fixes release. Two contributors didn't just report bugs — they diagnosed them to the exact line and submitted the fixes that shipped: **voice cloning on mlx-audio's CSM model works for the first time**, and **macOS live recording finally gets its microphone permission prompt** (both @MahdiHedhli). A third reporter's A/B analysis fixed **cross-language dubs speaking the wrong language**. On top of that: a backend shutdown race that produced confusing crash-on-quit reports is fixed, the Linux AppImage stops shipping a stale WebKitGTK that white-screened current distros, and a new OpenAI-compatible transcription backend opens a path to Qwen3-ASR today. Thank you to everyone who filed, diagnosed, and contributed — this release is mostly yours.
### Added
- **A path to Qwen3-ASR today: generic OpenAI-compatible transcription.** The direct integration is still blocked on `transformers>=5.13` stabilizing upstream, but a community member proposed splitting the work — add a backend that talks to any OpenAI-compatible transcription server right now. Point OmniVoice at a self-hosted Qwen3-ASR/FunASR/SenseVoice server, or OpenAI's own API, configured in Settings → Models. No install; audio does leave your machine to whichever server you configure, unlike every other ASR engine. (#877)
### Fixed
- **The Linux AppImage no longer white-screens on current distros with a healthy system WebKitGTK.** The release build ran on an older CI base image, and the resulting AppImage bundles whatever `libwebkit2gtk` that image's apt repos resolve — which the AppImage's own `LD_LIBRARY_PATH` then prioritizes over your system's newer, healthy copy at runtime. A from-source build (which links straight against your system library) worked fine on the exact same machine where the shipped AppImage didn't — that split was the tell. Bumped the release build to a current Ubuntu LTS. Raises the AppImage's minimum host to glibc 2.39 (Ubuntu 24.04+); no reports from anyone on an older distro. (#961)
- **Backend shutdown no longer races a still-loading model, surfacing a confusing crash on restart.** Quitting the app while a model was still loading in the background let shutdown report itself "done" while a background thread was still mid-import; tearing the process down under that thread produced a misleading error (a generic transformers import-failure message, unrelated to the real cause) that looked like a real crash rather than a timing issue. All background tasks are now properly cancelled and awaited before shutdown proceeds. (#1000, likely the same class behind #941 and #979)
- **Cross-language dub no longer speaks the source-language reference line verbatim.** Auto-generated speaker clones pair an audio slice with the ASR segment's own text field, assuming the two agree — but ASR segment text and its timestamps routinely drift (a trailing word audible in the clip but missing from the text, or vice versa). A mismatched (reference audio, reference text) pair breaks zero-shot TTS prompt priming badly enough that the clone can emit the reference text itself instead of the target-language line it was asked to speak. Each reference clip is now re-transcribed after it's written, so the pair matches by construction — reported with an exceptionally clear root-cause diagnosis and a working A/B repro. (#1004)
- **Voice Gallery errors now say what actually went wrong.** "Use voice", "Preview", search, upload, save, delete, and trim in the Gallery all showed the same hardcoded guess ("the engine may be loading") on ANY failure — a 500, a validation error, a genuinely unrelated bug — discarding the real, already-clean backend error message in the process. Every one of those now shows the actual error.
- **Voice cloning on mlx-audio's CSM model no longer crashes with an opaque "list index out of range".** `MLXAudioBackend.generate()` read `voice`/`ref_audio`/`language`/`speed` from its kwargs but silently dropped `ref_text` — CSM only builds its cloning context when both `ref_audio` and `ref_text` are present, so cloning on this engine could never have worked as shipped. Reported with the exact root cause and a working fix. (#1012, #1013)
- **A dub segment's free-text style tags no longer 400 the segment preview.** A validator-safe instruct builder already keeps Studio and Clone generation from round-tripping a 400 on unsupported free-text (a preset's raw attrs, an old profile's stray descriptive phrase) — but the Dub tab's segment preview, and saving a profile from a clone or from history, built their instruct strings directly and skipped it. Same guard now applies everywhere an instruct string is sent. (#1010)
- **The dub editor's play button no longer sticks permanently disabled after an audio-decode hiccup.** When the initial WaveSurfer decode fails, the timeline falls back to loading pre-computed peaks — the waveform draws fine, but the button's enabled state only relied on the `ready` event firing again for that recovery load, which it didn't reliably do. Each fallback path now confirms readiness explicitly once it settles.
- **macOS: live recording finally works — the microphone permission prompt now actually appears.** The app never showed up in System Settings → Privacy & Security → Microphone because macOS never saw a legitimate request: Tauri enables Hardened Runtime by default, which blocks microphone hardware access unless the matching entitlement is in the signed bundle — and it wasn't. Diagnosed to the exact mechanism and fixed by a community contributor (@MahdiHedhli), who also corrected our initial mis-read of this as an upstream WebKit limitation. (#1013, #1016)
- **Quitting during a slow model load waits longer before giving up.** A post-merge code review of the shutdown-race fix flagged that its 3-second wait could still be outrun by a cold model import on a slow disk, reproducing the original confusing-crash-on-quit in rare cases. The wait is now 20 seconds — imperceptible on a normal quit (tasks finish or cancel in milliseconds), only felt in the exact case it protects. (#1020)
### Changed
- **Removed the donate heart from the nav rail.** Support OmniVoice is still one click away from Settings and the Contact page.
### CI
- **The "flaky trio" is root-caused and neutralized.** Three tests failed intermittently on CI — never locally — across unrelated PRs, costing a re-run each time. Cause: a leaked half-precision torch default from some earlier test in CI's ordering (the giveaway: a failing assertion's observed value was exactly float16(0.1)). An autouse test-suite guard now resets the leak between tests and names the offending test in CI output when it fires. (#1021)
## [0.3.12] — 2026-07-08
A community-issue sweep — nineteen open reports triaged in one pass, most fixed same-day. The through-line: **your active engine selection is now honored everywhere** (dubbing, batch, and — new in this release — MLX-Audio's own curated models are finally selectable instead of always silently defaulting to Kokoro), **first-run stops dead-ending users on restricted networks or behind corporate TLS proxies**, and a run of sharp community diagnoses (a one-line ROCm index fix, a Windows-only focus-stealing bug, a genuine crash regression) got fixed largely because reporters did the hard diagnostic work themselves. Thank you.
### Added
- **MLX-Audio's other 6 curated models are finally selectable.** The engine multiplexes Kokoro, CSM, Qwen3-TTS, Dia, Chatterbox, MeloTTS, and OuteTTS, but there was no way anywhere in the UI or API to pick which one loads — downloading a model via Settings → Models did nothing, since the backend always defaulted to Kokoro regardless. Settings → Engines now shows a model picker on the mlx-audio row; switching takes effect immediately, no restart needed. (#981)
### Fixed
- **First-run no longer dead-ends behind restricted networks (e.g. China).** The system check probed hardcoded huggingface.co, and any failure locked the Continue button — users behind the Great Firewall were stuck on the very first screen, even when they had already configured a working mirror. The check now probes the Hugging Face endpoint actually in effect, an unreachable endpoint is a warning instead of a blocker (models already on disk keep working offline), and when huggingface.co is blocked but the hf-mirror.com community mirror answers, the wizard says so and offers a one-click mirror switch right on the check screen — no restart needed. (#984)
- **Installs behind a corporate or antivirus TLS-inspecting proxy no longer fail with a raw SSL error.** `SSLV3_ALERT_HANDSHAKE_FAILURE` happens when a proxy re-signs HTTPS traffic with a root CA your OS trusts but Python's bundled certificate list doesn't — a different failure mode from the network-blocking case above. OmniVoice now trusts your OS's certificate store directly, which should resolve the handshake outright rather than just explain it better. (#976)
- **The loaded-models panel now says when a resident model is not your active engine.** Switching TTS engines keeps the previous model in VRAM (so switching back is instant) — but the panel showed it with no context, so "OmniVoice TTS — 1.9 GB" after selecting VoxCPM2 looked like the selection was ignored. A field report confirmed the confusion. Resident-but-inactive models are now tagged "not active — safe to unload", and the API self-describes each entry's engine. (#985)
- **Voices no longer ship with a hidden echo.** Every non-raw synthesis was getting a small room reverb baked in by the mastering pre-stage — on top of whatever effect preset you chose, so even "Podcast" (which promises *no reverb*) had some, and Cinematic/Warm got it twice. A field report ("a lot of echo/reverb on some of the voices") led straight to it. The mastering stage is now highpass + compressor only; reverb happens only when a preset explicitly declares it. Also documented: cloned voices reproduce the reference clip's room acoustics — dry, close-mic references clone cleanest. (#986)
- **Your engine selection now actually applies to Dubbing and Batch TTS.** Both hardcoded OmniVoice regardless of what was picked in Settings → Engines — pick VoxCPM2, dub anyway with OmniVoice, no error. Both now resolve the active engine up front; an engine that can't clone from reference audio (KittenTTS, Sherpa-ONNX, Supertonic 3 — fixed preset voices only) fails the job immediately with a clear message naming which engines do support it, instead of silently substituting OmniVoice or mis-cloning every speaker into one voice. Batch only requires cloning when a specific voice is pinned — an unpinned batch job runs on any engine. (#987)
- **AMD ROCm torch install no longer silently falls back to CPU.** A community member (Kaihui-AMD) diagnosed it precisely: the ROCm wheel index we pointed at tops out at PyTorch 2.5.1, but the app pins `torch==2.8.0` — the reinstall was unsatisfiable and silently kept the default CUDA build, which runs on CPU on an AMD GPU. Bumped the default index to one that actually carries the pinned version. (#972)
- **mlx-audio no longer crashes on unsupported languages.** Selecting a language like Dutch, Spanish, or Portuguese with mlx-audio's Kokoro model crashed with a raw, unreadable internal-details dump instead of a real error — the code was guessing an ISO language code by truncating the language name, which only worked by coincidence for a few languages. Unsupported languages now fail cleanly with a message naming what's actually supported, and no engine can leak a raw crash-internals dump into an error message again. (#977)
- **The voice-design panel no longer crashes on certain saved voice profiles.** A genuine regression: an earlier translation fix accidentally introduced a crash when a saved design profile's data was incomplete (possible from an older app version or a partial save). Fixed at every layer — the render no longer crashes, both places that restore saved data complete it first, and profiles can no longer be *saved* with incomplete data in the first place. (#983)
- **Windows: the dictation pill no longer steals focus.** Pressing the dictation shortcut activated the pill window, which meant the auto-paste landed back in OmniVoice instead of whatever app you were dictating into, and the pill would get stuck on screen. Precisely diagnosed by a community reporter; fixed to match how this already worked on macOS. (#982)
- **The nemo-parakeet ASR engine's install hint no longer breaks your backend.** Following the in-app "pip install nemo_toolkit[asr]" instruction silently downgraded core packages your backend needs to start — the install reported success, and the breakage only showed up on the next restart. The hint now says plainly that this isn't safe to install into the shared environment. (#974)
- **A stuck generate now tells you the actual fix.** When a job times out from GPU/VRAM contention, the error explained why but never mentioned Flush/Unload — the one action that actually resolves it, and one the sibling ASR-timeout error already recommended. (#939)
### Changed
- **README and Linux docs no longer advertise a `.deb` package that isn't published.** `.deb` bundling is disabled in the release pipeline pending a tauri-cli fix; the docs now say so honestly instead of pointing at a file that was never in any release. (#961 investigation, #990)
- **Linux install docs mention `yt-dlp` as an optional prerequisite** — previously only surfaced via an in-app warning after the fact. (#973)
- **A benign Tauri startup warning no longer looks like an app problem.** On some Windows configurations, Tauri's own internal IPC fallback logs a warning that's fully harmless (it silently and successfully falls back to another transport) — it was spuriously flipping the Settings → Logs footer to show "1 warning" on every launch. Filtered out of the diagnostic capture. (#975)
## [0.3.11] — 2026-07-05
The multi-language release — dubbing into several languages at once is finally a mature, honest workflow: **"Generate N dubs" now translates each language before rendering it** (with visible per-language progress), **switching languages never destroys your work** (every track keeps its own text, subtitles, and audio cache), completed tracks always show their tabs, and dialogue stops starting seconds early because of footsteps — a community reporter's theory, confirmed exactly. Around it, a reliability sweep driven by same-day field reports: your LLM provider finally survives a restart, SOCKS-proxy users can synthesize again (installed models now load without touching the network at all), timeline boxes are visible on every WebView2 runtime, running from source works again — and when the backend crashes, **it now tells you the exit code and attaches the evidence to your bug report automatically**.
### Added
- **Backend crashes are now self-documenting.** When the local backend process dies (a native GPU abort, an out-of-memory kill), the app used to show only "Can't reach the backend" — undiagnosable without logs nobody sends. The launcher now records every unexpected backend death (exit code, how long it ran, the last 40 log lines), tells you honestly that it *crashed* and is restarting, offers a "View crash details" panel, attaches the evidence to in-app bug reports automatically (paths scrubbed), and stops silent crash-loops after 3 deaths in 10 minutes with the details on screen. Intentional shutdowns, restarts, and app quits are never misreported as crashes. (#969)
- **"Generate N dubs" now actually translates each language first.** Multi-language generation used to synthesize every track from whatever text was in the editor — so at most one of your N dubs was really in its language. The batch now runs translate → generate per language with a visible "Translating → Bengali (2/3)…" phase, skips (and reports) any language whose translation fails instead of rendering a wrong-language track, and your multi-language picks and export-track selection are saved with the project instead of vanishing on tab switch. (#957)
- **Switching dub languages no longer destroys your work — every track keeps its own text and audio.** Translations are now stored per language (switching the target swaps the editor text non-destructively; manual edits stay with their language), subtitles export each track's own text instead of N identical files, burned-in subs match their track, and the per-segment audio cache is keyed by language — "Regen changed" can no longer splice another language's audio into the track you're rebuilding, and staleness is tracked per track. Fully backward-compatible: existing projects and caches keep working; a pre-upgrade project's first "Regen changed" simply regenerates cleanly once. (#958)
### Fixed
- **Timeline segment boxes are visible on every WebView2 runtime.** The v0.3.10 flicker fix switched box colors to a newer CSS feature (`color-mix`) applied as an inline style — on WebView2 runtimes older than ~March 2023 (pinned enterprise/offline installs) that renders as *fully transparent*, turning "flickering boxes" into "no boxes at all" while looking perfect on up-to-date machines. Colors are now pre-blended in plain JavaScript to universally-supported `rgb()` values — pixel-identical on modern runtimes, theme-aware, and guarded by a test that fails if an engine-dependent color ever reaches the timeline again. (#968)
- **Dubbed dialogue stops starting seconds early because of footsteps.** Dialogue starts are snapped to the first detected sound — and a single 20 ms burst (footsteps, a door, a sigh) counted as "speech", with no limit on how far a start could jump, and the snap even ran on the raw mix when vocal separation had failed. Onsets now require sustained speech-like energy, long jumps are only allowed across genuinely silent spans (so the original fix for whisper's stretched starts keeps working), and snapping turns off entirely when vocals weren't separated. Credit to the community reporter whose "footsteps theory" was exactly right. (#967)
- **Completed dub tracks always show their video tabs.** Opening a project with a finished dubbed track hid the Original/track switcher until you re-selected the language — visibility was keyed to the language dropdown instead of the project's tracks, and restored projects couldn't set the language because the history database froze it at empty forever. Tabs now render from the tracks themselves, history keeps its language (existing projects heal without migration), restoring a project can no longer 404 the video preview, and track pills gained duration/timing tooltips plus an accurate now-playing indicator. (#956)
- **Running from source works again, and the install docs stop lying.** `bun run desktop-prod` broke when the frontend became a workspace (`bunx` could fetch the wrong "tauri" package from npm — fixed everywhere including CI); the Linux white-screen guidance now leads with the variable that actually fixes modern Ubuntu (`WEBKIT_DISABLE_DMABUF_RENDERER=1`, with the exact `EGL_BAD_PARAMETER` error quoted); Windows docs now state plainly that GPU acceleration is NVIDIA-only there; the Linux docs document the ROCm support that already shipped (the "planned follow-up" note was stale); and prerequisites are split installer-vs-source with git and curl included. (#964)
- **Your LLM provider now survives a restart.** Setting up Ollama (or any provider), testing it, and saving looked like it worked — then a restart forgot the selection: only the separate "Save & use for translation" button ever persisted it, and a leftover setting from the retired (≤0.3.7) translation panel could silently steal the choice back to "Custom" on every launch. An explicit save now activates the provider when none was chosen yet, the leftover legacy settings are migrated into the Custom provider once and removed, and the panel says "Saved — not yet used for translation" instead of staying silent when your edit isn't the active provider. (#965)
- **SOCKS-proxy users can synthesize again — and an installed model can never again be blocked by a broken network stack.** With a system-wide SOCKS proxy set, clicking Synthesize 500'd with a raw "socksio not installed" error: loading an already-downloaded model still constructed a network session first, which failed at creation. The app now ships SOCKS support (including in the packaged installers), resolves installed models **cache-first** (no network session when the files are already on disk — the local-first guarantee at the loader level), warms up at startup even when the online check fails, degrades LLM extras instead of crashing on proxy errors, and classifies the error with an actionable hint if it ever does surface. (#966)
## [0.3.10] — 2026-07-05
The listening release — nine fixes in twenty-four hours, almost all driven by your v0.3.9 field reports (several with same-day turnaround). The dubbing pipeline stops lying: **Cinematic and Autofit can no longer invent dialogue**, the **speaker count you set is honored on every path** (and auto-cloning stops fabricating voices from guessed labels), and the timeline stops flashing invisible on Windows. Audiobook chapters with pauses render again. And one fix everyone should want: **updating can no longer leave you secretly running the old version** — a leftover backend from a previous install holding the port is now detected and replaced at launch. Plus: the Dub tab's LLM engine finally runs on the provider you configured in Settings, history timestamps stop reading "20617d ago", and the Engines page can't crash under concurrent load.
+13 -174
View File
@@ -3,7 +3,7 @@
**OmniVoice Studio**
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The latest stable release is **v0.3.5**; `main` rolls ahead at **v0.3.6** (latest release + 1 patch — see the Versioning rule below).
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The current version lives in `frontend/package.json` (the single source of truth — see Versioning); the latest stable tag is on the [Releases page](https://github.com/debpalash/OmniVoice-Studio/releases/latest). With `AUTO_VERSION_BUMP` off (the current owner setting), `main` holds at the released version between releases.
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
@@ -16,175 +16,21 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release (currently **v0.3.5**). ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
<!-- GSD:project-end -->
<!-- GSD:stack-start source:research/STACK.md -->
## Technology Stack
## Recommended Stack — Per Capability
### Capability 1 — HuggingFace Token Persistence (issue #35)
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| `huggingface_hub` (already pinned transitively by `transformers>=5.3.0`) | `≥1.12.x` (latest 2026) | Auth + cache + token storage | Canonical, used by every HF library already in the stack. `HfFolder` is **superseded** in v1.x by the higher-level `login()` / `auth_list()` / `auth_switch()` API. |
| `keyring` (Python) | `≥25.x` | Optional OS-keychain backing | Only adopt if a future hardening pass wants Keychain/Credential-Manager/SecretService. **Not recommended for this milestone** — adds a native dep (`dbus`, `pywin32`) per platform with no real security win over `0600` file storage in `HF_HOME`. |
| Shell | One-liner to persist `HF_TOKEN` |
|-------|---------------------------------|
| macOS zsh (default since 10.15) | `echo 'export HF_TOKEN=hf_xxx' >> ~/.zshrc && source ~/.zshrc` |
| Linux bash | `echo 'export HF_TOKEN=hf_xxx' >> ~/.bashrc && source ~/.bashrc` |
| Windows PowerShell (user scope) | `[Environment]::SetEnvironmentVariable("HF_TOKEN","hf_xxx","User")` (new shells only) |
| Windows cmd | `setx HF_TOKEN "hf_xxx"` (user scope, new shells only) |
- [HF environment variables docs](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables) — HIGH confidence (official, current)
- [HF authentication API docs](https://huggingface.co/docs/huggingface_hub/en/package_reference/authentication) — HIGH confidence
- [Microsoft `setx` docs](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/setx) — HIGH confidence
### Capability 2 — In-App Structured Bug Reporting (opt-in, GitHub Issues)
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| GitHub REST API `POST /repos/{owner}/{repo}/issues` | `2026-03-10` API version | Server-side issue creation | Official, stable. Requires auth. |
| **Prefilled-URL pattern** (`github.com/{owner}/{repo}/issues/new?title=…&body=…&labels=…`) | n/a | Zero-auth fallback | **This is the recommended primary path for v0.3.x.** No token needed, no GitHub App registration needed, user's browser opens with a prefilled form, they review and click Submit. They own the issue, the OSS project gets the report, and OmniVoice never holds a credential. |
| `gh-app-jwt` + GitHub App (Rust crate `octocrab` or Python `pygithub`) | only if we later want fully-automated submission | Programmatic posting under an app identity | **Defer to a later milestone.** Requires registering a public GitHub App, hosting a token-exchange endpoint, and managing rate-limit quotas — disproportionate for stabilization scope. |
| `platform`, `psutil`, `torch.cuda` (already in deps) | already pinned | Capture OS, CPU/GPU/VRAM info | No new deps. |
| `httpx` (already in `dev-dependencies`, promote to runtime if needed) | `≥0.28.1` | HTTP for the API call path (if/when we add auth) | Modern async-first, already used in test suite. |
- ✓ No token storage in OmniVoice → no security surface
- ✓ Opt-in by definition (user has to click Submit on github.com)
- ✓ User owns the issue → can be replied to, edited, closed by them
- ✓ Zero infra cost — no proxy, no app, no rate-limit management
- ✓ Works identically on macOS / Windows / Linux via Tauri's `shell.open`
- ✓ Survives our project being forked (just change the URL)
- OS name + version (`platform.platform()`)
- Python version (`sys.version`)
- OmniVoice version (`pyproject.toml`)
- Backend git SHA (if installed from source) or installer build ID
- CPU model, RAM (`psutil.cpu_count()`, `psutil.virtual_memory()`)
- GPU vendor/model/VRAM (`torch.cuda.get_device_name()`, `torch.cuda.mem_get_info()`, MPS detect)
- Active TTS engine + list of installed engines
- Frontend: bun version, OS shell
- Last error message + stack trace if launched from an error toast
- Audio file contents (privacy — reference samples may contain user's voice)
- File paths containing `/Users/<name>/` (strip home dir → `~/`)
- HF token, OpenAI keys, any env var matching `*TOKEN*|*KEY*|*SECRET*`
- [GitHub URL query parameters for issues](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/creating-an-issue#creating-an-issue-from-a-url-query) — HIGH confidence
- [sindresorhus/new-github-issue-url](https://github.com/sindresorhus/new-github-issue-url) — HIGH (widely used reference impl)
- [GitHub REST API: Create an issue](https://docs.github.com/en/rest/issues/issues#create-an-issue) — HIGH confidence (for the future auto-submit path)
- [sentry-tauri](https://github.com/timfish/sentry-tauri) — reviewed, **rejected for milestone** due to local-first constraint
### Capability 3 — `uv venv` Mirror Fallback for Restricted Networks (issues #57, #60)
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| `uv` (already used) | `≥0.5.x` | Python+venv bootstrap | Existing dep. |
| `UV_PYTHON_INSTALL_MIRROR` env var | uv `0.4.x`+ | Override python-build-standalone download URL | **Official, current.** Replaces `https://github.com/astral-sh/python-build-standalone/releases/download/...` in download URL construction. No built-in fallback if mirror fails. |
| `UV_PYTHON_PREFERENCE=only-system` (or CLI flag `--python-preference only-system`) | uv `0.4.x`+ | Skip the python-build-standalone download entirely; use the user's system Python | **The reliable escape hatch** when no mirror works. Requires a compatible Python `>=3.11` to already be on PATH. |
| `UV_HTTP_TIMEOUT`, `UV_HTTP_CONNECT_TIMEOUT`, `UV_HTTP_RETRIES` | uv `0.4.x`+ | Tune retry behavior for flaky links | Defaults are 30s / 10s / 3 — bump to 120s / 30s / 5 for restricted networks. |
# Pseudocode for the bootstrap
# Final fallback: don't download Python at all
- `UV_DEFAULT_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple` (Tsinghua — fastest in China)
- `UV_DEFAULT_INDEX=https://mirrors.aliyun.com/pypi/simple` (Aliyun fallback)
- Russia: no major government-blessed PyPI mirror; users typically tunnel via VPN. Document this honestly rather than ship a broken default.
- [uv environment variables reference](https://docs.astral.sh/uv/reference/environment/) — HIGH (official)
- [uv issue #5224 — python-build-standalone mirror support](https://github.com/astral-sh/uv/issues/5224) — HIGH (the feature was added)
- [uv issue #14187 — venv on Chinese network](https://github.com/astral-sh/uv/issues/14187) — HIGH (confirms real user pain, no built-in fallback)
- [uv python-versions concepts](https://github.com/astral-sh/uv/blob/main/docs/concepts/python-versions.md) — HIGH (documents `python-preference` semantics)
- [dautovri/mirrors-china](https://github.com/dautovri/mirrors-china) — MEDIUM (community-maintained mirror list; verify each URL still works before shipping)
### Capability 4 — Supertonic-3 TTS Engine
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| `supertonic` (PyPI) | `1.3.1` (latest, May 18 2026 — Phase 3 Wave 1 to verify constructor signature before bump) | Official Supertonic-3 inference SDK | Authoritative wrapper from Supertone Inc. Wraps the ONNX session orchestration so we don't have to. |
| `onnxruntime` | `≥1.17.x` (any recent) | ONNX inference runtime | Already a transitive dep of WhisperX (via CTranslate2 path is separate, but `onnxruntime` itself ships for kittentts and audioseal). Verify with `uv tree` after adding — should resolve cleanly. |
| `huggingface_hub` (already pinned) | `≥1.12.x` | Model weight download (~400 MB on first use) | Reuses existing HF token + cache infrastructure. The user's existing `HF_TOKEN` (Capability 1) works for the Supertonic model download too. |
| `numpy`, `soundfile` (already pinned) | already pinned | Audio I/O + array math | No new deps. |
- `text_encoder.onnx`
- `latent_denoiser.onnx`
- `voice_decoder.onnx`
- 44.1 kHz sample rate, 24-dim latent, 128-dim style
- ~99M parameters total
- Tokenizer: `AutoTokenizer.from_pretrained(model_path)` — loads from `tokenizer.json` shipped with model
- [Supertone/supertonic-3 model card](https://huggingface.co/Supertone/supertonic-3) — HIGH (official)
- [supertone-inc/supertonic GitHub](https://github.com/supertone-inc/supertonic) — HIGH (official)
- [supertonic PyPI page](https://pypi.org/project/supertonic/) — HIGH (`1.3.1` confirmed 2026-05-18; same publisher, MIT, same 4 deps)
- [onnx-community/Supertonic-TTS-ONNX](https://huggingface.co/onnx-community/Supertonic-TTS-ONNX) — HIGH (ONNX file structure details)
### Capability 5 — Cross-Platform Documentation Tooling
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| Plain Markdown in `docs/` + GitHub-rendered (current state) | n/a | Install tutorial, troubleshooting | Zero new infra. Renders inline on GitHub for issue-replies. No build step to break. |
| Existing `scripts/smoke-test.sh` + Playwright `tests/` (already in `package.json`) | already pinned | Verify install paths actually work | **This is the real solution to "docs drift."** If smoke-test exercises the install path described in docs, docs that drift will break CI. |
| **Future** (defer): Astro Starlight | `≥0.30` | Standalone docs site at `docs.omnivoice.studio` | Adopt only when docs exceed ~20 markdown files and need search/versioning. Tauri, the framework OmniVoice already depends on, uses Starlight — well-traveled choice. Material for MkDocs entered maintenance mode in November 2025 per Docsio's 2026 review — **avoid** for new docs. |
| Project | What they do |
|---------|--------------|
| **OBS Studio** | Docs at `obsproject.com/docs` (Sphinx, separate repo). Install paths in README, wiki for community-contributed. CI doesn't gate on docs drift. |
| **Audacity** | Manual at `manual.audacityteam.org` (MediaWiki). README is minimal. Install path = "use the installer." No automated sync. |
| **Tauri** | Docs at `v2.tauri.app` (Astro Starlight, separate repo `tauri-apps/tauri-docs`). README is minimal. Heavy reliance on community contributions and PR review. |
| **VS Code** | Docs at `code.visualstudio.com/docs` (separate repo, Markdown). README is minimal. Manual sync; docs team is staffed. |
- [Tauri docs (Astro Starlight)](https://github.com/tauri-apps/tauri-docs) — HIGH (reference for "if we ever move off README")
- [OBS Studio docs](https://docs.obsproject.com/) — HIGH (Sphinx, separate site reference)
- [Audacity Manual](https://manual.audacityteam.org/) — HIGH (MediaWiki reference)
- [Docsio: Material for MkDocs 2026 review (maintenance mode)](https://docsio.co/blog/mkdocs-material) — MEDIUM (third-party review, but signal aligns with project's own GitHub activity)
- [Docsio: Starlight 2026 review](https://docsio.co/blog/starlight-docs) — MEDIUM
## Installation
# No new Python dependencies needed for Capabilities 1, 2, 3, 5.
# Only Capability 4 adds a runtime dep:
# Verify no regressions:
# Should show single versions of each; no duplicates.
## Alternatives Considered
| Recommended | Alternative | When to Use Alternative |
|-------------|-------------|-------------------------|
| HF token via in-app Settings → `huggingface_hub.login()` | OS keyring via `keyring` package | Only if a security hardening milestone later demands OS-native credential storage. Not worth the cross-platform native-dep cost for v0.3.x. |
| Prefilled-URL GitHub Issues | GitHub App + device flow + authenticated POST | When milestone budget can afford registering a public GitHub App and hosting a token-exchange function. Defer. |
| Prefilled-URL GitHub Issues | Sentry / `sentry-tauri` | Never — violates the "no third-party telemetry endpoint" constraint in PROJECT.md. |
| `UV_PYTHON_INSTALL_MIRROR` chain + `only-system` fallback | Bundle Python in the Tauri installer | Adds ~30 MB to every installer for ~5% of users. Revisit if the bootstrap is still a top complaint in v0.4. |
| In-repo Markdown docs | Astro Starlight standalone site | When docs grow past ~20 pages and need full-text search. Tauri provides a precedent if/when we get there. |
| In-repo Markdown docs | MkDocs / Material for MkDocs | **Avoid** for new sites — Material for MkDocs is in maintenance mode as of Nov 2025. |
## What NOT to Use
| Avoid | Why | Use Instead |
|-------|-----|-------------|
| `HfFolder.save_token()` directly | Older API; v1.x `login()` does the same plus git-credential integration and is the documented path | `huggingface_hub.login(token=val, add_to_git_credential=False)` |
| Setting `HF_TOKEN` via shell rc files as the *only* persistence mechanism | Different per OS, fragile, opaque to the user, breaks in installer-launched processes that don't source shell rc | Write to `$HF_HOME/token` via `login()`. Document env var as override only. |
| `setx` for HF token persistence | Doesn't propagate to current shell; common source of "I set it but it's empty" bug reports | `[Environment]::SetEnvironmentVariable(...,"User")` in PowerShell, or the in-app Settings field |
| PAT-based GitHub Issues posting from OmniVoice | Would require shipping or asking for a token; breaks local-first promise | Prefilled-URL pattern (user submits from their browser) |
| `sentry-tauri` for OmniVoice | Third-party telemetry endpoint — violates PROJECT.md constraint | Local-only `backend.log` rotation + opt-in prefilled-URL reporter |
| `hf_transfer` for downloads | Deprecated in favor of `hf-xet` per HF docs | Default `huggingface_hub` (uses `hf-xet` automatically when available) |
| `--python-preference managed` (default) without mirror config in restricted-network installers | Hits GitHub CDN, times out, user sees raw `uv` error | Configure `UV_PYTHON_INSTALL_MIRROR` + retry chain + `only-system` final fallback |
| Material for MkDocs as a *new* docs choice | Entered maintenance mode November 2025 | If docs site is eventually needed, use Astro Starlight (Tauri precedent) |
## Stack Patterns by Variant
- Set `UV_PYTHON_INSTALL_MIRROR` to one of the gh-proxy URLs at install time
- Set `UV_DEFAULT_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple` (China) or document VPN requirement (Russia)
- Fall back to `UV_PYTHON_PREFERENCE=only-system` if all mirrors fail
- Increase `UV_HTTP_TIMEOUT=120`, `UV_HTTP_RETRIES=5`
- Default path: in-app Settings field → `login()` → file at `$HF_HOME/token`
- Power-user path: `export HF_TOKEN=...` in shell rc (documented but not promoted)
- Both paths are read at HF library import time; env var wins on conflict
- Default path: in-app "Report a bug" → prefilled GitHub Issues URL → user reviews + submits in browser
- All optional capture toggles default ON except "include reproduction file" (privacy)
- No path posts to any URL except `github.com/{owner}/{repo}/issues/new` (rendered locally as a URL, opened via `shell.open`)
- `uv add supertonic` → new TTSBackend subclass in `backend/services/tts_backend.py`
- Auto-detected and added to the engine picker in Settings
- ~400 MB model download on first synthesize call, cached in `$HF_HUB_CACHE`
- Existing IndexTTS/CosyVoice/etc. installs are untouched (no shared model weights)
## Version Compatibility
| Package A | Compatible With | Notes |
|-----------|-----------------|-------|
| `supertonic@1.3.1` | `onnxruntime>=1.17`, `numpy>=1.24`, `huggingface_hub>=0.20` | All deps already satisfied transitively by current `pyproject.toml`. |
| `huggingface_hub>=1.12` | `transformers>=5.3.0` (current pin) | `HfFolder` retained as deprecated alias; `login()`/`get_token()` are the canonical APIs. |
| `uv>=0.5` | `UV_PYTHON_INSTALL_MIRROR`, `UV_PYTHON_PREFERENCE` | Both env vars stable since uv 0.4.x. |
| Tauri v2 + `@tauri-apps/api/shell` | `shell.open()` for the prefilled-URL pattern | Already in the desktop app; no new permission needed beyond what the existing "open external link" plugin grants. |
## Sources
- [Hugging Face Hub environment variables](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables) — HIGH (verified against v1.12.1 docs, current 2026)
- [Hugging Face Hub authentication API](https://huggingface.co/docs/huggingface_hub/en/package_reference/authentication) — HIGH (verified `login()` is the canonical 1.x API)
- [Microsoft `setx` reference](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/setx) — HIGH (confirms "current shell" gotcha)
- [PowerShell `about_Environment_Variables`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables) — HIGH
- [uv environment variables reference](https://docs.astral.sh/uv/reference/environment/) — HIGH (verified all mirror + retry env vars)
- [uv issue #5224 — python-build-standalone mirror](https://github.com/astral-sh/uv/issues/5224) — HIGH (feature shipped)
- [uv issue #14187 — venv on Chinese network](https://github.com/astral-sh/uv/issues/14187) — HIGH (confirms user pain, justifies fallback chain)
- [uv `python-preference` semantics](https://github.com/astral-sh/uv/blob/main/docs/concepts/python-versions.md) — HIGH
- [Supertone/supertonic-3 model card](https://huggingface.co/Supertone/supertonic-3) — HIGH (official, 99M params, 31 languages, OpenRAIL-M)
- [supertone-inc/supertonic GitHub](https://github.com/supertone-inc/supertonic) — HIGH (official inference API)
- [supertonic 1.3.1 on PyPI](https://pypi.org/project/supertonic/) — HIGH (released 2026-05-18, MIT code license; bumped from 1.2.3 after Phase 3 research)
- [onnx-community/Supertonic-TTS-ONNX](https://huggingface.co/onnx-community/Supertonic-TTS-ONNX) — HIGH (ONNX file structure)
- [GitHub Docs: Authenticating to the REST API](https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api) — HIGH
- [GitHub Docs: Generating a user access token for a GitHub App](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-a-user-access-token-for-a-github-app) — HIGH (device flow reference)
- [sindresorhus/new-github-issue-url](https://github.com/sindresorhus/new-github-issue-url) — HIGH (canonical prefilled-URL reference impl)
- [sentry-tauri](https://github.com/timfish/sentry-tauri) — MEDIUM (reviewed, rejected on PROJECT.md constraint, not on quality)
- [dautovri/mirrors-china](https://github.com/dautovri/mirrors-china) — MEDIUM (community-maintained, verify URLs are still live before pinning in production)
- [Tauri 2 docs (Astro Starlight reference)](https://v2.tauri.app/) — HIGH (precedent for docs framework if we ever migrate)
- [Docsio: Material for MkDocs entered maintenance mode Nov 2025](https://docsio.co/blog/mkdocs-material) — MEDIUM (third-party review, but signal aligns with the project's own GitHub commit activity)
The May-2026 stack research that used to live here served five capabilities that have all since shipped (HF-token Settings panel, prefilled-URL bug reporting, uv mirror fallback for restricted networks, the Supertonic-3 engine, in-repo Markdown docs). Follow the patterns in the code itself; the durable *don'ts* that research established:
- **No third-party telemetry endpoints, ever** (`sentry-tauri` was evaluated and rejected) — bug reporting stays opt-in via prefilled GitHub-issue URLs.
- **No PAT/token-based GitHub posting from the app** — the user submits from their own browser.
- **Don't recommend `setx` for env vars on Windows** (silent truncation, no current-shell propagation) — use the in-app Settings panel or PowerShell `[Environment]::SetEnvironmentVariable`.
- **Don't adopt Material for MkDocs** for any future docs site (maintenance mode since Nov 2025) — Astro Starlight is the precedent if docs ever outgrow the repo.
- **`hf_transfer` is deprecated** — default `huggingface_hub` (hf-xet) handles downloads.
For anything new: prefer what's already pinned in `pyproject.toml` / `frontend/package.json`, and check `uv tree` for conflicts before adding a dependency.
<!-- GSD:stack-end -->
<!-- GSD:conventions-start source:CONVENTIONS.md -->
@@ -222,16 +68,9 @@ No project skills found. Add skills to any of: `.claude/skills/`, `.agents/skill
<!-- GSD:skills-end -->
<!-- GSD:workflow-start source:GSD defaults -->
## GSD Workflow Enforcement
## Workflow
Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.
Use these entry points:
- `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks
- `/gsd-debug` for investigation and bug fixing
- `/gsd-execute-phase` for planned phase work
Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.
Direct repo edits are authorized (owner decision, 2026-07-08). The GSD command gate that used to live here referenced `/gsd-quick` / `/gsd-debug` / `/gsd-execute-phase` skills that are not installed in this environment; the owner chose to keep working directly rather than restore them. The working conventions that matter are in **Conventions** above — versioning, docs-sync, changelog, localization, fix quality, keep-main-green — plus: gate every merge on the "Tests (backend + frontend)" check passing and the PR being MERGEABLE, and check the open-PR queue before implementing any community-reported fix (contributors may have already submitted one).
<!-- GSD:workflow-end -->
+1
View File
@@ -18,6 +18,7 @@ Thanks for your interest in improving OmniVoice Studio! This guide covers everyt
### Prerequisites
- [Git](https://git-scm.com/)
- `curl` (used by the Bun / uv / rustup install one-liners on macOS and Linux)
- [Bun](https://bun.sh/) (frontend package manager)
- [uv](https://docs.astral.sh/uv/) (Python environment manager)
- [ffmpeg](https://ffmpeg.org/) (audio/video processing)
+36 -9
View File
@@ -170,7 +170,7 @@ The eight headliners — and twelve more waiting under the fold.
- 📦 **Batch Queue** — drop 50 videos, walk away; per-job progress bars.
- 🛡️ **AI Watermark** — AudioSeal (Meta): invisible, survives compression.
- 🔬 **Diagnostics** — self-check suite, error journal, scrubbed diagnostic bundles.
-**GPU Auto-Detect** — CUDA · MPS · ROCm · CPU; ≤8 GB VRAM auto-offloads.
-**GPU Auto-Detect** — CUDA · MPS · ROCm (Linux, opt-in) · CPU; ≤8 GB VRAM auto-offloads.
- 🧭 **Engine routing** — preflight GPU check per engine; no silent CPU fallback.
- 🧩 **Extensible** — subclass `TTSBackend`, add any engine in ~50 lines.
- 🎒 **Portable personas** — export voices as `.ovsvoice` bundles: identity + watermark.
@@ -190,7 +190,6 @@ The eight headliners — and twelve more waiting under the fold.
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
<br/>
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy &amp; Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
<br/>
@@ -244,7 +243,7 @@ ElevenLabs charges **$5$330/mo** and processes your audio on their servers. O
| **Video Dubbing** | ✅ Cloud-only | ✅ Fully local |
| **Data Privacy** | Audio sent to cloud | **Nothing leaves your machine** |
| **API Keys** | Required | Not needed |
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm (Linux) · CPU |
| **Desktop App** | ❌ | ✅ macOS · Windows · Linux |
| **TTS Engines** | 1 | **14** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS) |
| **ASR Engines** | 1 | **9** (WhisperX, Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet, Moonshine, FunASR, isolated Faster-Whisper, sherpa-onnx live dictation) |
@@ -267,16 +266,19 @@ Professional-grade voice AI, minus the subscription and the cloud.
| | **Minimum** | **Recommended** |
|---|---|---|
| **OS** | Windows 10, macOS 12+ (Apple Silicon), Ubuntu 20.04+ | Any modern 64-bit OS |
| **OS** | Windows 10, macOS 12+ (Apple Silicon), Ubuntu 24.04+ (glibc 2.39+) | Any modern 64-bit OS |
| **RAM** | 8 GB | 16 GB+ |
| **VRAM (GPU)** | 4 GB (auto-offloads TTS to CPU) | 8 GB+ (NVIDIA RTX 3060+) |
| **Disk** | 10 GB free (models + cache) | 20 GB+ SSD |
| **Python** | 3.10+ (managed by `uv`) | 3.113.12 |
| **GPU** | Optional — CPU works | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm |
| **GPU** | Optional — CPU works | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm (Linux only) |
> [!TIP]
> On GPUs with **≤8 GB VRAM**, OmniVoice automatically offloads TTS to CPU during transcription — no config needed. A dedicated GPU is not required; the entire pipeline runs on CPU (just slower).
> [!NOTE]
> **AMD GPUs:** ROCm acceleration is **Linux-only and opt-in** — pick **"AMD GPU (ROCm)"** on the first-run setup screen or set `OMNIVOICE_TORCH_VARIANT=rocm` ([docs/install/linux.md](docs/install/linux.md#amd-gpu-rocm)). **On Windows, AMD GPUs (incl. Ryzen AI iGPUs) run CPU-only**: PyTorch has no Windows ROCm wheels, so Windows GPU acceleration is NVIDIA/CUDA-only ([docs/install/windows.md](docs/install/windows.md#gpu-support)).
> [!IMPORTANT]
> **macOS Intel (x86_64) is unsupported for the local backend:** the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac wheels ([#889](https://github.com/debpalash/OmniVoice-Studio/issues/889)). Intel-Mac users can still point the UI at a remote backend on another machine — see [docs/install/macos.md](docs/install/macos.md).
@@ -284,7 +286,7 @@ Professional-grade voice AI, minus the subscription and the cloud.
### 🗣️ TTS Engines
**14 engines, one picker.** OmniVoice (default, 600+ languages) is always available; CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, and Sherpa-ONNX are opt-in and auto-detected — plus six lazy-installed heavyweights (IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var.
**14 engines, one picker.** OmniVoice (default, 600+ languages) is always available; CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, and Sherpa-ONNX are opt-in and auto-detected — plus six lazy-installed heavyweights (IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var — the selection applies everywhere synthesis happens: single-clip generation, Voice Cloning, Video Dubbing, and Batch TTS.
<details>
<summary><b>📊 The full matrix</b> — 14 engines × platform × clone/instruct × license</summary>
@@ -310,6 +312,8 @@ Professional-grade voice AI, minus the subscription and the cloud.
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only · ⚡ = lazy-registered (installed on first use)
>
> **Clone** matters beyond single-clip generation: Video Dubbing (and any Batch job with a pinned voice) needs reference-audio cloning to preserve speaker identity, so picking a Clone-less engine (KittenTTS, Sherpa-ONNX, Supertonic 3) as the active engine fails those jobs up front with an actionable message instead of silently falling back to OmniVoice.
>
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path. **Confucius4-TTS** (14-language cross-lingual zero-shot cloning) is similar — its own Python 3.10 venv from a clone; CUDA recommended, CPU validated end-to-end (slow, ~17× realtime; no MPS — tested slower than CPU); see [Confucius4-TTS](docs/engines/confucius4-tts.md).
</details>
@@ -318,10 +322,10 @@ Professional-grade voice AI, minus the subscription and the cloud.
### 🎧 ASR Engines
**9 engines, all fully local** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Settings → ASR Engine** or via the `OMNIVOICE_ASR_BACKEND` env var.
**10 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Settings → Engines** (the ASR Engines table — same picker TTS has), or pin one with the `OMNIVOICE_ASR_BACKEND` env var (the env var wins over the Settings pick). Nine run fully on-device; one (OpenAI-compatible) is an optional remote client for pointing at Qwen3-ASR or another compatible server — see below.
<details>
<summary><b>📊 The full lineup</b> — 9 engines, what each is best at, and compute-type notes</summary>
<summary><b>📊 The full lineup</b> — 10 engines, what each is best at, and compute-type notes</summary>
<br/>
@@ -336,6 +340,7 @@ Professional-grade voice AI, minus the subscription and the cloud.
| **Moonshine** | `moonshine` | English | Edge / low-latency, ONNX |
| **FunASR** | `funasr` | 50+ | All-in-one multilingual — built-in VAD + inline speaker diarization (SenseVoice) |
| **sherpa-onnx** (live dictation) | `sherpa-onnx-asr` | 25 EU + 90+ | Live, faster-than-real-time dictation — small streaming/offline ONNX models (Parakeet TDT v3/v2, streaming Zipformer & Paraformer, Whisper Tiny), CPU, identical on macOS / Windows / Linux. Picked per-model in **Settings → Voice**. |
| **OpenAI-compatible** ⚠️ remote | `openai-compat-asr` | Server-dependent | A path to **Qwen3-ASR** today (self-hosted server, no transformers wait), any OpenAI-compatible transcription endpoint, or OpenAI's own API — no install, configure in **Settings → Models**. Audio leaves your machine to whatever server you point it at; see [docs/engines/openai-compatible-asr.md](docs/engines/openai-compatible-asr.md). |
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. **sherpa-onnx** powers the live dictation model picker — you talk and text appears as you speak. Every engine runs on-device — no API keys, no cloud.
@@ -392,6 +397,20 @@ print(result.text)
Want the whole surface (100+ endpoints)? The full REST API reference is embedded in the app — **Settings → OpenAPI Reference** (Scalar-powered), or the `{}` button in the footer.
### 📓 Run on Google Colab (community)
No local GPU? A community member ([@shakib30](https://github.com/shakib30)) maintains a working Colab notebook: [shakib30/OmniVoice-Studio-google-colab](https://github.com/shakib30/OmniVoice-Studio-google-colab). Community-maintained — issues with the notebook go there; issues with OmniVoice itself come here.
### 🤝 Agent Skills
Teach your AI agent (Claude Code, Cursor, Codex, …) to use OmniVoice with one command:
```sh
npx skills add debpalash/omnivoice-studio
```
Ships two [skills](https://skills.sh): **`omnivoice`** — speak and transcribe through your local install (including your cloned voices) from any agent, free and offline; and **`oss-maintainer`** — the maintainer methodology this project is run with, for anyone running their own OSS project with an agent.
---
## 🗺️ Roadmap
@@ -520,7 +539,15 @@ Yes please — bug fixes, new TTS engine adapters, UI improvements, docs, transl
<details>
<summary><b>Is this really as good as ElevenLabs?</b></summary>
<br/>
For voice cloning and dubbing, yes — OmniVoice uses a state-of-the-art diffusion TTS model with 646 languages (ElevenLabs supports 32). Quality is comparable for most use cases. Where ElevenLabs wins is in their polished cloud API and pre-made voice library. OmniVoice wins on privacy, cost, language coverage, and customizability.
Honest answer: <b>it depends on what you're doing.</b>
<b>Where OmniVoice is genuinely competitive:</b> voice cloning from a clean reference clip (state-of-the-art open diffusion TTS), language coverage (646 languages vs. their 32), and everything structural — no per-character billing, no usage caps, no audio leaving your machine, full pipeline customizability (10 TTS engines, 10 ASR engines, your choice of translation).
<b>Where ElevenLabs still wins:</b> out-of-the-box consistency and polish, especially for English TTS. Their one model is heavily tuned; our quality depends on which engine you pick, your hardware, and — for cloning — the reference audio (a dry, close-mic clip clones dramatically better than a noisy or echoey one).
<b>For dubbing specifically:</b> a dub is a chain — transcription → translation → cloning → synthesis — and the output is only as good as its weakest link on <i>your</i> source material. Noisy or accented source audio degrades transcription, which degrades everything downstream; some language pairs translate better than others. If parts of a dub come out incoherent, check the segment table's <i>original</i> text first: if the transcription is already wrong there, switch the ASR engine (Settings → Engines) or use cleaner source audio — that's usually the fix, not the voice.
Try it on your real material — it's free and takes one download. Many users find it replaces ElevenLabs outright; some keep both for different jobs. Both outcomes are fine with us.
</details>
<details>
+13
View File
@@ -33,6 +33,14 @@ hiddenimports = [
'uvicorn.lifespan', 'uvicorn.lifespan.on',
'fastapi', 'fastapi.responses', 'starlette',
'multipart',
# SOCKS proxy support (#959). httpx imports socksio lazily inside a
# try/except (only when a socks5:// proxy env var is set), so
# PyInstaller's static tracer never sees it — without this entry the
# frozen installers keep raising "Using SOCKS proxy, but the 'socksio'
# package is not installed" on every model load under a SOCKS proxy,
# even though pyproject.toml ships the package. Guarded by
# tests/test_socks_proxy.py.
'socksio',
# Core
'uuid', 'asyncio',
@@ -61,6 +69,11 @@ hiddenimports = [
# Pipeline
'yt_dlp', 'demucs', 'demucs.separate',
# Numbers→words for the pre-TTS text normalization pass
# (services/text_normalization.py). Imported inside a function (lazy),
# so pin it explicitly rather than trusting the tracer.
'num2words',
# OmniVoice's own package
'omnivoice', 'omnivoice.models', 'omnivoice.models.omnivoice',
]
+63 -24
View File
@@ -311,34 +311,60 @@ async def _prepare_synth(default_voice: str | None, language: str | None = None)
return info["synth"], info["sample_rate"], resolve, engine_id
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None):
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None,
language=None):
"""Render one chapter, content-addressed so a re-run reuses it (resume).
Returns ``(wav_path, duration_s, was_cached)``. The WAV lives at
``cache_dir/<key>.wav`` where ``key`` is :func:`chapter_cache_key` over the
chapter's spans + sample rate + engine + each voice's resolved signature
(+ the lexicon, so a lexicon edit re-renders), so an unchanged chapter is
never re-synthesized. Runs in the GPU-pool executor.
Returns ``(wav_path, duration_s, was_cached, seg_stats)``. Two cache
layers:
* Outer the WAV at ``cache_dir/<key>.wav`` where ``key`` is
:func:`chapter_cache_key` over the chapter's spans + sample rate +
engine + each voice's resolved signature (+ the lexicon, so a lexicon
edit re-renders). A fully-unchanged chapter hits here and never touches
segment files; the key derivation is unchanged, so chapter caches
written by released versions keep hitting. ``seg_stats`` is ``None``.
* Inner on a chapter miss, each spoken span goes through the
:class:`services.longform_render.SegmentCache` under
``cache_dir/segments``: cached segments load from disk, only the
edited/missing ones synthesize, and each fresh segment persists the
moment it renders (an interrupted chapter resumes from them).
``seg_stats`` is ``{"total": spoken_spans, "cached": reused}``.
Span text is normalized (``services.text_normalization``) up front BEFORE
either cache key and BEFORE ``synthesize_chapter``'s lexicon pass, so the
per-project dictionary operates on normalized text and toggling / changing
normalization output naturally invalidates cached chapters and segments.
Runs in the GPU-pool executor.
"""
import json
import wave
from services.audio_io import atomic_save_wav
from services.longform_render import chapter_cache_key
from services.audiobook import Span
from services.longform_render import SegmentCache, chapter_cache_key
from services.pronunciation import normalize_lexicon
from services.text_normalization import normalize_for_tts
spans = [Span(voice_id=s.voice_id, text=normalize_for_tts(s.text, language),
pause_ms_after=s.pause_ms_after, speed=getattr(s, "speed", None))
for s in chapter.spans]
spans_tuples = [(s.voice_id, s.text, s.pause_ms_after, getattr(s, "speed", None))
for s in chapter.spans]
sig: dict = {}
for s in chapter.spans:
for s in spans]
voice_sigs: dict = {}
for s in spans:
k = s.voice_id or ""
if k not in sig:
if k not in voice_sigs:
v = resolve(s.voice_id)
sig[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
voice_sigs[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
sig: dict = dict(voice_sigs)
lex_sig = ""
if lexicon:
# Fold the lexicon into the cache key so editing pronunciations
# invalidates cached chapters (reserved key can't collide with a voice id).
sig["\x00lexicon"] = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
lex_sig = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
sig["\x00lexicon"] = lex_sig
key = chapter_cache_key(spans_tuples, sample_rate=sr, engine_id=engine_id, voice_sig=sig)
wav_path = os.path.join(cache_dir, f"{key}.wav")
@@ -346,13 +372,17 @@ def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, le
try:
with wave.open(wav_path, "rb") as w:
dur = w.getnframes() / float(w.getframerate() or sr)
return wav_path, dur, True
return wav_path, dur, True, None
except Exception:
pass # corrupt cache entry — fall through and re-render
audio, dur = synthesize_chapter(chapter.spans, synth, sr, lexicon=lexicon)
seg_cache = SegmentCache(cache_dir, sample_rate=sr, engine_id=engine_id,
voice_sig=voice_sigs, extra_sig=lex_sig)
audio, dur = synthesize_chapter(spans, synth, sr, lexicon=lexicon,
segment_cache=seg_cache)
atomic_save_wav(wav_path, audio, sr)
return wav_path, dur, False
return wav_path, dur, False, {"total": seg_cache.hits + seg_cache.misses,
"cached": seg_cache.hits}
class AudiobookPreviewRequest(BaseModel):
@@ -383,14 +413,15 @@ async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
chapter = plan.chapters[req.chapter_index]
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
os.makedirs(cache_dir, exist_ok=True)
resolved_lang = _resolve_default_language(req.language, req.default_voice)
synth, sr, resolve, engine_id = await _prepare_synth(
req.default_voice,
language=_resolve_default_language(req.language, req.default_voice),
language=resolved_lang,
)
loop = asyncio.get_running_loop()
wav_path, dur, was_cached = await loop.run_in_executor(
wav_path, dur, was_cached, _seg_stats = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
req.lexicon,
req.lexicon, resolved_lang,
)
return {
"output": os.path.relpath(wav_path, OUTPUTS_DIR), # served via /audio
@@ -495,8 +526,9 @@ async def _render_longform_sse(
loop = asyncio.get_running_loop()
try:
resolved_lang = _resolve_default_language(language, default_voice)
synth, sr, resolve, engine_id = await _prepare_synth(
default_voice, language=_resolve_default_language(language, default_voice)
default_voice, language=resolved_lang
)
total = len(plan.chapters)
@@ -508,9 +540,10 @@ async def _render_longform_sse(
for i, chapter in enumerate(plan.chapters):
try:
wav_path, dur, was_cached = await loop.run_in_executor(
wav_path, dur, was_cached, seg_stats = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached,
chapter, synth, sr, engine_id, resolve, cache_dir, lexicon,
resolved_lang,
)
except Exception: # isolate a bad chapter — keep going
logger.warning("[%s] chapter %d (%s) failed to render",
@@ -522,9 +555,15 @@ async def _render_longform_sse(
chapter_files.append(wav_path)
chapters_meta.append((chapter.title, int(round(dur * 1000))))
cached_n += 1 if was_cached else 0
yield _emit({"type": "chapter", "index": i, "total": total,
"title": chapter.title, "duration_s": round(dur, 2),
"cached": was_cached})
ev = {"type": "chapter", "index": i, "total": total,
"title": chapter.title, "duration_s": round(dur, 2),
"cached": was_cached}
if seg_stats is not None:
# Additive fields (old clients ignore them): segment-level
# reuse inside a re-rendered chapter.
ev["segments"] = seg_stats["total"]
ev["cached_segments"] = seg_stats["cached"]
yield _emit(ev)
if not chapter_files:
yield _emit({"type": "error", "error": "all chapters failed to render"})
+26 -13
View File
@@ -179,6 +179,21 @@ async def _run_batch_pipeline(job_id: str, job: dict):
job["status"] = "failed"
return
# ── Engine resolution (issue #312 class) ────────────────────────────
# Batch used to hardcode OmniVoice via get_model() regardless of the
# engine selected in Settings → Engines. require_cloning only when a
# specific voice is pinned (job["voice_id"]) — an unpinned job is fine on
# any active engine. Resolved ONCE for the whole job (every language
# below shares the same active engine); an uncaught ValueError here
# propagates to _worker()'s existing except-Exception handling, which
# already records a structured job failure via core.failure.build_failure.
from services.tts_backend import resolve_generation_backend
backend = await resolve_generation_backend(
require_cloning=bool(job.get("voice_id")),
cloning_purpose="this batch job's pinned voice",
)
sr = backend.sample_rate
# ── 3. Translate + Generate per language ───────────────────────────
total_langs = len(langs)
outputs = {}
@@ -243,13 +258,10 @@ async def _run_batch_pipeline(job_id: str, job: dict):
total_segments=len(translated_segments),
)
from services.model_manager import get_model
from services.audio_dsp import apply_mastering, normalize_audio
from services.audio_io import atomic_save_wav
import torch
_model = await get_model()
sr = _model.sampling_rate
total_samples = int(duration * sr)
full_audio = torch.zeros(1, total_samples)
total_segs = len(translated_segments)
@@ -275,6 +287,13 @@ async def _run_batch_pipeline(job_id: str, job: dict):
continue
def _gen(text=seg_text, lang=target_lang, dur=seg_duration):
# Normalize once at the segment's text→engine choke point —
# the same pre-pass as /generate and dub_generate's _gen.
# `lang` is the job's target language code. Pref-gated,
# idempotent, never raises.
from services.text_normalization import normalize_for_tts
text = normalize_for_tts(text, lang)
ref_audio = None
ref_text = None
@@ -295,22 +314,16 @@ async def _run_batch_pipeline(job_id: str, job: dict):
ref_text = row.get("ref_text")
try:
audios = _model.generate(
audio_out = backend.generate(
text=text, language=lang,
ref_audio=ref_audio, ref_text=ref_text,
duration=dur, num_step=16,
guidance_scale=2.0, speed=1.0,
denoise=True, postprocess_output=True,
)
audio_out = audios[0]
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(
audio_out,
sample_rate=sr,
)
return normalize_audio(mastered, target_dBFS=-2.0)
if not getattr(backend, "applies_own_mastering", False):
audio_out = apply_mastering(audio_out, sample_rate=sr)
return normalize_audio(audio_out, target_dBFS=-2.0)
except Exception as e:
logger.warning("TTS failed for seg %d (lang=%s): %s", i, lang, e)
return torch.zeros(1, int(dur * sr))
+51 -6
View File
@@ -452,6 +452,11 @@ async def dub_transcribe_stream(
asr_audio_target = job.get("vocals_path")
if not asr_audio_target or not os.path.exists(asr_audio_target):
asr_audio_target = job.get("audio_path")
# #963: onset snapping is only trustworthy on the Demucs vocals
# track. When separation failed/was skipped, dub_pipeline sets
# vocals_path to the mixed audio_path — so compare paths instead
# of trusting the key's presence.
asr_on_vocals = bool(asr_audio_target) and asr_audio_target != job.get("audio_path")
if not asr_audio_target or not os.path.exists(asr_audio_target):
preflight_error = "No audio available for transcription."
else:
@@ -649,9 +654,12 @@ async def dub_transcribe_stream(
# leading music/silence (classic case: speech begins at 0:03,
# transcript says 0.0 → the dub plays 3 s early). Snap starts
# forward to the actual speech onset. `audio_np` is the same
# track ASR ran on — vocals.wav when Demucs succeeded.
# track ASR ran on — vocals.wav when Demucs succeeded. #963:
# when it didn't (mixed audio), snapping is disabled — every
# footstep/sigh/score cue is a false onset candidate there.
try:
snap_segment_starts(chunk_segs, audio_np, sr)
snap_segment_starts(chunk_segs, audio_np, sr,
separated_vocals=asr_on_vocals)
except Exception as e:
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
# Provisional per-chunk labels for the streaming UI only — the
@@ -998,6 +1006,24 @@ async def dub_transcribe_stream(
clones = done.pop().result()
break
yield _sse_event("ping", {})
if clones:
from services.speaker_clone import refine_ref_texts
# Bound the re-transcribe like every other ASR dispatch in
# this file (#730): a wedged transcribe would otherwise hold
# the GPU-pool worker forever and starve later work into a
# "can't reach backend". On timeout the guard resets the pool
# and raises — keep the original (unrefined) clones, matching
# refine_ref_text's own "failure is a strict no-op" fallback.
try:
clones = await run_transcribe_guarded(
_gpu_pool,
lambda: refine_ref_texts(clones, _asr_backend),
what="Dub clone ref-text refine",
)
except ASRTimeoutError as e:
logger.warning(
"clone ref-text refine timed out; keeping original ref_text: %s", e
)
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
# own reference from the vocals so the dub of each line matches the
# prosody of its source line. Short lines fall back to the
@@ -1017,6 +1043,19 @@ async def dub_transcribe_stream(
),
)
if seg_clones:
from services.speaker_clone import refine_ref_texts
# Same guard as the per-speaker refine above (#730):
# keep the original seg_clones on a wedge/timeout.
try:
seg_clones = await run_transcribe_guarded(
_gpu_pool,
lambda: refine_ref_texts(seg_clones, _asr_backend),
what="Dub segment ref-text refine",
)
except ASRTimeoutError as e:
logger.warning(
"segment ref-text refine timed out; keeping original ref_text: %s", e
)
job["segment_clones"] = seg_clones
except Exception as e:
logger.warning("per-segment clone refs skipped: %s", e)
@@ -1125,11 +1164,14 @@ async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
_model = await get_model()
def _transcribe():
asr_audio_target = job.get("vocals_path")
if not asr_audio_target or not os.path.exists(asr_audio_target):
asr_audio_target = job.get("audio_path")
# #963: same source-awareness as the SSE endpoint — vocals_path
# falls back to the mixed audio_path when Demucs failed/skipped.
asr_on_vocals = bool(asr_audio_target) and asr_audio_target != job.get("audio_path")
import torch
detected_lang = None
@@ -1173,10 +1215,13 @@ async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
# #280: snap segment starts forward to the actual speech onset so the
# dub doesn't begin seconds before the original speaker does.
# dub doesn't begin seconds before the original speaker does. #963:
# only on the separated vocals track — on mixed audio every ambient
# sound is a false onset candidate, so snapping is disabled.
try:
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
snap_segment_starts(segments, audio_for_onset, onset_sr)
snap_segment_starts(segments, audio_for_onset, onset_sr,
separated_vocals=asr_on_vocals)
except Exception as e:
logger.warning("onset alignment skipped: %s", e)
+78 -15
View File
@@ -170,8 +170,42 @@ async def dub_list_tracks(job_id: str):
return {"tracks": job.get("dubbed_tracks", {})}
def _segments_for_lang(job: dict, lang: "str | None") -> list:
"""Job segments with `text` overlaid from ``job["segments_i18n"][lang]``.
P1.2 ``job["segments"]`` is single-slot: it holds whichever language was
generated LAST, so exporting subtitles for track A after generating track B
emitted B's text under A's language label (the "N identical subtitle
files" class). ``segments_i18n`` ({lang: {segKey: text}}, written by
``dub_generate._sync_job_segments``) preserves each generated track's text;
this overlays it non-destructively when present.
Back-compat: no lang requested, no ``segments_i18n`` on the job (predates
the field), no entry for this lang, or no text for a given segment each
falls back to the segment as-is, i.e. exactly today's behaviour.
Segment keys are the stable id (str) with the list index (str) as the
legacy fallback, mirroring how the map is written.
"""
segments = job.get("segments", [])
if not lang:
return segments
i18n = job.get("segments_i18n")
lang_texts = i18n.get(lang) if isinstance(i18n, dict) else None
if not isinstance(lang_texts, dict) or not lang_texts:
return segments
out = []
for i, seg in enumerate(segments):
key = str(seg.get("id")) if seg.get("id") is not None else str(i)
txt = lang_texts.get(key)
if txt is None:
txt = lang_texts.get(str(i))
out.append(dict(seg, text=txt) if isinstance(txt, str) and txt.strip() else seg)
return out
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
fitted_segments: "list[dict] | None" = None) -> str | None:
fitted_segments: "list[dict] | None" = None,
lang: "str | None" = None) -> str | None:
"""Build a temp SRT from job segments for use with ffmpeg's subtitles filter.
Returned path is already ffmpeg-filter-safe (plain ASCII basename under exports_dir).
@@ -181,8 +215,11 @@ def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
fitted timeline when provided, cue times come from there instead of
the original ``job["segments"]`` timings, so burned subs track the
retimed video / fitted audio rather than the source timeline.
``lang`` (P1.2): burn the named track's text (see ``_segments_for_lang``)
instead of whatever language generated last.
"""
segments = job.get("segments", [])
segments = _segments_for_lang(job, lang)
if not segments:
return None
if fitted_segments:
@@ -486,7 +523,9 @@ async def dub_download(
# Smart Fit: cue times come from the fitted timeline — that's where the
# dubbed audio actually sits, whether or not the video retime succeeds.
fitted_segments = _fitted_segments_for(job, default_track) if default_track and default_track != "original" else None
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments) if burn_subs else None
# Burn the DEFAULT track's text (P1.2) — it's the audio the viewer hears.
_burn_lang = default_track if default_track and default_track != "original" else None
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments, lang=_burn_lang) if burn_subs else None
# ── Smart Fit video retime (two-tier) ─────────────────────────────────
# Tier 1 (≤48 chunks): single filter_complex graph inlined into the mux
@@ -1125,20 +1164,40 @@ async def dub_get_audio(job_id: str):
raise HTTPException(status_code=404, detail="Audio file not found")
return FileResponse(audio, media_type="audio/wav")
def _seg_wav_candidates(job: dict, lang: "str | None", seg_keys: tuple) -> list:
"""Per-segment WAV name candidates, language-keyed first (P1.3).
Generation writes ``seg_{lang}_{id}.wav`` now; ``lang`` defaults to the
job's last-generated track. Legacy un-keyed names (``seg_{id}.wav`` /
``seg_{index}.wav``) stay as fallbacks so jobs rendered by previous
builds keep serving their audio these read-only endpoints keep the
permissive fallback that matches their historic behaviour (the strict
single-track gate lives on the generate splice path, where a wrong-
language read would be baked into a track).
"""
lang = lang or job.get("language_code")
keys = []
if lang:
keys.extend(f"{lang}_{k}" for k in seg_keys)
keys.extend(seg_keys)
return keys
@router.get("/dub/preview/{job_id}/{segment_index}")
async def dub_preview_segment(job_id: str, segment_index: int):
async def dub_preview_segment(job_id: str, segment_index: int, lang: str = Query(None)):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Resolve the stable-id-named WAV via the render manifest; fall back to the
# legacy index name for jobs rendered before id-based naming (#185). Each
# candidate is realpath-normalised and containment-checked BEFORE any
# filesystem access, so the guard dominates every path sink.
# Resolve the stable-id-named WAV via the render manifest — language-keyed
# name first (P1.3), then the legacy id/index names for jobs rendered
# before per-language (and before id-based, #185) naming. Each candidate
# is realpath-normalised and containment-checked BEFORE any filesystem
# access, so the guard dominates every path sink.
order = job.get("seg_order") or []
seg_id = order[segment_index] if 0 <= segment_index < len(order) else segment_index
base = os.path.realpath(DUB_DIR)
seg_path = None
for _sid in (seg_id, segment_index):
for _sid in _seg_wav_candidates(job, lang, (seg_id, segment_index)):
cand = os.path.realpath(dub_seg_path(job_id, _sid))
if cand.startswith(base + os.sep) and os.path.exists(cand):
seg_path = cand
@@ -1342,13 +1401,16 @@ def _fitted_cue_times(job: dict, lang: str | None) -> list | None:
async def dub_export_srt(
job_id: str,
dual: bool = False,
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
lang: str = Query(None, description="Track language code. Emits that track's text (segments_i18n) when the job carries it; when that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
segments = job.get("segments", [])
# P1.2 — text follows the REQUESTED track, not whichever language was
# generated last (job["segments"] is single-slot). Legacy jobs without
# segments_i18n fall back to today's behaviour.
segments = _segments_for_lang(job, lang)
if not segments:
raise HTTPException(status_code=400, detail="No transcript segments available")
@@ -1391,13 +1453,14 @@ def _format_vtt_time(seconds):
async def dub_export_vtt(
job_id: str,
dual: bool = False,
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
lang: str = Query(None, description="Track language code. Emits that track's text (segments_i18n) when the job carries it; when that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
segments = job.get("segments", [])
# Same per-track text resolution as /dub/srt (see comment there, P1.2).
segments = _segments_for_lang(job, lang)
if not segments:
raise HTTPException(status_code=400, detail="No transcript segments available")
@@ -1427,7 +1490,7 @@ async def dub_export_vtt(
@router.get("/dub/export-segments/{job_id}")
async def dub_export_segments_zip(job_id: str):
async def dub_export_segments_zip(job_id: str, lang: str = Query(None)):
import zipfile
job = _get_job(job_id)
if not job:
@@ -1445,7 +1508,7 @@ async def dub_export_segments_zip(job_id: str):
seg_id = order[i] if i < len(order) else i
# realpath + containment guard before any filesystem access.
seg_path = None
for _sid in (seg_id, i):
for _sid in _seg_wav_candidates(job, lang, (seg_id, i)):
cand = os.path.realpath(dub_seg_path(job_id, _sid))
if cand.startswith(base + os.sep) and os.path.exists(cand):
seg_path = cand
+206 -67
View File
@@ -11,7 +11,8 @@ from core.db import db_conn
from core.config import DUB_DIR, VOICES_DIR, dub_seg_path
from core.tasks import task_manager
from schemas.requests import DubRequest
from services.model_manager import get_model, _gpu_pool, run_on_gpu_pool_guarded
from services.model_manager import _gpu_pool, run_on_gpu_pool_guarded
from services.tts_backend import resolve_generation_backend
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
from services.audio_io import atomic_save_wav, _safe_torchaudio_save
from services.ffmpeg_utils import (
@@ -87,6 +88,62 @@ def _sync_job_segments(job: dict, req: DubRequest) -> None:
merged.append(row)
job["segments"] = merged
# P1.2 — per-language text, additively. `job["segments"]` stays the flat
# single-slot map every existing consumer reads (last generated language);
# `job["segments_i18n"]` preserves EACH generated track's text so
# /dub/srt|vtt?lang= can emit that language instead of N identical files.
# Shape: { langCode: { segKey: text } } where segKey is the segment's
# stable id (str) or, for id-less legacy segments, its list index (str).
# The whole per-language map is rebuilt on every generate of that language
# (the request always carries the full segment list), so deleted segments
# never linger. Jobs predating this field simply lack it — every reader
# falls back to `job["segments"]`.
lang = (req.language_code or "und").strip() or "und"
i18n = job.setdefault("segments_i18n", {})
i18n[lang] = {
(str(row["id"]) if row.get("id") is not None else str(i)): row["text"]
for i, row in enumerate(merged)
}
def _seg_hashes_by_lang(job: dict) -> dict:
"""Per-language segment fingerprints: { langCode: { segId: hash } }.
Additive migration (P1.3): jobs written by previous builds carry ONE flat
`seg_hashes` map that was overwritten by whichever language generated
last. That flat map can only describe the job's last-generated track, so
it is attributed to `job["language_code"]` (which generate has always
kept in lock-step with the last run). When even that is unknown the
legacy hashes are dropped segments then read as stale and regenerate
cleanly, which is safer than guessing a language and splicing wrong-track
audio. Note the legacy hashes also predate language-scoped fingerprints
(see services.incremental.segment_fingerprint), so they compare stale
once regardless carrying them over just preserves the job shape.
"""
by_lang = job.get("seg_hashes_by_lang")
if not isinstance(by_lang, dict):
by_lang = {}
legacy = job.get("seg_hashes")
prev_lang = job.get("language_code")
if isinstance(legacy, dict) and legacy and prev_lang:
by_lang[prev_lang] = dict(legacy)
job["seg_hashes_by_lang"] = by_lang
return by_lang
def _legacy_seg_cache_ok(job: dict, lang_code: str) -> bool:
"""May this run reuse legacy un-keyed ``seg_<id>.wav`` files?
Only when no OTHER language's audio could be sitting in them: the job has
no dubbed track in a different language. Single-language jobs rendered by
previous builds therefore keep their whole on-disk cache; the moment a
job carries a second language the un-keyed files are ambiguous (they hold
whichever language wrote them last) and must never be spliced into a
track again the P1.3 cross-contamination class.
"""
tracks = job.get("dubbed_tracks") or {}
return not any(lc != lang_code for lc in tracks)
router = APIRouter()
@@ -100,13 +157,37 @@ async def dub_generate(job_id: str, req: DubRequest):
detail="This dub session has expired or was never created. Re-upload the video to start a new one.",
)
_model = await get_model()
# ── Engine resolution (issue #312 class) ────────────────────────────────
# Dub used to hardcode OmniVoice via get_model() regardless of the engine
# selected in Settings → Engines — a SILENT fallback. Every real dub
# segment's ref_audio resolves to either an auto:<speaker>/auto-seg:<id>
# clone cut from the source video or a saved voice-profile row (see
# `_gen` below), so require_cloning=True: an engine that can't clone
# would either mis-clone per segment or fail deep into the job. Checked
# ONCE here, before the streaming task starts, so a doomed job fails fast
# with one clear message instead of N per-segment ones.
try:
backend = await resolve_generation_backend(require_cloning=True)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
async def _stream(task_id):
total = len(req.segments)
all_segment_wavs = []
sync_scores = []
# Track language for this run. Everything per-track — the per-segment
# WAV cache, fingerprints, seg_wav_kind — is keyed by it (P1.3) so a
# multi-language job's tracks can't cross-contaminate.
lang_code = req.language_code or "und"
def _seg_lang_path(seg_key) -> str:
# Per-language per-segment WAV: seg_{lang}_{id}.wav. Built through
# dub_seg_path so the sanitisation + DUB_DIR containment guard
# apply to the combined key. Legacy un-keyed seg_{id}.wav files
# remain readable via the gated fallback (_legacy_seg_cache_ok).
return dub_seg_path(job_id, f"{lang_code}_{seg_key}")
# Throttle the device cache flush. empty_cache() is a synchronous
# device stall, so calling it every segment (as the old code did)
# serialised the GPU loop; the batched-I/O design it replaced kept
@@ -236,7 +317,16 @@ async def dub_generate(job_id: str, req: DubRequest):
# double-compress. Force one full regen; afterwards seg_wav_kind is
# "natural" and partial regen / fit-only re-mix (regen_only=[]) work.
# Jobs predating this field have unknown kind → also regen once.
if strategy == "smart_fit" and regen_only is not None and job.get("seg_wav_kind") != "natural":
# P1.3: the kind is per-track now (each language renders under its own
# strategy); the flat job["seg_wav_kind"] is only consulted for jobs
# written before the per-language map existed — once the map is
# present, a language without an entry has unknown-kind WAVs (or none
# at all) and must regen once, exactly like the pre-field case.
_kind_map = job.get("seg_wav_kind_by_lang")
_wav_kind = (
_kind_map.get(lang_code) if isinstance(_kind_map, dict) else job.get("seg_wav_kind")
)
if strategy == "smart_fit" and regen_only is not None and _wav_kind != "natural":
regen_only = None
# Manifest: stable segment id per current index. Per-segment WAVs are
# named by stable id (dub_seg_path) so regen reuses the right audio after
@@ -248,6 +338,13 @@ async def dub_generate(job_id: str, req: DubRequest):
# retain every generated tensor in RAM until final assembly.
_pending_seg_writes: list[tuple] = []
# Calibration records for the pre-synthesis duration planner
# (services/duration_planner.py): text length + the NATURAL-rate TTS
# duration of every freshly synthesized segment. Only meaningful for
# the natural-rate strategies — strict_slot forces the audio to the
# slot length, which would poison the observed chars-per-second.
_natural_dur_records: dict[str, dict] = {}
# Phase 4.1 bench instrumentation: measure where incremental time goes.
# Only prints when regen_only is active (real-user incremental path).
_t_start = time.perf_counter()
@@ -266,7 +363,7 @@ async def dub_generate(job_id: str, req: DubRequest):
seg_duration = seg.end - seg.start
if seg_duration <= 0.05 or not seg.text.strip():
sr = _model.sampling_rate
sr = backend.sample_rate
# max(0, …): a zero/negative-duration slot must not feed a
# negative length to torch.zeros (raises) — _store_mix_wav
# turns the empty buffer into a harmless in-memory entry.
@@ -283,31 +380,38 @@ async def dub_generate(job_id: str, req: DubRequest):
# Partial regen: if this segment isn't in the allow-list, reuse its
# previously-rendered WAV so the final mix still covers the timeline.
if regen_only is not None and seg_id not in regen_only:
seg_wav_path = dub_seg_path(job_id, seg_id)
if not os.path.exists(seg_wav_path):
# Back-compat: jobs rendered before id-named files used seg_{index}.wav.
_legacy = dub_seg_path(job_id, i)
if os.path.exists(_legacy):
seg_wav_path = _legacy
# This track's own cache first (seg_{lang}_{id}.wav). Legacy
# un-keyed files (seg_{id}.wav / seg_{index}.wav) are reused
# ONLY when no other-language track exists on the job — a
# multi-track job's un-keyed files hold whichever language
# rendered last, and splicing them here was exactly how
# "Regen N changed" mixed language B into track A (P1.3).
seg_wav_path = _seg_lang_path(seg_id)
if not os.path.exists(seg_wav_path) and _legacy_seg_cache_ok(job, lang_code):
for _legacy_key in (seg_id, i):
_legacy = dub_seg_path(job_id, _legacy_key)
if os.path.exists(_legacy):
seg_wav_path = _legacy
break
if os.path.exists(seg_wav_path):
try:
_t_cache_0 = time.perf_counter()
cached_wav, cached_sr = torchaudio.load(seg_wav_path)
if cached_sr != _model.sampling_rate:
if cached_sr != backend.sample_rate:
import torchaudio.functional as AF
cached_wav = AF.resample(cached_wav, cached_sr, _model.sampling_rate)
cached_wav = AF.resample(cached_wav, cached_sr, backend.sample_rate)
# Pad/trim to slot — except smart_fit, whose mix
# loop needs the natural-rate length to compute the
# audio/video split (the seg_wav_kind guard above
# guarantees these cached WAVs are natural-rate).
if strategy != "smart_fit":
target_samples = int(seg_duration * _model.sampling_rate)
target_samples = int(seg_duration * backend.sample_rate)
current_samples = cached_wav.shape[-1]
if target_samples > current_samples:
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
elif current_samples > target_samples:
cached_wav = cached_wav[..., :target_samples]
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, cached_wav, _model.sampling_rate, f"mix_{seg_id}"))
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, cached_wav, backend.sample_rate, f"mix_{seg_id}"))
try:
del cached_wav
except Exception:
@@ -320,7 +424,7 @@ async def dub_generate(job_id: str, req: DubRequest):
# Fall through to a silent placeholder if the cached WAV
# is broken — cleaner than aborting the whole mix.
yield f"data: {json.dumps({'type': 'warning', 'segment': i, 'message': f'cached seg lost, padding silence: {str(e)[:120]}'})}\n\n"
sr = _model.sampling_rate
sr = backend.sample_rate
silence = torch.zeros(1, max(0, int(seg_duration * sr)))
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, silence, sr, f"mix_{seg_id}"))
try:
@@ -332,6 +436,12 @@ async def dub_generate(job_id: str, req: DubRequest):
continue
def _gen(text, lang, instruct_str, dur_s, nstep, cfg, spd, profile_id, effect_preset):
# Normalize once at the segment's text→engine choke point
# (covers the OOM-retry generate below too, which reuses this
# closure's `text`). Pref-gated, idempotent, never raises.
from services.text_normalization import normalize_for_tts
text = normalize_for_tts(text, lang)
ref_audio = None
ref_text = None
used_seed = None
@@ -404,25 +514,23 @@ async def dub_generate(job_id: str, req: DubRequest):
torch.manual_seed(used_seed)
try:
audios = _model.generate(
audio_out = backend.generate(
text=text, language=lang if lang != "Auto" else None,
ref_audio=ref_audio, ref_text=ref_text,
instruct=instruct_str if instruct_str else None,
duration=dur_s, num_step=nstep, guidance_scale=cfg,
speed=spd, denoise=True, postprocess_output=True,
)
audio_out = audios[0]
sr = _model.sampling_rate if hasattr(_model, 'sampling_rate') else 24000
sr = backend.sample_rate
# Apply per-segment DSP effect preset (default: broadcast)
seg_effect_preset = effect_preset or "broadcast"
if seg_effect_preset == "raw":
return audio_out
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
mastered_audio = audio_out
if not getattr(backend, "applies_own_mastering", False):
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
effect_chain = get_effect_chain(seg_effect_preset)
if effect_chain:
mastered_audio = apply_effects_chain(
@@ -453,24 +561,22 @@ async def dub_generate(job_id: str, req: DubRequest):
nstep, retry_steps,
)
try:
audios = _model.generate(
audio_out = backend.generate(
text=text, language=lang if lang != "Auto" else None,
ref_audio=ref_audio, ref_text=ref_text,
instruct=instruct_str if instruct_str else None,
duration=dur_s, num_step=retry_steps, guidance_scale=cfg,
speed=spd, denoise=True, postprocess_output=True,
)
audio_out = audios[0]
sr = _model.sampling_rate if hasattr(_model, 'sampling_rate') else 24000
sr = backend.sample_rate
seg_effect_preset = effect_preset or "broadcast"
if seg_effect_preset == "raw":
return audio_out
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
mastered_audio = audio_out
if not getattr(backend, "applies_own_mastering", False):
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
effect_chain = get_effect_chain(seg_effect_preset)
if effect_chain:
mastered_audio = apply_effects_chain(
@@ -557,7 +663,7 @@ async def dub_generate(job_id: str, req: DubRequest):
yield f"data: {json.dumps({'type': 'cancelled', 'segments_processed': i + 1})}\n\n"
return
target_samples = int(seg_duration * _model.sampling_rate)
target_samples = int(seg_duration * backend.sample_rate)
current_samples = audio_tensor.shape[-1]
if strategy == "strict_slot":
@@ -575,15 +681,27 @@ async def dub_generate(job_id: str, req: DubRequest):
# trim, slip, stretch the video, or split audio/video
# retiming (smart_fit) to accommodate it.
generated_dur = audio_tensor.shape[-1] / _model.sampling_rate
generated_dur = audio_tensor.shape[-1] / backend.sample_rate
sync_ratio = round(generated_dur / max(seg_duration, 0.01), 3)
sync_scores.append(sync_ratio)
# Duration-planner calibration sample: this text length spoke
# for this long at natural rate. Keyed by stable seg id and
# merged into the per-language job map after the loop.
if strategy != "strict_slot" and seg.text.strip() and generated_dur > 0:
_natural_dur_records[str(seg_id)] = {
"chars": len(seg.text.strip()),
"dur": round(generated_dur, 4),
}
# Build the fingerprint now (cheap) but defer the disk write
# and job flush to the batch-write phase after the GPU loop.
_seg_fp = None
try:
# track_lang scopes the hash to THIS track (P1.3); the
# client-side recompute (/tools/incremental) sends the
# same code, so parity (#281 class) holds per language.
_seg_fp = segment_fingerprint({
"text": seg.text,
"target_lang": getattr(seg, "target_lang", None),
@@ -592,24 +710,24 @@ async def dub_generate(job_id: str, req: DubRequest):
"speed": getattr(seg, "speed", None),
"direction": getattr(seg, "direction", None),
"effect_preset": getattr(seg, "effect_preset", None),
})
}, track_lang=lang_code)
except Exception as e:
logger.debug("seg fingerprint skipped for %s: %s", seg_id, e)
_pending_seg_writes.append((i, _model.sampling_rate, seg_id, _seg_fp, _num_step))
_pending_seg_writes.append((i, backend.sample_rate, seg_id, _seg_fp, _num_step))
# RVC needs the WAV on disk, so write it immediately only
# when RVC is active (uncommon path).
if rvc_is_enabled():
seg_wav_path = dub_seg_path(job_id, seg_id)
atomic_save_wav(seg_wav_path, audio_tensor, _model.sampling_rate)
seg_wav_path = _seg_lang_path(seg_id)
atomic_save_wav(seg_wav_path, audio_tensor, backend.sample_rate)
try:
await loop.run_in_executor(_gpu_pool, apply_rvc, seg_wav_path)
rvc_wav, rvc_sr = torchaudio.load(seg_wav_path)
if rvc_sr == _model.sampling_rate:
if rvc_sr == backend.sample_rate:
audio_tensor = rvc_wav
target_samples = int(seg_duration * _model.sampling_rate)
target_samples = int(seg_duration * backend.sample_rate)
current_samples = audio_tensor.shape[-1]
if target_samples > current_samples:
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, target_samples - current_samples))
@@ -619,31 +737,32 @@ async def dub_generate(job_id: str, req: DubRequest):
yield f"data: {json.dumps({'type': 'warning', 'segment': i, 'message': f'RVC skipped: {str(e)[:120]}'})}\n\n"
# Watermark this FRESH TTS output exactly once, right before it
# is persisted. The same seg_<id>.wav is BOTH the downloadable
# per-segment file AND the assembly input for the final track,
# so marking it here (and nowhere else) gives the downloadable
# WAV its mark back and the final mix inherits it — no double-
# mark. Cached-reuse audio is already marked; silence/zero slots
# carry no speech to mark, so neither is re-watermarked.
audio_tensor = embed_watermark(audio_tensor, _model.sampling_rate)
# is persisted. The same seg_{lang}_{id}.wav is BOTH the
# downloadable per-segment file AND the assembly input for the
# final track, so marking it here (and nowhere else) gives the
# downloadable WAV its mark back and the final mix inherits it —
# no double-mark. Cached-reuse audio is already marked;
# silence/zero slots carry no speech to mark, so neither is
# re-watermarked.
audio_tensor = embed_watermark(audio_tensor, backend.sample_rate)
seg_wav_path = dub_seg_path(job_id, seg_id)
seg_wav_path = _seg_lang_path(seg_id)
try:
# Keep the existing per-segment WAV contract for previews
# and partial regeneration, but do not keep the tensor in RAM.
atomic_save_wav(seg_wav_path, audio_tensor, _model.sampling_rate)
atomic_save_wav(seg_wav_path, audio_tensor, backend.sample_rate)
except Exception as e:
logger.warning("seg write failed for %s: %s", seg_id, e)
# If the durable segment write fails, still preserve a mix
# copy so this generation can finish.
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, audio_tensor, _model.sampling_rate, f"mix_{seg_id}"))
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, audio_tensor, backend.sample_rate, f"mix_{seg_id}"))
try:
del audio_tensor
except Exception:
pass
_release_audio_tensors()
else:
all_segment_wavs.append((seg.start, seg.end, seg_wav_path, _model.sampling_rate))
all_segment_wavs.append((seg.start, seg.end, seg_wav_path, backend.sample_rate))
try:
del audio_tensor
except Exception:
@@ -651,7 +770,7 @@ async def dub_generate(job_id: str, req: DubRequest):
_release_audio_tensors()
except Exception as e:
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error': str(e)})}\n\n"
sr = _model.sampling_rate
sr = backend.sample_rate
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, torch.zeros(1, max(0, int(seg_duration * sr))), sr, f"mix_{seg_id}"))
sync_scores.append(1.0)
@@ -663,17 +782,31 @@ async def dub_generate(job_id: str, req: DubRequest):
# Per-segment WAVs were written during the loop to keep RAM bounded.
# Flush only lightweight fingerprints/quality metadata here.
_t_diskw_0 = time.perf_counter()
hashes = job.setdefault("seg_hashes", {})
# P1.3 — fingerprints live per language so each track's staleness is
# judged against ITS OWN last generate. The flat job["seg_hashes"] is
# kept as a mirror of the CURRENT track's map: every existing consumer
# (the `done` event, dub-history restore, older frontends) already
# treats it as "the hashes of the language generated last", which is
# exactly what it now provably contains.
hashes = _seg_hashes_by_lang(job).setdefault(lang_code, {})
quality_map = job.setdefault("seg_num_step", {})
for (_si, _sr, _sid, _fp, _nstep) in _pending_seg_writes:
if _fp is not None:
hashes[_sid] = _fp
quality_map[_sid] = _nstep
job["seg_hashes"] = dict(hashes)
# Duration-planner calibration: per-language (chars, natural dur)
# records. update() (not replace) so partial regens keep accumulating
# samples from earlier runs of this track.
if _natural_dur_records:
job.setdefault("seg_natural_durs_by_lang", {}).setdefault(
lang_code, {},
).update(_natural_dur_records)
# Single job flush instead of one per 8 segments.
_save_job(job_id, job)
_t_diskw = time.perf_counter() - _t_diskw_0
sr = _model.sampling_rate
sr = backend.sample_rate
slot_fit = (req.slot_fit or "time_stretch").lower()
overflow_budget_s = max(0.0, float(req.overflow_budget_s or 0.0))
@@ -759,7 +892,6 @@ async def dub_generate(job_id: str, req: DubRequest):
# not from the plan — so subtitles land exactly on the audio.
fitted_cues: list[dict] = []
lang_code = req.language_code or "und"
track_path = os.path.join(DUB_DIR, job_id, f"dubbed_{lang_code}.wav")
os.makedirs(os.path.dirname(track_path), exist_ok=True)
@@ -1049,8 +1181,12 @@ async def dub_generate(job_id: str, req: DubRequest):
job["dubbed_tracks"][lang_code]["fit_fp"] = fit_fp
# Record what kind of per-segment WAVs are on disk so a later
# smart_fit run knows whether partial regen / fit-only re-mix can
# reuse them ("natural") or must regen once ("slotted").
job["seg_wav_kind"] = "slotted" if strategy == "strict_slot" else "natural"
# reuse them ("natural") or must regen once ("slotted"). Per-track
# (P1.3) — each language renders under its own strategy; the flat
# field stays in lock-step for older readers.
_kind = "slotted" if strategy == "strict_slot" else "natural"
job.setdefault("seg_wav_kind_by_lang", {})[lang_code] = _kind
job["seg_wav_kind"] = _kind
_save_job(job_id, job)
_t_total = time.perf_counter() - _t_start
@@ -1098,7 +1234,13 @@ async def preview_segment(job_id: str, req: SegmentPreviewRequest):
if not job:
raise HTTPException(status_code=404, detail="Job not found")
_model = await get_model()
# See the /dub/generate/{job_id} resolution above (issue #312 class) —
# a segment preview resolves ref_audio from the same auto-clone /
# voice-profile sources, so it needs the same cloning-capable gate.
try:
backend = await resolve_generation_backend(require_cloning=True)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
def _gen():
ref_audio = None
@@ -1133,8 +1275,11 @@ async def preview_segment(job_id: str, req: SegmentPreviewRequest):
instruct_str = row["instruct"]
lang = req.language if req.language != "Auto" else None
audios = _model.generate(
text=req.text,
# Same normalization as the full dub render above, so a preview
# sounds exactly like the final segment. Pref-gated, never raises.
from services.text_normalization import normalize_for_tts
audio_out = backend.generate(
text=normalize_for_tts(req.text, lang),
language=lang,
ref_audio=ref_audio,
ref_text=ref_text,
@@ -1146,21 +1291,15 @@ async def preview_segment(job_id: str, req: SegmentPreviewRequest):
denoise=True,
postprocess_output=True,
)
audio_out = audios[0]
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(
audio_out,
sample_rate=getattr(_model, "sampling_rate", 24000),
)
return normalize_audio(mastered, target_dBFS=-2.0)
if not getattr(backend, "applies_own_mastering", False):
audio_out = apply_mastering(audio_out, sample_rate=backend.sample_rate)
return normalize_audio(audio_out, target_dBFS=-2.0)
# Bounded + pool-reset on hang so a wedged preview generate can't starve the
# GPU pool and brick the backend (#730 class).
audio_tensor = await run_on_gpu_pool_guarded(_gen, what="Dub preview generate")
sr = getattr(_model, "sampling_rate", 24000)
sr = backend.sample_rate
buf = io.BytesIO()
_safe_torchaudio_save(buf, audio_tensor, sr, format="wav")
buf.seek(0)
+235 -4
View File
@@ -9,7 +9,7 @@ from fastapi.responses import JSONResponse
from schemas.requests import TranslateRequest
from services.model_manager import _cpu_pool, _gpu_pool
from services.translator import cinematic_available, cinematic_refine_many, _cinematic_budget
from api.routers.dub_core import _get_job
from api.routers.dub_core import _get_job, _save_job
router = APIRouter()
logger = logging.getLogger("omnivoice.api")
@@ -202,6 +202,46 @@ def _resolve_source_lang(req: TranslateRequest) -> str:
return _guess_lang_from_text(getattr(req, "segments", None)) or "en"
def _resolve_translation_context(req, client, model_name: str, timeout: float,
src_lang: str) -> Optional[dict]:
"""Cached auto-glossary context (theme + terms) for this job/target.
Cache lives on the dub job dict (``job["translation_context"][target]``)
and persists through the existing ``job_data`` JSON blob via ``_save_job``
no schema change. A transcript fingerprint keys the cache so an edited
transcript re-extracts; an unchanged transcript costs zero LLM calls on
re-translate. Any failure returns None translation proceeds without
context, never fails because of it. Blocking; run in an executor.
"""
from services import translation_quality as tq
texts = [(s.text or "") for s in req.segments]
fp = tq.transcript_fingerprint(texts)
job = _get_job(req.job_id) if getattr(req, "job_id", None) else None
if job is not None:
cached = (job.get("translation_context") or {}).get(req.target_lang)
if isinstance(cached, dict) and cached.get("fingerprint") == fp:
return cached
ctx = tq.extract_context_sync(
client, model_name, timeout,
segment_texts=texts,
source_lang=src_lang,
target_lang=req.target_lang,
source_name=LANG_NAMES.get(src_lang, src_lang),
target_name=LANG_NAMES.get(req.target_lang, req.target_lang),
)
if ctx is None:
return None
ctx = {**ctx, "fingerprint": fp}
if job is not None:
try:
job.setdefault("translation_context", {})[req.target_lang] = ctx
_save_job(req.job_id, job)
except Exception: # noqa: BLE001 — persistence is best-effort
logger.debug("translation context persist skipped", exc_info=True)
return ctx
def _unload_nllb():
"""Release NLLB VRAM so TTS model can reload."""
global _nllb_model, _nllb_tokenizer
@@ -366,6 +406,27 @@ async def dub_translate(req: TranslateRequest):
)
return JSONResponse(status_code=400, content={"error": friendly})
from services import translation_quality as tq
# Two-stage quality toggles. None (old clients) = ON — an LLM
# translator is active on this branch by definition.
auto_glossary_on = req.auto_glossary if req.auto_glossary is not None else True
reflect_on = req.reflect if req.reflect is not None else True
# Stage 1 — auto-glossary: ONE pass over the full transcript for a
# theme summary + terminology map (cached per job/target/transcript),
# merged with the user's manual glossary (user entries win) and
# injected into every per-segment prompt below. With the toggle off
# the manual glossary still rides along — that costs no extra call.
auto_ctx = None
if auto_glossary_on:
auto_ctx = await loop.run_in_executor(
_cpu_pool, _resolve_translation_context,
req, client, model_name, llm_timeout, src_lang,
)
merged_terms = tq.merge_glossary(req.glossary, (auto_ctx or {}).get("terms"))
context_extra = tq.context_clause((auto_ctx or {}).get("theme", ""), merged_terms)
def _build_prompt(src_code: str, tgt_code: str) -> str:
"""Build a system prompt that resists hallucinations on small
local LLMs. Three things matter:
@@ -396,13 +457,19 @@ async def dub_translate(req: TranslateRequest):
dia_clause = ""
if req.dialect and str(req.dialect).lower().startswith(str(tgt_code).lower()[:2]):
dia_clause = dialect_clause(req.dialect)
return (
base = (
f"You are a professional dubbing translator. "
f"Translate the user's text from {src_name} into "
f"{tgt_name}.{script_clause}{dia_clause} "
f"Reply ONLY with the translated {tgt_name} text, do not "
f"add quotes, notes, headers, explanations, or commentary."
)
# Auto-glossary theme + merged terminology (user terms win) —
# every segment prompt carries the same brief, so recurring
# names/terms come out consistent across the whole dub.
if context_extra:
base = base + "\n\n" + context_extra
return base
def _translate_llm(seg):
if not seg.text or not seg.text.strip():
@@ -446,6 +513,33 @@ async def dub_translate(req: TranslateRequest):
seg.id, attempt + 1, last_err,
)
continue
# Stage 2 — reflect pass: critique→rewrite the direct
# translation into natural spoken dialogue. Returns None
# on ANY failure/timeout/divergence, in which case the
# direct translation stands — refinement can never fail
# a segment that already translated fine. The belt-and-
# braces except keeps that guarantee even if the helper
# itself ever raised: without it, the enclosing attempt
# handler would burn a retry on a segment that already
# translated successfully.
if reflect_on:
polished = None
try:
polished = tq.reflect_translation_sync(
client, model_name, llm_timeout,
source_text=seg.text,
direct_text=out_text,
source_lang=src_lang,
target_lang=tgt_code,
target_name=LANG_NAMES.get(tgt_code, tgt_code),
extra_clause=context_extra,
)
except Exception as e: # noqa: BLE001
logger.warning("reflect pass skipped for %s: %s",
seg.id, e)
if polished:
return {"id": seg.id, "text": polished,
"literal": out_text}
return {"id": seg.id, "text": out_text}
except Exception as e:
last_err = f"{type(e).__name__}: {e}"
@@ -624,6 +718,132 @@ async def dub_translate(req: TranslateRequest):
return JSONResponse(status_code=500, content={"error": str(e)})
def _stamp_duration_plan(rows, req) -> None:
"""Attach a pre-synthesis duration-plan verdict to every row (in place).
Pure planning (services/duration_planner.py): estimate the natural
speech duration of each row's FINAL text — self-calibrated from this
job's already-synthesized segments when possible — and classify it
against slot + borrowable gap using fit_planner's own caps. The verdict
rides on the row as ``plan`` so the segment table can badge tight/
impossible segments BEFORE any GPU time is spent. Informational only
generation is never blocked. Never raises.
"""
try:
from services.duration_planner import calibration_from_job, classify_segments
timed = [
s for s in req.segments
if getattr(s, "start", None) is not None and getattr(s, "end", None) is not None
]
if not timed:
return # old client — no timeline info, no plan
text_by_id = {str(r["id"]): (r.get("text") or "") for r in rows}
segs = sorted(
(
{
"id": str(s.id),
"start": float(s.start),
"end": float(s.end),
"text": text_by_id.get(str(s.id), ""),
}
for s in timed
),
key=lambda d: d["start"],
)
calib = None
total_dur = 0.0
if getattr(req, "job_id", None):
job = _get_job(req.job_id)
if job:
calib = calibration_from_job(job, req.target_lang)
total_dur = float(job.get("duration") or 0.0)
verdicts = {
v["id"]: v
for v in classify_segments(
segs, req.target_lang, calibration=calib, total_dur_s=total_dur,
)
}
for row in rows:
v = verdicts.get(str(row["id"]))
if v is None or row.get("error") or not (row.get("text") or "").strip():
continue
row["plan"] = {
"status": v["status"],
"est_dur_s": v["est_dur_s"],
"available_s": v["available_s"],
"est_overrun_s": v["est_overrun_s"],
"calibrated": v["calibrated"],
}
except Exception as e: # noqa: BLE001 — planning must never sink a translate
logger.debug("duration-plan stamping skipped: %s", e)
async def _apply_condense_pass(rows, req, loop) -> None:
"""Opt-in LLM condensation for ``impossible`` rows (in place).
Fans ``condense_for_slot`` out on the CPU pool under the same wall-clock
budget the cinematic phase uses, so a slow LLM can't hang the translate.
Suggestions land as ``plan.suggested_text`` the user applies them per
segment; the row's ``text`` is never touched here. Every failure mode
(no LLM, LLM error, divergent reply, budget) degrades to no suggestion.
"""
targets = [
row for row in rows
if (row.get("plan") or {}).get("status") == "impossible"
and (row.get("text") or "").strip() and not row.get("error")
]
if not targets:
return
try:
from services.duration_planner import calibration_from_job, condense_for_slot
calib = None
if getattr(req, "job_id", None):
job = _get_job(req.job_id)
if job:
calib = calibration_from_job(job, req.target_lang)
source_by_id = {str(s.id): s.text for s in req.segments}
sem = asyncio.Semaphore(int(os.environ.get("OMNIVOICE_LLM_CONCURRENCY", "6")))
async def _one(row):
async with sem:
res = await loop.run_in_executor(
_cpu_pool,
lambda: condense_for_slot(
row["text"],
available_s=float(row["plan"]["available_s"]),
target_lang=req.target_lang,
source_text=source_by_id.get(str(row["id"])),
calibration=calib,
),
)
if res.get("applied") and res.get("text"):
row["plan"]["suggested_text"] = res["text"]
row["plan"]["suggested_est_dur_s"] = res.get("est_dur_s")
tasks = [asyncio.ensure_future(_one(row)) for row in targets]
budget = _cinematic_budget()
done, pending = await asyncio.wait(
tasks, timeout=budget if budget and budget > 0 else None,
)
for task in pending:
task.cancel() # abandon the executor thread (#730 pattern)
for task in done:
exc = task.exception()
if exc is not None:
logger.warning("condense pass segment failed: %s", exc)
except Exception as e: # noqa: BLE001 — a suggestion pass must never sink a translate
logger.warning("condense pass skipped: %s", e)
async def _finalize_duration_plan(rows, req, loop) -> None:
"""Stamp plan verdicts on the FINAL row texts, then (opt-in) condense."""
_stamp_duration_plan(rows, req)
if getattr(req, "condense", False):
await _apply_condense_pass(rows, req, loop)
def _stamp_predicted_rate_ratio(translated, req) -> None:
"""Stamp a predicted ``rate_ratio`` on every row that has a known slot.
@@ -717,8 +937,10 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
"quality_used": "fast",
**_dialect_flags(req, applied=(already_llm and bool(dialect_hint)))}
# Fast (and anything unrecognised) returns the plain translation unchanged.
# Fast (and anything unrecognised) returns the plain translation unchanged
# (plus the pre-synthesis duration-plan badges — no LLM needed for those).
if quality not in ("cinematic", "autofit"):
await _finalize_duration_plan(translated, req, loop)
return base
source_by_id: dict[str, str] = {str(s.id): s.text for s in req.segments}
@@ -738,15 +960,19 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
if already_llm:
merged = []
for row in translated:
# A reflect-pass row already carries its pre-polish direct
# translation as `literal` — keep it instead of clobbering.
out = {"id": row["id"],
"text": row.get("text", "") or "",
"literal": row.get("text", "") or ""}
"literal": row.get("literal") or row.get("text", "") or ""}
if row.get("error"):
out["error"] = row["error"]
if "rate_ratio" in row:
out["rate_ratio"] = row["rate_ratio"]
merged.append(out)
await _apply_fit_pass(merged, req, slots_by_id, source_by_id, quality, loop, deadline)
# Plan AFTER the fit pass — verdicts must describe the final text.
await _finalize_duration_plan(merged, req, loop)
return {"translated": merged, "target_lang": req.target_lang,
"source_lang": src_lang, "quality_used": quality,
**_dialect_flags(req, applied=bool(dialect_hint))}
@@ -756,6 +982,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
if not cinematic_available():
logger.warning("%s requested but no LLM configured — returning Fast result.", quality)
base["cinematic_skipped"] = "no-llm-configured"
await _finalize_duration_plan(translated, req, loop)
return base
directions: dict[str, str] = {
@@ -774,6 +1001,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
pairs.append((seg_id, source_by_id.get(seg_id, ""), literal))
if not pairs:
await _finalize_duration_plan(translated, req, loop)
return base
refined = await cinematic_refine_many(
@@ -810,6 +1038,9 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
# Phase 4.4 speech-rate fit pass — now concurrent + bounded (see helper).
await _apply_fit_pass(merged, req, slots_by_id, source_by_id, quality, loop, deadline)
# Plan AFTER the fit pass — verdicts must describe the final text.
await _finalize_duration_plan(merged, req, loop)
return {
"translated": merged,
"target_lang": req.target_lang,
+24
View File
@@ -16,6 +16,7 @@ Environment variables (`OMNIVOICE_TTS_BACKEND`, `OMNIVOICE_ASR_BACKEND`,
a backend without Settings silently undoing it.
"""
import os
import re
import threading
from time import perf_counter
@@ -429,6 +430,11 @@ def engine_selftest(engine_id: str):
class SelectEngineRequest(BaseModel):
family: str # "tts" | "asr" | "llm"
backend_id: str
# Only meaningful for family="tts", backend_id="mlx-audio" (#981) — picks
# which of mlx-audio's curated models is actually loaded. A curated key
# ("kokoro") or a raw HF repo id ("mlx-community/Kokoro-82M-bf16") — the
# same tolerance MLXAudioBackend.__init__ already has. Ignored otherwise.
model_id: str | None = None
class SelectEngineResponse(BaseModel):
@@ -471,6 +477,24 @@ def select_engine(req: SelectEngineRequest):
f"Backend {req.backend_id} can't run on this machine: {why}. "
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
)
# #981: mlx-audio multiplexes 7+ curated models behind one backend id —
# persist the model pick alongside the backend id so the UI can actually
# select which curated model gets loaded (previously it always defaulted
# to Kokoro no matter what the user downloaded in Settings → Models).
if req.family == "tts" and req.backend_id == "mlx-audio" and req.model_id is not None:
known_keys = tts_backend.MLXAudioBackend.CURATED_MODELS
# Accept a curated key OR a raw HF repo id ("owner/name") — the same
# tolerance MLXAudioBackend.__init__ already has for power users.
# Anything else (typo'd key, malformed id) is rejected outright
# rather than silently persisted as a "custom repo" that then fails
# to resolve at load time.
if req.model_id not in known_keys and not re.fullmatch(r"[\w.-]+/[\w.-]+", req.model_id):
raise HTTPException(
400,
f"Unknown mlx-audio model: {req.model_id!r}. Expected one of "
f"{sorted(known_keys)} or a HF repo id like 'owner/name'.",
)
prefs.set_("mlx_audio_model_id", req.model_id)
prefs.set_(pref_key, req.backend_id)
return {
"family": req.family,
+247 -16
View File
@@ -12,6 +12,7 @@ import traceback
from typing import Optional
from fastapi import APIRouter, File, Form, UploadFile, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
import sqlite3
from core.db import db_conn, ensure_schema
@@ -105,8 +106,8 @@ def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering
``skip_mastering`` honors a backend's ``applies_own_mastering`` flag
(issue #312): studio engines (e.g. VoxCPM2's native 48 kHz output)
opt out of the broadcast Compressor + Reverb chain that's tuned for
OmniVoice's 24 kHz clone output. Loudness normalization still runs —
opt out of the broadcast highpass + Compressor pre-stage that's tuned
for OmniVoice's 24 kHz clone output. Loudness normalization still runs —
it's a benign peak scale. Mirrors ``_run_tts`` in openai_compat.py.
"""
from services.audio_dsp import (
@@ -143,6 +144,28 @@ def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering
return normalize_audio(audio_out, target_dBFS=-2.0)
def _safe_exc_text(e: BaseException) -> str:
"""``f"{type(e).__name__}: {e}"`` — the house style used for
unrecognized-error formatting throughout the backend (grep
``type(e).__name__`` in settings.py / asr_backend.py / model_manager.py
/ engines.py) with a guard against leaking a raw container repr.
#977: an AssertionError raised deep inside a vendored dependency
(mlx-audio's Kokoro pipeline) had ``.args`` shaped like
``('du', {'a': 'American English', ...})`` a tuple containing a dict.
``str(e)`` on that renders the WHOLE table straight into the user-facing
message. Any engine's ``generate()`` can raise something shaped like
this (not just Kokoro), so guard generically: if any element of
``e.args`` is a container rather than a plain string, don't interpolate
``str(e)`` at all name the exception type and point at the log
instead.
"""
args = getattr(e, "args", ())
if any(isinstance(a, (dict, list, tuple, set, frozenset)) for a in args):
return f"{type(e).__name__} — see Settings → Logs → Backend for details"
return f"{type(e).__name__}: {e}"
def _exception_chain(e):
"""Yield ``e`` plus every ``__cause__``/``__context__`` beneath it
(cycle-safe). Engines and hub libraries routinely wrap the original
@@ -412,7 +435,7 @@ def _oom_friendly_reraise(e):
raise RuntimeError(
f"TTS engine stopped mid-generation with an error OmniVoice doesn't "
f"recognize. Retry once; if it keeps failing, please report it with "
f"the full trace. Underlying error: {e}"
f"the full trace. Underlying error: {_safe_exc_text(e)}"
) from e
@@ -576,6 +599,32 @@ def _run_backend_inference(
_oom_friendly_reraise(e)
def _persist_profile_ref_text(profile_id: str, ref_text: str) -> None:
"""Cache an auto-transcribed reference transcript onto its profile row.
#1032 perf regression: profiles saved without a transcript re-ran a FULL
ASR model load + transcribe on every /generate (the #308 auto-transcribe
path). Persisting the first transcript makes subsequent generates read it
from the row like a user-entered one. The guarded UPDATE only ever fills
an empty column it can never overwrite a transcript the user typed or a
lock wrote and a failure is logged, never raised (best-effort, same
contract as the transcribe itself)."""
try:
with db_conn() as conn:
updated = conn.execute(
"UPDATE voice_profiles SET ref_text=? "
"WHERE id=? AND (ref_text IS NULL OR ref_text='')",
(ref_text, profile_id),
).rowcount
if updated:
event_bus.emit("profiles", {"action": "updated", "id": profile_id})
except Exception as e: # noqa: BLE001 — cache write must not break generate
logger.warning(
"could not persist auto-transcribed ref_text onto profile %s: %s",
profile_id, e,
)
@router.post("/generate")
async def generate_speech(
text: str = Form(...),
@@ -671,11 +720,51 @@ async def generate_speech(
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
# ── #1033/#1037: warm the engine under the LOAD budget, not the generate
# budget. A cold adapter lazily loads (and possibly downloads multi-GB
# weights) inside generate(), so a fresh install's first request burned
# its whole OMNIVOICE_GENERATE_TIMEOUT_S window on the download and died
# with a misleading "too heavy for the available compute" 503 (#1014
# measured it: 0% GPU util for the full 300s). Model loading gets its own,
# larger budget (OMNIVOICE_MODEL_LOAD_TIMEOUT, default 1200s) — the same
# split get_model() already has for the native engine. Once warm, this is
# a no-op per request.
if _backend is not None:
from services.model_manager import _model_load_timeout
try:
await run_on_gpu_pool_guarded(
_backend.ensure_ready,
what=f"TTS engine '{engine_id}' model load",
timeout=_model_load_timeout(),
)
# Builtin TimeoutError base, not GpuJobTimeoutError — reload-proof
# class identity (see the twin catch in openai_compat.py).
except TimeoutError as exc:
logger.warning("engine load exceeded the model-load budget: %s", exc)
raise HTTPException(
status_code=503,
detail=(
f"TTS engine '{engine_id}' did not finish loading within its "
f"model-load budget — on a first run this usually means the "
f"weight download is slow or stalled (check Settings → Models "
f"for progress), not that generation failed. Retry once the "
f"model shows as installed."
),
) from exc
ref_audio_path = None
cleanup_ref = False
used_seed = seed
resolved_profile_id = None
history_mode = None # profile.kind when a profile drives; else inferred at insert
# #1032: profile id to persist an auto-transcribed reference transcript to.
# Set only for a plain (unlocked) clone profile whose stored ref_text is
# empty — the case where every /generate re-ran a full ASR model load +
# transcribe of the same clip. Locked profiles are excluded (their ref
# audio is the locked take, and unlocking would leave a mismatched
# transcript paired with the original reference); design profiles are
# excluded (a re-render replaces the sample, stranding a stale transcript).
persist_ref_text_profile_id = None
if profile_id:
with db_conn() as conn:
@@ -721,6 +810,11 @@ async def generate_speech(
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
if not ref_text and row["ref_text"]:
ref_text = row["ref_text"]
elif ref_audio_path and not ref_text:
# Empty stored transcript → the auto-transcribe below will
# run; cache its result onto the profile so it runs ONCE,
# not on every generate (#1032 perf regression).
persist_ref_text_profile_id = profile_id
if not instruct and row["instruct"]:
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
@@ -770,6 +864,11 @@ async def generate_speech(
except GpuJobTimeoutError as e:
logger.warning("reference transcribe hung (%s); using model ASR fallback", e)
ref_text = None
# #1032: cache the transcript onto its clone profile so the ASR model
# load + transcribe above happens once per profile, not per generate.
# Only fills an empty column — a user-entered transcript always wins.
if ref_text and persist_ref_text_profile_id:
_persist_profile_ref_text(persist_ref_text_profile_id, ref_text)
# #526: materialize a concrete seed when none was supplied (and no profile
# pinned one) so the take is reproducible and we can hand it back via the
@@ -779,6 +878,15 @@ async def generate_speech(
if used_seed is None:
used_seed = random.randint(0, 2**31 - 1)
# Engine-agnostic text normalization (junk strip, numbers→words,
# abbreviations) — AFTER `language` is fully resolved, and BEFORE the
# pronunciation dictionary so user dictionary entries operate on
# normalized text and respellings are never re-mangled (ordering rationale
# in services/text_normalization.py). Pref-gated (default ON), idempotent,
# never raises; applied exactly once per request, at this choke point.
from services.text_normalization import normalize_for_tts
text = normalize_for_tts(text, language)
# Expressive-TTS Spec 01: apply the user pronunciation dictionary + inline
# [[…]] one-off overrides to the text, here — AFTER `language` is fully
# resolved (a profile may fill it above) so per-language entries match the
@@ -883,6 +991,13 @@ async def generate_speech(
logger.warning("history write still failed after schema heal; returning audio anyway: %s", e2)
except Exception as e:
logger.warning("generation history write failed; returning audio anyway: %s", e)
# Retention cap: without it, takes (rows + WAVs in OUTPUTS_DIR) grow
# unbounded forever. Best-effort — a prune failure must never affect
# the generation that just succeeded.
try:
_prune_history_over_cap()
except Exception as e: # noqa: BLE001
logger.warning("history retention prune failed (non-fatal): %s", e)
event_bus.emit("generation_history", {"action": "created", "id": audio_id})
buffer = io.BytesIO()
@@ -934,7 +1049,7 @@ async def generate_speech(
status_code=500,
detail=(
f"Couldn't synthesize audio. See Settings → Logs → Backend for the full trace. "
f"Underlying error: {e}"
f"Underlying error: {_safe_exc_text(e)}"
),
)
finally:
@@ -955,17 +1070,101 @@ def _safe_output_path(name):
return candidate
def _remove_wav_if_unreferenced(conn, audio_path, exclude_ids=()):
"""Delete a history WAV from OUTPUTS_DIR — but only when no *other*
generation_history row still references the same file.
History WAVs are uniquely owned by their row (lock/save-as-profile COPY
into VOICES_DIR, exports copy to the user's destination), so this guard is
normally a no-op it exists so any future path that duplicates a row can
never make a delete/prune yank audio out from under a surviving take."""
if not audio_path:
return
p = _safe_output_path(audio_path)
if not p or not os.path.exists(p):
return
placeholders = ",".join("?" for _ in exclude_ids)
others = conn.execute(
"SELECT COUNT(*) FROM generation_history WHERE audio_path=?"
+ (f" AND id NOT IN ({placeholders})" if exclude_ids else ""),
(audio_path, *exclude_ids),
).fetchone()[0]
if others:
return
with contextlib.suppress(OSError):
os.remove(p)
# How many takes to keep before pruning the oldest UNstarred ones (rows + their
# WAVs). User-tunable via Settings → Storage; 0 = unlimited. The pref key is
# shared with api/routers/settings.py (the GET/PUT endpoint) — same pattern as
# perf.torch_compile_disabled, which settings.py and engine_env.py both name.
HISTORY_CAP_PREF_KEY = "generation_history_cap"
DEFAULT_HISTORY_CAP = 200
def _history_cap() -> int:
from core import prefs
try:
cap = int(prefs.get(HISTORY_CAP_PREF_KEY, DEFAULT_HISTORY_CAP))
except (TypeError, ValueError):
return DEFAULT_HISTORY_CAP
return max(0, cap)
def _prune_history_over_cap() -> int:
"""Retention: keep the newest ``_history_cap()`` takes; delete the oldest
UNstarred rows over the cap plus their WAVs (via the unreferenced guard).
Starred takes are never pruned even when they alone exceed the cap.
Returns the number of rows pruned."""
cap = _history_cap()
if cap <= 0:
return 0 # 0 = unlimited
with db_conn() as conn:
total = conn.execute("SELECT COUNT(*) FROM generation_history").fetchone()[0]
excess = total - cap
if excess <= 0:
return 0
victims = conn.execute(
"SELECT id, audio_path FROM generation_history "
"WHERE COALESCE(starred, 0)=0 ORDER BY created_at ASC LIMIT ?",
(excess,),
).fetchall()
if not victims:
return 0
victim_ids = [r["id"] for r in victims]
conn.executemany(
"DELETE FROM generation_history WHERE id=?", [(i,) for i in victim_ids]
)
for r in victims:
_remove_wav_if_unreferenced(conn, r["audio_path"], exclude_ids=victim_ids)
logger.info("history retention: pruned %d takes over the %d cap", len(victims), cap)
return len(victims)
@router.get("/history")
def list_history():
"""Newest 50 generations whose audio still exists on disk.
"""The newest 50 generations plus every starred take, newest first, kept to
rows whose audio still exists on disk.
Rows whose WAV was deleted out-of-band (cleared outputs dir, manual
cleanup) used to come back anyway and render dead players that 404 on
every fetch; prune them here so the UI never sees them again."""
Starred takes ride along past the 50-row window so a keeper can never age
off the rail. Rows whose WAV was deleted out-of-band (cleared outputs dir,
manual cleanup) used to come back anyway and render dead players that 404
on every fetch; prune them here so the UI never sees them again."""
query = (
"SELECT * FROM generation_history WHERE COALESCE(starred, 0)=1 "
"OR id IN (SELECT id FROM generation_history ORDER BY created_at DESC LIMIT 50) "
"ORDER BY created_at DESC"
)
with db_conn() as conn:
rows = conn.execute(
"SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50"
).fetchall()
try:
rows = conn.execute(query).fetchall()
except sqlite3.OperationalError:
# Same class as #710/#552: a DB that missed init or the additive
# `starred` column. Heal once and retry inside this connection.
ensure_schema()
rows = conn.execute(query).fetchall()
alive, stale_ids = [], []
for r in rows:
p = _safe_output_path(r["audio_path"]) if r["audio_path"] else None
@@ -981,6 +1180,39 @@ def list_history():
logger.info("pruned %d stale history rows (audio file gone)", len(stale_ids))
return alive
class _StarBody(BaseModel):
starred: bool
@router.put("/history/{history_id}/starred")
def set_history_starred(history_id: str, body: _StarBody):
"""Star/unstar a take. Starred takes survive the retention cap and always
appear in GET /history regardless of the recency window."""
def _update():
with db_conn() as conn:
cur = conn.execute(
"UPDATE generation_history SET starred=? WHERE id=?",
(1 if body.starred else 0, history_id),
)
return cur.rowcount
try:
changed = _update()
except sqlite3.OperationalError as e:
# `no such column: starred` on a pre-migration DB (or the #710
# missing-table class) — heal the schema and retry once.
logger.warning("star update failed (%s); healing schema + retrying", e)
ensure_schema()
changed = _update()
if not changed:
raise HTTPException(
status_code=404,
detail="That take no longer exists — it may have been pruned or deleted.",
)
event_bus.emit("generation_history", {"action": "starred", "id": history_id})
return {"id": history_id, "starred": body.starred}
@router.delete("/history")
def clear_history():
with db_conn() as conn:
@@ -998,11 +1230,10 @@ def clear_history():
def delete_single_history(history_id: str):
with db_conn() as conn:
row = conn.execute("SELECT audio_path FROM generation_history WHERE id=?", (history_id,)).fetchone()
if row and row["audio_path"]:
p = _safe_output_path(row["audio_path"])
if p and os.path.exists(p):
with contextlib.suppress(OSError):
os.remove(p)
conn.execute("DELETE FROM generation_history WHERE id=?", (history_id,))
if row:
# Row first, file second — the WAV goes only if no surviving take
# still references it (see _remove_wav_if_unreferenced).
_remove_wav_if_unreferenced(conn, row["audio_path"], exclude_ids=(history_id,))
event_bus.emit("generation_history", {"action": "deleted", "id": history_id})
return {"deleted": True}
+67 -5
View File
@@ -108,6 +108,23 @@ class SpeechRequest(BaseModel):
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
)
# #1014: these two were silently DISCARDED before (pydantic ignores
# undeclared fields) — a 200 OK that quietly dropped the caller's quality
# knobs. Declared now and passed through, matching the native /generate
# form fields (defaults there: num_step=16, guidance_scale=2.0; the
# model's documented "quality" preset is num_step=32).
num_step: Optional[int] = Field(
default=None,
ge=1,
le=128,
description="OmniVoice extension: iterative unmasking steps (app default 16; 32 = the model's documented quality preset).",
)
guidance_scale: Optional[float] = Field(
default=None,
gt=0,
le=20,
description="OmniVoice extension: classifier-free guidance scale (app default 2.0).",
)
class TranscriptionResponse(BaseModel):
@@ -237,10 +254,10 @@ def _run_tts(backend, text: str, kw: dict):
sr = backend.sample_rate
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
# That chain's Compressor + 8% Reverb is tuned for OmniVoice's 24 kHz clone
# output; applied to a studio engine it adds an audible level pump and a
# reverb tail that degrade the very output we want clean. Loudness
# normalisation still runs — it's a benign peak scale, not dynamics.
# That chain's highpass + Compressor is tuned for OmniVoice's 24 kHz clone
# output; applied to a studio engine it adds an audible level pump that
# degrades the very output we want clean. Loudness normalisation still
# runs — it's a benign peak scale, not dynamics.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr)
wav = normalize_audio(wav, target_dBFS=-2.0)
@@ -274,6 +291,10 @@ async def create_speech(req: SpeechRequest):
kw["chunk_duration"] = req.chunk_duration
if req.chunk_threshold is not None:
kw["chunk_threshold"] = req.chunk_threshold
if req.num_step is not None:
kw["num_step"] = req.num_step
if req.guidance_scale is not None:
kw["guidance_scale"] = req.guidance_scale
if req.language:
kw["language"] = req.language
if req.instruct:
@@ -311,11 +332,52 @@ async def create_speech(req: SpeechRequest):
# Not a profile ID — might be a KittenTTS preset or similar
kw["voice"] = voice
# Engine-agnostic text normalization (junk strip, numbers→words,
# abbreviations) at this route's text→engine choke point — the same
# pre-pass as /generate, applied exactly once per request. `req.language`
# is everything this route knows about the language (None → universal
# safety filters only). Pref-gated (default ON), idempotent, never raises.
from services.text_normalization import normalize_for_tts
text = normalize_for_tts(req.input, req.language)
# ── #1033/#1037/#1014: warm the engine under the LOAD budget before the
# generate clock starts. The T4 verification (#1014) measured a fresh
# install's first /v1/audio/speech burning its whole 300s generate budget
# on the multi-GB checkpoint download (0% GPU util throughout) and dying
# with a misleading "too heavy for the available compute" error. Model
# loading gets OMNIVOICE_MODEL_LOAD_TIMEOUT (default 1200s); once warm
# this is a per-request no-op.
from services.model_manager import _model_load_timeout
try:
await run_on_gpu_pool_guarded(
backend.ensure_ready,
what=f"TTS engine '{backend.id}' model load",
timeout=_model_load_timeout(),
)
# Catch the BUILTIN TimeoutError base, not GpuJobTimeoutError by name:
# several tests reload services.model_manager mid-suite, so a class
# imported at call time can differ in identity from the one the guard
# (bound at this module's import) actually raises — the except would
# silently miss. The builtin base has one identity forever. (Caught by
# this exact test failing CI-only, in full-suite order.)
except TimeoutError as e:
logger.warning("engine load exceeded the model-load budget: %s", e)
raise HTTPException(
status_code=503,
detail=(
f"TTS engine '{backend.id}' did not finish loading within its "
f"model-load budget — on a first run this usually means the weight "
f"download is slow or stalled (check Settings → Models for "
f"progress), not that generation failed. Retry once the model "
f"shows as installed."
),
) from e
try:
# Bounded + pool-reset on hang so a wedged TTS request can't starve the
# GPU pool and brick the backend (#730 class).
wav, sr = await run_on_gpu_pool_guarded(
lambda: _run_tts(backend, req.input, kw), what="OpenAI TTS generate")
lambda: _run_tts(backend, text, kw), what="OpenAI TTS generate")
except Exception as e:
logger.exception("OpenAI TTS failed: %s", e)
raise HTTPException(status_code=500, detail=str(e))
+11
View File
@@ -73,6 +73,17 @@ async def create_profile(
raise ValueError("not an object")
except ValueError:
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
# Root-cause close for #983: a design profile must never be PERSISTED
# with a partial vd_states shape, regardless of which client (older
# frontend build, hand-edited payload, third-party API caller) created
# it — a missing category key crashes DesignMethodPanel's render on
# every future client that selects this profile. CATEGORY_ORDER is the
# same single source of truth the frontend's CATEGORIES keys mirror
# (core/describe_voice.py), so this can't drift from the picker.
from core.describe_voice import CATEGORY_ORDER
for _cat in CATEGORY_ORDER:
parsed.setdefault(_cat, "Auto")
vd_states = _json.dumps(parsed)
# An all-Auto design (every category left on "Auto") yields an empty
# instruct — that's still a valid, saveable voice: synthesis falls back
# to neutral instruct-only conditioning (see generation.py design path).
+98 -2
View File
@@ -124,6 +124,46 @@ def set_torch_compile_disabled(body: _TorchCompileBody):
return _torch_compile_state()
# ── Generation-history retention (Studio takes rail) ──────────────────────
class _HistoryRetentionBody(BaseModel):
cap: int = Field(
...,
ge=0,
le=100000,
description="Max takes kept before the oldest UNstarred ones (rows + WAVs) are pruned; 0 = unlimited",
)
def _history_retention_state() -> dict:
from api.routers.generation import DEFAULT_HISTORY_CAP, _history_cap
return {"cap": _history_cap(), "default": DEFAULT_HISTORY_CAP}
@router.get("/history-retention")
def get_history_retention():
"""Current generation-history retention cap (Settings → Storage)."""
return _history_retention_state()
@router.put("/history-retention")
def set_history_retention(body: _HistoryRetentionBody):
"""Persist the retention cap. Enforced after every generation: the oldest
unstarred takes over the cap are pruned (rows + their audio files);
starred takes are never pruned. 0 disables pruning entirely."""
from core import prefs
from api.routers.generation import HISTORY_CAP_PREF_KEY
try:
prefs.set_(HISTORY_CAP_PREF_KEY, int(body.cap))
except Exception:
logger.exception("set_history_retention failed")
raise HTTPException(status_code=500, detail="Failed to persist setting")
return _history_retention_state()
# ── Dictation refinement (parity program Wave 2.1 / Spec 3 phase 2) ───────
@@ -282,7 +322,8 @@ def save_llm_provider(provider_id: str, body: _LLMProviderBody):
A None field is left unchanged; an empty api_key clears the stored key.
"""
from services import llm_providers
if llm_providers.get_provider(provider_id) is None:
p = llm_providers.get_provider(provider_id)
if p is None:
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
if body.api_key is not None:
llm_providers.save_key(provider_id, body.api_key.strip())
@@ -290,7 +331,17 @@ def save_llm_provider(provider_id: str, body: _LLMProviderBody):
provider_id, base_url=body.base_url, model=body.model,
account_id=body.account_id,
)
if body.make_active:
# An explicit save also claims the active slot when the user has never
# chosen a provider (#963). Without this, a saved-and-tested local
# provider (Ollama/LM Studio) evaporates on restart: active_provider_id()
# deliberately excludes local providers from auto-select, so the plain
# "Save" left nothing persisted to resolve. Gated on the STORED selection
# only — an explicit prior choice is never stolen by a plain save, and an
# unconfigured provider can't claim the slot.
if body.make_active or (
llm_providers.stored_active_provider_id() is None
and llm_providers.is_configured(p)
):
llm_providers.set_active_provider(provider_id)
return list_llm_providers()
@@ -739,6 +790,51 @@ def set_hf_mirror(body: _HFMirrorBody):
return {"configured": url, "restart_required": changed, "presets": _HF_MIRROR_PRESETS}
# ── OpenAI-compatible remote ASR (#877) ─────────────────────────────────────
# A path to Qwen3-ASR/FunASR/SenseVoice — or OpenAI's own Whisper API — today,
# without waiting on transformers to ship a direct Qwen3-ASR integration.
# base_url/model are plain settings_store text rows; the key is encrypted via
# settings_store.set_secret — same convention as /llm-providers, never
# returned to the client, '' clears it, omitted/None leaves it unchanged.
class _ASROpenAICompatBody(BaseModel):
base_url: str | None = None
model: str | None = None
api_key: str | None = Field(None, description="'' clears it, None leaves unchanged")
@router.get("/asr-openai-compat")
def get_asr_openai_compat():
from services import asr_backend
return {
"base_url": asr_backend.resolve_openai_compat_asr_base_url(),
"model": asr_backend.resolve_openai_compat_asr_model(),
"has_key": asr_backend.openai_compat_asr_has_key(),
}
@router.put("/asr-openai-compat")
def set_asr_openai_compat(body: _ASROpenAICompatBody):
from services import asr_backend, settings_store
if body.base_url is not None:
url = body.base_url.strip().rstrip("/")
if url and not url.startswith(("http://", "https://")):
raise HTTPException(status_code=400, detail="Base URL must start with http(s)://")
settings_store.set_text(asr_backend._ASR_OPENAI_COMPAT_BASE_URL_KEY, url)
if body.model is not None:
settings_store.set_text(
asr_backend._ASR_OPENAI_COMPAT_MODEL_KEY, body.model.strip() or "whisper-1"
)
if body.api_key is not None:
settings_store.set_secret(
asr_backend._ASR_OPENAI_COMPAT_SECRET_NAME, body.api_key.strip()
)
return get_asr_openai_compat()
# ── Updates panel: shipped changelog + pre-migration DB backup state ────────
# (feat/safe-updates). Both are read-only, local-first surfaces for
# Settings → Updates: the "What's new" viewer reads the CHANGELOG.md that
+5 -3
View File
@@ -530,14 +530,16 @@ async def install_model(req: InstallModelRequest):
_install_cooldowns[req.repo_id] = _time_fail.time()
# #874: when the install failed because the configured HF mirror is
# unreachable, name the mirror + the setting instead of leaking the
# raw connectivity error. No-op for every other failure.
from core.failure import append_hf_mirror_hint
# raw connectivity error. #959: likewise for the SOCKS-proxy class
# (missing socksio fails the download's session construction).
# No-op for every other failure.
from core.failure import append_hint
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_error",
"error": append_hf_mirror_hint(str(e)),
"error": append_hint(str(e)),
})
finally:
_cancelled.discard(req.repo_id)
+68 -10
View File
@@ -168,16 +168,40 @@ def _detect_gpu() -> dict:
return info
def _probe_network(host: str = "huggingface.co", timeout: float = 2.0) -> bool:
def _probe_network(host: str = "huggingface.co", port: int = 443, timeout: float = 2.0) -> bool:
"""Tiny TCP connect test."""
import socket
try:
with socket.create_connection((host, 443), timeout=timeout):
with socket.create_connection((host, port), timeout=timeout):
return True
except Exception:
return False
def _hf_endpoint_host() -> tuple[str, int]:
"""Host/port of the Hugging Face endpoint actually in effect.
Mirror-aware: restricted-network users (e.g. behind the Great Firewall)
point HF_ENDPOINT at a mirror via Settings Models Hugging Face
mirror. Probing hardcoded huggingface.co would fail them even when their
configured mirror works fine.
"""
try:
from core.failure import configured_hf_mirror
mirror = configured_hf_mirror()
except Exception:
mirror = ""
if mirror:
try:
from urllib.parse import urlsplit
u = urlsplit(mirror)
if u.hostname:
return u.hostname, u.port or (80 if u.scheme == "http" else 443)
except Exception:
pass
return "huggingface.co", 443
def _ram_gb() -> float:
try:
import psutil
@@ -400,15 +424,49 @@ def preflight():
"status": r_status, "detail": r_detail, "fix": r_fix,
})
# ── Network
net_ok = _probe_network()
# ── Network — probes the HF endpoint actually in effect (mirror-aware),
# and a dead network is a WARNING, not a blocker. The app is local-first:
# already-downloaded models work offline, and a hard fail here dead-ends
# restricted-network users (e.g. China, where huggingface.co is blocked)
# on the very first screen — before they can reach the mirror setting
# that fixes it. Model downloads surface their own actionable errors.
net_host, net_port = _hf_endpoint_host()
net_ok = _probe_network(net_host, net_port)
mirror_reachable = False
if not net_ok and net_host == "huggingface.co":
# Official endpoint blocked — if the community mirror is reachable,
# tell the user exactly which switch unblocks them.
mirror_reachable = _probe_network("hf-mirror.com")
if net_ok:
net_fix = None
elif mirror_reachable:
net_fix = (
"huggingface.co is blocked on this network, but the hf-mirror.com "
"community mirror is reachable — apply it below and re-check. "
"Model downloads will use the mirror immediately."
)
elif net_host != "huggingface.co":
net_fix = (
f"Your configured Hugging Face mirror ({net_host}) is unreachable "
"— it may be down or blocked. Pick another mirror or the official "
"endpoint below, or continue offline: models already downloaded "
"keep working."
)
else:
net_fix = (
"Check internet connection, VPN, or corporate firewall whitelist "
"for huggingface.co. You can continue — models already downloaded "
"keep working offline; new downloads need a connection or a "
"mirror (configurable below)."
)
checks.append({
"id": "network", "label": "Network (huggingface.co)",
"status": "pass" if net_ok else "fail",
"detail": "Reachable" if net_ok else "Unreachable on port 443",
"fix": None if net_ok else
"Check internet connection, VPN, or corporate firewall "
"whitelist for huggingface.co.",
"id": "network", "label": f"Network ({net_host})",
"status": "pass" if net_ok else "warn",
"detail": "Reachable" if net_ok else f"Unreachable on port {net_port}",
"fix": net_fix,
# Frontend affordance hint: the wizard offers the mirror quick-pick
# when the endpoint is unreachable (PreflightCheck allows extras).
"mirror_reachable": mirror_reachable,
})
# Aggregate
+5
View File
@@ -77,6 +77,10 @@ async def probe(req: ProbeReq):
class IncrementalReq(BaseModel):
segments: list[dict]
stored_hashes: Optional[dict[str, str]] = None
# P1.3 — the ACTIVE track's language code. When set, fingerprints are
# scoped to that language (pass that language's stored hashes alongside);
# omitted → legacy language-agnostic hashing, kept for old callers.
lang: Optional[str] = None
@router.post("/tools/incremental")
@@ -84,6 +88,7 @@ def plan_incremental(req: IncrementalReq):
return incremental.plan_incremental(
req.segments,
stored_hashes=req.stored_hashes or {},
track_lang=req.lang,
)
+16 -1
View File
@@ -136,7 +136,10 @@ async def ws_tts(websocket: WebSocket):
kw["emo_text"] = data["emo_text"]
if data.get("emo_audio"):
kw["emo_audio"] = data["emo_audio"]
if data.get("emo_alpha") != 1.0:
# Default 1.0 when absent: a missing key must not trip the
# `!= 1.0` branch into a KeyError (any minimal request that
# omitted emo_alpha got an error frame instead of audio).
if data.get("emo_alpha", 1.0) != 1.0:
kw["emo_alpha"] = data["emo_alpha"]
# Resolve voice profile
@@ -168,6 +171,18 @@ async def ws_tts(websocket: WebSocket):
except Exception:
kw["voice"] = voice
# Engine-agnostic text normalization (junk strip,
# numbers→words, abbreviations) — the same pre-pass as
# /generate, applied exactly ONCE per request, on the whole
# text BEFORE the sentence chunker fans it out (so per-sentence
# generates never re-normalize, and expanded abbreviations
# can't confuse the sentence splitter). The request's
# `language` is all this route knows (None → universal safety
# filters only). Pref-gated (default ON), idempotent, never
# raises.
from services.text_normalization import normalize_for_tts
text = normalize_for_tts(text, data.get("language"))
# Wave 1.4: split the request into sentences so the first
# sentence's audio streams while later sentences are still
# synthesizing — this is the time-to-first-audio win. The
+1
View File
@@ -70,6 +70,7 @@ _BASE_SCHEMA = """
duration_seconds REAL,
generation_time REAL,
seed INTEGER DEFAULT NULL,
starred INTEGER DEFAULT 0,
created_at REAL,
FOREIGN KEY (profile_id) REFERENCES voice_profiles(id)
);
+64
View File
@@ -41,9 +41,12 @@ _HINTS: dict[str, str] = {
"COMPUTE_TYPE_UNSUPPORTED": "Your GPU doesn't support float16 — OmniVoice retried on int8. If transcription still fails, set OMNIVOICE/ASR_COMPUTE_TYPE=int8 or use CPU.",
"TRANSFORMERS_IMPORT": "Your transformers install is incomplete. Reinstall it (`uv pip install --reinstall transformers`) or switch ASR to faster-whisper (Settings → Models).",
"OS_INVALID_ARGUMENT": "The OS rejected a file operation (Errno 22 / invalid argument) — in the transcribe path this is the temporary WAV write before ASR. It's almost always the temp directory: missing, read-only, on a full or removed drive, or blocked by antivirus. Check that your system TEMP/TMP folder exists and is writable and the drive has free space (add an OmniVoice antivirus exclusion if you use one), then retry.",
"SOCKS_PROXY_SUPPORT_MISSING": "A SOCKS proxy is configured in your environment (ALL_PROXY/HTTPS_PROXY=socks5://…) and the backend's HTTP client is missing SOCKS support. Newer OmniVoice builds ship SOCKS support (the socksio package) — update the app. If you still see this, unset ALL_PROXY/HTTPS_PROXY for OmniVoice, or run `uv pip install 'httpx[socks]'` in the backend venv, then restart.",
"SSL_HANDSHAKE_FAILURE": "A corporate or antivirus proxy is intercepting HTTPS traffic and re-signing certificates with its own CA — your OS trusts that CA, but Python's bundled certifi CA list doesn't, so the TLS handshake fails even though the connection reached the server. Newer OmniVoice builds trust the OS certificate store at startup (the truststore package), which should already fix this — update the app and retry. If you still see this, add an HTTPS-scanning exclusion for OmniVoice/Python in your antivirus, or ask IT for the proxy's CA bundle and set SSL_CERT_FILE to it, then restart.",
"UNSUPPORTED_VIDEO_URL": "This link isn't a directly downloadable video. Paste a direct video page (e.g. a youtube.com/watch?v=… or douyin.com/video/<id> link), not a share/profile/feed link — or download the file and drop it in directly.",
"VIDEO_DOWNLOAD_NETWORK": "The connection to the video server dropped mid-download (often a transient CDN/network blip or a regional rate-limit). Just retry — OmniVoice already cleaned up the partial download. If it keeps failing, check your network/VPN.",
"BROKEN_VENV": "The Python backend environment was moved or damaged. OmniVoice rebuilds it automatically on the next launch; if it keeps failing, use Clean & Retry on the setup screen.",
"MODEL_CACHE_CORRUPT": "The model cache had broken file links — snapshot entries that no longer point at their downloaded data (interrupted renames or antivirus interference can cause this). OmniVoice repairs this automatically and retries the load once. If the error persists, quit OmniVoice, delete the model's models--<org>--<name> folder inside the Hugging Face cache, and restart — the model re-downloads automatically.",
# HF_MIRROR_UNREACHABLE has a DYNAMIC hint (it names the configured mirror)
# — see hf_mirror_hint(); build_failure special-cases it.
}
@@ -168,6 +171,34 @@ def append_hf_mirror_hint(text: str) -> str:
return f"{text}{hint}" if hint else text
# Classes whose hint is safe to attach on the CONTEXT-FREE surfaces (the
# global 500 handler in main.py, the model-install SSE in setup/download.py),
# where all we have is a raw error string with no stage. Only classes whose
# classify() trigger is unmistakable belong here — e.g. VIDEO_DOWNLOAD_NETWORK
# must NOT be added: its bare "timed out" trigger would stamp a "video server"
# hint on a model-load timeout that leaks through the 500 handler.
_CONTEXT_FREE_HINT_CLASSES = frozenset({
"SOCKS_PROXY_SUPPORT_MISSING",
"SSL_HANDSHAKE_FAILURE",
})
def append_hint(text: str) -> str:
"""``"{text}{hint}"`` for raw-string surfaces (the global 500 handler,
the model-install SSE): the dynamic mirror hint (#874) when that class
applies, else a context-free static class hint (#959). ``text`` unchanged
otherwise a no-op for every other error. Never raises."""
try:
hint = hf_mirror_hint(text)
if not hint:
topic = classify(text)
if topic in _CONTEXT_FREE_HINT_CLASSES:
hint = _HINTS.get(topic, "")
except Exception:
return text
return f"{text}{hint}" if hint else text
def classify(reason: str) -> str:
"""Map a failure reason to a docs-taxonomy key, or "" when unknown.
@@ -201,6 +232,17 @@ def classify(reason: str) -> str:
# transformers + site-packages markers, which this signature lacks.
if "errno 22" in low:
return "OS_INVALID_ARGUMENT"
# An HF cache whose snapshot entries don't resolve (dangling symlinks /
# zero-byte stand-ins): transformers reports the weights missing ("does
# not appear to have a file named pytorch_model.bin or model.safetensors")
# even though the blobs are fully on disk. model_manager self-heals this
# (delete broken entries → snapshot_download → retry once); the class here
# covers both the raw transformers wording (any load surface can leak it)
# and OmniVoice's own repair messages, so the user-facing error and the
# auto bug report name the class and its automatic repair.
if ("does not appear to have a file named" in low
or "broken file link" in low):
return "MODEL_CACHE_CORRUPT"
if (
"could not import module" in low
or "autofeatureextractor" in low
@@ -220,6 +262,28 @@ def classify(reason: str) -> str:
)
):
return "TRANSFORMERS_IMPORT"
# #959: httpx raises ImportError AT CLIENT CONSTRUCTION ("Using SOCKS
# proxy, but the 'socksio' package is not installed. Make sure to install
# httpx using `pip install httpx[socks]`.") when ALL_PROXY/HTTPS_PROXY is
# socks5:// and socksio isn't importable. It surfaced from
# huggingface_hub's get_session() inside model load — a bare 500 on
# /generate with no next step. Checked BEFORE the HF-auth/mirror rules so
# a message that also carries HF wording still names this class.
if "socks proxy" in low or "socksio" in low:
return "SOCKS_PROXY_SUPPORT_MISSING"
# #976: a TLS handshake failing AFTER the TCP connection succeeds — the
# signature of a corporate/antivirus proxy that TLS-inspects traffic and
# re-signs certificates with a CA the OS trusts but Python's bundled
# certifi list doesn't (a different failure mode from #984's TCP-level
# "can't reach the host at all"). Requires "ssl" plus a handshake/cert-
# verify marker so a generic connection error isn't mislabelled.
if "ssl" in low and (
"handshake" in low
or "certificate verify failed" in low
or "sslv3_alert" in low
or "sslcertverificationerror" in low
):
return "SSL_HANDSHAKE_FAILURE"
if ("huggingface" in low or "hf_token" in low or "401" in low or "unauthorized" in low) and (
"token" in low or "auth" in low or "401" in low or "unauthorized" in low
):
+1 -1
View File
@@ -24,7 +24,7 @@ from pathlib import Path
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
# release.yml's version-bump job, so it stays equal to
# pyproject/tauri.conf/Cargo/package.json.
_FALLBACK_VERSION = "0.3.10"
_FALLBACK_VERSION = "0.3.16"
def _fallback_version() -> str:
+1
View File
@@ -70,6 +70,7 @@ class Supertonic3Backend(SubprocessBackend):
id = "supertonic3"
display_name = "Supertonic-3 (31 langs, CPU ONNX, 7 preset voices, OpenRAIL-M)"
supports_voice_design = False # preset voices only
supports_cloning = False # preset voices only; generate() never reads ref_audio
# TTS-04: honest hardware reporting. Supertonic-3 has no CUDA / MPS
# path in the SDK ‑‑ ONNX Runtime CPU EP only.
gpu_compat: tuple[str, ...] = ("cpu",)
+96 -14
View File
@@ -139,6 +139,24 @@ os.environ.setdefault("HF_HUB_DISABLE_XET", "1")
os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "15")
os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "30")
# ── OS trust store for TLS (#976) ───────────────────────────────────────────
# Users behind a corporate/antivirus proxy that TLS-inspects HTTPS traffic get
# a raw "[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] ssl/tls alert handshake failure"
# on every model install — the TCP connection succeeds (a different failure
# mode from #984's TCP-level blocked-host case), but the proxy re-signs the
# certificate with its own root CA, which the OS trusts (Windows CryptoAPI/
# SChannel) and Python's bundled `certifi` CA list does not. `inject_into_ssl`
# patches `ssl.SSLContext` process-wide to verify against the OS trust store
# instead, which is the actual fix (not just a nicer error message). Must run
# here — at MODULE level, before huggingface_hub/requests/httpx do any network
# I/O — not inside lifespan(), which runs too late. Not platform-gated: it's a
# correctness improvement everywhere. Best-effort: never block startup.
try:
import truststore
truststore.inject_into_ssl()
except Exception:
pass
# Prevent torchaudio from lazy-importing torchcodec (broken on some installs).
# Proper fix = exclude torchcodec in pyproject.toml; this is a belt-and-braces guard.
@@ -155,6 +173,16 @@ from logging.handlers import RotatingFileHandler
# written to prefs.json so they survive backend restarts. Read them back
# here — before any user code reads os.environ — so the values are available
# from startup.
#
# Legacy (≤v0.3.7) Translation-LLM rows (env.TRANSLATE_*) must migrate into
# the custom LLM provider's settings store BEFORE the re-import below — once
# TRANSLATE_BASE_URL lands in os.environ it hijacks the LLM provider
# selection for the whole session (#963). Real env vars are untouched.
try:
from services.llm_providers import migrate_legacy_translate_prefs
migrate_legacy_translate_prefs()
except Exception:
pass # never block startup on the migration; it retries next launch
_PERSISTED_ENV_PREFIX = "env."
try:
from core.prefs import _load as _load_all_prefs
@@ -476,6 +504,35 @@ async def _start_mcp_session_manager(session_manager, *, timeout: float):
return task, stop, mounted
async def _cancel_and_await_tasks(*tasks, timeout: float = 3.0) -> None:
"""Cancel each background task and give it a bounded chance to actually
finish before shutdown proceeds ``None`` entries are skipped (a task
that's conditionally created, e.g. ``capture_preload_task``, may not
exist).
``task.cancel()`` alone is not enough for a task awaiting
``run_in_executor()``: once the underlying OS thread is inside blocking
native/import work, cancellation can't stop it, so cancel-and-move-on lets
shutdown finish while that thread is still running invisible to
asyncio, but very much alive when the interpreter starts tearing down
module state under it (#1000 class). Awaiting with a bound (instead of
just cancelling) gives an early-stage task a real chance to exit cleanly
first; a task that's genuinely still deep in blocking work times out here
same as before, and the caller's own GPU-pool reset handles that case.
"""
for t in tasks:
if t is None:
continue
t.cancel()
for t in tasks:
if t is None:
continue
try:
await asyncio.wait_for(t, timeout=timeout)
except (asyncio.CancelledError, asyncio.TimeoutError):
pass
@asynccontextmanager
async def lifespan(app: FastAPI):
# Startup watchdog (#632): a silent hang during startup (e.g. a model-load /
@@ -550,6 +607,7 @@ async def lifespan(app: FastAPI):
# lean and the first dictation is instant instead of a cold model load.
# OMNIVOICE_PRELOAD_CAPTURE_ASR=0 opts out; the warm-up is also skipped
# under 4 GB free RAM (checked at warm time, not boot time).
capture_preload_task = None # only assigned when the preload actually runs (#1000 class)
if _env_flag("OMNIVOICE_PRELOAD_CAPTURE_ASR", default=True):
async def _preload_capture_asr():
await asyncio.sleep(_capture_preload_delay_s())
@@ -618,14 +676,33 @@ async def lifespan(app: FastAPI):
pass
except Exception:
pass
idle_task.cancel()
worker_task.cancel()
# Wait for tasks to finish their current iteration
for t in (idle_task, worker_task):
try:
await asyncio.wait_for(t, timeout=3.0)
except (asyncio.CancelledError, asyncio.TimeoutError):
pass
# preload_task/capture_preload_task matter most here (#1000 class): a quit
# mid-preload used to fall straight through to "Shutdown: done." while the
# model load was still running on a GPU-pool thread — cancel() can't stop
# a thread already inside blocking import/load work, so the process
# reported a clean exit while that background thread was still mid-
# `import transformers`, and got torn down by interpreter finalization
# instead. That surfaced as a misleading "Could not import module
# 'AutoFeatureExtractor'" — transformers' own generic lazy-import wrapper,
# not a real dependency problem. Awaiting here lets an early-stage load
# (still importing, not yet mid weight-download) finish cleanly before we
# report done; a load that's genuinely deep into a multi-GB download still
# times out — _reset_gpu_pool() below abandons it either way.
#
# 20s, not the original 3s (code-review finding post-merge): a cold
# transformers import alone can take longer than 3s on a slow disk or a
# first-ever launch, so the original bound left a real residual window —
# cancellation detaches the asyncio task, but the underlying OS thread
# keeps running past it, and shutdown could still report "done" while
# that thread was alive. Python cannot forcibly kill a running thread, so
# no finite bound eliminates this outright — 20s just shrinks the window
# from "any preload" to "an unusually slow cold-import," which is the
# practical ceiling before a longer shutdown itself becomes the
# complaint. A thread that's still running past 20s was never going to
# finish in a shutdown-appropriate timeframe regardless.
await _cancel_and_await_tasks(
idle_task, worker_task, preload_task, capture_preload_task, timeout=20.0,
)
# Unload the model and free GPU memory
try:
import services.model_manager as mm
@@ -633,6 +710,10 @@ async def lifespan(app: FastAPI):
mm.model = None
logger.info("Shutdown: model unloaded.")
mm.free_vram()
# Abandon a still-running preload's GPU-pool thread (Python can't kill
# a thread mid blocking call) so it can't outlive this shutdown block
# holding a reference into module state that's about to be torn down.
mm._reset_gpu_pool()
except Exception:
pass
# Run GC to release any remaining references
@@ -718,13 +799,14 @@ async def global_exception_handler(request: Request, exc: Exception):
# #874: a model download that failed because the CONFIGURED Hugging Face
# mirror (HF_ENDPOINT) is unreachable used to leak the raw transformers
# message ("We couldn't connect to 'https://hf-mirror.com' …") as the 500
# detail with no next step. Appending the shared mirror hint HERE covers
# every route that can leak a model-load/download error (generate, dub,
# archetypes, …), not just TTS generate. append_hf_mirror_hint is a no-op
# for every other error and never raises.
from core.failure import append_hf_mirror_hint
# detail with no next step. #959: same story for the SOCKS-proxy class
# ("Using SOCKS proxy, but the 'socksio' package is not installed").
# Appending the shared hints HERE covers every route that can leak a
# model-load/download error (generate, dub, archetypes, …), not just TTS
# generate. append_hint is a no-op for every other error and never raises.
from core.failure import append_hint
return JSONResponse(
{"detail": append_hf_mirror_hint(str(exc)), "error_class": _entry.get("error_class")},
{"detail": append_hint(str(exc)), "error_class": _entry.get("error_class")},
status_code=500,
headers=headers,
)
@@ -0,0 +1,65 @@
"""Generation takes: starred flag on generation_history
Revision ID: 0009_generation_history_starred
Revises: 0008_pronunciation_dictionary
Create Date: 2026-07-10 00:00:00.000000
Adds ``generation_history.starred INTEGER DEFAULT 0`` the "keep this
take" flag behind the Studio takes rail. Starred takes are exempt from the
retention cap that prunes old generations, and star/unstar round-trips through
``PUT /history/{id}/starred``.
Additive + idempotent (guarded by PRAGMA table_info, matching 0002/0003), so
re-running on a fresh-install DB where ``_BASE_SCHEMA`` already declares the
column is a no-op (Backward-compatible project data constraint). The same
column is mirrored into ``core/db.py::_BASE_SCHEMA`` so fresh installs and
migrated DBs converge on an identical end-state and DBs where alembic can't
run at all pick it up via ``_reconcile_additive_columns`` (the #552/#547
self-heal), the dual-path discipline.
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0009_generation_history_starred"
down_revision: Union[str, None] = "0008_pronunciation_dictionary"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _has_column(table: str, column: str) -> bool:
bind = op.get_bind()
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
return any(r[1] == column for r in rows)
def _has_table(name: str) -> bool:
bind = op.get_bind()
row = bind.execute(
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
{"n": name},
).fetchone()
return row is not None
def upgrade() -> None:
# A DB that somehow missed init has no generation_history at all — the
# startup self-heal (#710) creates it with the column already present, so
# ALTERing here would be both impossible and unnecessary.
if not _has_table("generation_history"):
return
if not _has_column("generation_history", "starred"):
# nullable + DEFAULT 0 to byte-match _BASE_SCHEMA's declaration
# (`starred INTEGER DEFAULT 0`) — the dual-path convergence test
# compares table shape between a migrated DB and a fresh install.
op.add_column(
"generation_history",
sa.Column("starred", sa.Integer(), nullable=True, server_default="0"),
)
def downgrade() -> None:
if _has_table("generation_history") and _has_column("generation_history", "starred"):
op.drop_column("generation_history", "starred")
+22
View File
@@ -122,6 +122,12 @@ class TranslateSegment(BaseModel):
# Available time slot (end - start, seconds) for rate-ratio prediction
# and the cinematic slot-fit pass. Same silent-drop fix as `direction`.
slot_seconds: Optional[float] = None
# Timeline position (seconds) — lets the duration planner borrow silence
# from the gap to the NEXT segment when classifying fits/tight/impossible
# (services/duration_planner.py). Optional: old clients that only send
# slot_seconds still get rate_ratio badges, just no plan verdicts.
start: Optional[float] = None
end: Optional[float] = None
class TranslateRequest(BaseModel):
segments: List[TranslateSegment]
@@ -137,6 +143,22 @@ class TranslateRequest(BaseModel):
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
# Google) can't honor it; the response then carries dialect_applied=false.
dialect: Optional[str] = None
# Two-stage LLM translation quality (provider="openai" only; MT engines
# ignore both). None = default ON for the LLM engine.
# auto_glossary — one up-front LLM pass over the full transcript extracts
# a theme summary + terminology map, merged with `glossary` (user
# entries win) and injected into every per-segment prompt.
# reflect — per-segment critique→rewrite polish after the direct
# translation (2 extra LLM calls per segment; failures silently keep
# the direct translation).
auto_glossary: Optional[bool] = None
reflect: Optional[bool] = None
# Opt-in LLM condensation (default OFF): for segments the duration
# planner classifies "impossible", ask the configured LLM for a shorter
# meaning-preserving rewrite and attach it as plan.suggested_text — a
# per-segment suggestion the user applies manually, never auto-applied.
# No LLM configured / LLM failure → silently no suggestion.
condense: Optional[bool] = False
class DubIngestUrlRequest(BaseModel):
url: str
+222 -1
View File
@@ -29,6 +29,8 @@ import os
import re
import threading
from abc import ABC, abstractmethod
from collections import OrderedDict
from typing import Optional
logger = logging.getLogger("omnivoice.asr")
@@ -1634,6 +1636,161 @@ class FunASRBackend(ASRBackend):
pass
# ── OpenAI-compatible remote transcription (#877 — Qwen3-ASR / FunASR / any
# compatible server, today, without waiting on transformers to catch up) ──
#
# transformers doesn't yet ship a stable Qwen3-ASR integration (issue #877),
# but a self-hosted Qwen3-ASR/FunASR/SenseVoice server exposing an
# OpenAI-compatible `POST /v1/audio/transcriptions` endpoint — or OpenAI's own
# Whisper API — is usable right now. This backend is a pure network client:
# no model runs locally, so it needs no install and claims no GPU.
#
# Settings mirror the LLM-providers convention exactly (services/
# llm_providers.py): base_url/model are plain settings_store text rows; the
# API key is Fernet-encrypted via settings_store.set_secret/get_secret — never
# a .env row, never echoed back to the client. Optional: some self-hosted
# servers (vLLM, LM Studio-style) don't check the key at all.
_ASR_OPENAI_COMPAT_BASE_URL_KEY = "asr.openai_compat.base_url"
_ASR_OPENAI_COMPAT_MODEL_KEY = "asr.openai_compat.model"
_ASR_OPENAI_COMPAT_SECRET_NAME = "asr_openai_compat_key"
def resolve_openai_compat_asr_base_url() -> str:
from services import settings_store
return (
os.environ.get("ASR_OPENAI_COMPAT_BASE_URL")
or settings_store.get_text(_ASR_OPENAI_COMPAT_BASE_URL_KEY)
or ""
)
def resolve_openai_compat_asr_model() -> str:
from services import settings_store
return (
os.environ.get("ASR_OPENAI_COMPAT_MODEL")
or settings_store.get_text(_ASR_OPENAI_COMPAT_MODEL_KEY)
or "whisper-1"
)
def resolve_openai_compat_asr_api_key() -> Optional[str]:
"""Env → encrypted stored key → None. Unlike LLM providers, no 'local'
sentinel: many self-hosted transcription servers accept an empty/omitted
Authorization header outright, so the OpenAI SDK is constructed with
``api_key="not-needed"`` (a non-empty placeholder the SDK requires) when
this returns None, rather than treating a keyless server as unconfigured.
"""
from services import settings_store
return os.environ.get("ASR_OPENAI_COMPAT_API_KEY") or settings_store.get_secret(
_ASR_OPENAI_COMPAT_SECRET_NAME
)
def openai_compat_asr_has_key() -> bool:
"""Whether a key is configured, without ever decrypting it — mirrors
llm_providers.has_key()'s no-plaintext-round-trip contract."""
from services import settings_store
if os.environ.get("ASR_OPENAI_COMPAT_API_KEY"):
return True
return _ASR_OPENAI_COMPAT_SECRET_NAME in settings_store.list_secret_names()
class OpenAICompatASRBackend(ASRBackend):
"""Remote transcription via any OpenAI-compatible server.
Adapts whatever the server returns into this module's expected shape.
Prefers `response_format="verbose_json"` for real per-segment timestamps
(OpenAI's own API and most compatible servers support it); falls back to
plain text with rough single-segment bounds mirroring
MoonshineASRBackend's degraded shape — for minimal servers that reject it.
"""
id = "openai-compat-asr"
display_name = "OpenAI-compatible (remote server)"
gpu_compat = ("cpu",) # network client only — no local compute
def __init__(self):
self._base_url = resolve_openai_compat_asr_base_url()
self._model = resolve_openai_compat_asr_model()
@classmethod
def is_available(cls) -> tuple[bool, str]:
if not resolve_openai_compat_asr_base_url():
return False, "Configure a server endpoint in Settings → Engines"
try:
import openai # noqa: F401
except ImportError:
return False, "openai package not installed. Install with: uv pip install openai"
return True, "ready"
def _client(self):
from openai import OpenAI
api_key = resolve_openai_compat_asr_api_key() or "not-needed"
# max_retries=0: mirrors llm_skills.resolve_skill_client — a
# rate-limited/slow server retrying inside the SDK would blow past
# whatever bounded timeout the caller (dub transcribe, dictation)
# expects from a single call.
return OpenAI(base_url=self._base_url, api_key=api_key, max_retries=0)
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
logger.info(
"OpenAI-compat ASR transcribing %s (base_url=%s, model=%s)",
audio_path, self._base_url, self._model,
)
client = self._client()
try:
with open(audio_path, "rb") as f:
try:
resp = client.audio.transcriptions.create(
file=f, model=self._model, response_format="verbose_json",
)
except Exception:
# Minimal/older compatible servers reject verbose_json
# outright — retry plain before treating it as a real
# failure. Re-open: the SDK may have partially consumed
# the file handle on the first attempt.
f.seek(0)
resp = client.audio.transcriptions.create(
file=f, model=self._model, response_format="json",
)
except Exception as exc:
# Never leak a raw SDK/httpx exception object (auth headers,
# connection internals) straight into a user-facing message —
# same convention as generation.py's _safe_exc_text (#977 class).
raise RuntimeError(
f"OpenAI-compatible ASR server at {self._base_url!r} failed: "
f"{type(exc).__name__}: {exc}"
) from exc
return self._adapt_response(resp)
@staticmethod
def _adapt_response(resp) -> dict:
segments_out = []
# verbose_json: resp.segments is a list of objects with start/end/text.
raw_segments = getattr(resp, "segments", None)
if raw_segments:
for seg in raw_segments:
seg_dict = seg if isinstance(seg, dict) else seg.model_dump()
segments_out.append({
"text": (seg_dict.get("text") or "").strip(),
"start": seg_dict.get("start", 0.0),
"end": seg_dict.get("end", 0.0),
"words": [], # word-level timing isn't part of this API
})
else:
# Plain text response (json/text format) — single-segment shape,
# matching MoonshineASRBackend's degraded fallback exactly.
text = (getattr(resp, "text", None) or "").strip()
if text:
segments_out.append({"text": text, "start": 0.0, "end": None, "words": []})
chunks = [
{"text": seg["text"], "timestamp": (seg["start"], seg["end"])}
for seg in segments_out
]
language = getattr(resp, "language", None) or "en"
return {"chunks": chunks, "segments": segments_out, "language": language}
def _isolated_faster_whisper():
"""Lazy import so the subprocess_asr → subprocess_backend chain isn't
pulled in at registry definition time."""
@@ -1689,6 +1846,7 @@ _REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
"moonshine": MoonshineASRBackend,
"funasr": FunASRBackend,
"sherpa-onnx-asr": SherpaDictationBackend,
"openai-compat-asr": OpenAICompatASRBackend,
# "faster-whisper-isolated": resolved lazily (crash-isolated subprocess).
})
@@ -1700,10 +1858,26 @@ _INSTALL_HINTS: dict[str, str] = {
"faster-whisper": "pip install faster-whisper (CTranslate2; cross-platform, CUDA or CPU)",
"mlx-whisper": "pip install mlx-whisper (Apple Silicon only)",
"pytorch-whisper": "Bundled with transformers — no extra install (CUDA/MPS/CPU)",
"nemo-parakeet": "pip install nemo_toolkit[asr] (NVIDIA Parakeet; CUDA or CPU)",
"nemo-parakeet": (
"No safe install path in this app yet — nemo_toolkit's ASR extras pin "
"transformers>=4.57,<4.58, which conflicts with OmniVoice's own "
"transformers>=5.3 requirement and WILL break the backend "
"(ImportError on startup) if installed into this shared venv. Do NOT "
"install nemo_toolkit here. If you want to try Parakeet TDT, set it "
"up in a separate/dedicated Python environment — not the one "
"OmniVoice manages; in-app isolation for this engine is tracked "
"separately."
),
"moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)",
"funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)",
"sherpa-onnx-asr": "uv add sherpa-onnx (ONNX live dictation; CPU, cross-platform)",
"openai-compat-asr": (
"No install needed — configure a server endpoint in Settings → "
"Engines. Points OmniVoice at any OpenAI-compatible transcription "
"server (a self-hosted Qwen3-ASR/FunASR/SenseVoice server, OpenAI's "
"own Whisper API, or similar) — a path to Qwen3-ASR today, without "
"waiting on a direct transformers integration."
),
"faster-whisper-isolated": (
"No extra install (reuses faster-whisper). Escape hatch for hanging "
"transcribes: runs ASR in a separate process that can be force-killed "
@@ -1853,6 +2027,38 @@ def get_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
return cls()
# ── Reference-transcript cache (#1032) ──────────────────────────────────────
# `get_active_asr_backend()` returns a FRESH backend instance per call for the
# whisper family, so every `transcribe_reference` used to reload whisper
# weights from scratch — a multi-second (CPU: tens of seconds) hit on EVERY
# /generate whose reference clip has no stored transcript (#308 introduced the
# call; profiles saved without a transcript hit it per request). The reference
# audio is identical across those requests, so cache the *transcript* keyed by
# the file's content hash: no model or VRAM is held, repeated generates with
# the same clip skip ASR entirely. Bounded LRU; failures (None) are never
# cached so a transient ASR problem still retries next request.
_REF_TRANSCRIPT_CACHE_MAX = 64
_ref_transcript_cache: "OrderedDict[str, str]" = OrderedDict()
_ref_transcript_lock = threading.Lock()
def _ref_audio_fingerprint(audio_path: str) -> str | None:
"""sha256 of the clip's bytes, or None when unreadable (→ no caching).
Content-keyed (not path-keyed) because ad-hoc clone uploads land in a new
NamedTemporaryFile per request the path changes, the bytes don't.
Reference clips are seconds long, so hashing is negligible next to ASR."""
import hashlib
try:
h = hashlib.sha256()
with open(audio_path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
except OSError:
return None
def transcribe_reference(audio_path: str) -> str | None:
"""Transcribe a voice-clone reference clip with the active ASR backend.
@@ -1864,7 +2070,16 @@ def transcribe_reference(audio_path: str) -> str | None:
the model-attached pipeline is only reached when it is genuinely the last
resort. Returns ``None`` on any failure callers pass ``ref_text=None``
through and the model's built-in fallback still gets its chance.
Results are cached by audio content (#1032) — see the cache notes above.
"""
fingerprint = _ref_audio_fingerprint(audio_path)
if fingerprint is not None:
with _ref_transcript_lock:
cached = _ref_transcript_cache.get(fingerprint)
if cached is not None:
_ref_transcript_cache.move_to_end(fingerprint)
return cached
try:
backend = get_active_asr_backend()
except Exception as e: # noqa: BLE001 — never let ASR break generation
@@ -1888,6 +2103,12 @@ def transcribe_reference(audio_path: str) -> str | None:
(seg.get("text") or "").strip() for seg in result.get("segments", [])
)
text = (text or "").strip()
if text and fingerprint is not None:
with _ref_transcript_lock:
_ref_transcript_cache[fingerprint] = text
_ref_transcript_cache.move_to_end(fingerprint)
while len(_ref_transcript_cache) > _REF_TRANSCRIPT_CACHE_MAX:
_ref_transcript_cache.popitem(last=False)
return text or None
+62 -18
View File
@@ -1,9 +1,12 @@
"""
Audio DSP pipeline broadcast-grade mastering + configurable effects chain.
The default `apply_mastering()` is the same chain shipped since v0.1.0
(highpass + compressor + light reverb). The new `apply_effects_chain()`
lets callers build custom pipelines from a list of named effects.
`apply_mastering()` is the shared pre-stage that runs before the user's
effect preset: highpass + gentle compression only (see `MASTERING_CHAIN`).
Reverb is deliberately NOT part of it it is preset-declared only (e.g.
cinematic, warm); a hidden reverb here used to bake echo into every non-raw
synthesis, which field reports flagged. `apply_effects_chain()` lets callers
build custom pipelines from a list of named effects.
All effects use Spotify's `pedalboard` library. When pedalboard isn't
installed, every function degrades gracefully (returns audio unmodified).
@@ -97,24 +100,26 @@ def get_effect_chain(preset_id: str) -> list[dict]:
# ── Core DSP functions ──────────────────────────────────────────────────
#: Shared pre-preset mastering stage: highpass + gentle compression ONLY.
#: Reverb must never live here — a hidden Reverb in this chain baked echo
#: into every non-raw synthesis regardless of the chosen preset (field
#: reports of echoey voices; the podcast preset even promises "no reverb").
#: Reverb is preset-declared only (see EFFECT_PRESETS: cinematic, warm).
MASTERING_CHAIN = [
{"type": "highpass", "cutoff_hz": 60},
{"type": "compressor", "threshold_db": -15, "ratio": 1.5, "attack_ms": 2.0, "release_ms": 100},
]
def apply_mastering(audio_tensor, sample_rate=24000):
"""Applies professional Broadcast-grade DSP (EQ, Compressor, light Reverb) to the clone voice."""
"""Applies the broadcast pre-stage (highpass + gentle compression) to the clone voice.
Reverb is intentionally absent only user-chosen effect presets declare
it. Degrades gracefully: pedalboard missing or any DSP error returns the
input unmodified.
"""
try:
from pedalboard import Pedalboard, Compressor, Reverb, HighpassFilter
import numpy as np
board = Pedalboard([
HighpassFilter(cutoff_frequency_hz=60),
Compressor(threshold_db=-15, ratio=1.5, attack_ms=2.0, release_ms=100),
Reverb(room_size=0.10, wet_level=0.08, dry_level=0.95)
])
audio_np = audio_tensor.cpu().numpy()
if audio_np.ndim == 1:
audio_np = audio_np[np.newaxis, :]
effected = board(audio_np, sample_rate, reset=False)
return torch.from_numpy(effected).to(audio_tensor.device)
except ImportError:
return audio_tensor # Fail gracefully if pedalboard isn't installed
return apply_effects_chain(audio_tensor, sample_rate, MASTERING_CHAIN)
except Exception as e:
logger.warning("Mastering DSP Error: %s", e)
return audio_tensor
@@ -142,6 +147,45 @@ def normalize_audio(audio_tensor, target_dBFS=-2.0):
return audio_tensor
def trim_trailing_silence(
audio_tensor: torch.Tensor,
sample_rate: int,
keep_tail_s: float = 0.3,
) -> torch.Tensor:
"""Trim trailing near-silence from a generated clip, keeping a short
natural tail of ``keep_tail_s`` seconds after the last voiced sample.
Amplitude-based SILENCE trim only no content analysis of any kind.
Uses the same -50 dBFS silence floor as :func:`normalize_audio`: the last
sample above that floor marks the end of speech, and everything more than
``keep_tail_s`` past it is dropped.
Guaranteed no-op cases (input returned as-is, same object):
the trailing quiet span is already ``keep_tail_s`` (clean output);
the entire clip sits below the floor (dead render downstream
dead-render guards own that case, we must not shrink their evidence);
empty input.
Accepts ``(n,)`` or ``(channels, n)`` tensors; the returned tensor keeps
the input's shape convention.
"""
if audio_tensor.numel() == 0:
return audio_tensor
# -50 dBFS ≈ 0.00316 linear — matches normalize_audio's silence floor.
floor = 10 ** (-50.0 / 20.0)
envelope = torch.abs(audio_tensor)
if envelope.ndim > 1:
envelope = envelope.amax(dim=tuple(range(envelope.ndim - 1)))
voiced = torch.nonzero(envelope > floor)
if voiced.numel() == 0:
return audio_tensor
last_voiced = int(voiced[-1].item())
end = last_voiced + 1 + int(keep_tail_s * sample_rate)
if end >= audio_tensor.shape[-1]:
return audio_tensor
return audio_tensor[..., :end]
def apply_effects_chain(audio_tensor, sample_rate: int, chain: list[dict]) -> torch.Tensor:
"""Apply a chain of named effects to an audio tensor.
+20 -7
View File
@@ -101,6 +101,7 @@ def synthesize_chapter(
*,
crossfade_ms: int = 50,
lexicon: Optional[dict] = None,
segment_cache: Optional["object"] = None,
):
"""Render a chapter's spans to one waveform via an injected ``synth``.
@@ -111,6 +112,12 @@ def synthesize_chapter(
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
given) respells each span's text before chunking so the engine pronounces
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
``segment_cache`` (when given a :class:`services.longform_render.
SegmentCache`) is consulted per spoken span: a cached segment WAV is reused
instead of synthesizing, and every freshly rendered span is stored the
moment it finishes so a one-sentence edit re-renders one segment and an
interrupted chapter resumes from its finished segments. Pauses are
synthesized silence and never touch the cache.
Returns ``(audio_tensor, duration_seconds)``. torch + chunked_tts are
imported lazily so this module stays import-light for the pure parser path.
@@ -122,13 +129,19 @@ def synthesize_chapter(
items: list = [] # ("a", tensor) for audio, ("s", n_samples) for silence
for span in spans:
if span.text:
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
if len(rendered) == 1:
items.append(("a", rendered[0]))
elif rendered:
items.append(("a", concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms)))
audio = segment_cache.load(span) if segment_cache is not None else None
if audio is None:
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
if len(rendered) == 1:
audio = rendered[0]
elif rendered:
audio = concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms)
if audio is not None and segment_cache is not None:
segment_cache.store(span, audio)
if audio is not None:
items.append(("a", audio))
if span.pause_ms_after > 0:
n = int(sample_rate * span.pause_ms_after / 1000.0)
if n > 0:
+10
View File
@@ -187,6 +187,14 @@ def put_job(job_id: str, job: dict) -> None:
def save_job(job_id: str, job: dict, filename: str = "", duration: float = 0.0, content_hash: str = "") -> None:
"""Persist dub job state to SQLite so it survives restarts. Uses UPSERT
on `id` so repeated saves in a session keep the latest snapshot.
language / language_code / content_hash only update when the incoming
value is non-empty: the ingest-time insert runs before the target
language is known (both columns ""), generation sets them on the job
dict, and a later save from a job that lost them (e.g. hydrated from an
old row) must not clobber the healed columns back to "". The frontend
keys history restore off language_code, so a frozen "" hid finished
tracks until the user re-picked a language.
"""
try:
segments = job.get("segments") or []
@@ -200,6 +208,8 @@ def save_job(job_id: str, job: dict, filename: str = "", duration: float = 0.0,
filename=excluded.filename,
duration=excluded.duration,
segments_count=excluded.segments_count,
language=CASE WHEN excluded.language != '' THEN excluded.language ELSE dub_history.language END,
language_code=CASE WHEN excluded.language_code != '' THEN excluded.language_code ELSE dub_history.language_code END,
tracks=excluded.tracks,
job_data=excluded.job_data,
content_hash=CASE WHEN excluded.content_hash != '' THEN excluded.content_hash ELSE dub_history.content_hash END""",
+329
View File
@@ -0,0 +1,329 @@
"""Pre-synthesis duration planning for dub segments.
The Smart Fit planner (services/fit_planner.py) reconciles dubbed audio
with the timeline AFTER synthesis by then a doomed segment has already
burned GPU time and can only be sped up or trimmed. This module predicts
BEFORE TTS whether a translated segment can possibly fit its slot, so the
UI can badge it (and optionally offer a shorter rewrite) while the text is
still cheap to change. It never blocks generation it informs.
Three pieces, all pure and unit-testable:
1. **Estimator** predict the natural speech duration of target-language
text. Self-calibrating: segments already synthesized in this job carry
``(chars, natural duration)`` records (written by dub_generate for every
natural-rate strategy), and the median chars-per-second of those is a
far better predictor for *this* voice/engine/language than any table.
With no (or too little) calibration data it falls back to the
conservative static per-language rate table in ``services.speech_rate``
(the same one the rate-ratio badge uses).
2. **Classifier** per segment, compare the estimate against the
*available* time: the slot plus silence borrowable from the gap to the
next segment (mirroring fit_planner's slack absorption, but with a
deliberate cap see ``GAP_BORROW_MAX_S``). The verdict thresholds are
derived from the SAME ``FitParams`` caps fit_planner enforces, so:
fits need max_audio_only_rate absorbed imperceptibly
tight need what the caps absorb audible speed-up and/or
video slow-down
impossible beyond the caps fit_planner will trim
3. **Condensation** (optional, caller-gated) for ``impossible`` segments,
ask the configured LLM for a meaning-preserving shorter rewrite
targeting the available duration. Strictly best-effort: no LLM, an LLM
error, or a divergent reply all degrade to a no-op.
No I/O, no torch; the only side-effectful function is ``condense_for_slot``
(network LLM call), which callers opt into explicitly.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Iterable, Optional
from services.fit_planner import MAX_AUDIO_RATE_HARD, FitParams
from services.llm_backend import OffBackend, get_active_llm_backend
from services.speech_rate import expected_duration
# Shared LLM-output divergence guard (target-script + length window +
# critique-echo) — same seam speech_rate's Autofit pass uses.
from services.translator import refine_output_ok
logger = logging.getLogger("omnivoice.duration_planner")
# LLM Skills registry id — condensation is the same "make the line fit its
# slot" skill family as the Autofit pass, so it routes (and can be disabled)
# through the same Settings → LLM Skills entry.
_SKILL_ID = "slot_fitting"
# ── Calibration ─────────────────────────────────────────────────────────
# A calibration only counts once this many usable samples exist — below
# that, one odd segment (a sound effect, a mumbled clone ref) would swing
# the estimate more than the static table's error.
MIN_CALIBRATION_SAMPLES = 3
# Per-sample sanity floor: shorter/tinier segments carry more silence
# padding and TTS ramp-up than speech, so their chars/sec is noise.
MIN_SAMPLE_DUR_S = 0.4
MIN_SAMPLE_CHARS = 4
# How far a segment may borrow into the silent gap before the next segment
# (or the video tail). fit_planner itself absorbs the WHOLE gap, so this cap
# makes the pre-synthesis verdict deliberately conservative: a huge gap
# (scene change, music bed) is real slack at mix time, but planning speech
# to sprawl seconds past its slot is rarely what the user wants — and the
# estimate is fuzzy enough that promising it would over-sell.
GAP_BORROW_MAX_S = 3.0
@dataclass(frozen=True)
class Calibration:
"""Observed speech rate for one (job, language) pair."""
cps: float # chars per second at natural TTS rate
samples: int # how many segments backed it
def calibrate_cps(samples: Iterable[tuple[float, float]]) -> Optional[Calibration]:
"""Derive a chars-per-second calibration from ``(chars, natural_dur_s)``
pairs of already-synthesized segments.
Median of the per-segment rates robust against the occasional outlier
(a segment that's mostly a breath, an engine hiccup) that would drag a
mean. Returns None when fewer than ``MIN_CALIBRATION_SAMPLES`` usable
samples exist; callers then fall back to the static table.
"""
rates: list[float] = []
for chars, dur in samples:
try:
chars = float(chars)
dur = float(dur)
except (TypeError, ValueError):
continue
if dur >= MIN_SAMPLE_DUR_S and chars >= MIN_SAMPLE_CHARS:
rates.append(chars / dur)
if len(rates) < MIN_CALIBRATION_SAMPLES:
return None
rates.sort()
n = len(rates)
mid = n // 2
median = rates[mid] if n % 2 else (rates[mid - 1] + rates[mid]) / 2.0
if median <= 0:
return None
return Calibration(cps=median, samples=n)
def calibration_from_job(job: dict, lang: str) -> Optional[Calibration]:
"""Build a Calibration from the ``seg_natural_durs_by_lang`` records
dub_generate persists on the job. Tolerates any legacy/partial shape."""
try:
recs = (job.get("seg_natural_durs_by_lang") or {}).get(lang) or {}
return calibrate_cps(
(r.get("chars", 0), r.get("dur", 0))
for r in recs.values()
if isinstance(r, dict)
)
except Exception as e: # noqa: BLE001 — calibration is best-effort by design
logger.debug("calibration_from_job skipped: %s", e)
return None
# ── Estimator ───────────────────────────────────────────────────────────
def estimate_natural_duration(
text: str, lang: str, calibration: Optional[Calibration] = None,
) -> float:
"""Predicted natural-rate speech duration (seconds) of ``text``.
Calibrated rate when available, else the static per-language table
(``speech_rate.expected_duration``, 13 cps default for unknown codes).
"""
text = (text or "").strip()
if not text:
return 0.0
if calibration is not None and calibration.cps > 0:
return len(text) / calibration.cps
return expected_duration(text, lang)
# ── Classifier ──────────────────────────────────────────────────────────
def absorb_caps(params: FitParams) -> tuple[float, float]:
"""(fits_cap, absorb_cap) need-ratios aligned with fit_planner.
``fits_cap``: up to here the audio-only speed-up is imperceptible.
``absorb_cap``: up to here fit_planner's knobs absorb the overrun
(audio cap × video cap in hybrid mode; the legacy hard audio ceiling
when video retiming is off). Beyond it, fit_planner trims.
"""
if params.allow_video_retime:
return params.max_audio_only_rate, params.audio_rate_cap * params.video_slow_cap
return params.max_audio_only_rate, MAX_AUDIO_RATE_HARD
def classify_segments(
segments: list[dict],
target_lang: str,
*,
calibration: Optional[Calibration] = None,
fit_params: Optional[FitParams] = None,
total_dur_s: float = 0.0,
gap_borrow_max_s: float = GAP_BORROW_MAX_S,
) -> list[dict]:
"""Classify each segment's translated text against its timeline slot.
``segments``: chronological dicts with ``id``, ``start``, ``end``
(seconds) and ``text`` (the translated text about to be synthesized).
``total_dur_s``: original video duration (0/unknown the last segment
gets no tail borrow), mirroring ``fit_planner.plan_fit``.
Returns one dict per segment::
{id, status, est_dur_s, available_s, est_overrun_s, calibrated}
``status`` {"fits", "tight", "impossible"}; ``est_overrun_s`` is the
predicted seconds of speech past the available time (0 when it fits).
Pure function: no I/O, deterministic.
"""
params = fit_params or FitParams()
fits_cap, cap = absorb_caps(params)
n = len(segments)
out: list[dict] = []
for i, seg in enumerate(segments):
start = float(seg["start"])
end = float(seg["end"])
slot = max(0.0, end - start)
# Borrowable silence — fit_planner's slack absorption, capped.
if i + 1 < n:
gap = max(0.0, float(segments[i + 1]["start"]) - end)
borrow = min(max(0.0, gap - params.gap_guard_s), gap_borrow_max_s)
elif total_dur_s > 0:
borrow = min(max(0.0, float(total_dur_s) - end), gap_borrow_max_s)
else:
borrow = 0.0
available = slot + borrow
est = estimate_natural_duration(seg.get("text") or "", target_lang, calibration)
if est <= 0.0:
status = "fits"
overrun = 0.0
elif available <= 0.0:
status = "impossible"
overrun = est
else:
need = est / available
# Same boundary tolerance as fit_planner's _EPS: a need that
# lands exactly on a cap is absorbed, not escalated.
if need <= fits_cap + 1e-9:
status = "fits"
elif need <= cap + 1e-9:
status = "tight"
else:
status = "impossible"
overrun = max(0.0, est - available)
out.append({
"id": str(seg.get("id", f"seg_{i}")),
"status": status,
"est_dur_s": round(est, 3),
"available_s": round(available, 3),
"est_overrun_s": round(overrun, 3),
"calibrated": calibration is not None,
})
return out
# ── Optional LLM condensation ───────────────────────────────────────────
_CONDENSE_PROMPT = """\
You are a dubbing writer. The user will give you a translated line that is
TOO LONG for its time slot. Rewrite it shorter so it can be read aloud
within the target duration: cut filler words, tighten phrasing, and drop
the least essential clauses but preserve the meaning. Never change
character names, proper nouns, numbers, or technical terms. Stay in the
same language as the line.
Reply with ONLY the rewritten line. No quotes, no commentary."""
# Bound the LLM loop — condensation is a per-segment *suggestion*, not a
# fit guarantee, so two shots are plenty before degrading to a no-op.
_CONDENSE_ATTEMPTS = 2
def condense_for_slot(
text: str,
*,
available_s: float,
target_lang: str,
source_text: Optional[str] = None,
calibration: Optional[Calibration] = None,
) -> dict:
"""Meaning-preserving shorter rewrite of ``text`` targeting ``available_s``.
Returns ``{"text", "applied", "est_dur_s"}`` (+ ``"error"`` on the no-op
paths). ``applied=False`` keeps the input text untouched no LLM
configured, LLM failure, and divergent/too-aggressive replies all
degrade there. The best (shortest-estimate) candidate that passes the
divergence guard AND is actually shorter than the input wins; a reply
that fits ``available_s`` returns immediately.
"""
text = (text or "").strip()
base_est = estimate_natural_duration(text, target_lang, calibration)
if not text or available_s <= 0:
return {"text": text, "applied": False, "est_dur_s": round(base_est, 3),
"error": "nothing-to-condense"}
if base_est <= available_s:
return {"text": text, "applied": False, "est_dur_s": round(base_est, 3),
"error": "already-fits"}
from services import llm_skills
# `active=` forwards this module's (monkeypatch-able) name so the
# no-override path matches the plain get_active_llm_backend behavior.
llm = llm_skills.skill_backend(_SKILL_ID, active=lambda: get_active_llm_backend())
if isinstance(llm, OffBackend):
return {"text": text, "applied": False, "est_dur_s": round(base_est, 3),
"error": "no-llm"}
best: Optional[tuple[str, float]] = None # (candidate, est)
for attempt in range(1, _CONDENSE_ATTEMPTS + 1):
user_lines = [
f"Target language: {target_lang}",
f"Target duration: {available_s:.2f}s",
f"Current line: {text}",
f"Current reading duration: ~{base_est:.2f}s",
]
if source_text:
user_lines.append(f"Source line (for meaning): {source_text}")
if attempt > 1 and best is not None:
user_lines.append(
f"Your previous rewrite was still ~{best[1]:.2f}s. Cut further."
)
try:
reply = llm.chat(
system=_CONDENSE_PROMPT, user="\n".join(user_lines),
temperature=0.2, # pinned like Autofit — default 1.0 drifts/invents
)
except Exception as e: # noqa: BLE001 — LLM failure must no-op, never raise
logger.warning("condense attempt %d failed: %s", attempt, e)
break
candidate = (reply or "").strip()
if not candidate:
continue
ok, reason = refine_output_ok(text, candidate, target_lang)
if not ok:
logger.warning("condense attempt %d rejected (%s)", attempt, reason)
continue
est = estimate_natural_duration(candidate, target_lang, calibration)
if est >= base_est:
continue # not actually shorter — useless as a suggestion
if best is None or est < best[1]:
best = (candidate, est)
if est <= available_s:
break # fits — done
if best is None:
return {"text": text, "applied": False, "est_dur_s": round(base_est, 3),
"error": "condense-failed"}
return {"text": best[0], "applied": True, "est_dur_s": round(best[1], 3)}
+292
View File
@@ -0,0 +1,292 @@
"""Self-heal for HF cache snapshots whose entries no longer resolve.
The Hugging Face hub cache stores each file's bytes once under
``models--<org>--<name>/blobs/<hash>`` and exposes every revision as
``snapshots/<rev>/<filename>`` entries that link into ``blobs/``. Several
real-world events leave a snapshot entry *broken* a dangling symlink (its
blob target doesn't exist) or a zero-byte stand-in file — while the actual
bytes are safely on disk under ``blobs/``: a blob-naming mismatch between
download modes, an interrupted rename mid-download, antivirus interference.
``os.path.isfile()`` on a dangling symlink is False, so transformers concludes
the weights are missing ("… does not appear to have a file named
pytorch_model.bin or model.safetensors") even though the multi-GB download
completed. A plain ``snapshot_download`` doesn't reliably fix this — depending
on hub version and platform symlink support, the existing-but-broken entry can
short-circuit the restore. Deleting exactly the broken entries first makes
``snapshot_download`` deterministically restore them (reusing completed blobs
where the naming matches, re-downloading only where it doesn't).
Conservative by design, repairing STATE rather than chasing one cause:
* never touches ``blobs/`` (the downloaded bytes),
* never touches snapshot entries that resolve,
* never force-redownloads healthy files,
* never raises any internal failure logs and returns a summary,
* a healthy cache is a cheap lstat/stat walk of ``snapshots/`` (no hashing,
no network) on every platform; the heal is generic, not Windows-gated.
"""
from __future__ import annotations
import logging
import os
logger = logging.getLogger("omnivoice.hf_cache_repair")
# Snapshot entries that are never legitimately zero bytes: weight formats and
# JSON/sentencepiece config-tokenizer files (an empty file is not valid JSON /
# not a valid serialized model). Zero-byte files with any OTHER suffix — an
# empty .txt, .md, .gitattributes, a marker file a repo genuinely ships empty —
# are left alone: when unsure, don't flag.
_NEVER_EMPTY_SUFFIXES = frozenset({
# weights / tensors
".safetensors", ".bin", ".pt", ".pth", ".ckpt", ".onnx", ".gguf",
".msgpack", ".h5", ".pb", ".tflite",
# config / tokenizer
".json", ".model", ".spm",
})
def _env_flag(name: str) -> bool:
return (os.environ.get(name) or "").strip().lower() in {"1", "true", "yes", "on"}
def hf_cache_home() -> str:
"""The hub cache root in effect. Mirrors huggingface_hub's resolution
(``HF_HUB_CACHE`` > ``HF_HOME``/hub > default) but reads the env at call
time hub's constants freeze at import, which is too early for tests and
for the Windows short-cache redirect in ``core.config``."""
env = (os.environ.get("HF_HUB_CACHE") or "").strip()
if env:
return env
hf_home = (os.environ.get("HF_HOME") or "").strip()
if hf_home:
return os.path.join(hf_home, "hub")
try:
from huggingface_hub.constants import HF_HUB_CACHE
return HF_HUB_CACHE
except Exception:
return os.path.join(os.path.expanduser("~"), ".cache", "huggingface", "hub")
def repo_cache_dir(repo_id: str, cache_dir: str | None = None) -> str:
"""The ``models--<org>--<name>`` folder for ``repo_id`` (repo_type=model)."""
return os.path.join(cache_dir or hf_cache_home(),
"models--" + repo_id.replace("/", "--"))
def _is_dangling_symlink(path: str) -> bool:
# islink() uses lstat (True even when the target is gone); exists()
# resolves the link — False for a dangling one. Never raises for a path
# that came out of os.walk.
return os.path.islink(path) and not os.path.exists(path)
def _is_suspicious_zero_byte(path: str) -> bool:
"""A zero-byte REGULAR file standing where model content must be.
Conservative: only weight/config-typed names are flagged (those are never
legitimately empty the bytes to restore them live in ``blobs/`` or on
the Hub); anything else is presumed intentional and left alone."""
if os.path.islink(path):
return False # resolving symlinks are handled by the dangling check
try:
if not os.path.isfile(path) or os.path.getsize(path) != 0:
return False
except OSError:
return False
return os.path.splitext(path)[1].lower() in _NEVER_EMPTY_SUFFIXES
def find_dangling_entries(repo_cache_dir: str) -> list[str]:
"""Broken entries under ``<repo_cache_dir>/snapshots/*/``: dangling
symlinks plus suspicious zero-byte regular files (see above).
Returns absolute paths. On a healthy cache this is a no-op scan a pure
lstat/stat walk of ``snapshots/`` (``blobs/`` is never visited), no
hashing, no network. Never raises."""
broken: list[str] = []
snapshots = os.path.join(repo_cache_dir, "snapshots")
if not os.path.isdir(snapshots):
return broken
try:
# followlinks=False: a dangling symlink is not a dir, so os.walk lists
# it among the files of its parent — exactly where we scan.
for root, _dirs, files in os.walk(snapshots, followlinks=False):
for name in files:
path = os.path.join(root, name)
if _is_dangling_symlink(path) or _is_suspicious_zero_byte(path):
broken.append(path)
except OSError as walk_err: # pragma: no cover - defensive
logger.warning("HF cache scan of %s aborted: %s", snapshots, walk_err)
return broken
def _force_copy_mode(cache_root: str) -> bool:
"""Best-effort: make huggingface_hub materialize snapshot entries as real
file COPIES instead of symlinks for the rest of this process.
Why: hub's ``are_symlinks_supported()`` probe can succeed in-process while
real snapshot symlink creation fails or produces broken links (Windows
without Developer Mode is the reported case) and the result is memoized
in the private ``file_download._are_symlinks_supported_in_dir`` dict, so a
plain ``snapshot_download`` retry would recreate the SAME dangling links.
Pre-seeding that memo with False flips hub into copy mode. It's private
API, so any failure (attribute/shape changed across hub versions) is
logged and reported as False the caller then skips the copy-mode pass
rather than crash. Deliberately NOT undone: on a host where links come
out broken, every later download should use copies too."""
try:
from pathlib import Path
import huggingface_hub.file_download as _fd
memo = getattr(_fd, "_are_symlinks_supported_in_dir", None)
if not isinstance(memo, dict):
raise TypeError(
f"_are_symlinks_supported_in_dir is {type(memo).__name__}, expected dict"
)
# Same key normalization hub's are_symlinks_supported() applies.
memo[str(Path(cache_root).expanduser().resolve())] = False
return True
except Exception as e:
logger.warning(
"Could not force copy-mode for the HF cache (%s) — "
"huggingface_hub's private memo may have changed; skipping the "
"copy-mode repair pass.", e,
)
return False
def repair_repo_cache(repo_id: str, cache_dir: str | None = None) -> dict:
"""Repair a repo's cache: delete broken snapshot entries (and ONLY those),
then ``snapshot_download`` to restore the missing files hub reuses
completed blobs where the naming matches and re-downloads otherwise.
Verified after the fact: if the restore recreated dangling links (a host
where hub's symlink probe passes but real links come out broken — Windows
without Developer Mode), force copy mode and repair once more so the
snapshot ends up with real files.
Returns a summary dict; never raises:
``found`` broken entries detected up front,
``removed`` entries actually deleted (both passes),
``restored`` True when a snapshot_download completed,
``outcome`` "healthy" | "healed_with_links" | "healed_with_copies"
| "repair_failed",
``ok`` True unless outcome == "repair_failed",
``error`` "" or why the repair failed.
"""
summary: dict = {
"repo_id": repo_id,
"repo_dir": "",
"found": 0,
"removed": 0,
"restored": False,
"outcome": "repair_failed",
"ok": False,
"error": "",
}
try:
cache_root = cache_dir or hf_cache_home()
repo_dir = repo_cache_dir(repo_id, cache_root)
summary["repo_dir"] = repo_dir
broken = find_dangling_entries(repo_dir)
summary["found"] = len(broken)
if not broken:
summary["ok"] = True # nothing broken → nothing to do
summary["outcome"] = "healthy"
return summary
if _env_flag("HF_HUB_OFFLINE") or _env_flag("TRANSFORMERS_OFFLINE"):
# Don't delete what we can't restore: offline mode means the
# follow-up snapshot_download is off the table.
summary["error"] = (
"Hugging Face offline mode is enabled "
"(HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE) — cannot restore files"
)
logger.warning(
"Model cache for %s has %d broken snapshot entr%s but HF "
"offline mode is set — skipping repair.",
repo_id, len(broken), "y" if len(broken) == 1 else "ies",
)
return summary
def _remove(paths: list[str]) -> int:
n = 0
for path in paths:
try:
os.remove(path) # removes the link/file itself, never a blob
n += 1
logger.info(
"HF cache self-heal: removed broken snapshot entry %s", path
)
except OSError as rm_err:
logger.warning(
"HF cache self-heal: could not remove broken entry %s: %s",
path, rm_err,
)
return n
summary["removed"] = _remove(broken)
if summary["removed"] == 0:
summary["error"] = "broken entries could not be removed"
return summary
from huggingface_hub import snapshot_download
dl_kwargs: dict = {"repo_id": repo_id}
if cache_dir:
dl_kwargs["cache_dir"] = cache_dir
endpoint = os.environ.get("HF_ENDPOINT")
if endpoint:
dl_kwargs["endpoint"] = endpoint
snapshot_download(**dl_kwargs)
summary["restored"] = True
# Verify-after-repair: hub's memoized symlink probe can claim support
# while the links it just recreated dangle again. If so, force copy
# mode and repair once more so real files land in the snapshot.
still_broken = find_dangling_entries(repo_dir)
if not still_broken:
summary["ok"] = True
summary["outcome"] = "healed_with_links"
logger.info(
"HF cache self-heal for %s: removed %d broken snapshot entr%s "
"and restored the snapshot from existing blobs / the Hub.",
repo_id, summary["removed"],
"y" if summary["removed"] == 1 else "ies",
)
return summary
logger.warning(
"HF cache self-heal for %s: the restore recreated %d broken "
"link(s) — forcing copy-mode and repairing once more.",
repo_id, len(still_broken),
)
if not _force_copy_mode(cache_root):
summary["error"] = (
"the snapshot restore recreated broken links and copy-mode "
"could not be forced"
)
return summary
summary["removed"] += _remove(still_broken)
snapshot_download(**dl_kwargs)
remaining = find_dangling_entries(repo_dir)
if remaining:
summary["error"] = (
f"{len(remaining)} snapshot entr"
f"{'y is' if len(remaining) == 1 else 'ies are'} still broken "
"after the copy-mode repair"
)
return summary
summary["ok"] = True
summary["outcome"] = "healed_with_copies"
logger.info(
"HF cache self-heal for %s: healed with real file copies "
"(symlinks on this host come out broken; hub stays in copy-mode "
"for the rest of this run).", repo_id,
)
return summary
except Exception as e: # never raise — repair is best-effort
summary["error"] = f"{type(e).__name__}: {e}"
logger.warning(
"HF cache self-heal for %s failed: %s", repo_id, summary["error"],
)
return summary
+22 -2
View File
@@ -49,7 +49,7 @@ def _canon_value(field: str, value):
return value
def segment_fingerprint(seg: dict) -> str:
def segment_fingerprint(seg: dict, track_lang: str | None = None) -> str:
"""Deterministic hash of the inputs that actually affect TTS output.
Any change to `_GEN_INPUT_FIELDS` flips the hash and the segment becomes
@@ -61,8 +61,20 @@ def segment_fingerprint(seg: dict) -> str:
so a fingerprint computed from the generate request (server defaults
filled in) matches one recomputed later from the client's raw segment
state the root cause of #281's "1 edit re-dubs all N lines".
``track_lang`` (P1.3) is the TRACK's language code (`req.language_code`,
e.g. "es"). It is part of the fingerprint because the same segment text
renders different audio per language without it, a bn hash could
vouch for an es WAV on a multi-track job. It is only mixed in when
provided, so hashes computed by legacy callers (and hashes stored by
previous builds, which never carried a language) keep their old values;
a legacy hash therefore never matches a lang-scoped fingerprint and the
segment reads as stale the safe direction (one clean regen, never a
wrong-language splice).
"""
payload = {k: _canon_value(k, seg.get(k)) for k in _GEN_INPUT_FIELDS}
if track_lang:
payload["track_lang"] = str(track_lang)
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
@@ -120,6 +132,7 @@ def plan_incremental(
segments: list[dict],
*,
stored_hashes: dict[str, str] | None = None,
track_lang: str | None = None,
) -> dict:
"""Return `{stale, fresh, total, fingerprints}` where:
@@ -133,6 +146,13 @@ def plan_incremental(
`stored_hashes` may come from the caller's own bookkeeping (e.g. the
`dub_history.job_data["seg_hashes"]` we'll start writing in Phase 4.5).
When missing, every segment is considered stale (first run).
`track_lang` (P1.3) scopes the plan to ONE dub track: pass the track's
language code together with THAT language's stored hashes
(`job_data["seg_hashes_by_lang"][lang]`) so staleness is judged against
the active track, never against whatever language was generated last.
Must match the language the generate run hashed with, or every segment
reads stale (#281 parity class).
"""
stored = stored_hashes or {}
stale: list[str] = []
@@ -142,7 +162,7 @@ def plan_incremental(
sid = str(seg.get("id", ""))
if not sid:
continue
fp = segment_fingerprint(seg)
fp = segment_fingerprint(seg, track_lang=track_lang)
fingerprints[sid] = fp
prev = stored.get(sid)
if prev == fp:
+84 -3
View File
@@ -24,10 +24,13 @@ users.
"""
from __future__ import annotations
import logging
import os
from dataclasses import dataclass
from typing import Optional
logger = logging.getLogger("omnivoice.llm_providers")
# Settings-store row names (non-secret overrides live in the plaintext table;
# keys live in the encrypted secret table under ``llm_key.<id>``).
_ACTIVE_PROVIDER_KEY = "llm.active_provider"
@@ -248,6 +251,19 @@ def is_configured(p: Provider) -> bool:
# ── Active provider selection ─────────────────────────────────────────────
def stored_active_provider_id() -> Optional[str]:
"""The user's explicitly-persisted selection ONLY — no env pin, no legacy
TRANSLATE_* fallback, no auto-detect.
``None`` means the user has never chosen a provider. This is what gates
save-activates in the settings router (#963): an explicit save may claim
the *empty* slot, but must never steal it from a made choice.
"""
from services import settings_store
stored = settings_store.get_text(_ACTIVE_PROVIDER_KEY)
return stored if stored and stored in _BY_ID else None
def active_provider_id() -> Optional[str]:
"""The provider Cinematic/Autofit should use.
@@ -255,12 +271,11 @@ def active_provider_id() -> Optional[str]:
configured provider None. Legacy ``TRANSLATE_BASE_URL`` users with no
explicit selection resolve to ``custom`` (its envs are TRANSLATE_*).
"""
from services import settings_store
env_pick = os.environ.get("LLM_DEFAULT_PROVIDER")
if env_pick and env_pick in _BY_ID:
return env_pick
stored = settings_store.get_text(_ACTIVE_PROVIDER_KEY)
if stored and stored in _BY_ID:
stored = stored_active_provider_id()
if stored:
return stored
# Legacy: a lone TRANSLATE_BASE_URL means the old single-endpoint setup.
if os.environ.get("TRANSLATE_BASE_URL"):
@@ -354,3 +369,69 @@ def describe(p: Provider) -> dict:
d["account_id"] = resolve_account_id(p)
d["account_from_env"] = bool(p.account_env and os.environ.get(p.account_env))
return d
# ── Legacy TRANSLATE_* prefs migration (#963) ──────────────────────────────
# prefs.json row → the custom-provider field it becomes.
_LEGACY_TRANSLATE_PREFS: tuple[tuple[str, str], ...] = (
("env.TRANSLATE_BASE_URL", "base_url"),
("env.TRANSLATE_MODEL", "model"),
("env.TRANSLATE_API_KEY", "api_key"),
)
def migrate_legacy_translate_prefs() -> bool:
"""Move the retired (≤v0.3.7) Translation-LLM panel's prefs rows into the
``custom`` provider's own settings-store rows, then delete them.
Those ``env.TRANSLATE_*`` rows in prefs.json are re-imported into
``os.environ`` on every launch (main.py), and a live ``TRANSLATE_BASE_URL``
makes :func:`active_provider_id` resolve to ``custom`` ahead of the stored
selection fallbacks silently hijacking the active slot on every restart
(issue #963, "Ollama works until I restart"). Must run BEFORE main.py's
prefsenv import so the rows never reach the environment.
Semantics:
* Each value is copied only where the store has no value yet a user's
later edit of the custom provider always wins over legacy leftovers.
* The prefs row is deleted afterwards either way, so it can never be
re-imported as env again (the migration is one-shot per row).
* Real process env vars are NEVER touched a shell/.env
``TRANSLATE_BASE_URL`` keeps its documented override behavior.
* A row whose store write fails is kept in prefs (it still works via the
env import this launch and the migration retries next launch).
Returns True if any prefs row was migrated/removed.
"""
from core import prefs
from services import settings_store
changed = False
for prefs_key, field in _LEGACY_TRANSLATE_PREFS:
try:
raw = prefs.get(prefs_key)
except Exception:
logger.exception("legacy TRANSLATE prefs read failed (%s)", prefs_key)
return changed
if raw is None:
continue
val = str(raw).strip()
try:
if val:
if field == "base_url":
if not settings_store.get_text(_BASE_URL_KEY + "custom"):
save_overrides("custom", base_url=val)
elif field == "model":
if not settings_store.get_text(_MODEL_KEY + "custom"):
save_overrides("custom", model=val)
else: # api_key — encrypted store, never overwrite an existing one
if not _key_in_store("custom"):
save_key("custom", val)
prefs.delete(prefs_key)
changed = True
except Exception:
# Store not ready (e.g. settings table missing) — keep the prefs
# row so the legacy env import still works and we retry next boot.
logger.exception("legacy TRANSLATE prefs migration failed (%s)", prefs_key)
return changed
+17 -1
View File
@@ -242,8 +242,24 @@ def resolve_skill_client(skill_id: str) -> Optional[SkillClient]:
# skill's wall-clock budget (the cinematic pass budget, the glossary call
# timeout) from inside one request. Fail fast — the per-call timeout and the
# pass-level budget are the only bounds we want. Mirrors OpenAICompatBackend.
#
# #959 class guard: OpenAI() eagerly builds its httpx client, which can
# raise AT CONSTRUCTION for environment-shaped reasons — the reported one
# is httpx's ImportError under ALL_PROXY/HTTPS_PROXY=socks5:// without
# socksio; a malformed proxy URL or broken cert bundle fails the same way.
# The contract here is already "None == LLM unavailable, degrade" — a bad
# proxy env must degrade the skill, never 500 the calling feature.
try:
client = OpenAI(max_retries=0, **kw)
except Exception as exc:
logger.warning(
"LLM client construction failed for skill %s (provider %s): %s"
"treating the skill as unavailable.",
skill_id, res.provider.id, exc,
)
return None
return SkillClient(
client=OpenAI(max_retries=0, **kw),
client=client,
model=llm_providers.resolve_model(res.provider),
provider_id=res.provider.id,
timeout=_default_timeout(),
+146 -21
View File
@@ -18,10 +18,16 @@ reimplement it:
(+ optional cover art, loudness filter), output as ``m4b`` or ``mp3``.
* ``chapter_cache_key`` deterministic content hash so a re-run reuses
already-rendered chapters (resume) and re-renders only what changed.
* ``segment_cache_key`` / ``SegmentCache`` the inner cache layer: each
spoken span's WAV is content-addressed under ``<cache_dir>/segments`` so
editing one sentence re-renders one segment (not the chapter) and an
interrupted chapter render resumes from its finished segments.
Every function here is pure (string/argv in, string/argv out) so it's unit
tested without ffmpeg, torch, or a GPU. The impure ffmpeg run lives in the
caller (the audiobook router today; the stories job tomorrow).
The builders are pure (string/argv in, string/argv out) so they're unit tested
without ffmpeg, torch, or a GPU; the cache helpers (``prune_cache_dir``,
``SegmentCache``) touch only local files and import torch lazily. The impure
ffmpeg run lives in the caller (the audiobook router today; the stories job
tomorrow).
"""
from __future__ import annotations
@@ -65,30 +71,29 @@ def _escape_meta(value: str) -> str:
def prune_cache_dir(cache_dir: str, max_bytes: int = _CACHE_MAX_BYTES) -> tuple[int, int]:
"""Evict the oldest files in ``cache_dir`` until the total size is within
``max_bytes`` (LRU by mtime). The content-addressed chapter cache otherwise
``max_bytes`` (LRU by mtime). The content-addressed render cache otherwise
grows without bound uncompressed WAVs accumulate across every render.
Best-effort: returns ``(remaining_bytes, removed_count)`` and never raises
(a missing dir / unstattable file is just skipped). Call it *before* writing
a job's chapters so the fresh ones are never the eviction target.
Walks the whole tree, so chapter WAVs at the root and segment WAVs under
``segments/`` share ONE byte budget the cap holds no matter which layer
grew. Best-effort: returns ``(remaining_bytes, removed_count)`` and never
raises (a missing dir / unstattable file is just skipped). Call it *before*
writing a job's files so the fresh ones are never the eviction target.
"""
try:
names = os.listdir(cache_dir)
except OSError:
return (0, 0)
entries: list[tuple[float, int, str]] = []
total = 0
for name in names:
p = os.path.join(cache_dir, name)
try:
if not os.path.isfile(p):
for root, _dirs, names in os.walk(cache_dir):
for name in names:
p = os.path.join(root, name)
try:
if not os.path.isfile(p):
continue
size = os.path.getsize(p)
mtime = os.path.getmtime(p)
except OSError:
continue
size = os.path.getsize(p)
mtime = os.path.getmtime(p)
except OSError:
continue
entries.append((mtime, size, p))
total += size
entries.append((mtime, size, p))
total += size
if total <= max_bytes:
return (total, 0)
entries.sort() # oldest first
@@ -136,6 +141,126 @@ def chapter_cache_key(
return hashlib.sha1(raw.encode("utf-8"), usedforsecurity=False).hexdigest()[:20]
# ── Segment cache (sub-chapter granularity) ─────────────────────────────────
#: Segment WAVs live in a subdirectory of the chapter cache dir so both layers
#: share one root — and one byte cap (``prune_cache_dir`` walks the tree).
SEGMENT_SUBDIR = "segments"
def segment_cache_key(
text: str,
*,
sample_rate: int,
engine_id: str,
voice_id: Optional[str] = None,
voice_sig: str = "",
speed: Optional[float] = None,
extra_sig: str = "",
) -> str:
"""Deterministic content hash for ONE rendered segment (a single spoken
span). Same dimensions as :func:`chapter_cache_key` minus span order and
pauses (pauses are synthesized silence never cached): text, voice
identity (id + resolved signature), speed, sample rate, engine, plus
``extra_sig`` for anything else that changes the rendered audio (the
pronunciation lexicon today). Any change new key re-synthesize just
this segment.
"""
payload = {
"sr": int(sample_rate),
"engine": engine_id or "",
"voice": voice_id or "",
"text": text or "",
"speed": speed,
"voice_sig": voice_sig or "",
"extra": extra_sig or "",
}
raw = json.dumps(payload, sort_keys=True, ensure_ascii=False)
# Content-addressing only — not a security digest (see chapter_cache_key).
return hashlib.sha1(raw.encode("utf-8"), usedforsecurity=False).hexdigest()[:20]
class SegmentCache:
"""Content-addressed per-segment WAV store under ``cache_dir/segments``.
The chapter cache stays the fast outer layer a fully-unchanged chapter
hits at the chapter key and never touches segment files. This inner layer
makes a *changed* chapter cheap: only the edited/new segments synthesize
(the rest load from disk), and an interrupted chapter render resumes from
the segments that already finished, because each segment is persisted the
moment it renders.
``voice_sig`` maps ``voice_id or ""`` resolved-profile signature (same
strings the chapter key uses) so a profile edit invalidates segments too.
Load/store are best-effort: a missing/corrupt/foreign-rate file is a clean
cache miss (re-render), and a failed store never fails the render so
caches written by any app version degrade safely. torch/torchaudio import
lazily to keep this module import-light for the pure-builder callers.
"""
def __init__(
self,
cache_dir: str,
*,
sample_rate: int,
engine_id: str,
voice_sig: Optional[dict] = None,
extra_sig: str = "",
) -> None:
self.dir = os.path.join(cache_dir, SEGMENT_SUBDIR)
self.sample_rate = int(sample_rate)
self.engine_id = engine_id or ""
self.voice_sig = dict(voice_sig or {})
self.extra_sig = extra_sig or ""
self.hits = 0
self.misses = 0
def _path(self, span) -> str:
key = segment_cache_key(
span.text,
sample_rate=self.sample_rate,
engine_id=self.engine_id,
voice_id=span.voice_id,
voice_sig=self.voice_sig.get(span.voice_id or "", ""),
speed=getattr(span, "speed", None),
extra_sig=self.extra_sig,
)
return os.path.join(self.dir, f"{key}.wav")
def load(self, span):
"""Cached audio tensor for ``span``, or ``None`` (miss). A hit bumps
the file's mtime so LRU eviction sees the segment as recently used."""
path = self._path(span)
if not os.path.isfile(path):
self.misses += 1
return None
try:
import torchaudio
audio, sr = torchaudio.load(path)
except Exception:
self.misses += 1
return None # unreadable/corrupt entry — clean miss, re-render
if int(sr) != self.sample_rate or audio.numel() == 0:
self.misses += 1
return None # foreign-rate/empty entry — clean miss, re-render
try:
os.utime(path, None)
except OSError:
pass
self.hits += 1
return audio
def store(self, span, audio) -> None:
"""Persist a freshly rendered segment. Best-effort — a full disk or
unwritable cache dir must never fail the chapter render."""
try:
from services.audio_io import atomic_save_wav
os.makedirs(self.dir, exist_ok=True)
atomic_save_wav(self._path(span), audio, self.sample_rate)
except Exception:
pass
# ── Loudness normalization ──────────────────────────────────────────────────
@dataclass(frozen=True)
+24
View File
@@ -45,11 +45,33 @@ def _asr_device() -> str:
return "cpu"
def _active_tts_id() -> Optional[str]:
"""Configured TTS engine id, or None if it can't be resolved. Attribution
is advisory a prefs/import hiccup must never break /model/loaded."""
try:
from services.tts_backend import active_backend_id
return active_backend_id()
except Exception:
return None
def _tts_attribution(engine_id: str, active: Optional[str]) -> dict:
"""Per-entry engine attribution for TTS-family models. A model can stay
resident in VRAM after the user switches engines (freed only by unload/
idle-evict), so the panel needs to know which entry synthesis actually
routes to. ``is_active_engine`` is None when the active id is unknown."""
return {
"engine_id": engine_id,
"is_active_engine": (engine_id == active) if active is not None else None,
}
def list_loaded() -> dict:
"""Enumerate every currently-loaded model. Shape: ``{"models": [...],
"count": n}`` with per-model id/name/checkpoint/device/vram_mb/unloadable
(+ optional ``note``)."""
models: list[dict] = []
active_tts = _active_tts_id()
# 1. In-process TTS model (OmniVoice)
if mm.model is not None:
@@ -64,6 +86,7 @@ def list_loaded() -> dict:
"device": device,
"vram_mb": round(_tts_vram_mb(), 1),
"unloadable": True,
**_tts_attribution("omnivoice", active_tts),
})
# 2. ASR (WhisperX) — co-loaded with and released alongside the TTS model.
@@ -105,6 +128,7 @@ def list_loaded() -> dict:
"device": get_best_device(),
"vram_mb": round(float(s.get("vram_mb") or 0), 1),
"unloadable": True,
**_tts_attribution(s["id"], active_tts),
})
except Exception:
pass
+191 -48
View File
@@ -277,9 +277,11 @@ def _timeout_guidance(what: str, timeout: float) -> str:
)
return common + (
"most often the GPU is VRAM-starved (a resident model and this job "
"contend for memory). For a durable fix try shorter text, a lighter "
"engine, or set the engine to CPU in Settings → Models. (Raise "
"OMNIVOICE_GENERATE_TIMEOUT_S for very long single generations.)"
"contend for memory). For a durable fix, Flush caches / Unload the "
"resident model (top toolbar or Settings → Models) before retrying, "
"try shorter text, a lighter engine, or set the engine to CPU in "
"Settings → Models. (Raise OMNIVOICE_GENERATE_TIMEOUT_S for very "
"long single generations.)"
)
@@ -655,6 +657,78 @@ def _hf_offline() -> bool:
return _env_flag("HF_HUB_OFFLINE") or _env_flag("TRANSFORMERS_OFFLINE")
# ── Broken-snapshot-link self-heal ───────────────────────────────────
# A sibling of the incomplete-cache class above: the blobs are FULLY
# downloaded, but the snapshots/<rev>/ entries pointing at them are dangling
# symlinks (0 KB) or zero-byte stand-ins — blob-naming mismatches between
# download modes, interrupted renames, or antivirus interference all produce
# this state (reported on Windows, where the NTFS links show as 0 KB, but the
# heal is generic). os.path.isfile() on a dangling link is False, so
# transformers raises the same "does not appear to have a file named …"
# signature even though the bytes are on disk. The resume repair below can't
# fix it (snapshot_download may trust/short-circuit on the existing broken
# entry), so rung 0 of the recovery ladder deletes exactly the broken entries
# and restores them — see services.hf_cache_repair.
# Repos this process already attempted the link self-heal for — the retry
# after a repair may only happen ONCE per repo per process, so a cache that
# stays broken can't loop repair↔retry.
_LINK_REPAIR_ATTEMPTED: set[str] = set()
def _selfheal_broken_snapshot_links(checkpoint: str) -> bool:
"""Rung 0 of cache recovery: delete-and-restore broken snapshot entries.
Returns True only when broken entries were found, removed AND restored
i.e. retrying the load is worth it. At most one attempt per repo per
process. Never raises; when it returns False the legacy resume/force
ladder still runs."""
if checkpoint in _LINK_REPAIR_ATTEMPTED:
return False
_LINK_REPAIR_ATTEMPTED.add(checkpoint)
if os.path.isdir(checkpoint):
return False # a local-directory checkpoint doesn't use the hub cache
try:
from services.hf_cache_repair import repair_repo_cache
summary = repair_repo_cache(checkpoint)
except Exception as repair_err: # repair must never break the ladder
logger.warning("Snapshot-link self-heal for %s errored: %s",
checkpoint, repair_err)
return False
if summary.get("removed") and summary.get("ok"):
logger.warning(
"Model cache for %s had %d broken file link(s) — repaired "
"automatically (%s), retrying the load.",
checkpoint, summary["removed"],
summary.get("outcome") or "healed",
)
return True
if summary.get("found"):
logger.warning(
"Model cache for %s has %d broken file link(s) that could not be "
"auto-repaired (%s).",
checkpoint, summary["found"], summary.get("error") or "unknown",
)
return False
def _manual_cache_delete_hint(checkpoint: str) -> str:
"""Names the exact on-disk folder to delete when every auto-repair rung
failed "delete the model" is only actionable if the user can find it.
Empty for local-directory checkpoints (they don't live in the hub cache)."""
try:
if os.path.isdir(checkpoint):
return ""
from services.hf_cache_repair import repo_cache_dir
return (
f" If the problem persists, quit OmniVoice, delete "
f"{repo_cache_dir(checkpoint)} and restart — the model "
"re-downloads automatically."
)
except Exception:
return ""
# Why the LAST _repair_model_cache run failed ("" when it succeeded / hasn't
# run). #886: the "could not be auto-repaired" message used to drop the cause
# entirely, so a mirror outage, offline mode, or a full disk all read the same.
@@ -850,50 +924,76 @@ def _load_model_sync():
# cache never reaches this branch, so the fast path is untouched).
if not _is_incomplete_cache_error(e):
raise
_set_loading("loading_weights", "Repairing incomplete model cache…")
if not _repair_model_cache(checkpoint):
raise RuntimeError(
f"The TTS model cache for {checkpoint} is incomplete "
"(weights missing — usually an interrupted download)."
f"{_repair_failure_detail()} "
"Open Settings → Models, delete the OmniVoice TTS model, "
"and install it again."
) from e
_set_loading("loading_weights", f"Loading TTS weights on {device}")
try:
_model = _load()
except OSError as e2:
# Resume-repair ran but the cache is still unusable. The usual
# cause beyond "repo genuinely lacks weights" is a blob that's
# present with the right size but corrupt — snapshot_download's
# resume trusts it and never re-fetches it (#739). Force a full
# re-download (replaces corrupt blobs) and retry once more before
# falling back to the manual delete-and-reinstall message.
if _is_incomplete_cache_error(e2):
_set_loading("loading_weights", "Re-downloading model files…")
if _repair_model_cache(checkpoint, force=True):
try:
_model = _load()
except OSError as e3:
# Rung 0: broken snapshot links — the blobs are on disk but the
# snapshot entries don't resolve (dangling symlinks / zero-byte
# stand-ins). Delete exactly the broken entries, restore, and
# retry the load ONCE (guarded per repo per process). A cache
# without broken links falls straight through to the resume
# ladder below.
_model = None
if _selfheal_broken_snapshot_links(checkpoint):
_set_loading(
"loading_weights",
"Model cache had broken file links — repaired "
"automatically, retrying…",
)
try:
_model = _load()
except OSError as e_link:
if not _is_incomplete_cache_error(e_link):
raise
logger.warning(
"Load still failing after snapshot-link repair of %s"
"falling back to resume repair.", checkpoint,
)
e = e_link
_model = None
if _model is None:
_set_loading("loading_weights", "Repairing incomplete model cache…")
if not _repair_model_cache(checkpoint):
raise RuntimeError(
f"The TTS model cache for {checkpoint} is incomplete "
"(weights missing — usually an interrupted download)."
f"{_repair_failure_detail()} "
"Open Settings → Models, delete the OmniVoice TTS model, "
f"and install it again.{_manual_cache_delete_hint(checkpoint)}"
) from e
_set_loading("loading_weights", f"Loading TTS weights on {device}")
try:
_model = _load()
except OSError as e2:
# Resume-repair ran but the cache is still unusable. The usual
# cause beyond "repo genuinely lacks weights" is a blob that's
# present with the right size but corrupt — snapshot_download's
# resume trusts it and never re-fetches it (#739). Force a full
# re-download (replaces corrupt blobs) and retry once more before
# falling back to the manual delete-and-reinstall message.
if _is_incomplete_cache_error(e2):
_set_loading("loading_weights", "Re-downloading model files…")
if _repair_model_cache(checkpoint, force=True):
try:
_model = _load()
except OSError as e3:
raise RuntimeError(
f"The TTS model cache for {checkpoint} is incomplete "
"and could not be auto-repaired. Open Settings → "
"Models, delete the OmniVoice TTS model, and install "
f"it again.{_manual_cache_delete_hint(checkpoint)}"
) from e3
else:
raise RuntimeError(
f"The TTS model cache for {checkpoint} is incomplete "
"and could not be auto-repaired. Open Settings → "
"Models, delete the OmniVoice TTS model, and install "
"it again."
) from e3
f"The TTS model cache for {checkpoint} is incomplete and "
f"could not be auto-repaired.{_repair_failure_detail()} "
"Open Settings → Models, delete the OmniVoice TTS model, "
f"and install it again.{_manual_cache_delete_hint(checkpoint)}"
) from e2
else:
raise RuntimeError(
f"The TTS model cache for {checkpoint} is incomplete and "
f"could not be auto-repaired.{_repair_failure_detail()} "
"Open Settings → Models, delete the OmniVoice TTS model, "
"and install it again."
"could not be auto-repaired. Open Settings → Models, delete "
"the OmniVoice TTS model, and install it again."
f"{_manual_cache_delete_hint(checkpoint)}"
) from e2
else:
raise RuntimeError(
f"The TTS model cache for {checkpoint} is incomplete and "
"could not be auto-repaired. Open Settings → Models, delete "
"the OmniVoice TTS model, and install it again."
) from e2
try:
# plan-02 (#65): gate on Triton availability (+ user setting), not
@@ -955,7 +1055,13 @@ def _load_model_sync():
except Exception: # never let failure-formatting mask the real error
err_msg = str(exc)
_set_loading("error", "Model loading failed", error=err_msg)
logger.error("Model loading failed: %s", str(exc))
# #1000 class: transformers' lazy-import machinery wraps ANY disruption
# to an inner import (including one interrupted by process teardown)
# in a generic "Could not import module X. Are this object's
# requirements defined correctly?" — logging only str(exc) discarded
# the real cause in __cause__/__context__ and made a shutdown race
# look like a broken install. exc_info surfaces the full chain.
logger.error("Model loading failed: %s", str(exc), exc_info=exc)
raise
finally:
unregister_listener(lid)
@@ -1022,6 +1128,22 @@ async def get_model():
return model
def _checkpoint_in_local_cache(checkpoint: str) -> bool:
"""True when ``checkpoint`` is loadable with NO network: an existing local
directory, or a COMPLETE HF cache snapshot. ``snapshot_download(...,
local_files_only=True)`` never constructs an HTTP session, so a broken
proxy env (#959: ``ALL_PROXY``/``HTTPS_PROXY=socks5://`` without socksio)
can't false-negative this probe. Never raises."""
if os.path.isdir(checkpoint):
return True
try:
from huggingface_hub import snapshot_download
snapshot_download(checkpoint, local_files_only=True)
return True
except Exception:
return False
async def preload_model():
"""Background model warm-up — call from lifespan startup.
@@ -1042,10 +1164,27 @@ async def preload_model():
try:
from huggingface_hub import model_info
model_info(checkpoint, timeout=5)
except Exception:
# Model not downloaded yet — skip preload
logger.info("Preload skipped: %s not available locally.", checkpoint)
return
except Exception as probe_err:
# The probe failing does NOT mean the model isn't installed — it
# means the Hub API wasn't reachable from this process. The #959
# class: under ALL_PROXY/HTTPS_PROXY=socks5:// without socksio,
# hf_hub's get_session() raises ImportError AT CLIENT CONSTRUCTION;
# same story for offline mode, DNS, or firewall failures. Fall back
# to a cache-only probe (no HTTP session involved) and warm up
# anyway when the model is locally present, instead of silently
# skipping and letting the first /generate eat the full load.
if not _checkpoint_in_local_cache(checkpoint):
logger.info(
"Preload skipped: %s not available locally (network probe "
"failed: %s: %s).",
checkpoint, type(probe_err).__name__, probe_err,
)
return
logger.warning(
"Network probe for %s failed (%s: %s) — model found in the "
"local cache; warming up from cache.",
checkpoint, type(probe_err).__name__, probe_err,
)
logger.info("Preloading TTS model in background…")
_last_used = time.time()
@@ -1054,7 +1193,11 @@ async def preload_model():
model = await _load_model_with_timeout()
logger.info("Preload complete — model ready.")
except Exception as e:
logger.warning("Model preload failed (non-fatal): %s", e)
# See the matching exc_info note on the _load_model_sync handler above
# (#1000 class) — the full chain, not just str(e), is what actually
# distinguishes a real dependency problem from a shutdown-interrupted
# import.
logger.warning("Model preload failed (non-fatal): %s", e, exc_info=e)
def get_model_status():
is_loaded = model is not None
+111 -7
View File
@@ -10,8 +10,8 @@ plays the moment the video begins and everything feels desynchronised.
``snap_segment_starts`` post-processes segments against the actual audio
(ideally the Demucs-isolated vocals track, which the dub pipeline already
produces): for each segment it scans the waveform inside ``[start, end]``
for the first frame whose RMS rises above an adaptive threshold and moves
``start`` forward to just before that onset.
for the first *sustained* rise of frame RMS above an adaptive threshold
and moves ``start`` forward to just before that onset.
Design constraints:
@@ -23,6 +23,27 @@ Design constraints:
(no frame above the absolute floor) are left untouched.
* **Pure NumPy.** No model, no platform-specific code identical
behaviour on macOS / Windows / Linux, trivially unit-testable.
Robustness against non-speech onsets (#963): a field report showed dubbed
lines starting seconds off because "when a noise is heard (a sigh or
footsteps), it's interpreted as the start of the conversation". Three
guards address that class of failure:
* **Sustained energy.** A frame only counts as an onset when the energy
stays up for a speech-like duration (``SUSTAIN_MIN_S`` within the
following ``SUSTAIN_WINDOW_S``). Footsteps/door thuds/clicks light up
one or two 20 ms frames and die; syllables keep the energy up.
* **Bounded snap distance.** Shifts beyond ``MAX_SNAP_S`` are only
trusted when everything being skipped is (near-)silence the genuine
#280 whisper start-stretch, where Demucs removed the leading music and
left real silence on the vocals track. Jumping far over *audible*
content (e.g. quiet speech sitting under the relative threshold) would
play the dub seconds late, so it is refused.
* **Source-aware.** Snapping only runs on a separated vocals track
(``separated_vocals=True``). On mixed/original audio Demucs skipped
or failed music, ambience and room tone are all legitimate sustained
energy, so any detected "onset" is as likely the score as the speaker;
whisper's own timestamps beat a confidently wrong snap.
"""
from __future__ import annotations
@@ -51,6 +72,23 @@ RELATIVE_THRESHOLD = 0.10
# the whole window is treated as silence and left alone (we'd only be
# snapping to noise).
ABS_RMS_FLOOR = 1e-3
# An onset must be *sustained* to count as speech (#963): within the
# SUSTAIN_WINDOW_S that follows a candidate frame, at least SUSTAIN_MIN_S
# worth of frames must also sit above the threshold. A ~100 ms footstep
# burst fails this; real speech (syllables every few hundred ms) passes.
SUSTAIN_WINDOW_S = 0.30
SUSTAIN_MIN_S = 0.16
# Snaps larger than this are only trusted when the skipped span is
# (near-)silence — see _region_mostly_silent (#963).
MAX_SNAP_S = 1.5
# The skipped span counts as "mostly silent" when at most this fraction of
# its frames is audible. Non-zero so an isolated transient bleeding through
# separation (a footstep) doesn't block a genuine long silence-trim…
SKIPPED_AUDIBLE_FRAC = 0.10
# …where "audible" = above max(ABS_RMS_FLOOR, this fraction of the span's
# own peak); the relative term keeps a slightly raised residual noise floor
# from reading as content.
SKIPPED_FLOOR_PEAK_FRAC = 0.02
def _frame_rms(x: np.ndarray, frame_len: int) -> np.ndarray:
@@ -70,6 +108,15 @@ def detect_speech_onset(
) -> float | None:
"""Return the absolute time (s) of the first speech-like frame inside
``[start_s, end_s]``, or ``None`` when the window is empty / silent.
"Speech-like" requires *sustained* energy (#963): within the
``SUSTAIN_WINDOW_S`` look-ahead after a candidate frame, at least
``SUSTAIN_MIN_S`` worth of frames must also exceed the threshold.
Short broadband transients footsteps, door thuds, mouse clicks
light up one or two 20 ms frames and then die, so they no longer read
as "the conversation started here"; real speech keeps the energy up
across syllables. A candidate too close to the window's end to prove
sustain is rejected (conservative: the ASR timestamp stands).
"""
if sr <= 0 or end_s <= start_s:
return None
@@ -86,10 +133,21 @@ def detect_speech_onset(
if peak < ABS_RMS_FLOOR:
return None # whole window is effectively silent
threshold = max(RELATIVE_THRESHOLD * peak, ABS_RMS_FLOOR)
above = np.nonzero(rms >= threshold)[0]
if above.size == 0:
above = rms >= threshold
candidates = np.nonzero(above)[0]
if candidates.size == 0:
return None
return start_s + float(above[0]) * (frame_len / sr)
frame_s = frame_len / sr
win_frames = max(1, int(round(SUSTAIN_WINDOW_S / frame_s)))
need_frames = max(1, int(round(SUSTAIN_MIN_S / frame_s)))
# counts[k] = above-threshold frames within rms[c : c + win_frames]
# for candidate c — O(n) via a cumulative sum, no per-candidate scan.
cum = np.concatenate(([0], np.cumsum(above)))
counts = cum[np.minimum(candidates + win_frames, above.size)] - cum[candidates]
sustained = candidates[counts >= need_frames]
if sustained.size == 0:
return None # only transient bursts in this window
return start_s + float(sustained[0]) * frame_s
# Hysteresis for full-track onset listing: after a frame crosses the
@@ -138,21 +196,60 @@ def detect_speech_onsets(audio: np.ndarray, sr: int) -> list[float]:
return onsets
def _region_mostly_silent(
audio: np.ndarray,
sr: int,
start_s: float,
end_s: float,
) -> bool:
"""True when ``[start_s, end_s]`` contains (almost) no audible content.
Gates long snaps (> ``MAX_SNAP_S``, #963): jumping far forward is only
trustworthy when everything being skipped is silence the genuine
whisper start-stretch of #280, where Demucs stripped the leading music
and left real silence on the vocals track. A small fraction of audible
frames is tolerated so an isolated transient bleeding through
separation (a footstep) doesn't block the trim; *sustained* audible
content e.g. quiet speech sitting below the relative onset
threshold does block it, because skipping past it would desync the
dub by the full jump.
"""
i0 = max(0, int(start_s * sr))
i1 = min(len(audio), int(end_s * sr))
if i1 <= i0:
return True
rms = _frame_rms(audio[i0:i1], max(1, int(FRAME_S * sr)))
if rms.size == 0:
return True
floor = max(ABS_RMS_FLOOR, SKIPPED_FLOOR_PEAK_FRAC * float(rms.max()))
return float((rms >= floor).mean()) <= SKIPPED_AUDIBLE_FRAC
def snap_segment_starts(
segments: Sequence[dict],
audio: np.ndarray,
sr: int,
*,
min_shift_s: float = MIN_SHIFT_S,
separated_vocals: bool = True,
) -> int:
"""Snap each segment's ``start`` forward to the actual speech onset.
Mutates the segment dicts in place (the shape the dub pipeline passes
around). Returns the number of segments adjusted.
``audio`` should be mono float; the Demucs vocals track gives the best
signal but the mixed track still beats nothing.
``audio`` should be the mono-float **separated vocals** track. When the
caller only has mixed/original audio (Demucs skipped or failed), pass
``separated_vocals=False``: snapping is then disabled entirely (#963) —
on a mixed track music, ambience and footsteps are all sustained energy,
so a detected "onset" is as likely the score as the speaker, and
whisper's own timestamps beat a confidently wrong snap.
"""
if not separated_vocals:
logger.info(
"onset-align: skipped — audio is not a separated vocals track "
"(Demucs unavailable/failed); keeping ASR timestamps as-is")
return 0
if sr <= 0 or audio is None or len(audio) == 0:
return 0
if audio.ndim > 1:
@@ -174,6 +271,13 @@ def snap_segment_starts(
shift = new_start - start
if shift < min_shift_s:
continue
if shift > MAX_SNAP_S and not _region_mostly_silent(audio, sr, start, onset):
# Long jump over audible content (#963): the "onset" is more
# likely a louder late event than the true start — quiet speech
# under the relative threshold would be skipped wholesale and
# the dub would play seconds LATE. Bounded corrections are fine;
# unbounded ones only over true silence (the #280 case).
continue
# Preserve a minimum playable duration.
new_start = min(new_start, end - MIN_SEG_DUR_S)
if new_start - start < min_shift_s:
+54
View File
@@ -223,6 +223,60 @@ def extract_segment_refs(
return out
def refine_ref_text(ref_audio_path: str, asr_backend, fallback_text: str) -> str:
"""Re-transcribe a written reference clip and return that transcript.
`extract_speaker_clones`/`extract_segment_refs` pair each audio slice with
the ASR segment's OWN text field, on the assumption that the segment's
timestamps and its transcribed text agree. They routinely don't — Whisper
(and friends) frequently drift on segment boundaries: a trailing word
audible in `[start, end]` but missing from `text`, or vice versa. When the
(ref_audio, ref_text) pair disagrees, zero-shot TTS prompt-priming breaks
down and the clone can speak the mismatched reference text itself instead
of the target-language text it was given to synthesize (issue #1004).
Re-transcribing the *actual written clip* guarantees the pair matches by
construction the model doesn't care whether the original ASR text was
right, only that ref_text is what's really in ref_audio. `asr_backend` is
the caller's already-loaded active backend (duck-typed:
`.transcribe(path, word_timestamps=...) -> dict` with a `chunks` list of
`{"text": ...}`); the model is already warm, so this costs one more short
transcribe call, not a fresh load. Falls back to `fallback_text` never
raises so a re-transcribe failure is a strict no-op, never a regression
from the original (matching) behavior.
"""
if asr_backend is None:
return fallback_text
try:
result = asr_backend.transcribe(ref_audio_path, word_timestamps=False)
text = " ".join(
(c.get("text") or "").strip() for c in (result.get("chunks") or [])
).strip()
return text or fallback_text
except Exception as e:
logger.warning(
"speaker_clone: re-transcribe of %s failed, keeping original ref_text: %s",
ref_audio_path, e,
)
return fallback_text
def refine_ref_texts(clones: dict[str, dict], asr_backend) -> dict[str, dict]:
"""Apply `refine_ref_text` to every entry's `ref_text` in place.
Batches the whole dict (per-speaker `clones` from `extract_speaker_clones`
or per-segment `seg_clones` from `extract_segment_refs`) into the single
executor round-trip the caller submits to the GPU pool, rather than one
dispatch per reference. Mutates and returns `clones` for a convenient
call-and-reassign at the call site.
"""
for entry in clones.values():
entry["ref_text"] = refine_ref_text(
entry["ref_audio"], asr_backend, entry.get("ref_text", "")
)
return clones
# ── Internals ───────────────────────────────────────────────────────────────
+492
View File
@@ -0,0 +1,492 @@
"""Engine-agnostic text normalization — a conservative pre-pass before TTS.
Raw user text trips TTS engines: digits, clock times, and title abbreviations
mispronounce; zero-width junk and pathological repeat runs cause hallucinations
and long dead air. This module cleans text *once*, at the point where each
pipeline hands text to an engine (single-shot /generate, dub segments,
longform chapters), so every engine benefits equally.
Design rules (load-bearing):
* **Conservative.** A false negative (digits left alone) is fine; a false
positive (mangled meaning) is not. Anything ambiguous thousands-grouped
numbers ("1,000"), ranges ("3-5"), version strings ("v2", "3.5.1"),
leading-zero codes ("007"), 7+-digit IDs is left unchanged. Roman
numerals are out of scope entirely ("I" is a pronoun).
* **Idempotent.** ``normalize_text(normalize_text(x)) == normalize_text(x)``:
number/abbreviation output contains no digits or matchable tokens and the
safety filters are fixed-point by construction, so an accidental second
pass through a pipeline is harmless.
* **Per-language.** Numbers go through ``num2words`` only for languages it
supports (``_NUM2WORDS_LANGS``; the request's ``language`` is a full
display name from frontend/src/languages.json or an ISO-ish code both
resolve via :func:`_num2words_lang`). Everything else keeps its digits.
Clock times / ordinals / currency are English-only (their spoken form is
language-specific); decimals only for locales whose num2words rendering
was vetted. CJK scripts pass through the safety filters untouched no
CJK punctuation is stripped and no words are injected into unsegmented
text.
* **Markup-safe.** The single-bracket grammar (``[voice:]``, ``[pause ]``,
SSML-lite) and inline ``[[]]`` pronunciation overrides are never touched:
the language passes skip every ``[]`` span (same shape as chunked_tts's
``_BRACKET_TAG_RE``), so ``[pause 300ms]`` / ``[rate 0.9]`` stay parseable.
Ordering vs. the pronunciation dictionary (audited 2026-07-10): normalization
runs **BEFORE** ``services.pronunciation.apply_pronunciation`` (and before the
audiobook ``apply_lexicon`` overlay). Rationale from the code:
1. Dictionary respellings are the user's explicit, final say. If
normalization ran second it would re-process them a respelling that
deliberately contains digits or an abbreviation must reach the engine
verbatim.
2. Users already write lexicon entries against display text (the lexicon
docstring's own example is ``{"Dr": "Doctor"}``); entries keyed on
normalized words keep firing, and the dictionary stays the override for
anything the normalizer produced.
3. Inline ``[[]]`` overrides resolve last inside ``apply_pronunciation``
(and their bracketed content is masked here), so the user retains a
per-occurrence override over any normalizer output.
Pinned by ``tests/test_text_normalization.py`` (dictionary-order test).
Gate: prefs key ``text_normalization_enabled`` (default ON) with env override
``OMNIVOICE_TEXT_NORMALIZATION`` the same env-wins contract as
``OMNIVOICE_PRONUNCIATION`` ("0"/"false"/"no"/"off" disable).
:func:`normalize_for_tts` is the gated entry point every pipeline calls; it
never raises normalization is never allowed to break synthesis.
"""
from __future__ import annotations
import logging
import os
import re
from typing import Callable, Optional
logger = logging.getLogger("omnivoice.text_normalization")
ENV_VAR = "OMNIVOICE_TEXT_NORMALIZATION"
PREF_KEY = "text_normalization_enabled"
# ── Language resolution ───────────────────────────────────────────────────────
#
# The `language` kwarg across the app is normally a full display name from
# frontend/src/languages.json ("English", "German", …) — see
# resolve_kokoro_lang_code in services/tts_backend.py — but ISO-ish codes
# ("en", "pt-BR") also flow through dub/API callers. Map both to a num2words
# locale; anything unmapped keeps its digits (false negatives are fine).
_FULL_NAME_TO_CODE = {
"english": "en",
"german": "de",
"spanish": "es",
"french": "fr",
"italian": "it",
"portuguese": "pt",
"dutch": "nl",
"russian": "ru",
"ukrainian": "uk",
"polish": "pl",
"turkish": "tr",
"czech": "cs",
"danish": "da",
"finnish": "fi",
"swedish": "sv",
"norwegian": "no",
"norwegian bokmål": "no",
"norwegian nynorsk": "no",
"romanian": "ro",
"hungarian": "hu",
"indonesian": "id",
"lithuanian": "lt",
"latvian": "lv",
"slovenian": "sl",
"serbian": "sr",
"hebrew": "he",
"persian": "fa",
"azerbaijani": "az",
"vietnamese": "vi",
"kazakh": "kz",
"standard arabic": "ar",
}
# ISO codes whose num2words locale name differs.
_ISO_ALIASES = {"kk": "kz"}
# Locales verified against the pinned num2words (cardinal + basic rendering).
# zh/ja/ko/th are deliberately absent: unsegmented scripts where injecting
# space-delimited words is wrong, and their engines read digits natively.
_NUM2WORDS_LANGS = frozenset({
"en", "de", "es", "fr", "it", "pt", "nl", "ru", "uk", "pl", "tr", "cs",
"da", "fi", "sv", "no", "ro", "hu", "id", "lt", "lv", "sl", "sr", "ar",
"he", "fa", "az", "vi", "kz",
})
# Locales whose num2words decimal rendering was vetted ("drei Komma fünf",
# "три целых пять десятых", …). tr/vi are excluded on purpose: their 0.5
# renders as "fifty" (wrong), so decimals keep their digits there.
_DECIMAL_LANGS = frozenset({
"en", "de", "es", "fr", "it", "pt", "nl", "ru", "uk", "pl", "cs", "da",
"no", "sv", "fi", "ro", "hu", "id",
})
# "50%" → "fifty <word>" only where the spoken percent word is unambiguous.
_PERCENT_WORD = {
"en": "percent",
"de": "Prozent",
"es": "por ciento",
"fr": "pour cent",
"it": "per cento",
"pt": "por cento",
"nl": "procent",
}
_ISO_CODE_RE = re.compile(r"^([a-z]{2,3})(?:[-_]|$)")
def _num2words_lang(language: Optional[str]) -> Optional[str]:
"""Resolve a request language (display name or ISO-ish code) to a
num2words locale, or ``None`` when digits should be left alone."""
if not language:
return None
s = str(language).strip().lower()
if not s or s == "auto":
return None
code = _FULL_NAME_TO_CODE.get(s)
if code:
return code
m = _ISO_CODE_RE.match(s)
if m:
c = _ISO_ALIASES.get(m.group(1), m.group(1))
if c in _NUM2WORDS_LANGS:
return c
return None
# ── Universal safety filters (all languages) ─────────────────────────────────
# Zero-width & bidi controls, C0/C1 controls (except \t \n \r), BOM, U+FFFD.
_ZW_CONTROL_RE = re.compile(
"[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f-\u009f"
"\u200b-\u200f\u202a-\u202e\u2060-\u2064\ufeff\ufffd]"
)
# A tiny, unambiguous HTML-entity leftover set. `&amp;` is decoded only when
# NOT followed by a letter/`#` — so double-encoded junk ("&amp;nbsp;") is left
# alone rather than decoded one layer per pass (idempotency).
_ENTITIES = {
"&nbsp;": " ",
"&quot;": '"',
"&#39;": "'",
"&apos;": "'",
"&hellip;": "",
"&mdash;": "",
"&ndash;": "",
}
_ENTITY_RE = re.compile(
"(?:" + "|".join(re.escape(k) for k in _ENTITIES) + "|&amp;(?![a-zA-Z#]))"
)
# Same ASCII punctuation char repeated more than 3 times → capped at 3
# ("!!!!!!!!" / "........." cause dead air and babble). CJK punctuation and
# letters are deliberately untouched ("Nooooo" is expressive).
_REPEAT_RE = re.compile(r"([!?.,;:~_*#=-])\1{3,}")
_HSPACE_RE = re.compile(r"[^\S\n]+") # horizontal whitespace runs → one space
_NEWLINE_RE = re.compile(r"\n{3,}") # blank-line floods → one blank line
def _safety_filters(text: str) -> str:
out = _ZW_CONTROL_RE.sub("", text)
out = _ENTITY_RE.sub(lambda m: _ENTITIES.get(m.group(0), "&"), out)
out = _REPEAT_RE.sub(lambda m: m.group(1) * 3, out)
out = _HSPACE_RE.sub(" ", out)
out = _NEWLINE_RE.sub("\n\n", out)
return out.strip()
# ── Bracket masking ──────────────────────────────────────────────────────────
#
# Language passes must never rewrite `[…]` spans: `[pause 300ms]` /
# `[rate 0.9]` / `[voice:NAME]` are grammar, and `[[term|replacement]]`
# belongs to the pronunciation layer. Bounded repetition keeps it linear.
_BRACKET_SPAN_RE = re.compile(r"\[[^\][\n]{0,128}\]")
def _outside_brackets(text: str, fn: Callable[[str], str]) -> str:
if "[" not in text:
return fn(text)
parts: list[str] = []
last = 0
for m in _BRACKET_SPAN_RE.finditer(text):
parts.append(fn(text[last:m.start()]))
parts.append(m.group(0))
last = m.end()
parts.append(fn(text[last:]))
return "".join(parts)
# ── Abbreviation expansion ────────────────────────────────────────────────────
#
# Per-language (key, expansion, guard) triples. Matching is case-sensitive
# (a lowercase "st." is NOT the title "St."); lowercase connective keys
# ("e.g.") get an auto-added sentence-initial variant. Guards:
# "cap" — only before a capitalized word (titles precede names; leaves
# street-suffix "Elm St." / "Elm Dr." untouched).
# "digit" — only before a number ("No. 5"; leaves the word "No." alone).
_ABBREVIATIONS: dict[str, list[tuple[str, str, Optional[str]]]] = {
"en": [
("Dr.", "Doctor", "cap"),
("Mr.", "Mister", "cap"),
("Mrs.", "Missus", "cap"),
("Prof.", "Professor", "cap"),
("St.", "Saint", "cap"),
("Mt.", "Mount", "cap"),
("Jr.", "Junior", None),
("Sr.", "Senior", None),
("vs.", "versus", None),
("etc.", "et cetera", None),
("e.g.", "for example", None),
("i.e.", "that is", None),
("approx.", "approximately", None),
("No.", "number", "digit"),
],
"de": [
("Dr.", "Doktor", "cap"),
("Prof.", "Professor", "cap"),
("Nr.", "Nummer", "digit"),
("z.B.", "zum Beispiel", None),
("z. B.", "zum Beispiel", None),
("d.h.", "das heißt", None),
("d. h.", "das heißt", None),
("usw.", "und so weiter", None),
("bzw.", "beziehungsweise", None),
("ca.", "circa", None),
],
"es": [
("Sr.", "Señor", "cap"),
("Sra.", "Señora", "cap"),
("Srta.", "Señorita", "cap"),
("Dr.", "Doctor", "cap"),
("Dra.", "Doctora", "cap"),
("Ud.", "usted", None),
("Uds.", "ustedes", None),
("etc.", "etcétera", None),
("núm.", "número", "digit"),
],
"fr": [
# "M." is deliberately absent: indistinguishable from a middle initial.
("Mme", "Madame", "cap"),
("Mmes", "Mesdames", "cap"),
("Mlle", "Mademoiselle", "cap"),
("Mlles", "Mesdemoiselles", "cap"),
("etc.", "et cetera", None),
("", "numéro", "digit"),
("", "Numéro", "digit"),
],
}
_GUARD_LOOKAHEAD = {
None: "",
"cap": r"(?=\s+[A-ZÀ-ÖØ-Þ])",
"digit": r"(?=\s*\d)",
}
def _compile_abbreviations() -> dict[str, tuple[re.Pattern, dict[str, str]]]:
compiled: dict[str, tuple[re.Pattern, dict[str, str]]] = {}
for lang, entries in _ABBREVIATIONS.items():
entries = list(entries)
# Sentence-initial variants for lowercase connectives ("E.g." → …).
for key, expansion, guard in list(entries):
if key[:1].islower():
cap_key = key[0].upper() + key[1:]
if not any(k == cap_key for k, _, _ in entries):
entries.append((cap_key, expansion[0].upper() + expansion[1:], guard))
entries.sort(key=lambda e: len(e[0]), reverse=True) # longest key wins
lookup = {key: expansion for key, expansion, _ in entries}
alts = []
for key, _, guard in entries:
suffix = r"(?!\w)" if key[-1:].isalnum() else ""
alts.append(f"{re.escape(key)}{suffix}{_GUARD_LOOKAHEAD[guard]}")
# Literal alternation with per-key guards; no nested quantifiers.
pattern = re.compile(r"(?<![\w.])(?:" + "|".join(alts) + ")")
compiled[lang] = (pattern, lookup)
return compiled
_ABBREV_COMPILED = _compile_abbreviations()
def _expand_abbreviations(text: str, lang: str) -> str:
entry = _ABBREV_COMPILED.get(lang)
if entry is None:
return text
pattern, lookup = entry
def _repl(m: re.Match) -> str:
return lookup.get(m.group(0), m.group(0))
return pattern.sub(_repl, text)
# ── Numbers → words ──────────────────────────────────────────────────────────
#
# Every pattern requires clean word boundaries: digits glued to letters
# ("MP3", "v2"), separators ("1,000", "3-5", "1/2", "12:34:56"), leading
# zeros ("007") or 7+ digits (IDs, phone numbers) are all left alone.
# EN-only clock time: H:MM, 0-23 hours. Rejects H:MM:SS (durations).
_TIME_RE = re.compile(r"(?<![\d:.,])([01]?\d|2[0-3]):([0-5]\d)(?![\d:])")
# EN-only ordinal, suffix verified in the callback ("2th" stays as-is).
_ORDINAL_RE = re.compile(r"(?<![\w.,])(\d{1,4})(st|nd|rd|th)\b")
# EN-only dollars: $N or $N.CC. "$1,000" is blocked by the lookahead.
_CURRENCY_RE = re.compile(r"(?<!\w)\$(\d{1,6})(?:\.(\d{2}))?(?![\d.,])")
_PERCENT_RE = re.compile(r"(?<![\w.,])(\d{1,6}(?:\.\d{1,4})?)\s?%")
_DECIMAL_RE = re.compile(
r"(?<![\w.,:/$%-])(\d{1,6})\.(\d{1,6})(?![\w:/%-])(?![.,]\d)"
)
_INTEGER_RE = re.compile(
r"(?<![\w.,:/$%-])(?!0\d)(\d{1,6})(?![\w:/%-])(?![.,]\d)"
)
_ORDINAL_SUFFIX = {1: "st", 2: "nd", 3: "rd"}
def _correct_ordinal_suffix(n: int) -> str:
if 10 <= n % 100 <= 13:
return "th"
return _ORDINAL_SUFFIX.get(n % 10, "th")
def _numbers_to_words(text: str, lang: str) -> str:
try:
from num2words import num2words
except ImportError: # pragma: no cover — direct dependency; belt & braces
return text
def _safe(m: re.Match, render: Callable[[re.Match], str]) -> str:
# Any num2words hiccup leaves this occurrence untouched.
try:
return render(m)
except Exception: # noqa: BLE001 — conservative: never mangle
return m.group(0)
if lang == "en":
def _time(m: re.Match) -> str:
h, mm = int(m.group(1)), int(m.group(2))
hw = num2words(h, lang="en")
if mm == 0:
return f"{hw} o'clock"
if mm < 10:
return f"{hw} oh {num2words(mm, lang='en')}"
return f"{hw} {num2words(mm, lang='en')}"
text = _TIME_RE.sub(lambda m: _safe(m, _time), text)
def _ordinal(m: re.Match) -> str:
n = int(m.group(1))
if m.group(2) != _correct_ordinal_suffix(n):
return m.group(0)
return num2words(n, lang="en", to="ordinal")
text = _ORDINAL_RE.sub(lambda m: _safe(m, _ordinal), text)
def _currency(m: re.Match) -> str:
dollars = int(m.group(1))
if m.group(2) is not None:
amount = float(f"{m.group(1)}.{m.group(2)}")
return num2words(amount, lang="en", to="currency", currency="USD")
unit = "dollar" if dollars == 1 else "dollars"
return f"{num2words(dollars, lang='en')} {unit}"
text = _CURRENCY_RE.sub(lambda m: _safe(m, _currency), text)
percent_word = _PERCENT_WORD.get(lang)
if percent_word:
def _percent(m: re.Match) -> str:
raw = m.group(1)
if "." in raw:
if lang not in _DECIMAL_LANGS:
return m.group(0)
value: object = float(raw)
else:
value = int(raw)
return f"{num2words(value, lang=lang)} {percent_word}"
text = _PERCENT_RE.sub(lambda m: _safe(m, _percent), text)
if lang in _DECIMAL_LANGS:
def _decimal(m: re.Match) -> str:
return num2words(float(f"{m.group(1)}.{m.group(2)}"), lang=lang)
text = _DECIMAL_RE.sub(lambda m: _safe(m, _decimal), text)
def _integer(m: re.Match) -> str:
raw = m.group(1)
n = int(raw)
if len(raw) == 4 and 1500 <= n <= 2099:
# Bare 4-digit numbers in this range read as years
# ("nineteen eighty-four"); fall back to cardinal where the
# locale has no year form (sv, vi).
try:
return num2words(n, lang=lang, to="year")
except Exception: # noqa: BLE001
pass
return num2words(n, lang=lang)
return _INTEGER_RE.sub(lambda m: _safe(m, _integer), text)
# ── Public API ───────────────────────────────────────────────────────────────
def normalize_text(text: str, language: Optional[str] = None) -> str:
"""Pure, idempotent normalization pass (no pref gate — see
:func:`normalize_for_tts` for the gated entry point pipelines call)."""
if not text:
return text or ""
out = _safety_filters(text)
lang = _num2words_lang(language)
if lang:
if lang in _ABBREV_COMPILED:
out = _outside_brackets(out, lambda t: _expand_abbreviations(t, lang))
out = _outside_brackets(out, lambda t: _numbers_to_words(t, lang))
return out
def normalization_enabled() -> bool:
"""Env wins (power-user override, mirrors OMNIVOICE_PRONUNCIATION);
otherwise the ``text_normalization_enabled`` pref, default ON."""
env = os.environ.get(ENV_VAR)
if env is not None:
return env.strip().lower() not in ("0", "false", "no", "off", "")
try:
from core import prefs
return bool(prefs.get(PREF_KEY, True))
except Exception: # noqa: BLE001 — prefs unreadable → default ON
return True
def normalize_for_tts(text: str, language: Optional[str] = None) -> str:
"""Gated + hardened entry point: pref/env toggle, never raises.
Every TTS pipeline calls this exactly once, at its textengine choke
point, BEFORE the pronunciation dictionary (see module docstring).
"""
if not text:
return text or ""
if not normalization_enabled():
return text
try:
return normalize_text(text, language)
except Exception: # noqa: BLE001 — normalization must never break synth
logger.warning("text normalization failed; using raw text", exc_info=True)
return text
+282
View File
@@ -0,0 +1,282 @@
"""
Two-stage translation quality for the LLM dub engine (provider="openai").
Stage 1 auto-glossary. ONE up-front LLM pass over the full transcript
extracts a short theme summary plus a sourcetarget terminology map for the
target language. The caller merges it with the user's manual glossary
(user entries always win) and injects the result into every per-segment
translation prompt, so recurring names/terms are rendered the same way in
segment 3 and segment 300. The extraction result is cached on the dub job
dict (``job["translation_context"][target_lang]``) and rides the existing
``job_data`` JSON blob no schema change; a transcript fingerprint keys the
cache so edited segments re-extract.
Stage 2 reflect pass. After a segment's direct LLM translation, a
critique-then-rewrite step reviews the draft for wordiness / stiff or
unnatural register and produces the final natural line. It runs on the SAME
client/model the translation used (the dub_translation skill's provider).
Failure policy for BOTH stages: refinement must never fail a segment. Any
error, timeout, empty output, or divergent rewrite silently keeps the direct
translation callers get ``None`` back and move on.
MT engines (argos/nllb/google/deepl/) never reach this module: they have no
prompts to inject into and no LLM to critique with. The Cinematic/Autofit
refine for those engines lives in ``services/translator.py``.
"""
from __future__ import annotations
import hashlib
import logging
import os
from typing import Iterable, Optional
logger = logging.getLogger("omnivoice.translation_quality")
# ── Prompts ──────────────────────────────────────────────────────────────────
# The context pass runs ONCE per (job, target language, transcript); the
# reflect prompts run twice per segment — keep them short, verbosity = wall time.
_CONTEXT_PROMPT = """\
You are a dubbing terminology editor preparing a translation brief. The user
gives you the full source-language transcript of one video. Reply in this
exact plain-text format (no JSON, no code fences, no commentary):
THEME: one or two sentences what the video is about, its register
(casual / formal / technical) and audience.
TERM: SOURCE || TARGET
TERM: SOURCE || TARGET
TERM lines list proper nouns (people, places, brands, product names) and
recurring domain terms that must be translated identically every time, each
with your preferred {target_name} rendering. At most {max_terms} TERM lines;
fewer is better. Skip one-off words and anything trivially consistent."""
_REVIEW_PROMPT = """\
You are a dubbing script reviewer. The user gives you a source line and its
draft {target_name} translation. In 1-2 short sentences, point out where the
draft is wordy, stiff, or uses a register nobody would use in spoken
dialogue, and whether recurring terms follow the brief. If the draft already
sounds natural, say so. Reply ONLY with the critique no headers, no lists,
no code fences."""
_POLISH_PROMPT = """\
You are a dubbing script writer. Rewrite the draft translation using the
reviewer's notes so it reads like natural spoken {target_name}. Keep the
meaning faithful to the source line, keep required terminology, and never add
content that is not in the source. Prefer the same length or shorter than the
draft. The output MUST stay in the same language and script as the draft
never switch language or transliterate. Reply ONLY with the final translation
no quotes, no notes, no commentary."""
def _chat(client, model: str, timeout: float, *, system: str, user: str) -> str:
"""One-shot chat completion on the caller's client. Raises on failure."""
res = client.chat.completions.create(
model=model,
timeout=timeout,
temperature=0.2, # pinned like the direct-translate path — 1.0 drifts
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
)
return (res.choices[0].message.content or "").strip()
# ── Stage 1: auto-glossary (theme + terminology) ────────────────────────────
def transcript_fingerprint(segment_texts: Iterable[str]) -> str:
"""Stable hash of the transcript, so the per-job context cache invalidates
when the user edits segments between translate runs."""
h = hashlib.sha256()
for t in segment_texts:
h.update((t or "").strip().encode("utf-8", errors="replace"))
h.update(b"\x00")
return h.hexdigest()[:16]
def extract_context_sync(
client,
model: str,
timeout: float,
*,
segment_texts: Iterable[str],
source_lang: str,
target_lang: str,
source_name: Optional[str] = None,
target_name: Optional[str] = None,
max_terms: int = 30,
) -> Optional[dict]:
"""One LLM pass over the whole transcript → ``{"theme", "terms"}``.
``terms`` is ``[{"source", "target"}]``. Returns None on ANY failure or
when the response yields neither a theme nor terms the caller proceeds
without context, never errors. Blocking; run in an executor.
"""
text = "\n".join(t.strip() for t in segment_texts if t and t.strip())
if not text:
return None
# Same cap as the explicit glossary auto-extract endpoint — one shared
# knob for "how much transcript may ride a single LLM context call".
try:
max_chars = int(os.environ.get("OMNIVOICE_GLOSSARY_MAX_CHARS", "12000"))
except ValueError:
max_chars = 12000
if len(text) > max_chars:
text = text[:max_chars] + "\n…[truncated]"
system = _CONTEXT_PROMPT.format(
target_name=target_name or target_lang, max_terms=max_terms,
)
user = (
f"Source language: {source_name or source_lang}\n"
f"Target language: {target_name or target_lang}\n"
f"Transcript:\n{text}"
)
try:
body = _chat(client, model, timeout, system=system, user=user)
except Exception as e: # noqa: BLE001 — context is an enhancement, never a gate
logger.warning("auto-glossary context pass failed: %s", e)
return None
theme = ""
terms: list[dict] = []
for line in body.splitlines():
line = line.strip()
if not line:
continue
upper = line.upper()
if upper.startswith("THEME:"):
theme = line[len("THEME:"):].strip()
continue
if upper.startswith("TERM:"):
line = line[len("TERM:"):].strip()
if "||" not in line:
continue
parts = [p.strip() for p in line.split("||")]
if len(parts) < 2 or not parts[0] or not parts[1]:
continue
terms.append({"source": parts[0], "target": parts[1]})
if len(terms) >= max_terms:
break
if not theme and not terms:
logger.warning("auto-glossary context pass returned nothing parseable")
return None
return {"theme": theme, "terms": terms}
def merge_glossary(
user_terms: Optional[Iterable[dict]],
auto_terms: Optional[Iterable[dict]],
) -> list[dict]:
"""Merge manual + auto glossaries. User entries ALWAYS win: an auto term
whose source matches a user source (case-insensitive) is dropped."""
merged: list[dict] = []
seen: set[str] = set()
for entry in user_terms or []:
src = (entry.get("source") or "").strip()
tgt = (entry.get("target") or "").strip()
if not src or not tgt:
continue
merged.append(entry)
seen.add(src.lower())
for entry in auto_terms or []:
src = (entry.get("source") or "").strip()
tgt = (entry.get("target") or "").strip()
if not src or not tgt or src.lower() in seen:
continue
merged.append({"source": src, "target": tgt})
seen.add(src.lower())
return merged
def context_clause(theme: str, terms: Optional[Iterable[dict]]) -> str:
"""Prompt fragment carrying the theme + merged glossary into every
per-segment translation prompt. Empty string when there's nothing."""
parts: list[str] = []
theme = (theme or "").strip()
if theme:
parts.append(f"Video context: {theme}")
lines = []
for entry in terms or []:
src = (entry.get("source") or "").strip()
tgt = (entry.get("target") or "").strip()
if not src or not tgt:
continue
note = (entry.get("note") or "").strip()
lines.append(f"- {src}{tgt}" + (f" (note: {note})" if note else ""))
if lines:
parts.append(
"Terminology — render every occurrence of a source term exactly "
"as its target:\n" + "\n".join(lines)
)
return "\n".join(parts)
# ── Stage 2: reflect pass (critique → rewrite) ──────────────────────────────
def reflect_translation_sync(
client,
model: str,
timeout: float,
*,
source_text: str,
direct_text: str,
source_lang: str,
target_lang: str,
target_name: Optional[str] = None,
extra_clause: str = "",
) -> Optional[str]:
"""Critique-then-rewrite the direct translation of one segment.
Returns the polished line, or None whenever the direct translation should
stand: any LLM failure/timeout, an empty rewrite, or a rewrite that
diverged from the draft (wrong script, runaway length, critique echoed
back the shared ``refine_output_ok`` guard). Never raises. Blocking;
run in an executor.
"""
if not direct_text or not direct_text.strip():
return None
tgt_name = target_name or target_lang
def _with_clause(base: str) -> str:
return base + "\n\n" + extra_clause if extra_clause.strip() else base
try:
review_user = (
f"Source ({source_lang}): {source_text}\n"
f"Draft translation ({target_lang}): {direct_text}"
)
critique = _chat(
client, model, timeout,
system=_with_clause(_REVIEW_PROMPT.format(target_name=tgt_name)),
user=review_user,
)
polish_user = review_user + f"\nReviewer's notes: {critique}"
polished = _chat(
client, model, timeout,
system=_with_clause(_POLISH_PROMPT.format(target_name=tgt_name)),
user=polish_user,
)
except Exception as e: # noqa: BLE001 — refinement must never fail a segment
logger.warning("reflect pass failed (%s) — keeping direct translation", e)
return None
polished = (polished or "").strip()
if not polished or polished == direct_text:
return None
# Same divergence guard the Cinematic ADAPT step uses: wrong script,
# runaway length, or the critique leaking through as the "translation".
from services.translator import refine_output_ok
ok, reason = refine_output_ok(direct_text, polished, target_lang, critique=critique)
if not ok:
logger.warning(
"reflect pass diverged for %s (%s) — keeping direct translation",
target_lang, reason,
)
return None
return polished
+468 -19
View File
@@ -6,7 +6,7 @@ A uniform protocol for every TTS engine. Today we ship:
OmniVoiceBackend wraps the current k2-fsa/OmniVoice model. Zero
behaviour change for existing callers.
VoxCPM2Backend thin stub that raises with a clear install hint
until `pip install voxcpm` is present and enabled.
until `pip install "voxcpm>=2.0.3"` is present and enabled.
Callers should use `get_active_tts_backend()` to pick the configured engine
instead of importing a specific class. The selection is controlled by the
@@ -142,13 +142,40 @@ class TTSBackend(ABC):
#: (e.g. "young female, warm tone, British accent") without reference audio.
supports_voice_design: bool = False
def ensure_ready(self) -> None:
"""Load model weights now (blocking), so callers can separate the
LOAD budget from the GENERATE budget (#1033/#1037 class).
Every adapter lazily loads inside ``generate()`` via a private
``_ensure_loaded()`` which meant a cold first call spent its whole
``OMNIVOICE_GENERATE_TIMEOUT_S`` window (default 300s) downloading /
loading weights and got killed with a misleading "too heavy for the
available compute" error (measured in the wild on a fresh install:
multi-GB checkpoint download, 0% GPU util, #1014). Routes call this
first under the model-load budget (``OMNIVOICE_MODEL_LOAD_TIMEOUT``,
default 1200s), then start the generate clock on an already-warm
engine. Default implementation dispatches to the adapter's own
``_ensure_loaded`` when present; engines without lazy state no-op.
Must be called on the GPU pool (it's blocking), same as generate.
"""
loader = getattr(self, "_ensure_loaded", None)
if callable(loader):
loader()
#: Whether this engine already emits mastered, studio-grade audio and should
#: therefore skip the shared apply_mastering() chain (Compressor + Reverb,
#: therefore skip the shared apply_mastering() chain (highpass + Compressor,
#: tuned for OmniVoice's 24 kHz output). Studio engines like VoxCPM2 (native
#: 48 kHz) set this True so their clean output isn't pumped/reverbed. Loudness
#: 48 kHz) set this True so their clean output isn't pumped. Loudness
#: normalisation is applied regardless — it's a benign peak scale.
applies_own_mastering: bool = False
#: Whether this engine can clone an arbitrary voice from reference audio
#: (`ref_audio=`), as opposed to only offering a fixed set of preset
#: voices. Default True — most engines clone. Dub/batch gate on this
#: (issue #312 class) before committing to a job that needs it, instead
#: of silently falling back to OmniVoice or mis-cloning per segment.
supports_cloning: bool = True
#: GPU/accelerator targets the engine can run on. Surfaced via the
#: Engine Compatibility Matrix (Plan 02-04 / ENGINE-06) so users can
#: tell at a glance which engines will use their hardware. Defaults to
@@ -380,9 +407,155 @@ class OmniVoiceBackend(TTSBackend):
# ── VoxCPM2 adapter (optional, scaffolded) ──────────────────────────────────
#: Minimum recommended `voxcpm` package version. 2.0.3 fixed an audio-quality
#: bug on Apple Silicon (low-precision dtypes on the MPS device produced
#: degraded output). A floor, NOT a pin: newer versions are fine, and an
#: already-installed older version keeps working — we only surface an upgrade
#: hint (is_available reason + load-time warning), never force a reinstall.
_VOXCPM_MIN_VERSION = "2.0.3"
#: Reference-clip cap for VoxCPM2 cloning (seconds). The `voxcpm` package no
#: longer trims reference audio internally, so an unbounded user clip would
#: condition the model on minutes of audio (slow, and past a point it stops
#: helping voice similarity). 30 s is a conservative upper bound.
_VOXCPM_REF_MAX_S = 30.0
#: Silence pad kept around the voiced region when trimming a reference clip —
#: a hard cut exactly at the first/last voiced sample clips consonant onsets.
_VOXCPM_REF_EDGE_PAD_S = 0.05
def _version_tuple(v: str) -> Optional[tuple[int, ...]]:
"""Parse the leading numeric components of a version string ("2.0.3"
(2, 0, 3), "2.1rc1" (2, 1)). Returns None when nothing numeric parses
callers treat that as 'unknown, assume fine' rather than failing."""
parts: list[int] = []
for piece in v.split("."):
digits = ""
for ch in piece:
if not ch.isdigit():
break
digits += ch
if not digits:
break
parts.append(int(digits))
return tuple(parts) if parts else None
def _voxcpm_installed_version() -> Optional[str]:
"""Installed `voxcpm` dist version, or None when undeterminable
(not installed, or importable without package metadata)."""
try:
from importlib.metadata import version
return version("voxcpm")
except Exception:
return None
def _voxcpm_upgrade_hint() -> Optional[str]:
"""Actionable upgrade hint when the installed `voxcpm` is older than
:data:`_VOXCPM_MIN_VERSION`, else None. Never raises; an unparseable or
unknown version yields None (don't nag users we can't be sure about)."""
installed = _voxcpm_installed_version()
if installed is None:
return None
have = _version_tuple(installed)
want = _version_tuple(_VOXCPM_MIN_VERSION)
if have is None or want is None or have >= want:
return None
return (
f"installed voxcpm {installed} is older than {_VOXCPM_MIN_VERSION}, "
"which fixed an audio-quality bug on Apple Silicon (low-precision "
"dtypes on MPS). The engine still works, but upgrading is "
'recommended: pip install --upgrade "voxcpm>=2.0.3"'
)
# Prepared-reference cache: (abspath, mtime_ns, size) → prepared path (which
# may be the original path itself when no trim/cap applied). Keeps repeat
# generations from re-reading + re-writing the same clip, and keeps the temp
# dir from filling with one copy per generate() call.
_VOXCPM_REF_PREP_CACHE: dict[tuple, str] = {}
def _prepare_voxcpm_ref(path: str) -> str:
"""Prepare a cloning reference clip for VoxCPM2.
The `voxcpm` package used to trim reference audio itself but no longer
does raw user clips reach the model unconditioned. This applies the
minimal, conservative preparation the model expects:
trim leading/trailing near-silence (amplitude threshold at the same
-50 dBFS floor `audio_dsp.normalize_audio` uses, with a small
:data:`_VOXCPM_REF_EDGE_PAD_S` pad kept on each side), and
cap the reference at :data:`_VOXCPM_REF_MAX_S` seconds from the
trimmed start.
Returns a path to the prepared WAV. Deliberately non-destructive and
fail-open: the ORIGINAL path is returned unchanged when the clip needs no
meaningful trim/cap (short clean clips pass through untouched), when the
whole clip sits below the silence floor (nothing to anchor a trim on), or
when anything at all goes wrong reference prep must never be the reason
a generation fails.
"""
try:
import numpy as np
import soundfile as sf
abspath = os.path.abspath(path)
st = os.stat(abspath)
cache_key = (abspath, st.st_mtime_ns, st.st_size)
cached = _VOXCPM_REF_PREP_CACHE.get(cache_key)
if cached is not None and (cached == abspath or os.path.exists(cached)):
return cached
audio, sr = sf.read(abspath, dtype="float32", always_2d=True) # (n, ch)
n = audio.shape[0]
if n == 0 or sr <= 0:
return path
# Silence floor: -50 dBFS, matching audio_dsp.normalize_audio. A clip
# that never rises above it is left alone (fail-open, see docstring).
floor = 10 ** (-50.0 / 20.0)
envelope = np.abs(audio).max(axis=1)
voiced = np.flatnonzero(envelope > floor)
if voiced.size == 0:
_VOXCPM_REF_PREP_CACHE[cache_key] = abspath
return path
pad = int(_VOXCPM_REF_EDGE_PAD_S * sr)
start = max(0, int(voiced[0]) - pad)
end = min(n, int(voiced[-1]) + 1 + pad)
cap = int(_VOXCPM_REF_MAX_S * sr)
end = min(end, start + cap)
# No-op path: nothing meaningful to cut (>0.1 s total) — hand the
# original file to the model byte-identical.
if (start + (n - end)) <= int(0.1 * sr):
_VOXCPM_REF_PREP_CACHE[cache_key] = abspath
return path
import tempfile
fd, prepared = tempfile.mkstemp(prefix="voxcpm_ref_", suffix=".wav")
os.close(fd)
sf.write(prepared, audio[start:end], sr)
_VOXCPM_REF_PREP_CACHE[cache_key] = prepared
logger.info(
"VoxCPM2: prepared reference clip %s%s (%.2fs → %.2fs; "
"silence trimmed, cap %.0fs)",
path, prepared, n / sr, (end - start) / sr, _VOXCPM_REF_MAX_S,
)
return prepared
except Exception as e: # noqa: BLE001 — prep is best-effort by contract
logger.warning(
"VoxCPM2: reference-clip preparation failed for %s — using the "
"raw clip: %s", path, e,
)
return path
class VoxCPM2Backend(TTSBackend):
"""OpenBMB VoxCPM2 wrapper — `pip install voxcpm` required.
"""OpenBMB VoxCPM2 wrapper — `pip install "voxcpm>=2.0.3"` required.
Ships as a scaffold: the class loads and reports unavailability cleanly
when the dep isn't installed, so Settings UI can gate the engine selector
@@ -410,10 +583,17 @@ class VoxCPM2Backend(TTSBackend):
import voxcpm # noqa: F401
except ImportError:
return False, (
"voxcpm package not installed. Install with `pip install voxcpm` "
"voxcpm package not installed. Install with "
'`pip install "voxcpm>=2.0.3"` '
"(requires Python ≥3.10, PyTorch ≥2.5). CUDA ≥12 recommended "
"for full speed; MPS (Apple Silicon) and CPU also supported."
)
# Version FLOOR, not pin: an older install still reports available
# (no forced reinstall), but the reason carries the upgrade hint and
# _ensure_loaded() logs it at load time.
hint = _voxcpm_upgrade_hint()
if hint:
return True, f"ready — {hint}"
return True, "ready"
@property
@@ -435,6 +615,9 @@ class VoxCPM2Backend(TTSBackend):
ok, msg = self.is_available()
if not ok:
raise RuntimeError(f"VoxCPM2 unavailable: {msg}")
hint = _voxcpm_upgrade_hint()
if hint:
logger.warning("VoxCPM2: %s", hint)
from voxcpm import VoxCPM # type: ignore[import-not-found]
checkpoint = os.environ.get("OMNIVOICE_VOXCPM_MODEL", "openbmb/VoxCPM2")
logger.info("Loading VoxCPM2 from %s", checkpoint)
@@ -464,14 +647,16 @@ class VoxCPM2Backend(TTSBackend):
cfg_value=kw.get("guidance_scale", 2.0),
inference_timesteps=kw.get("num_step", 10),
)
if isinstance(wav, np.ndarray):
wav = torch.from_numpy(wav).float()
if wav.ndim == 1:
wav = wav.unsqueeze(0)
return wav
return self._finalize(wav)
# ── Standard clone / instruct mode ──────────────────────────────
# Map our instruct prop onto VoxCPM2's inline "(instruct)prompt" prefix.
# The reference clip is prepared first (edge-silence trim + length
# cap) — the model no longer trims it internally, so a raw user clip
# would condition generation on dead air. Fail-open: on any prep
# problem the raw path is used, exactly as before.
if ref_audio:
ref_audio = _prepare_voxcpm_ref(ref_audio)
prompt = text
if instruct:
prompt = f"({instruct}){text}"
@@ -483,11 +668,26 @@ class VoxCPM2Backend(TTSBackend):
prompt_wav_path=ref_audio if ref_text else None,
prompt_text=ref_text,
)
return self._finalize(wav)
def _finalize(self, wav) -> torch.Tensor:
"""Normalize model output to a (1, n) float tensor and apply the
trailing-silence guard.
The guard is a SILENCE trim only: generations often end with a long
near-silent tail, which this cuts (keeping a short ~0.3 s natural
tail). It deliberately does NOT attempt to detect or judge trailing
*content* an output that ends in audible audio, wanted or not,
passes through unchanged, as does any output without a silent tail.
"""
import numpy as np
from services.audio_dsp import trim_trailing_silence
if isinstance(wav, np.ndarray):
wav = torch.from_numpy(wav).float()
if wav.ndim == 1:
wav = wav.unsqueeze(0)
return wav
return trim_trailing_silence(wav, self.sample_rate)
# ── MOSS-TTS-Nano adapter (tiny, CPU-friendly, 20 langs) ────────────────────
@@ -603,6 +803,7 @@ class KittenTTSBackend(TTSBackend):
display_name = "KittenTTS (English, 8 preset voices, CPU realtime)"
# KittenTTS ships as an ONNX CPU graph; no CUDA/MPS path today.
gpu_compat = ("cpu",)
supports_cloning = False # fixed preset voices only; ref_audio is ignored
PRESET_VOICES = [
"expr-voice-2-m", "expr-voice-2-f",
@@ -683,6 +884,54 @@ class KittenTTSBackend(TTSBackend):
# ── MLX-Audio (mac-ARM engine multiplexer) ──────────────────────────────────
# #977: Kokoro's own ALIASES table (mlx_audio.tts.models.kokoro.pipeline) only
# recognizes ISO-ish tokens ("en", "es", "fr-fr", "pt-br", …) — it has no idea
# what a full language name is. OmniVoice's `language` kwarg is normally a
# full display name from frontend/src/languages.json (e.g. "Dutch",
# "Spanish"), forwarded verbatim by the frontend and by
# `OmniVoiceBackend.generate()`. Translate the subset Kokoro actually
# supports to the ISO token its own ALIASES expects; a caller that already
# passes an ISO code (or one of Kokoro's own single-letter codes) is
# resolved unchanged by `resolve_kokoro_lang_code()` below.
_KOKORO_ISO_BY_FULL_NAME = {
"english": "en",
"spanish": "es",
"french": "fr",
"hindi": "hi",
"italian": "it",
"portuguese": "pt",
"japanese": "ja",
"chinese": "zh",
}
def resolve_kokoro_lang_code(language: str) -> str:
"""Map a full language name / ISO code to Kokoro's single-letter
`lang_code`, against the AUTHORITATIVE table read from the installed
mlx-audio package (never a hardcoded guess the vendored table is the
only source of truth and can change across mlx-audio versions).
Raises ``ValueError`` for anything Kokoro doesn't support, naming what
it *does* support instead of forwarding a bogus code into Kokoro's
`assert lang_code in LANG_CODES`, which crashes with an unreadable
``(lang_code, LANG_CODES)`` tuple/dict repr (#977).
"""
from mlx_audio.tts.models.kokoro.pipeline import ALIASES, LANG_CODES
key = language.strip().lower()
iso = _KOKORO_ISO_BY_FULL_NAME.get(key, key)
code = ALIASES.get(iso, iso)
if code not in LANG_CODES:
supported = ", ".join(sorted(name.title() for name in _KOKORO_ISO_BY_FULL_NAME))
raise ValueError(
f"mlx-audio's Kokoro model (mlx-community/Kokoro-82M-bf16) doesn't "
f"support language={language!r}. Kokoro supports: {supported}. "
f"Pick one of those, leave language as 'Auto', or switch to a "
f"multilingual engine (e.g. OmniVoice) for other languages."
)
return code
class MLXAudioBackend(TTSBackend):
"""Blaizzy/mlx-audio — Apple-Silicon-only wrapper over 14+ TTS engines
(Kokoro, CSM, Dia, Qwen3-TTS, Chatterbox, MeloTTS, OuteTTS, Spark,
@@ -721,7 +970,16 @@ class MLXAudioBackend(TTSBackend):
def __init__(self):
self._model = None
self._sr = 24000 # most mlx-audio engines emit 24 kHz mono
key = os.environ.get("OMNIVOICE_MLX_AUDIO_MODEL", self.DEFAULT_MODEL_KEY)
# Env var > persisted UI choice (#981 — Settings → Engines curated-
# model picker) > default. Mirrors active_backend_id()'s resolution
# order exactly so power-users can still pin a model without the UI
# silently undoing it.
from core import prefs
key = prefs.resolve(
"mlx_audio_model_id",
env="OMNIVOICE_MLX_AUDIO_MODEL",
default=self.DEFAULT_MODEL_KEY,
)
# Accept either a curated key ("kokoro") or a full HF repo id
# ("mlx-community/Kokoro-82M-bf16") — flexibility for power users.
self._model_id = self.CURATED_MODELS.get(key, key)
@@ -759,6 +1017,18 @@ class MLXAudioBackend(TTSBackend):
# silently ignores languages it doesn't know.
return ["multi"]
@property
def supports_cloning(self) -> bool:
"""Model-dependent — this adapter multiplexes 7+ curated models and
only some take a reference-audio speaker prompt. `generate()` passes
`ref_audio` through when present (~kwargs below) but silently retries
without it on a TypeError, so an engine picked for cloning that's
actually running Kokoro/Qwen3-TTS/etc. would clone nothing. Of the
curated set, only CSM (`mlx-community/csm-1b-8bit`) is confirmed to
accept a reference prompt default False for every other model,
curated or user-supplied, until positively confirmed."""
return self._model_id == self.CURATED_MODELS.get("csm")
def _ensure_loaded(self):
if self._model is not None:
return
@@ -772,6 +1042,7 @@ class MLXAudioBackend(TTSBackend):
voice = kw.get("voice")
ref_audio = kw.get("ref_audio")
ref_text = kw.get("ref_text")
language = kw.get("language")
speed = float(kw.get("speed", 1.0))
@@ -782,7 +1053,31 @@ class MLXAudioBackend(TTSBackend):
kwargs = {"text": text, "speed": speed}
if voice: kwargs["voice"] = voice
if ref_audio: kwargs["ref_audio"] = ref_audio
if language: kwargs["lang_code"] = language[:2].lower()
# CSM (sesame.py) only builds its cloning context when BOTH ref_audio
# AND ref_text are present — with ref_text missing, its context list
# stays empty and indexing into it raises an opaque
# "IndexError: list index out of range" deep inside mlx-audio,
# instead of ever attempting the clone. Community-diagnosed (#1012).
if ref_audio and ref_text: kwargs["ref_text"] = ref_text
if language and language != "Auto":
if self._model_id == self.CURATED_MODELS.get("kokoro"):
# Kokoro's vendored pipeline hard-asserts `lang_code` against
# its own single-letter table — a bogus code crashes with an
# unreadable AssertionError instead of failing cleanly
# (#977). Resolve against the authoritative installed table
# instead of guessing via `language[:2]`.
kwargs["lang_code"] = resolve_kokoro_lang_code(language)
else:
# `lang_code`-as-2-letter-truncation is Kokoro's own
# convention, not mlx-audio's in general — other curated
# models either ignore unrecognized kwargs (CSM/Dia/OuteTTS
# accept **kwargs and drop it) or expect something else
# entirely (Qwen3-TTS's own docstring: "lang_code: Language
# code (auto, chinese, english, etc.)" — a full name, not a
# 2-letter code). Kokoro's strict validation doesn't apply to
# them, so don't reject a language that's valid for whatever
# model is actually active.
kwargs["lang_code"] = language[:2].lower()
pieces = []
try:
@@ -1111,6 +1406,7 @@ class SherpaOnnxBackend(TTSBackend):
# Sherpa-ONNX uses the onnxruntime providers — CPU is the universal
# baseline; CUDA provider is available on Linux/Windows installs.
gpu_compat = ("cuda", "cpu")
supports_cloning = False # VITS speaker-id only; no ref_audio support
def __init__(self):
self._tts = None
@@ -1336,7 +1632,7 @@ _INSTALL_HINTS: dict[str, str] = {
"cosyvoice": "git clone --recursive FunAudioLLM/CosyVoice + pip install -r requirements.txt + SoX",
"kittentts": "pip install kittentts (ONNX, CPU-only, ~80 MB)",
"mlx-audio": "pip install mlx-audio (Apple Silicon only)",
"voxcpm2": "pip install voxcpm (CPU/MPS supported; CUDA recommended for speed)",
"voxcpm2": 'pip install "voxcpm>=2.0.3" (floor: 2.0.3 fixed Apple-Silicon audio quality; CPU/MPS supported, CUDA recommended for speed)',
"moss-tts-nano": "git clone OpenMOSS/MOSS-TTS-Nano && pip install -e . (not on PyPI)",
"indextts2": "git clone index-tts/index-tts && uv pip install -e . (NOT uv sync --all-extras)",
"gpt-sovits": "External API server — start api_v2.py on port 9880",
@@ -1366,6 +1662,21 @@ _SETUP_SNIPPETS: dict[str, str] = {
}
# Short, readable labels for mlx-audio's curated models (#981) — surfaced in
# the Settings → Engines model picker so users see more than a bare key.
# Single-sourced here rather than on MLXAudioBackend.CURATED_MODELS itself so
# the class dict stays a plain key → repo-id map (what __init__ needs).
_MLX_AUDIO_MODEL_LABELS: dict[str, str] = {
"kokoro": "Kokoro (default, fast)",
"csm": "CSM (voice cloning)",
"qwen3-tts": "Qwen3-TTS (voice design)",
"dia": "Dia",
"chatterbox": "Chatterbox",
"melotts": "MeloTTS (lightweight)",
"outetts": "OuteTTS",
}
def list_backends() -> list[dict]:
"""Enumerate every registered backend with its availability state.
@@ -1449,6 +1760,22 @@ def list_backends() -> list[dict]:
# effective_device / routing_status / routing_reason (scrubbed):
**routing_fields(gpu_compat, caps),
})
# #981: mlx-audio multiplexes 7+ curated models behind one backend id
# — surface the roster + the currently-active pick so Settings can
# render a model picker instead of always defaulting to Kokoro.
# mlx-audio ONLY; every other backend loads a single fixed model.
if bid == "mlx-audio":
from core import prefs
active_model = prefs.resolve(
"mlx_audio_model_id",
env="OMNIVOICE_MLX_AUDIO_MODEL",
default=cls.DEFAULT_MODEL_KEY,
)
out[-1]["curated_models"] = [
{"key": key, "label": _MLX_AUDIO_MODEL_LABELS.get(key, key), "repo_id": repo_id}
for key, repo_id in cls.CURATED_MODELS.items()
]
out[-1]["active_model_id"] = active_model
return out
@@ -1458,6 +1785,29 @@ def get_backend_class(backend_id: str) -> type[TTSBackend]:
return _REGISTRY[backend_id]
def cloning_capable_engine_ids() -> list[str]:
"""Engine ids that support reference-audio voice cloning — used to build
an actionable error when the active engine can't (dub/batch gating).
Iterates the same registry ``list_backends()`` uses, via ``.items()`` so
lazy entries resolve through ``_LazyRegistry``'s snapshot-safe iteration
(see ``_LazyRegistry.__iter__``) exactly like every other registry scan
in this module.
A class-level ``getattr`` on a *property* returns the descriptor object
itself (always truthy) rather than its computed value so a
model-dependent adapter like ``MLXAudioBackend`` (only some of its 7+
curated models can clone) would always show up here regardless of which
model is actually configured. Excluded rather than falsely recommended:
``isinstance(..., bool)`` is False for a descriptor, True for a plain
class attribute.
"""
return [
bid for bid, cls in _REGISTRY.items()
if isinstance((v := getattr(cls, "supports_cloning", True)), bool) and v
]
def active_routing() -> dict | None:
"""Routing verdict for the currently-active TTS engine, or ``None`` if it
can't be determined (no engine / probe failure).
@@ -1526,15 +1876,21 @@ def active_backend_id() -> str:
# call the outgoing engine's unload() before switching.
_active_instance: "TTSBackend | None" = None
_active_instance_id: "str | None" = None
# mlx-audio multiplexes 7+ curated models behind one backend id — a model-only
# switch (same "mlx-audio" id, different curated model) must also invalidate
# the cache, or picking a different model in Settings has no effect until the
# app restarts (#981). Only meaningful when _active_instance_id == "mlx-audio".
_active_mlx_model_key: "str | None" = None
def reset_active_backend() -> None:
"""Unload + clear the cached active backend. For app shutdown and tests.
Idempotent and best-effort a raising unload() never propagates."""
global _active_instance, _active_instance_id
global _active_instance, _active_instance_id, _active_mlx_model_key
inst = _active_instance
_active_instance = None
_active_instance_id = None
_active_mlx_model_key = None
if inst is not None:
try:
inst.unload()
@@ -1552,13 +1908,32 @@ def get_active_tts_backend(*, model=None) -> TTSBackend:
``model=`` (caller already holds a loaded model), we return a fresh view
over the shared singleton rather than caching it but a switch *away from*
a different engine still triggers that engine's unload().
For mlx-audio specifically, the backend id alone doesn't capture *which*
curated model is loaded (#981) — so we also track the resolved model key
and treat a model-only change as a switch, reusing the exact same
unload-and-reconstruct path as an id switch.
"""
global _active_instance, _active_instance_id
global _active_instance, _active_instance_id, _active_mlx_model_key
bid = active_backend_id()
# Switching engines: release the outgoing one first. Best-effort so a bad
# unload() can never block the switch.
if _active_instance is not None and _active_instance_id != bid:
mlx_model_key = None
if bid == "mlx-audio":
from core import prefs
mlx_model_key = prefs.resolve(
"mlx_audio_model_id",
env="OMNIVOICE_MLX_AUDIO_MODEL",
default=MLXAudioBackend.DEFAULT_MODEL_KEY,
)
# Switching engines (or, for mlx-audio, switching curated models): release
# the outgoing one first. Best-effort so a bad unload() can never block
# the switch.
switching = _active_instance is not None and (
_active_instance_id != bid
or (bid == "mlx-audio" and mlx_model_key != _active_mlx_model_key)
)
if switching:
try:
_active_instance.unload()
except Exception as exc: # noqa: BLE001
@@ -1566,6 +1941,7 @@ def get_active_tts_backend(*, model=None) -> TTSBackend:
type(_active_instance).__name__, exc)
_active_instance = None
_active_instance_id = None
_active_mlx_model_key = None
cls = get_backend_class(bid)
if cls is OmniVoiceBackend and model is not None:
@@ -1577,9 +1953,82 @@ def get_active_tts_backend(*, model=None) -> TTSBackend:
if _active_instance is None or _active_instance_id != bid:
_active_instance = OmniVoiceBackend(model=model) if cls is OmniVoiceBackend else cls()
_active_instance_id = bid
_active_mlx_model_key = mlx_model_key
return _active_instance
# ── Shared generation-time engine resolution (issue #312 class) ───────────
#
# dub_generate.py and batch.py used to call services.model_manager.get_model()
# directly, hardcoding OmniVoice regardless of the engine selected in
# Settings → Engines — a SILENT fallback: pick VoxCPM2, dub anyway with
# OmniVoice, no error. This is the single resolution path both routers now
# call instead, mirroring generation.py's /generate resolution (engine id →
# is_available() → routing gate) plus a voice-cloning capability gate that
# /generate doesn't need (OmniVoice's native path always clones).
async def resolve_generation_backend(
*, require_cloning: bool = False, cloning_purpose: str = "dubbing",
) -> TTSBackend:
"""Resolve + validate the active TTS engine for a generation call.
Returns the live backend instance (:func:`get_active_tts_backend`)
cached, and properly unload()ed on an engine switch. Raises ``ValueError``
with an actionable message (never silently falls back to OmniVoice) when:
* the configured engine id is unknown (bad env var / stale pref),
* the engine reports itself unavailable (``is_available()``),
* the engine needs an accelerator this host lacks and has no CPU path
(``routing_status == "unavailable"``),
* ``require_cloning`` is True and the resolved backend can't clone
from reference audio (``supports_cloning`` False) checked on the
live *instance*, not the class, so a model-dependent adapter like
MLX-Audio (Kokoro vs. CSM) is judged by what's actually loaded.
"""
engine_id = active_backend_id()
try:
backend_cls = get_backend_class(engine_id)
except ValueError as e:
raise ValueError(
f"Active TTS engine '{engine_id}' is not a recognized backend ({e}). "
"Check Settings → Engines or the OMNIVOICE_TTS_BACKEND env var."
) from e
try:
ok, msg = backend_cls.is_available()
except Exception as exc: # noqa: BLE001 — surface as an actionable ValueError
ok, msg = False, f"{type(exc).__name__}: {exc}"
if not ok:
raise ValueError(f"TTS engine '{engine_id}' is not available: {_mask_hf_tokens(msg)}")
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing
routing = resolve_routing(getattr(backend_cls, "gpu_compat", ("cpu",)), detect_host_caps())
if routing["routing_status"] == "unavailable":
raise ValueError(routing["routing_reason"])
_model = None
if backend_cls is OmniVoiceBackend:
# OmniVoice needs its model pre-loaded before construction: called
# from an async context, OmniVoiceBackend._ensure_loaded() refuses to
# bootstrap its own event loop (see its docstring) — same reason
# generation.py's /generate special-cases this backend.
from services.model_manager import get_model
_model = await get_model()
backend = get_active_tts_backend(model=_model)
if require_cloning and not getattr(backend, "supports_cloning", True):
raise ValueError(
f"The active TTS engine '{engine_id}' doesn't support voice cloning, "
f"so {cloning_purpose} can't preserve speaker voices. Switch to one "
f"of: {', '.join(cloning_capable_engine_ids())} in Settings → "
"Engines, or use OmniVoice for this job."
)
return backend
# ── PEP 562 lazy attribute re-export ───────────────────────────────────────
#
# Allows ``from services.tts_backend import IndexTTS2Backend`` to keep
+39 -6
View File
@@ -39,6 +39,27 @@ _audioseal_available: Optional[bool] = None
# This is our signature — every OmniVoice-generated audio carries it.
OMNI_MESSAGE = [0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1]
# Watermark ops run chunk-by-chunk: AudioSeal's activation memory grows
# linearly with input length — a single multi-minute waveform demands a
# multi-GB CPU buffer, which OOM'd a 16 GB machine mid-generate (#1045).
# 30 s bounds each call to tens of MB; the 16-bit message repeats throughout
# the audio, so per-chunk embedding/detection is equivalent.
_CHUNK_SECONDS = 30
def _iter_chunks(audio: torch.Tensor, sample_rate: int):
"""Yield ≤ ~_CHUNK_SECONDS slices of (batch, channels, samples) audio
along the time axis. A sub-second tail is folded into the previous chunk
(AudioSeal embeds poorly on very short segments)."""
total = audio.shape[-1]
step = _CHUNK_SECONDS * sample_rate
starts = list(range(0, total, step))
if len(starts) > 1 and total - starts[-1] < sample_rate:
starts.pop()
for i, start in enumerate(starts):
end = starts[i + 1] if i + 1 < len(starts) else total
yield audio[..., start:end]
def _check_available() -> bool:
"""Check if AudioSeal is installed and importable."""
@@ -135,7 +156,13 @@ def embed_watermark(
# AudioSeal operates at 16kHz internally; it handles resampling, but
# we need to inform it of the source rate for correct embedding.
watermarked = generator(audio, sample_rate=sample_rate, message=msg)
watermarked = torch.cat(
[
generator(seg, sample_rate=sample_rate, message=msg)
for seg in _iter_chunks(audio, sample_rate)
],
dim=-1,
)
# Restore original shape
if len(original_shape) == 2:
@@ -189,11 +216,17 @@ def detect_watermark(
else:
audio = waveform
result = detector.detect_watermark(audio, sample_rate=sample_rate, message_threshold=0.5)
# result is (detection_confidence, decoded_message)
confidence = float(result[0]) if isinstance(result, tuple) else 0.0
decoded_msg = result[1] if isinstance(result, tuple) and len(result) > 1 else None
# Detect per chunk and keep the best hit: bounds memory the same way
# embedding does, and a splice where only part of the file is
# OmniVoice audio still registers (a whole-file average would dilute it).
best_conf, decoded_msg = -1.0, None
for seg in _iter_chunks(audio, sample_rate):
result = detector.detect_watermark(seg, sample_rate=sample_rate, message_threshold=0.5)
seg_conf = float(result[0]) if isinstance(result, tuple) else 0.0
if seg_conf > best_conf:
best_conf = seg_conf
decoded_msg = result[1] if isinstance(result, tuple) and len(result) > 1 else None
confidence = max(best_conf, 0.0)
# Decode message bits
message_bits = ""
+1 -1
View File
@@ -16,7 +16,7 @@
},
"frontend": {
"name": "omnivoice-studio",
"version": "0.3.10",
"version": "0.3.11",
"dependencies": {
"@fontsource-variable/inter": "^5.2.8",
"@fontsource-variable/source-serif-4": "^5.2.9",
+6 -2
View File
@@ -82,8 +82,12 @@ If `huggingface.co` is slow or blocked, point the client at a mirror:
HF_ENDPOINT=https://hf-mirror.com
```
Set it as an environment variable (or in **Settings → environment**) before
downloading. Caveats:
Set it in **Settings → Models → Hugging Face mirror** (quick-pick presets
included), or as an environment variable before launching. On first run, the
setup wizard offers the same mirror quick-pick right on the system-check
screen when the endpoint is unreachable — the network check is a warning, not
a blocker, so an offline or firewalled machine can still finish setup once
models are available (mirror, or manual download below). Caveats:
- A mirror serves the **classic** download path, **not Xet** — you lose
chunk-dedup and Xet's parallel fetch, but you gain reachability. On the
+40
View File
@@ -86,6 +86,46 @@ translation is produced:
Cinematic and Autofit **require an LLM** (below). If none is configured, they
fall back to Fast with a notice.
## Two-stage quality on the LLM engine (auto-glossary + reflect pass)
When the **LLM (OpenAI-compatible)** engine is the active translator, two extra
quality stages run by default. Both have checkboxes next to the Quality control
in the Dub tab's translation settings (they only appear for the LLM engine —
MT engines can't run either stage):
- **Auto glossary** — before the per-segment translation, ONE extra LLM pass
reads the whole transcript and extracts a short theme summary plus a
source → target terminology map. That brief rides every segment's translation
prompt, so character names, places, and recurring domain terms come out the
same in segment 3 and segment 300. It's merged with your manual glossary —
**your entries always win** on a clashing term. The result is cached with the
dub project per target language, so re-translating an unchanged transcript
costs zero extra calls; editing segments re-extracts.
- **Reflect pass** — after each segment's direct translation, the LLM critiques
the draft for wordiness and stiff/unnatural register, then rewrites it as
natural spoken dialogue. **This uses 3 LLM calls per segment instead of 1**
turn it off for long videos on slow or metered providers. If any refinement
step fails or times out, the direct translation is kept silently; refinement
can never fail a segment.
### Fit prediction (all quality levels)
Every translation additionally gets a **pre-synthesis fit check** — no LLM
needed. For each segment, OmniVoice predicts how long the translated line will
take to speak (self-calibrating to your voice/engine from segments already
generated in the job, with a per-language rate table as the cold-start
fallback) and compares it against the slot plus the silence it can borrow
before the next line. Segments the Smart Fit caps can only absorb with an
audible speed-up get a **Tight fit** badge; segments no fitting can save get a
**Won't fit +Ns** badge — so you can shorten the text *before* burning GPU
time on a line that would end up trimmed. Badges are informational only:
generation is never blocked.
**Suggest shorter lines** (checkbox under Quality, off by default) goes one
step further: for every "Won't fit" segment it asks the configured LLM for a
meaning-preserving shorter rewrite and offers it on the row as a one-click
**Use shorter rewrite** suggestion. It never rewrites anything automatically,
and with no LLM configured (or on any LLM error) it simply does nothing.
## LLM Providers (for Cinematic / Autofit)
**Settings → System → LLM Providers** is the one place to set up the LLM. Pick a
+43
View File
@@ -0,0 +1,43 @@
# OmniVoice Studio — OpenAI-Compatible Remote ASR
A path to Qwen3-ASR, a self-hosted FunASR/SenseVoice server, or OpenAI's own
Whisper API — today, without waiting on `transformers` to ship a direct
Qwen3-ASR integration (tracked separately). Unlike every other ASR engine,
this one runs no model locally: it's a pure network client that calls any
server exposing an OpenAI-compatible `POST /v1/audio/transcriptions`
endpoint.
## Setup
No install step — configure it directly:
1. Open **Settings → Models** and find **OpenAI-compatible ASR (remote
server)**.
2. Set **Server URL** to your server's base URL (e.g.
`http://localhost:8000/v1` for a local Qwen3-ASR/FunASR server, or
`https://api.openai.com/v1` for OpenAI's own API).
3. Set **Model** to whatever your server expects (`whisper-1` for OpenAI's
API; check your self-hosted server's docs otherwise).
4. **API key** is optional — many self-hosted servers accept requests
without one. Set it if your server requires auth, or if you're using
OpenAI's own API.
5. Activate the engine in **Settings → Engines** — click **Use** on
**OpenAI-compatible ASR** in the ASR Engines table (the same picker TTS
engines have). Power users can pin it instead by setting
`OMNIVOICE_ASR_BACKEND=openai-compat-asr` before launching — the env var
always wins over the Settings pick.
## Response format
The backend prefers `response_format=verbose_json` for real per-segment
timestamps (OpenAI's API and most compatible servers support it) and falls
back to plain text automatically if your server rejects that format. Neither
path returns word-level timestamps — that's not part of this API.
## Privacy note
Unlike every other ASR engine in OmniVoice, audio sent through this backend
leaves your machine — to whatever server you configured. If that's a
self-hosted server on your own network, nothing leaves your control; if
it's a third-party API (OpenAI's, or someone else's), review their data
handling before sending anything sensitive.
+2
View File
@@ -74,6 +74,8 @@ asr_engines:
readme: FunASR
- id: sherpa-onnx-asr
readme: "**sherpa-onnx** (live dictation)"
- id: openai-compat-asr
readme: "**OpenAI-compatible** ⚠️ remote"
# Doc files that must exist (the install path users are sent to).
docs:
+2
View File
@@ -56,6 +56,8 @@ Priority: `duration` > `speed`.
| `preprocess_prompt` | bool | True | Whether to apply preprocessing to the voice-clone prompt audio (remove long silences in reference audio, add punctuation in the end of reference text). |
| `postprocess_output` | bool | True | Apply post-processing to generated audio (remove long silences). |
> **Tip — reference-clip quality transfers.** Zero-shot cloning mirrors the acoustics of the reference clip, not just the voice: a clip recorded in an echoey room clones echoey. Record dry and close-mic for clean output. No effect preset adds reverb unless you choose one that declares it (Cinematic, Warm).
## Long-Form Generation
To support stable long-form speech generation with low VRAM consumption, the text is automatically split into smaller segments when the estimated duration of the generated speech exceeds `audio_chunk_duration`, with each segment producing approximately `audio_chunk_duration` seconds of audio. This approach allows the model to accept arbitrarily long text and generate arbitrarily long speech with near-constant VRAM consumption.
+60
View File
@@ -0,0 +1,60 @@
# Verified Tesla T4 (16GB) inference notes
Measured on a real NVIDIA Tesla T4 (16GB, Turing/sm_75), driver 550.163.01 (CUDA 12.8), torch
2.8.0+cu128, transformers 5.3.0, Python 3.11.15 (uv-managed). Engine under test: the default
`omnivoice` TTS backend (`OMNIVOICE_TTS_BACKEND=omnivoice`).
## Cold-cache first call can time out at 300s
The first `generate()` call lazily downloads the ~2.3GB `k2-fsa/OmniVoice` checkpoint, and that
download happens *inside* the `OMNIVOICE_GENERATE_TIMEOUT_S` budget (default 300s). On a fresh
install, the very first `POST /v1/audio/speech` can fail like this even though the GPU isn't
actually short on memory:
```
ERROR [omnivoice.openai_compat] OpenAI TTS failed: OpenAI TTS generate exceeded 300s and was
abandoned — the backend is running, but the job was too heavy for the available compute.
... most often the GPU is VRAM-starved ...
```
VRAM sampling during the failure showed a flat ~2GB with 0% GPU utilization for the whole 300s —
consistent with waiting on a download, not compute. Once the checkpoint is cached, the identical
request succeeds in ~1s (reproduced 5x: 1.574s / 1.034s / 1.065s / 0.995s / 0.911s).
**Workaround (no code change needed, both already exist):**
- For headless/API-only setups, pre-fetch the checkpoint before your first real TTS request:
```bash
curl -X POST http://localhost:3900/models/install \
-H "Content-Type: application/json" \
-d '{"repo_id": "k2-fsa/OmniVoice"}'
```
(`repo_id` is required — `InstallModelRequest` in `backend/api/schemas.py` rejects a bare/empty
body — and must match one of the entries in `KNOWN_MODELS`, e.g. the default engine's
`k2-fsa/OmniVoice`.) Progress streams over the existing `/setup/download-stream` SSE feed.
- Or raise `OMNIVOICE_GENERATE_TIMEOUT_S` for the first request.
## OpenAI-compatible endpoint doesn't expose `num_step` / `guidance_scale`
`POST /v1/audio/speech`'s request schema doesn't declare `num_step` or `guidance_scale` fields —
sending them in the JSON body returns `200 OK` but they're silently discarded (pydantic's default
`extra=ignore` behavior). The native multipart `POST /generate` endpoint *does* expose both as
explicit form fields, so use that endpoint if you need to control them.
Separately: the app's own default for `num_step` is 16 — half of the model's documented default of
32 (see `docs/generation-parameters.md`, "Use 16 for faster inference"). Not a bug, just not stated
that the app already runs the "fast" preset unless you override it via `/generate`.
## T4 acceleration checklist
| Option | Status |
|---|---|
| dtype | `torch.float16` hardcoded for the `omnivoice` engine (`model_manager.py`) — correct for Turing (no bf16 tensor cores this generation). No env var override for this engine specifically (ASR engines have `ASR_COMPUTE_TYPE`; `dots_tts`/`indextts` have their own precision vars; `omnivoice` doesn't). |
| Attention | `sdpa`, selected automatically since `flash_attn` isn't installed (`_supports_flash_attn_2=True` is declared but the package itself is absent) — safe on T4. |
| int8 | No int8 path for this engine (ASR's CTranslate2 `int8` and `sherpa-onnx`'s int8 ONNX models are separate/unrelated). |
| CUDA Graphs | No direct API usage in the app. Reachable indirectly via `torch.compile(mode="reduce-overhead")`, which the app attempts **by default** on this GPU (T4/sm_75 isn't in the framework's compile-exclusion list, unlike newer/Blackwell GPUs). The numbers above were measured with `TORCH_COMPILE_DISABLE=1` for a clean eager baseline. |
| torch.compile | Attempted by default on T4 (see above) — not evaluated further here. |
## VRAM
Peak measured: 2487 MiB (`nvidia-smi`) / 2.050 GB (`torch.cuda.max_memory_allocated()`) for the
default `omnivoice` engine — comfortably fits even the README's stated "minimum" (4GB) tier.
+109 -33
View File
@@ -5,13 +5,32 @@ working OmniVoice Studio install on a Debian / Ubuntu / Fedora / Arch host.
## Prerequisites
### Using the AppImage
- **Linux x86_64** with a desktop session (X11 or Wayland) capable of running
a Tauri / WebKitGTK app.
- **~10 GB free disk** for the app, its Python environment, and model weights.
- Optional: an **NVIDIA driver** for CUDA GPU acceleration — the app runs
CPU-only without one. For AMD GPUs see [AMD GPU (ROCm)](#amd-gpu-rocm).
- Optional: **yt-dlp** for downloading YouTube/video clips directly in the
Voice Gallery and Dub tabs — `sudo apt install yt-dlp` (Debian/Ubuntu),
`sudo dnf install yt-dlp` (Fedora), or `sudo pacman -S yt-dlp` (Arch).
Without it those downloads fail; everything else works fine.
That's it — Python, FFmpeg, and the model weights are bundled or bootstrapped
by the app itself on first launch. No toolchain needed.
### Building from source
Everything above, plus the toolchain:
- **git**`sudo apt install git` (Debian/Ubuntu), `sudo dnf install git` (Fedora), or `sudo pacman -S git` (Arch).
- **curl** — usually preinstalled; used by the Bun and rustup install one-liners below.
- **Python 3.11+** — typically `sudo apt install python3.11` on Debian/Ubuntu,
`sudo dnf install python3.11` on Fedora, or already installed on Arch.
- **Bun**`curl -fsSL https://bun.sh/install | bash`.
- **FFmpeg**`sudo apt install ffmpeg` (Debian/Ubuntu), `sudo dnf install ffmpeg-free` (Fedora), or `sudo pacman -S ffmpeg` (Arch).
- **Rust / Cargo** (required for building from source only)`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` or via your package manager (e.g., `sudo apt install rustc cargo`).
- **Rust / Cargo**`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` or via your package manager (e.g., `sudo apt install rustc cargo`).
If you use rustup, reopen the shell or source `"$HOME/.cargo/env"` before running `bun run desktop-prod`.
- **GTK/WebKit deps** for the Tauri shell:
@@ -58,12 +77,15 @@ No FUSE? Use `--appimage-extract-and-run`:
./OmniVoice.Studio_*.AppImage --appimage-extract-and-run
```
## Install (.deb)
## .deb package
```bash
sudo apt install ./OmniVoice.Studio_*.amd64.deb
omnivoice-studio
```
Not currently published: `.deb` bundling is disabled in the release pipeline
because of a `tauri-cli` bug (`Failed to create control scripts`) — see the
comment in `.github/workflows/release.yml` for the tracking note. The
AppImage above is the supported Linux install path until a `tauri-cli`
version resolves it. `apt install`-able `.deb`s shipped before v0.3 (see
[.deb ffprobe conflict](#deb-ffprobe-conflict) below) if you're upgrading
from one of those.
The desktop app uses these canonical paths (kept in sync with
`scripts/desktop-prod.sh` by the docs-drift CI gate):
@@ -74,24 +96,49 @@ APP_ID="com.debpalash.omnivoice-studio"
APP_NAME="OmniVoice Studio"
```
## AppImage white-screen on Fedora 44 / Ubuntu 24.04
## AppImage white screen / EGL errors (Fedora 44, Ubuntu 24.04+, 26.04)
<a id="appimage-white-screen-on-fedora-44--ubuntu-2404"></a>
Newer distros ship WebKitGTK 2.44 / 2.46, which has a compositing-mode
regression that lands the Tauri window as a fully-white frame with no UI.
Two separate WebKitGTK rendering issues land the Tauri window as a
fully-white frame with no UI. Which one you have depends on your WebKitGTK
version (`pkg-config --modversion webkit2gtk-4.1` prints it).
**Workaround:** set `WEBKIT_DISABLE_COMPOSITING_MODE=1` before launching:
**Modern WebKitGTK (2.48+ — Ubuntu 24.04 and newer, incl. 26.04): try this
first.** WebKit's DMA-BUF renderer fails against some GPU drivers; the
terminal typically shows:
```
Could not create default EGL display: EGL_BAD_PARAMETER
```
Disable the DMA-BUF renderer before launching:
```bash
WEBKIT_DISABLE_DMABUF_RENDERER=1 ./OmniVoice.Studio_*.AppImage
```
**WebKitGTK 2.44 / 2.46 (Fedora 44, Ubuntu 24.04 at release):** a
compositing-mode regression blanks the surface on first paint. Disable
compositing mode instead:
```bash
WEBKIT_DISABLE_COMPOSITING_MODE=1 ./OmniVoice.Studio_*.AppImage
```
OmniVoice's AppRun launcher autodetects the broken WebKitGTK range and sets
this for you (shipped in v0.3+). The manual env-var path remains the documented
fallback when running from a checked-out source tree.
OmniVoice's AppRun launcher autodetects the broken 2.44/2.46 range and sets
this second variable for you (shipped in v0.3+). The manual env-var path
remains the documented fallback when running from a checked-out source tree.
Tracking issue: [#62](https://github.com/debpalash/OmniVoice-Studio/issues/62).
**Last resort** — if neither variable alone helps, force software rendering
(slower, but always paints):
```bash
WEBKIT_DISABLE_DMABUF_RENDERER=1 LIBGL_ALWAYS_SOFTWARE=1 ./OmniVoice.Studio_*.AppImage
```
Tracking issues: [#62](https://github.com/debpalash/OmniVoice-Studio/issues/62),
[#961](https://github.com/debpalash/OmniVoice-Studio/issues/961).
## .deb ffprobe conflict
@@ -134,36 +181,67 @@ that picks these defaults automatically; for v0.3 set them by hand.
<a id="amd-gpu-rocm"></a>
OmniVoice **auto-detects AMD GPUs**`get_best_device()` returns the GPU when a
ROCm build of PyTorch is installed (ROCm-built PyTorch reports through
`torch.cuda.is_available()`), and OmniVoice auto-sets `HSA_OVERRIDE_GFX_VERSION`
for consumer cards whose GFX ID isn't in the official ROCm support matrix. No
code changes or flags are needed.
ROCm support is **Linux-only and opt-in**. The **default install ships the
CUDA build** of PyTorch (the `pytorch-cuda` index in `pyproject.toml`), so on
an AMD-only machine `torch.cuda.is_available()` is `False` and OmniVoice runs
on CPU until you opt into the ROCm variant. (On Windows there is no ROCm path
at all — PyTorch publishes no Windows ROCm wheels; see
[windows.md](windows.md#gpu-support).)
The catch: the **default install ships the CUDA build** of PyTorch (the
`pytorch-cuda` index in `pyproject.toml`), so on an AMD-only machine
`torch.cuda.is_available()` is `False` and OmniVoice falls back to CPU. To use
your AMD GPU, replace torch with the ROCm wheel **after** the first-run install
populates the venv:
Three ways to opt in, in order of preference:
**1. First-run setup screen (recommended).** On Linux the setup screen's
**Compute** card offers **"AMD GPU (ROCm, Linux)"** next to the default
**Auto**. When OmniVoice detects an AMD GPU *and* the ROCm userspace
(`/opt/rocm` present, or `rocminfo` on PATH), the ROCm option is pre-selected;
with an AMD GPU but no ROCm runtime it stays offered-but-unselected — install
ROCm first (or continue on CPU). Choosing ROCm makes the bootstrap reinstall
`torch`/`torchaudio` from the ROCm wheel index
(`https://download.pytorch.org/whl/rocm6.4` by default) right after the
dependency sync — matched to the app's pinned `torch==2.8.0` (the rocm6.2
index only ever published up to torch 2.5.1, so it silently failed the
reinstall and left the CPU-only CUDA build in place).
**2. Environment variable (existing installs / headless).** Set
`OMNIVOICE_TORCH_VARIANT=rocm` before launching — the next bootstrap performs
the same ROCm reinstall. `OMNIVOICE_TORCH_INDEX=<url>` overrides the wheel
index when you need a different ROCm version — e.g. AMD publishes newer
driver-matched builds (7.2.x) at `repo.radeon.com` as a `--find-links` page
rather than a PyPI-style index:
```bash
uv pip install --reinstall torch==2.8.0 torchaudio==2.8.0 \
--find-links https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.4/
```
run that manually if you want a specific ROCm point release; the
`OMNIVOICE_TORCH_INDEX` env var only accepts a PEP 503 index URL, not a
find-links page. If the reinstall fails (network, unsupported card), OmniVoice
keeps the default torch build and warns instead of breaking the install.
**3. Manual wheel swap (fallback).** Replace torch with the ROCm wheel
**after** the first-run install populates the venv:
```bash
# From the project directory (source install), into OmniVoice's uv venv.
# Current stable is ROCm 6.2 — match your installed ROCm/driver version
# (https://pytorch.org/get-started/locally/ lists available wheels).
# Matches the app's torch==2.8.0 pin — a different ROCm point release
# (e.g. rocm6.2, rocm7.x) may not carry that exact torch build.
uv pip install --reinstall torch torchaudio \
--index-url https://download.pytorch.org/whl/rocm6.2
--index-url https://download.pytorch.org/whl/rocm6.4
```
Then relaunch — the Settings → System panel should now report the GPU device
instead of `cpu`. Verify the wheel sees your card:
Once a ROCm build of PyTorch is in the venv, detection is automatic —
`get_best_device()` returns the GPU (ROCm-built PyTorch reports through
`torch.cuda.is_available()`), and OmniVoice auto-sets
`HSA_OVERRIDE_GFX_VERSION` for consumer cards whose GFX ID isn't in the
official ROCm support matrix. Relaunch and the Settings → System panel should
report the GPU device instead of `cpu`. Verify the wheel sees your card:
```bash
uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
```
Notes:
- ROCm is **Linux-only** and **opt-in** — the default cross-platform behavior
(CUDA on NVIDIA, MPS on Apple, CPU elsewhere) is unchanged.
- ROCm is exercised far less than the default CUDA/MPS/CPU paths — it works,
but expect rough edges on consumer cards and report what you hit.
- Unsupported GFX (e.g. some consumer RDNA cards): if it still won't run, set
`HSA_OVERRIDE_GFX_VERSION` yourself (e.g. `export HSA_OVERRIDE_GFX_VERSION=11.0.0`)
to the nearest supported architecture before launching.
@@ -171,8 +249,6 @@ Notes:
native ROCm wheel.
Tracking issue: [#124](https://github.com/debpalash/OmniVoice-Studio/issues/124).
An installer-integrated, env-var-driven ROCm wheel selection is a planned
follow-up; until then this manual step is the supported path.
## Hugging Face token (optional but recommended)
+15 -2
View File
@@ -17,13 +17,26 @@ working OmniVoice Studio install on macOS (Apple Silicon).
## Prerequisites
### Using the DMG
- **macOS 12 (Monterey) or newer** — Apple Silicon (Intel: UI only, see the
note above).
- **~10 GB free disk** for the app, its Python environment, and model weights.
That's it — GPU acceleration (Apple MPS) is automatic on Apple Silicon, and
Python, FFmpeg, and the model weights are bundled or bootstrapped by the app
itself on first launch. No toolchain needed.
### Building from source
Everything above, plus the toolchain:
- **Xcode Command Line Tools**`xcode-select --install` (includes **git**
and the C toolchain; `curl` ships with macOS).
- **Python 3.11+**`brew install python@3.11` (or use `pyenv` / the system Python if you already have ≥3.11).
- **Bun**`curl -fsSL https://bun.sh/install | bash`.
- **Xcode Command Line Tools**`xcode-select --install`.
- **FFmpeg** (used by the dubbing + capture pipelines) — `brew install ffmpeg`.
- **Rust / Cargo** (required for building from source only)`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` or `brew install rust`.
- **Rust / Cargo**`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` or `brew install rust`.
If you use rustup, reopen the terminal or source `"$HOME/.cargo/env"` before running `bun run desktop-prod`.
Optional but recommended:
+71 -6
View File
@@ -115,13 +115,22 @@ quarantines every download.
**Fix:** see [macos.md#gatekeeper-quarantine](macos.md#gatekeeper-quarantine).
## 4. AppImage white screen on Fedora 44 / Ubuntu 24.04
## 4. AppImage white screen / EGL errors (Fedora 44, Ubuntu 24.04+, 26.04)
**Symptom:** the AppImage window opens fully white. No UI ever appears.
**Symptom:** the AppImage window opens fully white. No UI ever appears. On
newer distros (Ubuntu 24.04 and later, incl. 26.04) the terminal often shows
`Could not create default EGL display: EGL_BAD_PARAMETER`.
**Cause:** WebKitGTK 2.44 / 2.46 compositing-mode regression.
**Cause:** WebKitGTK rendering regressions — the DMA-BUF renderer on modern
WebKitGTK (2.48+), or the 2.44 / 2.46 compositing mode.
**Fix:** see [linux.md#appimage-white-screen-on-fedora-44--ubuntu-2404](linux.md#appimage-white-screen-on-fedora-44--ubuntu-2404).
**Fix:** try `WEBKIT_DISABLE_DMABUF_RENDERER=1` first (modern WebKitGTK / the
EGL error), then `WEBKIT_DISABLE_COMPOSITING_MODE=1` — full walkthrough incl.
the software-rendering last resort:
[linux.md#appimage-white-screen-on-fedora-44--ubuntu-2404](linux.md#appimage-white-screen-on-fedora-44--ubuntu-2404).
**Linked issues:** [#62](https://github.com/debpalash/OmniVoice-Studio/issues/62),
[#961](https://github.com/debpalash/OmniVoice-Studio/issues/961)
## 5. Windows Triton / torch.compile OOM
@@ -233,6 +242,15 @@ for the dedicated CosyVoice path.
**Linked issue:** [#55](https://github.com/debpalash/OmniVoice-Studio/issues/55)
**Same class, ASR side:** the `nemo-parakeet` ASR engine has the identical
problem and currently has **no safe install path** at all — `nemo_toolkit[asr]`
hard-pins `transformers>=4.57,<4.58`, which is unsatisfiable alongside
OmniVoice's own `transformers>=5.3` requirement. Installing it into the
shared venv breaks the backend outright. Do not `pip install nemo_toolkit`
into OmniVoice's environment; if you want to try it, use a separate Python
environment. Isolated-venv support for this engine (matching CosyVoice/
dots-tts) is tracked in [#974](https://github.com/debpalash/OmniVoice-Studio/issues/974).
## 12. CUDA PyTorch wheel download fails on first run
**Symptom:** first-run setup stops at **Installing dependencies** with a failure
@@ -305,8 +323,10 @@ order:
files are a common false-positive quarantine), then re-enable.
- **Connection** — use a stable, direct connection; pause any VPN; avoid
corporate/school networks.
- **Region mirror** — if `huggingface.co` is slow/blocked where you are, set a
mirror **before** launching and relaunch:
- **Region mirror** — if `huggingface.co` is slow/blocked where you are, pick a
mirror in-app (**Settings → Models → Hugging Face mirror**, or the quick-pick
the first-run system check offers when the endpoint is unreachable), or set
it as an env var before launching and relaunch:
- macOS/Linux: `export HF_ENDPOINT=https://hf-mirror.com`
- Windows (PowerShell): `[Environment]::SetEnvironmentVariable("HF_ENDPOINT","https://hf-mirror.com","User")`
@@ -382,6 +402,16 @@ overhead. It reuses your existing faster-whisper install (nothing extra to
download). OmniVoice never switches engines automatically — this stays your
call.
> **Seeing "The backend crashed (exit code …)" instead?** That's the other
> failure mode: the backend **process died** (native CUDA abort, out-of-memory
> kill, DLL crash) rather than hanging. Newer desktop builds detect the death,
> restart the backend automatically (giving up after 3 crashes in 10 minutes),
> and show a crash notice with a **View crash details** button (exit code +
> the last error output). Use **Report this bug** from that notice — the crash
> evidence is attached to the prefilled GitHub issue automatically, with home
> paths scrubbed. The raw markers live next to the backend logs in
> `backend_crash_markers.json`.
## 15. Stuck at "preparing" forever after a crash / BSOD (Windows)
**Symptom:** after an unclean shutdown (Windows BSOD, forced power-off), every
@@ -415,6 +445,41 @@ quit OmniVoice Studio, delete the folder below, then start the app again.
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\com.debpalash.omnivoice-studio\EBWebView"
```
## 16. macOS: microphone permission never prompts, OmniVoice never appears in System Settings
**Symptom:** clicking record shows "Microphone access denied. macOS: open
System Settings → Privacy & Security → Microphone and enable OmniVoice" —
but OmniVoice never appears in that list, so there's nothing to enable.
`NSMicrophoneUsageDescription` is present in the app's `Info.plist`, and
resetting the permission (`tccutil reset Microphone
com.debpalash.omnivoice-studio`) followed by a relaunch changes nothing — no
system prompt ever appears.
**Cause:** the app bundle was missing the Hardened Runtime *entitlement* for
microphone access. An earlier revision of this section blamed an upstream
Tauri/WebKit limitation — that was wrong (a community contributor,
[@MahdiHedhli](https://github.com/MahdiHedhli), read the sources more
carefully and found the real gap). wry's `WKUIDelegate` already grants the
WebKit-layer media-capture request; but Tauri's macOS bundler enables
Hardened Runtime by default, and Hardened Runtime blocks microphone hardware
access unless `com.apple.security.device.audio-input` is present in the
signed binary's entitlements — regardless of `Info.plist`'s
`NSMicrophoneUsageDescription` (that only supplies the prompt *text*).
Without the entitlement, macOS's TCC layer never registers a request, which
is exactly why the app never appears in the System Settings list.
**Fix:** ships in the release after v0.3.12 (the bundle now carries
`src-tauri/entitlements.plist` — [#1016](https://github.com/debpalash/OmniVoice-Studio/pull/1016),
contributed by the same person who diagnosed it). Update and live recording
works, with a normal macOS permission prompt on first use.
**Workaround on older builds (≤ v0.3.12):** record your voice sample in any
other app (Voice Memos, QuickTime, etc.) and upload the resulting file in
OmniVoice instead of using live recording — upload-based cloning is
unaffected and works normally.
**Linked issue:** [#1013](https://github.com/debpalash/OmniVoice-Studio/issues/1013)
## Dub: "translation engine needs the optional … package"
**Symptom:** in the Dub tab, translating fails with e.g. *"The 'google'
+55 -5
View File
@@ -5,7 +5,24 @@ working OmniVoice Studio install on Windows 10 / 11 (x64).
## Prerequisites
### Using the MSI installer
- **Windows 10 (21H2 or newer) or Windows 11**, x64.
- **~10 GB free disk** for the app, its Python environment, and model weights.
- Optional: an **NVIDIA GPU + driver** for CUDA acceleration — see
[GPU support on Windows](#gpu-support). AMD GPUs run CPU-only on Windows.
That's it — Python, FFmpeg, and the model weights are bundled or bootstrapped
by the app itself on first launch. No toolchain needed.
### Building from source
Everything above, plus the toolchain:
- **Git for Windows**`winget install --id Git.Git -e`. Needed for
`git clone`, and it includes **Git Bash**, which `bun run desktop-prod`
uses to run its build-and-launch script. Without it, `desktop-prod` stops
with an error telling you to install it.
- **Python 3.11+**`winget install Python.Python.3.11` (or download from
[python.org](https://www.python.org/downloads/windows/)).
- **Microsoft C++ Build Tools** — required by some PyPI source distributions
@@ -14,13 +31,24 @@ working OmniVoice Studio install on Windows 10 / 11 (x64).
with the **"Desktop development with C++"** workload checked.
- **Bun**`powershell -c "irm bun.sh/install.ps1 | iex"`.
- **FFmpeg**`winget install Gyan.FFmpeg`.
- **Git for Windows** (from-source installs only) — `winget install --id Git.Git -e`.
You need it for `git clone` anyway, and it includes **Git Bash**, which
`bun run desktop-prod` uses to run its build-and-launch script. Without it,
`desktop-prod` stops with an error telling you to install it.
- **Rust / Cargo** (required for building from source only) — `winget install Rust.Rustup` or download `rustup-init.exe` from [rustup.rs](https://rustup.rs/).
- **Rust / Cargo**`winget install Rust.Rustup` or download `rustup-init.exe` from [rustup.rs](https://rustup.rs/).
After installing Rustup, close and reopen PowerShell before running `bun run desktop-prod`.
## GPU support on Windows
<a id="gpu-support"></a>
**GPU acceleration on Windows is NVIDIA/CUDA-only.** The Windows install
ships the CUDA build of PyTorch; with an NVIDIA GPU and a regular NVIDIA
driver it's picked up automatically (no CUDA Toolkit install needed).
**AMD GPUs — including Ryzen / Ryzen AI integrated Radeon graphics — run
CPU-only on Windows.** ROCm is not supported on Windows: PyTorch publishes no
Windows ROCm wheels, and OmniVoice's ROCm option is Linux-only. (The Ryzen AI
NPU is likewise not used.) Everything still works on CPU, just slower. If you
have an AMD GPU and want GPU acceleration, run OmniVoice on Linux instead —
see [linux.md — AMD GPU (ROCm)](linux.md#amd-gpu-rocm).
## Install (from source)
Run from a regular (non-admin) PowerShell:
@@ -49,6 +77,28 @@ Download the latest MSI from the
run it, follow the wizard. The shortcut lands in the Start menu as
**OmniVoice Studio**.
### Installing to a different drive
<a id="install-other-drive"></a>
The wizard's **directory picker** lets you install the app to any **local**
drive (D:, E:, …). Two caveats:
- **Mapped network drives (Z: → a share) are not supported** — this is a
Windows Installer limitation, not an OmniVoice bug: MSI custom actions run
as a service account that doesn't see per-user drive mappings, so the
install fails or rolls back. Install to a local drive instead.
- The install location only moves the ~200 MB app itself. The big data
(models, voices, projects — tens of GB) lives in the **data directory**,
which you move independently: **Settings → Storage → Models directory**
in-app, or `OMNIVOICE_DATA_DIR` / [Portable mode](#portable-install) for
the whole data tree.
If an install to a local non-C: drive fails anyway, capture a log with
`msiexec /i OmniVoice*.msi /L*V install.log` and
[open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) with it
— that log shows exactly which step rolled back.
## Portable install (Windows)
<a id="portable-install"></a>
@@ -6,7 +6,7 @@ Today the longform renderer (Audiobook + Stories) applies a **single-pass** `lou
Upgrade to **two-pass** `loudnorm`: a first **measure** pass (`print_format=json`, output to `-f null -`) parses the clip's `input_i / input_tp / input_lra / input_thresh / target_offset`, then a second **apply** pass feeds those measured values back as `measured_*` + `offset` + `linear=true`. This lands the output accurately on the preset target. The change is a **runner enhancement** layered over the existing pure builders — the pure `build_loudnorm_filter()` and `LOUDNESS_PRESETS` stay; we add a measure-filter builder, a measured-apply-filter builder, a JSON parser, a measure-cmd argv builder, and an async two-pass orchestrator that runs in `_render_longform_sse` (`backend/api/routers/audiobook.py:345`) between the chapter renders and the final mux. Loudness stays **opt-in** (`loudness: None` default on both `AudiobookRequest` `:151` and `LongformRenderRequest` `:510`), so default cross-platform behavior is unchanged.
> **Naming note (grounded):** "mastering" already exists in this codebase as `services.audio_dsp.apply_mastering()` (`backend/services/audio_dsp.py:101`) — a per-clip pedalboard EQ/Compressor/Reverb chain used by `/generate`, `/dub`, batch, and stream paths. That is a **different** operation and **is not called** in the longform path (`_render_longform_sse` muxes chapter WAVs straight from `synthesize_chapter`, no `apply_mastering`). The two-pass loudnorm here is the *only* loudness operation in the longform renderer. To avoid conflating the two, the new SSE event is named `"mastering"` deliberately as the user-facing loudness step for longform; this is harmless because the longform stream never emits anything else by that name, but reviewers should know the term is overloaded across the repo.
> **Naming note (grounded):** "mastering" already exists in this codebase as `services.audio_dsp.apply_mastering()` (`backend/services/audio_dsp.py:101`) — a per-clip pedalboard highpass/Compressor chain used by `/generate`, `/dub`, batch, and stream paths. That is a **different** operation and **is not called** in the longform path (`_render_longform_sse` muxes chapter WAVs straight from `synthesize_chapter`, no `apply_mastering`). The two-pass loudnorm here is the *only* loudness operation in the longform renderer. To avoid conflating the two, the new SSE event is named `"mastering"` deliberately as the user-facing loudness step for longform; this is harmless because the longform stream never emits anything else by that name, but reviewers should know the term is overloaded across the repo.
## Problem
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "omnivoice-studio",
"version": "0.3.10",
"version": "0.3.16",
"private": true,
"license": "AGPL-3.0-only",
"type": "module",
+2 -1
View File
@@ -2941,7 +2941,7 @@ dependencies = [
[[package]]
name = "omnivoice-studio"
version = "0.3.10"
version = "0.3.16"
dependencies = [
"arboard",
"dirs-next",
@@ -2969,6 +2969,7 @@ dependencies = [
"walkdir",
"webkit2gtk",
"webview2-com",
"windows 0.61.3",
"windows-core 0.61.2",
"zip 2.4.2",
]
+8 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "omnivoice-studio"
version = "0.3.10"
version = "0.3.16"
description = "OmniVoice Studio AI voice cloning & dubbing desktop app"
authors = ["Debpalash"]
license = "AGPL-3.0-only"
@@ -71,6 +71,13 @@ zip = { version = "2", default-features = false, features = ["deflate"] }
# Versions match what wry already locks — no new native code is pulled in.
webview2-com = "0.38"
windows-core = "0.61"
# Raw HWND access (GetWindowLongPtrW/SetWindowLongPtrW/ShowWindow) to mark the
# dictation pill WS_EX_NOACTIVATE so showing it doesn't steal Win32 foreground
# activation (#982 — Windows counterpart of #287's macOS focus-steal fix).
# Version pinned to match what `tauri` itself already resolves to (0.61.x) so
# `WebviewWindow::hwnd()`'s return type and our syscalls share the exact same
# `HWND` type — no second copy of the crate enters the dependency graph.
windows = { version = "0.61", features = ["Win32_Foundation", "Win32_UI_WindowsAndMessaging"] }
[target.'cfg(unix)'.dependencies]
libc = "0.2"
+19 -1
View File
@@ -22,9 +22,27 @@ HERE="$(dirname -- "$(readlink -f -- "$0")")"
# Sourced by AppRun.test.sh — keep this function pure so unit tests can stub
# `pkg-config`, source the file, call _detect_webkit_workaround, and inspect
# the resulting environment without exec'ing the binary.
#
# Version source (#961 follow-up): the WebKitGTK that actually RUNS is the
# BUNDLED copy (LD_LIBRARY_PATH below puts $HERE/usr/lib first) — NOT the
# host's. Asking the host's pkg-config therefore reads the wrong number
# whenever host and bundle diverge (e.g. a user who builds from source has
# dev packages installed, so pkg-config answers with their system's healthy
# 2.48 while the bundle runs an older lib — skipping a workaround the running
# library needs). inject-apprun.sh stamps the bundled version into
# .bundled-webkitgtk-version at build time, where it is knowable by
# construction; the host pkg-config path survives only as a fallback for
# bundles predating the stamp. OMNIVOICE_APPRUN_WK_MARKER exists for the
# unit tests to point at a fixture marker.
_detect_webkit_workaround() {
local wk_version="0.0"
if command -v pkg-config >/dev/null 2>&1; then
local marker="${OMNIVOICE_APPRUN_WK_MARKER:-$HERE/.bundled-webkitgtk-version}"
if [ -r "$marker" ]; then
# Empty/unreadable marker content → "0.0" (unknown) → fail-safe workaround,
# same philosophy as the missing-pkg-config branch below.
wk_version="$(cat "$marker" 2>/dev/null | tr -d '[:space:]')"
[ -n "$wk_version" ] || wk_version="0.0"
elif command -v pkg-config >/dev/null 2>&1; then
wk_version="$(pkg-config --modversion webkit2gtk-4.1 2>/dev/null \
|| pkg-config --modversion webkit2gtk-4.0 2>/dev/null \
|| echo "0.0")"
@@ -72,6 +72,56 @@ run_case "2.46 (broken)" "2.46.1" "1"
run_case "2.48 (healthy)" "2.48.0" "unset"
run_case "pkg-config absent" "0.0" "1" "no"
# ── Bundled-version marker cases (#961 follow-up) ───────────────────────────
# inject-apprun.sh stamps the bundle's actual WebKitGTK version into
# .bundled-webkitgtk-version at build time; AppRun must prefer that marker
# over the host's pkg-config (which reports the SYSTEM version — wrong
# whenever it diverges from the bundled copy, e.g. on a machine with newer
# dev packages installed).
run_marker_case() {
local label="$1" marker_content="$2" pkg_output="$3" expected="$4"
local marker_file
marker_file="$(mktemp)"
printf '%s\n' "$marker_content" > "$marker_file"
local actual
actual=$(
bash -c '
set +e
pkg_output="'"$pkg_output"'"
export OMNIVOICE_APPRUN_WK_MARKER="'"$marker_file"'"
pkg-config() { echo "$pkg_output"; }
export -f pkg-config
exec() { :; }
export -f exec
# shellcheck disable=SC1090
source "'"$THIS_DIR"'/AppRun" >/dev/null 2>&1 || true
echo "${WEBKIT_DISABLE_COMPOSITING_MODE:-unset}"
'
)
rm -f "$marker_file"
if [[ "$actual" == "$expected" ]]; then
echo "PASS [$label]"
PASS_COUNT=$((PASS_COUNT + 1))
else
echo "FAIL [$label]: expected '$expected' got '$actual'" >&2
FAIL_COUNT=$((FAIL_COUNT + 1))
fi
}
# Marker says broken → workaround applies, even though host pkg-config says healthy.
run_marker_case "marker 2.46 beats host 2.48" "2.46.1" "2.48.0" "1"
# Marker says healthy → no workaround, even though host pkg-config says broken
# (the exact #961 inversion: from-source user with old system lib, new bundle).
run_marker_case "marker 2.48 beats host 2.44" "2.48.0" "2.44.3" "unset"
# Empty marker → treated as unknown → fail-safe workaround.
run_marker_case "empty marker fails safe" "" "2.48.0" "1"
echo
echo "─── AppRun test summary: $PASS_COUNT pass / $FAIL_COUNT fail ───"
if [[ $FAIL_COUNT -ne 0 ]]; then
+28
View File
@@ -0,0 +1,28 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<!--
Tauri's macOS bundle defaults `hardenedRuntime` to true. Hardened
Runtime blocks camera/microphone hardware access unless the matching
entitlement is present here — regardless of Info.plist's
NSMicrophoneUsageDescription and regardless of wry's own WKUIDelegate
already granting the request at the WebKit/JS layer
(WryWebViewUIDelegate::request_media_capture_permission unconditionally
calls WKPermissionDecision::Grant). Without this entitlement, TCC
never even registers a request for the app — nothing shows up in
System Settings → Privacy & Security → Microphone to enable, because
the OS never saw a legitimately-entitled process ask.
-->
<key>com.apple.security.device.audio-input</key>
<true/>
<!--
Matches Info.plist's forward-looking NSCameraUsageDescription — no
current feature uses the camera, but ship the entitlement now so a
future getUserMedia({video: true}) call doesn't hit this same bug.
-->
<key>com.apple.security.device.camera</key>
<true/>
</dict>
</plist>
+184 -40
View File
@@ -12,6 +12,7 @@ use serde::Serialize;
use tauri::{Emitter, Manager};
use crate::config::get_effective_region;
use crate::crash::BackendExit;
use crate::tools::resolve_uv;
use crate::{AppFlags, BackendState, backend_port};
@@ -164,6 +165,7 @@ pub fn retry_bootstrap(app: tauri::AppHandle, state: tauri::State<'_, BootstrapS
if v.is_empty() { "<unknown>" } else { v.as_str() },
env!("CARGO_PKG_VERSION"),
);
set_backend_kill_intended(true); // deliberate kill, not a crash (#941)
crate::backend::kill_orphan_on_port(backend_port());
std::thread::sleep(Duration::from_millis(500));
}
@@ -171,6 +173,7 @@ pub fn retry_bootstrap(app: tauri::AppHandle, state: tauri::State<'_, BootstrapS
}
if crate::backend::port_in_use(backend_port()) {
log::warn!("Port {} in use — taking ownership", backend_port());
set_backend_kill_intended(true); // deliberate kill, not a crash (#941)
crate::backend::kill_orphan_on_port(backend_port());
std::thread::sleep(Duration::from_millis(500));
}
@@ -192,9 +195,7 @@ pub fn spawn_backend_and_wait(app: &tauri::AppHandle, stage_handle: &Arc<Mutex<B
let mut venv_heal_attempted = false;
'bootstrap: loop {
let child = crate::backend::spawn_backend(app, Some(stage_handle));
if let Ok(mut guard) = app.state::<BackendState>().process.lock() {
*guard = child;
}
track_backend_child(app, child);
let start = std::time::Instant::now();
while start.elapsed() < Duration::from_secs(300) {
if crate::backend::backend_healthy(backend_port()) {
@@ -213,20 +214,41 @@ pub fn spawn_backend_and_wait(app: &tauri::AppHandle, stage_handle: &Arc<Mutex<B
}
return;
}
let process_dead = if let Ok(mut guard) = app.state::<BackendState>().process.lock() {
match guard.as_mut() {
Some(child) => match child.try_wait() {
Ok(Some(status)) => Some(status.to_string()),
Ok(None) => None,
Err(_) => Some("unknown".to_string()),
},
None => Some("never started".to_string()),
}
} else {
None
};
if let Some(exit_info) = process_dead {
let process_dead: Option<(String, Option<BackendExit>)> =
if let Ok(mut guard) = app.state::<BackendState>().process.lock() {
match guard.as_mut() {
Some(child) => match child.try_wait() {
Ok(Some(status)) => {
let exit = BackendExit::from_status(status);
Some((exit.description.clone(), Some(exit)))
}
Ok(None) => None,
// try_wait errored — the death is real but its
// shape is unknown; no exit code for the marker.
Err(_) => Some(("unknown".to_string(), None)),
},
// Spawn itself failed — no process ever ran, so this
// is a spawn failure (spawn_failure_diagnostic owns
// it), NOT a crash: no marker.
None => Some(("never started".to_string(), None)),
}
} else {
None
};
if let Some((exit_info, real_exit)) = process_dead {
let err_tail = crate::backend::read_error_log_tail(30);
// #941: persist the forensics for every true process death —
// startup crashes included — unless the app is shutting down
// or a retry flow deliberately killed the child.
if let Some(ref exit) = real_exit {
if !app_is_quitting(app) && !backend_kill_intended() {
crate::crash::record_crash(crate::crash::marker_now(
exit,
backend_uptime_s(app),
crate::backend::read_error_log_tail(CRASH_STDERR_TAIL_LINES),
));
}
}
// #314: a backend that dies because the venv itself is broken
// can only be healed by rebuilding the venv — do that once
// instead of failing into an unwinnable retry loop.
@@ -308,12 +330,34 @@ pub fn spawn_backend_and_wait(app: &tauri::AppHandle, stage_handle: &Arc<Mutex<B
/// first to reach Ready claims this and the rest fall through.
static SUPERVISOR_ACTIVE: AtomicBool = AtomicBool::new(false);
/// Give up (surface Failed) if the backend dies this many times within
/// `RESTART_WINDOW` — a deterministic startup crash must not become a
/// fork-bomb. The #314 broken-venv self-heal stays the venv-failure path; the
/// #941: set while a retry/clean-retry flow deliberately kills the backend to
/// replace it, so the death watchers (startup poll + supervisor) never write a
/// crash marker for — or respawn against — an *intentional* kill. Cleared the
/// moment a fresh child is spawned and tracked (`track_backend_child`).
static BACKEND_KILL_INTENDED: AtomicBool = AtomicBool::new(false);
pub fn set_backend_kill_intended(value: bool) {
BACKEND_KILL_INTENDED.store(value, Ordering::SeqCst);
}
fn backend_kill_intended() -> bool {
BACKEND_KILL_INTENDED.load(Ordering::SeqCst)
}
/// How much of backend_err.log rides inside a crash marker (#941). ~40 lines
/// is enough for a Python traceback or a native abort banner without bloating
/// the marker file or the bug-report URL (the frontend truncates further).
const CRASH_STDERR_TAIL_LINES: usize = 40;
/// Crash-loop escalation guard (#941, supersedes the #567 5-in-60s budget):
/// give up (surface Failed with the crash details) once the backend has died
/// `MAX_RESTARTS` times inside `RESTART_WINDOW`. The longer 10-minute window
/// catches *slow* crash loops (e.g. an engine that OOMs a couple of minutes
/// into every generation) that the old 60-second window let spin silently
/// forever. The #314 broken-venv self-heal stays the venv-failure path; the
/// supervisor only handles post-Ready deaths.
const MAX_RESTARTS: usize = 5;
const RESTART_WINDOW: Duration = Duration::from_secs(60);
const MAX_RESTARTS: usize = 3;
const RESTART_WINDOW: Duration = Duration::from_secs(600);
fn app_is_quitting(app: &tauri::AppHandle) -> bool {
app.try_state::<AppFlags>()
@@ -321,17 +365,39 @@ fn app_is_quitting(app: &tauri::AppHandle) -> bool {
.unwrap_or(false)
}
/// Returns `Some(exit description)` if the tracked backend child has exited,
/// Store the freshly spawned backend child (and its spawn time, for the crash
/// marker's `uptime_s`), and re-arm the death watchers: any deliberate-kill
/// window ends the moment a new child is tracked.
fn track_backend_child(app: &tauri::AppHandle, child: Option<std::process::Child>) {
let state = app.state::<BackendState>();
if let Ok(mut guard) = state.process.lock() {
*guard = child;
}
if let Ok(mut spawned) = state.spawned_at.lock() {
*spawned = Some(Instant::now());
}
set_backend_kill_intended(false);
}
/// Seconds since the tracked backend child was spawned (0 when unknown).
fn backend_uptime_s(app: &tauri::AppHandle) -> u64 {
app.try_state::<BackendState>()
.and_then(|s| s.spawned_at.lock().ok().and_then(|g| *g))
.map(|t| t.elapsed().as_secs())
.unwrap_or(0)
}
/// Returns `Some(BackendExit)` if the tracked backend child has exited,
/// `None` if it is still running (or none is tracked — which we never treat as
/// a death to respawn, to avoid fighting a deliberate teardown).
fn backend_child_exit(app: &tauri::AppHandle) -> Option<String> {
fn backend_child_exit(app: &tauri::AppHandle) -> Option<BackendExit> {
let state = app.try_state::<BackendState>()?;
let mut guard = state.process.lock().ok()?;
match guard.as_mut() {
Some(child) => match child.try_wait() {
Ok(Some(status)) => Some(status.to_string()),
Ok(Some(status)) => Some(BackendExit::from_status(status)),
Ok(None) => None,
Err(e) => Some(format!("try_wait error: {e}")),
Err(e) => Some(BackendExit::unknown(&format!("try_wait error: {e}"))),
},
None => None,
}
@@ -358,21 +424,39 @@ fn supervise_backend(app: &tauri::AppHandle, stage_handle: &Arc<Mutex<BootstrapS
if app_is_quitting(app) {
return;
}
let exit_info = match backend_child_exit(app) {
Some(info) => info,
let exit = match backend_child_exit(app) {
Some(exit) => exit,
None => continue, // still running
};
// The exit may have raced with a shutdown that killed the child.
if app_is_quitting(app) {
return;
}
// A retry/clean-retry flow killed the child on purpose and owns the
// respawn — no crash marker, and step aside so the retry's own
// spawn_backend_and_wait claims the supervisor slot at Ready (#941).
if backend_kill_intended() {
log::info!("Backend exit was a deliberate replace — supervisor yielding to the retry flow");
return;
}
let exit_info = exit.description.clone();
// #941: make the death self-documenting BEFORE any restart attempt —
// the marker (exit code/signal + stderr tail + uptime) is what turns
// the next "Can't reach the backend" report into a diagnosable one.
let uptime_s = backend_uptime_s(app);
crate::crash::record_crash(crate::crash::marker_now(
&exit,
uptime_s,
crate::backend::read_error_log_tail(CRASH_STDERR_TAIL_LINES),
));
if restart_budget_exhausted(&mut restart_times, Instant::now()) {
let tail = crate::backend::read_error_log_tail(30);
let msg = format!(
"The backend kept crashing ({} times in {}s) and couldn't be kept running. \
Use Clean & Retry, or check Settings Logs Backend.{}",
"The backend kept crashing ({} times in {} min; last death: {}) and couldn't \
be kept running. Use Clean & Retry, or check Settings Logs Backend.{}",
MAX_RESTARTS,
RESTART_WINDOW.as_secs(),
RESTART_WINDOW.as_secs() / 60,
exit.label(),
if tail.is_empty() { String::new() } else { format!("\n\nLast output:\n{tail}") },
);
log::error!("Backend supervisor giving up: {msg}");
@@ -393,9 +477,7 @@ fn supervise_backend(app: &tauri::AppHandle, stage_handle: &Arc<Mutex<BootstrapS
std::thread::sleep(Duration::from_millis(300));
}
let child = crate::backend::spawn_backend(app, Some(stage_handle));
if let Ok(mut guard) = app.state::<BackendState>().process.lock() {
*guard = child;
}
track_backend_child(app, child);
// Wait (bounded) for the respawn to become healthy. If it dies again
// immediately, bail early so the next loop counts it toward the cap.
let start = Instant::now();
@@ -430,6 +512,7 @@ pub fn clean_and_retry_bootstrap(app: tauri::AppHandle, state: tauri::State<'_,
// project dir, otherwise bootstrap will "attach" to the stale process.
if crate::backend::port_in_use(backend_port()) {
log::warn!("Clean retry: killing stale backend on port {}", backend_port());
set_backend_kill_intended(true); // deliberate kill, not a crash (#941)
crate::backend::kill_orphan_on_port(backend_port());
std::thread::sleep(Duration::from_millis(500));
}
@@ -621,9 +704,35 @@ fn sync_failure_is_torch_download(tail: &str) -> bool {
|| (low.contains("torch") && (low.contains("failed to download") || low.contains("failed to fetch")))
}
/// Default PyTorch ROCm wheel index for the opt-in AMD path (#124). ROCm 6.2 is
/// the current stable wheel set; overridable via OMNIVOICE_TORCH_INDEX.
const ROCM_TORCH_INDEX: &str = "https://download.pytorch.org/whl/rocm6.2";
/// Default PyTorch ROCm wheel index for the opt-in AMD path (#124).
/// ROCm 6.4, not 6.2: the app's pinned `torch==2.8.0` (pyproject.toml) has no
/// build on the rocm6.2 index (it tops out at 2.5.1), so that index silently
/// failed the reinstall and left the default CUDA build in place — which runs
/// on CPU on an AMD GPU (#972). rocm6.4 carries a matching 2.8.0 build.
/// Overridable via OMNIVOICE_TORCH_INDEX (e.g. a `--find-links` URL for
/// distro-matched ROCm builds torch's own index doesn't carry).
const ROCM_TORCH_INDEX: &str = "https://download.pytorch.org/whl/rocm6.4";
/// Args for the routine update-drift sync (#307 path) — the one that runs on
/// every app update when `uv.lock` changed. `--inexact` is the fix for #1029:
/// plain `uv sync` UNINSTALLS every package not in the lockfile, which
/// silently deleted user-pip-installed optional engines (voxcpm, kittentts —
/// packages the app's own Settings → Engines hints tell users to install
/// into this venv) on every single update. `--inexact` still installs/
/// upgrades everything the lockfile demands — locked deps stay exactly
/// correct — it just stops removing extras the user added on purpose.
///
/// Deliberately NOT applied to the repair sync (`repair_sync_args`): repair
/// runs when the venv is *broken*, and a user-installed extra is a plausible
/// cause — healing must restore the known-good locked state, extras
/// included-out. An engine lost to a repair is re-installable; a venv that
/// repair can't actually repair is a support thread.
const DRIFT_SYNC_ARGS: [&str; 5] = ["sync", "--frozen", "--inexact", "--no-dev", "--verbose"];
/// Exact-sync args for the venv-repair path — see `DRIFT_SYNC_ARGS` for why
/// repair stays exact while the update-drift sync preserves user extras.
const REPAIR_SYNC_ARGS_LOCKED: [&str; 4] = ["sync", "--frozen", "--no-dev", "--verbose"];
const REPAIR_SYNC_ARGS_UNLOCKED: [&str; 3] = ["sync", "--no-dev", "--verbose"];
/// `uv pip install` args that replace the default CUDA torch build with the AMD
/// ROCm wheel (#124). Opt-in (gated on OMNIVOICE_TORCH_VARIANT=rocm by the
@@ -1166,7 +1275,7 @@ manually, then relaunch.",
drift_cmd.env("UV_INDEX_URL", "https://mirrors.aliyun.com/pypi/simple/");
}
drift_cmd
.args(["sync", "--frozen", "--no-dev", "--verbose"])
.args(DRIFT_SYNC_ARGS)
.current_dir(&project_dir);
match run_streaming(app, "installing_deps", &mut drift_cmd) {
Ok(ref s) if s.success() => {
@@ -1241,9 +1350,9 @@ the existing venv; newly added dependencies may be missing (#307)",
apply_uv_http_env(&mut repair_cmd);
let has_lockfile = project_dir.join("uv.lock").is_file();
if has_lockfile {
repair_cmd.args(["sync", "--frozen", "--no-dev", "--verbose"]);
repair_cmd.args(REPAIR_SYNC_ARGS_LOCKED);
} else {
repair_cmd.args(["sync", "--no-dev", "--verbose"]);
repair_cmd.args(REPAIR_SYNC_ARGS_UNLOCKED);
}
repair_cmd.current_dir(&project_dir);
let repair_status = run_streaming(app, "installing_deps", &mut repair_cmd);
@@ -1624,6 +1733,29 @@ mod tests {
use super::*;
use std::collections::HashMap;
#[test]
fn update_drift_sync_preserves_user_installed_engines() {
// #1029: the routine update sync must carry --inexact so a
// user-pip-installed optional engine (voxcpm, kittentts — packages
// the app's own Settings → Engines hints tell users to install into
// this venv) survives every update instead of being silently
// uninstalled. --frozen must stay (lockfile is the resolution truth).
assert!(DRIFT_SYNC_ARGS.contains(&"--inexact"),
"update-drift sync lost --inexact — user-installed engines get wiped on every update (#1029)");
assert!(DRIFT_SYNC_ARGS.contains(&"--frozen"));
}
#[test]
fn repair_sync_stays_exact() {
// Deliberate asymmetry with the drift sync: repair runs when the venv
// is BROKEN and a user-installed extra is a plausible cause — healing
// must restore the known-good locked state, extras included-out.
assert!(!REPAIR_SYNC_ARGS_LOCKED.contains(&"--inexact"),
"repair sync must stay exact — it's the recovery path when an extra broke the venv");
assert!(!REPAIR_SYNC_ARGS_UNLOCKED.contains(&"--inexact"));
assert!(REPAIR_SYNC_ARGS_LOCKED.contains(&"--frozen"));
}
#[test]
fn scrub_python_env_removes_bundled_runtime_vars() {
// #144: every uv/venv/pip subprocess must drop the AppImage's bundled
@@ -1668,6 +1800,15 @@ mod tests {
assert_eq!(envs.get("UV_HTTP_RETRIES").map(String::as_str), Some("5"));
}
#[test]
fn crash_loop_policy_is_three_deaths_in_ten_minutes() {
// #941 escalation guard: ≥3 crashes inside 10 min must stop the
// respawn loop and land on the Failed screen with the crash details —
// the old 5-in-60s budget let slow crash loops spin silently forever.
assert_eq!(MAX_RESTARTS, 3);
assert_eq!(RESTART_WINDOW, Duration::from_secs(600));
}
#[test]
fn restart_budget_caps_respawns_and_prunes_old_ones() {
// Supervisor backoff policy (#567): fewer than MAX_RESTARTS deaths
@@ -1725,7 +1866,10 @@ mod tests {
assert!(args.iter().any(|a| a == "torch"));
assert!(args.iter().any(|a| a == "torchaudio"));
let i = args.iter().position(|a| a == "--index-url").expect("has --index-url");
assert!(args[i + 1].contains("rocm6.2"), "default index is the rocm6.2 wheel set");
// rocm6.4, not rocm6.2: rocm6.2's index tops out at torch 2.5.1 and
// can't satisfy the app's torch==2.8.0 pin (#972) — a regression to
// rocm6.2 here would silently resurrect the CPU-fallback bug.
assert!(args[i + 1].contains("rocm6.4"), "default index is the rocm6.4 wheel set (matches torch==2.8.0)");
}
#[test]
+318
View File
@@ -0,0 +1,318 @@
//! Backend crash forensics (#941).
//!
//! When the backend PROCESS dies (native CUDA abort, OOM kill, DLL crash),
//! the user used to see only "Can't reach the local OmniVoice backend" — and
//! the evidence (exit code, stderr tail) evaporated with the process. Every
//! such report was undiagnosable without asking for logs nobody sends.
//!
//! This module makes every backend death self-documenting: the death watchers
//! in `bootstrap.rs` (the startup health poll and the post-Ready supervisor)
//! call [`record_crash`] with the exit status and captured stderr tail, which
//! persists a small JSON **crash marker** next to the backend logs. The
//! frontend reads the newest marker via the `get_last_backend_crash` command
//! to replace the vague unreachable-toast with the honest story ("the backend
//! crashed (exit code X)…"), and the bug-report prefill attaches it so the
//! next #941-class GitHub issue arrives WITH the evidence.
//!
//! Only the last [`MAX_MARKERS`] crashes are kept. Acknowledgment is a
//! persisted timestamp (not deletion!) so viewing the crash details doesn't
//! destroy the evidence a subsequent bug report needs.
use std::fs;
use std::path::{Path, PathBuf};
use std::process::ExitStatus;
use serde::{Deserialize, Serialize};
/// How many crash markers to retain (newest first).
pub const MAX_MARKERS: usize = 3;
// ── Exit-status decomposition ──────────────────────────────────────────────
/// Structured view of how the backend child ended: the numeric exit code (or
/// Unix signal) for the marker, plus the human-readable `ExitStatus` display
/// for logs and bootstrap messages.
#[derive(Clone, Debug, PartialEq)]
pub struct BackendExit {
pub code: Option<i32>,
pub signal: Option<i32>,
pub description: String,
}
impl BackendExit {
pub fn from_status(status: ExitStatus) -> Self {
#[cfg(unix)]
let signal = {
use std::os::unix::process::ExitStatusExt;
status.signal()
};
#[cfg(not(unix))]
let signal = None;
BackendExit { code: status.code(), signal, description: status.to_string() }
}
/// For deaths we can't decompose (`try_wait` errored).
pub fn unknown(description: &str) -> Self {
BackendExit { code: None, signal: None, description: description.to_string() }
}
/// Short human label — "exit code 3221226505" / "signal 6" — for messages.
pub fn label(&self) -> String {
match (self.code, self.signal) {
(Some(c), _) => format!("exit code {}", c),
(None, Some(s)) => format!("signal {}", s),
(None, None) => self.description.clone(),
}
}
}
// ── Marker model ───────────────────────────────────────────────────────────
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct CrashMarker {
/// Unix seconds when the death was detected.
pub ts: u64,
/// Process exit code, when the OS reported one.
pub exit_code: Option<i32>,
/// Unix signal that killed the process (None on Windows / normal exits).
pub signal: Option<i32>,
/// Human-readable `ExitStatus` display ("exit status: 134", …).
pub exit_desc: String,
/// App/backend version (lockstep per the versioning rule).
pub backend_version: String,
/// Seconds the backend had been running when it died.
pub uptime_s: u64,
/// Tail of backend_err.log captured at death time.
pub last_stderr: String,
}
/// The single on-disk store: newest-first markers plus the acknowledgment
/// watermark. One file keeps rotation + ack updates atomic-ish and avoids
/// filename collisions for same-second crashes.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct CrashStore {
/// `ts` of the newest marker the user has acknowledged (seen). Markers
/// with `ts <= acked_ts` are "old news" for UI purposes but are retained
/// for bug-report attachment.
#[serde(default)]
pub acked_ts: u64,
/// Newest first, capped at [`MAX_MARKERS`].
#[serde(default)]
pub markers: Vec<CrashMarker>,
}
/// Prepend `marker` and keep only the newest [`MAX_MARKERS`]. Pure so the
/// rotation policy is unit-tested without touching the filesystem.
pub fn push_marker(store: &mut CrashStore, marker: CrashMarker) {
store.markers.insert(0, marker);
store.markers.truncate(MAX_MARKERS);
}
/// Newest marker + whether the user has already acknowledged it.
pub fn newest_with_ack(store: &CrashStore) -> Option<(CrashMarker, bool)> {
store.markers.first().map(|m| (m.clone(), m.ts <= store.acked_ts))
}
// ── Persistence ────────────────────────────────────────────────────────────
/// The marker store lives next to the backend logs (same rationale: it's
/// forensic output of the backend process, discoverable alongside
/// backend.log / backend_err.log).
pub fn markers_path() -> PathBuf {
crate::backend::backend_log_path().with_file_name("backend_crash_markers.json")
}
pub fn load_store_from(path: &Path) -> CrashStore {
fs::read_to_string(path)
.ok()
.and_then(|s| serde_json::from_str(&s).ok())
.unwrap_or_default()
}
pub fn save_store_to(path: &Path, store: &CrashStore) {
match serde_json::to_string_pretty(store) {
Ok(json) => {
if let Err(e) = fs::write(path, json) {
log::warn!("Could not persist crash marker to {}: {}", path.display(), e);
}
}
Err(e) => log::warn!("Could not serialize crash marker: {}", e),
}
}
fn now_unix_s() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// Build a marker for a death detected right now.
pub fn marker_now(exit: &BackendExit, uptime_s: u64, last_stderr: String) -> CrashMarker {
CrashMarker {
ts: now_unix_s(),
exit_code: exit.code,
signal: exit.signal,
exit_desc: exit.description.clone(),
backend_version: env!("CARGO_PKG_VERSION").to_string(),
uptime_s,
last_stderr,
}
}
/// Persist an unexpected backend death. Called by the death watchers in
/// `bootstrap.rs` AFTER they have ruled out intentional shutdowns (app quit,
/// deliberate retry/clean-retry kills).
pub fn record_crash(marker: CrashMarker) {
log::error!(
"Backend process died unexpectedly ({}, uptime {} s). Crash marker written. Stderr tail:\n{}",
marker.exit_desc,
marker.uptime_s,
if marker.last_stderr.is_empty() { "<none captured>" } else { &marker.last_stderr },
);
let path = markers_path();
let mut store = load_store_from(&path);
push_marker(&mut store, marker);
save_store_to(&path, &store);
}
// ── Tauri commands ─────────────────────────────────────────────────────────
/// Newest crash marker + its acknowledgment state, as returned to the
/// frontend (`get_last_backend_crash`).
#[derive(Clone, Debug, Serialize)]
pub struct CrashNotice {
#[serde(flatten)]
pub marker: CrashMarker,
pub acknowledged: bool,
}
/// Newest backend crash marker, or null when the backend has never crashed.
/// `acknowledged` tells the UI whether the user already viewed/dismissed it.
#[tauri::command]
pub fn get_last_backend_crash() -> Option<CrashNotice> {
let store = load_store_from(&markers_path());
newest_with_ack(&store).map(|(marker, acknowledged)| CrashNotice { marker, acknowledged })
}
/// Mark the newest crash as seen. Deliberately does NOT delete the marker —
/// the bug-report prefill still needs the evidence after the user viewed it.
#[tauri::command]
pub fn acknowledge_backend_crash() {
let path = markers_path();
let mut store = load_store_from(&path);
if let Some(newest_ts) = store.markers.first().map(|m| m.ts) {
if store.acked_ts < newest_ts {
store.acked_ts = newest_ts;
save_store_to(&path, &store);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn marker(ts: u64) -> CrashMarker {
CrashMarker {
ts,
exit_code: Some(1),
signal: None,
exit_desc: format!("exit status: 1 (#{ts})"),
backend_version: "0.0.0-test".into(),
uptime_s: 42,
last_stderr: "Traceback…".into(),
}
}
#[test]
fn rotation_keeps_only_the_last_three_newest_first() {
// #941: write 4 markers → only the newest MAX_MARKERS survive.
let mut store = CrashStore::default();
for ts in [1, 2, 3, 4] {
push_marker(&mut store, marker(ts));
}
assert_eq!(store.markers.len(), MAX_MARKERS);
let kept: Vec<u64> = store.markers.iter().map(|m| m.ts).collect();
assert_eq!(kept, vec![4, 3, 2], "newest first, oldest dropped");
}
#[test]
fn ack_semantics_survive_newer_crashes() {
let mut store = CrashStore::default();
push_marker(&mut store, marker(100));
// Fresh crash → unacknowledged.
let (m, acked) = newest_with_ack(&store).expect("has a marker");
assert_eq!(m.ts, 100);
assert!(!acked, "a fresh crash must be unacknowledged");
// Viewing acks the newest…
store.acked_ts = 100;
assert!(newest_with_ack(&store).unwrap().1, "viewed crash is acknowledged");
// …but a NEWER crash re-arms the notice, and the marker itself is
// retained (evidence survives the ack — bug reports still attach it).
push_marker(&mut store, marker(200));
let (m2, acked2) = newest_with_ack(&store).unwrap();
assert_eq!(m2.ts, 200);
assert!(!acked2, "a newer crash must surface again");
assert_eq!(store.markers.len(), 2, "ack never deletes markers");
}
#[test]
fn store_roundtrips_through_json_and_defaults_when_missing() {
let dir = std::env::temp_dir().join(format!("omnivoice-test-941-{}", std::process::id()));
let _ = fs::remove_dir_all(&dir);
fs::create_dir_all(&dir).unwrap();
let path = dir.join("backend_crash_markers.json");
// Missing file → default store, never an error (first run).
assert_eq!(load_store_from(&path), CrashStore::default());
// Corrupt file → default store (a truncated write must not wedge the
// whole forensics path).
fs::write(&path, "{not json").unwrap();
assert_eq!(load_store_from(&path), CrashStore::default());
let mut store = CrashStore::default();
push_marker(
&mut store,
CrashMarker {
ts: 1,
exit_code: None,
signal: Some(6), // SIGABRT — the native-CUDA-abort shape
exit_desc: "signal: 6 (SIGABRT)".into(),
backend_version: "0.3.10".into(),
uptime_s: 7,
last_stderr: "CUDA error: an illegal memory access".into(),
},
);
store.acked_ts = 0;
save_store_to(&path, &store);
let loaded = load_store_from(&path);
assert_eq!(loaded, store, "Option fields (code=None, signal=Some) must roundtrip");
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn backend_exit_labels_code_signal_and_unknown() {
let coded = BackendExit { code: Some(-1073740791), signal: None, description: "x".into() };
assert_eq!(coded.label(), "exit code -1073740791");
let signaled = BackendExit { code: None, signal: Some(9), description: "x".into() };
assert_eq!(signaled.label(), "signal 9");
let unknown = BackendExit::unknown("try_wait error: gone");
assert_eq!(unknown.label(), "try_wait error: gone");
}
#[cfg(unix)]
#[test]
fn backend_exit_decomposes_real_exit_statuses() {
use std::os::unix::process::ExitStatusExt;
// Normal exit with code 3.
let e = BackendExit::from_status(ExitStatus::from_raw(3 << 8));
assert_eq!(e.code, Some(3));
assert_eq!(e.signal, None);
// Killed by SIGABRT (6) — code is None, signal carries the story.
let k = BackendExit::from_status(ExitStatus::from_raw(6));
assert_eq!(k.code, None);
assert_eq!(k.signal, Some(6));
assert_eq!(k.label(), "signal 6");
}
}
+168 -12
View File
@@ -13,6 +13,7 @@ pub mod bootstrap;
pub mod tools;
pub mod backend;
pub mod commands;
pub mod crash;
pub mod updater_channel;
use std::process::Child;
@@ -41,6 +42,9 @@ pub fn backend_port() -> u16 {
pub struct BackendState {
pub process: Mutex<Option<Child>>,
/// When the tracked child was spawned — feeds the crash marker's
/// `uptime_s` (#941). Set alongside `process` in bootstrap.rs.
pub spawned_at: Mutex<Option<std::time::Instant>>,
}
pub struct AppFlags {
@@ -75,9 +79,18 @@ pub const TRAY_ICON_RECORDING: &[u8] = include_bytes!("../icons/tray-recording.p
// applies on top.
// - Linux (WebKitGTK): media-stream must be enabled per-WebView and the
// permission request answered programmatically.
// - macOS (WKWebView): nothing to do here — wry grants media-capture to the
// app origin and the user-visible consent is the system TCC prompt driven
// by NSMicrophoneUsageDescription in src-tauri/Info.plist.
// - macOS (WKWebView): nothing to do here in code — wry's own WKUIDelegate
// (WryWebViewUIDelegate::request_media_capture_permission) already grants
// every media-capture request unconditionally at the WebKit/JS layer. But
// that alone isn't sufficient (#1013): Tauri's macOS bundle defaults
// `hardenedRuntime` to true, and Hardened Runtime blocks camera/microphone
// hardware access unless the matching entitlement is present — without it,
// TCC never even registers a request, so the app never appears in System
// Settings → Privacy & Security → Microphone for the user to enable. See
// src-tauri/entitlements.plist (wired in via tauri.conf.json's
// bundle.macOS.entitlements) for the actual grant; NSMicrophoneUsageDescription
// in Info.plist only supplies the *prompt text* TCC shows, it doesn't
// substitute for the entitlement.
/// True for origins the app itself serves: the Tauri custom-protocol origin
/// in production and the Vite dev server / loopback in `tauri dev`.
@@ -201,6 +214,120 @@ mod media_permission_tests {
}
}
// ── Windows: dictation pill must never take foreground focus (#982) ────────
//
// Windows counterpart of #287 (macOS auto-paste — don't steal focus). The
// pill is `.always_on_top(true).skip_taskbar(true)` and is documented above
// (see `grant_webview_media_permissions`) as "deliberately unfocused so the
// auto-paste lands in the target app" — true on macOS, but on Windows,
// showing an always-on-top top-level window gives it Win32 foreground
// activation by default (ordinary Windows window-manager behavior; macOS
// doesn't force-activate a shown window the same way). Nothing marked the
// pill non-activating, so on Windows it stole foreground on every show —
// the synthesized Ctrl+V from `simulate_paste` landed back in the pill
// instead of the app the user was dictating into, and because the pill
// wrongly held focus for the whole session the target app never got it back
// until the pill's auto-dismiss timer eventually hid it.
//
// Two pieces, both required (verified by reading how `.show()` is used at
// the call sites below — several are followed by an explicit `set_focus()`
// that would fight the style bit on its own):
// 1. WS_EX_NOACTIVATE on the HWND, applied once right after creation, so
// the OS never grants this window foreground activation implicitly.
// 2. `ShowWindow(SW_SHOWNOACTIVATE)` in place of `WebviewWindow::show()` at
// the pill's dictation-trigger call sites, and the explicit
// `set_focus()` calls at those same sites are skipped on Windows (the
// same way they already are on macOS below).
//
// The flag math (`with_noactivate_style`) is a plain function so it's
// unit-testable on every platform — the actual Win32 syscalls that use it
// are Windows-only and can't run under `cargo test` on a non-Windows runner.
/// `WS_EX_NOACTIVATE` (winuser.h: `#define WS_EX_NOACTIVATE 0x08000000L`).
/// Hardcoded rather than imported from the `windows` crate so `with_noactivate_style`
/// below stays free of the Windows-only dependency and is testable everywhere.
/// Only consumed by Windows-only code (or the platform-agnostic test module
/// below) — `#[allow(dead_code)]` elsewhere, same as `is_app_origin` above.
#[cfg_attr(not(windows), allow(dead_code))]
const WS_EX_NOACTIVATE_BIT: isize = 0x0800_0000;
/// OR `WS_EX_NOACTIVATE` into an existing extended window style, preserving
/// every other bit already set (topmost, layered, etc. — the pill's
/// `always_on_top(true)` sets one of these). Pure so it's unit-testable
/// without a real HWND. See module comment above for why this exists.
#[cfg_attr(not(windows), allow(dead_code))]
fn with_noactivate_style(current_ex_style: isize) -> isize {
current_ex_style | WS_EX_NOACTIVATE_BIT
}
/// Mark the pill's HWND `WS_EX_NOACTIVATE`, once, right after creation — this
/// holds for every later `.show()` regardless of call site (belt-and-braces
/// alongside `show_pill_noactivate` below, which some call sites also need
/// because they pair `.show()` with an explicit `set_focus()`).
#[cfg(target_os = "windows")]
fn mark_pill_noactivate(win: &tauri::WebviewWindow) {
use windows::Win32::UI::WindowsAndMessaging::{
GetWindowLongPtrW, SetWindowLongPtrW, GWL_EXSTYLE,
};
let Ok(hwnd) = win.hwnd() else {
log::warn!("pill: could not resolve HWND to apply WS_EX_NOACTIVATE (#982)");
return;
};
unsafe {
let current = GetWindowLongPtrW(hwnd, GWL_EXSTYLE);
SetWindowLongPtrW(hwnd, GWL_EXSTYLE, with_noactivate_style(current));
}
}
/// Show the pill without granting it foreground activation. Used instead of
/// `WebviewWindow::show()` at the pill's dictation-trigger call sites on
/// Windows — `.show()` maps to plain `ShowWindow(SW_SHOW)`, which relies on
/// the NOACTIVATE style alone to suppress activation; `SW_SHOWNOACTIVATE` is
/// the explicit, documented way to show a window without activating it and
/// costs nothing extra now that the style bit is also set (#982).
#[cfg(target_os = "windows")]
fn show_pill_noactivate(win: &tauri::WebviewWindow) {
use windows::Win32::UI::WindowsAndMessaging::{ShowWindow, SW_SHOWNOACTIVATE};
let Ok(hwnd) = win.hwnd() else {
log::warn!("pill: could not resolve HWND for non-activating show (#982)");
return;
};
unsafe {
let _ = ShowWindow(hwnd, SW_SHOWNOACTIVATE);
}
}
#[cfg(test)]
mod pill_noactivate_tests {
use super::{with_noactivate_style, WS_EX_NOACTIVATE_BIT};
#[test]
fn adds_noactivate_bit_without_clobbering_existing_style() {
// Stand-in for whatever bits the pill's always_on_top/skip_taskbar
// window already carries (e.g. WS_EX_TOPMOST = 0x00000008) —
// NOACTIVATE must be added on top, never replace them.
let topmost = 0x0000_0008isize;
let updated = with_noactivate_style(topmost);
assert_eq!(
updated & WS_EX_NOACTIVATE_BIT,
WS_EX_NOACTIVATE_BIT,
"NOACTIVATE bit must be set"
);
assert_eq!(updated & topmost, topmost, "pre-existing style bits must survive");
}
#[test]
fn idempotent_if_already_noactivate() {
assert_eq!(with_noactivate_style(WS_EX_NOACTIVATE_BIT), WS_EX_NOACTIVATE_BIT);
}
#[test]
fn matches_documented_win32_value() {
// winuser.h: #define WS_EX_NOACTIVATE 0x08000000L
assert_eq!(WS_EX_NOACTIVATE_BIT, 0x0800_0000);
}
}
// ── Tauri entry ───────────────────────────────────────────────────────────
#[cfg_attr(mobile, tauri::mobile_entry_point)]
@@ -271,6 +398,8 @@ pub fn run() {
commands::get_launch_as_widget,
commands::set_launch_as_widget,
commands::clear_webview_cache_and_relaunch,
crash::get_last_backend_crash,
crash::acknowledge_backend_crash,
])
.setup(move |app| {
app.handle().plugin(tauri_plugin_dialog::init())?;
@@ -328,9 +457,15 @@ pub fn run() {
.skip_taskbar(true)
.center()
.build();
if let Err(e) = result {
if let Err(e) = &result {
log::error!("Failed to create widget window: {e:?}");
}
// Windows: mark the pill non-activating right away so it holds
// for every later `.show()` regardless of call site (#982).
#[cfg(target_os = "windows")]
if let Ok(win) = &result {
mark_pill_noactivate(win);
}
}
app.manage(AppFlags {
@@ -364,12 +499,17 @@ pub fn run() {
if win.move_window(Position::BottomCenter).is_err() {
let _ = win.center();
}
// Windows: show without granting foreground activation
// (#982) — `.show()` on other platforms is unaffected.
#[cfg(target_os = "windows")]
show_pill_noactivate(&win);
#[cfg(not(target_os = "windows"))]
let _ = win.show();
// Don't steal focus on macOS: the simulated ⌘V from
// simulate_paste() must land in the app the user is
// dictating into — focusing the widget would swallow
// it (#287).
#[cfg(not(target_os = "macos"))]
// Don't steal focus on macOS or Windows: the simulated
// ⌘V/Ctrl+V from simulate_paste() must land in the app
// the user is dictating into — focusing the widget would
// swallow it (#287 macOS, #982 Windows).
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
let _ = win.set_focus();
}
let _ = app_handle.emit("tray-dictate", ());
@@ -520,7 +660,9 @@ pub fn run() {
// + focus the widget BEFORE emitting tray-dictate so
// the user sees the pill instead of silent recording.
// Positioning mirrors the global-shortcut handler:
// bottom-center (WhisperFlow style).
// bottom-center (WhisperFlow style). Windows skips the
// focus (and uses a non-activating show) for the same
// reason the global-shortcut handler does — see #982.
if let Some(win) = app.get_webview_window("widget") {
if win.is_visible().unwrap_or(false) {
let _ = app.emit("tray-dictate-stop", ());
@@ -528,8 +670,13 @@ pub fn run() {
if win.move_window(Position::BottomCenter).is_err() {
let _ = win.center();
}
let _ = win.show();
let _ = win.set_focus();
#[cfg(target_os = "windows")]
show_pill_noactivate(&win);
#[cfg(not(target_os = "windows"))]
{
let _ = win.show();
let _ = win.set_focus();
}
let _ = app.emit("tray-dictate", ());
}
} else {
@@ -639,6 +786,7 @@ pub fn run() {
app.manage(bootstrap_state);
app.manage(BackendState {
process: Mutex::new(None),
spawned_at: Mutex::new(None),
});
let app_handle = app.handle().clone();
@@ -737,6 +885,14 @@ pub fn run() {
app.run(|app_handle, event| {
if let tauri::RunEvent::ExitRequested { .. } = event {
// Raise the quitting flag FIRST: exits that don't pass through the
// tray Quit item (macOS ⌘Q, OS session end) would otherwise let a
// death watcher observe our own SIGTERM below and record a false
// "backend crashed" marker (#941).
app_handle
.state::<AppFlags>()
.quitting
.store(true, Ordering::SeqCst);
if let Ok(mut lock) = app_handle.state::<BackendState>().process.lock() {
if let Some(ref mut child) = *lock {
let pid = child.id();
+2 -1
View File
@@ -85,7 +85,8 @@
],
"macOS": {
"minimumSystemVersion": "12.0",
"signingIdentity": "-"
"signingIdentity": "-",
"entitlements": "entitlements.plist"
}
},
"plugins": {
+129 -7
View File
@@ -42,6 +42,8 @@ import WorkspaceVoices from './components/WorkspaceVoices';
import WorkspaceProjects from './components/WorkspaceProjects';
import ErrorBoundary from './components/ErrorBoundary';
import FloatingPill from './components/FloatingPill';
import GlobalAudioPlayer from './components/GlobalAudioPlayer';
import BackendCrashNotice from './components/BackendCrashNotice';
// RemoteAuthGate is mounted at the true outermost provider in main-app.jsx so
// it covers all app states (setup check / wizard / bootstrap), not just the
// main studio return below. Do not re-wrap here double-gating renders two
@@ -72,6 +74,7 @@ import {
CLONE_MAX_SECONDS,
} from './utils/constants';
import { LANG_CODES } from './utils/languages';
import { restoreProjectExtras } from './utils/projectState';
import { API, apiFetch } from './api/client';
import { flushMemory as apiFlushMemory } from './api/system';
import {
@@ -81,6 +84,12 @@ import {
renameProject as apiRenameProject,
} from './api/projects';
import { exportAction, exportReveal, exportRecord } from './api/exports';
import {
clearHistory as apiClearHistory,
setHistoryStarred as apiSetHistoryStarred,
audioUrlWithCacheBust,
} from './api/generate';
import { clearDubHistory as apiClearDubHistory } from './api/dub';
import { isTauri, doubleClickMaximize, fileToMediaUrl, playBlobAudio } from './utils/media';
import { browserDownload } from './utils/download';
@@ -416,9 +425,15 @@ function App() {
const defaultTrack = useAppStore((s) => s.defaultTrack);
const setDefaultTrack = useAppStore((s) => s.setDefaultTrack);
const exportTracks = useAppStore((s) => s.exportTracks);
const setExportTracks = useAppStore((s) => s.setExportTracks);
const previewSegIds = useAppStore((s) => s.previewSegIds);
const speakerClones = useAppStore((s) => s.speakerClones);
const setSpeakerClones = useAppStore((s) => s.setSpeakerClones);
// Multi-language batch picks (P1.4) saved with the project payload.
const multiLangMode = useAppStore((s) => s.multiLangMode);
const setMultiLangMode = useAppStore((s) => s.setMultiLangMode);
const multiLangs = useAppStore((s) => s.multiLangs);
const setMultiLangs = useAppStore((s) => s.setMultiLangs);
const setGlossaryTerms = useAppStore((s) => s.setGlossaryTerms);
const dualSubs = useAppStore((s) => s.dualSubs);
@@ -450,6 +465,8 @@ function App() {
closeDirection,
saveDirection,
setLastGenFingerprints,
fingerprintsByLang,
setFingerprintsByLang,
incrementalPlan,
recomputeIncremental,
} = useSegmentEditing();
@@ -579,7 +596,7 @@ function App() {
fd.append('num_step', '16');
const res = await apiFetch(`${API}/generate`, { method: 'POST', body: fd });
const blob = await res.blob();
await playBlobAudio(blob);
await playBlobAudio(blob, { label: i18n.t('player.generated_audio') });
toast.success(i18n.t('firstrun.first_sound_done'), { duration: 7000 });
} catch {
/* silent — see above */
@@ -952,6 +969,17 @@ function App() {
preserveBg,
defaultTrack,
speakerClones,
// P1.4 multi-language batch setup + export-track prefs travel with
// the project. Additive: loaders default them when absent (see
// utils/projectState.js).
multiLangMode,
multiLangs,
exportTracks,
// P1.3 per-language segment fingerprints, so reopening a project
// keeps every track's "Regen N changed" plan. Additive: legacy
// loaders ignore the key; segments' `translations` maps ride along
// inside dubSegments above.
segHashesByLang: fingerprintsByLang,
},
};
try {
@@ -993,9 +1021,24 @@ function App() {
setDubStep(s.dubStep === 'done' ? 'done' : s.dubSegments?.length ? 'editing' : 'idle');
// Phase 4.5 rehydrate per-segment fingerprints. The incremental plan
// immediately shows "N segments changed" for any segments edited after
// the last generate.
setLastGenFingerprints(s.segHashes || {});
// the last generate. P1.3: prefer the per-language map; a legacy flat
// `segHashes` can only describe the project's saved target language.
if (
s.segHashesByLang &&
typeof s.segHashesByLang === 'object' &&
!Array.isArray(s.segHashesByLang)
) {
setFingerprintsByLang(s.segHashesByLang);
} else {
setLastGenFingerprints(s.segHashes || {}, s.dubLangCode || 'en');
}
setSpeakerClones(s.speakerClones || {});
// P1.4 restore multi-lang picks; legacy payloads default to off/empty
// and leave the in-session exportTracks untouched (null sentinel).
const extras = restoreProjectExtras(s);
setMultiLangMode(extras.multiLangMode);
setMultiLangs(extras.multiLangs);
if (extras.exportTracks) setExportTracks(extras.exportTracks);
toast.success(i18n.t('app.toast_opened', { name: data.name }));
} catch (err) {
toast.error(err.message);
@@ -1045,14 +1088,31 @@ function App() {
})),
);
setDubTranscript(job.full_transcript || '');
setDubLang(item.language || 'Auto');
setDubLangCode(item.language_code || 'und');
// Older DBs froze the language/language_code COLUMNS at the ingest-time
// "" (the UPSERT didn't update them until #P0 fixed it), but the job_data
// JSON always carried the value generation set. Falling back to job_data
// restores existing rows correctly without a migration.
setDubLang(item.language || job.language || 'Auto');
setDubLangCode(item.language_code || job.language_code || 'und');
setDubTracks(Object.keys(job.dubbed_tracks || {}));
setDubStep(Object.keys(job.dubbed_tracks || {}).length > 0 ? 'done' : 'editing');
// Phase 4.5 seg_hashes are written per successful segment by
// dub_generate.py. Reloading a half-generated dub lets the "Regen N
// changed" button resume right where the crash happened.
setLastGenFingerprints(job.seg_hashes || {});
// changed" button resume right where the crash happened. P1.3: prefer
// the per-language map (multi-track jobs); a legacy flat map belongs to
// the job's last-generated language the code restored just above.
if (
job.seg_hashes_by_lang &&
typeof job.seg_hashes_by_lang === 'object' &&
!Array.isArray(job.seg_hashes_by_lang)
) {
setFingerprintsByLang(job.seg_hashes_by_lang);
} else {
setLastGenFingerprints(
job.seg_hashes || {},
item.language_code || job.language_code || 'und',
);
}
// Rehydrate the auto-extracted speaker clones so the CAST dropdown's
// "🎤 From video" option reappears after a reload. Projects that
// predate the speaker-clone feature have an empty map; the Extract
@@ -1082,6 +1142,33 @@ function App() {
toast.success(i18n.t('app.toast_restored_state'));
};
// Generation takes: star/unstar a take so it survives the retention cap and
// never ages off the rail. Optimistic errors only the WS
// generation_history event refreshes the list on success.
const toggleStarHistory = async (item) => {
try {
await apiSetHistoryStarred(item.id, !item.starred);
loadHistory();
} catch (err) {
toast.error(err.message);
}
};
// Load a past take back as the active output: fetch its WAV and hand it to
// the same global mini-player a fresh generation plays through.
const playTakeAsOutput = async (item) => {
try {
const res = await apiFetch(audioUrlWithCacheBust(item.audio_path));
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const blob = await res.blob();
await playBlobAudio(blob, {
label: item.text || i18n.t('player.generated_audio'),
});
} catch (err) {
toast.error(i18n.t('history.load_take_failed', { message: err.message || '' }));
}
};
const deleteHistory = async (id, type) => {
if (!(await askConfirm('Delete this history item?'))) return;
try {
@@ -1098,6 +1185,27 @@ function App() {
}
};
// Clear-all for the workspace history panels (#1032). The control lived in
// the old left Sidebar; the workspace UX overhaul (#374) moved history into
// the right-side WorkspaceHistory panels and the button was dropped in the
// move restore it, scoped per workspace (voice = synth rows, dub = dubs).
const clearWorkspaceHistory = async (type) => {
const count = type === 'dub' ? dubHistory.length : history.length;
if (!(await askConfirm(i18n.t('sidebar.clear_confirm', { count })))) return;
try {
if (type === 'dub') {
await apiClearDubHistory();
loadDubHistory();
} else {
await apiClearHistory();
loadHistory();
}
toast.success(i18n.t('sidebar.history_cleared'));
} catch (err) {
toast.error(err.message);
}
};
// Install-plan screen outranks everything both on a true first run and
// when explicitly requested via `--setup`. Without this, a live backend
// answering /setup/status would route straight to the model wizard and the
@@ -1217,6 +1325,10 @@ function App() {
<FloatingPill />
{/* #941: honest surfacing of backend process crashes (exit code +
stderr tail from the shell's crash marker), with ack-on-view. */}
<BackendCrashNotice />
<Header
mode={mode}
setMode={setMode}
@@ -1441,6 +1553,7 @@ function App() {
dubHistory={dubHistory}
restoreDubHistory={restoreDubHistory}
deleteHistory={deleteHistory}
clearHistory={() => clearWorkspaceHistory('dub')}
/>
</div>
)}
@@ -1537,6 +1650,9 @@ function App() {
handleNativeExport={handleNativeExport}
restoreHistory={restoreHistory}
deleteHistory={deleteHistory}
clearHistory={() => clearWorkspaceHistory('synth')}
toggleStarHistory={toggleStarHistory}
playTakeAsOutput={playTakeAsOutput}
/>
</div>
</div>
@@ -1643,6 +1759,12 @@ function App() {
</Suspense>
)}
{/* GLOBAL AUDIO MINI-PLAYER (grid row 3, above the footer)
Subsumes the #1032 PlaybackStopPill: waveform + seek + time + stop
for every playBlobAudio playback that has no on-screen player. As a
real grid row it can never overlap row-2 content or the footer. */}
<GlobalAudioPlayer />
{/* ═══ BOTTOM LOGS PANEL (VSCode-style) ═══ */}
<Suspense fallback={null}>
<LogsFooter />
+32
View File
@@ -8,6 +8,16 @@
// and the API, so a remote device on http://<host>:<share-port> must hit
// that same origin — NOT a hardcoded :3900, which is cross-origin (CORS)
// and loopback-only/unreachable from another machine.
// Explicit .ts extension: tests/frontend/apiClient.test.mjs loads this module
// under `node --experimental-strip-types`, whose ESM resolver requires real
// file extensions (tsconfig has allowImportingTsExtensions for tsc).
import {
getUnacknowledgedBackendCrash,
describeCrashExit,
crashAge,
type BackendCrashMarker,
} from '../utils/backendCrash.ts';
const viteEnv = import.meta.env ?? {};
// Remote-backend settings (Wave 2.3): user-configured in Settings → Sharing.
// localStorage so the choice survives restarts; read once at module load —
@@ -143,6 +153,28 @@ export async function apiFetch(path: string, opts: RequestInit = {}): Promise<Re
await new Promise((r) => setTimeout(r, TRANSPORT_RETRY_BACKOFF_MS[attempt]));
continue;
}
// #941: if the desktop shell recorded an unacknowledged backend crash,
// tell the honest story instead of the vague "can't reach" — and let
// BackendCrashNotice raise its "View crash details" affordance.
let crash: BackendCrashMarker | null = null;
try {
crash = await getUnacknowledgedBackendCrash();
} catch {
/* forensics unavailable — fall through to the generic message */
}
if (crash) {
try {
window.dispatchEvent(new CustomEvent('ov:backend-crashed', { detail: crash }));
} catch {
/* no window (tests) — the ApiError below still tells the story */
}
throw new ApiError(
`The local OmniVoice backend crashed (${describeCrashExit(crash)}) ${crashAge(crash)} ago ` +
'and is being restarted — this request could not reach it. ' +
'Open the crash notice for the error output, or check Settings → Logs → Backend.',
{ status: 0, detail: lastDetail },
);
}
throw new ApiError(
"Can't reach the local OmniVoice backend — it may still be starting up, or it stopped. " +
'Wait a few seconds and try again; if it persists, restart the app (or check Settings → Logs → Backend).',
+17
View File
@@ -111,6 +111,23 @@ export async function clearDubHistory(): Promise<Response> {
return apiFetch('/dub/history', { method: 'DELETE' });
}
export interface DubTrackInfo {
path?: string;
language?: string;
language_code?: string;
duration?: number;
timing_strategy?: string;
}
/** Per-track metadata (duration, timing strategy, ) keyed by language code.
* Backs the track-pill tooltips; the store only carries the track codes. */
export async function dubListTracks(jobId: string): Promise<Record<string, DubTrackInfo>> {
const res = await apiJson<{ tracks?: Record<string, DubTrackInfo> }>(
`/dub/tracks/${encodeURIComponent(jobId)}`,
);
return res?.tracks || {};
}
export interface DubQCResponse {
engine: string;
total: number;
+8 -1
View File
@@ -46,8 +46,15 @@ export async function listEngines(): Promise<AllEnginesResponse> {
export async function selectEngine(
family: EngineFamily,
backendId: string,
modelId?: string,
): Promise<SelectEngineResponse> {
return apiPost<SelectEngineResponse>('/engines/select', { family, backend_id: backendId });
return apiPost<SelectEngineResponse>('/engines/select', {
family,
backend_id: backendId,
// Only mlx-audio's curated-model picker (#981) sets this — omit
// entirely rather than send `undefined`/null for every other call site.
...(modelId ? { model_id: modelId } : {}),
});
}
/**
+8
View File
@@ -16,6 +16,14 @@ export async function clearHistory(): Promise<Response> {
return apiFetch('/history', { method: 'DELETE' });
}
export async function setHistoryStarred(id: string, starred: boolean): Promise<unknown> {
return apiJson(`/history/${id}/starred`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ starred }),
});
}
export function audioUrl(filename: string): string {
return `${API}/audio/${filename}`;
}
+24
View File
@@ -40,6 +40,19 @@ interface EngineBackend {
effective_device?: EffectiveDevice;
routing_status?: RoutingStatus;
routing_reason?: string | null;
// #981 — mlx-audio ONLY: it multiplexes 7+ curated models behind one
// backend id, so its entry also carries the roster + current pick so
// Settings can render a model picker. Absent on every other backend.
curated_models?: CuratedModel[];
active_model_id?: string;
}
// #981 — one of mlx-audio's curated models (see backend
// MLXAudioBackend.CURATED_MODELS / _MLX_AUDIO_MODEL_LABELS).
export interface CuratedModel {
key: string;
label: string;
repo_id: string;
}
interface EngineFamilyResponse {
@@ -231,6 +244,17 @@ export interface DubTranslateResponse {
text_original?: string;
rate_ratio?: number;
rate_error?: string;
/** Pre-synthesis duration plan (backend services/duration_planner.py). */
plan?: {
status: 'fits' | 'tight' | 'impossible';
est_dur_s: number;
available_s: number;
est_overrun_s: number;
calibrated: boolean;
/** Opt-in LLM condensation suggestion (request condense=true only). */
suggested_text?: string;
suggested_est_dur_s?: number;
};
}[];
}
@@ -0,0 +1,153 @@
import { useCallback, useEffect, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { AlertTriangle, X } from 'lucide-react';
import { Button, Dialog } from '../ui';
import {
acknowledgeBackendCrash,
crashAge,
describeCrashExit,
getUnacknowledgedBackendCrash,
} from '../utils/backendCrash';
import { openExternal } from '../api/external';
import { buildBugReportUrl } from '../utils/bugReport';
/**
* BackendCrashNotice the honest half of #941.
*
* When the backend PROCESS dies, the desktop shell records a crash marker
* (src-tauri/src/crash.rs). This component surfaces it: a banner naming the
* exit code and when it happened, with a "View crash details" affordance that
* shows the captured stderr tail and a report path. Sources:
* - `ov:backend-crashed` window events, dispatched by api/client.ts when a
* request fails against a freshly crashed backend, and
* - a mount-time check, so a crash that happened with no request in flight
* (or a crash-loop that forced an app restart) still gets told.
*
* Viewing or dismissing acknowledges the marker (it is retained on disk so
* bug reports can still attach the evidence). Outside the Tauri shell the
* marker getters resolve null and this renders nothing.
*/
export default function BackendCrashNotice() {
const { t } = useTranslation();
const [marker, setMarker] = useState(null);
const [showDetails, setShowDetails] = useState(false);
useEffect(() => {
let cancelled = false;
getUnacknowledgedBackendCrash()
.then((m) => {
if (!cancelled && m) setMarker(m);
})
.catch(() => {});
const onCrash = (e) => {
if (e?.detail) setMarker(e.detail);
};
window.addEventListener('ov:backend-crashed', onCrash);
return () => {
cancelled = true;
window.removeEventListener('ov:backend-crashed', onCrash);
};
}, []);
const view = useCallback(() => {
setShowDetails(true);
// Ack on view the user has seen the honest story; the marker itself
// stays on disk for bug-report attachment.
acknowledgeBackendCrash().catch(() => {});
}, []);
const dismiss = useCallback(() => {
acknowledgeBackendCrash().catch(() => {});
setShowDetails(false);
setMarker(null);
}, []);
if (!marker) return null;
const exit = describeCrashExit(marker);
const ago = crashAge(marker);
return (
<>
<div
role="alert"
className="fixed left-1/2 top-[var(--space-4)] z-[70] flex w-[min(600px,92vw)] -translate-x-1/2 items-center gap-[var(--space-3)] rounded-lg border border-border bg-bg-elev-1 px-[var(--space-4)] py-[var(--space-3)] shadow-lg backdrop-blur-md"
>
<AlertTriangle size={16} className="shrink-0 text-danger" aria-hidden />
<span className="flex-1 text-[length:var(--text-sm)] text-fg">
{t('crash.notice', { exit, ago })}
</span>
<Button variant="subtle" size="sm" onClick={view}>
{t('crash.view')}
</Button>
<Button
variant="ghost"
size="sm"
iconSize="sm"
onClick={dismiss}
title={t('crash.dismiss')}
>
<X size={12} />
</Button>
</div>
<Dialog
open={showDetails}
onClose={() => {
setShowDetails(false);
setMarker(null);
}}
title={t('crash.details_title')}
size="lg"
footer={
<>
<Button
variant="subtle"
onClick={async () => {
try {
// buildBugReportUrl attaches the crash marker (exit code +
// scrubbed stderr tail) automatically the report arrives
// WITH the evidence.
await openExternal(
await buildBugReportUrl({ title: `[Crash] Backend died (${exit})` }),
);
} catch (e) {
console.warn('[BackendCrashNotice] report action failed', e);
}
}}
>
{t('errors.report')}
</Button>
<Button variant="primary" onClick={dismiss}>
{t('common.close')}
</Button>
</>
}
>
<div className="flex flex-col gap-[var(--space-4)]">
<p className="m-0 text-[length:var(--text-sm)] text-fg-muted">
{t('crash.details_intro', { exit, ago })}
</p>
<dl className="m-0 grid grid-cols-[max-content_1fr] gap-x-[var(--space-5)] gap-y-[var(--space-2)] text-[length:var(--text-sm)]">
<dt className="text-fg-subtle">{t('crash.field_exit')}</dt>
<dd className="m-0 font-mono text-fg">{exit}</dd>
<dt className="text-fg-subtle">{t('crash.field_when')}</dt>
<dd className="m-0 text-fg">{new Date(marker.ts * 1000).toLocaleString()}</dd>
<dt className="text-fg-subtle">{t('crash.field_uptime')}</dt>
<dd className="m-0 text-fg">{t('crash.uptime_value', { count: marker.uptime_s })}</dd>
<dt className="text-fg-subtle">{t('crash.field_version')}</dt>
<dd className="m-0 text-fg">{marker.backend_version}</dd>
</dl>
<div>
<div className="mb-[var(--space-2)] text-[length:var(--text-sm)] text-fg-subtle">
{t('crash.stderr_title')}
</div>
<pre className="m-0 max-h-[40vh] overflow-auto rounded-md border border-border bg-bg-elev-2 p-[var(--space-3)] font-mono text-[length:var(--text-xs)] leading-relaxed text-fg whitespace-pre-wrap">
{marker.last_stderr || t('crash.no_stderr')}
</pre>
</div>
</div>
</Dialog>
</>
);
}
@@ -0,0 +1,82 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import BackendCrashNotice from './BackendCrashNotice';
import { acknowledgeBackendCrash, getUnacknowledgedBackendCrash } from '../utils/backendCrash';
// #941: the crash-notice branch a recorded backend death must surface the
// honest message (exit code + age) with a "View crash details" affordance,
// and viewing/dismissing must acknowledge the marker.
vi.mock('../utils/backendCrash', async (importOriginal) => {
const actual = await importOriginal();
return {
...actual,
getUnacknowledgedBackendCrash: vi.fn().mockResolvedValue(null),
acknowledgeBackendCrash: vi.fn().mockResolvedValue(undefined),
};
});
vi.mock('../utils/bugReport', () => ({
buildBugReportUrl: vi.fn().mockResolvedValue('https://example.test/issues/new'),
}));
vi.mock('../api/external', () => ({
openExternal: vi.fn().mockResolvedValue(undefined),
}));
const MARKER = {
ts: Math.floor(Date.now() / 1000) - 12,
exit_code: 134,
signal: null,
exit_desc: 'exit status: 134',
backend_version: '0.3.10',
uptime_s: 87,
last_stderr: 'CUDA error: an illegal memory access was encountered',
acknowledged: false,
};
describe('BackendCrashNotice', () => {
beforeEach(() => {
vi.clearAllMocks();
getUnacknowledgedBackendCrash.mockResolvedValue(null);
});
it('renders nothing when the shell reports no crash', async () => {
const { container } = render(<BackendCrashNotice />);
await waitFor(() => expect(getUnacknowledgedBackendCrash).toHaveBeenCalled());
expect(container).toBeEmptyDOMElement();
});
it('shows the honest message and the details affordance for a fresh marker', async () => {
getUnacknowledgedBackendCrash.mockResolvedValue(MARKER);
render(<BackendCrashNotice />);
const alert = await screen.findByRole('alert');
// Honest: names the exit code instead of a vague "can't reach".
expect(alert.textContent).toContain('crashed');
expect(alert.textContent).toContain('exit code 134');
expect(screen.getByRole('button', { name: /view crash details/i })).toBeInTheDocument();
});
it('surfaces a crash pushed via the ov:backend-crashed event', async () => {
render(<BackendCrashNotice />);
await waitFor(() => expect(getUnacknowledgedBackendCrash).toHaveBeenCalled());
window.dispatchEvent(new CustomEvent('ov:backend-crashed', { detail: MARKER }));
const alert = await screen.findByRole('alert');
expect(alert.textContent).toContain('exit code 134');
});
it('acks on view and shows the stderr tail in the details dialog', async () => {
getUnacknowledgedBackendCrash.mockResolvedValue(MARKER);
render(<BackendCrashNotice />);
fireEvent.click(await screen.findByRole('button', { name: /view crash details/i }));
expect(acknowledgeBackendCrash).toHaveBeenCalledTimes(1);
expect(await screen.findByText(/illegal memory access/)).toBeInTheDocument();
expect(screen.getByText('Backend crash details')).toBeInTheDocument();
});
it('ack + clear on dismiss', async () => {
getUnacknowledgedBackendCrash.mockResolvedValue(MARKER);
render(<BackendCrashNotice />);
await screen.findByRole('alert');
fireEvent.click(screen.getByRole('button', { name: /dismiss/i }));
expect(acknowledgeBackendCrash).toHaveBeenCalledTimes(1);
await waitFor(() => expect(screen.queryByRole('alert')).not.toBeInTheDocument());
});
});
+36
View File
@@ -275,6 +275,42 @@ function DubSegmentRow({
📖 {seg.rate_ratio.toFixed(2)}×
</span>
)}
{/* Pre-synthesis duration plan (backend duration_planner): warn about
tight/impossible segments BEFORE GPU time is spent. Informational
only generation is never blocked. */}
{seg.plan && (seg.plan.status === 'tight' || seg.plan.status === 'impossible') && (
<span
className="text-[0.48rem] mt-[1px] inline-flex items-center gap-[1px]"
style={{ color: seg.plan.status === 'impossible' ? '#fb4934' : '#fabd2f' }}
title={t(
seg.plan.status === 'impossible'
? 'segment.plan_impossible_title'
: 'segment.plan_tight_title',
{
est: (seg.plan.est_dur_s || 0).toFixed(1),
avail: (seg.plan.available_s || 0).toFixed(1),
seconds: (seg.plan.est_overrun_s || 0).toFixed(1),
},
)}
>
<AlertCircle size={8} />{' '}
{seg.plan.status === 'impossible'
? t('segment.plan_impossible', {
seconds: (seg.plan.est_overrun_s || 0).toFixed(1),
})
: t('segment.plan_tight')}
</span>
)}
{seg.plan && seg.plan.suggested_text && seg.plan.suggested_text !== seg.text && (
<button
onClick={() => onEditField(seg.id, 'text', seg.plan.suggested_text)}
disabled={disabled}
title={t('segment.plan_apply_title', { text: seg.plan.suggested_text })}
className="bg-transparent border-none text-[#83a598] cursor-pointer p-0 mt-[1px] text-[0.48rem] text-left"
>
{t('segment.plan_apply')}
</button>
)}
</span>
<input
@@ -17,7 +17,7 @@ import { useTranslation } from 'react-i18next';
import { listEngines, getEngineHealth, selfTestEngine } from '../api/engines';
import { copyText } from '../utils/copyText';
import { ChevronRight } from 'lucide-react';
import { Badge, Button, Segmented, Table } from '../ui';
import { Badge, Button, Segmented, Select, Table } from '../ui';
import { cn } from '@/lib/utils';
import SupertonicLicenseDialog from './SupertonicLicenseDialog';
@@ -63,12 +63,18 @@ function reasonMentionsLicense(reason) {
*
* Props:
* - family: 'tts' | 'asr' | 'llm' default 'tts'
* - onSelect?: (family, backendId) => Promise<void> optional when
* provided, a "Use" button appears next to "Test engine" for
* - onSelect?: (family, backendId, modelId?) => Promise<void> optional
* when provided, a "Use" button appears next to "Test engine" for
* available, non-active rows. Lets the matrix double as an engine
* picker so Settings doesn't need a parallel table.
* picker so Settings doesn't need a parallel table. The optional third
* arg is set only by mlx-audio's curated-model picker (#981).
* - activeId?: string the currently-active backend id for this
* family. Used to render the "active" badge.
* - showFamilyTabs?: boolean default true. When false, the matrix is
* pinned to `family` no TTS/ASR/LLM switcher, and the header names
* the family ("ASR Engines") instead of the generic matrix title.
* Settings Engines stacks one pinned matrix per family so the ASR
* and LLM pickers are visible instead of tucked behind a tab.
*/
const FAMILY_META = {
tts: { label: 'TTS', icon: Cpu },
@@ -139,6 +145,10 @@ function normalizeEntry(entry) {
effective_device: entry.effective_device || null,
routing_status: entry.routing_status || null,
routing_reason: entry.routing_reason || null,
// #981 mlx-audio ONLY: the curated-model roster + current pick.
// null/absent on every other backend, which never renders a picker.
curated_models: Array.isArray(entry.curated_models) ? entry.curated_models : null,
active_model_id: entry.active_model_id || null,
};
}
@@ -153,8 +163,10 @@ export default function EngineCompatibilityMatrix({
family = 'tts',
onSelect = null,
activeId = null,
// Test-friendly overrides let the RTL suite mock the API layer
// without resorting to module-level vi.mock incantations.
showFamilyTabs = true,
// Injectable API layer lets the RTL suite mock it without module-level
// vi.mock incantations, and lets EnginesTab share one in-flight
// GET /engines across its stacked per-family matrices.
apiListEngines = listEngines,
apiGetEngineHealth = getEngineHealth,
apiSelfTestEngine = selfTestEngine,
@@ -291,6 +303,18 @@ export default function EngineCompatibilityMatrix({
setTimeout(() => setCopiedId((c) => (c === id ? null : c)), 1500);
}, []);
// #981 mlx-audio's curated-model picker. Reuses the same onSelect the
// "Use" button calls, with the curated model key as the optional third
// arg, then reloads so active_model_id reflects the new pick immediately.
const changeModel = useCallback(
async (id, modelId) => {
if (!onSelect || !modelId) return;
await onSelect(activeFamily, id, modelId);
reload();
},
[onSelect, activeFamily, reload],
);
const COLUMNS = [
{ key: 'name', label: t('engines.matrixTitle').split(' ')[0] || 'Engine', flex: 3 },
{ key: 'status', label: t('engines.status'), width: 130, align: 'center' },
@@ -330,12 +354,19 @@ export default function EngineCompatibilityMatrix({
// TTS-05: the license dialog registered for the engine awaiting acceptance
// (or null). Capitalized so JSX renders it as a component below.
const LicenseDialog = licenseDialogFor ? LICENSE_DIALOGS[licenseDialogFor] : null;
// Pinned mode: the header names the family (with its icon) since there is
// no switcher to say which family this table is.
const familyMeta = FAMILY_META[activeFamily] || FAMILY_META.tts;
const TitleIcon = showFamilyTabs ? Layers : familyMeta.icon;
return (
<section className="engine-matrix flex flex-col gap-[var(--space-3,8px)]">
<header className="engine-matrix__head flex items-center justify-between gap-[12px]">
<h3 className="engine-matrix__title inline-flex items-center gap-[6px] m-0 text-[13px] font-semibold text-[color:var(--chrome-fg,currentColor)]">
<Layers size={14} /> {t('engines.matrixTitle')}
<TitleIcon size={14} />{' '}
{showFamilyTabs
? t('engines.matrixTitle')
: t('engines.familyMatrixTitle', { family: familyMeta.label })}
</h3>
<Button
size="sm"
@@ -348,7 +379,7 @@ export default function EngineCompatibilityMatrix({
</Button>
</header>
{families.length > 1 && (
{showFamilyTabs && families.length > 1 && (
<Segmented
size="sm"
value={activeFamily}
@@ -413,6 +444,35 @@ export default function EngineCompatibilityMatrix({
<code className="engine-matrix__id font-mono text-[11px] text-[color:var(--chrome-fg-muted,#888)]">
{b.id}
</code>
{/* #981 mlx-audio multiplexes 7+ curated models behind this
one backend id (Kokoro, CSM, OuteTTS, ); without this
picker there's no way to load anything but the default
(Kokoro) even after downloading a different model's
weights in Settings Models. Disabled while the row
itself isn't available/selectable, matching the "Use"
button's gating. */}
{b.curated_models && b.curated_models.length > 0 && (
<div className="engine-matrix__model-picker flex items-center gap-[6px] mt-[2px]">
<span className="text-[11px] text-[color:var(--chrome-fg-muted,#888)]">
{t('engines.curatedModelLabel')}
</span>
<Select
size="sm"
className="w-auto min-w-[150px]"
value={b.active_model_id || ''}
disabled={!onSelect || !b.available}
onChange={(e) => changeModel(b.id, e.target.value)}
aria-label={t('engines.curatedModelAria', { engine: b.display_name })}
data-testid={`curated-model-select-${b.id}`}
>
{b.curated_models.map((m) => (
<option key={m.key} value={m.key}>
{m.label}
</option>
))}
</Select>
</div>
)}
{/* For available rows, show install_hint inline (one line usually
a parenthetical like "(bundled — no extra install needed)").
For unavailable rows, collapse reason + install_hint + last_error
+6 -2
View File
@@ -259,7 +259,11 @@ export default function ExportModal({
onClose?.();
};
const runClips = () => {
handleAudioExport?.(`${API}/dub/export-segments/${jobId}`, 'segments.zip');
// Ask for the ACTIVE track's per-segment clips (P1.3 the cache is
// language-keyed now); omitted lang falls back to the last-generated
// track server-side, which is all a legacy single-track job has.
const langQ = dubLangCode ? `?lang=${encodeURIComponent(dubLangCode)}` : '';
handleAudioExport?.(`${API}/dub/export-segments/${jobId}${langQ}`, 'segments.zip');
onClose?.();
};
@@ -282,7 +286,7 @@ export default function ExportModal({
return createPortal(
<div
className="pointer-events-none fixed inset-x-0 bottom-[var(--logs-footer-height,28px)] z-[90] flex justify-center"
className="pointer-events-none fixed inset-x-0 bottom-[calc(var(--logs-footer-height,28px)+var(--audio-dock-height,0px))] z-[90] flex justify-center"
role="dialog"
aria-modal="false"
aria-label={t('exportModal.export_options')}
@@ -0,0 +1,223 @@
/**
* GlobalAudioPlayer persistent bottom mini-player for "invisible" audio.
*
* `playBlobAudio` (playback source 'output') plays the generate auto-play,
* profile & dub-segment previews, story lines, gallery voices and Projects
* renders through a bare Audio()/AudioContext with no on-screen player. Its
* only global affordance used to be the stop-only PlaybackStopPill (#1032)
* this bar subsumes it: waveform (peaks decoded once from the blob already in
* hand), click/drag/keyboard seek, play/pause, elapsed/total time, a source
* label and a stop button, on every page (mounted once in App.jsx).
*
* Exclusion semantics are the pill's, unchanged: ONLY source 'output'
* renders here. Sources with their own visible player UI (WaveformPlayer
* instances, 'design-preview', 'demo-output') stay in-place.
*
* Layout: a real grid row of .app-container (row 3, directly above the
* LogsFooter see index.css). Content in row 2 physically ends at the bar's
* top edge, so the fixed-overlay overlap class the pill had at 1440×900
* (covering the studio's Production Overrides row) is impossible by
* construction. While visible it publishes --audio-dock-height so the fixed
* overlays that anchor above the footer (FloatingPill, VoicePreview,
* ExportModal, compare drawer) ride above the bar too.
*/
import React, { useEffect, useRef, useState } from 'react';
import { Pause, Play, Square } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import {
pauseActivePlayback,
resumeActivePlayback,
seekActivePlayback,
stopActivePlayback,
usePlaybackTrack,
} from '../utils/playback';
const DOCK_H = 44; // collapsed-chrome scale: header/footer bars are 28px, player needs touch room
const fmt = (s) => {
if (!isFinite(s) || s < 0) s = 0;
const m = Math.floor(s / 60);
const sec = Math.floor(s % 60);
return `${m}:${String(sec).padStart(2, '0')}`;
};
// Same visual language as WaveformPlayer's wavesurfer config (bar width 2,
// gap 1, wave/progress colors), just hand-drawn on a canvas the peaks are
// precomputed in utils/media.js, so no wavesurfer instance (and no second
// decode/fetch) is needed here.
const WAVE_COLOR = 'rgba(168,153,132,0.45)';
const PROGRESS_COLOR = 'rgba(211,134,155,0.75)';
const CURSOR_COLOR = '#d3869b';
function WaveCanvas({ peaks, progress }) {
const wrapRef = useRef(null);
const canvasRef = useRef(null);
const [width, setWidth] = useState(0);
useEffect(() => {
const el = wrapRef.current;
if (!el || typeof ResizeObserver === 'undefined') return undefined;
const ro = new ResizeObserver(() => setWidth(el.clientWidth));
ro.observe(el);
setWidth(el.clientWidth);
return () => ro.disconnect();
}, []);
useEffect(() => {
const canvas = canvasRef.current;
const ctx = canvas?.getContext?.('2d');
if (!ctx) return; // jsdom / very old engines seek + time still work
const w = width || canvas.clientWidth;
const h = canvas.clientHeight || 28;
if (!w || !h) return;
const dpr = window.devicePixelRatio || 1;
canvas.width = w * dpr;
canvas.height = h * dpr;
ctx.scale(dpr, dpr);
ctx.clearRect(0, 0, w, h);
const playedX = Math.max(0, Math.min(1, progress)) * w;
if (peaks && peaks.length) {
const barW = 2;
const gap = 1;
const count = Math.max(1, Math.floor(w / (barW + gap)));
for (let i = 0; i < count; i++) {
const x = i * (barW + gap);
const peak = peaks[Math.floor((i / count) * peaks.length)] || 0;
const barH = Math.max(2, peak * (h - 2));
ctx.fillStyle = x + barW <= playedX ? PROGRESS_COLOR : WAVE_COLOR;
ctx.fillRect(x, (h - barH) / 2, barW, barH);
}
} else {
// No peaks (decode unavailable e.g. the Tauri streamed fallback):
// a plain progress track, same colors.
ctx.fillStyle = WAVE_COLOR;
ctx.fillRect(0, h / 2 - 1.5, w, 3);
ctx.fillStyle = PROGRESS_COLOR;
ctx.fillRect(0, h / 2 - 1.5, playedX, 3);
}
// Playhead cursor.
ctx.fillStyle = CURSOR_COLOR;
ctx.fillRect(Math.min(playedX, w - 1), 0, 1.5, h);
}, [peaks, progress, width]);
return (
<div ref={wrapRef} className="w-full h-full">
<canvas ref={canvasRef} className="block w-full h-full" aria-hidden="true" />
</div>
);
}
function PlayerBar({ track }) {
const { t } = useTranslation();
const { label, paused, currentTime, duration, peaks, canSeek, canPause } = track;
const scrubbingRef = useRef(false);
const seekable = canSeek && duration > 0;
const seekToClientX = (target, clientX) => {
const rect = target.getBoundingClientRect();
if (!rect.width) return;
const frac = Math.max(0, Math.min(1, (clientX - rect.left) / rect.width));
seekActivePlayback(frac * duration);
};
const onPointerDown = (e) => {
if (!seekable) return;
scrubbingRef.current = true;
e.currentTarget.setPointerCapture?.(e.pointerId);
seekToClientX(e.currentTarget, e.clientX);
};
const onPointerMove = (e) => {
if (!seekable || !scrubbingRef.current) return;
seekToClientX(e.currentTarget, e.clientX);
};
const endScrub = () => {
scrubbingRef.current = false;
};
const onKeyDown = (e) => {
if (!seekable) return;
if (e.key === 'ArrowRight') seekActivePlayback(Math.min(duration, currentTime + 5));
else if (e.key === 'ArrowLeft') seekActivePlayback(Math.max(0, currentTime - 5));
else if (e.key === 'Home') seekActivePlayback(0);
else if (e.key === 'End') seekActivePlayback(duration);
else return;
e.preventDefault();
};
return (
<div
className="global-audio-dock flex items-center gap-[10px] px-[10px] [background:var(--chrome-bg)] [border-top:1px_solid_var(--chrome-border)] [color:var(--chrome-fg)] select-none"
style={{ height: DOCK_H }}
role="region"
aria-label={t('player.now_playing')}
data-testid="global-audio-player"
>
{canPause && (
<button
type="button"
className="wf-player__btn shrink-0 inline-flex items-center justify-center w-[28px] h-[28px] border-none rounded-full cursor-pointer text-[color:var(--color-fg-inverse)] bg-[var(--color-brand)] [transition:background_0.15s_ease,transform_0.1s_ease] hover:bg-[var(--color-brand-hover)] active:scale-[0.94]"
onClick={paused ? resumeActivePlayback : pauseActivePlayback}
aria-label={paused ? t('player.play') : t('player.pause')}
>
{paused ? <Play size={14} /> : <Pause size={14} />}
</button>
)}
<span
className="shrink-0 max-w-[220px] truncate text-[11.5px] [color:var(--chrome-fg-muted)]"
title={label || t('player.untitled')}
>
{label || t('player.untitled')}
</span>
<div
className={`flex-1 min-w-0 h-[28px] ${seekable ? 'cursor-pointer' : 'cursor-default'}`}
role="slider"
tabIndex={seekable ? 0 : -1}
aria-label={t('player.seek')}
aria-valuemin={0}
aria-valuemax={Math.round(duration)}
aria-valuenow={Math.round(currentTime)}
aria-valuetext={`${fmt(currentTime)} / ${fmt(duration)}`}
aria-disabled={!seekable}
onPointerDown={onPointerDown}
onPointerMove={onPointerMove}
onPointerUp={endScrub}
onPointerCancel={endScrub}
onKeyDown={onKeyDown}
>
<WaveCanvas peaks={peaks} progress={duration > 0 ? currentTime / duration : 0} />
</div>
<span className="shrink-0 [font-variant-numeric:tabular-nums] text-[11px] [color:var(--chrome-fg-muted)] whitespace-nowrap">
{fmt(currentTime)} / {fmt(duration)}
</span>
<button
type="button"
className="shrink-0 flex items-center justify-center w-[var(--chrome-icon-btn)] h-[var(--chrome-icon-btn)] rounded-[3px] bg-transparent border-0 cursor-pointer [color:var(--chrome-fg-muted)] hover:[color:var(--chrome-fg)] hover:[background:var(--chrome-hover-bg)] focus-visible:[outline:2px_solid_var(--chrome-accent)] focus-visible:[outline-offset:1px]"
onClick={stopActivePlayback}
title={t('player.stop')}
aria-label={t('player.stop')}
>
<Square size={12} />
</button>
</div>
);
}
export default function GlobalAudioPlayer() {
const track = usePlaybackTrack();
// Exact PlaybackStopPill routing: only bare 'output' playback docks here.
const visible = track?.source === 'output';
// Publish the dock height so fixed overlays anchored above the LogsFooter
// (--logs-footer-height consumers) stack above the bar instead of over it.
useEffect(() => {
document.documentElement.style.setProperty(
'--audio-dock-height',
visible ? `${DOCK_H}px` : '0px',
);
return () => {
document.documentElement.style.setProperty('--audio-dock-height', '0px');
};
}, [visible]);
if (!visible) return null;
return <PlayerBar track={track} />;
}
+7
View File
@@ -376,6 +376,13 @@ export default function Header({
<div className="flex flex-col gap-[1px] min-w-0">
<span className="text-[12px] text-[var(--color-fg)] font-medium">
{m.name}
{/* Resident-but-not-routed engine (e.g. OmniVoice still in
VRAM after switching to another backend) say so. */}
{m.is_active_engine === false && (
<span className="ml-[6px] text-[10px] font-normal text-[var(--color-fg-subtle)] [font-family:var(--font-mono)]">
{t('header.model_not_active')}
</span>
)}
</span>
<span className="text-[10px] text-[var(--color-fg-subtle)] [font-family:var(--font-mono)]">
{m.device} {m.vram_mb > 0 ? `· ${m.vram_mb.toFixed(0)} MB` : ''}
-27
View File
@@ -73,9 +73,6 @@ export default function NavRail({ mode, setMode, side = 'left', onFlipSide }) {
[t],
);
const donateLabel = t('donate.pill', { defaultValue: 'Support OmniVoice' });
const donateActive = mode === 'donate';
// `nav-rail` is retained purely as the layout hook the (out-of-scope)
// `.app-container > .nav-rail` grid rules position by; all visual styling now
// lives in the utilities below. Border flips to the inner edge when on the right.
@@ -84,17 +81,6 @@ export default function NavRail({ mode, setMode, side = 'left', onFlipSide }) {
? '[border-left:1px_solid_var(--chrome-border)]'
: '[border-right:1px_solid_var(--chrome-border)]';
// Quiet "Support" pill (was `.rail-btn.donate-pill`): neutral at rest, warms to
// the accent on hover/active.
const donateState = donateActive
? 'text-[var(--chrome-accent)] bg-[var(--chrome-accent-bg)] [border:1px_solid_var(--chrome-accent-border)]'
: 'bg-transparent text-[var(--chrome-fg-dim)] [border:1px_solid_transparent] hover:bg-[color-mix(in_srgb,var(--chrome-accent)_10%,transparent)] hover:text-[var(--chrome-accent)]';
const heartBase =
'text-[16px] leading-none [transition:filter_0.16s,opacity_0.16s,transform_0.16s] group-hover:[transform:scale(1.1)] motion-reduce:[transition:none] motion-reduce:group-hover:[transform:none]';
const heartState = donateActive
? 'opacity-100 [filter:grayscale(0)]'
: 'opacity-75 [filter:grayscale(0.55)] group-hover:opacity-100 group-hover:[filter:grayscale(0)]';
return (
<aside
className={`nav-rail z-50 flex select-none flex-col items-center gap-[6px] bg-[var(--chrome-bg)] py-[8px] ${asideBorder}`}
@@ -111,19 +97,6 @@ export default function NavRail({ mode, setMode, side = 'left', onFlipSide }) {
))}
</div>
<div className="flex flex-col items-center gap-[4px]">
{/* Quiet "Support" pill warms to the accent on hover, opens the
donate page. Sits with the footer nav (Settings / flip). (#007) */}
<button
onClick={() => setMode('donate')}
title={donateLabel}
aria-label={donateLabel}
className={`${RAIL_BTN_BASE} ${donateState}`}
>
<span className={`${heartBase} ${heartState}`} aria-hidden="true">
🩷
</span>
<span className={railLabelCls(side)}>{donateLabel}</span>
</button>
{footerItems.map((it) => (
<RailBtn
key={it.id}
+17 -7
View File
@@ -1,8 +1,17 @@
import React, { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react';
import React, {
useCallback,
useEffect,
useLayoutEffect,
useMemo,
useRef,
useState,
useSyncExternalStore,
} from 'react';
import { useTranslation } from 'react-i18next';
import { Play, Headphones } from 'lucide-react';
import {
REGION_COLORS,
getRegionColors,
subscribeRegionColors,
SNAP_PX,
visibleSegmentRange,
snapTime,
@@ -135,13 +144,14 @@ export default function SegmentTrack({
[effSegments, viewStart, viewEnd],
);
// Palette snapshot re-blends against the new --chrome-bg on theme change
// (#963) new array identity per re-blend, so the memo below recolors.
const regionColors = useSyncExternalStore(subscribeRegionColors, getRegionColors);
const speakerColor = useMemo(() => {
const speakers = [...new Set(segments.map((s) => s.speaker_id).filter(Boolean))];
const bySpeaker = new Map(
speakers.map((sp, i) => [sp, REGION_COLORS[i % REGION_COLORS.length]]),
);
return (seg, idx) => bySpeaker.get(seg.speaker_id) || REGION_COLORS[idx % REGION_COLORS.length];
}, [segments]);
const bySpeaker = new Map(speakers.map((sp, i) => [sp, regionColors[i % regionColors.length]]));
return (seg, idx) => bySpeaker.get(seg.speaker_id) || regionColors[idx % regionColors.length];
}, [segments, regionColors]);
// Onset tick strip (one viewport-sized canvas, non-interactive)
useEffect(() => {
+42 -2
View File
@@ -1,6 +1,6 @@
import React from 'react';
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen, fireEvent } from '@testing-library/react';
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { render, screen, fireEvent, act } from '@testing-library/react';
import SegmentTrack from './SegmentTrack';
// Mocked transport: fixed pxPerSec/scrollLeft, no WaveSurfer. jsdom has no
@@ -211,6 +211,46 @@ describe('SegmentTrack — compositor-safe positioning (#373)', () => {
});
});
describe('SegmentTrack — engine-independent box paint (#963)', () => {
const root = document.documentElement;
afterEach(async () => {
// Restore the default theme and let the palette observer settle so the
// module-level cache can't leak into other tests in this file.
await act(async () => {
root.style.removeProperty('--chrome-bg');
root.removeAttribute('data-theme');
await new Promise((resolve) => setTimeout(resolve, 0));
});
});
it('inline background is a literal opaque rgb() — no color-mix/var() the CSSOM could reject', () => {
// WebView2/Chromium < 111 rejects a color-mix() inline-style assignment
// wholesale, and .seg-track__box declares no fallback background the
// boxes rendered fully transparent (#963). The inline value must be
// plain rgb() so every engine parses it.
setup();
for (const el of screen.getAllByRole('option')) {
expect(el.style.background).toMatch(/^rgb\(\d{1,3}, \d{1,3}, \d{1,3}\)$/);
}
// Default theme, first palette slot: 0.45·rgb(211,134,155) over #0f1011.
expect(box(0).style.background).toBe('rgb(103, 69, 79)');
});
it('boxes re-blend live when the theme changes ([data-theme] on <html>)', async () => {
setup();
expect(box(0).style.background).toBe('rgb(103, 69, 79)');
await act(async () => {
// Same seam App.jsx uses: swap --chrome-bg and flag the theme.
root.style.setProperty('--chrome-bg', '#1e293b');
root.setAttribute('data-theme', 'slate');
await new Promise((resolve) => setTimeout(resolve, 0)); // flush MutationObserver
});
// round(0.45·[211,134,155] + 0.55·[30,41,59])
expect(box(0).style.background).toBe('rgb(111, 83, 102)');
});
});
describe('SegmentTrack — pointer + selection', () => {
it('pointerdown selects the segment (table sync)', () => {
const { onSelectSeg } = setup();
+21 -31
View File
@@ -54,6 +54,7 @@ import {
import { parseScript } from '../utils/parseScript';
import { importToText } from '../utils/importStory';
import { generateSpeech, audioUrl } from '../api/generate';
import { playBlobAudio } from '../utils/media';
import { encodeAudio } from '../api/stories';
import { longformRender } from '../api/audiobook';
import { exportStems } from '../utils/storyExport';
@@ -423,14 +424,6 @@ export default function StoriesEditor({ profiles = [] }) {
return res.blob();
}, []);
const fetchChunkAudio = useCallback(
async (text, profileId, speed = 1.0) => {
const blob = await fetchChunkBlob(text, profileId, speed);
return URL.createObjectURL(blob);
},
[fetchChunkBlob],
);
const previewTrack = useCallback(
async (track) => {
const raw = (track.text || '').trim();
@@ -443,14 +436,18 @@ export default function StoriesEditor({ profiles = [] }) {
if (!hasStoryMarkers(raw)) {
try {
const url = await fetchChunkAudio(raw, pid, spd);
const blob = await fetchChunkBlob(raw, pid, spd);
const url = URL.createObjectURL(blob);
setTracks((prev) =>
prev.map((tk) =>
tk.id === track.id ? { ...tk, audioUrl: url, generating: false } : tk,
),
);
const audio = new Audio(url);
audio.play().catch(() => {});
// Shared playback path (labelled with the line text): registers with
// the single-playback manager + global mini-player, and unlike the
// old bare `new Audio(blobUrl)` actually plays under Tauri's
// WebKit, where blob: URLs are dead in media elements.
playBlobAudio(blob, { label: raw }).catch(() => {});
} catch (err) {
console.warn('Stories preview failed:', err);
setTracks((prev) =>
@@ -462,17 +459,15 @@ export default function StoriesEditor({ profiles = [] }) {
const parsed = parseStoryText(raw, pid);
try {
const audioUrls = await Promise.all(
const chunkBlobs = await Promise.all(
parsed.map((seg) =>
seg.type === 'chunk'
? fetchChunkAudio(seg.text, seg.profileId, spd)
? fetchChunkBlob(seg.text, seg.profileId, spd)
: Promise.resolve(null),
),
);
let cursor = 0;
const finish = () => {
for (let i = cursor; i < audioUrls.length; i++)
if (audioUrls[i]) URL.revokeObjectURL(audioUrls[i]);
setTracks((prev) =>
prev.map((tk) =>
tk.id === track.id ? { ...tk, generating: false, audioUrl: null } : tk,
@@ -482,26 +477,21 @@ export default function StoriesEditor({ profiles = [] }) {
const step = () => {
while (cursor < parsed.length) {
const seg = parsed[cursor];
const url = audioUrls[cursor];
const blob = chunkBlobs[cursor];
cursor++;
if (seg.type === 'pause') {
setTimeout(step, seg.seconds * 1000);
return;
}
if (seg.type === 'chunk' && url) {
const audio = new Audio(url);
audio.onended = () => {
URL.revokeObjectURL(url);
step();
};
audio.onerror = () => {
URL.revokeObjectURL(url);
step();
};
audio.play().catch(() => {
URL.revokeObjectURL(url);
step();
});
if (seg.type === 'chunk' && blob) {
// Chained through the shared playback path: each chunk claims
// the global manager (mini-player shows the line), a natural
// end (or a broken chunk) advances the chain, and stopping from
// the player/another claim cancels the rest of the chain.
playBlobAudio(blob, {
label: raw,
onDone: (reason) => (reason === 'stopped' ? finish() : step()),
}).catch(() => step());
return;
}
}
@@ -515,7 +505,7 @@ export default function StoriesEditor({ profiles = [] }) {
);
}
},
[fetchChunkAudio, cast, globalSpeed, setTracks],
[fetchChunkBlob, cast, globalSpeed, setTracks],
);
// Deliver a stitched WAV in the chosen format. MP3 routes through the backend
+1 -1
View File
@@ -97,7 +97,7 @@ export default function VoicePreview({
if (!open) return null;
return (
<div className="fixed bottom-[calc(var(--logs-footer-height,28px)+16px)] right-[16px] z-[900] w-[320px] bg-[var(--chrome-bg)] border border-solid border-transparent rounded-[12px] [box-shadow:0_8px_32px_rgba(0,0,0,0.4)] flex flex-col overflow-hidden animate-[voice-preview-in_0.2s_ease-out]">
<div className="fixed bottom-[calc(var(--logs-footer-height,28px)+var(--audio-dock-height,0px)+16px)] right-[16px] z-[900] w-[320px] bg-[var(--chrome-bg)] border border-solid border-transparent rounded-[12px] [box-shadow:0_8px_32px_rgba(0,0,0,0.4)] flex flex-col overflow-hidden animate-[voice-preview-in_0.2s_ease-out]">
<div className="flex items-center justify-between py-[10px] px-[14px] border-b border-solid border-b-transparent">
<span className="flex items-center gap-[6px] [font-family:var(--font-mono)] text-[0.72rem] font-semibold uppercase [letter-spacing:0.04em] text-[color:var(--chrome-fg)]">
<Volume2 size={13} /> {t('voicePreview.title')}
+17 -3
View File
@@ -351,7 +351,14 @@ function WaveformTimeline(
console.warn('WebKit audio decode not supported, using media element directly');
try {
const emptyPeaks = new Float32Array(1000).fill(0);
ws.load(undefined, [emptyPeaks], mediaEl.duration || 60);
// Don't rely solely on the 'ready' event firing again for this
// recovery load the play button stayed permanently disabled
// when it didn't (the waveform still rendered from the peaks, so
// there was no visible sign anything was wrong). Confirm
// readiness explicitly once this load settles either way.
Promise.resolve(ws.load(undefined, [emptyPeaks], mediaEl.duration || 60))
.then(() => setReady(true))
.catch(() => setReady(true));
} catch (_) {
setReady(true);
}
@@ -372,7 +379,12 @@ function WaveformTimeline(
})
.then((audioBuffer) => {
const channelData = audioBuffer.getChannelData(0);
ws.load(undefined, [channelData], audioBuffer.duration);
// Same explicit-readiness guard as the NotSupportedError branch
// above don't depend on the 'ready' event re-firing for this
// manually-decoded recovery load.
Promise.resolve(ws.load(undefined, [channelData], audioBuffer.duration))
.then(() => setReady(true))
.catch(() => setReady(true));
})
.catch((decodeErr) => {
// HTTP 404 on the companion audio means the source file is
@@ -391,7 +403,9 @@ function WaveformTimeline(
console.warn('Audio decode fallback failed, loading with empty peaks:', decodeErr);
try {
const emptyPeaks = new Float32Array(1000).fill(0);
ws.load(undefined, [emptyPeaks], mediaEl.duration || 60);
Promise.resolve(ws.load(undefined, [emptyPeaks], mediaEl.duration || 60))
.then(() => setReady(true))
.catch(() => setReady(true));
} catch (_) {
setLoadError(true);
}
@@ -0,0 +1,43 @@
import { describe, it, expect } from 'vitest';
import { readFileSync } from 'node:fs';
import path from 'node:path';
// Regression guard: the dub editor's play button stayed permanently disabled
// (disabled={!ready}) whenever the initial WaveSurfer decode failed and the
// component fell back to a peaks-only ws.load(undefined, [peaks], duration)
// call — the waveform still rendered from those peaks (so nothing looked
// visibly broken), but `ready` was only ever set from the 'ready' event
// re-firing on that recovery load, which this component's own error-handling
// code never actually confirmed. Each fallback load must now explicitly
// confirm readiness once it settles, instead of assuming the event fires.
//
// Driving WaveSurfer + a real decode-failure/recovery sequence through jsdom
// is brittle (see WaveformTimeline.unlock.test.js), so this is a
// source-level contract guard, same house pattern: every `ws.load(undefined,
// ...)` recovery call inside the `ws.on('error', ...)` handler must be
// followed by an explicit setReady(true) confirmation.
const src = readFileSync(
path.resolve(process.cwd(), 'src/components/WaveformTimeline.jsx'),
'utf8',
);
describe('WaveformTimeline error-recovery ready confirmation', () => {
it("confirms readiness explicitly after every fallback ws.load() call, not just via the 'ready' event", () => {
const errorHandler = /ws\.on\('error', \(err\) => \{([\s\S]*?)\n \}\);/.exec(src)?.[1];
expect(errorHandler, "ws.on('error', ...) handler not found").toBeTruthy();
// Every recovery load in this handler passes peaks explicitly
// (`ws.load(undefined, [...], ...)`) — each occurrence must be
// immediately confirmed ready via a .then()/.catch() pair (or an
// unconditional setReady in a synchronous catch), not left to hope the
// 'ready' event re-fires on its own.
const loadCalls = [...errorHandler.matchAll(/ws\.load\(undefined, \[[^\]]*\][^)]*\)/g)];
expect(loadCalls.length).toBeGreaterThanOrEqual(3);
for (const match of loadCalls) {
const tail = errorHandler.slice(match.index, match.index + 220);
expect(tail, `no readiness confirmation after: ${match[0]}`).toMatch(/setReady\(true\)/);
}
});
});
+69 -7
View File
@@ -22,6 +22,8 @@ import {
Lock,
Download as DownloadIcon,
FolderOpen,
Play,
Star,
Trash2,
} from 'lucide-react';
import WaveformPlayer from './WaveformPlayer';
@@ -31,6 +33,7 @@ const FILTERS = [
{ id: 'all', label: 'All' },
{ id: 'clone', label: 'Clone' },
{ id: 'design', label: 'Design' },
{ id: 'starred', label: 'Starred' },
];
/**
@@ -78,15 +81,35 @@ export default function WorkspaceHistory({
handleNativeExport,
restoreHistory,
deleteHistory,
clearHistory, // clear-all for this workspace's history (#1032)
toggleStarHistory, // generation takes: keep this take past the retention cap
playTakeAsOutput, // generation takes: replay a take as the active output
}) {
const { t } = useTranslation();
const [filter, setFilter] = useState('all');
const [expanded, setExpanded] = useState(null); // row id with un-clamped title
// Clear-all affordance (#1032): the old left Sidebar had one; the workspace
// UX overhaul (#374) moved history here and dropped it. Confirm + endpoint
// live in App.jsx (clearWorkspaceHistory) this only renders the button.
const clearAllButton = (count) =>
clearHistory && count > 0 ? (
<button
type="button"
className="history-action-btn danger flex-[0_0_auto]"
onClick={clearHistory}
title={t('sidebar.clear_history')}
>
<Trash2 size={10} /> {t('sidebar.clear_history')}
</button>
) : null;
// Voice workspace = clone + design generations (dub lives in its own workspace).
const items = useMemo(() => {
const synth = history.filter((h) => h.mode === 'clone' || h.mode === 'design');
return filter === 'all' ? synth : synth.filter((h) => h.mode === filter);
if (filter === 'all') return synth;
if (filter === 'starred') return synth.filter((h) => !!h.starred);
return synth.filter((h) => h.mode === filter);
}, [history, filter]);
// Dub variant: a flat list of dub jobs, no clone/design filter.
@@ -94,9 +117,12 @@ export default function WorkspaceHistory({
return (
<aside className="flex-[1_1_0] flex flex-col min-h-0 overflow-hidden">
<div className="flex-[0_0_auto] flex flex-col gap-[8px] py-[10px] px-[12px]">
<span className="inline-flex items-center gap-[6px] [font-family:var(--chrome-font-mono,var(--font-mono))] text-[0.72rem] font-semibold [letter-spacing:0.04em] uppercase text-[color:var(--chrome-fg-muted)]">
<History size={13} /> {t('history.dub_title', { defaultValue: 'Dub history' })}
</span>
<div className="flex items-center justify-between gap-[6px]">
<span className="inline-flex items-center gap-[6px] [font-family:var(--chrome-font-mono,var(--font-mono))] text-[0.72rem] font-semibold [letter-spacing:0.04em] uppercase text-[color:var(--chrome-fg-muted)]">
<History size={13} /> {t('history.dub_title', { defaultValue: 'Dub history' })}
</span>
{clearAllButton(dubHistory.length)}
</div>
</div>
<div className="flex-[1_1_auto] min-h-0 overflow-y-auto flex flex-col gap-[8px] p-[8px]">
{dubHistory.length === 0 ? (
@@ -156,9 +182,12 @@ export default function WorkspaceHistory({
return (
<aside className="flex-[1_1_0] flex flex-col min-h-0 overflow-hidden">
<div className="flex-[0_0_auto] flex flex-col gap-[8px] py-[10px] px-[12px]">
<span className="inline-flex items-center gap-[6px] [font-family:var(--chrome-font-mono,var(--font-mono))] text-[0.72rem] font-semibold [letter-spacing:0.04em] uppercase text-[color:var(--chrome-fg-muted)]">
<History size={13} /> {t('history.title', { defaultValue: 'History' })}
</span>
<div className="flex items-center justify-between gap-[6px]">
<span className="inline-flex items-center gap-[6px] [font-family:var(--chrome-font-mono,var(--font-mono))] text-[0.72rem] font-semibold [letter-spacing:0.04em] uppercase text-[color:var(--chrome-fg-muted)]">
<History size={13} /> {t('history.title', { defaultValue: 'History' })}
</span>
{clearAllButton(history.length)}
</div>
<div className="flex flex-wrap gap-[4px]">
{FILTERS.map((f) => (
<button
@@ -223,6 +252,39 @@ export default function WorkspaceHistory({
) : null}
{item.audio_path ? (
<div className="history-actions">
{toggleStarHistory ? (
<button
className={`history-action-btn history-action-icon ${item.starred ? 'accent' : ''}`}
onClick={(e) => {
e.stopPropagation();
toggleStarHistory(item);
}}
aria-pressed={!!item.starred}
data-testid={`take-star-${item.id}`}
title={
item.starred
? t('history.unstar_take', { defaultValue: 'Unstar — allow cleanup' })
: t('history.star_take', { defaultValue: 'Star — keep this take' })
}
>
<Star size={10} fill={item.starred ? 'currentColor' : 'none'} />
</button>
) : null}
{playTakeAsOutput ? (
<button
className="history-action-btn accent history-action-icon"
onClick={(e) => {
e.stopPropagation();
playTakeAsOutput(item);
}}
data-testid={`take-play-${item.id}`}
title={t('history.play_take', {
defaultValue: 'Load as active output',
})}
>
<Play size={10} />
</button>
) : null}
<button
className="history-action-btn accent"
onClick={(e) => {
@@ -0,0 +1,92 @@
// Generation takes: the Studio history rail's star / load-as-output actions.
import { describe, it, expect, vi, beforeAll } from 'vitest';
import { fireEvent, render, screen } from '@testing-library/react';
import React from 'react';
import WorkspaceHistory from './WorkspaceHistory';
beforeAll(() => {
// LazyWaveform defers the real <WaveformPlayer> behind an IntersectionObserver;
// a no-op stub keeps rows rendered without ever mounting the audio fetch.
global.IntersectionObserver = class {
observe() {}
disconnect() {}
unobserve() {}
};
});
const takes = [
{
id: 'aa1',
mode: 'clone',
text: 'first take',
audio_path: 'aa1.wav',
starred: 0,
created_at: 2,
},
{
id: 'bb2',
mode: 'design',
text: 'second take',
audio_path: 'bb2.wav',
starred: 1,
created_at: 1,
},
];
const noop = () => {};
function renderRail(overrides = {}) {
return render(
<WorkspaceHistory
history={takes}
handleSaveHistoryAsProfile={noop}
handleLockProfile={noop}
handleNativeExport={noop}
restoreHistory={noop}
deleteHistory={noop}
toggleStarHistory={noop}
playTakeAsOutput={noop}
{...overrides}
/>,
);
}
describe('WorkspaceHistory takes actions', () => {
it('star button reflects the starred state and calls the handler', () => {
const toggleStarHistory = vi.fn();
renderRail({ toggleStarHistory });
const unstarred = screen.getByTestId('take-star-aa1');
const starred = screen.getByTestId('take-star-bb2');
expect(unstarred).toHaveAttribute('aria-pressed', 'false');
expect(starred).toHaveAttribute('aria-pressed', 'true');
fireEvent.click(unstarred);
expect(toggleStarHistory).toHaveBeenCalledTimes(1);
expect(toggleStarHistory.mock.calls[0][0].id).toBe('aa1');
});
it('load-as-output button hands the take to the player handler', () => {
const playTakeAsOutput = vi.fn();
renderRail({ playTakeAsOutput });
fireEvent.click(screen.getByTestId('take-play-bb2'));
expect(playTakeAsOutput).toHaveBeenCalledTimes(1);
expect(playTakeAsOutput.mock.calls[0][0].id).toBe('bb2');
});
it('starred filter narrows the rail to starred takes only', () => {
renderRail();
fireEvent.click(screen.getByRole('button', { name: 'Starred' }));
expect(screen.queryByTestId('take-star-aa1')).toBeNull();
expect(screen.getByTestId('take-star-bb2')).toBeInTheDocument();
});
it('omits the takes actions when no handlers are passed (dub rail safety)', () => {
renderRail({ toggleStarHistory: undefined, playTakeAsOutput: undefined });
expect(screen.queryByTestId('take-star-aa1')).toBeNull();
expect(screen.queryByTestId('take-play-aa1')).toBeNull();
});
});
@@ -147,6 +147,13 @@ export default function DesignMethodPanel({
{Object.entries(CATEGORIES).map(([key, options]) => {
const many = options.length > 6;
const optLabel = (val) => {
// #983: a profile/localStorage-restored vdStates can carry a
// partial shape (missing category keys) val is then undefined
// here even though the 'Auto' check above only catches the
// literal sentinel. Guard before .replace() rather than crash;
// 'Auto' matches how the rest of the component (the ternary
// above, the chip/select fallbacks) treats an unset category.
if (typeof val !== 'string' || !val) return 'Auto';
const tKey = `clone.opt_${val.replace(/[ -]/g, '_')}`;
const tl = t(tKey);
return tl !== tKey ? tl : val;
@@ -180,9 +187,13 @@ export default function DesignMethodPanel({
aria-label={t(`clone.cat_${key}`)}
>
{options.map((opt, i) => {
const optTKey = `clone.opt_${opt.replace(/[ -]/g, '_')}`;
// `opt` is always a hardcoded CATEGORIES string today,
// never undefined guarded anyway for consistency with
// the identical pattern above (#983).
const safeOpt = typeof opt === 'string' && opt ? opt : 'Auto';
const optTKey = `clone.opt_${safeOpt.replace(/[ -]/g, '_')}`;
const optTl = t(optTKey);
const optLabel = optTl !== optTKey ? optTl : opt;
const optLabel = optTl !== optTKey ? optTl : safeOpt;
const checked = vdStates[key] === opt;
// Roving tabindex: the checked chip is the group's
// single tab stop (first chip if nothing matches).
@@ -0,0 +1,68 @@
import React from 'react';
import { describe, it, expect, vi } from 'vitest';
import { render, screen } from '@testing-library/react';
import DesignMethodPanel from './DesignMethodPanel';
// #983: "Cannot read properties of undefined (reading 'replace')" the
// identity panel crashed whenever vdStates was missing one of the 6
// CATEGORIES keys (a design profile saved by an older/foreign client, or a
// stale localStorage shape). The label helper called `val.replace(...)` on
// an undefined category value. This regression-tests the render guard added
// to DesignMethodPanel.jsx directly, independent of the upstream data-shape
// fixes in useProfiles.js / useAppData.js / profiles.py.
// A minimal i18next-compatible mock: returns the defaultValue if given, else
// echoes the key back (mirrors i18next's behavior for a missing translation,
// which is what `optLabel`'s `tl !== tKey` check relies on).
const t = (key, opts) => opts?.defaultValue ?? key;
function setup(vdStates, props = {}) {
return render(
<DesignMethodPanel
t={t}
describeText=""
onDescribeChange={vi.fn()}
describeMatchedAny={false}
describeUnmatched={[]}
chipPersonalities={[]}
activePersonality={null}
applyPersonality={vi.fn()}
applyPreset={vi.fn()}
identityOpen={true}
setIdentityOpen={vi.fn()}
identityRecipe="test recipe"
vdStates={vdStates}
setVdStates={vi.fn()}
onChipKeyDown={vi.fn()}
showSaveProfile={false}
setShowSaveProfile={vi.fn()}
profileName=""
setProfileName={vi.fn()}
handleSaveDesignProfile={vi.fn()}
instruct=""
language="Auto"
{...props}
/>,
);
}
describe('DesignMethodPanel — #983 partial vdStates crash', () => {
it('does not throw when vdStates is missing 5 of the 6 CATEGORIES keys', () => {
// Only Gender is set Age, Pitch, Style, EnglishAccent, ChineseDialect
// are all undefined, exercising both the chip-based and <select>-based
// ("many" options) render paths.
expect(() => setup({ Gender: 'male' })).not.toThrow();
});
it('does not throw when vdStates is a fully empty object', () => {
expect(() => setup({})).not.toThrow();
});
it('still renders category labels and the identity recipe with a partial shape', () => {
const { container } = setup({ Gender: 'male' });
expect(screen.getByText('test recipe')).toBeInTheDocument();
// The label text sits alongside a sibling <span> kicker, so assert via
// textContent rather than getByText's exact-node matching.
expect(container.textContent).toContain('clone.cat_Gender');
});
});
@@ -34,8 +34,13 @@ export default function ScriptPanel({
}) {
return (
<div className="flex flex-col gap-[6px] flex-none min-h-0 relative z-[2]">
{/* overflow-visible: the Insert popover opens above the textarea and
must escape the panel's box instead of being clipped (#481). */}
{/* overflow-visible: the Insert popover opens BELOW the textarea and
must escape the panel's box instead of being clipped (#481). It
used to open upward but the script input sits at the very top of
the clone modal, so the tag list (max-h 280px) climbed straight out
of the viewport with no way to see or scroll it (owner-reported,
screenshot showed the CMU chips clipped). Below always has room
here: the panel is the topmost element in every mount. */}
<div className={`${STUDIO_PANEL} relative z-[10] overflow-visible`}>
<div className="label-row">
<Command className="label-icon" size={14} />{' '}
@@ -100,7 +105,7 @@ export default function ScriptPanel({
)}
{insertOpen && (
<div
className="absolute right-[8px] bottom-[60px] z-20 flex flex-wrap gap-1 max-w-[min(360px,calc(100vw-16px))] max-h-[min(280px,calc(100vh-120px))] overflow-y-auto overscroll-contain p-2 bg-[var(--chrome-bg)] border border-transparent rounded-[10px] shadow-[0_8px_24px_rgba(0,0,0,0.45)]"
className="absolute right-[8px] top-[calc(100%+6px)] z-20 flex flex-wrap gap-1 max-w-[min(360px,calc(100vw-16px))] max-h-[min(280px,calc(100vh-120px))] overflow-y-auto overscroll-contain p-2 bg-[var(--chrome-bg)] border border-transparent rounded-[10px] shadow-[0_8px_24px_rgba(0,0,0,0.45)]"
role="menu"
>
{TAGS.map((tag) => (
+109 -101
View File
@@ -25,6 +25,7 @@ export default function DubHeader({
handleDubStop,
dubProgress,
onGenerateClick,
isTranslating,
multiLangMode,
multiLangs,
incrementalPlan,
@@ -34,121 +35,128 @@ export default function DubHeader({
setExportOpen,
}) {
return (
<div className="flex flex-wrap justify-between items-center gap-x-[var(--space-2)] gap-y-[4px] min-w-0 px-[10px] py-[4px] shrink-0 bg-[var(--color-bg-elev-1)] rounded-md mb-[2px]">
{/* Pipeline spine, inlined onto the header row (Upload → … → Export). */}
<DubPipelineStepper dubStep={dubStep} inline />
<div className="label-row dub-head__title !gap-[6px]">
<FileText className="label-icon" size={11} />
<span className="font-medium text-[0.78rem] min-w-0 overflow-hidden text-ellipsis whitespace-nowrap text-fg normal-case">
{dubFilename}
</span>
<span className="text-fg-muted font-normal whitespace-nowrap text-[0.68rem] normal-case shrink-0">
· {formatTime(dubDuration)} · {dubSegments.length} {t('dub.segs')}
</span>
{activeProjectName && activeProjectName !== dubFilename && (
<span className="text-[#b8bb26] ml-[var(--space-2)] whitespace-nowrap text-[0.68rem] normal-case overflow-hidden text-ellipsis min-w-0">
{activeProjectName}
<div className="flex flex-col gap-[2px] min-w-0 px-[10px] py-[4px] shrink-0 bg-[var(--color-bg-elev-1)] rounded-md mb-[2px]">
{/* Row 1: project title (left) + actions (right). Row 2: the pipeline
spine (Upload Export) sits directly under the title with a
tight 2px gap title-first, owner-requested order. */}
<div className="flex flex-wrap justify-between items-center gap-x-[var(--space-2)] gap-y-[4px] min-w-0">
<div className="label-row dub-head__title !gap-[6px]">
<FileText className="label-icon" size={11} />
<span className="font-medium text-[0.78rem] min-w-0 overflow-hidden text-ellipsis whitespace-nowrap text-fg normal-case">
{dubFilename}
</span>
)}
</div>
<div className="flex gap-[6px] items-center shrink-0">
{/* Icon-only secondary actions (tooltips carry the labels);
<span className="text-fg-muted font-normal whitespace-nowrap text-[0.68rem] normal-case shrink-0">
· {formatTime(dubDuration)} · {dubSegments.length} {t('dub.segs')}
</span>
{activeProjectName && activeProjectName !== dubFilename && (
<span className="text-[#b8bb26] ml-[var(--space-2)] whitespace-nowrap text-[0.68rem] normal-case overflow-hidden text-ellipsis min-w-0">
{activeProjectName}
</span>
)}
</div>
<div className="flex gap-[6px] items-center shrink-0">
{/* Icon-only secondary actions (tooltips carry the labels);
Generate Dub keeps its label as the primary verb. */}
<Button
variant="subtle"
size="sm"
onClick={saveProject}
title={t('dub.save')}
aria-label={t('dub.save')}
>
<Save size={12} />
</Button>
<Button
variant="danger"
size="sm"
onClick={resetDub}
title={t('dub.reset')}
aria-label={t('dub.reset')}
>
<RotateCcw size={12} />
</Button>
{/* Primary actions live on the header bar (compact) — moved up from the footer. */}
<div className="flex gap-[6px] items-center pl-[var(--space-2)] ml-[2px]">
{dubStep === 'stopping' ? (
<FooterBtn
sm
tone="stopping"
disabled
icon={<Loader className="spinner" size={9} />}
label={t('dub.stopping')}
/>
) : dubStep === 'generating' ? (
<FooterBtn
sm
tone="danger"
onClick={handleDubStop}
icon={<Square size={9} />}
label={t('dub.stop_progress', {
current: dubProgress.current,
total: dubProgress.total,
})}
/>
) : (
<>
<Button
variant="subtle"
size="sm"
onClick={saveProject}
title={t('dub.save')}
aria-label={t('dub.save')}
>
<Save size={12} />
</Button>
<Button
variant="danger"
size="sm"
onClick={resetDub}
title={t('dub.reset')}
aria-label={t('dub.reset')}
>
<RotateCcw size={12} />
</Button>
{/* Primary actions live on the header bar (compact) — moved up from the footer. */}
<div className="flex gap-[6px] items-center pl-[var(--space-2)] ml-[2px]">
{dubStep === 'stopping' ? (
<FooterBtn
sm
tone={dubSegments.length ? 'pink' : 'idle'}
onClick={onGenerateClick}
disabled={!dubSegments.length}
icon={<Play size={11} />}
label={
multiLangMode && multiLangs.length > 1
? t('dub.generate_dub_multi', {
count: multiLangs.length,
defaultValue: 'Generate {{count}} dubs',
})
: t('dub.generate_dub')
}
tone="stopping"
disabled
icon={<Loader className="spinner" size={9} />}
label={t('dub.stopping')}
/>
{dubStep === 'done' && incrementalPlan && incrementalPlan.stale?.length > 0 && (
) : dubStep === 'generating' ? (
<FooterBtn
sm
tone="danger"
onClick={handleDubStop}
icon={<Square size={9} />}
label={t('dub.stop_progress', {
current: dubProgress.current,
total: dubProgress.total,
})}
/>
) : (
<>
<FooterBtn
sm
tone="pink"
onClick={() =>
handleDubGenerate({ regenOnly: incrementalPlan.stale, preview: true })
}
tone={dubSegments.length && !isTranslating ? 'pink' : 'idle'}
onClick={onGenerateClick}
// The multi-language batch translates between generates while
// dubStep briefly sits back at 'editing' keep the CTA inert
// during that phase so a re-click can't start a second batch.
disabled={!dubSegments.length || isTranslating}
icon={<Play size={11} />}
label={t('dub.regen_changed', { count: incrementalPlan.stale.length })}
label={
multiLangMode && multiLangs.length > 1
? t('dub.generate_dub_multi', {
count: multiLangs.length,
defaultValue: 'Generate {{count}} dubs',
})
: t('dub.generate_dub')
}
/>
)}
</>
)}
{dubStep === 'done' && (
{dubStep === 'done' && incrementalPlan && incrementalPlan.stale?.length > 0 && (
<FooterBtn
sm
tone="pink"
onClick={() =>
handleDubGenerate({ regenOnly: incrementalPlan.stale, preview: true })
}
icon={<Play size={11} />}
label={t('dub.regen_changed', { count: incrementalPlan.stale.length })}
/>
)}
</>
)}
{dubStep === 'done' && (
<FooterBtn
sm
tone="idle"
disabled={qcRunning || !dubSegments.length}
onClick={handleDubQc}
icon={
qcRunning ? <Loader className="spinner" size={11} /> : <ShieldCheck size={11} />
}
title={t('dub.qc_btn', { defaultValue: 'Verify dub timing (second-pass check)' })}
aria-label={t('dub.qc_btn', {
defaultValue: 'Verify dub timing (second-pass check)',
})}
/>
)}
<FooterBtn
sm
tone="idle"
disabled={qcRunning || !dubSegments.length}
onClick={handleDubQc}
icon={
qcRunning ? <Loader className="spinner" size={11} /> : <ShieldCheck size={11} />
}
title={t('dub.qc_btn', { defaultValue: 'Verify dub timing (second-pass check)' })}
aria-label={t('dub.qc_btn', {
defaultValue: 'Verify dub timing (second-pass check)',
})}
tone={dubStep === 'done' ? 'green' : 'idle'}
disabled={dubStep !== 'done' && !dubSegments.length}
onClick={() => setExportOpen(true)}
icon={<Download size={12} />}
title={t('dub.export_btn')}
aria-label={t('dub.export_btn')}
/>
)}
<FooterBtn
sm
tone={dubStep === 'done' ? 'green' : 'idle'}
disabled={dubStep !== 'done' && !dubSegments.length}
onClick={() => setExportOpen(true)}
icon={<Download size={12} />}
title={t('dub.export_btn')}
aria-label={t('dub.export_btn')}
/>
</div>
</div>
</div>
<DubPipelineStepper dubStep={dubStep} inline />
</div>
);
}
@@ -18,6 +18,7 @@ import { useAppStore } from '../../store';
import WaveformTimeline from '../WaveformTimeline';
import MultiLangPicker from '../MultiLangPicker';
import { API } from '../../api/client';
import { dubListTracks } from '../../api/dub';
import { LANG_CODES } from '../../utils/languages';
import ALL_LANGUAGES from '../../languages.json';
import { POPULAR_LANGS, PRESETS } from '../../utils/constants';
@@ -108,6 +109,16 @@ export default function DubLeftColumn({
// configured, we route the user straight to the LLM Providers setup instead
// of dead-ending on a toast (#838).
const openSettingsTab = useAppStore((s) => s.openSettingsTab);
// Two-stage LLM translation quality only meaningful (and only rendered)
// when the LLM engine is the active translator. Persisted prefs.
const autoGlossary = useAppStore((s) => s.autoGlossary);
const setAutoGlossary = useAppStore((s) => s.setAutoGlossary);
const reflectPass = useAppStore((s) => s.reflectPass);
const setReflectPass = useAppStore((s) => s.setReflectPass);
// Opt-in LLM condensation suggestions for segments the duration planner
// classifies as impossible to fit (default OFF needs an LLM).
const condenseSuggest = useAppStore((s) => s.condenseSuggest);
const setCondenseSuggest = useAppStore((s) => s.setCondenseSuggest);
// Frozen-build (packaged/signed, read-only site-packages) escape-hatch
// popover: pip install is impossible, so we surface the copyable command +
// a one-click switch to the always-bundled Argos engine + a docs deeplink.
@@ -143,6 +154,49 @@ export default function DubLeftColumn({
else toast.error(t('dub.copy_failed'));
};
// Per-track metadata (duration + timing strategy) for the pill tooltips.
// The store only carries the track codes, so hydrate lazily from the
// existing GET /dub/tracks/{job_id} once the editor shows tracks (re-runs
// when a new language finishes and dubTracks changes). Failure-silent:
// the pills render fine without tooltips.
const [trackInfo, setTrackInfo] = useState({});
useEffect(() => {
if (!hasDubbedTrack || !dubJobId) return undefined;
let cancelled = false;
dubListTracks(dubJobId)
.then((tracks) => {
if (!cancelled) setTrackInfo(tracks || {});
})
.catch(() => {
/* tooltip enrichment only — never block or toast */
});
return () => {
cancelled = true;
};
}, [hasDubbedTrack, dubJobId, dubTracks]);
const trackTooltip = (code) => {
const info = trackInfo[code];
if (!info) return undefined;
const parts = [];
if (Number.isFinite(info.duration) && info.duration > 0) {
parts.push(
t('dub.track_tip_duration', {
duration: fmtDur(Math.round(info.duration)),
defaultValue: 'Duration {{duration}}',
}),
);
}
if (info.timing_strategy) {
// Reuse the timing-strategy display names where they exist
// (dub.timing_<id>); unknown/future strategies fall back to the raw id.
const strategy = t(`dub.timing_${info.timing_strategy}`, {
defaultValue: info.timing_strategy,
});
parts.push(t('dub.track_tip_timing', { strategy, defaultValue: 'Timing {{strategy}}' }));
}
return parts.length ? parts.join(' · ') : undefined;
};
return (
<div className="studio-panel dub-panel-col">
{hasDubbedTrack && (
@@ -170,6 +224,7 @@ export default function DubLeftColumn({
aria-checked={previewMode === code}
className={`dub-lang-pill ${previewMode === code ? 'is-active' : ''}`}
onClick={() => setPreviewMode(code)}
title={trackTooltip(code)}
>
{label}
</button>
@@ -601,7 +656,54 @@ export default function DubLeftColumn({
{ value: 'cinematic', label: t('dub.cinematic_quality') },
]}
/>
{/* Opt-in (default OFF): when the duration planner marks a
translated line "impossible" for its slot, ask the LLM for a
shorter rewrite the user can apply per segment. */}
<label
className="flex items-center gap-[4px] mt-[3px] text-[0.55rem] text-fg-muted cursor-pointer select-none"
title={t('dub.condense_title')}
>
<input
type="checkbox"
checked={condenseSuggest}
onChange={(e) => setCondenseSuggest(e.target.checked)}
className="cursor-pointer"
/>
{t('dub.condense_label')}
</label>
</div>
{/* LLM engine only: auto-glossary + reflect pass. Both default ON;
the reflect tooltip is explicit that it multiplies LLM calls. */}
{translateProvider === 'openai' && (
<div
className={`${FIELD} flex-[0_0_auto] ${FIELD_RESP} justify-end gap-[2px] pb-[2px]`}
>
<label
className="flex items-center gap-[4px] text-[0.6rem] text-[var(--chrome-fg-muted)] cursor-pointer whitespace-nowrap"
title={t('dub.auto_glossary_title')}
>
<input
type="checkbox"
className="accent-[var(--color-brand)] cursor-pointer"
checked={autoGlossary}
onChange={(e) => setAutoGlossary(e.target.checked)}
/>
<span>{t('dub.auto_glossary_label')}</span>
</label>
<label
className="flex items-center gap-[4px] text-[0.6rem] text-[var(--chrome-fg-muted)] cursor-pointer whitespace-nowrap"
title={t('dub.reflect_title')}
>
<input
type="checkbox"
className="accent-[var(--color-brand)] cursor-pointer"
checked={reflectPass}
onChange={(e) => setReflectPass(e.target.checked)}
/>
<span>{t('dub.reflect_label')}</span>
</label>
</div>
)}
<div className={`${FIELD} flex-[1_1_90px] min-w-[64px] ${FIELD_RESP}`}>
<div className={FIELD_LABEL}>
<UserSquare2 className="label-icon" size={9} /> {t('dub.style')}{' '}
@@ -75,7 +75,7 @@ export default function CommunityZone({
onToggleFavorite={toggleFavorite}
onPreview={(item) =>
item.audio?.url
? onPlayAudio(item.audio.url, item.id)
? onPlayAudio(item.audio.url, item.id, item.name)
: flash(
t('gallery.no_preview', {
defaultValue: 'No preview — add it with "Use voice" to hear it.',
@@ -91,8 +91,13 @@ export default function CommunityZone({
name: r.name,
}),
);
} catch {
flash(t('gallery.use_failed', { defaultValue: 'Could not add that voice.' }));
} catch (e) {
flash(
t('gallery.use_failed', {
message: e?.message || String(e),
defaultValue: 'Could not create that voice: {{message}}',
}),
);
}
}}
onDesign={(item) =>
@@ -110,7 +110,12 @@ export default function ImportsZone({ t, playingId, loadingPreviewId, onPlayGall
const r = await searchYoutube(q, 'import', 10);
setResults(r.results || []);
} catch (e) {
flash(t('gallery.search_failed', { defaultValue: 'Search failed.' }));
flash(
t('gallery.search_failed', {
message: e?.message || String(e),
defaultValue: 'Search failed: {{message}}',
}),
);
} finally {
setIsSearching(false);
}
@@ -149,7 +154,12 @@ export default function ImportsZone({ t, playingId, loadingPreviewId, onPlayGall
await uploadVoiceClip(fd);
reload();
} catch (err) {
flash(t('gallery.upload_failed', { defaultValue: 'Upload failed.' }));
flash(
t('gallery.upload_failed', {
message: err?.message || String(err),
defaultValue: 'Upload failed: {{message}}',
}),
);
} finally {
if (fileRef.current) fileRef.current.value = '';
}
@@ -165,7 +175,12 @@ export default function ImportsZone({ t, playingId, loadingPreviewId, onPlayGall
}),
);
} catch (e) {
flash(t('gallery.save_failed', { defaultValue: 'Could not save profile.' }));
flash(
t('gallery.save_failed', {
message: e?.message || String(e),
defaultValue: 'Could not save profile: {{message}}',
}),
);
}
};
@@ -179,8 +194,13 @@ export default function ImportsZone({ t, playingId, loadingPreviewId, onPlayGall
try {
await deleteGalleryVoice(v.id);
reload();
} catch {
/* noop */
} catch (e) {
flash(
t('gallery.delete_failed', {
message: e?.message || String(e),
defaultValue: 'Could not delete: {{message}}',
}),
);
}
};
@@ -191,7 +211,12 @@ export default function ImportsZone({ t, playingId, loadingPreviewId, onPlayGall
const file = new File([blob], `${v.name}.wav`, { type: 'audio/wav' });
setTrimming({ voice: v, file });
} catch (e) {
flash(t('gallery.trim_load_failed', { defaultValue: 'Could not load audio for trimming.' }));
flash(
t('gallery.trim_load_failed', {
message: e?.message || String(e),
defaultValue: 'Could not load audio for trimming: {{message}}',
}),
);
}
};
@@ -209,7 +234,12 @@ export default function ImportsZone({ t, playingId, loadingPreviewId, onPlayGall
reload();
setTrimming(null);
} catch (e) {
flash(t('gallery.upload_failed', { defaultValue: 'Upload failed.' }));
flash(
t('gallery.upload_failed', {
message: e?.message || String(e),
defaultValue: 'Upload failed: {{message}}',
}),
);
}
};
@@ -0,0 +1,152 @@
/**
* Settings Models tab OpenAI-compatible remote ASR panel (#877).
*
* A path to Qwen3-ASR, a self-hosted FunASR/SenseVoice server, or OpenAI's
* own Whisper API today, without waiting on transformers to ship a direct
* Qwen3-ASR integration. Configures the `openai-compat-asr` backend's
* base_url/model/api_key; activating it as the active ASR engine still needs
* `OMNIVOICE_ASR_BACKEND=openai-compat-asr` (no in-app ASR engine picker
* exists yet for any ASR backend this panel only configures this one).
*
* Endpoints (loopback-only):
* GET /api/settings/asr-openai-compat {base_url, model, has_key}
* PUT /api/settings/asr-openai-compat body {base_url?, model?, api_key?}
* ('' clears api_key; omitted/null leaves it unchanged never returned)
*/
import React, { useCallback, useEffect, useState } from 'react';
import { Mic } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { apiJson, apiFetch } from '../../api/client';
import { SettingsSection, SettingRow, SettingsInput } from './primitives';
import { Button } from '../../ui';
export default function AsrOpenAICompatPanel() {
const { t } = useTranslation();
const [baseUrl, setBaseUrl] = useState('');
const [model, setModel] = useState('');
const [apiKey, setApiKey] = useState('');
const [hasKey, setHasKey] = useState(false);
const [saving, setSaving] = useState(false);
const [error, setError] = useState(null);
const refresh = useCallback(async () => {
setError(null);
try {
const d = await apiJson('/api/settings/asr-openai-compat');
setBaseUrl(d?.base_url || '');
setModel(d?.model || '');
setHasKey(Boolean(d?.has_key));
setApiKey(''); // the key is never returned the field always starts blank
} catch (e) {
setError(e?.message || t('models.asrOpenAICompatLoadError'));
}
}, [t]);
useEffect(() => {
refresh();
}, [refresh]);
const save = async () => {
setSaving(true);
setError(null);
try {
const res = await apiFetch('/api/settings/asr-openai-compat', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
base_url: baseUrl,
model,
// Only send api_key when the user actually typed something
// an untouched field must leave the stored key unchanged, not
// clear it (the field is always blank on load, so "unchanged"
// and "empty" would otherwise be indistinguishable).
...(apiKey ? { api_key: apiKey } : {}),
}),
});
const d = await res.json();
setBaseUrl(d.base_url || '');
setModel(d.model || '');
setHasKey(Boolean(d.has_key));
setApiKey('');
} catch (e) {
setError(e?.message || t('models.asrOpenAICompatSaveError'));
} finally {
setSaving(false);
}
};
return (
<SettingsSection
icon={Mic}
title={t('models.asrOpenAICompatTitle')}
description={t('models.asrOpenAICompatDescription')}
>
{error && (
<div className="perfpanel__error" role="alert">
{error}
</div>
)}
<SettingRow
stack
title={t('models.asrOpenAICompatBaseUrlTitle')}
hint={t('models.asrOpenAICompatBaseUrlHint')}
control={
<SettingsInput
mono
type="text"
value={baseUrl}
onChange={(e) => setBaseUrl(e.target.value)}
placeholder="http://localhost:8000/v1"
data-testid="asr-openai-compat-base-url"
/>
}
/>
<SettingRow
stack
title={t('models.asrOpenAICompatModelTitle')}
control={
<SettingsInput
mono
type="text"
value={model}
onChange={(e) => setModel(e.target.value)}
placeholder="whisper-1"
data-testid="asr-openai-compat-model"
/>
}
/>
<SettingRow
stack
title={t('models.asrOpenAICompatApiKeyTitle')}
hint={
hasKey ? t('models.asrOpenAICompatKeyConfigured') : t('models.asrOpenAICompatApiKeyHint')
}
control={
<>
<SettingsInput
mono
type="password"
value={apiKey}
onChange={(e) => setApiKey(e.target.value)}
placeholder={hasKey ? '••••••••' : t('models.asrOpenAICompatApiKeyOptional')}
data-testid="asr-openai-compat-api-key"
/>
<Button
variant="subtle"
size="sm"
onClick={save}
loading={saving}
disabled={saving}
data-testid="asr-openai-compat-save"
>
{t('common.save')}
</Button>
</>
}
/>
</SettingsSection>
);
}

Some files were not shown because too many files have changed in this diff Show More