Compare commits

...
10 Commits
Author SHA1 Message Date
29b6f30f2b release: freeze v0.3.15 — version bump, lockfiles, changelog (#1041)
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 04:59:59 +05:30
72e979f1d9 fix(tts): model-load time stops eating the generate timeout budget (#1039)
* fix(tts): model-load time stops eating the generate timeout budget (#1033, #1037)

The generate guard (OMNIVOICE_GENERATE_TIMEOUT_S, 300s) wrapped the
adapter's lazy _ensure_loaded() — weight download included — together
with the synthesis. A cold first request burned the whole window on
the download and died with the VRAM-guidance 503; #1014's T4
verification measured it (0% GPU util for the full 300s), and #1033 +
#1037 match the signature.

New public TTSBackend.ensure_ready() (dispatches to the adapter's
_ensure_loaded when present) runs FIRST under the model-load budget
(OMNIVOICE_MODEL_LOAD_TIMEOUT, 1200s) in both /generate's adapter path
and /v1/audio/speech — the same load/generate split get_model()
already gave the native engine. Warm engines no-op. A load exceeding
its own budget 503s with load-specific text pointing at Settings →
Models, never the misleading 'too heavy for compute' guidance.

Tests: end-to-end class test (load slower than a tiny generate budget
but inside the load budget → succeeds; fail-before verified), the
stalled-load error path, and the base-hook dispatch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* changelog entry for the load-budget split (#1033, #1037)

* catch the builtin TimeoutError base — reload-proof class identity (CI-only miss)

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 04:35:17 +05:30
6c6207a132 docs: link the community Colab notebook (#1038) (#1040)
@shakib30 built and tested a working Colab notebook for the project
and offered it upstream. Linking it from the README (community-
maintained, credited) makes the no-local-GPU path discoverable without
taking on notebook maintenance in-repo.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 04:19:54 +05:30
0effe485f4 fix(studio): restore Clear History, stoppable auto-play preview, + cached ref transcripts (#1032) (#1036)
Three-part fix for the v0.3.5-comparison report:

1. Perf: since v0.3.6 (#308), a clone reference without a stored
   transcript triggered a FULL ASR model load + transcribe on every
   /generate — get_active_asr_backend() builds a fresh whisper backend
   per call. Measured live: 92.7s wall vs 14.9s of actual TTS. Now the
   first auto-transcript is persisted onto the (unlocked, clone-kind)
   profile row, and transcribe_reference caches results by audio
   content hash (bounded LRU, no model/VRAM held), so the cost is paid
   once per clip, not per request. User-typed transcripts are never
   overwritten; locked/design profiles are excluded from the persist.

2. Clear History: the workspace UX overhaul (#374) moved history into
   the right-side WorkspaceHistory panels and dropped the old Sidebar's
   clear-all control (the Sidebar is now hidden in every mode). Both
   the Voice and Dub panels get a scoped Clear History button wired to
   the existing DELETE /history and /dub/history endpoints, with the
   same confirm dialog the Sidebar used.

3. Auto-play: the finished-render playback (playBlobAudio) has no
   on-screen player and the only stop lived in the Voice ActionBar's
   CTA morph — unstoppable from the Dub workspace, profile pages, or
   after navigating away. A global PlaybackStopPill now appears for any
   'output' playback on every page. The existing Settings → Appearance
   "Auto-play preview" pref (#667) now also gates the generate path,
   as its label always promised (default ON — no behavior change).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 18:43:33 +05:30
32407bc781 fix(api): /v1/audio/speech honors num_step + guidance_scale instead of silently dropping them (#1014) (#1035)
A contributor's measured Tesla T4 verification (PR #1014) caught that
POST /v1/audio/speech accepted num_step/guidance_scale in the JSON
body with a 200 OK and discarded both (pydantic's default
extra=ignore) — API callers could never reach the model's documented
quality preset (num_step=32) through the OpenAI-compatible surface,
while the native /generate exposes both as form fields.

Both are now declared as validated optional extensions (num_step 1-128,
guidance_scale 0-20) and passed through to the engine's generate()
kwargs — omitted means absent (engines that don't accept the kwargs
never see a stray None), exactly like the existing duration/seed
extensions.

Tests: passthrough reaches the engine kwargs; omitted stays absent.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 17:49:25 +05:30
df870e9ed3 docs(agents): add verified Tesla T4 (16GB) inference notes (#1014)
* docs: add AGENTS.md with verified Tesla T4 (16GB) inference notes

Documents two things found while verifying inference on a real T4:
1. Cold-cache first /v1/audio/speech call can hit the 300s
   OMNIVOICE_GENERATE_TIMEOUT_S because the checkpoint download happens
   inside that budget — workaround via existing POST /models/install or
   raising the timeout, no code change needed.
2. The OpenAI-compatible endpoint silently ignores num_step/guidance_scale
   (schema doesn't declare them) — use native /generate for those.

Also documents the T4 acceleration checklist (dtype/attention/int8/CUDA
graphs) and measured VRAM (peak 2.05GB). No code changes.

* fix(docs): make /models/install workaround command actually executable

Addresses Greptile review: the instruction omitted the required
repo_id body field (InstallModelRequest rejects an empty body).

* fix(docs): correct port in /models/install example (3900, not 8000)

The app serves on port 3900 (confirmed: /health returns 200 there,
connection refused on 8000). Verified the exact corrected curl command
returns 200 {"status":"install_started",...}.

* move T4 notes to docs/hardware-notes-tesla-t4.md — AGENTS.md is the auto-loaded agent-instructions filename

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 17:30:05 +05:30
stronghamjjiandstronghamjji 323892b36c fix(clone): bound the ref-text re-transcribe like every other ASR dispatch (#730) (#1031)
The re-transcribe added for the (ref_audio, ref_text) mismatch fix is
dispatched with a bare run_in_executor(_gpu_pool, ...), and refine_ref_text
calls asr_backend.transcribe() directly. Its try/except catches a raised
error but not a *hang* — a wedged whisperx/CTranslate2 transcribe (#730)
holds the GPU-pool worker forever. On a <=10 GB card the pool is 1 worker,
so that starves every later GPU job into the misleading "can't reach the
local backend", and there's no ping on the await so the EventSource drops.

Route both refine dispatches (per-speaker and per-segment) through the same
run_transcribe_guarded the rest of dub_core.py already uses (the chunk loop
and the whole-file "Dub" transcribe). On timeout it resets the pool and
raises ASRTimeoutError; keep the original clones, matching refine_ref_text's
own "failure is a strict no-op" fallback.

Adds a repro test: refine_ref_texts dispatched raw is unbounded on a hang;
through the guard it times out and falls back to the original ref_text.

Co-authored-by: stronghamjji <289942360+stronghamjji@users.noreply.github.com>
2026-07-09 17:10:23 +05:30
36ee06c7cc feat(skills): installable Agent Skills — npx skills add debpalash/omnivoice-studio (#1034)
Two skills in the standard skills/<name>/SKILL.md layout (vercel-labs/
skills CLI; listed on skills.sh via install telemetry):

- omnivoice — teaches any agent (Claude Code, Cursor, Codex, …) to
  speak and transcribe through the user's LOCAL install via the
  OpenAI-compatible API at localhost:3900: health preflight, TTS with
  cloned-voice-profile discovery via /v1/audio/voices, STT with
  srt/vtt subtitle formats, and the local-first rule (never silently
  fall back to a cloud API).
- oss-maintainer — the maintainer methodology this repo is actually
  run with, distilled from real sessions: absorbed-or-declined queue
  discipline, check-the-PR-queue-before-implementing, root-cause →
  fix-the-class → regression-test, structural merge gates with
  flaky-vs-real judgment, the release protocol, and
  thank-contributors-specifically.

Every endpoint/flag in the omnivoice skill verified against
backend/api/routers/openai_compat.py and the README's API section.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 17:06:14 +05:30
9bfd8f9a13 fix(update): app updates stop uninstalling user-added engines — drift sync goes --inexact (#1029) (#1030)
Every app update whose uv.lock changed ran `uv sync --frozen` to
reconcile the venv (#307 drift path) — and uv sync's exact mode
UNINSTALLS every package not in the lockfile. That silently deleted
user-pip-installed optional engines (voxcpm, kittentts — packages the
app's own Settings → Engines hints tell users to install into this
venv) on every single update. Reported as "VoxCPM2 is automatically
uninstalled after updating Studio."

Fix: the routine drift sync now carries --inexact — locked deps are
still installed/upgraded exactly per the lockfile, but extras the user
added on purpose are left alone. Deliberate asymmetry: the venv-REPAIR
sync stays exact, because repair runs when the venv is broken and a
user-installed extra is a plausible cause — healing must restore the
known-good locked state. First-run syncs are untouched (a fresh venv
has no extras; exact == inexact there).

Both sync arg sets are now named constants with contract tests pinning
the asymmetry, so neither side can silently regress.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 10:06:23 +05:30
cb6c72b409 docs(faq): honest ElevenLabs comparison — where each wins, and why dub quality varies (community question) (#1028)
Asked directly on Discord ('how it compares to something like 11 labs
in quality?'). The old answer ('yes, comparable for most use cases')
oversold — the honest version names where ElevenLabs still wins
(out-of-the-box English polish/consistency) and where OmniVoice is
genuinely competitive (cloning from clean references, 646 languages,
structural advantages), plus the dubbing-specific truth another
same-day report surfaced: a dub is a chain, and incoherent output
usually traces to transcription quality on the user's source audio —
with the check-the-original-text-first debugging step that actually
helps.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 09:13:11 +05:30
30 changed files with 1389 additions and 29 deletions
+18
View File
@@ -8,6 +8,24 @@ The bundled TTS model package (`pyproject.toml`) is versioned independently.
## [Unreleased]
## [0.3.15] — 2026-07-10
The cold-start release. Three "why is this broken on my machine" mysteries got solved at their roots: **first generations stop dying at 300 seconds** (the timeout was counting the model download as generation time — @moduvoice measured it on a Tesla T4: 0% GPU for the full window), **updates stop deleting engines you installed yourself** (the updater's dependency sync removed anything not in the app's lockfile — including things our own UI told you to install), and **the "slower than v0.3.5" regression is found and fixed** (clone profiles without a transcript were silently re-running a full Whisper transcription on every single generate). Also: Clear History is back, auto-played audio is finally stoppable, @stronghamjji hardened the dub pipeline against wedged transcribes, and @shakib30's community Colab notebook is now the linked no-GPU path. Thank you all.
### Added
- **Agent Skills: `npx skills add debpalash/omnivoice-studio`.** Two installable [skills](https://skills.sh) now ship in the repo — `omnivoice` teaches any AI agent (Claude Code, Cursor, Codex, …) to speak and transcribe through your local install via the OpenAI-compatible API, including your cloned voices; `oss-maintainer` packages the maintainer methodology this project is run with.
### Fixed
- **A fresh install's first generation no longer dies at 300 seconds while the model is still downloading.** The generate timeout was one clock around everything — including the engine's lazy multi-GB weight download on a cold start — so first requests burned the whole budget on the download (0% GPU the entire time, as a contributor's Tesla T4 verification measured) and failed with a misleading "too heavy for the available compute" error. Model loading now runs first under its own, much larger budget; the generate clock starts only once the engine is warm. A genuinely stalled download gets a new error that says so and points at Settings → Models. (#1033, #1037, evidence from #1014)
- **The OpenAI-compatible speech endpoint stops silently discarding quality settings.** `POST /v1/audio/speech` accepted `num_step` and `guidance_scale` in the request body with a 200 OK — and dropped them without a word, so API callers couldn't reach the model's documented quality preset (`num_step: 32`). Both are now declared, validated, and passed through to the engine, matching the native `/generate` endpoint. Caught by a contributor's measured Tesla T4 verification pass. (#1014)
- **Updating no longer uninstalls engines you added yourself.** Optional engines installed with pip into the app's environment (VoxCPM2, KittenTTS — exactly what Settings → Engines' own install hints say to do) were silently deleted by every app update, because the update's dependency sync removed anything not in the app's lockfile. Routine updates now leave your additions alone; the repair path ("Clean & Retry") still restores the exact known-good state, since a broken environment is sometimes *caused* by an extra package. (#1029)
- **Voice cloning stops re-transcribing the same reference clip on every generate.** Since v0.3.6, a profile saved without a transcript (the default) triggered a full ASR model load *plus* a transcription of the reference on every single synthesis — the "TTS got much slower than v0.3.5, same settings" regression. The first auto-transcription is now saved onto the profile, and repeated ad-hoc uploads of the same clip reuse a content-keyed transcript cache — so the cost is paid once, not per request. A transcript you typed yourself is never overwritten. (#1032)
- **The Clear History button is back.** The workspace redesign moved generation history into the right-side panels but dropped the old sidebar's clear-all control, leaving one-by-one deletion as the only way to empty a long history. Both the Voice and Dub history panels now have a Clear History button (with a confirmation), scoped to that workspace's history. (#1032)
- **The audio that auto-plays after a render can finally be stopped anywhere.** The finished-render playback has no on-screen player, and the only stop control lived in the Voice workspace's action bar — audio started from the Dub workspace, a profile preview, or after navigating away simply played to the end. A stop button now appears above the status area whenever such playback is active, on every page. The existing Settings → Appearance "Auto-play preview" toggle now also governs this playback, as its description always promised. (#1032)
## [0.3.14] — 2026-07-09
A fast follow to v0.3.13: **every engine family now has a visible picker.** Settings → Engines showed only a TTS table, with the ASR and LLM pickers hidden behind a low-discoverability tab — so the 10 transcription engines (including the new OpenAI-compatible backend) looked unswitchable without env vars. Now all three families get their own table. Also in: the Linux AppImage's white-screen auto-workaround now checks the WebKitGTK it actually ships (not whatever your system reports), and installing to a different drive on Windows is properly documented.
+23 -1
View File
@@ -397,6 +397,20 @@ print(result.text)
Want the whole surface (100+ endpoints)? The full REST API reference is embedded in the app — **Settings → OpenAPI Reference** (Scalar-powered), or the `{}` button in the footer.
### 📓 Run on Google Colab (community)
No local GPU? A community member ([@shakib30](https://github.com/shakib30)) maintains a working Colab notebook: [shakib30/OmniVoice-Studio-google-colab](https://github.com/shakib30/OmniVoice-Studio-google-colab). Community-maintained — issues with the notebook go there; issues with OmniVoice itself come here.
### 🤝 Agent Skills
Teach your AI agent (Claude Code, Cursor, Codex, …) to use OmniVoice with one command:
```sh
npx skills add debpalash/omnivoice-studio
```
Ships two [skills](https://skills.sh): **`omnivoice`** — speak and transcribe through your local install (including your cloned voices) from any agent, free and offline; and **`oss-maintainer`** — the maintainer methodology this project is run with, for anyone running their own OSS project with an agent.
---
## 🗺️ Roadmap
@@ -525,7 +539,15 @@ Yes please — bug fixes, new TTS engine adapters, UI improvements, docs, transl
<details>
<summary><b>Is this really as good as ElevenLabs?</b></summary>
<br/>
For voice cloning and dubbing, yes — OmniVoice uses a state-of-the-art diffusion TTS model with 646 languages (ElevenLabs supports 32). Quality is comparable for most use cases. Where ElevenLabs wins is in their polished cloud API and pre-made voice library. OmniVoice wins on privacy, cost, language coverage, and customizability.
Honest answer: <b>it depends on what you're doing.</b>
<b>Where OmniVoice is genuinely competitive:</b> voice cloning from a clean reference clip (state-of-the-art open diffusion TTS), language coverage (646 languages vs. their 32), and everything structural — no per-character billing, no usage caps, no audio leaving your machine, full pipeline customizability (10 TTS engines, 10 ASR engines, your choice of translation).
<b>Where ElevenLabs still wins:</b> out-of-the-box consistency and polish, especially for English TTS. Their one model is heavily tuned; our quality depends on which engine you pick, your hardware, and — for cloning — the reference audio (a dry, close-mic clip clones dramatically better than a noisy or echoey one).
<b>For dubbing specifically:</b> a dub is a chain — transcription → translation → cloning → synthesis — and the output is only as good as its weakest link on <i>your</i> source material. Noisy or accented source audio degrades transcription, which degrades everything downstream; some language pairs translate better than others. If parts of a dub come out incoherent, check the segment table's <i>original</i> text first: if the transcription is already wrong there, switch the ASR engine (Settings → Engines) or use cleaner source audio — that's usually the fix, not the voice.
Try it on your real material — it's free and takes one download. Many users find it replaces ElevenLabs outright; some keep both for different jobs. Both outcomes are fine with us.
</details>
<details>
+28 -6
View File
@@ -1008,9 +1008,22 @@ async def dub_transcribe_stream(
yield _sse_event("ping", {})
if clones:
from services.speaker_clone import refine_ref_texts
clones = await loop.run_in_executor(
_gpu_pool, lambda: refine_ref_texts(clones, _asr_backend),
)
# Bound the re-transcribe like every other ASR dispatch in
# this file (#730): a wedged transcribe would otherwise hold
# the GPU-pool worker forever and starve later work into a
# "can't reach backend". On timeout the guard resets the pool
# and raises — keep the original (unrefined) clones, matching
# refine_ref_text's own "failure is a strict no-op" fallback.
try:
clones = await run_transcribe_guarded(
_gpu_pool,
lambda: refine_ref_texts(clones, _asr_backend),
what="Dub clone ref-text refine",
)
except ASRTimeoutError as e:
logger.warning(
"clone ref-text refine timed out; keeping original ref_text: %s", e
)
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
# own reference from the vocals so the dub of each line matches the
# prosody of its source line. Short lines fall back to the
@@ -1031,9 +1044,18 @@ async def dub_transcribe_stream(
)
if seg_clones:
from services.speaker_clone import refine_ref_texts
seg_clones = await loop.run_in_executor(
_gpu_pool, lambda: refine_ref_texts(seg_clones, _asr_backend),
)
# Same guard as the per-speaker refine above (#730):
# keep the original seg_clones on a wedge/timeout.
try:
seg_clones = await run_transcribe_guarded(
_gpu_pool,
lambda: refine_ref_texts(seg_clones, _asr_backend),
what="Dub segment ref-text refine",
)
except ASRTimeoutError as e:
logger.warning(
"segment ref-text refine timed out; keeping original ref_text: %s", e
)
job["segment_clones"] = seg_clones
except Exception as e:
logger.warning("per-segment clone refs skipped: %s", e)
+76
View File
@@ -598,6 +598,32 @@ def _run_backend_inference(
_oom_friendly_reraise(e)
def _persist_profile_ref_text(profile_id: str, ref_text: str) -> None:
"""Cache an auto-transcribed reference transcript onto its profile row.
#1032 perf regression: profiles saved without a transcript re-ran a FULL
ASR model load + transcribe on every /generate (the #308 auto-transcribe
path). Persisting the first transcript makes subsequent generates read it
from the row like a user-entered one. The guarded UPDATE only ever fills
an empty column it can never overwrite a transcript the user typed or a
lock wrote and a failure is logged, never raised (best-effort, same
contract as the transcribe itself)."""
try:
with db_conn() as conn:
updated = conn.execute(
"UPDATE voice_profiles SET ref_text=? "
"WHERE id=? AND (ref_text IS NULL OR ref_text='')",
(ref_text, profile_id),
).rowcount
if updated:
event_bus.emit("profiles", {"action": "updated", "id": profile_id})
except Exception as e: # noqa: BLE001 — cache write must not break generate
logger.warning(
"could not persist auto-transcribed ref_text onto profile %s: %s",
profile_id, e,
)
@router.post("/generate")
async def generate_speech(
text: str = Form(...),
@@ -693,11 +719,51 @@ async def generate_speech(
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
# ── #1033/#1037: warm the engine under the LOAD budget, not the generate
# budget. A cold adapter lazily loads (and possibly downloads multi-GB
# weights) inside generate(), so a fresh install's first request burned
# its whole OMNIVOICE_GENERATE_TIMEOUT_S window on the download and died
# with a misleading "too heavy for the available compute" 503 (#1014
# measured it: 0% GPU util for the full 300s). Model loading gets its own,
# larger budget (OMNIVOICE_MODEL_LOAD_TIMEOUT, default 1200s) — the same
# split get_model() already has for the native engine. Once warm, this is
# a no-op per request.
if _backend is not None:
from services.model_manager import _model_load_timeout
try:
await run_on_gpu_pool_guarded(
_backend.ensure_ready,
what=f"TTS engine '{engine_id}' model load",
timeout=_model_load_timeout(),
)
# Builtin TimeoutError base, not GpuJobTimeoutError — reload-proof
# class identity (see the twin catch in openai_compat.py).
except TimeoutError as exc:
logger.warning("engine load exceeded the model-load budget: %s", exc)
raise HTTPException(
status_code=503,
detail=(
f"TTS engine '{engine_id}' did not finish loading within its "
f"model-load budget — on a first run this usually means the "
f"weight download is slow or stalled (check Settings → Models "
f"for progress), not that generation failed. Retry once the "
f"model shows as installed."
),
) from exc
ref_audio_path = None
cleanup_ref = False
used_seed = seed
resolved_profile_id = None
history_mode = None # profile.kind when a profile drives; else inferred at insert
# #1032: profile id to persist an auto-transcribed reference transcript to.
# Set only for a plain (unlocked) clone profile whose stored ref_text is
# empty — the case where every /generate re-ran a full ASR model load +
# transcribe of the same clip. Locked profiles are excluded (their ref
# audio is the locked take, and unlocking would leave a mismatched
# transcript paired with the original reference); design profiles are
# excluded (a re-render replaces the sample, stranding a stale transcript).
persist_ref_text_profile_id = None
if profile_id:
with db_conn() as conn:
@@ -743,6 +809,11 @@ async def generate_speech(
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
if not ref_text and row["ref_text"]:
ref_text = row["ref_text"]
elif ref_audio_path and not ref_text:
# Empty stored transcript → the auto-transcribe below will
# run; cache its result onto the profile so it runs ONCE,
# not on every generate (#1032 perf regression).
persist_ref_text_profile_id = profile_id
if not instruct and row["instruct"]:
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
@@ -792,6 +863,11 @@ async def generate_speech(
except GpuJobTimeoutError as e:
logger.warning("reference transcribe hung (%s); using model ASR fallback", e)
ref_text = None
# #1032: cache the transcript onto its clone profile so the ASR model
# load + transcribe above happens once per profile, not per generate.
# Only fills an empty column — a user-entered transcript always wins.
if ref_text and persist_ref_text_profile_id:
_persist_profile_ref_text(persist_ref_text_profile_id, ref_text)
# #526: materialize a concrete seed when none was supplied (and no profile
# pinned one) so the take is reproducible and we can hand it back via the
+54
View File
@@ -108,6 +108,23 @@ class SpeechRequest(BaseModel):
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
)
# #1014: these two were silently DISCARDED before (pydantic ignores
# undeclared fields) — a 200 OK that quietly dropped the caller's quality
# knobs. Declared now and passed through, matching the native /generate
# form fields (defaults there: num_step=16, guidance_scale=2.0; the
# model's documented "quality" preset is num_step=32).
num_step: Optional[int] = Field(
default=None,
ge=1,
le=128,
description="OmniVoice extension: iterative unmasking steps (app default 16; 32 = the model's documented quality preset).",
)
guidance_scale: Optional[float] = Field(
default=None,
gt=0,
le=20,
description="OmniVoice extension: classifier-free guidance scale (app default 2.0).",
)
class TranscriptionResponse(BaseModel):
@@ -274,6 +291,10 @@ async def create_speech(req: SpeechRequest):
kw["chunk_duration"] = req.chunk_duration
if req.chunk_threshold is not None:
kw["chunk_threshold"] = req.chunk_threshold
if req.num_step is not None:
kw["num_step"] = req.num_step
if req.guidance_scale is not None:
kw["guidance_scale"] = req.guidance_scale
if req.language:
kw["language"] = req.language
if req.instruct:
@@ -311,6 +332,39 @@ async def create_speech(req: SpeechRequest):
# Not a profile ID — might be a KittenTTS preset or similar
kw["voice"] = voice
# ── #1033/#1037/#1014: warm the engine under the LOAD budget before the
# generate clock starts. The T4 verification (#1014) measured a fresh
# install's first /v1/audio/speech burning its whole 300s generate budget
# on the multi-GB checkpoint download (0% GPU util throughout) and dying
# with a misleading "too heavy for the available compute" error. Model
# loading gets OMNIVOICE_MODEL_LOAD_TIMEOUT (default 1200s); once warm
# this is a per-request no-op.
from services.model_manager import _model_load_timeout
try:
await run_on_gpu_pool_guarded(
backend.ensure_ready,
what=f"TTS engine '{backend.id}' model load",
timeout=_model_load_timeout(),
)
# Catch the BUILTIN TimeoutError base, not GpuJobTimeoutError by name:
# several tests reload services.model_manager mid-suite, so a class
# imported at call time can differ in identity from the one the guard
# (bound at this module's import) actually raises — the except would
# silently miss. The builtin base has one identity forever. (Caught by
# this exact test failing CI-only, in full-suite order.)
except TimeoutError as e:
logger.warning("engine load exceeded the model-load budget: %s", e)
raise HTTPException(
status_code=503,
detail=(
f"TTS engine '{backend.id}' did not finish loading within its "
f"model-load budget — on a first run this usually means the weight "
f"download is slow or stalled (check Settings → Models for "
f"progress), not that generation failed. Retry once the model "
f"shows as installed."
),
) from e
try:
# Bounded + pool-reset on hang so a wedged TTS request can't starve the
# GPU pool and brick the backend (#730 class).
+1 -1
View File
@@ -24,7 +24,7 @@ from pathlib import Path
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
# release.yml's version-bump job, so it stays equal to
# pyproject/tauri.conf/Cargo/package.json.
_FALLBACK_VERSION = "0.3.14"
_FALLBACK_VERSION = "0.3.15"
def _fallback_version() -> str:
+48
View File
@@ -29,6 +29,7 @@ import os
import re
import threading
from abc import ABC, abstractmethod
from collections import OrderedDict
from typing import Optional
logger = logging.getLogger("omnivoice.asr")
@@ -2026,6 +2027,38 @@ def get_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
return cls()
# ── Reference-transcript cache (#1032) ──────────────────────────────────────
# `get_active_asr_backend()` returns a FRESH backend instance per call for the
# whisper family, so every `transcribe_reference` used to reload whisper
# weights from scratch — a multi-second (CPU: tens of seconds) hit on EVERY
# /generate whose reference clip has no stored transcript (#308 introduced the
# call; profiles saved without a transcript hit it per request). The reference
# audio is identical across those requests, so cache the *transcript* keyed by
# the file's content hash: no model or VRAM is held, repeated generates with
# the same clip skip ASR entirely. Bounded LRU; failures (None) are never
# cached so a transient ASR problem still retries next request.
_REF_TRANSCRIPT_CACHE_MAX = 64
_ref_transcript_cache: "OrderedDict[str, str]" = OrderedDict()
_ref_transcript_lock = threading.Lock()
def _ref_audio_fingerprint(audio_path: str) -> str | None:
"""sha256 of the clip's bytes, or None when unreadable (→ no caching).
Content-keyed (not path-keyed) because ad-hoc clone uploads land in a new
NamedTemporaryFile per request the path changes, the bytes don't.
Reference clips are seconds long, so hashing is negligible next to ASR."""
import hashlib
try:
h = hashlib.sha256()
with open(audio_path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
except OSError:
return None
def transcribe_reference(audio_path: str) -> str | None:
"""Transcribe a voice-clone reference clip with the active ASR backend.
@@ -2037,7 +2070,16 @@ def transcribe_reference(audio_path: str) -> str | None:
the model-attached pipeline is only reached when it is genuinely the last
resort. Returns ``None`` on any failure callers pass ``ref_text=None``
through and the model's built-in fallback still gets its chance.
Results are cached by audio content (#1032) — see the cache notes above.
"""
fingerprint = _ref_audio_fingerprint(audio_path)
if fingerprint is not None:
with _ref_transcript_lock:
cached = _ref_transcript_cache.get(fingerprint)
if cached is not None:
_ref_transcript_cache.move_to_end(fingerprint)
return cached
try:
backend = get_active_asr_backend()
except Exception as e: # noqa: BLE001 — never let ASR break generation
@@ -2061,6 +2103,12 @@ def transcribe_reference(audio_path: str) -> str | None:
(seg.get("text") or "").strip() for seg in result.get("segments", [])
)
text = (text or "").strip()
if text and fingerprint is not None:
with _ref_transcript_lock:
_ref_transcript_cache[fingerprint] = text
_ref_transcript_cache.move_to_end(fingerprint)
while len(_ref_transcript_cache) > _REF_TRANSCRIPT_CACHE_MAX:
_ref_transcript_cache.popitem(last=False)
return text or None
+20
View File
@@ -142,6 +142,26 @@ class TTSBackend(ABC):
#: (e.g. "young female, warm tone, British accent") without reference audio.
supports_voice_design: bool = False
def ensure_ready(self) -> None:
"""Load model weights now (blocking), so callers can separate the
LOAD budget from the GENERATE budget (#1033/#1037 class).
Every adapter lazily loads inside ``generate()`` via a private
``_ensure_loaded()`` which meant a cold first call spent its whole
``OMNIVOICE_GENERATE_TIMEOUT_S`` window (default 300s) downloading /
loading weights and got killed with a misleading "too heavy for the
available compute" error (measured in the wild on a fresh install:
multi-GB checkpoint download, 0% GPU util, #1014). Routes call this
first under the model-load budget (``OMNIVOICE_MODEL_LOAD_TIMEOUT``,
default 1200s), then start the generate clock on an already-warm
engine. Default implementation dispatches to the adapter's own
``_ensure_loaded`` when present; engines without lazy state no-op.
Must be called on the GPU pool (it's blocking), same as generate.
"""
loader = getattr(self, "_ensure_loaded", None)
if callable(loader):
loader()
#: Whether this engine already emits mastered, studio-grade audio and should
#: therefore skip the shared apply_mastering() chain (highpass + Compressor,
#: tuned for OmniVoice's 24 kHz output). Studio engines like VoxCPM2 (native
+60
View File
@@ -0,0 +1,60 @@
# Verified Tesla T4 (16GB) inference notes
Measured on a real NVIDIA Tesla T4 (16GB, Turing/sm_75), driver 550.163.01 (CUDA 12.8), torch
2.8.0+cu128, transformers 5.3.0, Python 3.11.15 (uv-managed). Engine under test: the default
`omnivoice` TTS backend (`OMNIVOICE_TTS_BACKEND=omnivoice`).
## Cold-cache first call can time out at 300s
The first `generate()` call lazily downloads the ~2.3GB `k2-fsa/OmniVoice` checkpoint, and that
download happens *inside* the `OMNIVOICE_GENERATE_TIMEOUT_S` budget (default 300s). On a fresh
install, the very first `POST /v1/audio/speech` can fail like this even though the GPU isn't
actually short on memory:
```
ERROR [omnivoice.openai_compat] OpenAI TTS failed: OpenAI TTS generate exceeded 300s and was
abandoned — the backend is running, but the job was too heavy for the available compute.
... most often the GPU is VRAM-starved ...
```
VRAM sampling during the failure showed a flat ~2GB with 0% GPU utilization for the whole 300s —
consistent with waiting on a download, not compute. Once the checkpoint is cached, the identical
request succeeds in ~1s (reproduced 5x: 1.574s / 1.034s / 1.065s / 0.995s / 0.911s).
**Workaround (no code change needed, both already exist):**
- For headless/API-only setups, pre-fetch the checkpoint before your first real TTS request:
```bash
curl -X POST http://localhost:3900/models/install \
-H "Content-Type: application/json" \
-d '{"repo_id": "k2-fsa/OmniVoice"}'
```
(`repo_id` is required — `InstallModelRequest` in `backend/api/schemas.py` rejects a bare/empty
body — and must match one of the entries in `KNOWN_MODELS`, e.g. the default engine's
`k2-fsa/OmniVoice`.) Progress streams over the existing `/setup/download-stream` SSE feed.
- Or raise `OMNIVOICE_GENERATE_TIMEOUT_S` for the first request.
## OpenAI-compatible endpoint doesn't expose `num_step` / `guidance_scale`
`POST /v1/audio/speech`'s request schema doesn't declare `num_step` or `guidance_scale` fields —
sending them in the JSON body returns `200 OK` but they're silently discarded (pydantic's default
`extra=ignore` behavior). The native multipart `POST /generate` endpoint *does* expose both as
explicit form fields, so use that endpoint if you need to control them.
Separately: the app's own default for `num_step` is 16 — half of the model's documented default of
32 (see `docs/generation-parameters.md`, "Use 16 for faster inference"). Not a bug, just not stated
that the app already runs the "fast" preset unless you override it via `/generate`.
## T4 acceleration checklist
| Option | Status |
|---|---|
| dtype | `torch.float16` hardcoded for the `omnivoice` engine (`model_manager.py`) — correct for Turing (no bf16 tensor cores this generation). No env var override for this engine specifically (ASR engines have `ASR_COMPUTE_TYPE`; `dots_tts`/`indextts` have their own precision vars; `omnivoice` doesn't). |
| Attention | `sdpa`, selected automatically since `flash_attn` isn't installed (`_supports_flash_attn_2=True` is declared but the package itself is absent) — safe on T4. |
| int8 | No int8 path for this engine (ASR's CTranslate2 `int8` and `sherpa-onnx`'s int8 ONNX models are separate/unrelated). |
| CUDA Graphs | No direct API usage in the app. Reachable indirectly via `torch.compile(mode="reduce-overhead")`, which the app attempts **by default** on this GPU (T4/sm_75 isn't in the framework's compile-exclusion list, unlike newer/Blackwell GPUs). The numbers above were measured with `TORCH_COMPILE_DISABLE=1` for a clean eager baseline. |
| torch.compile | Attempted by default on T4 (see above) — not evaluated further here. |
## VRAM
Peak measured: 2487 MiB (`nvidia-smi`) / 2.050 GB (`torch.cuda.max_memory_allocated()`) for the
default `omnivoice` engine — comfortably fits even the README's stated "minimum" (4GB) tier.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "omnivoice-studio",
"version": "0.3.14",
"version": "0.3.15",
"private": true,
"license": "AGPL-3.0-only",
"type": "module",
+1 -1
View File
@@ -2941,7 +2941,7 @@ dependencies = [
[[package]]
name = "omnivoice-studio"
version = "0.3.14"
version = "0.3.15"
dependencies = [
"arboard",
"dirs-next",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "omnivoice-studio"
version = "0.3.14"
version = "0.3.15"
description = "OmniVoice Studio AI voice cloning & dubbing desktop app"
authors = ["Debpalash"]
license = "AGPL-3.0-only"
+47 -3
View File
@@ -713,6 +713,27 @@ fn sync_failure_is_torch_download(tail: &str) -> bool {
/// distro-matched ROCm builds torch's own index doesn't carry).
const ROCM_TORCH_INDEX: &str = "https://download.pytorch.org/whl/rocm6.4";
/// Args for the routine update-drift sync (#307 path) — the one that runs on
/// every app update when `uv.lock` changed. `--inexact` is the fix for #1029:
/// plain `uv sync` UNINSTALLS every package not in the lockfile, which
/// silently deleted user-pip-installed optional engines (voxcpm, kittentts —
/// packages the app's own Settings → Engines hints tell users to install
/// into this venv) on every single update. `--inexact` still installs/
/// upgrades everything the lockfile demands — locked deps stay exactly
/// correct — it just stops removing extras the user added on purpose.
///
/// Deliberately NOT applied to the repair sync (`repair_sync_args`): repair
/// runs when the venv is *broken*, and a user-installed extra is a plausible
/// cause — healing must restore the known-good locked state, extras
/// included-out. An engine lost to a repair is re-installable; a venv that
/// repair can't actually repair is a support thread.
const DRIFT_SYNC_ARGS: [&str; 5] = ["sync", "--frozen", "--inexact", "--no-dev", "--verbose"];
/// Exact-sync args for the venv-repair path — see `DRIFT_SYNC_ARGS` for why
/// repair stays exact while the update-drift sync preserves user extras.
const REPAIR_SYNC_ARGS_LOCKED: [&str; 4] = ["sync", "--frozen", "--no-dev", "--verbose"];
const REPAIR_SYNC_ARGS_UNLOCKED: [&str; 3] = ["sync", "--no-dev", "--verbose"];
/// `uv pip install` args that replace the default CUDA torch build with the AMD
/// ROCm wheel (#124). Opt-in (gated on OMNIVOICE_TORCH_VARIANT=rocm by the
/// caller); the detection side (`get_best_device`) already routes ROCm through
@@ -1254,7 +1275,7 @@ manually, then relaunch.",
drift_cmd.env("UV_INDEX_URL", "https://mirrors.aliyun.com/pypi/simple/");
}
drift_cmd
.args(["sync", "--frozen", "--no-dev", "--verbose"])
.args(DRIFT_SYNC_ARGS)
.current_dir(&project_dir);
match run_streaming(app, "installing_deps", &mut drift_cmd) {
Ok(ref s) if s.success() => {
@@ -1329,9 +1350,9 @@ the existing venv; newly added dependencies may be missing (#307)",
apply_uv_http_env(&mut repair_cmd);
let has_lockfile = project_dir.join("uv.lock").is_file();
if has_lockfile {
repair_cmd.args(["sync", "--frozen", "--no-dev", "--verbose"]);
repair_cmd.args(REPAIR_SYNC_ARGS_LOCKED);
} else {
repair_cmd.args(["sync", "--no-dev", "--verbose"]);
repair_cmd.args(REPAIR_SYNC_ARGS_UNLOCKED);
}
repair_cmd.current_dir(&project_dir);
let repair_status = run_streaming(app, "installing_deps", &mut repair_cmd);
@@ -1712,6 +1733,29 @@ mod tests {
use super::*;
use std::collections::HashMap;
#[test]
fn update_drift_sync_preserves_user_installed_engines() {
// #1029: the routine update sync must carry --inexact so a
// user-pip-installed optional engine (voxcpm, kittentts — packages
// the app's own Settings → Engines hints tell users to install into
// this venv) survives every update instead of being silently
// uninstalled. --frozen must stay (lockfile is the resolution truth).
assert!(DRIFT_SYNC_ARGS.contains(&"--inexact"),
"update-drift sync lost --inexact — user-installed engines get wiped on every update (#1029)");
assert!(DRIFT_SYNC_ARGS.contains(&"--frozen"));
}
#[test]
fn repair_sync_stays_exact() {
// Deliberate asymmetry with the drift sync: repair runs when the venv
// is BROKEN and a user-installed extra is a plausible cause — healing
// must restore the known-good locked state, extras included-out.
assert!(!REPAIR_SYNC_ARGS_LOCKED.contains(&"--inexact"),
"repair sync must stay exact — it's the recovery path when an extra broke the venv");
assert!(!REPAIR_SYNC_ARGS_UNLOCKED.contains(&"--inexact"));
assert!(REPAIR_SYNC_ARGS_LOCKED.contains(&"--frozen"));
}
#[test]
fn scrub_python_env_removes_bundled_runtime_vars() {
// #144: every uv/venv/pip subprocess must drop the AppImage's bundled
+30
View File
@@ -42,6 +42,7 @@ import WorkspaceVoices from './components/WorkspaceVoices';
import WorkspaceProjects from './components/WorkspaceProjects';
import ErrorBoundary from './components/ErrorBoundary';
import FloatingPill from './components/FloatingPill';
import PlaybackStopPill from './components/PlaybackStopPill';
import BackendCrashNotice from './components/BackendCrashNotice';
// RemoteAuthGate is mounted at the true outermost provider in main-app.jsx so
// it covers all app states (setup check / wizard / bootstrap), not just the
@@ -83,6 +84,8 @@ import {
renameProject as apiRenameProject,
} from './api/projects';
import { exportAction, exportReveal, exportRecord } from './api/exports';
import { clearHistory as apiClearHistory } from './api/generate';
import { clearDubHistory as apiClearDubHistory } from './api/dub';
import { isTauri, doubleClickMaximize, fileToMediaUrl, playBlobAudio } from './utils/media';
import { browserDownload } from './utils/download';
@@ -1151,6 +1154,27 @@ function App() {
}
};
// Clear-all for the workspace history panels (#1032). The control lived in
// the old left Sidebar; the workspace UX overhaul (#374) moved history into
// the right-side WorkspaceHistory panels and the button was dropped in the
// move restore it, scoped per workspace (voice = synth rows, dub = dubs).
const clearWorkspaceHistory = async (type) => {
const count = type === 'dub' ? dubHistory.length : history.length;
if (!(await askConfirm(i18n.t('sidebar.clear_confirm', { count })))) return;
try {
if (type === 'dub') {
await apiClearDubHistory();
loadDubHistory();
} else {
await apiClearHistory();
loadHistory();
}
toast.success(i18n.t('sidebar.history_cleared'));
} catch (err) {
toast.error(err.message);
}
};
// Install-plan screen outranks everything both on a true first run and
// when explicitly requested via `--setup`. Without this, a live backend
// answering /setup/status would route straight to the model wizard and the
@@ -1270,6 +1294,10 @@ function App() {
<FloatingPill />
{/* #1032: global stop for playback that has no on-screen player (the
generate auto-play / profile & segment previews via playBlobAudio). */}
<PlaybackStopPill />
{/* #941: honest surfacing of backend process crashes (exit code +
stderr tail from the shell's crash marker), with ack-on-view. */}
<BackendCrashNotice />
@@ -1498,6 +1526,7 @@ function App() {
dubHistory={dubHistory}
restoreDubHistory={restoreDubHistory}
deleteHistory={deleteHistory}
clearHistory={() => clearWorkspaceHistory('dub')}
/>
</div>
)}
@@ -1594,6 +1623,7 @@ function App() {
handleNativeExport={handleNativeExport}
restoreHistory={restoreHistory}
deleteHistory={deleteHistory}
clearHistory={() => clearWorkspaceHistory('synth')}
/>
</div>
</div>
@@ -0,0 +1,36 @@
import { Square } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { stopActivePlayback, usePlaybackSource } from '../utils/playback';
/**
* PlaybackStopPill global stop affordance for "invisible" audio playback
* (#1032).
*
* `playBlobAudio` plays through a bare Audio()/AudioContext with no on-screen
* player the generate auto-play, profile previews, and dub segment previews
* all use it (playback source 'output'). The only visible stop control was the
* Voice workspace ActionBar's CTA morph (#316), so the same audio started from
* the Dub workspace, a profile page, or right after navigating away could not
* be stopped at all. This pill renders whenever an 'output' playback is
* active, on every page, and stops it via the global single-playback manager.
*
* Sources with their own visible player UI (WaveformPlayer instances,
* 'design-preview', 'demo-output', gallery previews) are deliberately NOT
* covered they already have in-place pause/stop controls.
*/
export default function PlaybackStopPill() {
const { t } = useTranslation();
const source = usePlaybackSource();
if (source !== 'output') return null;
return (
<button
type="button"
onClick={stopActivePlayback}
aria-label={t('clone.stop_playback')}
className="fixed left-1/2 -translate-x-1/2 bottom-[calc(var(--logs-footer-height,28px)+64px)] z-[var(--z-toast)] inline-flex items-center gap-[6px] py-[6px] px-[14px] rounded-[var(--radius-pill)] border border-[color:var(--color-border-strong)] bg-[var(--color-bg-elev-1)] text-[color:var(--color-fg)] [font-size:var(--text-sm)] shadow-[var(--shadow-lg)] cursor-pointer [backdrop-filter:var(--glass-blur-md)] hover:bg-[var(--color-bg-elev-2)] focus-visible:[outline:2px_solid_var(--chrome-accent)] focus-visible:[outline-offset:1px]"
>
<Square size={12} /> {t('clone.stop_playback')}
</button>
);
}
+28 -6
View File
@@ -78,11 +78,27 @@ export default function WorkspaceHistory({
handleNativeExport,
restoreHistory,
deleteHistory,
clearHistory, // clear-all for this workspace's history (#1032)
}) {
const { t } = useTranslation();
const [filter, setFilter] = useState('all');
const [expanded, setExpanded] = useState(null); // row id with un-clamped title
// Clear-all affordance (#1032): the old left Sidebar had one; the workspace
// UX overhaul (#374) moved history here and dropped it. Confirm + endpoint
// live in App.jsx (clearWorkspaceHistory) this only renders the button.
const clearAllButton = (count) =>
clearHistory && count > 0 ? (
<button
type="button"
className="history-action-btn danger flex-[0_0_auto]"
onClick={clearHistory}
title={t('sidebar.clear_history')}
>
<Trash2 size={10} /> {t('sidebar.clear_history')}
</button>
) : null;
// Voice workspace = clone + design generations (dub lives in its own workspace).
const items = useMemo(() => {
const synth = history.filter((h) => h.mode === 'clone' || h.mode === 'design');
@@ -94,9 +110,12 @@ export default function WorkspaceHistory({
return (
<aside className="flex-[1_1_0] flex flex-col min-h-0 overflow-hidden">
<div className="flex-[0_0_auto] flex flex-col gap-[8px] py-[10px] px-[12px]">
<span className="inline-flex items-center gap-[6px] [font-family:var(--chrome-font-mono,var(--font-mono))] text-[0.72rem] font-semibold [letter-spacing:0.04em] uppercase text-[color:var(--chrome-fg-muted)]">
<History size={13} /> {t('history.dub_title', { defaultValue: 'Dub history' })}
</span>
<div className="flex items-center justify-between gap-[6px]">
<span className="inline-flex items-center gap-[6px] [font-family:var(--chrome-font-mono,var(--font-mono))] text-[0.72rem] font-semibold [letter-spacing:0.04em] uppercase text-[color:var(--chrome-fg-muted)]">
<History size={13} /> {t('history.dub_title', { defaultValue: 'Dub history' })}
</span>
{clearAllButton(dubHistory.length)}
</div>
</div>
<div className="flex-[1_1_auto] min-h-0 overflow-y-auto flex flex-col gap-[8px] p-[8px]">
{dubHistory.length === 0 ? (
@@ -156,9 +175,12 @@ export default function WorkspaceHistory({
return (
<aside className="flex-[1_1_0] flex flex-col min-h-0 overflow-hidden">
<div className="flex-[0_0_auto] flex flex-col gap-[8px] py-[10px] px-[12px]">
<span className="inline-flex items-center gap-[6px] [font-family:var(--chrome-font-mono,var(--font-mono))] text-[0.72rem] font-semibold [letter-spacing:0.04em] uppercase text-[color:var(--chrome-fg-muted)]">
<History size={13} /> {t('history.title', { defaultValue: 'History' })}
</span>
<div className="flex items-center justify-between gap-[6px]">
<span className="inline-flex items-center gap-[6px] [font-family:var(--chrome-font-mono,var(--font-mono))] text-[0.72rem] font-semibold [letter-spacing:0.04em] uppercase text-[color:var(--chrome-fg-muted)]">
<History size={13} /> {t('history.title', { defaultValue: 'History' })}
</span>
{clearAllButton(history.length)}
</div>
<div className="flex flex-wrap gap-[4px]">
{FILTERS.map((f) => (
<button
+10 -3
View File
@@ -242,9 +242,16 @@ export default function useTTS({ selectedProfile, setSelectedProfile, loadHistor
}
const blob = new Blob(chunks, { type: 'audio/wav' });
try {
await playBlobAudio(blob);
} catch (e) {}
// #1032: honor the Settings → Appearance "Auto-play preview" pref here
// too. #667 added the toggle ("play the output as soon as a render
// finishes") but only wired the WaveformPlayer preview sites — the main
// generate path kept auto-playing unconditionally. getState() (not a
// subscription) so the freshest value is read at completion time.
if (useAppStore.getState().autoPlayPreview) {
try {
await playBlobAudio(blob);
} catch (e) {}
}
await loadHistory();
setSidebarTab('history');
+5 -3
View File
@@ -167,9 +167,11 @@ export interface PrefsSlice {
/**
* Auto-play the output preview as soon as a render finishes (Voice Clone /
* Design / profile preview). Default ON preserves the long-standing
* behavior. Users batch-generating segments (#666) can turn it off so each
* finished clip doesn't start playing on its own.
* Design / profile preview, AND the studio generate path in useTTS
* #1032 wired the latter; #666's toggle only covered the WaveformPlayer
* sites). Default ON preserves the long-standing behavior. Users
* batch-generating segments (#666) can turn it off so each finished clip
* doesn't start playing on its own.
*/
autoPlayPreview: boolean;
setAutoPlayPreview: (on: boolean) => void;
@@ -0,0 +1,56 @@
import { describe, it, expect, vi, afterEach } from 'vitest';
import { render, screen, fireEvent, act } from '@testing-library/react';
import PlaybackStopPill from '../components/PlaybackStopPill';
import { claimPlayback, stopActivePlayback } from '../utils/playback';
// #1032: playBlobAudio plays through a bare Audio()/AudioContext with no
// on-screen player (source 'output') the generate auto-play and profile /
// segment previews. Outside the Voice workspace's ActionBar there was no way
// to stop it. The global pill must appear for 'output' playback on any page,
// stop it on click, and stay out of the way for playback that already has a
// visible player (WaveformPlayer sources, demos).
afterEach(() => {
stopActivePlayback(); // never leak an active claim into the next test
});
describe('PlaybackStopPill (#1032)', () => {
it('renders nothing when idle', () => {
render(<PlaybackStopPill />);
expect(screen.queryByRole('button', { name: /stop playback/i })).toBeNull();
});
it('appears for an "output" playback and stops it on click', () => {
const stop = vi.fn();
render(<PlaybackStopPill />);
act(() => {
claimPlayback(stop, 'output');
});
const btn = screen.getByRole('button', { name: /stop playback/i });
fireEvent.click(btn);
expect(stop).toHaveBeenCalledTimes(1);
// Manager cleared the pill unmounts.
expect(screen.queryByRole('button', { name: /stop playback/i })).toBeNull();
});
it('disappears when playback ends on its own (release)', () => {
render(<PlaybackStopPill />);
let release;
act(() => {
release = claimPlayback(vi.fn(), 'output');
});
expect(screen.getByRole('button', { name: /stop playback/i })).toBeInTheDocument();
act(() => {
release();
});
expect(screen.queryByRole('button', { name: /stop playback/i })).toBeNull();
});
it('ignores sources that already have visible player UI', () => {
render(<PlaybackStopPill />);
act(() => {
claimPlayback(vi.fn(), 'design-preview');
});
expect(screen.queryByRole('button', { name: /stop playback/i })).toBeNull();
});
});
@@ -0,0 +1,76 @@
import { describe, it, expect, vi } from 'vitest';
import { render, screen, fireEvent } from '@testing-library/react';
import WorkspaceHistory from '../components/WorkspaceHistory';
// #1032: the old left Sidebar had a "Clear History" button; the workspace UX
// overhaul (#374) moved history into the right-side WorkspaceHistory panels
// and dropped it per-item delete was the only way to empty a long history.
// The panel must expose a clear-all affordance again (both variants), wired
// to the handler App.jsx passes in (which owns confirm + endpoint + reload).
const synthItem = (id, mode = 'clone') => ({
id,
mode,
text: `take ${id}`,
language: 'en',
generation_time: 1.2,
// no audio_path on purpose keeps LazyWaveform (IntersectionObserver)
// out of jsdom; the clear-all button lives in the header regardless.
});
describe('WorkspaceHistory clear-all (#1032)', () => {
it('voice variant: shows Clear History and calls the handler', () => {
const clearHistory = vi.fn();
render(
<WorkspaceHistory
history={[synthItem('a'), synthItem('b', 'design')]}
clearHistory={clearHistory}
deleteHistory={vi.fn()}
restoreHistory={vi.fn()}
handleSaveHistoryAsProfile={vi.fn()}
handleLockProfile={vi.fn()}
handleNativeExport={vi.fn()}
/>,
);
const btn = screen.getByRole('button', { name: /clear history/i });
fireEvent.click(btn);
expect(clearHistory).toHaveBeenCalledTimes(1);
});
it('voice variant: hidden when history is empty', () => {
render(<WorkspaceHistory history={[]} clearHistory={vi.fn()} deleteHistory={vi.fn()} />);
expect(screen.queryByRole('button', { name: /clear history/i })).toBeNull();
});
it('voice variant: hidden when no handler is provided (defensive)', () => {
render(<WorkspaceHistory history={[synthItem('a')]} deleteHistory={vi.fn()} />);
expect(screen.queryByRole('button', { name: /clear history/i })).toBeNull();
});
it('dub variant: shows Clear History and calls the handler', () => {
const clearHistory = vi.fn();
render(
<WorkspaceHistory
variant="dub"
dubHistory={[{ id: 'd1', filename: 'movie.mp4', segments_count: 3, duration: 12 }]}
clearHistory={clearHistory}
deleteHistory={vi.fn()}
restoreDubHistory={vi.fn()}
/>,
);
fireEvent.click(screen.getByRole('button', { name: /clear history/i }));
expect(clearHistory).toHaveBeenCalledTimes(1);
});
it('dub variant: hidden when dub history is empty', () => {
render(
<WorkspaceHistory
variant="dub"
dubHistory={[]}
clearHistory={vi.fn()}
deleteHistory={vi.fn()}
/>,
);
expect(screen.queryByRole('button', { name: /clear history/i })).toBeNull();
});
});
@@ -0,0 +1,78 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { renderHook, act } from '@testing-library/react';
import useTTS from '../hooks/useTTS';
import { useAppStore } from '../store';
import { playBlobAudio } from '../utils/media';
// #1032: Settings Appearance "Auto-play preview" ("play the output as soon
// as a render finishes", #666/#667) only gated the WaveformPlayer preview
// sites the main generate path (useTTS playBlobAudio) kept auto-playing
// unconditionally, with no visible way to stop it outside the Voice
// workspace. The pref must gate the generate auto-play too; default ON keeps
// the long-standing behavior.
vi.mock('../utils/media', async (importOriginal) => {
const actual = await importOriginal();
return {
...actual,
playBlobAudio: vi.fn().mockResolvedValue(undefined),
playPing: vi.fn(),
};
});
vi.mock('../api/generate', async (importOriginal) => {
const actual = await importOriginal();
return {
...actual,
generateSpeech: vi.fn().mockImplementation(async () => {
let served = false;
return {
body: {
getReader: () => ({
read: async () => {
if (served) return { done: true, value: undefined };
served = true;
return { done: false, value: new Uint8Array([0, 1, 2]) };
},
}),
},
headers: { get: () => null },
};
}),
};
});
const hookProps = () => ({
selectedProfile: null,
setSelectedProfile: vi.fn(),
loadHistory: vi.fn().mockResolvedValue(undefined),
profiles: [],
});
async function runGenerate() {
const { result } = renderHook(() => useTTS(hookProps()));
await act(async () => {
await result.current.handleGenerate();
});
}
beforeEach(() => {
vi.mocked(playBlobAudio).mockClear();
// Design path needs no reference audio; non-empty text passes validation.
useAppStore.setState({ text: 'Hello there', defineMethod: 'design' });
});
describe('useTTS auto-play pref (#1032)', () => {
it('auto-plays the finished render when autoPlayPreview is ON (default)', async () => {
useAppStore.setState({ autoPlayPreview: true });
await runGenerate();
expect(playBlobAudio).toHaveBeenCalledTimes(1);
expect(playBlobAudio.mock.calls[0][0]).toBeInstanceOf(Blob);
});
it('does NOT auto-play when autoPlayPreview is OFF', async () => {
useAppStore.setState({ autoPlayPreview: false });
await runGenerate();
expect(playBlobAudio).not.toHaveBeenCalled();
});
});
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "omnivoice"
version = "0.3.14"
version = "0.3.15"
description = "OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models"
readme = "README.md"
# Free and open-source under the GNU Affero General Public License v3 (see
+66
View File
@@ -0,0 +1,66 @@
---
name: omnivoice
description: Speak and transcribe through the user's local OmniVoice Studio — free, offline, no API key. Text-to-speech (including the user's cloned voices) and speech-to-text via the OpenAI-compatible API at localhost:3900.
---
# OmniVoice — local TTS & STT
The user runs [OmniVoice Studio](https://github.com/debpalash/OmniVoice-Studio), a fully-local voice app exposing an OpenAI-compatible audio API at `http://localhost:3900/v1`. Use it whenever the user asks to generate speech, narrate text, clone a voice, or transcribe audio — it costs nothing, works offline, and their audio never leaves the machine.
## Before the first call
Check the backend is up:
```sh
curl -sf http://localhost:3900/health
```
If it fails, tell the user to launch OmniVoice Studio (or `bun run desktop-prod` from a source checkout) — don't fall back to a cloud API without asking; local-first is why they installed it.
## Text-to-speech
```sh
curl -s http://localhost:3900/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model": "tts-1", "voice": "alloy", "input": "TEXT HERE", "response_format": "wav"}' \
--output speech.wav
```
- `model`: `tts-1` or `tts-1-hd` — both map to the user's active TTS engine.
- `voice`: OpenAI names (`alloy`, `echo`, `nova`, …) work, **but the real power is the user's own cloned voice-profile IDs** — discover them first (below) and prefer a named clone when the user says "my voice" / "the narrator voice" / a profile by name.
- `response_format`: `wav`, `mp3`, `flac`, `opus`, or `pcm`.
- Long texts are fine — the engine chunks at sentence boundaries internally.
## Discover the user's voices
```sh
curl -s http://localhost:3900/v1/audio/voices
```
Lists every cloned/designed voice profile (id + name) and the installed engines. Use a profile's id as the `voice` value in `/speech`.
## Speech-to-text
```sh
curl -s http://localhost:3900/v1/audio/transcriptions \
-F file=@clip.wav -F model=whisper-1 -F response_format=json
```
- `model`: `whisper-1` maps to the active ASR engine (WhisperX by default; the user picks in Settings → Engines).
- `response_format`: `json`, `text`, `verbose_json` (per-segment timestamps), `srt`, or `vtt` — use `srt`/`vtt` directly when the user wants subtitles.
## Python (openai SDK)
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:3900/v1", api_key="none") # any string; nothing checks it
audio = client.audio.speech.create(model="tts-1", voice="alloy", input="Hello!", response_format="wav")
text = client.audio.transcriptions.create(model="whisper-1", file=open("clip.wav", "rb")).text
```
## Notes
- **No API key, no rate limits, no billing** — it's the user's own hardware. First synthesis after a cold start may take longer (model loading); subsequent calls are fast.
- Anything beyond speech/transcription (video dubbing, batch jobs, voice design, audiobooks) lives in the full REST API — the interactive reference is embedded in the app at **Settings → OpenAPI Reference**, or ask the user to open it.
- If a call errors with an engine/model message, the actionable detail is usually in the response body — surface it to the user verbatim; OmniVoice's errors are written to be user-fixable (e.g. which Settings toggle to flip).
+56
View File
@@ -0,0 +1,56 @@
---
name: oss-maintainer
description: Run an open-source project's issue/PR/release loop like a careful human maintainer — triage to root cause, absorb community PRs before duplicating them, gate every merge, ship honest releases, and thank the people doing your QA for free.
---
# OSS Maintainer
You are operating an open-source project's maintenance loop: incoming issues, community PRs, CI, releases, and community channels. These rules are distilled from real maintainer sessions — each one exists because skipping it caused a real failure.
## The prime directive
**The queue has two exit states: absorbed or declined. Never limbo.** Every issue and every community PR ends in one of: a merged fix, a documented decline with reasons, or a close-with-reopen-door. "Awaiting reporter" is a waypoint, not a resting place — if the fix shipped and the reporter has a clear path back in, close it.
## Before you write any fix
1. **Check the open-PR queue first.** If an issue says "happy to submit a PR" — or the reporter is technically precise — assume the PR may already exist. Run `gh pr list` and search before implementing. Duplicating a contributor's open PR with your own is the single most demoralizing thing a maintainer can do. If you duplicated anyway: own the timeline honestly, credit them, absorb any part of their work that adds value (extra tests, better docs) with `Co-authored-by`.
2. **Read the actual code, not your memory of it.** Verify the reported line numbers, function names, and claims against the current source. Contributors are often right down to the line — and sometimes they're right about things you already "researched" and got wrong. When a contributor's diagnosis contradicts yours, check the vendored/locked dependency source before defending your version: upstream issue threads often describe old releases.
3. **Reproduce when possible; say so when you can't.** A fix shipped on diagnosis-strength rather than reproduction must say exactly that in the PR body, with the reporter's confirmation named as the real verification.
## Fix quality bar
- **Root-cause fully, then fix the class, not the instance.** If one call site dropped a parameter, grep for every sibling call site. If one error message lied, audit the whole error surface.
- **Every fix carries a fail-before/pass-after regression test.** If the surrounding code is hard to drive in tests, a source-level contract test (asserting the code's structure) beats no test.
- **Harden against recurrence.** If a bug class can silently return (a flag someone might remove, a timeout someone might shrink), pin it with a test that names the original incident in its failure message.
- The smallest correct change that is also recurrence-proof. Extra effort, not extra verbosity.
## Merging: gates, not vibes
- Merge on a **structural check**, evaluated at merge time, never sequentially assumed: required test check = pass AND mergeable = clean. Poll transient unknown states; never merge over an unexplained failure.
- **Flaky vs. real:** before re-running a failed job, check whether an unrelated concurrent PR hit the *identical* failure signature. Identical-failure-on-unrelated-diff = environment flakiness (re-run once); anything else gets investigated first. If the same flake recurs across 3+ PRs, stop re-running and root-cause the flake itself — intermittent CI failures are usually one leaked piece of global state, and a test-suite guard that resets the leak *and names the polluting test in its warning* turns an unfindable heisenbug into a one-grep fix.
- **Never trust a piped exit code.** `pytest | tail` exits with tail's status. Capture full output to a file and echo the real `$?` explicitly for anything gating a merge or release.
- Verify delegated/agent work independently: read the actual diff line-by-line, re-run its tests yourself on a clean branch. Never relay an agent's own success claims as your verification.
- Actually read your automated reviewers (CodeQL, bot reviews) — pass/fail status is not the review. Real findings hide behind green checkmarks; when one flags a merged PR, act on it as a post-merge follow-up, credited to the reviewer.
## Releases
- **Run the full gates BEFORE mutating any version file.** Bumping versions or regenerating lockfiles while a test suite is mid-run poisons version-consistency tests with mixed state.
- Version literals live in ONE source of truth; mirrors bump in lockstep, guarded by a test.
- **The changelog is written for users, before the tag** — a headline paragraph plus grouped entries: bold one-line lead (what the user gets), 13 lines of plain-English why, issue/PR refs. Never ship an auto-generated commit dump as release notes. Credit contributors by name in the headline when the release is theirs.
- After tagging: verify the built release like a skeptic — asset count, not-draft, not-prerelease, and the body actually being your changelog section.
- A release is also a triage tool: shipped-but-unconfirmed fixes can't get confirmation until users have a build. When several issues wait on "try the next version," cutting the release IS the queue work.
## Communication
- **Thank every issue and PR author — specifically.** Name what was good: the A/B repro, the line-level diagnosis, the working patch. Generic thanks reads as no thanks.
- **Lead with the outcome, stay honest.** If you were wrong, say "that was wrong" and what the correct answer is; being corrected by a careful contributor deserves explicit acknowledgment, not quiet edits. If a close was premature, correct the record plainly — don't gloss.
- Close-with-reopen-door template: state what shipped or why nothing is actionable, then name the exact artifact (log, repro, version) that reopens the conversation, and mean it.
- Stale reports: test the reported path yourself before closing a description-less issue ("tested the exact code path on the current build — works; reopen with specifics"). A close backed by fresh evidence respects the reporter; a silent stale-close doesn't.
- Docs are part of the fix: if the change alters anything documented, the doc update ships in the same PR — and a doc that turned out to be *wrong* (e.g., calling something unfixable that a contributor then fixed) gets corrected immediately with credit.
## Judgment defaults
- Old-version reports: ask the reporter to update past the relevant fixes before investigating deeply; close stale-version reports with an update path and reopen door.
- Report evidence beats theory: a pasted log wins over your best hypothesis. Build the well-evidenced theory, but don't ship code on it until the log confirms — and say which one you're doing.
- Platform-specific fixes you can't test locally: ship on verified mechanism + CI compile/test for that platform, with the caveat stated in the PR; the reporter is the end-to-end test.
- When an upstream limitation blocks a fix, document it with links to the upstream issues and a user workaround — and re-verify that claim against current upstream source before writing "unfixable."
+30
View File
@@ -117,3 +117,33 @@ def test_speed_passthrough(client, monkeypatch):
})
assert res.status_code == 200, res.text
assert fake.calls[0][1].get("speed") == pytest.approx(1.25)
def test_num_step_and_guidance_scale_passthrough(client, monkeypatch):
"""#1014: these were silently DISCARDED (200 OK, fields dropped) — a T4
hardware report caught it by comparing against the native /generate.
They must now reach the engine's generate() kwargs."""
fake = _make_fake_engine("fake-agent-quality")
monkeypatch.setitem(_tts_mod()._REGISTRY, "fake-agent-quality", fake)
res = client.post("/v1/audio/speech", json={
"model": "fake-agent-quality", "input": "Quality preset.",
"num_step": 32, "guidance_scale": 3.0,
})
assert res.status_code == 200, res.text
kw = fake.calls[0][1]
assert kw.get("num_step") == 32
assert kw.get("guidance_scale") == pytest.approx(3.0)
def test_num_step_and_guidance_scale_omitted_stay_absent(client, monkeypatch):
"""Engines that don't accept these kwargs must not suddenly receive
None values omitted means absent, exactly like duration/seed."""
fake = _make_fake_engine("fake-agent-defaults")
monkeypatch.setitem(_tts_mod()._REGISTRY, "fake-agent-defaults", fake)
res = client.post("/v1/audio/speech", json={
"model": "fake-agent-defaults", "input": "Defaults.",
})
assert res.status_code == 200, res.text
kw = fake.calls[0][1]
assert "num_step" not in kw
assert "guidance_scale" not in kw
+225
View File
@@ -0,0 +1,225 @@
"""#1032 perf regression: /generate must not re-transcribe a profile's
reference clip on every request.
The #308 auto-transcribe path (transcript-less reference → ASR registry) runs
per request, and `get_active_asr_backend()` builds a FRESH backend a full
whisper model load each time. A clone profile saved without a transcript
(POST /profiles defaults ref_text to "") therefore paid that load on EVERY
generate. The fix persists the first auto-transcript onto the profile row so
subsequent generates read it like a user-entered one.
Guards, tested here too:
* only an EMPTY ref_text column is ever filled (user text is never clobbered);
* a request-supplied ref_text skips the transcribe entirely (no persist);
* locked profiles are excluded their reference is the locked take, and
unlocking would strand a mismatched transcript against the original clip.
The engine layer is stubbed (no real model loads), matching
``tests/test_profile_language_propagation.py``.
"""
import importlib
import os
import uuid
import pytest
import torch
os.environ.setdefault("OMNIVOICE_MODEL", "test")
os.environ.setdefault("OMNIVOICE_DISABLE_FILE_LOG", "1")
def _tts_mod():
return importlib.import_module("services.tts_backend")
def _make_fake_engine(engine_id="fake-reftext-engine"):
class _FakeEngine(_tts_mod().TTSBackend):
id = engine_id
display_name = "Fake RefText Engine (test)"
applies_own_mastering = False
gpu_compat = ("cpu",)
calls: list = []
@property
def sample_rate(self) -> int:
return 24000
@property
def supported_languages(self) -> list[str]:
return ["multi"]
@classmethod
def is_available(cls):
return True, "ready"
def generate(self, text, **kw) -> torch.Tensor:
type(self).calls.append((text, kw))
return torch.zeros(1, 24000)
return _FakeEngine
class _CountingTranscribe:
def __init__(self, result="auto transcript words"):
self.calls = 0
self.result = result
def __call__(self, audio_path):
self.calls += 1
return self.result
@pytest.fixture()
def client():
from fastapi.testclient import TestClient
from main import app
return TestClient(app, client=("127.0.0.1", 50000))
@pytest.fixture()
def _init_db():
from core.db import init_db
init_db()
def _insert_profile(pid, **cols):
from core.db import db_conn
base = {
"id": pid, "name": "RefText Test", "kind": "clone", "created_at": 0.0,
"ref_text": "", "ref_audio_path": "reftext-test.wav",
}
base.update(cols)
keys = list(base)
with db_conn() as conn:
conn.execute(
f"INSERT INTO voice_profiles ({', '.join(keys)}) "
f"VALUES ({', '.join('?' * len(keys))})",
[base[k] for k in keys],
)
def _profile_ref_text(pid):
from core.db import db_conn
with db_conn() as conn:
return conn.execute(
"SELECT ref_text FROM voice_profiles WHERE id=?", (pid,)
).fetchone()["ref_text"]
@pytest.fixture()
def clone_profile(_init_db):
"""Unlocked clone profile with a reference clip but NO stored transcript —
the shape POST /profiles produces when the user doesn't type one."""
from core.db import db_conn
pid = f"vp-rt-{uuid.uuid4().hex[:8]}"
_insert_profile(pid)
yield pid
with db_conn() as conn:
conn.execute("DELETE FROM generation_history WHERE profile_id=?", (pid,))
conn.execute("DELETE FROM voice_profiles WHERE id=?", (pid,))
@pytest.fixture()
def locked_profile(_init_db):
"""Locked profile with an empty ref_text (legacy pre-lock-write rows)."""
from core.db import db_conn
pid = f"vp-rtl-{uuid.uuid4().hex[:8]}"
_insert_profile(pid, is_locked=1, locked_audio_path="locked-take.wav")
yield pid
with db_conn() as conn:
conn.execute("DELETE FROM generation_history WHERE profile_id=?", (pid,))
conn.execute("DELETE FROM voice_profiles WHERE id=?", (pid,))
def _generate(client, pid, fake_engine, **extra):
data = {"text": "Hello world", "profile_id": pid, "engine": fake_engine.id}
data.update(extra)
res = client.post("/generate", data=data)
assert res.status_code == 200, res.text
return res
def test_auto_transcript_persisted_and_reused(client, monkeypatch, clone_profile):
"""First generate transcribes ONCE and writes the transcript to the row;
the second reads it back no second transcribe. Before the fix the
counter hit 2 and the column stayed empty."""
import services.asr_backend as ab
fake = _make_fake_engine()
monkeypatch.setitem(_tts_mod()._REGISTRY, fake.id, fake)
fake.calls.clear()
counting = _CountingTranscribe()
monkeypatch.setattr(ab, "transcribe_reference", counting)
_generate(client, clone_profile, fake)
assert counting.calls == 1
assert _profile_ref_text(clone_profile) == "auto transcript words"
_generate(client, clone_profile, fake)
assert counting.calls == 1 # persisted row short-circuits the ASR path
# Both engine calls saw the transcript — the second from the DB row.
assert [kw.get("ref_text") for _, kw in fake.calls] == [
"auto transcript words", "auto transcript words",
]
def test_request_ref_text_wins_and_is_not_persisted(client, monkeypatch, clone_profile):
"""A request-supplied transcript must skip ASR and must NOT be written to
the profile (it may be a one-off override)."""
import services.asr_backend as ab
fake = _make_fake_engine()
monkeypatch.setitem(_tts_mod()._REGISTRY, fake.id, fake)
fake.calls.clear()
counting = _CountingTranscribe()
monkeypatch.setattr(ab, "transcribe_reference", counting)
_generate(client, clone_profile, fake, ref_text="typed by user")
assert counting.calls == 0
assert _profile_ref_text(clone_profile) == ""
def test_stored_transcript_never_overwritten(client, monkeypatch, _init_db):
"""A profile that already has a transcript is untouched (and untranscribed)."""
import services.asr_backend as ab
from core.db import db_conn
pid = f"vp-rts-{uuid.uuid4().hex[:8]}"
_insert_profile(pid, ref_text="the user's own words")
try:
fake = _make_fake_engine()
monkeypatch.setitem(_tts_mod()._REGISTRY, fake.id, fake)
fake.calls.clear()
counting = _CountingTranscribe()
monkeypatch.setattr(ab, "transcribe_reference", counting)
_generate(client, pid, fake)
assert counting.calls == 0
assert _profile_ref_text(pid) == "the user's own words"
finally:
with db_conn() as conn:
conn.execute("DELETE FROM generation_history WHERE profile_id=?", (pid,))
conn.execute("DELETE FROM voice_profiles WHERE id=?", (pid,))
def test_locked_profile_transcript_not_persisted(client, monkeypatch, locked_profile):
"""Locked profiles still transcribe (their audio is the locked take) but
never persist unlocking must not pair that transcript with the original
reference clip."""
import services.asr_backend as ab
fake = _make_fake_engine()
monkeypatch.setitem(_tts_mod()._REGISTRY, fake.id, fake)
fake.calls.clear()
counting = _CountingTranscribe(result="locked take words")
monkeypatch.setattr(ab, "transcribe_reference", counting)
_generate(client, locked_profile, fake)
assert counting.calls == 1
assert _profile_ref_text(locked_profile) == ""
+140
View File
@@ -0,0 +1,140 @@
"""Model LOAD time must not eat the GENERATE timeout budget (#1033/#1037 class).
Field evidence (#1014, measured on a Tesla T4): a fresh install's first
TTS request spent its entire OMNIVOICE_GENERATE_TIMEOUT_S window (300s)
downloading the multi-GB checkpoint 0% GPU utilization throughout and
died with the "too heavy for the available compute" 503. Two user reports
(#1033, #1037) match the signature. The generate guard was wrapping the
adapter's lazy `_ensure_loaded()` (weight download included) together with
the actual synthesis.
The fix gives loading its own, larger budget (OMNIVOICE_MODEL_LOAD_TIMEOUT,
default 1200s) via the new `TTSBackend.ensure_ready()` hook, dispatched
BEFORE the generate clock starts, in both /generate's adapter path and
/v1/audio/speech. These tests drive the class with a fake backend whose
"load" is slower than a tiny generate budget but inside the load budget
fail-before (GpuJobTimeoutError from the generate guard), pass-after.
Engine-stub pattern from tests/test_agentic_provider_contract.py.
"""
import os
import time
os.environ.setdefault("OMNIVOICE_MODEL", "test")
os.environ.setdefault("OMNIVOICE_DISABLE_FILE_LOG", "1")
import importlib
import pytest
import torch
def _tts_mod():
return importlib.import_module("services.tts_backend")
def _make_slow_loading_engine(engine_id, load_seconds):
class _SlowLoad(_tts_mod().TTSBackend):
id = engine_id
display_name = "Slow-Loading Engine (test)"
load_calls: list = []
def __init__(self):
self._loaded = False
@property
def sample_rate(self) -> int:
return 24000
@property
def supported_languages(self) -> list[str]:
return ["multi"]
@classmethod
def is_available(cls):
return True, "ready"
def _ensure_loaded(self):
if not self._loaded:
type(self).load_calls.append(time.monotonic())
time.sleep(load_seconds) # stands in for the weight download
self._loaded = True
def generate(self, text, **kw) -> torch.Tensor:
self._ensure_loaded()
return torch.zeros(1, 2400)
return _SlowLoad
@pytest.fixture()
def client():
from fastapi.testclient import TestClient
from main import app
return TestClient(app, client=("127.0.0.1", 50000))
def test_base_ensure_ready_dispatches_to_lazy_loader():
eng = _make_slow_loading_engine("slow-hookcheck", 0.01)()
assert not eng._loaded
eng.ensure_ready()
assert eng._loaded
def test_speech_survives_a_load_slower_than_the_generate_budget(client, monkeypatch):
"""The #1033/#1037 class, end to end: load (0.8s) > generate budget
(0.2s) but < load budget must succeed. Before the fix the generate
guard killed the request mid-'download' with the misleading 503."""
import services.model_manager as mm
import api.routers.openai_compat as oc
fake_cls = _make_slow_loading_engine("slow-load-engine", 0.8)
monkeypatch.setitem(_tts_mod()._REGISTRY, "slow-load-engine", fake_cls)
# Tiny generate budget; generous load budget — the exact asymmetry that
# used to be impossible because both ran on one clock.
monkeypatch.setattr(mm, "GPU_JOB_TIMEOUT_S", 0.2)
monkeypatch.setattr(oc, "run_on_gpu_pool_guarded", _patched_guard(mm, 0.2))
res = client.post("/v1/audio/speech", json={
"model": "slow-load-engine", "input": "Cold start.", "response_format": "wav",
})
assert res.status_code == 200, res.text
assert res.content[:4] == b"RIFF"
assert len(fake_cls.load_calls) == 1 # loaded exactly once, under the load budget
def _patched_guard(mm, generate_timeout):
"""run_on_gpu_pool_guarded with the module-default timeout shrunk, but
explicit `timeout=` (the load-budget call) respected mirrors how the
real default flows from GPU_JOB_TIMEOUT_S at call time vs. definition
time (the module default binds at def, so monkeypatching the constant
alone doesn't reach it)."""
real = mm.run_on_gpu_pool_guarded
async def _guard(fn, *, what="GPU job", timeout=None, executor=None):
return await real(
fn, what=what,
timeout=generate_timeout if timeout is None else timeout,
executor=executor,
)
return _guard
def test_speech_load_exceeding_load_budget_gets_the_load_error(client, monkeypatch):
"""A genuinely stalled download still fails — but with the load-specific
503 pointing at Settings Models, not the 'too heavy for compute' text."""
import services.model_manager as mm
import api.routers.openai_compat as oc
fake_cls = _make_slow_loading_engine("stalled-load-engine", 5.0)
monkeypatch.setitem(_tts_mod()._REGISTRY, "stalled-load-engine", fake_cls)
monkeypatch.setattr(mm, "_model_load_timeout", lambda: 0.2)
res = client.post("/v1/audio/speech", json={
"model": "stalled-load-engine", "input": "Stalled.", "response_format": "wav",
})
assert res.status_code == 503, res.text
detail = res.json()["detail"]
assert "model-load budget" in detail
assert "Settings → Models" in detail
assert "too heavy" not in detail
+62
View File
@@ -11,10 +11,16 @@ These tests pin the three guards:
Pure tests over a synthetic vocals wav no model, no main import.
"""
import asyncio
import threading
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import TimeoutError as FuturesTimeout
import numpy as np
import pytest
import soundfile as sf
from services.asr_backend import ASRTimeoutError, run_transcribe_guarded
from services.speaker_clone import (
ADJACENT_TURN_GUARD_S,
MIN_REF_DURATION_S,
@@ -206,3 +212,59 @@ class TestRefineRefTexts:
refine_ref_texts(clones, asr)
assert clones["Speaker 1"]["ref_text"] == "kept on failure"
assert clones["Speaker 2"]["ref_text"] == "buenos dias"
class _HangingASR:
"""An ASR backend whose .transcribe() *wedges* (blocks) instead of raising
the #730 whisperx/CTranslate2 hang. `refine_ref_text`'s try/except only
catches a raised Exception, so on its own this dispatch has no wall-clock
bound; it must go through the same `run_transcribe_guarded` every other
transcribe in dub_core.py uses."""
def __init__(self):
self.started = threading.Event()
self.release = threading.Event()
def transcribe(self, path, *, word_timestamps=True):
self.started.set()
self.release.wait() # blocks until the test releases it
return {"chunks": [{"text": "arrived too late"}]}
class TestRefineWedgeIsGuarded:
# Issue #730 class: a re-transcribe can hang rather than raise. The other
# transcribe dispatches in dub_core.py bound this via run_transcribe_guarded
# (chunk loop + whole-file "Dub"); the clone/segment refine dispatches must
# too, or a wedge holds the 1-worker GPU pool forever ("can't reach backend").
def test_refine_ref_texts_alone_has_no_wall_clock_bound(self):
# Proves the gap: dispatched raw (as #1008 did), a wedged transcribe
# never returns — refine_ref_text's except can't catch a hang.
asr = _HangingASR()
clones = {"S1": {"ref_audio": "/tmp/a.wav", "ref_text": "orig"}}
pool = ThreadPoolExecutor(max_workers=1)
fut = pool.submit(refine_ref_texts, clones, asr)
assert asr.started.wait(timeout=2.0)
with pytest.raises(FuturesTimeout):
fut.result(timeout=0.3) # still blocked — no internal bound
asr.release.set() # let the worker unwind before teardown
pool.shutdown(wait=False)
def test_run_transcribe_guarded_bounds_the_wedge_and_falls_back(self):
# Proves the fix: routing the same call through the guard bounds the
# hang, raises ASRTimeoutError, and the original ref_text is preserved
# (matching refine_ref_text's "failure is a strict no-op" fallback).
asr = _HangingASR()
clones = {"S1": {"ref_audio": "/tmp/a.wav", "ref_text": "orig"}}
pool = ThreadPoolExecutor(max_workers=1)
async def _go():
with pytest.raises(ASRTimeoutError):
await run_transcribe_guarded(
pool, lambda: refine_ref_texts(clones, asr),
what="Dub clone ref-text refine", timeout=0.3,
)
asyncio.run(_go())
assert clones["S1"]["ref_text"] == "orig" # fallback kept the original
asr.release.set()
pool.shutdown(wait=False)
+111 -1
View File
@@ -1,4 +1,5 @@
"""Voice-clone reference transcription goes through the ASR registry (#308).
"""Voice-clone reference transcription goes through the ASR registry (#308)
and caches transcripts by audio content (#1032).
A transcript-less reference used to fall through to OmniVoice's built-in
transformers pipeline (`load_asr_model`), which cannot load
@@ -6,7 +7,14 @@ whisper-large-v3-turbo on transformers 5.3 — even when whisperx /
faster-whisper / mlx-whisper were installed and working. `transcribe_reference`
must use the active registry backend, and degrade to None (the model fallback)
rather than raise.
#1032: the registry hands back a FRESH backend instance per call, so every
transcribe was a full whisper model load on EVERY /generate whose reference
had no stored transcript. Same clip same transcript, so results are cached
by content hash; a repeat call must not touch the ASR backend at all.
"""
import pytest
from services import asr_backend as ab
@@ -28,6 +36,14 @@ class _FakeBackend(ab.ASRBackend):
return self._result
@pytest.fixture(autouse=True)
def _clean_transcript_cache():
"""Each test starts (and leaves) with an empty content-hash cache."""
ab._ref_transcript_cache.clear()
yield
ab._ref_transcript_cache.clear()
def test_uses_active_backend_text(monkeypatch):
monkeypatch.setattr(
ab, "get_active_asr_backend",
@@ -76,3 +92,97 @@ def test_empty_result_degrades_to_none(monkeypatch):
lambda **kw: _FakeBackend(result={"text": " "}),
)
assert ab.transcribe_reference("ref.wav") is None
# ── #1032: content-keyed transcript cache ────────────────────────────────────
class _CountingFactory:
"""Stands in for get_active_asr_backend; counts backend constructions —
the expensive per-call model load the cache must eliminate."""
def __init__(self, result=None, exc=None):
self.calls = 0
self._result = result
self._exc = exc
def __call__(self, **kw):
self.calls += 1
return _FakeBackend(result=self._result, exc=self._exc)
def test_same_content_transcribed_once(monkeypatch, tmp_path):
"""Two calls on the same bytes → ONE backend construction/transcribe.
Before #1032 every call rebuilt (and thus reloaded) the ASR backend."""
factory = _CountingFactory(result={"text": "hello there"})
monkeypatch.setattr(ab, "get_active_asr_backend", factory)
clip = tmp_path / "ref.wav"
clip.write_bytes(b"RIFF-fake-audio-bytes")
assert ab.transcribe_reference(str(clip)) == "hello there"
assert ab.transcribe_reference(str(clip)) == "hello there"
assert factory.calls == 1
def test_same_content_different_path_hits_cache(monkeypatch, tmp_path):
"""Ad-hoc clone uploads land in a NEW temp file per request — the cache
must key on content, not path."""
factory = _CountingFactory(result={"text": "same clip"})
monkeypatch.setattr(ab, "get_active_asr_backend", factory)
a = tmp_path / "upload-1.wav"
b = tmp_path / "upload-2.wav"
a.write_bytes(b"identical-bytes")
b.write_bytes(b"identical-bytes")
assert ab.transcribe_reference(str(a)) == "same clip"
assert ab.transcribe_reference(str(b)) == "same clip"
assert factory.calls == 1
def test_different_content_not_conflated(monkeypatch, tmp_path):
factory = _CountingFactory(result={"text": "some words"})
monkeypatch.setattr(ab, "get_active_asr_backend", factory)
a = tmp_path / "a.wav"
b = tmp_path / "b.wav"
a.write_bytes(b"clip-A")
b.write_bytes(b"clip-B")
ab.transcribe_reference(str(a))
ab.transcribe_reference(str(b))
assert factory.calls == 2
def test_failure_not_cached(monkeypatch, tmp_path):
"""A transient ASR failure must retry on the next request, not stick."""
clip = tmp_path / "ref.wav"
clip.write_bytes(b"bytes")
failing = _CountingFactory(exc=RuntimeError("model load failed"))
monkeypatch.setattr(ab, "get_active_asr_backend", failing)
assert ab.transcribe_reference(str(clip)) is None
assert failing.calls == 1
working = _CountingFactory(result={"text": "recovered"})
monkeypatch.setattr(ab, "get_active_asr_backend", working)
assert ab.transcribe_reference(str(clip)) == "recovered"
assert working.calls == 1
def test_unreadable_path_still_transcribes_uncached(monkeypatch):
"""No fingerprint (unreadable file) → transcribe every time, cache nothing.
Keeps the pre-#1032 behavior for anything the hash can't see."""
factory = _CountingFactory(result={"text": "words"})
monkeypatch.setattr(ab, "get_active_asr_backend", factory)
assert ab.transcribe_reference("does-not-exist.wav") == "words"
assert ab.transcribe_reference("does-not-exist.wav") == "words"
assert factory.calls == 2
assert len(ab._ref_transcript_cache) == 0
def test_cache_is_bounded(monkeypatch, tmp_path):
factory = _CountingFactory(result={"text": "words"})
monkeypatch.setattr(ab, "get_active_asr_backend", factory)
for i in range(ab._REF_TRANSCRIPT_CACHE_MAX + 5):
clip = tmp_path / f"clip-{i}.wav"
clip.write_bytes(f"clip-{i}".encode())
ab.transcribe_reference(str(clip))
assert len(ab._ref_transcript_cache) == ab._REF_TRANSCRIPT_CACHE_MAX
Generated
+1 -1
View File
@@ -3207,7 +3207,7 @@ wheels = [
[[package]]
name = "omnivoice"
version = "0.3.14"
version = "0.3.15"
source = { editable = "." }
dependencies = [
{ name = "accelerate" },