Palash DebnathandClaude Opus 5
41722afe3b
* refactor(launchpad): quieter, borderless design refresh The launchpad carried decoration from an earlier direction: icon chips, corner-hung count badges, a permanently visible filled arrow, uppercase mono card titles, and a dotted stipple divider — plus a frame that had been invisible since the app-wide border tokens were zeroed. Rework it around what the borderless direction actually implies: - Feature tiles get a whisper-faint surface instead of a dead frame, and read as three bands (bare glyph + count / title + arrow / description). `--card-hue` is spent sparingly — the glyph at rest, the surface, count and arrow only once raised. Titles move to sans sentence case; counts are plain tabular numerals. Lift softened 4px -> 2px, coloured glow -> neutral shadow, plus an explicit focus ring and a staggered entrance. - Hero drops the boxed "646" pill and the filled A/B-Compare button for quiet type, with a hairline standing in for the separation. - Section labels trade the dotted stipple for a single fading hairline; rows are transparent until hover and reveal "Open" on hover/focus (it stays in the DOM, so AT and keyboard always reach it). - Hero, tiles, recent files, callout and project lists now share one 1180px column — previously only the top half was capped, so lists ran edge-to-edge on a wide display while the deck stayed centred. Two bugs found and fixed while doing it: - Buttons that had `border border-solid border-transparent` removed fell back to the UA default border and rendered a visible 1px outline. They now carry `border-0` explicitly. - `.lp-animate` used `animation-fill-mode: both`, so after the entrance it kept owning `transform` — and animation-origin declarations outrank normal ones, which silently killed the card hover lift. Now `backwards`, which still holds the from-state through the stagger delay. Also drops CSS the page has not rendered since #904: the cursor-spotlight layer, the breath ring, and the per-card waveform strip. Verified with headless renders at 1600/1280/940 and the empty state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(dictation): decode Wayland portal signals and show the capture pill The GlobalShortcuts portal declares Activated/Deactivated as (o session, s shortcut_id, t timestamp, a{sv} options). We decoded the timestamp as u32, so zbus rejected every signal with Signature mismatch: got `(osta{sv})`, expected `(osua{sv})` and the press was dropped as an invalid signal. Registration succeeded and the desktop even reported the bound chord back, so the hotkey looked wired up while doing nothing at all — on every Wayland compositor, for the whole life of the feature (#1490). Decode the 64-bit timestamp, and keep the 32-bit spelling as a fallback so a non-conforming portal degrades to working rather than to silence. With presses arriving, the second half of the failure showed: nothing had shown the widget window since it became a hidden recorder host, so a capture ran with no pill on screen — and a mic or Accessibility failure rendered into a window nobody could see. Add show_dictation_pill, which bottom-centres the capsule on the monitor under the pointer and shows it without taking focus (Windows keeps SW_SHOWNOACTIVATE so paste still lands in the user's document), and call it from the widget for every state but idle. Wayland denies clients their own placement, so the compositor picks the spot there; the pill still appears. dispatch_dictation_capture now logs whether a press was emitted or queued — a press that reaches Rust and produces nothing was otherwise indistinguishable from one the compositor never delivered. Tests: portal signals decode at both timestamp widths (the 64-bit case fails before this change with the exact production error); pill placement centres, respects a second monitor's origin, and clamps rather than going off-screen; the widget shows for a state needing the user, stays hidden while idle, and never shows for a press that arrives while dictation is disabled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: sync in-progress workspace changes Uncommitted work already in the tree, checkpointed so the branch matches the local machine: - Remote GPU workers: join-from-the-app flow, one-time secrets, QR join codes, a Compute control in the status bar, and the device-list Workers panel (#1516) - Model Catalogue workspace, with Settings pointing at it - Settings sidebar search and keyboard navigation - Demo assets for dubbing, dictation and voice design, plus the scripts that render them - Backend: validation-error handling, ASR request-path degradation, and the accompanying tests - CHANGELOG entries for the above and for the Wayland dictation fix Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): follow Engines to the Model Catalogue, and green the sweep - test_supertonic3 asserted the license gate points at "Settings" while the engine now names Model Catalogue → Engines, which is where the accept button actually lives. The assertion follows the move; what it pins is unchanged — the hint must name a place the user can reach it. - Carries the CJK allowlist entries for the rendered dub bundle (#1517) and the regenerated route snapshot for /workers/agent (#1516), both of which this branch inherits from the workspace sync. - docs/install/linux.md: the dictation capsule is bottom-anchored everywhere except Wayland, where the protocol gives applications no say in their placement. Documented rather than left as a surprise (CodeRabbit). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: stop a flaky dependency fetch from failing green runs en-core-web-sm resolves to a direct GitHub release URL, and github.com intermittently answers `http2 error: refused stream before processing any application logic`. uv's own three retries all land within the same few seconds and fail together, so the whole job dies on a dependency that has nothing to do with the change under test — it cost #1518 and #1517 an otherwise-green run tonight. Two changes: back off between whole `uv sync` attempts, which is what actually clears it, and pass --no-sync to the pytest steps. `uv run` re-resolves the environment before running, so every test step was a fresh chance to hit the same fetch even though the install step had already synced — that is exactly how #1518 failed, in the isolated backend/tests step, with all 5467 tests already passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: one retry seam for every uv sync, not just the job that failed last en-core-web-sm resolves to a direct GitHub *release* URL rather than a package index, and github.com intermittently answers `http2 error: refused stream before processing any application logic`. uv's own retries all land inside the same ~10 seconds and fail together, so a job dies on a dependency unrelated to the change under test. Tonight that cost four otherwise-green runs across #1515, #1517 and #1518 — and the first fix only covered the Tests job, so the next failure simply moved to Smoke (Linux), which syncs separately. The fetch is per-job, so the fix has to be per-job: scripts/uv-sync-retry.sh backs off between whole attempts (15s, 45s, 90s) and every workflow that syncs now goes through it — ci.yml (tests + the platform matrix), release.yml, security.yml, evals.yml. It still fails loudly after four attempts, so a genuinely broken lockfile is not disguised as a flake. The Tests job also lacked the UV_HTTP_TIMEOUT / UV_HTTP_RETRIES the smoke matrix has always set, which is part of why it was the one that kept dying; it has them now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(ci): pin the Intel-Mac contract by intent, not by command spelling test_ci_verifies_intel_mac_as_the_documented_remote_only_host asserted the literal line `run: uv sync --extra pockettts`, so routing every sync through scripts/uv-sync-retry.sh read as a broken Intel-Mac contract. The contract it exists to protect is that the pockettts extra installs ONLY on backend_supported legs — which the regex now pins, while leaving how the sync is invoked free to change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: keep every uv run out of the resolver, and bound the retry budget CodeRabbit, #1517: - `uv run` re-resolves before running, so the smoke suite, the worker-artifact tests, the release test run and the eval run were each a fresh chance to hit the flaky direct-URL fetch outside the retry loop. All of them pass --no-sync now; the environment is already synced by the step that owns the retries. security.yml's `uv run --with pip-audit` is deliberately left alone — it layers an ephemeral package rather than running the project's own tests. - The retry count multiplied uv's own budget (UV_HTTP_RETRIES=5 with a 120 s timeout on the smoke matrix). Three attempts and 60 s of total backoff outlast the refusals actually observed while staying well inside the jobs' timeout-minutes. - The Intel-Mac contract test pinned the smoke command literally too, so --no-sync tripped it exactly like the sync line did. Same fix: assert the contract (smoke runs only on backend_supported legs), not its spelling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
12 KiB
Expressive speech: breaths, laughter, and style
How to direct a performance — laughter, sighs, pauses, whispering, emotion, and the community favorite: "how do I make it take a sharp audible breath, like a person running out of breath?" Some of this is supported today (engine-dependent), some is spec'd but not shipped yet. This page tells you exactly which is which, so you don't burn an evening on tags an engine ignores.
The short version
| You want | Do this | Works on |
|---|---|---|
| A pause | Type [pause], [pause 500ms], or [pause 1.5s] in the text |
Every engine |
| Laughter or a sigh | ⊕ Insert → [laughter] / [sigh] |
Default engine (VoiceStudio) |
| An audible breath on demand | [breath] in the text |
CosyVoice 3 only (opt-in) — see Breaths |
| Whispering | Style → whisper (the voice-design/style field) |
Default engine |
| Emotion ("excited", "sad", graded intensity) | IndexTTS2's emotion controls — Audiobook tab's Production Overrides, or the /ws/tts API — or CosyVoice 3 instruct |
Opt-in engines only |
| The same take again | Pin the seed / lock the profile | Default engine |
Why bracket tags work at all (and when they don't)
Everything you type in the text box reaches the active engine verbatim — the pipeline goes out of its way not to break tags:
- The text-normalization pass (numbers, abbreviations) skips every
[…]span (backend/services/text_normalization.py). - The long-text chunker never cuts inside a bracket tag
(
backend/services/chunked_tts.py,_BRACKET_TAG_RE).
The flip side is just as important: unrecognized tags are not stripped.
An engine that doesn't know a tag receives it as literal text and will try to
speak it. Pasting an ElevenLabs script full of [excited] / [whispers]
degrades output on every engine we ship — those tags are on the roadmap (see
What's coming), not in the engines. Use only the tags listed
for your engine below.
What each engine can do today
Every engine
[pause Nms]markers —[pause](350 ms default),[pause 500ms],[pause 1s], up to 10 s. Rendered as real stitched silence, so it works identically on all engines.- Punctuation — ellipses, dashes, exclamation marks, and short fragments genuinely shape pacing and intonation. Cheap, underrated.
- The reference clip is a performance direction. Zero-shot cloning mirrors the delivery of the reference, not just the timbre — a flat reference clones flat, an animated one clones animated (see the tip in generation-parameters.md). This is the most reliable expressive control in the app.
- Pronunciation overrides —
[[Nuh-VAD-uh]]inline, or the pronunciation dictionary. Not expression, but often what a "it says this weirdly" problem actually needs.
Default engine (VoiceStudio)
Non-verbal tags. The bundled model natively tokenizes 13 reaction tags
(omnivoice/models/omnivoice.py, _NONVERBAL_PATTERN) — the ⊕ Insert button
at the corner of the Script box lists them all:
[laughter] [sigh] [confirmation-en] [question-en] [question-ah]
[question-oh] [question-ei] [question-yi] [surprise-ah]
[surprise-oh] [surprise-wa] [surprise-yo] [dissatisfaction-hnn]
Honest expectations: [laughter] and [sigh] are the broadly useful ones;
most of the interjection variants (-ah, -yi, -hnn) are tuned for
Mandarin-flavored speech. How convincingly a tag renders varies with the
voice — a tag that lands great on one reference clip can come out subdued on
another. There is no intensity control, and no [breath] tag in this
set.
Whispering. whisper is the one delivery style the instruct validator
accepts (the taxonomy is Gender / Age / Pitch / Style / Accent / Dialect —
see voice-design.md). [happy] / [sad]-style emotion
direction is not something the base model takes.
Sampling knobs + seed. The Voice workspace and the Audiobook tab each carry a Production Overrides panel exposing the sampling surface (defaults in parentheses; details in generation-parameters.md):
position_temperature(5.0) andclass_temperature(0.0) — 0 is greedy; higher is more random, which means more expressive variation and more artifacts.num_step— the Voice page defaults to 16 (fast); Audiobook renders default to 32 (cleaner), overridable in the Audiobook panel. Fewer steps = rougher, occasionally more "human-sounding" edges.- Seed — unpinned by default, so every render differs. The history rail shows the seed each take used; "Keep this seed" (Design tab) or locking a profile from history pins reference + seed, making the voice bit-reproducible. The Audiobook panel also takes a book-level seed directly.
postprocess_output(on) — removes long silences from the output. Turn it off when the silence is the performance.
The Audiobook panel adds two longform-only controls on top of that surface: IndexTTS2 emotion (see below), and Vary repeated lines — a cache opt-out that gives every identical line its own take instead of replaying one recording (off by default, so books stay byte-reproducible unless you ask for variety).
Longform-only tags. Audiobook and Stories additionally parse SSML-lite —
[slow]…[/slow], [fast]…[/fast], [emphasis]…[/emphasis], [spell] —
plus [voice:NAME] for multi-voice scripts
(backend/services/longform_parser.py). These are not parsed on the Voice
page.
CosyVoice 3 (opt-in)
The most direct paralinguistic control in the app, if you're willing to install it. CosyVoice 3 honors, inline in the text:
[breath]— an audible breath, exactly where you put it[laughter]<strong>word</strong>— emphasis
plus natural-language instruct ("speak with a Sichuan accent", "sound
exhausted") — the backend appends the model's required <|endofprompt|>
terminator for you (backend/services/tts_backend.py,
CosyVoiceBackend). One catch: the Studio style field whitelists instruct to
the default engine's taxonomy, so free-text instruct currently needs the API
(POST /generate with an instruct form field, or /ws/tts).
Setup: clone + install CosyVoice
(non-trivial: git clone --recursive, its requirements, SoX), then set
OMNIVOICE_COSYVOICE_MODEL to the model directory and select it in
Model Catalogue → Engines. CUDA or CPU; MPS is unverified upstream.
VoxCPM2 (opt-in)
VoxCPM2's native convention is an instruct prefix inside the text itself:
(speaking fast, out of breath) I can't stop now. The app maps the
instruct field onto that prefix (backend/services/tts_backend.py,
VoxCPM2Backend.generate), and because the convention is literally in-text,
typing the parenthetical at the start of your text works too. Treat it as
guidance, not a guarantee — adherence varies by voice and language.
IndexTTS2 (opt-in)
The only engine with graded emotion control: an 8-value emotion vector
(happy, angry, sad, afraid, disgusted, melancholic, surprised, calm), an
emotion reference clip whose delivery is mimicked (with a blend strength),
or a natural-language emotion description. The Audiobook tab's Production
Overrides expose the natural-language path — an Emotion description field
plus an Emotion strength slider (emo_alpha), shown only when IndexTTS2 is
the active engine so there are no dead controls. The full surface (the 8-float
emo_vector, an emo_audio reference clip) is on the streaming WebSocket API
(/ws/tts — emo_vector, emo_audio, emo_alpha, emo_text fields;
backend/api/routers/tts_stream.py). The single-shot Voice page does not carry
emotion controls yet.
Breaths, specifically
The honest answer to "how do I invoke a sharp inhale on demand?":
On the default engine — you can't yet, not directly. There is no
[breath] or [inhale] token in its tag set; [sigh] is the nearest
neighbor and it's an exhale. An engine-agnostic breath/reaction tag layer is
spec'd (specs/01-expressive-tts.md) but not
shipped — see below.
The direct route: CosyVoice 3. Its [breath] tag puts an audible breath
exactly where you type it. If on-demand breaths matter to your work, this is
the supported path today.
The coax-it recipe (default engine). Breaths can be elicited — this is exactly what v0.3.9 was doing by accident. Roughly in order of effectiveness:
- Put the breathing in the reference clip. Record 8–15 s of yourself (or your speaker) genuinely winded — audible inhales between phrases. The clone mirrors the delivery. This alone gets most of the way there.
- Write for it. Short gasping fragments with pauses:
I can't… [pause 300ms] I can't keep… [pause 200ms] keep running. - Turn off
postprocess_output(Production Overrides — Voice tab or Audiobook tab) so the silences — where breath artifacts live — aren't trimmed away. - Add randomness, then farm takes. Raise
class_temperatureto 0.3–0.7 (default is 0, fully greedy) and regenerate a few times — the seed is unpinned, so each take differs. Both controls live in the same Production Overrides panel, so this whole recipe now works for an audiobook chapter (audition it with the per-chapter Preview button), not just a single Voice render. In a book, the Vary repeated lines toggle farms takes across repeated lines automatically. - Keep the winner. When a take breathes right, its seed is on the history entry — lock the profile from there (or set the Audiobook panel's book-level seed) and every future generation uses the same reference + seed.
Tradeoffs, stated plainly: temperature cuts both ways (the same randomness that produces a great gasp produces slurred words and timbre drift), takes are non-repeatable until you pin the seed, and postprocess-off keeps all long silences, wanted or not. This is a workaround, not a feature — which is why the feature is spec'd.
Why v0.3.9-style random breaths faded
Users of v0.3.9 remember outputs that would spontaneously breathe, gasp, and
rustle — and noticed v0.3.15+ is smooth. Those breaths were never a feature:
they were uncontrolled sampling variance (unpinned seed + the default
position_temperature of 5.0) surviving an output chain that was, at the
time, less tidy. Then the chain got deliberately cleaner:
- v0.3.12 — the mastering pre-stage was cut down to highpass + compressor after a field report of hidden echo; every generation had been getting a small room reverb baked in, which made outputs sound roomier and breathier (#986).
- Silence post-processing (
postprocess_output, default on) removes long silences — the gaps where stray breath noise lived. - v0.3.16 — VoxCPM2 reference clips get edge-silence trimming before conditioning, and outputs get a trailing-silence trim (#1055), so that engine stopped inheriting dead air and its artifacts.
Net effect: the default output is now clean by design, and expressiveness is becoming something you ask for (tags, instruct, the recipe above) rather than something that happens to you.
What's coming
Spec 01 — Expressive TTS defines the plan: one
engine-agnostic tag surface ([excited], [whispers], reaction tags like
[breath]) that lowers to whatever the active engine can really do and
visibly degrades where it can't, plus an Expression panel (emotion
dropdown + intensity + emotion-reference clip) everywhere in the UI. The
pronunciation phases have shipped (dictionary + [[…]] overrides), and the
Audiobook tab now carries a first Expression surface — IndexTTS2 emotion
description + strength, engine-gated. Still spec'd, not shipped: the
engine-agnostic inline emotion/reaction tag grammar, the emotion-reference
clip picker, and the Expression panel on the single-shot Voice page. No
promised date — when each lands, this page gets updated in the same PR.