upstream-archive/VoiceStudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: cross-platform dictation delivery (#1610)

Browse Source 
Makes dictation delivery, capture, recovery, model fallback, AEC, and localized status behavior reliable across macOS, Windows, and Linux.
This commit is contained in:
Palash Debnath
2026-08-21 03:33:50 +00:00
committed by GitHub
parent 0687e13b57
commit 7718a7a10b
65 changed files with 5999 additions and 1029 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+11 -3
View File
@@ -10,12 +10,14 @@ the frozen-backend fallback mirror it for their toolchains.
**Highlights**
- The backend now answers within a second of launch and narrates its startup step by step
- Reporting a bug from an outdated build now offers the latest release first
- The backend is only announced ready once it can actually serve, and crash-loop restarts now pace themselves
- Dictation now stays bound to the app where it started and recovers locally from silent recognizer output (#1175)
- The backend now answers within a second of launch and narrates its startup step by step (#1550)
- Reporting a bug from an outdated build now offers the latest release first (#1547)
- The backend is only announced ready once it can actually serve, and crash-loop restarts now pace themselves (#1548)
- Invisible watermarking no longer stalls — or silently skips — the first take of a session (#1615)
### Changed
- Dictation now carries one native output session from shortcut-down through final delivery, restores text, HTML, image, or file-list clipboards only when untouched, keeps Wayland copy-safe unless current-focus insertion is explicitly enabled, and retries silent Sherpa speech only through an already-installed local ASR model (#1175)
- The backend binds its port immediately and reports startup progress live — `/health` answers 503-with-step and a new `/startup/progress` endpoint lists every step while PyTorch, API routes, and database migrations load in the background, so "starting at step X" is never mistakable for "dead"; the desktop splash narrates each step (#1550)
### Added
@@ -37,6 +39,12 @@ the frozen-backend fallback mirror it for their toolchains.
- The OmniVoice guide now covers combining style attributes with a reference clip (consistent instruct stabilizes cloning; the reference wins conflicts), inline pronunciation control (pinyin / CMU phonemes), and corrects the claim that the default engine can't do voice design — it can, from attributes (#1565)
### Fixed
- Dictation on a WebView that refuses a 16 kHz audio context (WKWebView) now low-passes before downsampling, so frequencies above 8 kHz stop folding into the speech the recognizer is fed (#1610)
- A microphone context that cannot be resumed now reports a mic error instead of leaving the dictation pill on "Listening" while capturing nothing (#1610)
- Dictation no longer retains a whole session's audio for silent-model recovery — an open mic grew that buffer by ~115 MB an hour; the recent two minutes are kept instead (#1610)
- The clipboard-delivery status is now translated in all 21 languages, so Wayland users — where clipboard delivery is the default — no longer see an English string (#1610)
- A native sherpa-onnx load failure of any exception type now degrades to "engine unavailable" instead of taking the dictation WebSocket down (#1610)
- Dictation now ships Whisper Tiny as its one cross-platform default, avoiding Parakeet's measured empty decoding on Windows while keeping Parakeet selectable behind runtime fallback (#1175)
- Re-mixing a dub no longer decodes, rewrites, and re-reads every cached segment — same-rate cached audio is reused directly (and rejected if truncated), switching timing modes can't reuse slot-truncated audio as natural-rate, and RVC respects natural-rate modes (#1594)
- PocketTTS French works again — pocket-tts only ships a 24-layer French model and rejected the name the sidecar asked for, so every French request failed at model load; French now always loads `french_24l` (#1613) — thanks @paoloantinori!
- Installing IndexTTS 2.5 no longer fails claiming an interrupted download — the weights repo ships `config.yaml` and VoiceStudio demanded a `config_v2_5.yaml` that exists in no upstream release; both names are accepted, so a hand-renamed checkout keeps working (#1611) — thanks @zuiaiyutu!
README.md
+1 -1
View File
@@ -103,7 +103,7 @@ Use `bun run dev` for the browser UI. See [Contributing](.github/CONTRIBUTING.md
| **Voice Design** | Create a voice from age, accent, pitch, style, and delivery instructions |
| **Video Dubbing** | Transcribe, translate, preserve speakers, synthesize, and export video |
| **Stories and audiobooks** | Multi-voice scripts · EPUB/PDF import · chapter rendering · `.m4b` export |
| **Dictation Widget** | System-wide shortcut, live transcription, optional local-LLM cleanup |
| **[Dictation Widget](docs/features/dictation.md)** | System-wide shortcut, live transcription, optional local-LLM cleanup |
| **Vocal Isolation** | Demucs speech/background separation |
| **Speaker Diarization** | Pyannote and WhisperX speaker assignment |
| **Batch Queue** | Queue large sets of audio and video jobs with per-job progress |
backend/api/routers/capture_ws.py
+253 -70
View File
@@ -27,6 +27,10 @@ Protocol:
"detail": "..."} — error ("detail"
kept for legacy)
Sherpa ``final`` frames additionally carry
``"final_kind": "utterance"|"summary"``. Utterances are mid-session
commits; the summary is the authoritative whole-session result at EOF.
Every ``final`` text is normalised by services.text_polish (leading
capital for Latin scripts, terminal punctuation, single-spaced) so the
pasted result reads like typed text. Partials are raw.
@@ -35,6 +39,7 @@ from __future__ import annotations
import asyncio
import logging
import math
import os
import tempfile
import time
@@ -70,17 +75,46 @@ _AEC_NEAR = 0x00 # microphone frame (clean it, then buffer for ASR)
_AEC_FAR = 0x01 # playback reference frame (feed the echo model only)
def _requested_pcm_sample_rate(query_params) -> int | None:
"""Return a bounded PCM rate for ``?pcm=1``/``?aec=1`` sessions."""
raw_pcm = query_params.get("pcm") in ("1", "true", "on")
aec = query_params.get("aec") in ("1", "true", "on")
if not raw_pcm and not aec:
return None
# Client-supplied ``?sr=`` values outside the range real capture devices use
# are replaced with 16 kHz. The rate sizes server-side state — RecoveryTail
# multiplies it by RECOVERY_TAIL_SECONDS to compute its byte ceiling — so an
# absurd rate must never be believed: it would re-open the unbounded-memory
# path the recovery-tail cap closed.
SR_MIN, SR_MAX = 8000, 96000
def _bounded_sample_rate(query_params) -> int:
try:
sample_rate = int(query_params.get("sr", "16000"))
except (TypeError, ValueError):
return 16000
return sample_rate if 8000 <= sample_rate <= 96000 else 16000
return sample_rate if SR_MIN <= sample_rate <= SR_MAX else 16000
def _requested_pcm_sample_rate(query_params) -> int | None:
"""Return the bounded rate when the client transport is raw PCM.
Sherpa clients omit ``pcm=1`` because the selected model already defines
that transport. If the model is demoted or its runtime is unavailable, the
legacy recognizer fallback must still decode those same bytes as PCM.
"""
raw_pcm = query_params.get("pcm") in ("1", "true", "on")
aec = query_params.get("aec") in ("1", "true", "on")
sherpa_pcm = False
requested_model = query_params.get("model")
if requested_model:
try:
from services.sherpa_dictation import is_sherpa_model
sherpa_pcm = is_sherpa_model(requested_model)
except Exception: # noqa: BLE001
# A broken sherpa install must not decide the framing question —
# sherpa_pcm stays False and the session negotiates the
# MediaRecorder path; availability is re-probed (and reported)
# when the model is actually selected.
sherpa_pcm = False
if not raw_pcm and not aec and not sherpa_pcm:
return None
return _bounded_sample_rate(query_params)
def _demux_aec_frame(data: bytes) -> tuple[str, bytes]:
@@ -137,16 +171,27 @@ def _select_sherpa_spec(websocket: WebSocket):
from services import sherpa_dictation as sd
except Exception:
return None
def _usable_spec(model_id):
spec = sd.get_spec(model_id)
if spec is not None and sd.is_demoted(spec.id):
logger.warning(
"dictation model %s is demoted — using the capture ASR fallback",
spec.id,
)
return None
return spec
requested = websocket.query_params.get("model")
if requested:
return sd.get_spec(requested) # explicit selection (may be None if bad)
return _usable_spec(requested) # explicit selection (may be unavailable)
# Fall back to the persisted dictation pref.
try:
from services.asr_backend import dictation_model_id
mid = dictation_model_id()
except Exception:
mid = None
return sd.get_spec(mid) if mid else None
return _usable_spec(mid) if mid else None
@router.websocket("/ws/transcribe")
@@ -422,6 +467,64 @@ SHERPA_OFFLINE_SILENCE_S = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_SILENC
SHERPA_OFFLINE_RMS_FLOOR = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_RMS", "0.01"))
#: Seconds of audio retained for silent-model recovery. Recovery only needs
#: enough speech to prove the model is broken and to re-transcribe what was
#: said; retaining the whole session grew ~115 MB/hour at 16 kHz on an open
#: mic, unbounded, and only ever got read when the fallback fired.
RECOVERY_TAIL_DEFAULT_SECONDS = 120.0
RECOVERY_TAIL_MAX_SECONDS = 300.0
def _bounded_recovery_tail_seconds(value: str | None) -> float:
"""Parse the recovery tail override without allowing unbounded buffers."""
try:
seconds = float(value) if value is not None else RECOVERY_TAIL_DEFAULT_SECONDS
except (TypeError, ValueError):
return RECOVERY_TAIL_DEFAULT_SECONDS
if not math.isfinite(seconds) or seconds <= 0:
return RECOVERY_TAIL_DEFAULT_SECONDS
return min(seconds, RECOVERY_TAIL_MAX_SECONDS)
RECOVERY_TAIL_SECONDS = _bounded_recovery_tail_seconds(
os.environ.get("OMNIVOICE_DICTATION_RECOVERY_TAIL_S")
)
class RecoveryTail:
"""The most recent ``RECOVERY_TAIL_SECONDS`` of session audio.
Keeps the *tail* rather than the head: a long dictation's useful speech is
what the user just said, and the silent-model check cares about how much
audio the session carried overall — which ``total_bytes`` still reports
truthfully after trimming.
"""
__slots__ = ("_buf", "_max", "total_bytes")
def __init__(self, sample_rate: int, seconds: float = RECOVERY_TAIL_SECONDS):
# int16 mono → 2 bytes/sample. Floor of one frame so a nonsense rate
# or seconds value can't produce a zero-length buffer.
self._max = max(2, int(seconds * max(1, sample_rate)) * 2)
self._buf = bytearray()
self.total_bytes = 0
def extend(self, pcm: bytes) -> None:
self._buf.extend(pcm)
self.total_bytes += len(pcm)
excess = len(self._buf) - self._max
if excess > 0:
# int16 mono: trim whole samples only. A split frame can carry an
# odd byte count, and an odd trim would leave the tail starting
# mid-sample — every later sample byte-shifted, and the recovery
# transcription fed noise.
excess += excess % 2
del self._buf[:excess]
def tail(self) -> bytes:
return bytes(self._buf)
def is_model_silent(text: str, heard_speech: bool, pcm_bytes: int) -> bool:
"""True when the dictation model produced NO text despite real speech.
@@ -448,19 +551,74 @@ def _pcm16_to_f32(pcm: bytes):
return np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
async def _sherpa_session(websocket: WebSocket):
"""Shared WS receive setup for the sherpa handlers.
def _pcm16_rms(pcm: bytes) -> float:
samples = _pcm16_to_f32(pcm)
if not len(samples):
return 0.0
return float((samples * samples).mean() ** 0.5)
Returns ``(get_frame, state)`` where ``get_frame`` is an async callable
that yields the next near-end (mic) PCM bytes, ``b""`` for a keepalive/ref
frame, or ``None`` on EOF/disconnect. ``state`` carries sample rate, AEC,
and the disconnect flag for the caller's finaliser.
"""
pcm_sr = 16000
async def _recover_silent_sherpa(
spec, pcm: bytes, pcm_sr: int,
) -> tuple[str, list[dict]]:
"""Retry a token-silent Sherpa session through an installed local ASR."""
logger.warning(
"dictation model %s decoded NOTHING from %.1fs of speech-level audio "
"— falling back to the capture ASR engine for this session",
spec.id, len(pcm) / float(max(1, pcm_sr) * 2),
)
try:
pcm_sr = int(websocket.query_params.get("sr", "16000"))
except (TypeError, ValueError):
pcm_sr = 16000
from services.asr_backend import asr_model_missing_error
fallback_missing = await asyncio.to_thread(
asr_model_missing_error,
purpose="dictation",
skip_sherpa=True,
require_installed=True,
)
if fallback_missing is not None:
logger.warning(
"dictation silent-model fallback is not installed (%s); "
"skipping recovery to avoid an automatic download",
fallback_missing.get("missing_repo_id", "unknown"),
)
return "", []
result = await _transcribe_buffer_full(
[pcm], pcm_sr=pcm_sr, skip_sherpa=True,
)
text = polish_text(_result_text(result))
if not text:
return "", []
# The RMS gate can fire on fan/keyboard noise. Only another recognizer
# producing words proves the audio held speech and makes persistent
# demotion safe.
try:
from services.sherpa_dictation import demote_model
if await asyncio.to_thread(demote_model, spec.id):
logger.error(
"dictation model %s demoted on this machine — it will no longer be "
"auto-selected. Pick it again in Settings to give it another chance.",
spec.id,
)
except Exception:
logger.exception("silent-model demotion failed")
segments = (result or {}).get("segments") or [
{"start": 0.0, "end": None, "text": text}
]
return text, segments
except Exception:
logger.exception("dictation silent-model fallback failed")
return "", []
async def _sherpa_session(websocket: WebSocket):
"""Shared WS setup for the sherpa handlers.
Returns ``(pcm_sr, aec)``: the bounded PCM sample rate for the session
and the echo canceller when ``?aec=1`` requested one (``None`` otherwise
or when AEC setup fails).
"""
pcm_sr = _bounded_sample_rate(websocket.query_params)
aec = None
if websocket.query_params.get("aec") in ("1", "true", "on"):
try:
@@ -569,6 +727,8 @@ async def _run_sherpa_streaming(websocket: WebSocket, spec):
last_partial = ""
committed: list[str] = [] # finalized utterances this session
session_pcm = RecoveryTail(pcm_sr) # bounded audio for silent-model recovery
heard_speech = False
client_disconnected = False
async def _send(payload) -> bool:
@@ -610,6 +770,9 @@ async def _run_sherpa_streaming(websocket: WebSocket, spec):
break
if kind == "skip":
continue
session_pcm.extend(pcm)
if not heard_speech and _pcm16_rms(pcm) >= SHERPA_OFFLINE_RMS_FLOOR:
heard_speech = True
text, endpoint = await asyncio.to_thread(_decode_after_feed, pcm)
if endpoint:
# Commit this utterance (polished — it gets pasted); reset
@@ -618,6 +781,7 @@ async def _run_sherpa_streaming(websocket: WebSocket, spec):
if text:
committed.append(text)
await _send({"type": "final", "text": text,
"final_kind": "utterance",
"segments": [{"start": 0.0, "end": None, "text": text}],
"language": "auto", "engine": backend.id})
rec.reset(stream)
@@ -644,7 +808,28 @@ async def _run_sherpa_streaming(websocket: WebSocket, spec):
# Pieces are already polished; the join is too (polish is idempotent).
full = " ".join(t for t in committed if t).strip()
segments = [{"start": 0.0, "end": None, "text": t} for t in committed if t]
model_silent = is_model_silent(full, heard_speech, session_pcm.total_bytes)
if model_silent:
recovered, recovered_segments = await _recover_silent_sherpa(
spec, session_pcm.tail(), pcm_sr,
)
if recovered:
full = recovered
segments = recovered_segments
if not client_disconnected:
payload = {"type": "final", "text": full, "final_kind": "summary",
"segments": segments,
"language": "auto", "engine": backend.id}
if model_silent:
payload["engine"] = "capture-asr-fallback" if full else backend.id
payload["model_silent"] = spec.id
payload["warning"] = (
f"The selected dictation model ({spec.id}) produced no text from your "
"speech. Switched to the fallback engine for this session — pick a "
"different model in Settings → Dictation."
)
if full:
# Hard-bounded refinement (~4s): never delays this summary `final`
# beyond OMNIVOICE_REFINE_TIMEOUT_S even with a dead LLM endpoint.
@@ -653,14 +838,9 @@ async def _run_sherpa_streaming(websocket: WebSocket, spec):
refined = await maybe_refine_async(full)
except Exception:
refined = None
payload = {"type": "final", "text": full, "segments": segments,
"language": "auto", "engine": backend.id}
if refined and refined != full:
payload["refined_text"] = refined
await _send(payload)
else:
await _send({"type": "final", "text": "", "segments": [],
"language": "auto", "engine": backend.id})
try:
await websocket.close()
except Exception:
@@ -697,7 +877,7 @@ async def _run_sherpa_offline(websocket: WebSocket, spec):
# whisper/zipformer transcribe the same bytes). Keep the whole session's
# audio and whether any of it was speech-level, so the finaliser can tell
# "user said nothing" (fine) from "model produced nothing" (broken).
session_pcm = bytearray()
session_pcm = RecoveryTail(pcm_sr)
heard_speech = False
running = True
client_disconnected = False
@@ -716,12 +896,6 @@ async def _run_sherpa_offline(websocket: WebSocket, spec):
client_disconnected = True
return False
def _rms(pcm: bytes) -> float:
samples = _pcm16_to_f32(pcm)
if not len(samples):
return 0.0
return float((samples * samples).mean() ** 0.5)
def _decode_window(pcm: bytes) -> str:
samples = _pcm16_to_f32(pcm)
if not len(samples):
@@ -740,7 +914,7 @@ async def _run_sherpa_offline(websocket: WebSocket, spec):
continue
buf.extend(pcm)
session_pcm.extend(pcm)
if not heard_speech and _rms(pcm) >= SHERPA_OFFLINE_RMS_FLOOR:
if not heard_speech and _pcm16_rms(pcm) >= SHERPA_OFFLINE_RMS_FLOOR:
heard_speech = True
last_audio = time.monotonic()
except WebSocketDisconnect:
@@ -766,6 +940,7 @@ async def _run_sherpa_offline(websocket: WebSocket, spec):
if text:
committed.append(text)
await _send({"type": "final", "text": text,
"final_kind": "utterance",
"segments": [{"start": 0.0, "end": None, "text": text}],
"language": "auto", "engine": backend.id})
@@ -777,8 +952,8 @@ async def _run_sherpa_offline(websocket: WebSocket, spec):
continue
snapshot = bytes(buf)
if len(snapshot) > sil_bytes and \
_rms(snapshot[-sil_bytes:]) < SHERPA_OFFLINE_RMS_FLOOR:
if _rms(snapshot[:-sil_bytes]) >= SHERPA_OFFLINE_RMS_FLOOR:
_pcm16_rms(snapshot[-sil_bytes:]) < SHERPA_OFFLINE_RMS_FLOOR:
if _pcm16_rms(snapshot[:-sil_bytes]) >= SHERPA_OFFLINE_RMS_FLOOR:
await _commit(snapshot)
else:
# Pure silence — drop it (keep the gate window for
@@ -824,39 +999,18 @@ async def _run_sherpa_offline(websocket: WebSocket, spec):
# quiet user — hand the session to the capture ASR backend so the user
# still gets their words, and say which model let them down. Bounded to
# this session; the pref is left alone so the user stays in control.
model_silent = is_model_silent(full, heard_speech, len(session_pcm))
model_silent = is_model_silent(full, heard_speech, session_pcm.total_bytes)
if model_silent:
logger.warning(
"dictation model %s decoded NOTHING from %.1fs of speech-level audio "
"— falling back to the capture ASR engine for this session",
spec.id, len(session_pcm) / float(max(1, pcm_sr) * 2),
recovered, recovered_segments = await _recover_silent_sherpa(
spec, session_pcm.tail(), pcm_sr,
)
# Demote it so the NEXT session doesn't repeat this round trip. The
# curated default can be broken on a platform we never tested (the
# NeMo-TDT decoder is, on Windows), and observing it beats guessing.
try:
from services.sherpa_dictation import demote_model
if demote_model(spec.id):
logger.error(
"dictation model %s demoted on this machine — it will no longer be "
"auto-selected. Pick it again in Settings to give it another chance.",
spec.id,
)
except Exception:
logger.exception("silent-model demotion failed")
try:
result = await _transcribe_buffer_full([bytes(session_pcm)], pcm_sr=pcm_sr)
fb_text = polish_text((result or {}).get("text", "") or "")
if fb_text:
full = fb_text
segments = (result or {}).get("segments") or [
{"start": 0.0, "end": None, "text": fb_text}
]
except Exception:
logger.exception("dictation silent-model fallback failed")
if recovered:
full = recovered
segments = recovered_segments
if not client_disconnected:
payload = {"type": "final", "text": full, "segments": segments,
payload = {"type": "final", "text": full, "final_kind": "summary",
"segments": segments,
"language": "auto", "engine": backend.id}
if model_silent:
# The client surfaces this so a silently-broken model can't look
@@ -884,6 +1038,35 @@ async def _run_sherpa_offline(websocket: WebSocket, spec):
pass
def _result_text(result: dict | None) -> str:
"""Normalize text from every ASR backend result shape.
Some backends return a top-level ``text`` value, while WhisperX, Faster
Whisper, Moonshine, and OpenAI-compatible ASR expose only ``segments`` and
``chunks``. Dictation partials and finals must interpret both contracts the
same way.
"""
if not isinstance(result, dict):
return ""
text = result.get("text")
if isinstance(text, str) and text.strip():
return text.strip()
for key in ("segments", "chunks"):
items = result.get(key)
if not isinstance(items, (list, tuple)):
continue
text = " ".join(
str(item.get("text", "")).strip()
for item in items
if isinstance(item, dict) and item.get("text")
).strip()
if text:
return text
return ""
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
"""Quick partial transcription of the current audio buffer."""
@@ -898,7 +1081,7 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
def _run():
backend = get_capture_asr_backend()
result = backend.transcribe(tmp, word_timestamps=False)
return result.get("text", "")
return _result_text(result)
# Bound dictation transcribes (#730): a wedged whisperx/CTranslate2 call
# must not hold its GPU-pool worker forever and starve TTS / other ASR
@@ -912,7 +1095,9 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
pass
async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict:
async def _transcribe_buffer_full(
chunks: list[bytes], *, pcm_sr: int | None = None, skip_sherpa: bool = False,
) -> dict:
"""Full transcription with timing info for the final result."""
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
if tmp is None:
@@ -924,15 +1109,13 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
def _run():
backend = get_capture_asr_backend()
backend = get_capture_asr_backend(skip_sherpa=skip_sherpa)
t0 = time.perf_counter()
result = backend.transcribe(tmp, word_timestamps=False)
elapsed = round(time.perf_counter() - t0, 2)
segments = result.get("segments", [])
full_text = result.get("text", "")
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
full_text = _result_text(result)
# Wave 1.1: strip Whisper hallucination loops from the final
# text (the string that gets auto-pasted). Segments keep the
backend/config/models.yaml
+10 -10
View File
@@ -159,17 +159,16 @@ models:
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8"
label: "Parakeet TDT v3 (sherpa-onnx — dictation, 25 EU langs)"
role: ASR
size_gb: 0.18
size_gb: 0.67
engine: sherpa-onnx
dictation_id: sherpa-parakeet-tdt-v3
tag: offline
curated_on: [all]
note: "Recommended live-dictation default. CPU, int8 ONNX. Requires sherpa-onnx."
note: "Multilingual European-language dictation. CPU, int8 ONNX. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8"
label: "Parakeet TDT v2 (sherpa-onnx — dictation, English)"
role: ASR
size_gb: 0.17
size_gb: 0.66
engine: sherpa-onnx
dictation_id: sherpa-parakeet-tdt-v2
tag: offline
@@ -178,7 +177,7 @@ models:
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20"
label: "Zipformer Bilingual (sherpa-onnx — streaming, zh+en)"
role: ASR
size_gb: 0.13
size_gb: 0.2
engine: sherpa-onnx
dictation_id: sherpa-zipformer-bilingual-zh-en
tag: streaming
@@ -187,7 +186,7 @@ models:
- repo_id: "csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en"
label: "Paraformer Bilingual (sherpa-onnx — streaming, zh+en)"
role: ASR
size_gb: 0.115
size_gb: 0.24
engine: sherpa-onnx
dictation_id: sherpa-paraformer-bilingual-zh-en
tag: streaming
@@ -196,7 +195,7 @@ models:
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17"
label: "Zipformer Streaming EN 20M (sherpa-onnx — streaming, English)"
role: ASR
size_gb: 0.128
size_gb: 0.044
engine: sherpa-onnx
dictation_id: sherpa-zipformer-en-20m
tag: streaming
@@ -205,7 +204,7 @@ models:
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23"
label: "Zipformer Streaming ZH 14M (sherpa-onnx — streaming, Chinese)"
role: ASR
size_gb: 0.074
size_gb: 0.025
engine: sherpa-onnx
dictation_id: sherpa-zipformer-zh-14m
tag: streaming
@@ -214,11 +213,12 @@ models:
- repo_id: "csukuangfj/sherpa-onnx-whisper-tiny"
label: "Whisper Tiny (sherpa-onnx — dictation, 90+ langs)"
role: ASR
size_gb: 0.116
size_gb: 0.104
engine: sherpa-onnx
dictation_id: sherpa-whisper-tiny
tag: offline
note: "Multilingual offline dictation (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx."
curated_on: [all]
note: "Recommended cross-platform dictation default (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx."
# ── Diarisation ───────────────────────────────────────────────────────
backend/services/asr_backend.py
+78 -15
View File
@@ -2995,7 +2995,7 @@ def _capture_prefers_parakeet() -> bool:
return _parakeet_mlx_installed()
def get_capture_asr_backend() -> ASRBackend:
def get_capture_asr_backend(*, skip_sherpa: bool = False) -> ASRBackend:
"""Pick the fastest ASR engine for capture / dictation.
Selection order:
@@ -3020,6 +3020,9 @@ def get_capture_asr_backend() -> ASRBackend:
Returns a cached singleton so the model stays warm between calls; the
singleton is rebuilt if the selected sherpa model changes.
``skip_sherpa`` is used only to validate a token-silent Sherpa result with
the installed capture fallback before persisting model demotion.
"""
global _capture_backend, _capture_backend_key
@@ -3028,7 +3031,7 @@ def get_capture_asr_backend() -> ASRBackend:
# call get_sherpa_dictation_backend concurrently) can't both build a model.
with _capture_backend_lock:
# 0. Honor an explicit sherpa dictation model selection.
sherpa_id = dictation_model_id()
sherpa_id = None if skip_sherpa else dictation_model_id()
if sherpa_id:
ok, _ = SherpaDictationBackend.is_available()
if ok:
@@ -3195,7 +3198,10 @@ def _capture_whisper_repo() -> str | None:
return os.environ.get("OMNIVOICE_PYTORCH_ASR_MODEL", _PYTORCH_ASR_DEFAULT)
def _recommended_asr_model(purpose: str, missing_repo: str | None) -> dict | None:
def _recommended_asr_model(
purpose: str, missing_repo: str | None, *, prefer_sherpa: bool = True,
excluded_sherpa_model_id: str | None = None,
) -> dict | None:
"""The catalog entry to offer in the download CTA.
Offline: the missing repo itself when it's in the catalog (guarantees
@@ -3215,20 +3221,38 @@ def _recommended_asr_model(purpose: str, missing_repo: str | None) -> dict | Non
by_id = {m["repo_id"]: m for m in KNOWN_MODELS}
exact = by_id.get(missing_repo) if missing_repo else None
want_sherpa = False
if purpose == "dictation":
if exact is not None and exact.get("engine") == "sherpa-onnx":
def _eligible(m: dict, *, sherpa: bool) -> bool:
if (m.get("engine") == "sherpa-onnx") != sherpa:
return False
if sherpa and m.get("dictation_id") == excluded_sherpa_model_id:
return False
return _model_supported(m)
if purpose != "dictation":
if exact is not None and _model_supported(exact):
return _shape(exact)
prefer_sherpa = False
if purpose == "dictation" and prefer_sherpa:
ok, _ = SherpaDictationBackend.is_available()
want_sherpa = ok
if not want_sherpa and exact is not None and _model_supported(exact):
if ok:
if exact is not None and _eligible(exact, sherpa=True):
return _shape(exact)
for m in KNOWN_MODELS:
if (m.get("role") == "ASR" and _eligible(m, sherpa=True)
and _model_curated(m)):
return _shape(m)
# No usable Sherpa recommendation remains (runtime unavailable, explicit
# fallback probe, or the sole curated entry is the demoted model). Offer
# the exact capture fallback so download → retry cannot loop.
if exact is not None and _eligible(exact, sherpa=False):
return _shape(exact)
for m in KNOWN_MODELS:
if m.get("role") != "ASR":
continue
if (m.get("engine") == "sherpa-onnx") != want_sherpa:
continue
if _model_curated(m) and _model_supported(m):
if _eligible(m, sherpa=False) and _model_curated(m):
return _shape(m)
return None
@@ -3259,7 +3283,9 @@ def _repo_installed(repo: str) -> bool:
def asr_model_missing_error(*, purpose: str = "transcribe",
sherpa_model_id: str | None = None,
backend_id: str | None = None) -> dict | None:
backend_id: str | None = None,
skip_sherpa: bool = False,
require_installed: bool = False) -> dict | None:
"""None when the active ASR selection can transcribe without downloading
anything; otherwise the typed ``{"error": "asr_model_missing", ...}``
payload for a 409 / SSE / WS error with a download CTA.
@@ -3271,6 +3297,11 @@ def asr_model_missing_error(*, purpose: str = "transcribe",
``?model=`` override. Installed state comes from the same HF-cache helpers
the model store uses (see :func:`_repo_installed`), so the answer matches
the Model Catalogue → Models install badges.
``skip_sherpa`` probes only the non-Sherpa capture fallback; silent-model
recovery uses it before deciding whether persistent demotion is warranted.
``require_installed`` makes unknown/custom selections fail closed for that
recovery path so it can never turn the normal fail-open policy into an
implicit model download.
FAIL-OPEN rule: a repo the model catalog doesn't know (a custom
``ASR_MODEL_*`` pin, pytorch-whisper's default repo, an unrecognized
@@ -3280,27 +3311,55 @@ def asr_model_missing_error(*, purpose: str = "transcribe",
a broken preflight must degrade to the old behaviour, not block ASR.
"""
try:
prefer_sherpa_recommendation = not skip_sherpa
excluded_sherpa_model_id = None
if purpose == "dictation":
sid = sherpa_model_id or dictation_model_id()
sid = None if skip_sherpa else (sherpa_model_id or dictation_model_id())
if sid:
ok, _ = SherpaDictationBackend.is_available()
if ok:
from services import sherpa_dictation as _sd
spec = _sd.get_spec(sid)
# A recognizer observed returning silence must follow the
# same capture fallback as execution, even when the
# frontend keeps sending its persisted `?model=` value.
if spec is not None:
if _sd.is_demoted(spec.id):
excluded_sherpa_model_id = spec.id
else:
if _sd.is_installed(spec):
return None
return {
"error": ASR_MODEL_MISSING,
"missing_repo_id": spec.repo_id,
"recommended": _recommended_asr_model(purpose, spec.repo_id),
"recommended": _recommended_asr_model(
purpose, spec.repo_id,
),
}
repo = _capture_whisper_repo()
else:
repo = _offline_asr_repo(backend_id)
if repo is None:
if require_installed:
return {
"error": ASR_MODEL_MISSING,
"missing_repo_id": "unresolved-capture-fallback",
"recommended": None,
}
return None # explicit opt-in engine — can't (and shouldn't) preflight
from api.routers.setup.models import get_model_catalog
if require_installed:
if _repo_installed(repo):
return None
return {
"error": ASR_MODEL_MISSING,
"missing_repo_id": repo,
"recommended": _recommended_asr_model(
purpose, repo,
prefer_sherpa=prefer_sherpa_recommendation,
excluded_sherpa_model_id=excluded_sherpa_model_id,
),
}
if get_model_catalog().get(repo) is None:
return None # not installable from the CTA — fail open (see docstring)
if _repo_installed(repo):
@@ -3308,7 +3367,11 @@ def asr_model_missing_error(*, purpose: str = "transcribe",
return {
"error": ASR_MODEL_MISSING,
"missing_repo_id": repo,
"recommended": _recommended_asr_model(purpose, repo),
"recommended": _recommended_asr_model(
purpose, repo,
prefer_sherpa=prefer_sherpa_recommendation,
excluded_sherpa_model_id=excluded_sherpa_model_id,
),
}
except Exception: # noqa: BLE001 — preflight is best-effort, never a blocker
logger.warning("ASR install preflight failed — proceeding without it",
backend/services/sherpa_dictation.py
+17 -10
View File
@@ -110,7 +110,7 @@ class SherpaModelSpec:
# the same HF tree API on 2026-08-07 — not estimated. Every one of the seven
# was wrong before, and in both directions, which is worse than uniformly
# optimistic: the two Parakeets under-reported by ~3.8x (0.18 -> 0.67 GB),
# so the recommended default quietly downloaded four times what the picker
# so installing v3 quietly downloaded four times what the picker
# promised on a metered or small-disk machine; but the two low-RAM
# zipformers OVER-reported by ~3x (0.128 -> 0.044), making the fallback
# models look bulkier than the heavyweights they exist to rescue users
@@ -129,7 +129,6 @@ _MODELS: dict[str, SherpaModelSpec] = {
kind="offline-transducer",
size_gb=0.67,
languages="25 European languages",
recommended=True,
heavy=True,
model_type="nemo_transducer",
files={
@@ -223,6 +222,7 @@ _MODELS: dict[str, SherpaModelSpec] = {
kind="offline-whisper",
size_gb=0.104,
languages="90+ languages (auto-detect)",
recommended=True,
files={
"encoder": "tiny-encoder.int8.onnx",
"decoder": "tiny-decoder.int8.onnx",
@@ -231,7 +231,7 @@ _MODELS: dict[str, SherpaModelSpec] = {
),
}
DEFAULT_MODEL_ID = "sherpa-parakeet-tdt-v3"
DEFAULT_MODEL_ID = "sherpa-whisper-tiny"
# repo_id → model id, so the model-store list (keyed by repo_id) can be
# enriched with the dictation metadata, and so capture can map either key.
@@ -261,6 +261,16 @@ def sherpa_available() -> tuple[bool, str]:
return True, "ready"
except ImportError as e:
return False, f"sherpa-onnx not installed: {e}. Install with: uv add sherpa-onnx"
except Exception as e: # noqa: BLE001 — an availability probe must fail closed
# Native wheel failures surface as OSError/RuntimeError rather than
# ImportError (missing DLL/dylib/so, loader or runtime init failure) —
# but the set is open-ended: an extension module is free to raise
# anything at init. This is an availability question, so ANY failure to
# import means "not available", never an exception escaping to the
# caller. SherpaDictationBackend.is_available() calls this directly and
# capture_ws.ws_transcribe calls that without a guard, so an unexpected
# type here took the WebSocket down instead of falling back (#1610).
return False, f"sherpa-onnx unavailable ({type(e).__name__}): {e}"
def _resolve_model_dir(spec: SherpaModelSpec, *, download: bool = True) -> str:
@@ -397,13 +407,10 @@ def build_online_recognizer(spec: SherpaModelSpec, *, download: bool = True):
# transcribe the same bytes. It is a defect inside sherpa-onnx that the app
# cannot fix by configuration.
#
# The curated default therefore cannot be trusted to WORK just because it is
# installed — and which platforms are affected is not knowable up front, so
# hard-coding a different default per OS would only be a guess. Instead the app
# learns from what it observes: when a session hears real speech and the model
# returns nothing, that model is demoted on THIS machine and stops being
# selected. Self-correcting wherever the breakage actually is, and a no-op
# everywhere it isn't.
# Installation alone therefore cannot prove that a recognizer works. When a
# session hears real speech and the model returns nothing, that model is
# demoted on this machine and stops being selected. This self-corrects wherever
# the decoder defect appears and is a no-op everywhere it does not.
#: prefs key holding the list of model ids demoted on this machine.
PREF_SILENT_MODELS = "dictation.silent_models"
backend/tests/test_dictation_model_demotion.py
+6 -7
View File
@@ -1,16 +1,15 @@
"""A dictation model that decodes nothing gets demoted, not re-selected forever.
`sherpa-parakeet-tdt-v3` is the curated default, and on Windows it installs
cleanly, loads without error, and returns an empty token list for clear speech
On Windows, `sherpa-parakeet-tdt-v3` installs cleanly, loads without error,
and returns an empty token list for clear speech
(both quantisations, both decoding methods, sherpa-onnx 1.13.3 and 1.13.4)
while whisper and zipformer transcribe the same bytes. The defect is inside
sherpa-onnx's NeMo-TDT decoder — unfixable from here by configuration.
Hard-coding a different default per OS would be a guess: we have evidence for
one platform only. So the app observes instead. When a session hears real
speech and the model returns nothing, that model is demoted ON THIS MACHINE and
stops being auto-selected, which self-corrects wherever the breakage actually
is and is a no-op everywhere it isn't.
Whisper Tiny is now the cross-platform default, while Parakeet remains
selectable. Runtime demotion still protects users who select a recognizer that
loads successfully but decodes nothing: it is demoted on this machine and the
next session follows the capture fallback.
These tests pin the demotion round trip and, critically, that the user can
always take back control by re-picking the model.
backend/tests/test_dictation_silent_model.py
+2 -2
View File
@@ -1,13 +1,13 @@
"""A dictation model that decodes NOTHING must fall back, not fail silently.
Found on Windows with the curated default `sherpa-parakeet-tdt-v3`: the model
Found on Windows with `sherpa-parakeet-tdt-v3`: the model
downloads, loads with zero errors, and is correctly detected as a TDT model
(`num_durations: 5`) — then returns an empty token list for clear speech.
Measured against the same 18.9s WAV, on the same machine, same sherpa-onnx:
sherpa-whisper-tiny -> "Alright, here we are. I hope that's all..."
sherpa-zipformer-en-20m -> "ANTS BOTH IN WHAT DISGUISED THIS THAT..."
parakeet-tdt-v3 (int8) -> '' <-- the curated default
parakeet-tdt-v3 (int8) -> ''
parakeet-tdt-v3 (fp32) -> ''
parakeet-tdt-v2 (int8) -> ''
docs/engines/nemo-parakeet.md
+3 -2
View File
@@ -20,8 +20,9 @@ instead — same model family, no NeMo dependency:
- **Apple Silicon:** [parakeet-mlx](parakeet-mlx.md) (installed by default on
mac-ARM source installs).
- **Any platform, CPU:** [sherpa-onnx-asr](sherpa-onnx-asr.md) — its default
dictation model is an int8 ONNX export of Parakeet TDT v3.
- **Any platform, CPU:** [sherpa-onnx-asr](sherpa-onnx-asr.md) — selectable
int8 ONNX exports of Parakeet TDT v2/v3; Whisper Tiny remains the
cross-platform dictation default.
## Selecting it
docs/engines/sherpa-onnx-asr.md
+9 -6
View File
@@ -11,10 +11,10 @@ partials either way.
- Ensure `sherpa-onnx` is installed (`uv add sherpa-onnx` on source installs).
- Pick a dictation model in the app (Model Catalogue → Models lists the
curated set below), or **Model Catalogue → Engines**, ASR tab → **Use**, or
selectable set below), or **Model Catalogue → Engines**, ASR tab → **Use**, or
pin `OMNIVOICE_ASR_BACKEND=sherpa-onnx-asr`.
- `OMNIVOICE_SHERPA_ASR_MODEL` selects the model — default
`sherpa-parakeet-tdt-v3`.
`sherpa-whisper-tiny`.
## Best at
@@ -25,17 +25,17 @@ partials either way.
timestamps, which makes it a dictation/notes tool rather than a dubbing
engine.
## The 7 curated models
## The 7 selectable models
| Id | Type | Languages | Download |
| --- | --- | --- | --- |
| `sherpa-parakeet-tdt-v3` (default) | offline | 25 European languages | 0.67 GB |
| `sherpa-parakeet-tdt-v3` | offline | 25 European languages | 0.67 GB |
| `sherpa-parakeet-tdt-v2` | offline | English | 0.66 GB |
| `sherpa-zipformer-bilingual-zh-en` | streaming | Chinese + English | 0.20 GB |
| `sherpa-paraformer-bilingual-zh-en` | streaming | Chinese + English | 0.24 GB |
| `sherpa-zipformer-en-20m` | streaming | English | 0.044 GB |
| `sherpa-zipformer-zh-14m` | streaming | Chinese | 0.025 GB |
| `sherpa-whisper-tiny` | offline | 90+ languages (auto-detect) | 0.104 GB |
| `sherpa-whisper-tiny` (default, recommended) | offline | 90+ languages (auto-detect) | 0.104 GB |
Sizes are measured on-disk download sizes. Weights are int8 ONNX checkpoints
that download on first use through the same HF cache as everything else —
@@ -45,7 +45,10 @@ allocator holds onto freed blocks).
## Platform support
CPU on every platform, by the strict cross-platform default-parity rule.
CPU on every platform, by the strict cross-platform default-parity rule. The
[upstream CPU wheels](https://k2-fsa.github.io/sherpa/onnx/python/install.html)
cover Linux, macOS, and Windows, and upstream documents Whisper as a
[supported non-streaming model family](https://k2-fsa.github.io/sherpa/onnx/pretrained_models/whisper/index.html).
`OMNIVOICE_SHERPA_ASR_PROVIDER` can override the ONNX provider on a verified
GPU build, but the default never diverges.
docs/features.yaml
+1
View File
@@ -95,4 +95,5 @@ docs:
- docs/install/linux.md
- docs/install/docker.md
- docs/install/troubleshooting.md
- docs/features/dictation.md
- docs/migration/real-time-voice-cloning.md
docs/features/dictation.md
+63
View File
@@ -0,0 +1,63 @@
# Dictation
VoiceStudio dictation records from the system-wide shortcut, transcribes
locally, and—where the desktop permits it—inserts the result into the app where
the shortcut was pressed. The pill never needs keyboard focus.
## Use it
1. Choose an installed dictation model in the Model Catalogue.
2. Set the shortcut and hold/toggle behavior in **Settings → Hotkey**.
3. Put the cursor in a text field, press the shortcut, speak, then release or
press again.
Whisper Tiny is the recommended default on macOS, Windows, and Linux. It
auto-detects more than 90 languages. Parakeet TDT v3 remains available for its
25 supported European languages, but it is not selected automatically.
The pill reports **Inserted** only after native delivery succeeds. **Copied**
means automatic insertion was unavailable and the complete final transcript is
ready for a normal paste. VoiceStudio retries a speech-level empty Sherpa decode
only through another ASR model whose weights are already installed; when that
fallback confirms the audio contains words, the silent model is demoted. This
recovery never starts a download.
## Destination and clipboard safety
The desktop captures the target at shortcut-down and carries its session ID
through partial, utterance, and summary messages. A late result from an older
session cannot use a newer session's target. macOS, Windows, and X11 validate
and reactivate the captured process/window before insertion. Wayland does not
expose a portable target identity or arbitrary foreign-window activation, so
the safe default leaves the transcript copied instead of guessing which app
should receive it. The GTK pill remains non-focusable.
For paste delivery, VoiceStudio snapshots text, HTML (with its plain-text
alternative), image, or file-list clipboard content, stages the transcript,
and restores the snapshot after the target consumes it.
Streaming segments share a generation-tracked lease: a stale restore cannot
win over a newer segment, and VoiceStudio never overwrites clipboard content
you copied during transcription. Unsupported clipboard formats cannot be
round-tripped; in that case the transcript remains on the clipboard instead of
attempting a lossy restore.
## Platform behavior
| Platform | Automatic insertion |
| --- | --- |
| macOS | Reactivates the captured application and sends Command-V. Without Accessibility permission, the result stays copied. |
| Windows | Validates the captured window and process, requests foreground activation, then sends Ctrl-V. If Windows denies activation, the result stays copied. |
| Linux X11 | Reactivates the captured X11 window through EWMH, verifies it, then sends Ctrl-V. |
| Wayland | Leaves the complete transcript copied because a portable captured-window identity is unavailable. |
| Browser mode | Copies the transcript; browsers cannot target another desktop app. |
Advanced Wayland users can set `VOICESTUDIO_WAYLAND_UNTARGETED_INSERT=1` to
insert into whichever client owns keyboard focus when transcription finishes.
wlroots compositors use `wtype`; KDE Plasma and GNOME can use clipboard paste
through `dotool` or `ydotool`. Helpers run with host loader variables from an
AppImage, have a bounded timeout, and never retry after one may have emitted
partial input. This opt-in cannot promise the shortcut-down target if focus
changes. `dotool` needs direct write access to `/dev/uinput`; `ydotool` 1.0+
needs a running `ydotoold` with that access and a user-readable socket.
VoiceStudio checks these prerequisites before selection. Tray-started Wayland
dictation always stays copy-only.
docs/install/linux.md
+16
View File
@@ -100,6 +100,22 @@ where the protocol gives applications no say in their own placement and the
compositor decides where it appears. The capsule works the same either way; only
its position is out of the app's hands there.
Wayland does not expose a portable identity for the app focused at shortcut
down, so VoiceStudio safely leaves the complete transcript on the clipboard and
the pill says **Copied** instead of risking insertion into a different app.
Advanced users can opt into current-focus insertion with
`VOICESTUDIO_WAYLAND_UNTARGETED_INSERT=1`. wlroots compositors such as Sway and
Hyprland use `wtype`; KDE Plasma and GNOME can use `dotool` or `ydotool` to
paste the Unicode clipboard payload. The
opt-in targets whichever client owns keyboard focus when transcription
finishes, not necessarily the app where dictation started. Tray-started
dictation remains copy-only. `dotool` needs direct write access to
`/dev/uinput` (normally through a distribution udev rule/group). `ydotool`
1.0+ needs the `ydotoold` daemon running with that access and its socket
available to the desktop user. VoiceStudio skips either helper when its
readiness check fails.
If the global shortcut stops working, restart your desktop's portal service,
then save the shortcut again in **Settings → Hotkey** to reopen consent. Portal
packages and support vary by desktop; use the backend recommended by your
docs/specs/2026-07-16-dictation-flow-program.md
+7 -7
View File
@@ -1,14 +1,14 @@
# Dictation Flow Program — local WhisperFlow-class dictation on Parakeet
# Dictation Flow Program — local cross-platform flow dictation
*Spec, 2026-07-16. Research inputs: three-agent study — product landscape (Wispr Flow, jamiepine/voicebox, Handy, VoiceInk, Whispering, Talon, Claude Code `/voice`), in-repo capability map, and Parakeet TDT/Nemotron feasibility (sherpa-onnx). Sources cited inline where load-bearing.*
*Spec, 2026-07-16. Research inputs: a multi-agent product-landscape study, in-repo capability map, and local ASR feasibility review. Sources are cited inline where load-bearing.*
## Why
Dictating prompts to AI agents is the fastest-growing text-input workload (Claude Code shipped built-in `/voice`; Wispr Flow raised at ~$2B on it) — and every polished option is **cloud** (Wispr: cloud-only, no Linux, one privacy scandal already; Claude Code voice: cloud-only, no SSH). The best open competitor, **jamiepine/voicebox** (41.7k★, MIT — our refinement layer is already adapted from it), only ships reliable auto-paste on macOS. VoiceStudio already has the hard parts: a Wispr-style pill, global hotkey, sherpa-onnx streaming WS, **Parakeet TDT v3 int8 as the shipped default**, clipboard-restoring paste, and local-LLM refinement. A local, cross-platform, private flow-dictation experience is reachable and strategically differentiating — the wedge is **local + Linux/Wayland + agent-prompting**, where nobody credible plays.
Dictating prompts to AI agents is a rapidly growing text-input workload, while polished options remain cloud-first and Linux support is uneven. VoiceStudio already has the hard parts: a compact capture pill, global hotkey, sherpa-onnx streaming WS, **Whisper Tiny int8 as the shipped cross-platform default**, clipboard-restoring paste, and local-LLM refinement. A local, cross-platform, private flow-dictation experience is reachable and strategically differentiating — the wedge is **local + Linux/Wayland + agent-prompting**.
## Current state (verified in-repo)
Widget: pill webview + `tauri-plugin-global-shortcut` (`CmdOrCtrl+Shift+Space`, toggle/hold) on macOS, Windows and X11 + the GlobalShortcuts desktop portal on Wayland + browser-mode keyboard fallback; `getUserMedia` → raw-PCM WS `/ws/transcribe`; paste via arboard+enigo with clipboard restore, macOS a11y fail-loud, Windows no-activate. Backend: 7 sherpa models (Parakeet TDT v3 default), streaming path (zipformer/paraformer) + chunked-offline path (0.8 s partial cadence, **RMS silence gate**), `text_polish` on finals, opt-in LLM refinement (Ollama/LM Studio, ≤4 s wall clock). Gaps: no real VAD, no dictionary/hotwords, no per-app awareness, no command grammar, no language picker, enigo-only Linux insertion, no comprehensive dictation feature guide beyond the Linux installation note, picker understates model size ~4×.
Widget: pill webview + `tauri-plugin-global-shortcut` (`CmdOrCtrl+Shift+Space`, toggle/hold) on macOS, Windows and X11 + the GlobalShortcuts desktop portal on Wayland + browser-mode keyboard fallback; `getUserMedia` → 16 kHz raw-PCM WS `/ws/transcribe`; a native session captures the destination before the pill appears, restores an untouched clipboard by generation, reactivates macOS/Windows/X11 targets, and uses a truthful copy fallback on Wayland unless current-focus insertion is explicitly enabled. Backend: 7 sherpa models (Whisper Tiny default), streaming path (zipformer/paraformer) + chunked-offline path (0.8 s partial cadence, **RMS silence gate**), shared speech-evidence model demotion with installed-only ASR fallback, `text_polish` on finals, opt-in LLM refinement (Ollama/LM Studio, ≤4 s wall clock). Gaps: no real VAD, no dictionary/hotwords, no per-app formatting profiles, no command grammar, no language picker, picker understates model size ~4×.
## Program phases
@@ -41,7 +41,7 @@ Parakeet's one real weakness is OOV technical terms — and the dictionary is Wi
### Phase 4 — insertion reliability + Wayland (beat everyone on Linux)
- **Reliability engineering** (the boring 20% that reads professional; Wispr does 5 retries): retry-with-backoff on paste, transcript stays on clipboard + toast on failure, password-field refusal, Windows elevated-window detection.
- **Wayland insertion chain** replacing bare enigo on Linux: kwtype→wtype→dotool→ydotool→wl-copy+notify fallback (Handy's proven cascade), IBus/Fcitx5 input-method commit path evaluated for GNOME (highest quality, nobody mainstream ships it), libei/RemoteDesktop-portal as the forward bet. Wispr has no Linux at all; voicebox has no Linux paste — this is the moat.
- **Wayland insertion chain** replacing bare enigo on Linux: wtype→dotool→ydotool→wl-copy+notify fallback, IBus/Fcitx5 input-method commit path evaluated for GNOME (highest quality, nobody mainstream ships it), libei/RemoteDesktop-portal as the forward bet. Reliable local Linux insertion is the moat.
### Phase 5 — command mode (headline, local-only differentiator)
Second hotkey → speak an instruction over selected text → local LLM rewrite → explicit Apply. Wispr charges for this; ours is local and free. Requires configured LLM; hidden otherwise (existing `llm_ready` plumbing).
@@ -56,10 +56,10 @@ Second hotkey → speak an instruction over selected text → local LLM rewrite
| Use case | Model | Partials | Final after pause | Disk/RAM |
|---|---|---|---|---|
| English, best feel | nemotron-streaming-en 160 ms (new, Ph. 1) | 200–400 ms | ~0.5–0.7 s | 0.66 GB / ~1.2 GB |
| Multilingual default | parakeet-tdt-v3 + silero-VAD (upgraded path) | 0.8 s cadence | ~0.4–0.7 s | 0.67 GB / ~1.2 GB |
| European languages (opt-in) | parakeet-tdt-v3 + silero-VAD (upgraded path) | 0.8 s cadence | ~0.4–0.7 s | 0.67 GB / ~1.2 GB |
| Multilingual streaming (opt-in) | Nemotron-3.5 320 ms (new) | ~400 ms | ~0.7 s | 0.68 GB / ~1.2 GB |
| Low-RAM | zipformer-20M (existing) | ~100 ms | ~0.6 s | 0.13 GB / ~0.3 GB |
| CJK / 90+ langs | whisper-tiny (existing; consider small) | n/a | seconds | 0.12 GB |
| Multilingual default / CJK | whisper-tiny (existing; consider small) | n/a | seconds | 0.104 GB |
## Top risks
frontend/src-tauri/Cargo.lock
Generated
+157 -1
View File
@@ -101,6 +101,7 @@ dependencies = [
"parking_lot",
"percent-encoding",
"windows-sys 0.60.2",
"wl-clipboard-rs",
"x11rb",
]
@@ -1040,6 +1041,12 @@ dependencies = [
"tendril",
]
[[package]]
name = "downcast-rs"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75b325c5dbd37f80359721ad39aca5a29fb04c89279657cffdda8736d0c0b9d2"
[[package]]
name = "dpi"
version = "0.1.2"
@@ -1287,6 +1294,12 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "fixedbitset"
version = "0.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99"
[[package]]
name = "flate2"
version = "1.1.9"
@@ -2582,6 +2595,15 @@ version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "650eef8c711430f1a879fdd01d4745a7deea475becfb90269c06775983bbf086"
[[package]]
name = "nom"
version = "8.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405"
dependencies = [
"memchr",
]
[[package]]
name = "ntapi"
version = "0.4.3"
@@ -2687,6 +2709,7 @@ checksum = "d49e936b501e5c5bf01fda3a9452ff86dc3ea98ad5f283e1455153142d97518c"
dependencies = [
"bitflags 2.13.0",
"block2 0.6.2",
"libc",
"objc2 0.6.4",
"objc2-core-foundation",
"objc2-core-graphics",
@@ -2948,8 +2971,11 @@ dependencies = [
"enigo",
"fs4",
"getrandom 0.3.4",
"gtk",
"libc",
"log",
"objc2 0.6.4",
"objc2-app-kit 0.3.2",
"reqwest",
"semver",
"serde",
@@ -2973,6 +2999,7 @@ dependencies = [
"webview2-com",
"windows 0.61.3",
"windows-core 0.61.2",
"x11rb",
"zbus",
"zip 2.4.2",
]
@@ -3017,6 +3044,16 @@ dependencies = [
"pin-project-lite",
]
[[package]]
name = "os_pipe"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967"
dependencies = [
"libc",
"windows-sys 0.45.0",
]
[[package]]
name = "osakit"
version = "0.3.1"
@@ -3097,6 +3134,17 @@ version = "2.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "petgraph"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455"
dependencies = [
"fixedbitset",
"hashbrown 0.15.5",
"indexmap 2.14.0",
]
[[package]]
name = "phf"
version = "0.13.1"
@@ -3181,7 +3229,7 @@ checksum = "092791278e026273c1b65bbdcfbba3a300f2994c896bd01ab01da613c29c46f1"
dependencies = [
"base64 0.22.1",
"indexmap 2.14.0",
"quick-xml",
"quick-xml 0.39.4",
"serde",
"time",
]
@@ -3369,6 +3417,15 @@ dependencies = [
"memchr",
]
[[package]]
name = "quick-xml"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1"
dependencies = [
"memchr",
]
[[package]]
name = "quinn"
version = "0.11.9"
@@ -5285,6 +5342,17 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "tree_magic_mini"
version = "3.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8765b90061cba6c22b5831f675da109ae5561588290f9fa2317adab2714d5a6"
dependencies = [
"memchr",
"nom",
"petgraph",
]
[[package]]
name = "try-lock"
version = "0.2.5"
@@ -5633,6 +5701,76 @@ dependencies = [
"semver",
]
[[package]]
name = "wayland-backend"
version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38a91b4eaddff87b1cd1074985e3713da4af2c49742d1b356b2c01670a67a078"
dependencies = [
"cc",
"downcast-rs",
"rustix",
"smallvec",
"wayland-sys",
]
[[package]]
name = "wayland-client"
version = "0.31.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3c36a0f861ad76d0901f2800b46321410d9f73f2ea88aac0650d86c32688073"
dependencies = [
"bitflags 2.13.0",
"rustix",
"wayland-backend",
"wayland-scanner",
]
[[package]]
name = "wayland-protocols"
version = "0.32.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "23d0c813de3daa2ed6520af85a3bd49b0e722a3078506899aa9686fea58dc4b6"
dependencies = [
"bitflags 2.13.0",
"wayland-backend",
"wayland-client",
"wayland-scanner",
]
[[package]]
name = "wayland-protocols-wlr"
version = "0.3.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eb04e52f7836d7c7976c78ca0250d61e33873c34156a2a1fc9474828ec268234"
dependencies = [
"bitflags 2.13.0",
"wayland-backend",
"wayland-client",
"wayland-protocols",
"wayland-scanner",
]
[[package]]
name = "wayland-scanner"
version = "0.31.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "338e30461b3a2b67d70eb30a6d89f8e0c93a833e07d2ae89085cd070c4a00ac0"
dependencies = [
"proc-macro2",
"quick-xml 0.41.0",
"quote",
]
[[package]]
name = "wayland-sys"
version = "0.31.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d8eab23fefc9e41f8e841df4a9c707e8a8c4ed26e944ef69297184de2785e3be"
dependencies = [
"pkg-config",
]
[[package]]
name = "web-sys"
version = "0.3.102"
@@ -6468,6 +6606,24 @@ dependencies = [
"wasmparser",
]
[[package]]
name = "wl-clipboard-rs"
version = "0.9.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9651471a32e87d96ef3a127715382b2d11cc7c8bb9822ded8a7cc94072eb0a3"
dependencies = [
"libc",
"log",
"os_pipe",
"rustix",
"thiserror 2.0.18",
"tree_magic_mini",
"wayland-backend",
"wayland-client",
"wayland-protocols",
"wayland-protocols-wlr",
]
[[package]]
name = "writeable"
version = "0.6.3"
frontend/src-tauri/Cargo.toml
+10 -2
View File
@@ -67,7 +67,13 @@ dirs-next = "2"
# Native (OS-side) clipboard write for dictation auto-paste: the widget window
# is unfocused on macOS so the simulated ⌘V reaches the target app, which makes
# the WebView clipboard APIs fail silently there (#287)
arboard = "3"
arboard = { version = "3", features = ["wayland-data-control"] }
[target.'cfg(target_os = "macos")'.dependencies]
# Capture the frontmost application at shortcut-down and reactivate that exact
# process before transcript delivery.
objc2 = "0.6"
objc2-app-kit = { version = "0.3", default-features = false, features = ["std", "libc", "NSRunningApplication", "NSWorkspace"] }
[target.'cfg(windows)'.dependencies]
zip = { version = "2", default-features = false, features = ["deflate"] }
@@ -84,13 +90,15 @@ windows-core = "0.61"
# `HWND` type — no second copy of the crate enters the dependency graph.
# Win32_System_Registry: check_microphone reads the CapabilityAccessManager
# ConsentStore mic toggle (RegGetValueW) for the permissions UX.
windows = { version = "0.61", features = ["Win32_Foundation", "Win32_UI_WindowsAndMessaging", "Win32_System_Registry"] }
windows = { version = "0.61", features = ["Win32_Foundation", "Win32_UI_WindowsAndMessaging", "Win32_System_Registry", "Win32_System_Threading"] }
[target.'cfg(unix)'.dependencies]
libc = "0.2"
[target.'cfg(target_os = "linux")'.dependencies]
webkit2gtk = "2.0"
gtk = "0.18"
x11rb = "0.13"
# Wayland compositors do not expose global keys through X11 grabs. Use the
# standard xdg-desktop-portal GlobalShortcuts interface there; zbus is already
# present transitively through Tauri's opener/single-instance plugins.
frontend/src-tauri/src/commands.rs
+175 -151
View File
@@ -7,13 +7,13 @@ use std::time::{Duration, Instant};
use serde::{Deserialize, Serialize};
use tauri::image::Image;
use tauri::Manager;
use tauri::{Emitter, Manager};
use tauri_plugin_dialog::DialogExt;
use crate::dictation_shortcut::{DictationShortcutManager, ShortcutInfo, update_tray_hint};
use crate::config::{load_config, save_config};
use crate::dictation_shortcut::{update_tray_hint, DictationShortcutManager, ShortcutInfo};
use crate::{AppFlags, TrayHandle};
use crate::{TRAY_ICON_DEFAULT, TRAY_ICON_RECORDING};
use crate::config::{load_config, save_config};
// ── Native host-path authorization ───────────────────────────────────────
@@ -65,10 +65,7 @@ fn remember_reveal_path<R: tauri::Runtime>(
Ok(())
}
fn reveal_path_is_authorized<R: tauri::Runtime>(
app: &tauri::AppHandle<R>,
target: &Path,
) -> bool {
fn reveal_path_is_authorized<R: tauri::Runtime>(app: &tauri::AppHandle<R>, target: &Path) -> bool {
if let Ok(data_root) = fs::canonicalize(
crate::setup::resolved_data_dir(app).unwrap_or_else(crate::setup::default_data_dir),
) {
@@ -89,12 +86,7 @@ fn reveal_path_is_authorized<R: tauri::Runtime>(
fn validate_host_path(kind: &str, path: PathBuf) -> Result<PathBuf, String> {
if !matches!(
kind,
"models_dir"
| "ffmpeg"
| "ffprobe"
| "dub_export"
| "soni_input"
| "soni_output_dir"
"models_dir" | "ffmpeg" | "ffprobe" | "dub_export" | "soni_input" | "soni_output_dir"
) {
return Err("Unsupported host-path capability".into());
}
@@ -216,7 +208,10 @@ pub async fn authorize_host_path(
kind,
path: validated.to_string_lossy().into_owned(),
};
fs::write(&target, serde_json::to_vec(&payload).map_err(|e| e.to_string())?)
fs::write(
&target,
serde_json::to_vec(&payload).map_err(|e| e.to_string())?,
)
.map_err(|e| format!("Could not authorize path: {e}"))?;
#[cfg(unix)]
{
@@ -247,7 +242,10 @@ mod host_path_authorization_tests {
#[test]
fn empty_models_path_is_the_authorized_default_reset() {
assert_eq!(validate_host_path("models_dir", PathBuf::new()).unwrap(), PathBuf::new());
assert_eq!(
validate_host_path("models_dir", PathBuf::new()).unwrap(),
PathBuf::new()
);
}
#[test]
@@ -259,11 +257,9 @@ mod host_path_authorization_tests {
destination,
);
assert!(validate_host_path("dub_export", PathBuf::from("relative/export.wav")).is_err());
assert!(validate_host_path(
"dub_export",
parent.join("missing-directory/export.wav"),
)
.is_err());
assert!(
validate_host_path("dub_export", parent.join("missing-directory/export.wav"),).is_err()
);
}
}
@@ -316,12 +312,14 @@ pub fn read_log_tail(source: String, tail: Option<usize>) -> LogTailPayload {
let path = match source.as_str() {
"backend" => backend_runtime_log_path(),
"tauri" => tauri_log_path(),
_ => return LogTailPayload {
_ => {
return LogTailPayload {
lines: vec![],
path: String::new(),
exists: false,
total_lines: 0,
},
}
}
};
let path_str = path.to_string_lossy().to_string();
@@ -363,14 +361,10 @@ fn backend_runtime_log_path() -> PathBuf {
let data_dir = if cfg!(target_os = "macos") {
dirs_data_dir().join("OmniVoice")
} else if cfg!(target_os = "windows") {
PathBuf::from(
std::env::var("APPDATA").unwrap_or_else(|_| ".".to_string()),
)
PathBuf::from(std::env::var("APPDATA").unwrap_or_else(|_| ".".to_string()))
.join("OmniVoice")
} else {
PathBuf::from(
std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()),
)
PathBuf::from(std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()))
.join(".omnivoice")
};
data_dir.join("omnivoice.log")
@@ -379,16 +373,12 @@ fn backend_runtime_log_path() -> PathBuf {
fn dirs_data_dir() -> PathBuf {
#[cfg(target_os = "macos")]
{
PathBuf::from(
std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()),
)
PathBuf::from(std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()))
.join("Library/Application Support")
}
#[cfg(not(target_os = "macos"))]
{
PathBuf::from(
std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()),
)
PathBuf::from(std::env::var("HOME").unwrap_or_else(|_| "/tmp".to_string()))
}
}
@@ -403,7 +393,10 @@ fn tauri_log_path() -> PathBuf {
.join("tauri.log")
} else if cfg!(target_os = "windows") {
let appdata = std::env::var("APPDATA").unwrap_or_else(|_| home.clone());
PathBuf::from(appdata).join(bid).join("logs").join("tauri.log")
PathBuf::from(appdata)
.join(bid)
.join("logs")
.join("tauri.log")
} else {
PathBuf::from(&home)
.join(".local/share")
@@ -508,22 +501,16 @@ fn hf_hub_cache_dir() -> PathBuf {
.join("hub")
}
// ── Simulate paste ────────────────────────────────────────────────────────
use enigo::{Direction, Enigo, Key, Keyboard, Settings as EnigoSettings};
// ── Dictation output ─────────────────────────────────────────────────────
/// Error-kind builder the dictation widget switches on. Kinds are a plain
/// string prefix ("a11y:" | "clipboard:" | "paste:") so the JS side can do
/// `err.split(':')[0]` without a serde enum crossing the IPC boundary.
/// string prefix ("a11y:" | "clipboard:" | "paste:" | "preflight:") so the
/// JS side can do `err.split(':')[0]` without a serde enum crossing the IPC
/// boundary.
fn kind_err(kind: &str, detail: impl std::fmt::Display) -> String {
format!("{kind}:{detail}")
}
/// How long the transcript must sit on the clipboard before the user's
/// previous clipboard is restored: ~300ms covers slow paste consumers
/// (Electron apps, remote desktops) without being user-noticeable.
const CLIPBOARD_RESTORE_DELAY: Duration = Duration::from_millis(300);
/// macOS Accessibility grant check — CGEvent key synthesis silently no-ops
/// without it. Direct FFI against ApplicationServices: one symbol, not worth
/// a crate.
@@ -705,7 +692,12 @@ pub fn open_microphone_settings() -> Result<(), String> {
.arg("x-apple.systempreferences:com.apple.preference.security?Privacy_Microphone")
.spawn()
.map(|_| ())
.map_err(|e| kind_err("settings", format!("failed to open microphone settings: {e}")))
.map_err(|e| {
kind_err(
"settings",
format!("failed to open microphone settings: {e}"),
)
})
}
#[cfg(target_os = "windows")]
{
@@ -718,7 +710,12 @@ pub fn open_microphone_settings() -> Result<(), String> {
.creation_flags(0x0800_0000) // CREATE_NO_WINDOW
.spawn()
.map(|_| ())
.map_err(|e| kind_err("settings", format!("failed to open microphone settings: {e}")))
.map_err(|e| {
kind_err(
"settings",
format!("failed to open microphone settings: {e}"),
)
})
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
{
@@ -744,7 +741,10 @@ pub fn open_input_monitoring_settings() -> Result<(), String> {
.spawn()
.map(|_| ())
.map_err(|e| {
kind_err("settings", format!("failed to open input monitoring settings: {e}"))
kind_err(
"settings",
format!("failed to open input monitoring settings: {e}"),
)
})
}
#[cfg(not(target_os = "macos"))]
@@ -757,71 +757,44 @@ pub fn open_input_monitoring_settings() -> Result<(), String> {
}
#[tauri::command]
pub fn simulate_paste(text: Option<String>) -> Result<(), String> {
// macOS: fail loud BEFORE touching the clipboard if Accessibility isn't
// granted — otherwise the ⌘V below silently goes nowhere and the caller
// can't tell (the old fire-and-forget behavior).
pub async fn simulate_paste(
text: String,
session_id: u64,
flags: tauri::State<'_, AppFlags>,
) -> Result<crate::dictation_output::DeliveryOutcome, String> {
// macOS: a revoked/missing Accessibility grant prevents synthesis, but it
// must not discard the result. Keep the full transcript copied and report
// the fallback truthfully.
#[cfg(target_os = "macos")]
if !accessibility_trusted() {
return Err(kind_err("a11y", "accessibility permission not granted"));
let output = flags.output.clone();
return tauri::async_runtime::spawn_blocking(move || {
output.copy_for_session(session_id, &text)
})
.await
.map_err(|error| kind_err("clipboard", format!("output worker failed: {error}")))?;
}
// Write the transcript to the clipboard natively first: the widget window
// is intentionally unfocused on macOS (so the simulated ⌘V reaches the
// target app), which makes the WebView clipboard APIs (navigator.clipboard
// / execCommand('copy')) fail silently there (#287). `text` is optional so
// call sites that already populated the clipboard keep working.
//
// Save what the user had there first (text only — restoring images/files
// isn't worth the platform-specific surface) so dictation doesn't clobber
// their clipboard.
let mut saved: Option<String> = None;
if let Some(t) = text {
let mut cb = arboard::Clipboard::new()
.map_err(|e| kind_err("clipboard", format!("init failed: {e}")))?;
saved = cb.get_text().ok();
cb.set_text(t)
.map_err(|e| kind_err("clipboard", format!("write failed: {e}")))?;
}
let output = flags.output.clone();
tauri::async_runtime::spawn_blocking(move || output.deliver(session_id, &text))
.await
.map_err(|error| kind_err("paste", format!("output worker failed: {error}")))?
}
std::thread::sleep(Duration::from_millis(80));
let mut enigo = Enigo::new(&EnigoSettings::default())
.map_err(|e| kind_err("paste", format!("failed to init keyboard sim: {e}")))?;
#[cfg(target_os = "macos")]
{
enigo.key(Key::Meta, Direction::Press)
.map_err(|e| kind_err("paste", format!("key press failed: {e}")))?;
enigo.key(Key::Unicode('v'), Direction::Click)
.map_err(|e| kind_err("paste", format!("key click failed: {e}")))?;
enigo.key(Key::Meta, Direction::Release)
.map_err(|e| kind_err("paste", format!("key release failed: {e}")))?;
}
#[cfg(not(target_os = "macos"))]
{
enigo.key(Key::Control, Direction::Press)
.map_err(|e| kind_err("paste", format!("key press failed: {e}")))?;
enigo.key(Key::Unicode('v'), Direction::Click)
.map_err(|e| kind_err("paste", format!("key click failed: {e}")))?;
enigo.key(Key::Control, Direction::Release)
.map_err(|e| kind_err("paste", format!("key release failed: {e}")))?;
}
// Best-effort restore of the user's clipboard once the target app has
// consumed the paste. Only on success — on a paste error the transcript
// stays on the clipboard so the user can ⌘V it manually as a fallback.
if let Some(prev) = saved {
std::thread::spawn(move || {
std::thread::sleep(CLIPBOARD_RESTORE_DELAY);
if let Ok(mut cb) = arboard::Clipboard::new() {
let _ = cb.set_text(prev);
}
});
}
Ok(())
/// Preserve the authoritative transcript without emitting any keyboard input.
/// Used after live typing may have left an unknown prefix in the target: a
/// second insertion would duplicate text, but losing the complete result is
/// not an acceptable fallback.
#[tauri::command]
pub async fn copy_dictation_output_session(
text: String,
session_id: u64,
flags: tauri::State<'_, AppFlags>,
) -> Result<crate::dictation_output::DeliveryOutcome, String> {
let output = flags.output.clone();
tauri::async_runtime::spawn_blocking(move || output.copy_for_session(session_id, &text))
.await
.map_err(|error| kind_err("clipboard", format!("output worker failed: {error}")))?
}
// ── Simulate live typing ──────────────────────────────────────────────────
@@ -833,18 +806,22 @@ pub fn simulate_paste(text: Option<String>) -> Result<(), String> {
/// revised), then `text` is typed. Either may be empty/zero, so a single call
/// can correct-then-type in one round trip.
///
/// Cross-platform: `enigo`'s `.text()` synthesizes Unicode key events on macOS
/// (CGEvent), Windows (`SendInput` w/ `KEYEVENTF_UNICODE`), and Linux (X11/
/// libei). Backspace is a plain virtual-key `Click`, identical on all three.
/// On macOS this reuses the SAME accessibility permission `simulate_paste`
/// already requires (both go through `enigo` → CGEvent); no new grant needed.
/// The session-bound output layer reactivates the destination captured at
/// shortcut-down before emitting input. On Wayland it selects one compatible
/// compositor helper before emission and never retries after a possible
/// partial write.
///
/// Returns `Err` if the input layer is unavailable (e.g. accessibility not
/// granted) so the JS caller can fall back to the clipboard+paste path for
/// that segment without double-inserting. Errors carry the same kind
/// prefixes as `simulate_paste` ("a11y:" | "paste:").
/// granted). Because a failed input call may already have emitted a prefix,
/// the JS caller suppresses later insertion for that session. `preflight:`
/// explicitly means nothing was emitted and a final paste remains safe.
#[tauri::command]
pub fn simulate_type(text: Option<String>, backspaces: Option<u32>) -> Result<(), String> {
pub async fn simulate_type(
text: String,
backspaces: Option<u32>,
session_id: u64,
flags: tauri::State<'_, AppFlags>,
) -> Result<crate::dictation_output::DeliveryOutcome, String> {
// Same a11y gate as simulate_paste — `.text()`/`.key()` go through the
// identical CGEvent path on macOS and would silently no-op without it.
#[cfg(target_os = "macos")]
@@ -852,24 +829,46 @@ pub fn simulate_type(text: Option<String>, backspaces: Option<u32>) -> Result<()
return Err(kind_err("a11y", "accessibility permission not granted"));
}
let mut enigo = Enigo::new(&EnigoSettings::default())
.map_err(|e| kind_err("paste", format!("failed to init keyboard sim: {e}")))?;
let output = flags.output.clone();
tauri::async_runtime::spawn_blocking(move || {
output.type_delta(session_id, &text, backspaces.unwrap_or(0))
})
.await
.map_err(|error| kind_err("paste", format!("output worker failed: {error}")))?
}
let n = backspaces.unwrap_or(0);
for _ in 0..n {
enigo
.key(Key::Backspace, Direction::Click)
.map_err(|e| kind_err("paste", format!("backspace failed: {e}")))?;
}
#[tauri::command]
pub async fn activate_dictation_output_session(
session_id: u64,
flags: tauri::State<'_, AppFlags>,
) -> Result<(), String> {
let output = flags.output.clone();
tauri::async_runtime::spawn_blocking(move || output.activate_session(session_id))
.await
.map_err(|error| kind_err("paste", format!("output worker failed: {error}")))?
}
if let Some(t) = text {
if !t.is_empty() {
enigo
.text(&t)
.map_err(|e| kind_err("paste", format!("type failed: {e}")))?;
}
}
#[tauri::command]
pub async fn reject_dictation_output_session(
session_id: u64,
flags: tauri::State<'_, AppFlags>,
) -> Result<(), String> {
let output = flags.output.clone();
tauri::async_runtime::spawn_blocking(move || output.reject_session_candidate(session_id))
.await
.map_err(|error| kind_err("paste", format!("output worker failed: {error}")))?;
Ok(())
}
#[tauri::command]
pub async fn finish_dictation_output_session(
session_id: u64,
flags: tauri::State<'_, AppFlags>,
) -> Result<(), String> {
let output = flags.output.clone();
tauri::async_runtime::spawn_blocking(move || output.finish_session(session_id))
.await
.map_err(|error| kind_err("paste", format!("output worker failed: {error}")))?;
Ok(())
}
@@ -889,11 +888,16 @@ pub fn set_tray_recording(
// permanently-hidden widget made meaningless.)
flags.dictating.store(recording, Ordering::SeqCst);
log::info!("Dictation recording state: {recording}");
let bytes = if recording { TRAY_ICON_RECORDING } else { TRAY_ICON_DEFAULT };
let bytes = if recording {
TRAY_ICON_RECORDING
} else {
TRAY_ICON_DEFAULT
};
let img = Image::from_bytes(bytes).map_err(|e| format!("decode tray icon: {e}"))?;
let lock = tray_handle.tray.lock().map_err(|_| "tray lock poisoned")?;
if let Some(ref tray) = *lock {
tray.set_icon(Some(img)).map_err(|e| format!("set_icon: {e}"))?;
tray.set_icon(Some(img))
.map_err(|e| format!("set_icon: {e}"))?;
}
update_tray_hint(&app, &shortcuts.info().display, recording);
Ok(())
@@ -987,7 +991,10 @@ fn place_dictation_pill(app: &tauri::AppHandle, win: &tauri::WebviewWindow) {
}
log::info!(
"pill: placed at {x},{y} ({}x{} on a {}x{} monitor)",
size.width, size.height, area.width, area.height
size.width,
size.height,
area.width,
area.height
);
}
@@ -1049,9 +1056,15 @@ pub fn mark_dictation_capture_ready(app: tauri::AppHandle) {
return;
};
capture.ready = true;
if let Some(action) = capture.pending.take() {
let pending = std::mem::take(&mut capture.pending);
drop(capture);
crate::dispatch_dictation_capture(&app, &action);
for event in pending {
if let Err(error) = app.emit(event.name, event.payload) {
log::warn!(
"Queued dictation event {} could not emit: {error}",
event.name
);
}
}
}
@@ -1198,7 +1211,8 @@ pub fn reveal_host_path(app: tauri::AppHandle, path: String) -> Result<(), Strin
let folder = if target.is_dir() {
target.clone()
} else {
target.parent()
target
.parent()
.ok_or_else(|| "That path has no containing folder".to_string())?
.to_path_buf()
};
@@ -1268,7 +1282,10 @@ const CLEAR_WEBVIEW_RETRY_DELAY: Duration = Duration::from_millis(500);
/// Windows — because step 2 runs before an `AppHandle` exists.
fn webview_cache_paths() -> Option<(PathBuf, PathBuf)> {
let base = dirs_next::data_local_dir()?.join(crate::config::BUNDLE_IDENTIFIER);
Some((base.join(CLEAR_WEBVIEW_MARKER), base.join(WEBVIEW_CACHE_DIR)))
Some((
base.join(CLEAR_WEBVIEW_MARKER),
base.join(WEBVIEW_CACHE_DIR),
))
}
#[tauri::command]
@@ -1281,7 +1298,10 @@ pub fn clear_webview_cache_and_relaunch(app: tauri::AppHandle) -> Result<(), Str
if let Some(parent) = marker.parent() {
let _ = fs::create_dir_all(parent);
}
fs::write(&marker, b"requested by the splash recovery panel (issue #879)\n")
fs::write(
&marker,
b"requested by the splash recovery panel (issue #879)\n",
)
.map_err(|e| format!("write {}: {e}", marker.display()))?;
log::warn!(
"WebView cache repair requested (#879) — relaunching to clear {}",
@@ -1301,7 +1321,12 @@ pub fn clear_webview_cache_if_marked() {
let Some((marker, cache)) = webview_cache_paths() else {
return;
};
clear_webview_cache_at(&marker, &cache, CLEAR_WEBVIEW_ATTEMPTS, CLEAR_WEBVIEW_RETRY_DELAY);
clear_webview_cache_at(
&marker,
&cache,
CLEAR_WEBVIEW_ATTEMPTS,
CLEAR_WEBVIEW_RETRY_DELAY,
);
}
/// Filesystem half of [`clear_webview_cache_if_marked`], parameterized over
@@ -1403,7 +1428,10 @@ mod webview_cache_repair_tests {
let cache = dir.path().join(super::WEBVIEW_CACHE_DIR);
fs::write(&marker, b"test").unwrap();
clear_webview_cache_at(&marker, &cache, FEW, NO_WAIT);
assert!(!marker.exists(), "marker consumed even with nothing to clear");
assert!(
!marker.exists(),
"marker consumed even with nothing to clear"
);
}
/// A cache that can't be deleted (Windows: WebView2 file locks; simulated
@@ -1419,7 +1447,10 @@ mod webview_cache_repair_tests {
// Deny writes on the cache dir so its entries can't be unlinked.
fs::set_permissions(&cache, fs::Permissions::from_mode(0o555)).unwrap();
clear_webview_cache_at(&marker, &cache, FEW, NO_WAIT);
assert!(!marker.exists(), "one-shot: marker consumed even on failure");
assert!(
!marker.exists(),
"one-shot: marker consumed even on failure"
);
assert!(cache.exists(), "a locked cache survives the failed repair");
// Restore permissions so TempDir can clean up.
fs::set_permissions(&cache, fs::Permissions::from_mode(0o755)).unwrap();
@@ -1428,7 +1459,7 @@ mod webview_cache_repair_tests {
#[cfg(test)]
mod paste_error_tests {
use super::{kind_err, CLIPBOARD_RESTORE_DELAY};
use super::kind_err;
#[test]
fn kind_err_prefixes_with_kind() {
@@ -1450,11 +1481,4 @@ mod paste_error_tests {
let e = kind_err("clipboard", "init failed: os error 5");
assert_eq!(e.split_once(':').map(|(k, _)| k), Some("clipboard"));
}
#[test]
fn restore_delay_is_about_300ms() {
// Contract with the widget layer: previous clipboard comes back
// ~300ms after the paste, long enough for slow paste consumers.
assert_eq!(CLIPBOARD_RESTORE_DELAY.as_millis(), 300);
}
}
frontend/src-tauri/src/dictation_output.rs
+1656
View File
File diff suppressed because it is too large Load Diff
frontend/src-tauri/src/lib.rs
+118 -26
View File
@@ -7,33 +7,36 @@
//! backend – spawn backend process, port probing, log paths
//! commands – Tauri IPC commands (sysinfo, logs, HF cache, paste, tray, dictation)
pub mod config;
pub mod setup;
pub mod bootstrap;
pub mod tools;
pub mod backend;
pub mod blank_guard;
pub mod bootstrap;
pub mod commands;
pub mod dictation_shortcut;
pub mod config;
pub mod crash;
pub mod dictation_output;
pub mod dictation_shortcut;
pub mod reset;
pub mod setup;
pub mod tools;
pub mod uninstall;
pub mod updater_channel;
pub mod blank_guard;
#[cfg(target_os = "linux")]
pub mod wayland_shortcut;
use std::collections::VecDeque;
use std::process::Child;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tauri::{Emitter, Manager};
use tauri::menu::{MenuBuilder, MenuItemBuilder};
use tauri::tray::TrayIconBuilder;
use tauri::{Emitter, Manager};
use tauri_plugin_positioner::{Position, WindowExt};
use crate::bootstrap::{BootstrapStage, BootstrapState, set_stage};
use crate::bootstrap::{set_stage, BootstrapStage, BootstrapState};
use crate::config::load_config;
use crate::dictation_output::CaptureOrigin;
use crate::dictation_shortcut::DictationShortcutManager;
// ── Port ──────────────────────────────────────────────────────────────────
@@ -63,11 +66,32 @@ pub struct AppFlags {
/// the tray icon), so that same call keeps this in step.
pub dictating: AtomicBool,
pub capture: Mutex<CaptureDispatchState>,
pub output: dictation_output::DictationOutput,
}
pub struct CaptureDispatchState {
pub ready: bool,
pub pending: Option<String>,
pub(crate) ready: bool,
pub(crate) pending: VecDeque<CaptureEvent>,
}
impl Default for CaptureDispatchState {
fn default() -> Self {
Self {
ready: false,
pending: VecDeque::new(),
}
}
}
#[derive(Clone, serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub(crate) struct DictationCapturePayload {
pub(crate) session_id: u64,
}
pub(crate) struct CaptureEvent {
pub(crate) name: &'static str,
pub(crate) payload: DictationCapturePayload,
}
pub struct TrayHandle {
@@ -84,8 +108,24 @@ fn dictation_capture_event(action: &str, dictating: bool) -> &'static str {
}
pub fn dispatch_dictation_capture(app: &tauri::AppHandle, action: &str) {
dispatch_dictation_capture_from(app, action, CaptureOrigin::Shortcut);
}
fn dispatch_dictation_capture_from(app: &tauri::AppHandle, action: &str, origin: CaptureOrigin) {
let flags = app.state::<AppFlags>();
let event = dictation_capture_event(action, flags.dictating.load(Ordering::SeqCst));
let session_id = if event == "tray-dictate" {
flags.output.begin_session(origin)
} else if let Some(session_id) = flags.output.current_session_id() {
session_id
} else {
log::warn!("Dictation capture '{action}' ignored — no active output session");
return;
};
let capture_event = CaptureEvent {
name: event,
payload: DictationCapturePayload { session_id },
};
let Ok(mut capture) = flags.capture.lock() else {
log::warn!("Dictation capture state lock poisoned");
return;
@@ -94,7 +134,7 @@ pub fn dispatch_dictation_capture(app: &tauri::AppHandle, action: &str) {
// A press that reaches Rust but produces no recording is otherwise
// indistinguishable from one the compositor never delivered, so say
// which side of the handshake the press left on.
if let Err(error) = app.emit(event, ()) {
if let Err(error) = app.emit(event, capture_event.payload) {
log::warn!("Dictation capture '{action}' could not emit {event}: {error}");
} else {
log::info!("Dictation capture '{action}' emitted as {event}");
@@ -103,21 +143,35 @@ pub fn dispatch_dictation_capture(app: &tauri::AppHandle, action: &str) {
log::warn!(
"Dictation capture '{action}' queued — the capture window has not registered yet"
);
capture.pending = Some(action.to_owned());
capture.pending.push_back(capture_event);
}
}
#[cfg(test)]
mod dictation_capture_tests {
use super::dictation_capture_event;
use super::{
dictation_capture_event, CaptureDispatchState, CaptureEvent, DictationCapturePayload,
};
#[test]
fn toggle_starts_when_idle_and_stops_when_recording() {
assert_eq!(dictation_capture_event("toggle", false), "tray-dictate");
assert_eq!(
dictation_capture_event("toggle", true),
"tray-dictate-stop"
);
assert_eq!(dictation_capture_event("toggle", true), "tray-dictate-stop");
}
#[test]
fn readiness_queue_preserves_press_then_release() {
let mut state = CaptureDispatchState::default();
state.pending.push_back(CaptureEvent {
name: "tray-dictate",
payload: DictationCapturePayload { session_id: 7 },
});
state.pending.push_back(CaptureEvent {
name: "tray-dictate-stop",
payload: DictationCapturePayload { session_id: 7 },
});
let names: Vec<_> = state.pending.into_iter().map(|event| event.name).collect();
assert_eq!(names, ["tray-dictate", "tray-dictate-stop"]);
}
}
@@ -374,12 +428,19 @@ mod pill_noactivate_tests {
WS_EX_NOACTIVATE_BIT,
"NOACTIVATE bit must be set"
);
assert_eq!(updated & topmost, topmost, "pre-existing style bits must survive");
assert_eq!(
updated & topmost,
topmost,
"pre-existing style bits must survive"
);
}
#[test]
fn idempotent_if_already_noactivate() {
assert_eq!(with_noactivate_style(WS_EX_NOACTIVATE_BIT), WS_EX_NOACTIVATE_BIT);
assert_eq!(
with_noactivate_style(WS_EX_NOACTIVATE_BIT),
WS_EX_NOACTIVATE_BIT
);
}
#[test]
@@ -408,7 +469,11 @@ pub fn run() {
if pill_mode {
log::info!(
"Starting in pill (dictation-only) mode (source: {})",
if cli_pill { "--pill flag" } else { "config.launch_as_widget" }
if cli_pill {
"--pill flag"
} else {
"config.launch_as_widget"
}
);
// On macOS, hide the Dock icon in pill mode so only the tray shows.
// This is handled after the app builds via set_activation_policy.
@@ -476,7 +541,11 @@ pub fn run() {
commands::read_log_tail,
commands::hf_cache_scan,
commands::simulate_paste,
commands::copy_dictation_output_session,
commands::simulate_type,
commands::activate_dictation_output_session,
commands::reject_dictation_output_session,
commands::finish_dictation_output_session,
commands::check_accessibility,
commands::open_accessibility_settings,
commands::check_microphone,
@@ -562,6 +631,7 @@ pub fn run() {
.decorations(false)
.always_on_top(true)
.visible(false)
.focused(false)
.skip_taskbar(true)
.center()
// Stamp the window's identity BEFORE any app script runs.
@@ -586,15 +656,23 @@ pub fn run() {
if let Ok(win) = &result {
mark_pill_noactivate(win);
}
// Wayland cannot reactivate an arbitrary foreign client. The
// GTK toplevel therefore must never accept focus when mapped.
#[cfg(target_os = "linux")]
if let Ok(win) = &result {
use gtk::prelude::GtkWindowExt;
if let Ok(gtk_window) = win.gtk_window() {
gtk_window.set_accept_focus(false);
gtk_window.set_focus_on_map(false);
}
}
}
app.manage(AppFlags {
quitting: AtomicBool::new(false),
dictating: AtomicBool::new(false),
capture: Mutex::new(CaptureDispatchState {
ready: false,
pending: None,
}),
capture: Mutex::new(CaptureDispatchState::default()),
output: dictation_output::DictationOutput::default(),
});
app.manage(TrayHandle {
tray: Mutex::new(None),
@@ -700,6 +778,20 @@ pub fn run() {
.icon(app.default_window_icon().unwrap().clone())
.menu(&tray_menu)
.tooltip(if pill_mode_tray { "VoiceStudio Dictation" } else { "VoiceStudio" })
.on_tray_icon_event(|tray, event| {
if matches!(
event,
tauri::tray::TrayIconEvent::Click {
button_state: tauri::tray::MouseButtonState::Down,
..
}
) {
tray.app_handle()
.state::<AppFlags>()
.output
.prime_tray_target();
}
})
.on_menu_event(move |app, event| {
match event.id().as_ref() {
"show" => {
@@ -760,9 +852,9 @@ pub fn run() {
// current by the frontend's existing
// `set_tray_recording` call on every start and stop.
if app.state::<AppFlags>().dictating.load(Ordering::SeqCst) {
dispatch_dictation_capture(app, "stop");
dispatch_dictation_capture_from(app, "stop", CaptureOrigin::Tray);
} else {
dispatch_dictation_capture(app, "start");
dispatch_dictation_capture_from(app, "start", CaptureOrigin::Tray);
}
}
"settings" => {
frontend/src-tauri/tests/backend_lifecycle.rs
+2 -1
View File
@@ -217,7 +217,8 @@ impl TestApp {
app.manage(AppFlags {
quitting: AtomicBool::new(false),
dictating: AtomicBool::new(false),
capture: Mutex::new(CaptureDispatchState { ready: false, pending: None }),
capture: Mutex::new(CaptureDispatchState::default()),
output: app_lib::dictation_output::DictationOutput::default(),
});
let stage = Arc::new(Mutex::new(BootstrapStage::Checking));
let logs: Arc<Mutex<Vec<LogPayload>>> = Arc::new(Mutex::new(Vec::new()));
frontend/src/components/CaptureWidget.jsx
+648 -151
View File
File diff suppressed because it is too large Load Diff
frontend/src/components/CaptureWidget.test.jsx
+982 -35
View File
File diff suppressed because it is too large Load Diff
frontend/src/i18n/locales/ar.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "جارٍ النسخ…",
"a11y_setup": "اسمح بتسهيلات الاستخدام حتى يتمكّن الإملاء من الكتابة نيابةً عنك",
"pasted": "تم لصقه",
"inserted": "تم الإدخال",
"copied": "تم النسخ إلى الحافظة",
"no_speech": "لم يتم اكتشاف أي كلام",
"mic_denied": "تم رفض الوصول إلى الميكروفون",
"mic_denied_toast": "تم رفض الوصول إلى الميكروفون. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "حذف نموذج الكلام",
"engine_unavailable": "محرك الإملاء المباشر غير متوفر في هذا التثبيت. يعود الإملاء إلى مسار النسخ القياسي.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "موصى به. إملاء سريع ودقيق عبر 25 لغة أوروبية.",
"sherpa-parakeet-tdt-v3": "إملاء سريع ودقيق عبر 25 لغة أوروبية.",
"sherpa-parakeet-tdt-v2": "إملاء سريع ودقيق باللغة الإنجليزية فقط.",
"sherpa-zipformer-bilingual-zh-en": "بث الصينية + الإنجليزية مع أجزاء حية.",
"sherpa-paraformer-bilingual-zh-en": "تدفق الصينية + الإنجليزية، نموذج مدمج.",
"sherpa-zipformer-en-20m": "نموذج صغير للبث باللغة الإنجليزية – أقل زمن وصول.",
"sherpa-zipformer-zh-14m": "النموذج الصيني المتدفق الصغير – أقل زمن وصول.",
"sherpa-whisper-tiny": "متعدد اللغات (+90 لغة) مع الكشف التلقائي عن اللغة."
"sherpa-whisper-tiny": "موصى به. إملاء متعدد اللغات لأكثر من 90 لغة مع اكتشاف اللغة تلقائيًا."
}
},
"profiles": {
frontend/src/i18n/locales/de.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Transkribieren…",
"a11y_setup": "Bedienungshilfen erlauben, damit das Diktat für Sie tippen kann",
"pasted": "Eingefügt",
"inserted": "Eingegeben",
"copied": "In die Zwischenablage kopiert",
"no_speech": "Keine Sprache erkannt",
"mic_denied": "Mikrofonzugriff verweigert",
"mic_denied_toast": "Mikrofonzugriff verweigert. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Sprachmodell löschen",
"engine_unavailable": "Die Live-Diktier-Engine ist bei dieser Installation nicht verfügbar. Beim Diktat wird auf den Standard-Transkriptionspfad zurückgegriffen.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Empfohlen. Schnelles und genaues Diktieren in 25 europäischen Sprachen.",
"sherpa-parakeet-tdt-v3": "Schnelles und genaues Diktieren in 25 europäischen Sprachen.",
"sherpa-parakeet-tdt-v2": "Schnelles, genaues Diktat nur auf Englisch.",
"sherpa-zipformer-bilingual-zh-en": "Streaming Chinesisch + Englisch mit Live-Teilabschnitten.",
"sherpa-paraformer-bilingual-zh-en": "Streaming Chinesisch + Englisch, kompaktes Modell.",
"sherpa-zipformer-en-20m": "Winziges englisches Streaming-Modell – niedrigste Latenz.",
"sherpa-zipformer-zh-14m": "Winziges chinesisches Streaming-Modell – niedrigste Latenz.",
"sherpa-whisper-tiny": "Mehrsprachig (über 90 Sprachen) mit automatischer Spracherkennung."
"sherpa-whisper-tiny": "Empfohlen. Mehrsprachiges Diktieren in über 90 Sprachen mit automatischer Spracherkennung."
}
},
"profiles": {
frontend/src/i18n/locales/en.json
+3 -2
View File
@@ -1038,6 +1038,7 @@
"listening_label": "Listening…",
"transcribing_label": "Transcribing…",
"pasted": "Pasted",
"inserted": "Inserted",
"copied": "Copied to clipboard",
"no_speech": "No speech detected",
"model_downloading": "Downloading voice model…",
@@ -1080,13 +1081,13 @@
"delete_confirm_title": "Delete speech model",
"engine_unavailable": "The live-dictation engine isn't available on this install. Dictation falls back to the standard transcription path.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Recommended. Fast, accurate dictation across 25 European languages.",
"sherpa-parakeet-tdt-v3": "Fast, accurate dictation across 25 European languages.",
"sherpa-parakeet-tdt-v2": "Fast, accurate English-only dictation.",
"sherpa-zipformer-bilingual-zh-en": "Streaming Chinese + English with live partials.",
"sherpa-paraformer-bilingual-zh-en": "Streaming Chinese + English, compact model.",
"sherpa-zipformer-en-20m": "Tiny streaming English model — lowest latency.",
"sherpa-zipformer-zh-14m": "Tiny streaming Chinese model — lowest latency.",
"sherpa-whisper-tiny": "Multilingual (90+ languages) with auto language detection."
"sherpa-whisper-tiny": "Recommended. Multilingual dictation across 90+ languages with automatic language detection."
}
},
"profiles": {
frontend/src/i18n/locales/es.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Transcribiendo…",
"a11y_setup": "Permite Accesibilidad para que el dictado pueda escribir por ti",
"pasted": "Pegado",
"inserted": "Insertado",
"copied": "Copiado al portapapeles",
"no_speech": "No se detectó voz",
"mic_denied": "Acceso al micrófono denegado",
"mic_denied_toast": "Acceso al micrófono denegado. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Eliminar modelo de voz",
"engine_unavailable": "El motor de dictado en vivo no está disponible en esta instalación. El dictado vuelve a la ruta de transcripción estándar.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Recomendado. Dictado rápido y preciso en 25 idiomas europeos.",
"sherpa-parakeet-tdt-v3": "Dictado rápido y preciso en 25 idiomas europeos.",
"sherpa-parakeet-tdt-v2": "Dictado rápido y preciso solo en inglés.",
"sherpa-zipformer-bilingual-zh-en": "Streaming chino + inglés con parciales en vivo.",
"sherpa-paraformer-bilingual-zh-en": "Streaming chino + inglés, modelo compacto.",
"sherpa-zipformer-en-20m": "Pequeño modelo de transmisión en inglés: latencia más baja.",
"sherpa-zipformer-zh-14m": "Pequeño modelo chino de transmisión: latencia más baja.",
"sherpa-whisper-tiny": "Multilingüe (más de 90 idiomas) con detección automática de idioma."
"sherpa-whisper-tiny": "Recomendado. Dictado multilingüe en más de 90 idiomas con detección automática del idioma."
}
},
"profiles": {
frontend/src/i18n/locales/fr.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Transcription…",
"a11y_setup": "Autorisez l'accessibilité pour que la dictée puisse écrire à votre place",
"pasted": "Collé",
"inserted": "Inséré",
"copied": "Copié dans le presse-papiers",
"no_speech": "Aucune parole détectée",
"mic_denied": "Accès au micro refusé",
"mic_denied_toast": "Accès au microphone refusé. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Supprimer le modèle vocal",
"engine_unavailable": "Le moteur de dictée en direct n'est pas disponible sur cette installation. La dictée revient au chemin de transcription standard.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Recommandé. Dictée rapide et précise dans 25 langues européennes.",
"sherpa-parakeet-tdt-v3": "Dictée rapide et précise dans 25 langues européennes.",
"sherpa-parakeet-tdt-v2": "Dictée rapide et précise en anglais uniquement.",
"sherpa-zipformer-bilingual-zh-en": "Streaming chinois + anglais avec partiels en direct.",
"sherpa-paraformer-bilingual-zh-en": "Streaming chinois + anglais, modèle compact.",
"sherpa-zipformer-en-20m": "Petit modèle anglais de streaming – latence la plus faible.",
"sherpa-zipformer-zh-14m": "Petit modèle chinois de streaming – latence la plus faible.",
"sherpa-whisper-tiny": "Multilingue (plus de 90 langues) avec détection automatique de la langue."
"sherpa-whisper-tiny": "Recommandé. Dictée multilingue dans plus de 90 langues avec détection automatique de la langue."
}
},
"profiles": {
frontend/src/i18n/locales/hi.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "प्रतिलेखन...",
"a11y_setup": "एक्सेसिबिलिटी की अनुमति दें ताकि श्रुतलेख आपके लिए टाइप कर सके",
"pasted": "चिपकाया गया",
"inserted": "दर्ज किया गया",
"copied": "क्लिपबोर्ड पर कॉपी किया गया",
"no_speech": "कोई भाषण नहीं मिला",
"mic_denied": "माइक का उपयोग अस्वीकृत",
"mic_denied_toast": "माइक्रोफ़ोन पहुंच अस्वीकृत. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "वाक् मॉडल हटाएँ",
"engine_unavailable": "इस इंस्टाल पर लाइव-डिक्टेशन इंजन उपलब्ध नहीं है। श्रुतलेखन मानक प्रतिलेखन पथ पर वापस आ जाता है।",
"model_desc": {
"sherpa-parakeet-tdt-v3": "अनुशंसित. 25 यूरोपीय भाषाओं में तेज़, सटीक श्रुतलेख।",
"sherpa-parakeet-tdt-v3": "25 यूरोपीय भाषाओं में तेज़ और सटीक श्रुतलेख।",
"sherpa-parakeet-tdt-v2": "तेज़, सटीक केवल अंग्रेज़ी श्रुतलेख।",
"sherpa-zipformer-bilingual-zh-en": "लाइव आंशिक भाग के साथ चीनी + अंग्रेजी स्ट्रीमिंग।",
"sherpa-paraformer-bilingual-zh-en": "स्ट्रीमिंग चीनी + अंग्रेजी, कॉम्पैक्ट मॉडल।",
"sherpa-zipformer-en-20m": "छोटा स्ट्रीमिंग अंग्रेजी मॉडल - सबसे कम विलंबता।",
"sherpa-zipformer-zh-14m": "छोटा स्ट्रीमिंग चीनी मॉडल - सबसे कम विलंबता।",
"sherpa-whisper-tiny": "ऑटो भाषा पहचान के साथ बहुभाषी (90+ भाषाएँ)।"
"sherpa-whisper-tiny": "अनुशंसित। स्वचालित भाषा पहचान के साथ 90 से अधिक भाषाओं में बहुभाषी श्रुतलेख।"
}
},
"profiles": {
frontend/src/i18n/locales/id.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Mentranskripsikan…",
"a11y_setup": "Izinkan Aksesibilitas agar dikte dapat mengetik untuk Anda",
"pasted": "Ditempel",
"inserted": "Dimasukkan",
"copied": "Disalin ke papan klip",
"no_speech": "Tidak ada ucapan yang terdeteksi",
"mic_denied": "Akses mikrofon ditolak",
"mic_denied_toast": "Akses mikrofon ditolak. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Hapus model ucapan",
"engine_unavailable": "Mesin pendiktean langsung tidak tersedia pada instalasi ini. Dikte kembali ke jalur transkripsi standar.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Direkomendasikan. Dikte yang cepat dan akurat dalam 25 bahasa Eropa.",
"sherpa-parakeet-tdt-v3": "Dikte cepat dan akurat dalam 25 bahasa Eropa.",
"sherpa-parakeet-tdt-v2": "Dikte khusus bahasa Inggris yang cepat dan akurat.",
"sherpa-zipformer-bilingual-zh-en": "Streaming bahasa Mandarin + Inggris dengan siaran langsung sebagian.",
"sherpa-paraformer-bilingual-zh-en": "Streaming bahasa Mandarin + Inggris, model ringkas.",
"sherpa-zipformer-en-20m": "Model streaming bahasa Inggris yang kecil — latensi terendah.",
"sherpa-zipformer-zh-14m": "Model streaming kecil Tiongkok — latensi terendah.",
"sherpa-whisper-tiny": "Multibahasa (90+ bahasa) dengan deteksi bahasa otomatis."
"sherpa-whisper-tiny": "Direkomendasikan. Dikte multibahasa dalam lebih dari 90 bahasa dengan deteksi bahasa otomatis."
}
},
"profiles": {
frontend/src/i18n/locales/it.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Trascrizione…",
"a11y_setup": "Consenti Accessibilità così la dettatura può digitare per te",
"pasted": "Incollato",
"inserted": "Inserito",
"copied": "Copiato negli appunti",
"no_speech": "Nessun parlato rilevato",
"mic_denied": "Accesso al microfono negato",
"mic_denied_toast": "Accesso al microfono negato. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Elimina modello vocale",
"engine_unavailable": "Il motore di dettatura dal vivo non è disponibile su questa installazione. La dettatura ritorna al percorso di trascrizione standard.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Consigliato. Dettatura rapida e accurata in 25 lingue europee.",
"sherpa-parakeet-tdt-v3": "Dettatura rapida e accurata in 25 lingue europee.",
"sherpa-parakeet-tdt-v2": "Dettatura veloce e accurata solo in inglese.",
"sherpa-zipformer-bilingual-zh-en": "Streaming cinese + inglese con parziali dal vivo.",
"sherpa-paraformer-bilingual-zh-en": "Streaming cinese + inglese, modello compatto.",
"sherpa-zipformer-en-20m": "Piccolo modello inglese di streaming: latenza più bassa.",
"sherpa-zipformer-zh-14m": "Piccolo modello cinese in streaming: latenza più bassa.",
"sherpa-whisper-tiny": "Multilingue (oltre 90 lingue) con rilevamento automatico della lingua."
"sherpa-whisper-tiny": "Consigliato. Dettatura multilingue in oltre 90 lingue con rilevamento automatico della lingua."
}
},
"profiles": {
frontend/src/i18n/locales/ja.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "文字起こし中…",
"a11y_setup": "音声入力が代わりに入力できるよう、アクセシビリティを許可してください",
"pasted": "貼り付けた",
"inserted": "入力しました",
"copied": "クリップボードにコピーしました",
"no_speech": "音声が検出されませんでした",
"mic_denied": "マイクアクセスが拒否されました",
"mic_denied_toast": "マイクへのアクセスが拒否されました。 {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "音声モデルの削除",
"engine_unavailable": "このインストールではライブディクテーション エンジンは利用できません。ディクテーションは標準の文字起こしパスにフォールバックします。",
"model_desc": {
"sherpa-parakeet-tdt-v3": "おすすめです。 25 のヨーロッパ言語にわたる高速かつ正確なディクテーション。",
"sherpa-parakeet-tdt-v3": "25のヨーロッパ言語に対応した高速で正確な音声入力。",
"sherpa-parakeet-tdt-v2": "高速かつ正確な英語のみのディクテーション。",
"sherpa-zipformer-bilingual-zh-en": "中国語 + 英語のライブ部分をストリーミングします。",
"sherpa-paraformer-bilingual-zh-en": "中国語+英語ストリーミング、コンパクトモデル。",
"sherpa-zipformer-en-20m": "小さなストリーミング英語モデル — レイテンシが最も低い。",
"sherpa-zipformer-zh-14m": "小さなストリーミング中国語モデル - 遅延が最も低い。",
"sherpa-whisper-tiny": "自動言語検出機能を備えた多言語 (90 以上の言語)。"
"sherpa-whisper-tiny": "おすすめ。90以上の言語に対応し、言語を自動検出する多言語音声入力。"
}
},
"profiles": {
frontend/src/i18n/locales/ko.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "스크립트 작성 중…",
"a11y_setup": "받아쓰기가 대신 입력할 수 있도록 손쉬운 사용을 허용하세요",
"pasted": "붙여넣음",
"inserted": "입력됨",
"copied": "클립보드에 복사됨",
"no_speech": "음성이 감지되지 않았습니다.",
"mic_denied": "마이크 액세스가 거부되었습니다.",
"mic_denied_toast": "마이크 액세스가 거부되었습니다. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "음성 모델 삭제",
"engine_unavailable": "The live-dictation engine isn't available on this install. Dictation falls back to the standard transcription path.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "추천합니다. Fast, accurate dictation across 25 European languages.",
"sherpa-parakeet-tdt-v3": "25개 유럽 언어를 지원하는 빠르고 정확한 받아쓰기입니다.",
"sherpa-parakeet-tdt-v2": "빠르고 정확한 영어 전용 받아쓰기.",
"sherpa-zipformer-bilingual-zh-en": "라이브 부분으로 중국어 + 영어 스트리밍.",
"sherpa-paraformer-bilingual-zh-en": "중국어 + 영어 스트리밍, 컴팩트 모델.",
"sherpa-zipformer-en-20m": "작은 스트리밍 영어 모델 - 지연 시간이 가장 낮습니다.",
"sherpa-zipformer-zh-14m": "작은 스트리밍 중국 모델 - 지연 시간이 가장 낮습니다.",
"sherpa-whisper-tiny": "Multilingual (90+ languages) with auto language detection."
"sherpa-whisper-tiny": "추천. 90개 이상의 언어를 지원하며 언어를 자동으로 감지하는 다국어 받아쓰기입니다."
}
},
"profiles": {
frontend/src/i18n/locales/nl.json
+6 -4
View File
@@ -350,7 +350,7 @@
"lines_one": "{{count}} regel",
"lines_other": "{{count}} regels",
"copy": "Kopiëren",
"copied": "Gekonieerd!",
"copied": "Gekopieerd!",
"waiting_output": "Wachten op uitvoer…",
"auto_detect": "Automatisch detecteren",
"suggest_lang": "Overschakelen naar het Nederlands?",
@@ -785,6 +785,8 @@
"transcribing_label": "Transcriberen…",
"a11y_setup": "Sta Toegankelijkheid toe zodat dicteren voor je kan typen",
"pasted": "Geplakt",
"inserted": "Ingevoegd",
"copied": "Gekopieerd naar klembord",
"no_speech": "Geen spraak gedetecteerd",
"mic_denied": "Microfoontoegang geweigerd",
"mic_denied_toast": "Microfoontoegang geweigerd. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Spraakmodel verwijderen",
"engine_unavailable": "De live-dicteerengine is niet beschikbaar bij deze installatie. Het dicteren valt terug op het standaard transcriptiepad.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Aanbevolen. Snel en nauwkeurig dicteren in 25 Europese talen.",
"sherpa-parakeet-tdt-v3": "Snel en nauwkeurig dicteren in 25 Europese talen.",
"sherpa-parakeet-tdt-v2": "Snel, nauwkeurig dicteren in het Engels.",
"sherpa-zipformer-bilingual-zh-en": "Streaming Chinees + Engels met live gedeeltelijke beelden.",
"sherpa-zipformer-bilingual-zh-en": "Streaming Chinees + Engels met live tussentijdse resultaten.",
"sherpa-paraformer-bilingual-zh-en": "Streaming Chinees + Engels, compact model.",
"sherpa-zipformer-en-20m": "Klein streaming Engels model - laagste latentie.",
"sherpa-zipformer-zh-14m": "Klein Chinees streamingmodel – laagste latentie.",
"sherpa-whisper-tiny": "Meertalig (90+ talen) met automatische taaldetectie."
"sherpa-whisper-tiny": "Aanbevolen. Meertalig dicteren in meer dan 90 talen met automatische taaldetectie."
}
},
"profiles": {
frontend/src/i18n/locales/pl.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Transkrypcja…",
"a11y_setup": "Zezwól na Dostępność, aby dyktowanie mogło pisać za Ciebie",
"pasted": "Wklejony",
"inserted": "Wstawiono",
"copied": "Skopiowano do schowka",
"no_speech": "Nie wykryto mowy",
"mic_denied": "Odmowa dostępu do mikrofonu",
"mic_denied_toast": "Odmowa dostępu do mikrofonu. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Usuń model mowy",
"engine_unavailable": "Mechanizm dyktowania na żywo nie jest dostępny w tej instalacji. Dyktowanie wraca do standardowej ścieżki transkrypcji.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Zalecane. Szybkie i dokładne dyktowanie w 25 językach europejskich.",
"sherpa-parakeet-tdt-v3": "Szybkie i dokładne dyktowanie w 25 językach europejskich.",
"sherpa-parakeet-tdt-v2": "Szybkie i dokładne dyktowanie wyłącznie w języku angielskim.",
"sherpa-zipformer-bilingual-zh-en": "Transmisja strumieniowa języka chińskiego i angielskiego z fragmentami na żywo.",
"sherpa-paraformer-bilingual-zh-en": "Przesyłanie strumieniowe w języku chińskim i angielskim, model kompaktowy.",
"sherpa-zipformer-en-20m": "Mały, angielski model przesyłania strumieniowego — najniższe opóźnienie.",
"sherpa-zipformer-zh-14m": "Mały chiński model do przesyłania strumieniowego — najniższe opóźnienie.",
"sherpa-whisper-tiny": "Wielojęzyczny (ponad 90 języków) z automatycznym wykrywaniem języka."
"sherpa-whisper-tiny": "Zalecane. Wielojęzyczne dyktowanie w ponad 90 językach z automatycznym wykrywaniem języka."
}
},
"profiles": {
frontend/src/i18n/locales/pt.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Transcrevendo…",
"a11y_setup": "Permita Acessibilidade para que o ditado possa digitar por você",
"pasted": "Colado",
"inserted": "Inserido",
"copied": "Copiado para a área de transferência",
"no_speech": "Nenhuma fala detectada",
"mic_denied": "Acesso ao microfone negado",
"mic_denied_toast": "Acesso ao microfone negado. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Excluir modelo de fala",
"engine_unavailable": "O mecanismo de ditado ao vivo não está disponível nesta instalação. O ditado volta ao caminho de transcrição padrão.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Recomendado. Ditado rápido e preciso em 25 idiomas europeus.",
"sherpa-parakeet-tdt-v3": "Ditado rápido e preciso em 25 idiomas europeus.",
"sherpa-parakeet-tdt-v2": "Ditado rápido e preciso somente em inglês.",
"sherpa-zipformer-bilingual-zh-en": "Streaming de chinês + inglês com parciais ao vivo.",
"sherpa-paraformer-bilingual-zh-en": "Streaming Chinês + Inglês, modelo compacto.",
"sherpa-zipformer-en-20m": "Modelo inglês de streaming minúsculo – latência mais baixa.",
"sherpa-zipformer-zh-14m": "Modelo chinês de streaming minúsculo – latência mais baixa.",
"sherpa-whisper-tiny": "Multilíngue (mais de 90 idiomas) com detecção automática de idioma."
"sherpa-whisper-tiny": "Recomendado. Ditado multilíngue em mais de 90 idiomas com detecção automática de idioma."
}
},
"profiles": {
frontend/src/i18n/locales/ru.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Расшифровка…",
"a11y_setup": "Разрешите Универсальный доступ, чтобы диктовка могла печатать за вас",
"pasted": "Вставлено",
"inserted": "Введено",
"copied": "Скопировано в буфер обмена",
"no_speech": "Речь не обнаружена",
"mic_denied": "Доступ к микрофону запрещен",
"mic_denied_toast": "Доступ к микрофону запрещен. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Удалить речевую модель",
"engine_unavailable": "Механизм живой диктовки недоступен в этой установке. Диктовка возвращается к стандартному пути транскрипции.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Рекомендуется. Быстрая и точная диктовка на 25 европейских языках.",
"sherpa-parakeet-tdt-v3": "Быстрая и точная диктовка на 25 европейских языках.",
"sherpa-parakeet-tdt-v2": "Быстрый и точный диктант только на английском языке.",
"sherpa-zipformer-bilingual-zh-en": "Потоковое вещание на китайском и английском языках с живыми фрагментами.",
"sherpa-paraformer-bilingual-zh-en": "Потоковое вещание на китайском и английском языках, компактная модель.",
"sherpa-zipformer-en-20m": "Миниатюрная потоковая английская модель — минимальная задержка.",
"sherpa-zipformer-zh-14m": "Миниатюрная потоковая китайская модель — самая низкая задержка.",
"sherpa-whisper-tiny": "Многоязычный (более 90 языков) с автоматическим определением языка."
"sherpa-whisper-tiny": "Рекомендуется. Многоязычная диктовка на более чем 90 языках с автоматическим определением языка."
}
},
"profiles": {
frontend/src/i18n/locales/sv.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Transkriberar...",
"a11y_setup": "Tillåt Hjälpmedel så att diktering kan skriva åt dig",
"pasted": "Klistras in",
"inserted": "Infogat",
"copied": "Kopierat till urklipp",
"no_speech": "Inget tal upptäckt",
"mic_denied": "Mikrofonåtkomst nekad",
"mic_denied_toast": "Mikrofonåtkomst nekad. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Ta bort talmodell",
"engine_unavailable": "Live-dikteringsmotorn är inte tillgänglig på den här installationen. Diktering faller tillbaka till standardtranskriptionsvägen.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Rekommenderas. Snabb, exakt diktering på 25 europeiska språk.",
"sherpa-parakeet-tdt-v3": "Snabb och exakt diktering på 25 europeiska språk.",
"sherpa-parakeet-tdt-v2": "Snabb, exakt diktering endast på engelska.",
"sherpa-zipformer-bilingual-zh-en": "Strömmande kinesiska + engelska med livepartialer.",
"sherpa-paraformer-bilingual-zh-en": "Streaming kinesiska + engelska, kompakt modell.",
"sherpa-zipformer-en-20m": "Liten strömmande engelsk modell — lägsta latens.",
"sherpa-zipformer-zh-14m": "Liten strömmande kinesisk modell — lägsta latens.",
"sherpa-whisper-tiny": "Flerspråkig (90+ språk) med automatisk språkdetektering."
"sherpa-whisper-tiny": "Rekommenderas. Flerspråkig diktering på över 90 språk med automatisk språkidentifiering."
}
},
"profiles": {
frontend/src/i18n/locales/th.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "กำลังถอดเสียง...",
"a11y_setup": "อนุญาตการช่วยการเข้าถึงเพื่อให้การป้อนตามคำบอกพิมพ์แทนคุณได้",
"pasted": "วางแล้ว",
"inserted": "แทรกแล้ว",
"copied": "คัดลอกไปยังคลิปบอร์ดแล้ว",
"no_speech": "ไม่พบคำพูด",
"mic_denied": "การเข้าถึงไมค์ถูกปฏิเสธ",
"mic_denied_toast": "การเข้าถึงไมโครโฟนถูกปฏิเสธ {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "ลบโมเดลคำพูด",
"engine_unavailable": "กลไกการเขียนตามคำบอกสดไม่พร้อมใช้งานในการติดตั้งนี้ การเขียนตามคำบอกจะกลับไปใช้เส้นทางการถอดเสียงมาตรฐาน",
"model_desc": {
"sherpa-parakeet-tdt-v3": "แนะนำ. การเขียนตามคำบอกที่รวดเร็วและแม่นยำใน 25 ภาษายุโรป",
"sherpa-parakeet-tdt-v3": "การเขียนตามคำบอกที่รวดเร็วและแม่นยำใน 25 ภาษายุโรป",
"sherpa-parakeet-tdt-v2": "การเขียนตามคำบอกภาษาอังกฤษเท่านั้นที่รวดเร็วและแม่นยำ",
"sherpa-zipformer-bilingual-zh-en": "สตรีมมิ่งภาษาจีน + อังกฤษพร้อมถ่ายทอดสดบางส่วน",
"sherpa-paraformer-bilingual-zh-en": "สตรีมมิ่งภาษาจีน+อังกฤษ รุ่นกะทัดรัด",
"sherpa-zipformer-en-20m": "โมเดลสตรีมมิ่งภาษาอังกฤษขนาดเล็ก — เวลาแฝงต่ำที่สุด",
"sherpa-zipformer-zh-14m": "โมเดลสตรีมมิ่งจีนขนาดเล็ก — เวลาแฝงต่ำที่สุด",
"sherpa-whisper-tiny": "หลายภาษา (90+ ภาษา) พร้อมการตรวจจับภาษาอัตโนมัติ"
"sherpa-whisper-tiny": "แนะนำ การเขียนตามคำบอกหลายภาษากว่า 90 ภาษา พร้อมการตรวจจับภาษาอัตโนมัติ"
}
},
"profiles": {
frontend/src/i18n/locales/tr.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Metne dönüştürülüyor…",
"a11y_setup": "Dikte sizin yerinize yazabilsin diye Erişilebilirlik'e izin verin",
"pasted": "Yapıştırıldı",
"inserted": "Eklendi",
"copied": "Panoya kopyalandı",
"no_speech": "Konuşma algılanmadı",
"mic_denied": "Mikrofon erişimi reddedildi",
"mic_denied_toast": "Mikrofon erişimi reddedildi. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Konuşma modelini sil",
"engine_unavailable": "Canlı dikte motoru bu kurulumda mevcut değil. Dikte, standart transkripsiyon yoluna geri döner.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Tavsiye edilir. 25 Avrupa dilinde hızlı, doğru dikte.",
"sherpa-parakeet-tdt-v3": "25 Avrupa dilinde hızlı ve doğru dikte.",
"sherpa-parakeet-tdt-v2": "Hızlı, doğru yalnızca İngilizce dikte.",
"sherpa-zipformer-bilingual-zh-en": "Canlı bölümlerle Çince + İngilizce akışı.",
"sherpa-paraformer-bilingual-zh-en": "Çince + İngilizce akışı, kompakt model.",
"sherpa-zipformer-en-20m": "Küçük akışlı İngilizce modeli — en düşük gecikme.",
"sherpa-zipformer-zh-14m": "Küçük akışlı Çin modeli — en düşük gecikme.",
"sherpa-whisper-tiny": "Otomatik dil algılamalı çok dilli (90'dan fazla dil)."
"sherpa-whisper-tiny": "Tavsiye edilir. Otomatik dil algılama ile 90'dan fazla dilde çok dilli dikte."
}
},
"profiles": {
frontend/src/i18n/locales/uk.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Транскрибування…",
"a11y_setup": "Дозвольте Доступність, щоб диктування могло друкувати за вас",
"pasted": "Вставив",
"inserted": "Введено",
"copied": "Скопійовано в буфер обміну",
"no_speech": "Мовлення не виявлено",
"mic_denied": "Доступ до мікрофона заборонено",
"mic_denied_toast": "Доступ до мікрофона заборонено. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Видалити модель мовлення",
"engine_unavailable": "Система живого диктування недоступна під час цієї інсталяції. Диктування повертається до стандартного шляху транскрипції.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Рекомендовано. Швидкий і точний диктант 25 європейськими мовами.",
"sherpa-parakeet-tdt-v3": "Швидке й точне диктування 25 європейськими мовами.",
"sherpa-parakeet-tdt-v2": "Швидкий і точний диктант лише англійською мовою.",
"sherpa-zipformer-bilingual-zh-en": "Потокова трансляція китайською та англійською мовами з живими частками.",
"sherpa-paraformer-bilingual-zh-en": "Потоковий китайський + англійський, компактна модель.",
"sherpa-zipformer-en-20m": "Маленька потокова англійська модель — найменша затримка.",
"sherpa-zipformer-zh-14m": "Маленька потокова китайська модель — найменша затримка.",
"sherpa-whisper-tiny": "Багатомовний (90+ мов) з автоматичним визначенням мови."
"sherpa-whisper-tiny": "Рекомендовано. Багатомовне диктування понад 90 мовами з автоматичним визначенням мови."
}
},
"profiles": {
frontend/src/i18n/locales/vi.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "Phiên âm…",
"a11y_setup": "Cho phép Trợ năng để đọc chính tả có thể gõ thay bạn",
"pasted": "Đã dán",
"inserted": "Đã nhập",
"copied": "Đã sao chép vào bảng nhớ tạm",
"no_speech": "Không phát hiện thấy giọng nói nào",
"mic_denied": "Quyền truy cập micrô bị từ chối",
"mic_denied_toast": "Quyền truy cập micrô bị từ chối. {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "Xóa mẫu giọng nói",
"engine_unavailable": "Công cụ đọc chính tả trực tiếp không khả dụng trên bản cài đặt này. Đọc chính tả quay trở lại đường dẫn phiên âm tiêu chuẩn.",
"model_desc": {
"sherpa-parakeet-tdt-v3": "Khuyến nghị. Đọc chính tả nhanh, chính xác trên 25 ngôn ngữ Châu Âu.",
"sherpa-parakeet-tdt-v3": "Đọc chính tả nhanh, chính xác bằng 25 ngôn ngữ châu Âu.",
"sherpa-parakeet-tdt-v2": "Đọc chính tả chỉ bằng tiếng Anh nhanh chóng, chính xác.",
"sherpa-zipformer-bilingual-zh-en": "Truyền phát tiếng Trung + tiếng Anh với các phần trực tiếp.",
"sherpa-paraformer-bilingual-zh-en": "Truyền phát tiếng Trung + tiếng Anh, model nhỏ gọn.",
"sherpa-zipformer-en-20m": "Mô hình phát trực tuyến nhỏ bằng tiếng Anh — độ trễ thấp nhất.",
"sherpa-zipformer-zh-14m": "Mô hình phát trực tuyến nhỏ của Trung Quốc — độ trễ thấp nhất.",
"sherpa-whisper-tiny": "Đa ngôn ngữ (hơn 90 ngôn ngữ) với tính năng tự động phát hiện ngôn ngữ."
"sherpa-whisper-tiny": "Khuyến nghị. Đọc chính tả đa ngôn ngữ với hơn 90 ngôn ngữ và tự động phát hiện ngôn ngữ."
}
},
"profiles": {
frontend/src/i18n/locales/zh-CN.json
+4 -2
View File
@@ -744,6 +744,8 @@
"transcribing_label": "正在抄写…",
"a11y_setup": "允许辅助功能，以便听写为您输入文字",
"pasted": "粘贴的",
"inserted": "已输入",
"copied": "已复制到剪贴板",
"no_speech": "未检测到语音",
"mic_denied": "麦克风访问被拒绝",
"mic_denied_toast": "麦克风访问被拒绝。 {{hint}}",
@@ -2477,13 +2479,13 @@
"delete_confirm_title": "删除语音模型",
"engine_unavailable": "实时听写引擎在此安装中不可用。听写将回退到标准转录流程。",
"model_desc": {
"sherpa-parakeet-tdt-v3": "推荐。快速、准确地听写 25 种欧洲语言。",
"sherpa-parakeet-tdt-v3": "支持 25 种欧洲语言的快速、准确听写。",
"sherpa-parakeet-tdt-v2": "快速、准确的纯英语听写。",
"sherpa-zipformer-bilingual-zh-en": "流式中英，带实时部分结果。",
"sherpa-paraformer-bilingual-zh-en": "流式中英，紧凑型模型。",
"sherpa-zipformer-en-20m": "小型流式英文模型 — 最低延迟。",
"sherpa-zipformer-zh-14m": "小型流式中文模型——最低延迟。",
"sherpa-whisper-tiny": "具有自动语言检测功能的多语言（90 多种语言）。"
"sherpa-whisper-tiny": "推荐。支持 90 多种语言并可自动检测语言的多语言听写。"
}
},
"profiles": {
frontend/src/i18n/locales/zh-TW.json
+4 -2
View File
@@ -785,6 +785,8 @@
"transcribing_label": "正在抄寫…",
"a11y_setup": "允許輔助使用，讓聽寫功能為您輸入文字",
"pasted": "貼上的",
"inserted": "已輸入",
"copied": "已複製到剪貼簿",
"no_speech": "未偵測到語音",
"mic_denied": "麥克風存取被拒絕",
"mic_denied_toast": "麥克風存取被拒絕。 {{hint}}",
@@ -2470,13 +2472,13 @@
"delete_confirm_title": "刪除語音模型",
"engine_unavailable": "即時聽寫引擎在此安裝中不可用。聽寫回到標準轉錄路徑。",
"model_desc": {
"sherpa-parakeet-tdt-v3": "推薦。快速、準確地聽寫 25 種歐洲語言。",
"sherpa-parakeet-tdt-v3": "支援 25 種歐洲語言的快速、準確聽寫。",
"sherpa-parakeet-tdt-v2": "快速、準確的純英語聽寫。",
"sherpa-zipformer-bilingual-zh-en": "串流中文 + 英文，並附有現場部分內容。",
"sherpa-paraformer-bilingual-zh-en": "串流中文+英文，緊湊型。",
"sherpa-zipformer-en-20m": "小型串流英文模型 — 最低延遲。",
"sherpa-zipformer-zh-14m": "小型串流媒體中國模型—最低延遲。",
"sherpa-whisper-tiny": "具有自動語言偵測功能的多語言（90 多種語言）。"
"sherpa-whisper-tiny": "推薦。支援 90 多種語言並可自動偵測語言的多語言聽寫。"
}
},
"profiles": {
frontend/src/store/prefsSlice.ts
+3 -3
View File
@@ -18,7 +18,7 @@ type DictationMode = 'toggle' | 'hold';
/** Default sherpa dictation model id — matches the backend
* `sherpa_dictation.DEFAULT_MODEL_ID`. Used only as the pre-hydration seed;
* the authoritative value comes from `GET /dictation/prefs`. */
const DEFAULT_DICTATION_MODEL_ID = 'sherpa-parakeet-tdt-v3';
const DEFAULT_DICTATION_MODEL_ID = 'sherpa-whisper-tiny';
/**
* Global UI font. Applied app-wide by overriding the `--font-sans` CSS custom
@@ -202,7 +202,7 @@ export interface PrefsSlice {
* • dictationMode — 'toggle' (press to start, press to stop) | 'hold'
* (record while the key is held).
* • dictationModelId — the selected sherpa-onnx model id (e.g.
* 'sherpa-parakeet-tdt-v3'); drives `?model=` on the
* 'sherpa-whisper-tiny'); drives `?model=` on the
* live `/ws/transcribe` socket.
*/
dictationEnabled: boolean;
@@ -289,7 +289,7 @@ export const createPrefsSlice: StateCreator<PrefsSlice, [], [], PrefsSlice> = (s
autoPlayPreview: true,
// Seeds only — overwritten by loadDictationPrefs() on init. The backend
// default is enabled:true / mode:'toggle' / model:Parakeet TDT v3.
// default is enabled:true / mode:'toggle' / model:Whisper Tiny.
dictationEnabled: true,
dictationMode: 'toggle',
dictationModelId: DEFAULT_DICTATION_MODEL_ID,
frontend/src/test/CaptureWidgetMicPreflight.test.jsx
+7 -5
View File
@@ -72,14 +72,16 @@ function stubInvoke({ mic = 'granted' } = {}) {
if (cmd === 'check_accessibility') return true;
if (cmd === 'request_dictation_capture') {
const event = payload?.action === 'stop' ? 'tray-dictate-stop' : 'tray-dictate';
if (eventHandlers[event]) return eventHandlers[event]();
captureState.pending = event;
const eventPayload =
event === 'tray-dictate' ? { payload: { sessionId: 'mic-preflight-session' } } : undefined;
if (eventHandlers[event]) return eventHandlers[event](eventPayload);
captureState.pending = { event, eventPayload };
return undefined;
}
if (cmd === 'mark_dictation_capture_ready' && captureState.pending) {
const pending = captureState.pending;
const { event, eventPayload } = captureState.pending;
captureState.pending = null;
return eventHandlers[pending]?.();
return eventHandlers[event]?.(eventPayload);
}
return undefined;
});
@@ -152,7 +154,7 @@ describe('CaptureWidget — mic permission pre-flight (Tauri)', () => {
});
render(<CaptureWidget />);
await waitFor(() => expect(eventHandlers['tray-dictate']).toBeTypeOf('function'));
eventHandlers['tray-dictate']();
eventHandlers['tray-dictate']({ payload: { sessionId: 'mic-preflight-session' } });
expect(await screen.findByText(/Mic access denied/)).toBeInTheDocument();
expect(gum).not.toHaveBeenCalled();
frontend/src/test/CaptureWidgetSetupRace.test.jsx
+18 -4
View File
@@ -8,15 +8,17 @@
*/
import React from 'react';
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import { render, screen, waitFor } from '@testing-library/react';
const { toastMock } = vi.hoisted(() => ({
const { toastMock, eventHandlers, eventState } = vi.hoisted(() => ({
toastMock: Object.assign(vi.fn(), {
error: vi.fn(),
success: vi.fn(),
dismiss: vi.fn(),
loading: vi.fn(),
}),
eventHandlers: {},
eventState: { pendingStart: false },
}));
vi.mock('react-hot-toast', () => ({ default: toastMock, toast: toastMock }));
@@ -25,7 +27,10 @@ vi.mock('@tauri-apps/api/core', () => ({
invoke: (...args) => invokeMock(...args),
}));
vi.mock('@tauri-apps/api/event', () => ({
listen: vi.fn(async () => () => {}),
listen: vi.fn(async (name, handler) => {
eventHandlers[name] = handler;
return () => delete eventHandlers[name];
}),
}));
vi.mock('@tauri-apps/api/window', () => ({
getCurrentWindow: () => ({ hide: async () => {} }),
@@ -103,7 +108,9 @@ class FakeWS {
}
function pressShortcut() {
fireEvent.keyDown(window, { code: 'Space', ctrlKey: true, shiftKey: true });
const handler = eventHandlers['tray-dictate'];
if (handler) handler({ payload: { sessionId: 'setup-race-session' } });
else eventState.pendingStart = true;
}
let realWebSocket;
@@ -114,8 +121,15 @@ beforeEach(() => {
invokeMock.mockImplementation(async (cmd) => {
if (cmd === 'check_microphone') return 'granted';
if (cmd === 'check_accessibility') return true;
if (cmd === 'mark_dictation_capture_ready' && eventState.pendingStart) {
eventState.pendingStart = false;
return eventHandlers['tray-dictate']?.({
payload: { sessionId: 'setup-race-session' },
});
}
return undefined;
});
eventState.pendingStart = false;
FakeWS.instances = [];
storeState.dictationModelId = 'sherpa-parakeet-v3';
realWebSocket = globalThis.WebSocket;
frontend/src/test/VoicePanel.test.jsx
+22 -20
View File
@@ -38,25 +38,25 @@ const MODELS = {
repo_id: 'org/parakeet-v3',
label: 'Parakeet TDT v3',
tag: 'offline',
recommended: true,
size_gb: 0.18,
recommended: false,
size_gb: 0.67,
languages: '25 European languages',
installed: true,
installed: false,
},
{
id: 'sherpa-whisper-tiny',
repo_id: 'org/whisper-tiny',
label: 'Whisper Tiny',
tag: 'offline',
recommended: false,
size_gb: 0.116,
recommended: true,
size_gb: 0.104,
languages: '90+ languages',
installed: false,
installed: true,
},
],
engine_available: true,
engine_reason: null,
default_model_id: 'sherpa-parakeet-tdt-v3',
default_model_id: 'sherpa-whisper-tiny',
};
function withI18n(node) {
@@ -83,7 +83,7 @@ describe('VoicePanel', () => {
return Promise.resolve({
enabled: true,
mode: 'toggle',
model_id: 'sherpa-parakeet-tdt-v3',
model_id: 'sherpa-whisper-tiny',
});
return Promise.resolve({});
});
@@ -91,7 +91,7 @@ describe('VoicePanel', () => {
useAppStore.setState({
dictationEnabled: true,
dictationMode: 'toggle',
dictationModelId: 'sherpa-parakeet-tdt-v3',
dictationModelId: 'sherpa-whisper-tiny',
dictationLoaded: true,
});
});
@@ -107,39 +107,41 @@ describe('VoicePanel', () => {
expect(screen.getByRole('switch', { name: 'Enable Voice Dictation' })).toBeChecked();
// The dropdown trigger shows the selected model once models load.
await waitFor(() =>
expect(screen.getByTestId('dictation-model-trigger')).toHaveTextContent('Parakeet TDT v3'),
expect(screen.getByTestId('dictation-model-trigger')).toHaveTextContent('Whisper Tiny'),
);
});
it('lists models with badges, size and install/delete affordances when expanded', async () => {
render(withI18n(<VoicePanel />));
await waitFor(() =>
expect(screen.getByTestId('dictation-model-trigger')).toHaveTextContent('Parakeet TDT v3'),
expect(screen.getByTestId('dictation-model-trigger')).toHaveTextContent('Whisper Tiny'),
);
fireEvent.click(screen.getByTestId('dictation-model-trigger'));
const v3 = screen.getByTestId('dictation-model-sherpa-parakeet-tdt-v3').closest('li');
expect(within(v3).getByText('recommended')).toBeInTheDocument();
expect(within(v3).getByText('offline')).toBeInTheDocument();
expect(within(v3).getByText('180 MB')).toBeInTheDocument();
const whisper = screen.getByTestId('dictation-model-sherpa-whisper-tiny').closest('li');
expect(within(whisper).getByText('recommended')).toBeInTheDocument();
expect(within(whisper).getByText('offline')).toBeInTheDocument();
expect(within(whisper).getByText('104 MB')).toBeInTheDocument();
// Installed → delete affordance.
expect(screen.getByTestId('dictation-delete-sherpa-parakeet-tdt-v3')).toBeInTheDocument();
expect(screen.getByTestId('dictation-delete-sherpa-whisper-tiny')).toBeInTheDocument();
// Not installed → download affordance.
expect(screen.getByTestId('dictation-install-sherpa-whisper-tiny')).toBeInTheDocument();
expect(screen.getByTestId('dictation-install-sherpa-parakeet-tdt-v3')).toBeInTheDocument();
});
it('writes the model pref and kicks off install when picking an uninstalled model', async () => {
render(withI18n(<VoicePanel />));
await waitFor(() => expect(screen.getByTestId('dictation-model-trigger')).toBeInTheDocument());
fireEvent.click(screen.getByTestId('dictation-model-trigger'));
fireEvent.click(screen.getByTestId('dictation-model-sherpa-whisper-tiny'));
fireEvent.click(screen.getByTestId('dictation-model-sherpa-parakeet-tdt-v3'));
// Pref write-through (POST /dictation/prefs with the new model id).
await waitFor(() =>
expect(apiPost).toHaveBeenCalledWith('/dictation/prefs', { model_id: 'sherpa-whisper-tiny' }),
expect(apiPost).toHaveBeenCalledWith('/dictation/prefs', {
model_id: 'sherpa-parakeet-tdt-v3',
}),
);
// Uninstalled → download started via the model-store install mutation.
expect(installMutate).toHaveBeenCalledWith('org/whisper-tiny');
expect(installMutate).toHaveBeenCalledWith('org/parakeet-v3');
});
it('toggles the enable switch through the store write-through', async () => {
frontend/src/test/captureSherpaFinal.test.js
+44 -14
View File
@@ -10,6 +10,8 @@ import { describe, it, expect } from 'vitest';
import {
isSherpaModel,
classifySherpaFinal,
sherpaSummaryTail,
aggregateDeliveryKind,
computeTypeDelta,
parsePasteError,
} from '../components/CaptureWidget';
@@ -26,30 +28,54 @@ describe('isSherpaModel', () => {
});
describe('classifySherpaFinal', () => {
it('treats the first non-empty offline final as a new utterance (then close finalises)', () => {
// Offline model (Parakeet v3 default): one final, nothing committed yet.
expect(classifySherpaFinal('hello world', [])).toBe('utterance');
it('uses the explicit utterance kind even when the text repeats', () => {
expect(
classifySherpaFinal({ type: 'final', final_kind: 'utterance', text: 'yes' }, ['yes']),
).toBe('utterance');
});
it('treats a streaming per-utterance final as an utterance', () => {
expect(classifySherpaFinal('second sentence', ['first sentence'])).toBe('utterance');
it('uses the explicit summary kind when EOF includes an uncommitted tail', () => {
expect(
classifySherpaFinal(
{ type: 'final', final_kind: 'summary', text: 'first sentence tail words' },
['first sentence'],
),
).toBe('summary');
});
it('detects the EOF summary (text === the committed join)', () => {
const committed = ['first sentence', 'second sentence'];
expect(classifySherpaFinal('first sentence second sentence', committed)).toBe('summary');
it('finalises on an explicit empty summary', () => {
expect(classifySherpaFinal({ type: 'final', final_kind: 'summary', text: '' }, [])).toBe(
'terminator',
);
});
it('detects a single-utterance summary (summary equals the one commit)', () => {
expect(classifySherpaFinal('hello world', ['hello world'])).toBe('summary');
it('ignores empty utterance and unknown final kinds', () => {
expect(classifySherpaFinal({ final_kind: 'utterance', text: '' }, [])).toBe('ignore');
expect(classifySherpaFinal({ text: 'legacy guess' }, [])).toBe('ignore');
});
});
describe('sherpaSummaryTail', () => {
it('returns only text not already committed', () => {
expect(sherpaSummaryTail('First sentence. Tail words.', ['First sentence.'])).toBe(
'Tail words.',
);
});
it('finalises on an empty no-speech terminator', () => {
expect(classifySherpaFinal('', [])).toBe('terminator');
it('returns the whole summary when no utterance was committed', () => {
expect(sherpaSummaryTail('Offline final.', [])).toBe('Offline final.');
});
it('ignores an empty final once utterances were committed (the summary covers it)', () => {
expect(classifySherpaFinal('', ['something'])).toBe('ignore');
it('does not duplicate a summary already delivered in full', () => {
expect(sherpaSummaryTail('One. Two.', ['One.', 'Two.'])).toBe('');
});
});
describe('aggregateDeliveryKind', () => {
it('keeps copied as the truthful session outcome', () => {
expect(aggregateDeliveryKind('pasted', 'inserted')).toBe('inserted');
expect(aggregateDeliveryKind('inserted', 'copied')).toBe('copied');
expect(aggregateDeliveryKind('copied', 'inserted')).toBe('copied');
});
});
@@ -155,6 +181,10 @@ describe('parsePasteError', () => {
kind: 'paste',
message: 'key event failed',
});
expect(parsePasteError('preflight: clipboard-only session')).toEqual({
kind: 'preflight',
message: 'clipboard-only session',
});
});
it('accepts Error objects (Tauri invoke may reject with either shape)', () => {
frontend/src/test/dictationPrefs.test.ts
+7 -1
View File
@@ -20,6 +20,7 @@ vi.mock('../api/client', () => ({
}));
import { useAppStore } from '../store';
import { createPrefsSlice } from '../store/prefsSlice';
function flush() {
// Let the write-through promise (.then) settle.
@@ -27,6 +28,11 @@ function flush() {
}
describe('dictation prefs store wiring', () => {
it('seeds the cross-platform default before backend hydration', () => {
const slice = createPrefsSlice(vi.fn() as any, vi.fn() as any, {} as any);
expect(slice.dictationModelId).toBe('sherpa-whisper-tiny');
});
beforeEach(() => {
apiJson.mockReset();
apiPost.mockReset();
@@ -34,7 +40,7 @@ describe('dictation prefs store wiring', () => {
useAppStore.setState({
dictationEnabled: true,
dictationMode: 'toggle',
dictationModelId: 'sherpa-parakeet-tdt-v3',
dictationModelId: 'sherpa-whisper-tiny',
dictationLoaded: false,
});
});
frontend/src/utils/aec/micCapture.js
+88 -7
View File
@@ -5,6 +5,58 @@
const WORKLET_URL = '/aec-worklet.js';
// Anti-alias filtering for the decimation below. resampleInterleavedFrame
// picks samples by linear interpolation, which is not a low-pass: taking a
// 48 kHz stream to 16 kHz that way folds everything above 8 kHz back down
// into the speech band as tones that were never spoken, and the ASR is fed
// the result. The browser only hands us 48 kHz when it refuses the requested
// 16 kHz AudioContext — WKWebView does — so this is the normal path there,
// not an edge case.
//
// Three cascaded Butterworth-Q biquads (~36 dB/octave) run in the audio
// graph rather than per frame, so the filter keeps its state across frame
// boundaries instead of restarting 50 times a second. The cutoff sits below
// Nyquist to leave room for the rolloff; speech has little energy up there.
const ANTIALIAS_STAGES = 3;
const ANTIALIAS_CUTOFF_RATIO = 0.4;
export function buildAntiAliasChain(ctx, targetRate) {
if (typeof ctx.createBiquadFilter !== 'function') return [];
const stages = [];
for (let stage = 0; stage < ANTIALIAS_STAGES; stage += 1) {
const filter = ctx.createBiquadFilter();
filter.type = 'lowpass';
filter.frequency.value = ANTIALIAS_CUTOFF_RATIO * targetRate;
filter.Q.value = Math.SQRT1_2; // Butterworth — flat passband, no resonant peak
stages.push(filter);
}
return stages;
}
export function resampleInterleavedFrame(frame, inputRate, outputRate, channels) {
if (inputRate === outputRate || frame.length === 0) return frame;
const inputFrames = Math.floor(frame.length / channels);
const outputFrames = Math.max(1, Math.round((inputFrames * outputRate) / inputRate));
const output = new Float32Array(outputFrames * channels);
const sourceStep = inputRate / outputRate;
for (let outputIndex = 0; outputIndex < outputFrames; outputIndex += 1) {
const sourcePosition = outputIndex * sourceStep;
const lowerIndex = Math.min(Math.floor(sourcePosition), inputFrames - 1);
const upperIndex = Math.min(lowerIndex + 1, inputFrames - 1);
const mix = sourcePosition - lowerIndex;
for (let channel = 0; channel < channels; channel += 1) {
const lower = frame[lowerIndex * channels + channel];
const upper = frame[upperIndex * channels + channel];
output[outputIndex * channels + channel] = lower + (upper - lower) * mix;
}
}
return output;
}
/**
* Start capturing ``stream`` as Float32 mono frames at ``sampleRate``.
*
@@ -20,15 +72,37 @@ export async function startMicCapture(
) {
const Ctx = window.AudioContext || window.webkitAudioContext;
const ctx = new Ctx({ sampleRate });
if (ctx.state === 'suspended') {
try {
await ctx.resume();
} catch {
/* reported below — a context that never runs emits no frames at all */
}
}
if (ctx.state === 'suspended') {
// Failing loudly beats the alternative: a suspended context runs no
// worklet, so the pill sits on "Listening" forever while not one frame
// is captured. The caller can surface this; silence cannot be surfaced.
try {
await ctx.close();
} catch {
/* ignore */
}
throw new Error('mic-suspended: the audio context could not be resumed');
}
await ctx.audioWorklet.addModule(WORKLET_URL);
const src = ctx.createMediaStreamSource(stream);
const sourceFrameSize = Math.max(1, Math.round((frameSize * ctx.sampleRate) / sampleRate));
const node = new AudioWorkletNode(ctx, 'aec-frame-emitter', {
processorOptions: { frameSize, channels },
processorOptions: { frameSize: sourceFrameSize, channels },
});
node.port.onmessage = (e) => onFrame(e.data);
// Mic → worklet only. Deliberately NOT connected to destination: we tap the
// mic, we don't want to play it back through the speakers.
src.connect(node);
node.port.onmessage = (e) =>
onFrame(resampleInterleavedFrame(e.data, ctx.sampleRate, sampleRate, channels));
// Mic → [anti-alias] → worklet. Only when the browser refused the requested
// rate; when it honors it there is no decimation and nothing to filter.
const antiAlias = ctx.sampleRate > sampleRate ? buildAntiAliasChain(ctx, sampleRate) : [];
const chain = [src, ...antiAlias, node];
for (let i = 0; i < chain.length - 1; i += 1) chain[i].connect(chain[i + 1]);
const stop = async function stop() {
try {
@@ -46,6 +120,13 @@ export async function startMicCapture(
} catch {
/* ignore */
}
for (const filter of antiAlias) {
try {
filter.disconnect();
} catch {
/* ignore */
}
}
try {
await ctx.close();
} catch {
@@ -53,8 +134,8 @@ export async function startMicCapture(
}
};
// Existing callers use this value as a function. The property lets generic
// PCM/WAV recording encode the frames at the AudioContext's actual rate.
stop.sampleRate = ctx.sampleRate;
// PCM/WAV recording encode the delivered frames at their actual rate.
stop.sampleRate = sampleRate;
stop.channels = channels;
return stop;
}
frontend/src/utils/aec/micCapture.test.js
+169
View File
@@ -0,0 +1,169 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { startMicCapture } from './micCapture';
let context;
let workletNode;
let contextSampleRate;
class FakeAudioContext {
constructor() {
context = this;
this.sampleRate = contextSampleRate;
this.state = 'suspended';
this.audioWorklet = { addModule: vi.fn().mockResolvedValue(undefined) };
this.resume = vi.fn(async () => {
this.state = 'running';
});
this.close = vi.fn().mockResolvedValue(undefined);
this.source = {
connect: vi.fn(),
disconnect: vi.fn(),
};
this.filters = [];
}
createMediaStreamSource() {
return this.source;
}
createBiquadFilter() {
const filter = {
type: '',
frequency: { value: 0 },
Q: { value: 0 },
connect: vi.fn(),
disconnect: vi.fn(),
};
this.filters.push(filter);
return filter;
}
}
class FakeAudioWorkletNode {
constructor(_context, _name, options) {
workletNode = this;
this.options = options;
this.port = { onmessage: null };
this.disconnect = vi.fn();
}
}
describe('startMicCapture', () => {
beforeEach(() => {
contextSampleRate = 48000;
vi.stubGlobal('AudioContext', FakeAudioContext);
vi.stubGlobal('AudioWorkletNode', FakeAudioWorkletNode);
});
afterEach(() => {
vi.unstubAllGlobals();
context = undefined;
workletNode = undefined;
contextSampleRate = undefined;
});
it('resumes a suspended AudioContext before capturing microphone frames', async () => {
const stop = await startMicCapture({}, vi.fn());
expect(context.resume).toHaveBeenCalledOnce();
expect(context.state).toBe('running');
await stop();
});
it('delivers fixed-size 16 kHz frames when the AudioContext runs at 48 kHz', async () => {
const frames = [];
const stop = await startMicCapture({}, (frame) => frames.push(frame), {
sampleRate: 16000,
frameSize: 4,
});
expect(workletNode.options.processorOptions).toEqual({ frameSize: 12, channels: 1 });
workletNode.port.onmessage({
data: new Float32Array([-1, -0.75, -0.5, -0.25, 0, 0.25, 0.5, 0.75, 1, 0.75, 0.5, 0.25]),
});
expect(frames).toEqual([new Float32Array([-1, -0.25, 0.5, 0.75])]);
expect(stop.sampleRate).toBe(16000);
await stop();
});
it('preserves worklet frames when the AudioContext honors the requested rate', async () => {
contextSampleRate = 16000;
const onFrame = vi.fn();
const stop = await startMicCapture({}, onFrame, { sampleRate: 16000, frameSize: 4 });
const frame = new Float32Array([-1, -0.25, 0.5, 0.75]);
workletNode.port.onmessage({ data: frame });
expect(onFrame).toHaveBeenCalledOnce();
expect(onFrame).toHaveBeenCalledWith(frame);
await stop();
});
it('low-passes before decimating when the browser refuses the requested rate', async () => {
// WKWebView hands back 48 kHz whatever we ask for; interpolating straight
// down to 16 kHz would fold everything above 8 kHz into the speech band.
const stop = await startMicCapture({}, vi.fn(), { sampleRate: 16000, frameSize: 4 });
expect(context.filters).toHaveLength(3);
for (const filter of context.filters) {
expect(filter.type).toBe('lowpass');
expect(filter.frequency.value).toBeLessThan(16000 / 2); // below the fold frequency
expect(filter.Q.value).toBeCloseTo(Math.SQRT1_2, 5); // Butterworth, no resonant peak
}
// Mic → filters → worklet, in order.
expect(context.source.connect).toHaveBeenCalledWith(context.filters[0]);
expect(context.filters[0].connect).toHaveBeenCalledWith(context.filters[1]);
expect(context.filters[1].connect).toHaveBeenCalledWith(context.filters[2]);
expect(context.filters[2].connect).toHaveBeenCalledWith(workletNode);
await stop();
for (const filter of context.filters) expect(filter.disconnect).toHaveBeenCalled();
});
it('skips the filter chain when the AudioContext honors the requested rate', async () => {
contextSampleRate = 16000;
const stop = await startMicCapture({}, vi.fn(), { sampleRate: 16000, frameSize: 4 });
// No decimation happens, so there is nothing to anti-alias.
expect(context.filters).toHaveLength(0);
expect(context.source.connect).toHaveBeenCalledWith(workletNode);
await stop();
});
it('fails loudly when the AudioContext cannot be resumed', async () => {
// A suspended context runs no worklet: every frame is silently lost and
// the pill sits on "Listening" forever. Better to surface it.
const failing = class extends FakeAudioContext {
constructor(...args) {
super(...args);
this.resume = vi.fn(async () => {
throw new Error('user gesture required');
});
}
};
vi.stubGlobal('AudioContext', failing);
await expect(startMicCapture({}, vi.fn())).rejects.toThrow(/mic-suspended/);
expect(context.close).toHaveBeenCalled();
});
it('fails loudly when resume resolves but the context stays suspended', async () => {
const stuck = class extends FakeAudioContext {
constructor(...args) {
super(...args);
this.resume = vi.fn(async () => {}); // resolves, state never changes
}
};
vi.stubGlobal('AudioContext', stuck);
await expect(startMicCapture({}, vi.fn())).rejects.toThrow(/mic-suspended/);
expect(context.close).toHaveBeenCalled();
});
});
frontend/src/utils/aec/playbackTap.js
+29 -4
View File
@@ -12,6 +12,7 @@
// the player never silences playback.
import { publishFarEnd } from './farEndBus';
import { buildAntiAliasChain, resampleInterleavedFrame } from './micCapture';
const WORKLET_URL = '/aec-worklet.js';
@@ -44,11 +45,21 @@ export async function attachPlaybackTap(mediaEl, { sampleRate = 16000, frameSize
/* gesture may be required; harmless */
}
}
const channels = 1;
const sourceFrameSize = Math.max(1, Math.round((frameSize * ctx.sampleRate) / sampleRate));
const node = new AudioWorkletNode(ctx, 'aec-frame-emitter', {
processorOptions: { frameSize },
processorOptions: { frameSize: sourceFrameSize, channels },
});
node.port.onmessage = (e) => publishFarEnd(e.data);
src.connect(node);
node.port.onmessage = (e) =>
publishFarEnd(resampleInterleavedFrame(e.data, ctx.sampleRate, sampleRate, channels));
// The far-end reference decimates exactly like the mic path, so it needs the
// same anti-alias low-pass before resampling — an aliased reference makes
// the AEC subtract tones the speaker never played. Filters sit only on the
// tap branch: the src → destination edge stays untouched, so what the user
// hears is unchanged.
const antiAlias = ctx.sampleRate > sampleRate ? buildAntiAliasChain(ctx, sampleRate) : [];
const chain = [src, ...antiAlias, node];
for (let i = 0; i < chain.length - 1; i += 1) chain[i].connect(chain[i + 1]);
return async function detach() {
try {
@@ -61,6 +72,20 @@ export async function attachPlaybackTap(mediaEl, { sampleRate = 16000, frameSize
} catch {
/* ignore */
}
// Intentionally leave ctx + src→destination intact (see header note).
for (const filter of antiAlias) {
try {
filter.disconnect();
} catch {
/* ignore */
}
}
// Detach the tap edge too: the ctx and src are memoised per element, so a
// filter left hanging off src would accumulate one dead chain per AEC
// toggle. The audible src→destination edge stays (see header note).
try {
src.disconnect(antiAlias[0] ?? node);
} catch {
/* ignore */
}
};
}
frontend/src/utils/aec/playbackTap.test.js
+116
View File
@@ -0,0 +1,116 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
vi.mock('./farEndBus', () => ({ publishFarEnd: vi.fn() }));
import { publishFarEnd } from './farEndBus';
import { attachPlaybackTap } from './playbackTap';
let context;
let workletNode;
let contextSampleRate;
class FakeAudioContext {
constructor() {
context = this;
this.sampleRate = contextSampleRate;
this.state = 'running';
this.destination = {};
this.audioWorklet = { addModule: vi.fn().mockResolvedValue(undefined) };
this.source = { connect: vi.fn(), disconnect: vi.fn() };
this.filters = [];
}
createMediaElementSource() {
return this.source;
}
createBiquadFilter() {
const filter = {
type: '',
frequency: { value: 0 },
Q: { value: 0 },
connect: vi.fn(),
disconnect: vi.fn(),
};
this.filters.push(filter);
return filter;
}
}
class FakeAudioWorkletNode {
constructor(_context, _name, options) {
workletNode = this;
this.options = options;
this.port = { onmessage: null };
this.disconnect = vi.fn();
}
}
describe('attachPlaybackTap', () => {
beforeEach(() => {
contextSampleRate = 48000;
vi.stubGlobal('AudioContext', FakeAudioContext);
vi.stubGlobal('AudioWorkletNode', FakeAudioWorkletNode);
publishFarEnd.mockClear();
});
afterEach(() => {
vi.unstubAllGlobals();
context = undefined;
workletNode = undefined;
contextSampleRate = undefined;
});
it('delivers fixed-size 16 kHz reference frames from a 48 kHz context', async () => {
const detach = await attachPlaybackTap({}, { sampleRate: 16000, frameSize: 4 });
expect(workletNode.options.processorOptions).toEqual({ frameSize: 12, channels: 1 });
workletNode.port.onmessage({
data: new Float32Array([-1, -0.75, -0.5, -0.25, 0, 0.25, 0.5, 0.75, 1, 0.75, 0.5, 0.25]),
});
expect(publishFarEnd).toHaveBeenCalledWith(new Float32Array([-1, -0.25, 0.5, 0.75]));
await detach();
expect(workletNode.disconnect).toHaveBeenCalledOnce();
expect(context.source.connect).toHaveBeenCalledWith(context.destination);
});
it('low-passes the far-end reference before decimating (48 kHz context)', async () => {
// The AEC reference decimates exactly like the mic path; without the
// filter, content above 8 kHz folds into the reference and the canceller
// subtracts tones the speaker never played.
const detach = await attachPlaybackTap({}, { sampleRate: 16000, frameSize: 4 });
expect(context.filters).toHaveLength(3);
for (const filter of context.filters) {
expect(filter.type).toBe('lowpass');
expect(filter.frequency.value).toBeLessThan(16000 / 2);
}
// Tap branch only: element → filters → worklet, while the audible
// element → destination edge stays direct.
expect(context.source.connect).toHaveBeenCalledWith(context.destination);
expect(context.source.connect).toHaveBeenCalledWith(context.filters[0]);
expect(context.filters[0].connect).toHaveBeenCalledWith(context.filters[1]);
expect(context.filters[1].connect).toHaveBeenCalledWith(context.filters[2]);
expect(context.filters[2].connect).toHaveBeenCalledWith(workletNode);
await detach();
for (const filter of context.filters) expect(filter.disconnect).toHaveBeenCalled();
// Only the tap edge is removed; the audible src→destination edge stays.
expect(context.source.disconnect).toHaveBeenCalledWith(context.filters[0]);
expect(context.source.disconnect).not.toHaveBeenCalledWith(context.destination);
});
it('skips the filter chain when the context honors the requested rate', async () => {
contextSampleRate = 16000;
const detach = await attachPlaybackTap({}, { sampleRate: 16000, frameSize: 4 });
expect(context.filters).toHaveLength(0);
expect(context.source.connect).toHaveBeenCalledWith(workletNode);
await detach();
expect(context.source.disconnect).toHaveBeenCalledWith(workletNode);
expect(context.source.disconnect).not.toHaveBeenCalledWith(context.destination);
});
});
tests/test_asr_model_missing.py
+83 -3
View File
@@ -19,6 +19,19 @@ import pytest
from fastapi.testclient import TestClient
@pytest.fixture(autouse=True)
def _clear_installed_repo_memo():
"""_repo_installed memoizes positives module-globally and never
invalidates, and test_installed_positive_is_memoized writes the very repo
the missing-model tests probe — so run order decided what these tests saw
(CodeRabbit on #1610). Deterministic now: empty before, empty after."""
from services import asr_backend
asr_backend._INSTALLED_REPO_MEMO.clear()
yield
asr_backend._INSTALLED_REPO_MEMO.clear()
@pytest.fixture(scope="module")
def client():
from main import app
@@ -94,8 +107,8 @@ class TestHelper:
assert payload is not None
assert payload["error"] == "asr_model_missing"
rec = payload["recommended"]
# The curated sherpa dictation entry, with the dictation_id the client
# needs to also set dictation.model_id so the retry picks it up.
# The explicitly selected sherpa entry, with the dictation_id the
# client needs to set so the retry picks it up.
assert rec["repo_id"] == "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8"
assert rec["dictation_id"] == "sherpa-parakeet-tdt-v3"
@@ -109,6 +122,55 @@ class TestHelper:
patch.object(sd, "is_installed", return_value=True):
assert asr_backend.asr_model_missing_error(purpose="dictation") is None
def test_demoted_sherpa_preflights_capture_fallback(self):
"""The next session follows demotion even when `?model=` persists."""
from api.routers.setup import models as setup_models
from services import asr_backend
from services import sherpa_dictation as sd
fallback_repo = "Systran/faster-whisper-large-v3"
with patch.object(asr_backend.SherpaDictationBackend, "is_available",
return_value=(True, "ready")), \
patch.object(sd, "is_demoted", return_value=True), \
patch.object(sd, "is_installed", return_value=True), \
patch.object(asr_backend, "_capture_whisper_repo",
return_value=fallback_repo), \
patch.object(setup_models, "is_cached", return_value=False), \
patch.object(setup_models, "cache_is_complete", return_value=True):
payload = asr_backend.asr_model_missing_error(
purpose="dictation",
sherpa_model_id="sherpa-parakeet-tdt-v3",
)
assert payload is not None
assert payload["missing_repo_id"] == fallback_repo
assert payload["recommended"]["repo_id"] == "csukuangfj/sherpa-onnx-whisper-tiny"
def test_demoted_default_never_recommends_itself(self):
"""Installing the CTA recommendation must escape, not clear, a demotion loop."""
from api.routers.setup import models as setup_models
from services import asr_backend
from services import sherpa_dictation as sd
fallback_repo = "Systran/faster-whisper-large-v3"
with patch.object(asr_backend.SherpaDictationBackend, "is_available",
return_value=(True, "ready")), \
patch.object(sd, "is_demoted",
side_effect=lambda mid: mid == "sherpa-whisper-tiny"), \
patch.object(asr_backend, "_capture_whisper_repo",
return_value=fallback_repo), \
patch.object(setup_models, "is_cached", return_value=False), \
patch.object(setup_models, "cache_is_complete", return_value=True):
payload = asr_backend.asr_model_missing_error(
purpose="dictation",
sherpa_model_id="sherpa-whisper-tiny",
)
assert payload is not None
assert payload["missing_repo_id"] == fallback_repo
assert payload["recommended"]["repo_id"] == fallback_repo
assert "dictation_id" not in payload["recommended"]
def test_never_raises(self):
from services import asr_backend
with patch.object(asr_backend, "active_backend_id",
@@ -125,6 +187,24 @@ class TestHelper:
patch.dict(os.environ, {"ASR_MODEL_FASTER": "someorg/custom-whisper"}):
assert asr_backend.asr_model_missing_error() is None
def test_silent_recovery_requires_even_a_custom_fallback_to_be_installed(self):
"""The recovery path never turns fail-open into an implicit download."""
from api.routers.setup import models as setup_models
from services import asr_backend
with patch.object(asr_backend, "_capture_whisper_repo",
return_value="someorg/custom-whisper"), \
patch.object(setup_models, "is_cached", return_value=False), \
patch.object(setup_models, "cache_is_complete", return_value=False):
payload = asr_backend.asr_model_missing_error(
purpose="dictation",
skip_sherpa=True,
require_installed=True,
)
assert payload is not None
assert payload["missing_repo_id"] == "someorg/custom-whisper"
def test_pytorch_whisper_default_repo_fails_open(self):
"""openai/whisper-large-v3-turbo (the pytorch-whisper default) is not
a catalog entry — the preflight stays out of the way (auto-download,
@@ -168,7 +248,7 @@ class TestHelper:
payload = asr_backend.asr_model_missing_error()
assert payload is not None
assert payload["missing_repo_id"] == (
"csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8"
"csukuangfj/sherpa-onnx-whisper-tiny"
)
def test_installed_positive_is_memoized(self):
tests/test_capture_ws.py
+91
View File
@@ -12,6 +12,7 @@ quality.
"""
import os
import time
import types
import pytest
@@ -67,6 +68,96 @@ def _audio_chunk(n_bytes: int = 20_000) -> bytes:
return b"\x00" * n_bytes
def test_select_sherpa_spec_ignores_demoted_query_override(monkeypatch):
"""A persisted frontend query must not resurrect a silent recognizer."""
from api.routers import capture_ws as cw
from services import sherpa_dictation as sd
model_id = "sherpa-parakeet-tdt-v3"
websocket = types.SimpleNamespace(query_params={"model": model_id})
monkeypatch.setattr(sd, "is_demoted", lambda mid: mid == model_id)
assert cw._select_sherpa_spec(websocket) is None
def test_demoted_sherpa_query_keeps_pcm_transport_for_legacy_fallback(
client, monkeypatch,
):
"""Demotion changes the recognizer, not the bytes already sent by the UI."""
from api.routers import capture_ws as cw
from services import sherpa_dictation as sd
model_id = "sherpa-parakeet-tdt-v3"
monkeypatch.setattr(sd, "is_demoted", lambda mid: mid == model_id)
sample_rates = []
async def fallback(_chunks, *, pcm_sr=None):
sample_rates.append(pcm_sr)
return {
"text": "legacy fallback heard pcm",
"segments": [],
"language": "en",
"engine": "stub",
}
monkeypatch.setattr(cw, "_transcribe_buffer_full", fallback)
with client.websocket_connect(
f"/ws/transcribe?model={model_id}&sr=16000"
) as ws:
ws.send_bytes(_audio_chunk())
ws.send_text("EOF")
for _ in range(10):
if ws.receive_json().get("type") == "final":
break
assert sample_rates == [16000]
@pytest.mark.asyncio
@pytest.mark.parametrize(
("result", "expected"),
[
({"text": "top-level text"}, "top-level text"),
(
{"segments": [{"text": "segment one"}, {"text": "segment two"}]},
"segment one segment two",
),
(
{"chunks": [{"text": "chunk one"}, {"text": "chunk two"}]},
"chunk one chunk two",
),
],
)
async def test_partial_text_normalizes_every_asr_result_shape(
monkeypatch, tmp_path, result, expected,
):
"""Live partials work for backends that expose only segments/chunks.
WhisperX, Faster Whisper, Moonshine, and OpenAI-compatible ASR do not add a
top-level ``text`` field. The capture seam must consume the shared ASR
result contract instead of silently dropping their partial transcript.
"""
from api.routers import capture_ws as cw
from services import asr_backend
wav = tmp_path / "partial.wav"
wav.write_bytes(b"placeholder")
class StubBackend:
def transcribe(self, _path, *, word_timestamps=False):
assert word_timestamps is False
return result
async def run_inline(_executor, fn, **_kwargs):
return fn()
monkeypatch.setattr(cw, "_pcm16_to_wav", lambda _pcm, _sr: str(wav))
monkeypatch.setattr(asr_backend, "get_capture_asr_backend", lambda: StubBackend())
monkeypatch.setattr(asr_backend, "run_transcribe_guarded", run_inline)
assert await cw._transcribe_buffer([b"\x00" * 4000], pcm_sr=16000) == expected
def test_eof_text_frame_triggers_final_without_disconnect(client):
"""Client sends audio + 'EOF' text frame, expects `final` over open socket."""
with client.websocket_connect("/ws/transcribe") as ws:
tests/test_dictation_model_copy.py
+24
View File
@@ -0,0 +1,24 @@
"""Keep localized dictation recommendation copy aligned with model policy."""
from __future__ import annotations
import json
from pathlib import Path
_LOCALES = Path(__file__).resolve().parents[1] / "frontend" / "src" / "i18n" / "locales"
def test_recommended_badge_copy_belongs_to_whisper_tiny_in_every_locale():
natural_prefixes = {
"ja": "おすすめ",
"uk": "рекомендовано",
"vi": "khuyến nghị",
}
for path in sorted(_LOCALES.glob("*.json")):
panel = json.loads(path.read_text(encoding="utf-8"))["voicePanel"]
badge = panel["badge_recommended"].casefold()
recommendation = natural_prefixes.get(path.stem, badge)
descriptions = panel["model_desc"]
assert recommendation not in descriptions["sherpa-parakeet-tdt-v3"].casefold(), path.name
assert recommendation in descriptions["sherpa-whisper-tiny"].casefold(), path.name
tests/test_dictation_recovery_tail_bound.py
+147
View File
@@ -0,0 +1,147 @@
"""Silent-model recovery audio must be bounded (#1610 review).
Both dictation WebSocket paths retained every PCM byte of a session so the
silent-model fallback could re-transcribe it. Nothing capped that: an open mic
at 16 kHz mono int16 added ~115 MB per hour, held for the life of the session
and only ever read when the fallback actually fired. Streaming and offline
both did it.
The tail is what matters — recovery re-transcribes what the user just said —
while the silent-model gate measures how much audio the session carried, so
the true total is tracked separately and stays truthful after trimming.
"""
from __future__ import annotations
import importlib
import pytest
@pytest.fixture
def ws():
return importlib.import_module("api.routers.capture_ws")
SR = 16000
BYTES_PER_S = SR * 2
def test_a_long_session_stops_growing(ws):
tail = ws.RecoveryTail(SR, seconds=2.0)
for _ in range(600): # 60 s of 100 ms frames
tail.extend(b"\x01\x02" * (SR // 10))
assert len(tail.tail()) == 2 * BYTES_PER_S
def test_the_true_total_survives_trimming(ws):
"""is_model_silent gates on how much audio the session carried; capping
the buffer must not make a long session look too short to be recoverable."""
tail = ws.RecoveryTail(SR, seconds=1.0)
for _ in range(30):
tail.extend(b"\x00\x01" * SR) # 1 s each
assert tail.total_bytes == 30 * BYTES_PER_S
assert len(tail.tail()) == BYTES_PER_S
assert ws.is_model_silent("", True, tail.total_bytes) is True
def test_the_retained_audio_is_the_most_recent(ws):
"""Head-trimming, not head-keeping — the useful speech is the latest."""
tail = ws.RecoveryTail(SR, seconds=1.0)
tail.extend(b"\xaa\xaa" * SR) # older
tail.extend(b"\xbb\xbb" * SR) # newer
assert tail.tail() == b"\xbb\xbb" * SR
def test_a_short_session_is_kept_whole(ws):
tail = ws.RecoveryTail(SR, seconds=120.0)
tail.extend(b"\x01\x02" * SR)
assert tail.tail() == b"\x01\x02" * SR
assert tail.total_bytes == BYTES_PER_S
@pytest.mark.parametrize(
("value", "expected"),
[
(None, 120.0), ("bad", 120.0), ("nan", 120.0), ("inf", 120.0),
("-1", 120.0), ("0", 120.0), ("60", 60.0), ("999999", 300.0),
],
)
def test_recovery_tail_environment_override_is_finite_and_bounded(ws, value, expected):
assert ws._bounded_recovery_tail_seconds(value) == expected
@pytest.mark.parametrize("sample_rate,seconds", [(0, 120.0), (16000, 0.0), (-1, -1.0)])
def test_a_nonsense_bound_still_yields_a_usable_buffer(ws, sample_rate, seconds):
"""A bad sr query param or env override must not produce a zero-length
buffer that silently disables recovery."""
tail = ws.RecoveryTail(sample_rate, seconds=seconds)
tail.extend(b"\x01\x02" * 100)
assert len(tail.tail()) >= 2
@pytest.mark.parametrize("sr", ["1000000000", "4000", "0", "-16000", "junk", ""])
def test_an_absurd_client_sample_rate_is_not_believed(ws, sr):
"""`?sr=` sizes RecoveryTail's byte ceiling (rate × RECOVERY_TAIL_SECONDS),
so an unclamped client value re-opens the unbounded-memory path (#1610
review). Out-of-range and garbage rates fall back to 16 kHz."""
assert ws._bounded_sample_rate({"sr": sr}) == 16000
def test_supported_client_sample_rates_pass_through(ws):
for sr in (8000, 16000, 44100, 48000, 96000):
assert ws._bounded_sample_rate({"sr": str(sr)}) == sr
def test_the_sherpa_paths_use_the_bounded_rate(ws):
"""Structural: both sherpa handlers get their rate from _sherpa_session,
which must parse via the clamped helper — a raw int() of the query param
is exactly the bug."""
import inspect
src = inspect.getsource(ws._sherpa_session)
assert "_bounded_sample_rate(" in src
assert 'int(websocket.query_params.get("sr"' not in src
def test_both_socket_paths_use_the_bounded_buffer(ws):
"""Structural: the streaming path and the offline path both had the leak,
so a fix applied to only one of them is not a fix."""
import inspect
src = inspect.getsource(ws)
assert src.count("RecoveryTail(pcm_sr)") == 2
assert "session_pcm = bytearray()" not in src
def test_trimming_never_splits_a_sample(ws):
"""int16 mono PCM: transport frames can carry odd byte counts (a sample
split across two WebSocket messages), but the *stream* stays aligned — a
sample starts at every even global offset. Trimming must remove an even
number of bytes so the retained tail still starts on a sample boundary.
The failure needs a stream that ends mid-sample (the session closed on a
torn frame): with an odd total, an odd-trimming buffer ends with an odd
cumulative removal, the tail starts mid-sample, and every decoded value
is byte-shifted garbage. (An even total self-rebalances across trims,
which is why the obvious version of this test cannot fail.)
"""
import struct
n = SR * 2 # 2 s of samples; sample k holds the value k
stream = b"".join(struct.pack("<h", k % 32000) for k in range(n)) + b"\x7f"
tail = ws.RecoveryTail(SR, seconds=1.0)
# Odd-sized chunks so extend() boundaries never align with samples.
i = 0
for size in (1, 3331, 7777, 32001):
tail.extend(stream[i:i + size])
i += size
tail.extend(stream[i:])
kept = tail.tail()
whole = kept[: (len(kept) // 2) * 2] # the torn final byte is half a sample
values = [v for (v,) in struct.iter_unpack("<h", whole)]
# Sample-aligned ⟺ the decoded values are a contiguous ascending run; a
# mid-sample start turns them into byte-shifted noise.
assert values == list(range(values[0], values[0] + len(values))), values[:5]
assert values[-1] == (n - 1) % 32000
tests/test_dictation_router.py
+3 -3
View File
@@ -35,7 +35,7 @@ def test_list_models_shape(client):
r = client.get("/dictation/models")
assert r.status_code == 200
body = r.json()
assert body["default_model_id"] == "sherpa-parakeet-tdt-v3"
assert body["default_model_id"] == "sherpa-whisper-tiny"
assert len(body["models"]) == 7
keys = {"id", "repo_id", "label", "tag", "recommended", "size_gb",
"languages", "kind", "installed"}
@@ -43,7 +43,7 @@ def test_list_models_shape(client):
assert keys <= set(m), f"missing keys in {m}"
assert m["tag"] in ("offline", "streaming")
rec = [m for m in body["models"] if m["recommended"]]
assert [m["id"] for m in rec] == ["sherpa-parakeet-tdt-v3"]
assert [m["id"] for m in rec] == ["sherpa-whisper-tiny"]
def test_list_models_omits_probe_diagnostic(client, monkeypatch):
@@ -64,7 +64,7 @@ def test_get_prefs_defaults(client):
assert r.status_code == 200
body = r.json()
assert body == {"enabled": True, "mode": "toggle",
"model_id": "sherpa-parakeet-tdt-v3"}
"model_id": "sherpa-whisper-tiny"}
def test_set_prefs_persists_and_validates(client):
tests/test_locale_parity.py
+47 -4
View File
@@ -99,12 +99,25 @@ _ENGINE_AGNOSTIC_KEYS = (
# Never raise one: if this fails after adding en.json keys, add the keys to
# every locale (translated) in the same change instead.
_MISSING_BASELINE = {
"ar": 494, "de": 494, "es": 494, "fr": 494, "hi": 494, "id": 494,
"it": 494, "ja": 494, "ko": 494, "nl": 494, "pl": 494, "pt": 494,
"ru": 494, "sv": 494, "th": 494, "tr": 494, "uk": 494, "vi": 494,
"zh-CN": 487, "zh-TW": 494,
"ar": 493, "de": 493, "es": 493, "fr": 493, "hi": 493, "id": 493,
"it": 493, "ja": 493, "ko": 493, "nl": 493, "pl": 493, "pt": 493,
"ru": 493, "sv": 493, "th": 493, "tr": 493, "uk": 493, "vi": 493,
"zh-CN": 486, "zh-TW": 493,
}
#: Keys every locale must carry regardless of the aggregate ratchet above.
#: The count alone is a weak guarantee — a locale can translate one new key
#: while dropping another and the total never moves. These are strings a user
#: on a default path actually reads, so they get pinned by name.
#: capture.copied is the Wayland clipboard-delivery status, and clipboard
#: delivery IS the Wayland default, so leaving it English broke that path for
#: every non-English Linux user (#1610 review).
_REQUIRED_IN_EVERY_LOCALE = (
"capture.copied",
"capture.inserted",
"capture.pasted",
)
def _locale_files():
return sorted(f for f in os.listdir(_LOCALES_DIR) if f.endswith(".json"))
@@ -388,3 +401,33 @@ def test_transliterated_brands_are_a_known_gap():
delete this test rather than weakening the one above.
"""
assert not _ENGINE_BRANDS.search("\u30a6\u30a3\u30b9\u30d1\u30fc\u3067\u6587\u5b57\u8d77\u3053\u3057\u4e2d")
def _lookup(tree, dotted):
node = tree
for part in dotted.split("."):
if not isinstance(node, dict) or part not in node:
return None
node = node[part]
return node
@pytest.mark.parametrize("locale", sorted(_MISSING_BASELINE))
@pytest.mark.parametrize("key", _REQUIRED_IN_EVERY_LOCALE)
def test_every_locale_carries_the_user_facing_dictation_status(locale, key):
"""Named-key parity, not just the aggregate count.
The ratchet in _MISSING_BASELINE measures totals, so a locale can gain one
translation and lose another without the number moving. These keys are
read by users on default paths, so they are asserted individually.
"""
value = _lookup(_load(locale), key)
english = _lookup(_load("en"), key)
assert isinstance(value, str) and value.strip(), (
f"{locale}.json is missing {key!r} — users on that locale see the raw key "
f"or the English string"
)
assert value != english, (
f"{locale}.json copies the English {key!r} verbatim ({english!r}); "
f"translate it or the ratchet is measuring nothing"
)
tests/test_models_catalog.py
+7
View File
@@ -101,3 +101,10 @@ def test_every_platform_has_a_curated_asr_pick():
and (not m.get("platforms") or set(m["platforms"]) & family_tags)
]
assert curated_asr, f"no curated ASR pick resolves for host tags {family_tags}"
def test_single_cross_platform_dictation_recommendation_is_whisper_tiny():
"""Installer curation must match the runtime/UI dictation default."""
sherpa = [m for m in _models() if m.get("engine") == "sherpa-onnx"]
recommended = [m["dictation_id"] for m in sherpa if "all" in (m.get("curated_on") or [])]
assert recommended == ["sherpa-whisper-tiny"]
tests/test_setup_recommendations.py
+7 -4
View File
@@ -74,7 +74,8 @@ def test_mac_arm_curates_mlx_whisper_not_ct2(client):
# The CT2 build stays available in the full catalog but is not the
# Apple Silicon curated pick — MLX is Metal-accelerated, CT2 is CPU-only there.
assert "Systran/faster-whisper-large-v3" not in ids
assert "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" in ids
assert "csukuangfj/sherpa-onnx-whisper-tiny" in ids
assert "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" not in ids
def test_cuda_curates_ct2_whisper_and_turbo(client):
@@ -82,7 +83,8 @@ def test_cuda_curates_ct2_whisper_and_turbo(client):
assert "k2-fsa/OmniVoice" in ids
assert "Systran/faster-whisper-large-v3" in ids
assert "deepdml/faster-whisper-large-v3-turbo-ct2" in ids
assert "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" in ids
assert "csukuangfj/sherpa-onnx-whisper-tiny" in ids
assert "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" not in ids
# MLX models never resolve off Apple Silicon.
assert not any(rid.startswith("mlx-community/") for rid in ids)
@@ -100,12 +102,13 @@ def test_rocm_curates_pytorch_whisper_gpu_path(client):
assert "deepdml/faster-whisper-large-v3-turbo-ct2" not in ids
def test_cpu_only_curates_ct2_and_parakeet(client):
def test_cpu_only_curates_ct2_and_whisper_tiny(client):
payload = _recommend(client, ["win32", "win32-AMD64", "cpu"])
ids = _ids(payload)
assert "Systran/faster-whisper-large-v3" in ids
assert "deepdml/faster-whisper-large-v3-turbo-ct2" in ids
assert "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" in ids
assert "csukuangfj/sherpa-onnx-whisper-tiny" in ids
assert "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8" not in ids
assert "openai/whisper-large-v3" not in ids # 3.1 GB PyTorch build: GPU hosts only
tests/test_sherpa_dictation.py
+28 -2
View File
@@ -11,6 +11,7 @@ Also pins the verified ONNX asset filenames so a silent registry typo (the
streaming zipformer repos use plain `encoder-epoch-99-avg-1.int8.onnx`, NOT a
`-chunk-16-left-64` variant) fails loudly.
"""
import builtins
import os
import sys
import types
@@ -139,8 +140,8 @@ def test_seven_models_registered():
}
# Exactly one recommended default.
rec = [s for s in specs if s.recommended]
assert [s.id for s in rec] == ["sherpa-parakeet-tdt-v3"]
assert sd.DEFAULT_MODEL_ID == "sherpa-parakeet-tdt-v3"
assert [s.id for s in rec] == ["sherpa-whisper-tiny"]
assert sd.DEFAULT_MODEL_ID == "sherpa-whisper-tiny"
def test_verified_filenames_pinned():
@@ -174,6 +175,31 @@ def test_get_spec_accepts_repo_id():
assert not sd.is_sherpa_model(None)
@pytest.mark.parametrize(
"native_error",
[
OSError("native library could not be loaded"),
RuntimeError("native runtime initialization failed"),
],
)
def test_sherpa_available_degrades_native_loader_failures(monkeypatch, native_error):
"""A broken platform DLL/dylib/so disables Sherpa without crashing APIs."""
from services import sherpa_dictation as sd
real_import = builtins.__import__
def import_with_broken_native(name, *args, **kwargs):
if name == "sherpa_onnx":
raise native_error
return real_import(name, *args, **kwargs)
monkeypatch.setattr(builtins, "__import__", import_with_broken_native)
available, reason = sd.sherpa_available()
assert available is False
assert type(native_error).__name__ in reason
def test_model_resolution_pins_offline_probe_and_download(monkeypatch, tmp_path):
from services import hf_revisions, sherpa_dictation as sd
import huggingface_hub
tests/test_sherpa_model_sizes.py
+33
View File
@@ -83,6 +83,39 @@ def test_declared_size_is_within_tolerance_of_the_measurement(sd, model_id):
)
def test_the_model_catalogue_carries_the_same_sizes(sd):
"""`backend/config/models.yaml` is a second copy of these numbers.
The Model Catalogue's download accounting reads the yaml, the dictation
picker reads `sd._MODELS` — the 2026-08 re-measure fixed only the latter
and the catalogue kept advertising the old 3-4x-wrong figures (#1610
review). Every yaml entry with a `dictation_id` must match the spec.
"""
import yaml
path = os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"backend", "config", "models.yaml",
)
with open(path, encoding="utf-8") as fh:
catalog = yaml.safe_load(fh)
entries = {
e["dictation_id"]: e for e in catalog["models"]
if isinstance(e, dict) and e.get("dictation_id")
}
assert set(entries) == set(sd._MODELS), (
"models.yaml and sherpa_dictation._MODELS list different models"
)
for model_id, entry in entries.items():
assert entry["size_gb"] == pytest.approx(
sd._MODELS[model_id].size_gb
), (
f"models.yaml advertises {entry['size_gb']} GB for {model_id} but "
f"the measured spec says {sd._MODELS[model_id].size_gb} GB — "
f"update the catalogue in the same change as the spec"
)
def test_the_small_fallbacks_are_smaller_than_the_heavy_models(sd):
"""The property that made the old numbers actively misleading.
tests/test_sherpa_probe_fails_closed.py
+54
View File
@@ -0,0 +1,54 @@
"""The sherpa availability probe must fail closed (#1610 review).
``sherpa_available()`` caught ImportError plus OSError/RuntimeError, the types
a broken native wheel usually raises. That set is open-ended — an extension
module can raise anything during init. ``SherpaDictationBackend.is_available()``
calls this directly and ``capture_ws.ws_transcribe`` calls that without a
guard, so an unlisted exception type didn't degrade to "engine unavailable",
it took the dictation WebSocket down.
"""
from __future__ import annotations
import builtins
import importlib
import pytest
@pytest.fixture
def sherpa():
return importlib.import_module("services.sherpa_dictation")
class _Boom(Exception):
"""A native init failure that is neither OSError nor RuntimeError."""
@pytest.mark.parametrize("exc", [
ImportError("no module named sherpa_onnx"),
OSError("cannot load libonnxruntime.so"),
RuntimeError("failed to initialize backend"),
_Boom("ctypes ArgumentError-shaped failure"),
ValueError("unexpected init failure"),
])
def test_any_import_failure_reports_unavailable(sherpa, monkeypatch, exc):
real_import = builtins.__import__
def fake_import(name, *args, **kwargs):
if name == "sherpa_onnx":
raise exc
return real_import(name, *args, **kwargs)
monkeypatch.setattr(builtins, "__import__", fake_import)
ok, detail = sherpa.sherpa_available()
assert ok is False
assert detail # the reason is always reported, never swallowed silently
assert type(exc).__name__ in detail or "not installed" in detail
def test_a_working_install_still_reports_ready(sherpa, monkeypatch):
import sys
import types
monkeypatch.setitem(sys.modules, "sherpa_onnx", types.ModuleType("sherpa_onnx"))
assert sherpa.sherpa_available() == (True, "ready")
tests/test_sherpa_ws_streaming.py
+251
View File
@@ -63,6 +63,25 @@ class _GrowingOnlineRecognizer:
self._endpoint_at = 999
class _SilentOnlineRecognizer:
"""Accepts clear speech but never returns a token."""
def create_stream(self):
return _GrowingStream()
def is_ready(self, s):
return False
def decode_stream(self, s):
pass
def get_result(self, s):
return ""
def is_endpoint(self, s):
return False
@pytest.fixture
def client(monkeypatch):
from fastapi.testclient import TestClient
@@ -140,12 +159,176 @@ def test_partials_before_final(client):
assert partials, f"no partials emitted; saw {types_seen}"
assert finals, f"no final emitted; saw {types_seen}"
assert types_seen.index("partial") < types_seen.index("final")
assert [m["final_kind"] for m in finals] == ["utterance", "summary"]
# Partials grow monotonically in length.
lengths = [len(p["text"]) for p in partials]
assert lengths == sorted(lengths)
def test_streaming_silent_model_falls_back_and_demotes(monkeypatch):
"""Speech into a token-silent streaming recognizer still returns text.
The offline Sherpa handler already detects this runtime failure class. The
streaming handler must expose the same recovery contract instead of
returning a successful-looking empty summary.
"""
import numpy as np
from fastapi.testclient import TestClient
from api.routers import capture_ws as cw
from services import asr_backend as ab
from services import sherpa_dictation as sd
spec = sd.get_spec("sherpa-zipformer-en-20m")
monkeypatch.setattr(cw, "_select_sherpa_spec", lambda ws: spec)
monkeypatch.setattr(ab.SherpaDictationBackend, "is_available",
classmethod(lambda cls: (True, "ready")))
monkeypatch.setattr(ab.SherpaDictationBackend, "ensure_loaded",
lambda self: setattr(self, "_rec", _SilentOnlineRecognizer()))
monkeypatch.setattr(sd, "is_installed", lambda _spec: True)
demoted = []
monkeypatch.setattr(sd, "demote_model", lambda model_id: demoted.append(model_id) or True)
fallback_calls = []
async def fallback(chunks, *, pcm_sr=None, skip_sherpa=False):
fallback_calls.append((b"".join(chunks), pcm_sr, skip_sherpa))
return {
"text": "fallback heard me",
"segments": [{"start": 0.0, "end": 0.25, "text": "fallback heard me"}],
"language": "en",
"engine": "stub-fallback",
}
monkeypatch.setattr(cw, "_transcribe_buffer_full", fallback)
monkeypatch.setitem(sys.modules, "services.refinement",
types.SimpleNamespace(maybe_refine_async=lambda t: None,
collapse_repetitive_artifacts=lambda t: t))
speech = np.full(3000, 3000, dtype=np.int16).tobytes()
from main import app
client = TestClient(app, client=("127.0.0.1", 50000))
with client.websocket_connect(
"/ws/transcribe?model=sherpa-zipformer-en-20m&sr=16000"
) as ws:
ws.send_bytes(speech)
ws.send_text("EOF")
final = None
for _ in range(10):
msg = ws.receive_json()
if msg.get("type") == "final":
final = msg
break
assert final is not None
assert final["text"] == "Fallback heard me."
assert final["final_kind"] == "summary"
assert final["engine"] == "capture-asr-fallback"
assert final["model_silent"] == spec.id
assert final["warning"]
assert demoted == [spec.id]
assert fallback_calls == [(speech, 16000, True)]
def test_streaming_silent_model_does_not_download_a_fallback(monkeypatch):
"""An installed silent Sherpa model must not trigger another model pull.
The session-start probe validates the selected Sherpa weights, but that
says nothing about the fallback. Before demoting to it, recovery must
separately prove the capture fallback is installed — otherwise it invokes
an ASR backend that auto-downloads on a cache miss, turning a failed
dictation into a surprise multi-gigabyte pull.
"""
import numpy as np
from fastapi.testclient import TestClient
from api.routers import capture_ws as cw
from services import asr_backend as ab
from services import sherpa_dictation as sd
spec = sd.get_spec("sherpa-zipformer-en-20m")
monkeypatch.setattr(cw, "_select_sherpa_spec", lambda ws: spec)
monkeypatch.setattr(ab.SherpaDictationBackend, "is_available",
classmethod(lambda cls: (True, "ready")))
monkeypatch.setattr(ab.SherpaDictationBackend, "ensure_loaded",
lambda self: setattr(self, "_rec", _SilentOnlineRecognizer()))
monkeypatch.setattr(sd, "is_installed", lambda _spec: True)
monkeypatch.setattr(sd, "demote_model", lambda _model_id: True)
probes = []
def probe(**kwargs):
probes.append(kwargs)
if kwargs.get("sherpa_model_id") == spec.id:
return None # selected Sherpa model is installed
return {
"error": "asr_model_missing",
"missing_repo_id": "local/fallback-not-installed",
"recommended": None,
}
monkeypatch.setattr(ab, "asr_model_missing_error", probe)
fallback_calls = []
async def fallback(_chunks, *, pcm_sr=None, skip_sherpa=False):
fallback_calls.append(pcm_sr)
return {"text": "this required a download", "segments": []}
monkeypatch.setattr(cw, "_transcribe_buffer_full", fallback)
speech = np.full(3000, 3000, dtype=np.int16).tobytes()
from main import app
client = TestClient(app, client=("127.0.0.1", 50000))
with client.websocket_connect(
"/ws/transcribe?model=sherpa-zipformer-en-20m&sr=16000"
) as ws:
ws.send_bytes(speech)
ws.send_text("EOF")
final = None
for _ in range(10):
msg = ws.receive_json()
if msg.get("type") == "final":
final = msg
break
assert final is not None
assert final["text"] == ""
assert final["model_silent"] == spec.id
assert fallback_calls == []
assert probes == [
{"purpose": "dictation", "sherpa_model_id": spec.id},
{"purpose": "dictation", "skip_sherpa": True, "require_installed": True},
]
@pytest.mark.asyncio
async def test_silent_recovery_needs_fallback_speech_before_demotion(monkeypatch):
"""Noise alone must not persistently disable an otherwise healthy model."""
from api.routers import capture_ws as cw
from services import asr_backend as ab
from services import sherpa_dictation as sd
spec = sd.get_spec("sherpa-zipformer-en-20m")
monkeypatch.setattr(ab, "asr_model_missing_error", lambda **_kwargs: None)
demoted = []
monkeypatch.setattr(sd, "demote_model", lambda model_id: demoted.append(model_id) or True)
async def silent_fallback(_chunks, *, pcm_sr=None, skip_sherpa=False):
assert pcm_sr == 16000
assert skip_sherpa is True
return {"text": "", "segments": []}
monkeypatch.setattr(cw, "_transcribe_buffer_full", silent_fallback)
recovered, segments = await cw._recover_silent_sherpa(
spec, b"\x01\x00" * 3000, 16000,
)
assert recovered == ""
assert segments == []
assert demoted == []
def test_non_streaming_model_uses_offline_handler(monkeypatch):
"""An offline-kind sherpa model routes to the offline cadence handler and
still finalizes (sanity that the kind branch wires up)."""
@@ -190,6 +373,72 @@ def test_non_streaming_model_uses_offline_handler(monkeypatch):
# Polished final (dictation v2): leading capital + terminal punctuation.
assert final["text"] == "Offline text."
assert final["engine"] == "sherpa-onnx-asr"
assert final["final_kind"] == "summary"
def test_offline_silent_model_does_not_download_a_fallback(monkeypatch):
"""Offline silent-model recovery observes the same local-only gate."""
import numpy as np
from fastapi.testclient import TestClient
from api.routers import capture_ws as cw
from services import asr_backend as ab
from services import sherpa_dictation as sd
spec = sd.get_spec("sherpa-whisper-tiny")
monkeypatch.setattr(cw, "_select_sherpa_spec", lambda ws: spec)
monkeypatch.setattr(ab.SherpaDictationBackend, "is_available",
classmethod(lambda cls: (True, "ready")))
monkeypatch.setattr(ab.SherpaDictationBackend, "ensure_loaded",
lambda self: setattr(self, "_rec", object()))
monkeypatch.setattr(ab.SherpaDictationBackend, "_decode_offline",
lambda self, samples, sr: "")
monkeypatch.setattr(sd, "is_installed", lambda _spec: True)
monkeypatch.setattr(sd, "demote_model", lambda _model_id: True)
probes = []
def probe(**kwargs):
probes.append(kwargs)
if kwargs.get("sherpa_model_id") == spec.id:
return None
return {
"error": "asr_model_missing",
"missing_repo_id": "local/fallback-not-installed",
"recommended": None,
}
monkeypatch.setattr(ab, "asr_model_missing_error", probe)
fallback_calls = []
async def fallback(_chunks, *, pcm_sr=None, skip_sherpa=False):
fallback_calls.append(pcm_sr)
return {"text": "this required a download", "segments": []}
monkeypatch.setattr(cw, "_transcribe_buffer_full", fallback)
speech = np.full(3000, 3000, dtype=np.int16).tobytes()
from main import app
client = TestClient(app, client=("127.0.0.1", 50000))
with client.websocket_connect(
"/ws/transcribe?model=sherpa-whisper-tiny&sr=16000"
) as ws:
ws.send_bytes(speech)
ws.send_text("EOF")
final = None
for _ in range(10):
msg = ws.receive_json()
if msg.get("type") == "final":
final = msg
break
assert final is not None
assert final["text"] == ""
assert final["model_silent"] == spec.id
assert fallback_calls == []
assert probes == [
{"purpose": "dictation", "sherpa_model_id": spec.id},
{"purpose": "dictation", "skip_sherpa": True, "require_installed": True},
]
# ── Utterance-windowed offline decoding (dictation v2) ───────────────────────
@@ -253,6 +502,8 @@ def test_offline_silence_gate_commits_mid_session(monkeypatch):
# final. The old behavior produced exactly one (everything at EOF).
assert len(finals) >= 2, f"silence gate never committed mid-session: {msgs}"
assert finals[0]["text"] == "Utterance one."
assert all(m["final_kind"] == "utterance" for m in finals[:-1])
assert finals[-1]["final_kind"] == "summary"
# EOF final = committed pieces + the drained live tail (utterance 2).
assert finals[-1]["text"] == "Utterance one. Utterance one."
# O(n²) fix: every decode was bounded by ONE utterance window — never a