Review follow-up. The budget matched model names by substring, so a custom
repo whose name contains "turbo", "small" or "base" (or a word such as
"database") got a reduced budget and could be admitted to CUDA without
enough memory. Reduced budgets now apply only to the exact OpenAI
checkpoint ids, .en variants included. Any other repository, fine-tunes
included, keeps the conservative 5.0 GB, as the engine doc says.
Review follow-up. A generation first waits in the GPU pool's queue, on its
own clock (GPU_QUEUE_TIMEOUT_S, 1800 s by default), before its execution
budget starts, so a 630 s wait could still give up before the backend
returned its queue error. The generate wait now adds the queue budget.
Transcription is unchanged: run_transcribe_guarded starts its 300 s clock
at submission, so queue time already counts against it.
Parity tests pin each tool's wait above the backend's own worst case,
from ASR_TRANSCRIBE_TIMEOUT_S, GPU_QUEUE_TIMEOUT_S and generate_timeout_s
on cpu, cuda and mps. A later backend change that outgrows the MCP wait
now fails a test. A new case covers a CPU budget larger than the GPU one.
Review follow-up. The shared loader resamples soundfile-readable audio to
16 kHz with a linear interpolation, which can alias 44.1/48 kHz input into
Whisper's band. PyTorch Whisper now reads such audio at its native rate,
as before, so the pipeline's band-limited resampler converts it. Only
files soundfile can't open (MP4/M4A) take the ffmpeg path, which resamples
properly. New tests: a 48 kHz stereo WAV reaches the pipeline at 48 kHz
without ffmpeg, a POST /transcribe with an .m4a returns 200, and the MCP
tool names M4A bytes .m4a.
The CUDA preflight demanded 5.0 GB of free VRAM for every model, sized for
full large-v3. The default model is large-v3-turbo, which the pipeline
loads in fp16: about 1.6 GB of weights, not the 3.2 GiB fp32 figure in the
old comment. So a 6 GB card with nothing else resident reported 5.0 GB
free and was sent to CPU every time, although CUDA ran the same audio in
37 s against minutes on CPU.
The budget is now fp16 weights + 1.5 GB workspace (batch 16) + 0.5 GB
headroom, per model and capped at the old 5.0 GB. That is 3.6 GB for turbo
and 5.0 GB for full large-v3 and any unrecognised model. Both CPU-fallback
warnings name the model and the OMNIVOICE_ASR_VRAM_PREFLIGHT=0 opt-out.
The engine doc lists the budgets.
Fixes#2041
The MCP tools gave up on a backend POST after OMNIVOICE_MCP_TIMEOUT_S,
default 120 s, while the backend's own budgets run longer: 300 s for ASR,
and 300 s or more for generation. A transcription the backend would have
finished came back as an empty client-side timeout, and the abandoned
job kept holding the worker. The docstring said the timeout followed
OMNIVOICE_GENERATE_TIMEOUT_S, but it never read it.
Unset, each tool now waits for the backend's budget plus 30 s, never less
than 120 s. transcribe follows OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S, and
generate_speech follows the larger of the GPU and CPU generation budgets,
scaled by text length like the backend. An explicit
OMNIVOICE_MCP_TIMEOUT_S still wins. docs/mcp.md says so.
Fixes#2040
PyTorchWhisperBackend.transcribe read uploads with soundfile, and
libsndfile cannot open MP4/M4A (AAC). /transcribe and the MCP tool both
accept .m4a, so every such upload failed with "Format not recognised"
before ASR ran. It now uses _load_audio_16k_mono_f32, the loader the
sherpa backend already uses: soundfile first, then the validated ffmpeg
path, returning the 16 kHz mono float32 Whisper expects.
Fixes#2039
The floors test checked only uv.lock, and pyproject still allowed
transformers>=5.5.0 and protobuf>=4.25. A fresh resolution, or an install
without the lock, could still pick a release below the advisory fixes.
The manifest now requires transformers>=5.10.0 and protobuf>=5.29.6. The
lock already resolved 5.15.1 and 7.36.0, so no package version changes;
only the recorded requirements do.
Palash Debnath