d91beef0fd314250d8d9b94de86dfea019a8bd96
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Palash DebnathandClaude Fable 5
|
2d5f2e800e |
feat(omnivoice): voice prompts that survive restarts + opt-in FlashInfer (~2.2x) (#1565)
* feat(omnivoice): port upstream VoiceClonePrompt persistence + FlashInfer opt-in Upstream k2-fsa teardown ports, verified with generated voice samples: - VoiceClonePrompt.save()/.load() (upstream format v1, weights_only-safe) on the vendored model, and a disk layer under the in-memory prompt LRU (DATA_DIR/prompt_cache, keyed by ref path+mtime+ref_text+preprocess, 32 newest kept, OMNIVOICE_PROMPT_DISK_CACHE=0 opts out). First generation of a session with a known voice skips the reference re-encode and any auto-transcription pass — verified across two real processes (encodes=1 then encodes=0, same voice). - omnivoice_flashinfer.py ported (packed CFG attention, fused kernels, optional CUDA graphs), schedule adapted to our num_step+1 divergence. Opt-in via OMNIVOICE_FLASHINFER=1|graph, CUDA-only, replaces torch.compile for the session; missing package / apply failure / runtime failure all degrade with a named reason (same #278 contract as compile: classify → unapply → retry once, session latch). Measured 2.20x at batch=1 on an RTX 4090 with byte-identical text and clean ASR round-trip. - Docs: OmniVoice guide gains instruct+reference combination semantics (consistent instruct stabilizes cloning, reference wins conflicts), inline pronunciation control (pinyin / CMU), prompt persistence, and corrects the 'no voice design' claim; performance.md documents both new env knobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: point changelog entries at the real PR number (#1565) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pr): harden FlashInfer lifecycle + prompt-cache writes per review Bot harvest round 1 (#1565): unapply on apply-failure (half-patched model could crash the next render); pin eager-mode FlashInfer inference to one thread too — the attention plan and packed position ids are per-generation module state, so interleaved _gpu_pool workers would corrupt each other; restore the CAPTURED pre-apply attention impl (could be flash_attention_2) instead of assuming sdpa; unique tmp name per prompt-cache write; correct the _forward_logits layout docstring; resolve VoiceClonePrompt at test runtime; docs — Known limits keeps only the limitation, performance.md states the VRAM cost and scopes the fallback claim to classified kernel failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pr): round-2 review — publish only a fully restored model, redact latch reason, tighten CPU-persistence test Greptile: the runtime fallback now unapplies BEFORE swapping generate, so a concurrent render keeps queuing behind the thread-affinity wrapper while teardown mutates modules. CodeRabbit: FlashInfer failure reasons pass through core.failure.sanitize before latching/logging (wheel paths embed the user's home); the save-portability test now creates the tokens on CUDA when available and asserts the persisted payload itself is CPU-resident. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pr): fail-closed latch reason when the sanitizer itself breaks CodeQL empty-except + CodeRabbit round 3: if core.failure.sanitize raises, the raw reason (home paths, wheel paths) was latched anyway. Now only the exception class survives with a fixed redaction note; two regression tests (normal redaction + sanitizer failure). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |