chore: untrack local backlog/ task tracker (gitignored)
This commit is contained in:
@@ -150,3 +150,6 @@ playwright-report/
|
||||
|
||||
# probe — generated HTML reports
|
||||
tests/probe/reports/
|
||||
|
||||
# local Backlog.md task tracker; never tracked upstream
|
||||
backlog/
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
project_name: "omnivoice-studio"
|
||||
default_status: "To Do"
|
||||
statuses: ["To Do", "In Progress", "Done"]
|
||||
labels: []
|
||||
date_format: yyyy-mm-dd
|
||||
max_column_width: 20
|
||||
default_editor: "vi"
|
||||
auto_open_browser: false
|
||||
default_port: 6420
|
||||
remote_operations: true
|
||||
auto_commit: false
|
||||
filesystem_only: false
|
||||
bypass_git_hooks: false
|
||||
check_active_branches: true
|
||||
active_branch_days: 30
|
||||
task_prefix: "task"
|
||||
@@ -1,42 +0,0 @@
|
||||
---
|
||||
id: TASK-1
|
||||
title: Investigate OmniVoice VRAM/lifecycle bug (stuck model loads, unkillable abandoned jobs)
|
||||
status: To Do
|
||||
assignee: []
|
||||
created_date: '2026-07-28 17:00'
|
||||
updated_date: '2026-07-28 17:00'
|
||||
labels:
|
||||
- omnivoice
|
||||
- bug
|
||||
- vram
|
||||
- lifecycle
|
||||
- mps
|
||||
dependencies: []
|
||||
priority: high
|
||||
ordinal: 1000
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
OmniVoice (MPS backend) is chronically VRAM-starved, causing a recurring lifecycle failure:
|
||||
|
||||
1. A heavy model load exceeds the 1200s execution-time budget.
|
||||
2. OmniVoice "abandons" the GPU-pool worker, but **cannot kill it**. The abandoned worker keeps running and **holds the MPS device**.
|
||||
3. Every subsequent synth (REST `/v1/audio/speech`, `/generate`, and even the MCP `generate_speech`) queues behind the device-holding abandoned worker and hangs (60s-180s+ timeouts).
|
||||
|
||||
Evidence from `~/Library/Application Support/OmniVoice/omnivoice.log` (read 2026-07-28):
|
||||
- `Model load exceeded 1200.0s; resetting GPU pool` appears **3 times**: 2026-07-20 11:34, 2026-07-28 16:22, 2026-07-28 16:58 (recurred ~36 min apart the same day, i.e. a retry cascade: stuck, retry, still stuck because the device is held).
|
||||
- `abandoned ... cannot be killed: it keeps running and keeps holding the device` appears **34 times**.
|
||||
- VRAM / memory-pressure events: **40 times**.
|
||||
|
||||
OmniVoice's own message references internal issues #730 / #1190.
|
||||
|
||||
The MCP path (`mcp__omnivoice__generate_speech`) works when the device is free (succeeded 2026-07-28 12:17 @35s and 12:30 @13s), but hangs once the abandoned worker holds the device (16:22 onward). v0.4.2 (released 2026-07-28) did NOT introduce this per its changelog (update-UX + model-repair + localization), so it predates the release.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Root cause of the VRAM starvation: which resident model + which load contends, and why the load exceeds 1200s of compute.
|
||||
- [ ] Root cause of the unkillable abandoned worker (the lifecycle bug behind #730/#1190): why OmniVoice cannot kill/clean up an abandoned GPU-pool worker.
|
||||
- [ ] A reproducer (input/state that reliably triggers the stuck load).
|
||||
- [ ] Fix so abandoned workers are actually killed (release the device) OR so loads cannot starve past the budget.
|
||||
- [ ] Verify the fix prevents recurrence under sustained use (no stuck loads in a long-run test).
|
||||
@@ -1,35 +0,0 @@
|
||||
---
|
||||
id: TASK-2
|
||||
title: Exploration - durable fixes for OmniVoice VRAM pressure / synth reliability
|
||||
status: To Do
|
||||
assignee: []
|
||||
created_date: '2026-07-28 17:00'
|
||||
updated_date: '2026-07-28 17:00'
|
||||
labels:
|
||||
- omnivoice
|
||||
- exploration
|
||||
- vram
|
||||
- reliability
|
||||
dependencies:
|
||||
- TASK-1
|
||||
priority: medium
|
||||
ordinal: 2000
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
While TASK-1 investigates the root lifecycle bug, evaluate these durable mitigations (from OmniVoice's own error message + the voice-on-reaction integration findings) so the TTS is reliable for unattended / reaction-triggered use. Document the tradeoff of each, then pick the one(s) to adopt.
|
||||
|
||||
Candidates to explore:
|
||||
|
||||
1. **Flush caches / Unload the resident model** (OmniVoice UI: Settings -> Models, or an API call) before a heavy/long synth, to free VRAM so the load does not starve. Question: can this be automated (API or script) so it runs before each long synth or on a schedule?
|
||||
2. **Set the engine to CPU** (Settings -> Models) to remove MPS VRAM contention; stable but slower. Quantify the speed/quality tradeoff and whether it is acceptable for the voice-on-reaction use case.
|
||||
3. **Raise `OMNIVOICE_GENERATE_TIMEOUT_S`** to tolerate long generations. Caveat: this does NOT fix the unkillable-worker hang (the device stays held regardless); evaluate whether it helps or only delays the failure.
|
||||
4. **Shorter text per synth**: chunk long messages into multiple shorter synths to reduce per-call load. Evaluate the chunking strategy and how to concatenate the audio.
|
||||
5. **Plugin-side PocketTTS fallback** (deployed 2026-07-28 in the hermes-agent `table_image_fallback` voice module, `_voice.py`): on the OmniVoice-trigger emoji, try OmniVoice; on timeout/failure, fall back to PocketTTS. Makes the voice-on-reaction feature robust regardless of OmniVoice's state (OmniVoice quality when healthy, PocketTTS speed as fallback). This complements, not replaces, TASK-1.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] For each candidate: documented tradeoff (reliability gain vs cost/complexity/quality).
|
||||
- [ ] Pick the durable fix(es) to adopt for unattended use; record the decision (and link TASK-1's root cause).
|
||||
- [ ] If a candidate is automated (e.g. auto-flush before a long synth), implement and verify it prevents the stuck-load recurrence.
|
||||
Reference in New Issue
Block a user