Files
VoiceStudio/tests/test_prefs_thread_safety.py
ff56865cf7 feat(engines): one-click IndexTTS-2 sidecar install from Settings → Engines (#1083)
* feat(engines): one-click IndexTTS-2 sidecar install from Settings → Engines

IndexTTS-2 required four manual terminal steps (git clone, uv venv,
uv pip install -e ., export OMNIVOICE_INDEXTTS_DIR). This turns that into
a guided in-app install:

- backend/services/sidecar_install.py — parametrized sidecar provisioner
  (SidecarSpec/SPECS so future sidecar engines are one entry, not another
  installer). Resumable background job with step-by-step status: disk-space
  preflight (needs-X/have-Y message), source fetch (git clone --depth 1
  primary, GitHub tarball fallback when git is absent/fails), dedicated
  venv via uv (OMNIVOICE_BUNDLED_UV → PATH resolution; transformers<5
  isolation preserved — the parent env is never touched), import-probe
  verification, IndexTeam/IndexTTS-2 weights into <checkout>/checkpoints
  (where the sidecar actually loads from) via snapshot_download with the
  auto-selected/configured HF endpoint + token — no hardcoded
  huggingface.co — and persistence of OMNIVOICE_INDEXTTS_DIR (os.environ
  for immediate use, prefs.json env.* for the next launch). Idempotent:
  partial installs repair, downloads resume, healthy installs (incl. a
  user's own clone) report already_installed and are never touched.
- API: POST /engines/{id}/install starts the job, GET
  /engines/{id}/install/status polls it, DELETE /engines/{id}/install
  removes an app-managed install (loopback-gated; refuses user-managed
  clones). list_backends() gains one_click_install.
- Frontend: Settings → Engines shows an Install button on the IndexTTS2
  row with per-step progress, live log tail, weight-download %, and
  error+remediation; the manual setup snippet is demoted to a collapsed
  "Manual install" fallback. All strings via i18n (en.json).
- OMNIVOICE_INDEXTTS_DIR joins the Settings env-var allowlist
  (single-sourced from the installer SPECS).
- Docs: docs/engines/indextts.md leads with the one-click flow; manual
  steps become the fallback section. CHANGELOG Unreleased entry added.
- Tests: tests/test_sidecar_install.py (24 cases — happy path, disk-space
  fail, git-absent/git-failing tarball fallback, partial-install repair,
  already-installed/running gating, uninstall safety, spec↔bootstrap
  contract, router wiring) + 6 new EngineCompatibilityMatrix RTL cases.
  API route snapshot regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engines): harden the sidecar installer — review findings

- Route namespace: /engines/sidecar/{id}/install — a dynamic
  /engines/{id}/install would shadow the literal
  POST /engines/sonitranslate/install (engines router registers first);
  regression-guarded by test_sidecar_routes_never_shadow_literal_engine_routes.
- Weights completion marker: a killed-mid-download multi-shard weights dir
  (config.yaml + plausible shards) no longer passes for healthy; the marker
  is written only after snapshot_download returns, so re-runs resume.
- _run_logged: drain thread + proc.wait(timeout) + POSIX process-group kill
  — a grandchild holding the stdout pipe can no longer hang the step past
  its timeout.
- Job log lock: the status poll's list(deque) copy no longer races the
  worker's appends (RuntimeError under active logging).
- Self-heal: a healthy managed install whose env var was lost (prefs wiped)
  is re-pointed by start_install instead of reported already_installed
  while the engine stays unavailable; legacy bootstrap installs (Probe-2
  venv) are trusted via the engine's own probe.
- Single-sourced uv/venv-layout resolution: engines.indextts.bootstrap now
  delegates _locate_uv/_venv_python_path to services.sidecar_install.
- Frontend: stable poll interval (keyed on the running-id set, not the
  status map), reload on a job that finishes before the first poll,
  re-attach to an in-flight job on remount, i18n'd Install aria-label,
  manual-install <details> auto-opens on failure, snippet block hoisted
  out of the JSX IIFE.
- list_backends: sidecar-installable set hoisted out of the per-engine
  loop; exhaustive-shape registry test updated for one_click_install.
- Tests rebind the live services.sidecar_install module per test (other
  suites purge sys.modules["services"], which made router tests
  order-dependent).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): fill in the PR ref (#1083)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): validated tarball fallback + scanner-clean installer

- The pre-filter= extractall fallback (Python < 3.11.4) now extracts
  member-by-member behind the same guards extractall(filter="data")
  enforces — regular files/dirs only, no absolute paths, no ../ escapes,
  resolved-path containment. Kills the new CodeQL py/tarslip (high) and
  Bandit B202 (error) alerts; regression-tested with a malicious tarball
  (test_safe_extract_members_blocks_tar_slip).
- snapshot_download tracks the weights repo's default branch on purpose
  (same policy as every other model download; artifacts are
  checksum-verified by hf_hub) — documented + B615 waived at the call.
- Explanatory comments on the intentional empty-except blocks
  (CodeQL py/empty-except notes).

Verified locally: bandit -ll -ii on the module reports 0 MEDIUM+ findings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engines): address Greptile review — Windows tree kill, prefs write race, poll robustness

- _kill_tree: Windows now uses taskkill /F /T so a git/uv helper spawned by
  the timed-out child can't keep writing into the checkout (POSIX already
  killed the process group). Unit-tested with os.name patched to nt.
- core/prefs: mutations (set_/delete) are serialized behind a module lock —
  the installer worker persisting its env.* key concurrently with a Settings
  write could previously drop whichever key saved first (whole-class fix:
  every threaded prefs writer, not just the installer). Fail-before/
  pass-after: tests/test_prefs_thread_safety.py.
- Matrix polling: at most one in-flight status request per engine (an old
  'running' response can no longer land after a newer 'succeeded' and
  restart the poller), and four consecutive poll failures drop the stale
  snapshot instead of showing "Installing…" and hammering a dead backend
  forever.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 01:13:35 +05:30

63 lines
2.1 KiB
Python

"""prefs.json mutation thread-safety.
Writers run on many threads — FastAPI's request threadpool plus background
workers (the sidecar-engine installer persists its install dir from a worker
thread). Each ``set_``/``delete`` is a load-modify-save of the whole JSON
file; before the module-level mutation lock, two concurrent writers could
interleave (both load, both save) and the later save silently dropped the
other's key. Fail-before/pass-after: this test loses keys reliably on the
unlocked implementation.
"""
import os
import threading
os.environ.setdefault("OMNIVOICE_MODEL", "test")
os.environ.setdefault("OMNIVOICE_DISABLE_FILE_LOG", "1")
from core import prefs
def test_concurrent_set_never_drops_keys(tmp_path, monkeypatch):
monkeypatch.setattr(prefs, "_PREFS_PATH", str(tmp_path / "prefs.json"))
n_threads, n_keys = 8, 25
barrier = threading.Barrier(n_threads)
def writer(tid: int) -> None:
barrier.wait() # maximize interleaving
for i in range(n_keys):
prefs.set_(f"t{tid}.k{i}", tid * 1000 + i)
threads = [threading.Thread(target=writer, args=(tid,)) for tid in range(n_threads)]
for th in threads:
th.start()
for th in threads:
th.join()
data = prefs._load()
expected = {f"t{tid}.k{i}" for tid in range(n_threads) for i in range(n_keys)}
missing = expected - set(data)
assert not missing, f"concurrent writers dropped {len(missing)} keys: {sorted(missing)[:5]}…"
def test_concurrent_set_and_delete_serialize(tmp_path, monkeypatch):
monkeypatch.setattr(prefs, "_PREFS_PATH", str(tmp_path / "prefs.json"))
prefs.set_("keep", 1)
barrier = threading.Barrier(2)
def setter():
barrier.wait()
for i in range(50):
prefs.set_(f"s{i}", i)
def deleter():
barrier.wait()
for i in range(50):
prefs.delete(f"absent{i}") # churns load-save alongside the setter
t1, t2 = threading.Thread(target=setter), threading.Thread(target=deleter)
t1.start(); t2.start(); t1.join(); t2.join()
data = prefs._load()
assert data.get("keep") == 1
assert all(data.get(f"s{i}") == i for i in range(50))