Files
Palash DebnathandClaude Opus 5 41c098e009 feat(demos): ship the demo audio and video the app already advertises (#1517)
* feat(demos): ship the demo audio and video the app already advertises

Every demo asset in the app was a dead link on anything but a Mac.

`personalities.py` has carried a `preview_url` for each of the seven
voice-design presets since they were added; DictationDemo.jsx posts three
bundled WAVs to /transcribe so the feature can be shown without microphone
permission; the Dub workspace reads a manifest and plays a source video plus
four dubbed languages. None of those files were committed, because the tooling
that renders them (scripts/build_demos.sh, scripts/build_dub_demo.sh) hard-
requires macOS `say` — it even carries a `TODO: add espeak-ng path for Linux
contributors`. So the presets returned 404, the replay buttons did nothing, and
the dubbing demo never loaded.

Rendered with VoiceStudio's own engine, which runs wherever the app does:

- 7 voice-design previews (2.2 MB)
- 3 dictation replay clips (1.1 MB) — verified by transcribing them back:
  the conversational and French clips round-trip exactly
- dubbing demo: source + 4 dubbed videos with subtitles and manifest (9.6 MB)

Tooling fixes this turned up:

- build_dub_demo.sh wrote to backend/assets/demo/dubbing, but main.py mounts
  backend/assets/samples at /demo_audio — so the frontend's
  /demo_audio/demo/dubbing/manifest.json could never have resolved even after
  a successful Mac build. Output moved under the mount.
- `say` is now the fallback rather than the requirement: the new
  scripts/render_dub_demo_audio.py renders the five tracks with the engine and
  the shell script picks them up.
- The five demo paragraphs lived in two files. They are now one JSON both read
  — two copies is one edit away from a video whose subtitles disagree with it.
- render_demos_omnivoice.py peak-normalized, which a single-sample transient
  defeats: the Helpdesk preset landed at -30 dB RMS against -17 dB for its
  neighbours, so the preview row played at wildly different volumes. Now EBU
  R128 at -18 LUFS with a -1.5 dBTP ceiling.
- …and pinning the output rate, because loudnorm resamples to 192 kHz
  internally and writes there unless told otherwise, which turned 2.1 MB of
  previews into 17.5 MB of identical-sounding audio.
- update_manifest() looked for a manifest at a path nothing writes, so it
  always printed "not found" and did nothing.
- Dictation is rendered here now too. It was excluded on the grounds that
  `say` was good enough and engine TTS was overkill — true only on macOS.

tests/test_demo_assets_exist.py resolves every advertised URL against the
directory main.py actually mounts, and checks each dubbing subtitle matches the
script its manifest entry claims. A missing static file is not an import error
and not a failing request; nothing would have caught this otherwise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(changelog): stamp the demo-asset entries with their PR ref

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(demos): watermark rendered demo audio, and harden the render scripts

Review findings on #1517:

- Greptile P1: the renderers wrote engine output straight to disk, so a
  re-render shipped demo audio with no provenance mark. These clips play
  back to users as VoiceStudio output — they are synthetic audio leaving
  the app like any other, and now go through mark_synthetic (#1169), the
  one chokepoint every producing route uses. It runs on the file AFTER
  loudnorm, since loudnorm re-encodes what it is handed, and says so
  loudly when marking is unavailable rather than committing an unmarked
  asset. The dubbing renderer shares the same helper.
- CodeRabbit: build_dub_demo.sh checked only source.src.wav before
  deciding it could run without macOS `say`, so a Linux or Windows run
  with four of five tracks present reached a missing one, called `say`,
  and left a half-built bundle. It now requires all five.
- CodeRabbit: shutil.move over an existing path delegates to os.rename,
  which raises FileExistsError on Windows — os.replace overwrites
  atomically everywhere.
- CodeRabbit: the preview test discovered presets in a parametrize
  argument, importing app code at collection time and leaving
  core.personalities in sys.modules for later tests. Discovery moved into
  the test body.

CI: the rendered dub bundle's zh/ja subtitles, its manifest and the
script source are dubbing CONTENT, not UI strings — allowlisted in
test_no_hardcoded_cjk.py with that justification.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(demos): a render that cannot be watermarked fails instead of warning

CodeRabbit and Greptile, #1517: mark_synthetic degrades rather than
raising — correct for generation, wrong for a render script, whose whole
job is to produce files a human then commits. A printed warning on a
scrolling console is not a gate, so both scripts exited 0 with unmarked
assets sitting on disk ready to commit. They now raise, with the reason
and the fix; OMNIVOICE_DEMO_ALLOW_UNMARKED=1 stays for a local listen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* ci: stop a flaky dependency fetch from failing green runs

en-core-web-sm resolves to a direct GitHub release URL, and github.com
intermittently answers `http2 error: refused stream before processing
any application logic`. uv's own three retries all land within the same
few seconds and fail together, so the whole job dies on a dependency
that has nothing to do with the change under test — it cost #1518 and
#1517 an otherwise-green run tonight.

Two changes: back off between whole `uv sync` attempts, which is what
actually clears it, and pass --no-sync to the pytest steps. `uv run`
re-resolves the environment before running, so every test step was a
fresh chance to hit the same fetch even though the install step had
already synced — that is exactly how #1518 failed, in the isolated
backend/tests step, with all 5467 tests already passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* ci: one retry seam for every uv sync, not just the job that failed last

en-core-web-sm resolves to a direct GitHub *release* URL rather than a
package index, and github.com intermittently answers `http2 error:
refused stream before processing any application logic`. uv's own
retries all land inside the same ~10 seconds and fail together, so a job
dies on a dependency unrelated to the change under test. Tonight that
cost four otherwise-green runs across #1515, #1517 and #1518 — and the
first fix only covered the Tests job, so the next failure simply moved
to Smoke (Linux), which syncs separately.

The fetch is per-job, so the fix has to be per-job: scripts/uv-sync-retry.sh
backs off between whole attempts (15s, 45s, 90s) and every workflow that
syncs now goes through it — ci.yml (tests + the platform matrix),
release.yml, security.yml, evals.yml. It still fails loudly after four
attempts, so a genuinely broken lockfile is not disguised as a flake.

The Tests job also lacked the UV_HTTP_TIMEOUT / UV_HTTP_RETRIES the smoke
matrix has always set, which is part of why it was the one that kept
dying; it has them now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(ci): pin the Intel-Mac contract by intent, not by command spelling

test_ci_verifies_intel_mac_as_the_documented_remote_only_host asserted
the literal line `run: uv sync --extra pockettts`, so routing every sync
through scripts/uv-sync-retry.sh read as a broken Intel-Mac contract. The
contract it exists to protect is that the pockettts extra installs ONLY
on backend_supported legs — which the regex now pins, while leaving how
the sync is invoked free to change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* ci: keep every uv run out of the resolver, and bound the retry budget

CodeRabbit, #1517:

- `uv run` re-resolves before running, so the smoke suite, the
  worker-artifact tests, the release test run and the eval run were each
  a fresh chance to hit the flaky direct-URL fetch outside the retry
  loop. All of them pass --no-sync now; the environment is already
  synced by the step that owns the retries. security.yml's
  `uv run --with pip-audit` is deliberately left alone — it layers an
  ephemeral package rather than running the project's own tests.
- The retry count multiplied uv's own budget (UV_HTTP_RETRIES=5 with a
  120 s timeout on the smoke matrix). Three attempts and 60 s of total
  backoff outlast the refusals actually observed while staying well
  inside the jobs' timeout-minutes.
- The Intel-Mac contract test pinned the smoke command literally too, so
  --no-sync tripped it exactly like the sync line did. Same fix: assert
  the contract (smoke runs only on backend_supported legs), not its
  spelling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 23:29:19 +00:00

194 lines
7.3 KiB
YAML

# Security scanning — runs on every PR, on push to main, and weekly.
#
# Complements the CodeRabbit + Greptile app reviews (which fire on PR creation)
# with deterministic, gating checks:
# • gitleaks — secret scanning. HARD FAIL: a leaked credential blocks merge.
# • CodeQL — Python + JS/TS SAST. Results land in the Security tab.
# • bandit — Python SAST (SARIF → Security tab). Reporting, non-gating.
# • pip-audit — Python dependency advisories. Reporting, non-gating.
# • bun audit — frontend dependency advisories. Reporting, non-gating.
#
# Only the secret scan gates the PR. Dependency advisories and bandit findings
# are surfaced as signal (Security tab / job log) rather than blocking every PR
# on a transitive upstream advisory — consistent with the "no ceremony,
# continuous-to-main" cadence.
name: Security
on:
pull_request:
branches: [main]
push:
branches: [main]
schedule:
# Mondays 06:00 UTC — catch advisories disclosed since the last PR.
- cron: "0 6 * * 1"
workflow_dispatch:
env:
# Match ci.yml: run JS actions on Node 24 (GH removes Node 20 in Sep 2026).
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
# Least privilege by default; jobs that upload SARIF opt into security-events.
permissions:
contents: read
# PR branches: a new push cancels the superseded scan (no wasted runners).
# main: every commit keeps its own group, so nothing is cancelled — a merge
# train used to leave a permanent red ✗ ("cancelled") on every intermediate
# commit in the history view even though nothing failed.
concurrency:
group: security-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || 'branch' }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
# ── Secret scanning (gating) ─────────────────────────────────────────────
# Full-history scan on push to main; PR-diff scan on pull_request (faster,
# and the action picks the right mode from the event automatically).
secrets:
name: Secret scan (gitleaks)
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
with:
# gitleaks needs full history to scan all commits on push events.
fetch-depth: 0
# No authed git needed after clone; don't persist GITHUB_TOKEN.
persist-credentials: false
- name: gitleaks
uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# GITLEAKS_LICENSE is only required for GitHub *organizations*; this is
# a personal public repo, so the action runs free without it.
# ── CodeQL SAST (Python + JS/TS) ─────────────────────────────────────────
codeql:
name: CodeQL (${{ matrix.language }})
runs-on: ubuntu-22.04
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: [python, javascript-typescript]
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
# Both targets are interpreted — no compiled build step needed.
build-mode: none
# Scope analysis to shipped product code. The excluded trees never
# ship in the installer's runtime path and produce the bulk of the
# note-level + false-positive findings (file-not-closed in eval
# harnesses, unused alembic migration globals, bind-all in tests,
# path sinks in the legacy Gradio research UI). Queries live in the
# inline config so there's a single source of truth next to
# paths-ignore. paths-ignore is supported here because build-mode is
# `none` (interpreted analysis).
config: |
queries:
- uses: security-and-quality
paths-ignore:
- omnivoice/eval
- research
- tests
- backend/migrations
- "**/*.test.js"
- "**/*.test.jsx"
- "**/*.test.ts"
- "**/*.test.tsx"
- name: Analyze
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{ matrix.language }}"
# ── Python SAST (bandit → SARIF) ─────────────────────────────────────────
bandit:
name: Python SAST (bandit)
runs-on: ubuntu-22.04
permissions:
contents: read
security-events: write
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Setup Python 3.11
uses: actions/setup-python@v5
with:
python-version: "3.11"
# -ll: report MEDIUM+ severity only. -ii: MEDIUM+ confidence only.
# Keeps the SARIF focused on findings worth a human look. The scan step
# is allowed to "fail" (findings present) without failing the job; the
# SARIF upload still runs so results reach the Security tab.
#
# NOTE: the `sarif` output format lives in the `bandit[sarif]` extra
# (pulls in sarif-om + jschema-to-python). Plain `bandit` rejects
# `-f sarif`, so install via the extra spec.
- name: Run bandit
continue-on-error: true
run: |
pipx run --spec 'bandit[sarif]' bandit -r backend/ -ll -ii -f sarif -o bandit.sarif
# continue-on-error: this job is reporting-only. If bandit can't write a
# SARIF for any reason (no findings dir, pipx hiccup), don't fail the job.
- name: Upload bandit SARIF
uses: github/codeql-action/upload-sarif@v3
if: always()
continue-on-error: true
with:
sarif_file: bandit.sarif
category: bandit
# ── Dependency advisories (reporting) ────────────────────────────────────
dependencies:
name: Dependency audit
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
- name: Setup Python 3.11
uses: actions/setup-python@v5
with:
python-version: "3.11"
# Audit the resolved Python environment. Non-gating: a transitive
# advisory with no fix available should not wall off every PR.
- name: pip-audit (Python)
continue-on-error: true
run: |
bash scripts/uv-sync-retry.sh
uv run --with pip-audit pip-audit
# Pin a floor: `bun audit` was added in bun 1.2.x, so guarantee it exists.
- name: Setup Bun
uses: oven-sh/setup-bun@v1
with:
bun-version: "1.2"
# `bun audit` reports advisories against the frontend lockfile. Non-gating
# for the same reason; also tolerant of older bun without the subcommand.
- name: bun audit (frontend)
continue-on-error: true
working-directory: frontend
run: |
bun install --frozen-lockfile
bun audit