* feat(dub): Smart Fit phase B — per-segment video retime export, drift absorption, fitted subtitles Executes the video side of the Smart Fit plans persisted by Phase A (job["fit_plans"], #347) at export and preview time. Backend: - services/video_retime.py (new, clean-room): two-tier retime executor. ≤48 chunks → the proven single-pass split/trim/setpts/concat filter_complex; above → batches of 40 chunks rendered to intermediate slices (identical libx264 medium/crf20 params, keyframe at t=0) joined losslessly with the concat demuxer. Slices are CFR-resampled (fps=) because setpts leaves VFR-ish timestamps that broke tpad and drifted a frame per retimed chunk on ffmpeg 7.x. Temp slices cleaned on success AND failure/abort. - Drift absorption: fitted track longer than retimed video → freeze-frame tail (tpad=stop_mode=clone) predicted into the last slice / single-pass graph, with residual mux-side tpad; video longer → silence-pad the dub audio chain (apad=whole_dur). ±50 ms tolerance. - VFR guard: probe r_frame_rate vs avg_frame_rate; normalise with fps= before trim/setpts; probe failure degrades gracefully. - Plan resolution: _video_retime_plan_for spans legacy video_stretch_plans (byte-identical resolution + command construction) and fit_plans, gated on the track's own timing_strategy so stale plans never retime a track re-generated under another strategy. - Fitted subtitles: /dub/srt + /dub/vtt accept ?lang= and serve cue times from fitted_segments for Smart Fit tracks; _write_burn_srt does the same for burn-in. burn_subs+retime is now allowed for smart_fit (burn runs AFTER the retime graph); still rejected for legacy stretch_video. - /dub/preview-video resolves the same plan so in-app preview matches export. - Fallback ladder: batch encode failure/timeouts → un-retimed export with a structured core.failure warning (X-Dub-Export-Warning header + job["last_export_warning"]); concat join rejection → one single-pass retry while ≤96 chunks; abort → 409 + proc kill via run_ffmpeg job_id registration (/dub/abort reaches export encodes now) + temp cleanup. Frontend: - Export drawer passes ?lang= on subtitle exports and shows an i18n'd re-encode cost note (~0.5–2× video length on CPU) when a retiming strategy is active — translated in all 21 locales. Tests: tests/test_smart_fit_export.py — plan resolution, batch math, graph parity + new stages, fitted-cue SRT/VTT/burn selection, burn policy, VFR detection; ffmpeg-gated integration renders both executor tiers (batch size forced to 2) and the real /dub/download endpoint, ffprobing durations within ±50 ms across both pad branches. All existing dub export/subtitle/preview/timing tests pass unchanged. Refs docs/competitive-analysis.md Action 1 (dub-length fitting v2); completes Smart Fit (Phase A = #347). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(security): sanitize Smart Fit retime work paths at every sink (CodeQL py/path-injection) The job_id-derived retime work path (retimed_*.mp4 / preview_retimed_*.tmp.mp4) flowed unguarded from dub_export into prepare_smart_fit_video / render_retimed_video and their derived slice/concat paths and ffmpeg argv. Apply the repo's proven inline realpath+startswith containment pattern (helpers/commonpath are not recognized — see #309/#328/#329/#348): - dub_export.py: validate work_path against DUB_DIR at both construction sites (export + preview) and pass the validated realpath onward. - video_retime.py: make both entry points self-defending — realpath + DUB_DIR containment on out_path/work_path before any derivation, raising RetimeError(stage="plan") on escape; slices_dir/slice_path/list_path and RetimeDecision.file_path now all derive from the sanitized value. DUB_DIR is read via module attribute so test fixtures reloading core.config work. - ffmpeg_utils.py: document that all caller-assembled argv paths are realpath-validated upstream. - tests: sandbox DUB_DIR in the executor integration tests (tmp_path) so the new guard sees the test workspace. No behavior change for valid (server-built) paths — the guard only fires on traversal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(smart-fit): patch DUB_DIR on video_retime's own config ref — survives suite-wide reload The retime guard reads video_retime._config.DUB_DIR at call time; the sandbox fixture patched a fresh 'import core.config' instead. Another test reloads core.config in the full suite, so the two module refs diverged — the patch missed and the guard rejected the test's tmp paths (green in isolation, red in CI's full run). Patch the exact ref the guard dereferences. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(dub): resolve DUB_DIR live at call time in retime guards — survive full-suite reload The path-containment guards bound DUB_DIR via a module-level 'from core import config as _config'. Other tests importlib.reload() core.config (sandboxing OMNIVOICE_DATA_DIR), after which the guard checked containment against a stale DUB_DIR while dub_export built the path under the reloaded one — every retime path then 'escaped the dub workspace' (green file-alone, red full-suite: the 5 integration failures CI hit). Re-import DUB_DIR locally in each guard so it always reads the current sys.modules value; simplify the sandbox fixture to patch the canonical module. Verified: full backend suite green on the Smart Fit tests (the 2 remaining settings_store failures are pre-existing on main, unrelated — local data-dir artifact). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(security): clear CodeQL alerts on Smart Fit export — job_id allowlist, proc-registry decouple - py/path-injection (8, video_retime.py): validate job_id with a strict inline regex allowlist (re.fullmatch [A-Za-z0-9_-]{1,64}) at the entry of dub_download and dub_preview_video, before it reaches any filesystem path or ffmpeg argv. The existing realpath containment guards stay as defense-in-depth; the regex barrier is the sanitizer CodeQL recognizes through the service-module call chain. - py/log-injection (4): newline-strip job_id inline at the logger calls in ffmpeg_utils.run_ffmpeg and the two retime-fallback logger.error sites in dub_export. - py/empty-except (3): best-effort cleanup os.remove handlers now log the OSError at debug instead of bare pass (video_retime + both dub_export mux finally blocks; _discard_tmp too for consistency). - py/cyclic-import (2): break the dub_pipeline <-> ffmpeg_utils cycle for real — the subprocess registry (register_proc/unregister_proc/ kill_job_procs/has_active_procs + state) moves to a new stdlib-only leaf module services/proc_registry.py. ffmpeg_utils now imports it at module top (no lazy import); dub_pipeline re-exports every name so dub_core aliases and tests keep working unchanged. No behavior change for valid inputs; invalid job ids now get a clean 400 instead of a 404/containment error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(dub): address #350 review — cancelled-vs-failed retime, logged best-effort excepts, redacted probe logs, narrowed test assert - rc<0 (killed by user cancel) now raises RetimeError(stage='aborted') instead of reporting an ordinary render failure (CodeRabbit) - best-effort cleanup/QC-event excepts log at debug instead of bare pass (CodeQL empty-except x3) - probe failure logs use basename, not full user paths (CodeRabbit/CodeQL) - test_render_cleans_slices_on_failure asserts RetimeError, not Exception Rebuttals (no change needed, see PR comment): fitted-cue subtitles track the fitted AUDIO timeline which is correct even on retime fallback; the planner only emits stretch ratios >1 so the early-exit guard is a true no-op check; '\'' is ffmpeg's own utility quoting for concat lists; has_active_procs is an intentional re-export (noqa'd). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: mergetest <test@local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
62 lines
2.0 KiB
Python
62 lines
2.0 KiB
Python
"""In-flight subprocess registry for dub jobs.
|
|
|
|
Tracks ffmpeg/demucs subprocesses per job id so ``POST /dub/abort/{id}``
|
|
can kill long-running work. This lives in its own stdlib-only leaf module
|
|
so both ``services.dub_pipeline`` and ``services.ffmpeg_utils`` can import
|
|
it at module top: ``run_ffmpeg(job_id=...)`` previously had to lazy-import
|
|
``register_proc``/``unregister_proc`` from dub_pipeline inside the function
|
|
body to dodge the dub_pipeline → ffmpeg_utils → dub_pipeline cycle.
|
|
|
|
``dub_pipeline`` re-exports every name here (including the private state)
|
|
for backward compatibility — ``api.routers.dub_core`` and tests alias them
|
|
through that module.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import threading
|
|
|
|
logger = logging.getLogger("omnivoice.api")
|
|
|
|
_active_procs: dict[str, list] = {}
|
|
_active_procs_lock = threading.Lock()
|
|
|
|
|
|
def register_proc(job_id: str, proc) -> None:
|
|
"""Track an in-flight subprocess so /dub/abort can kill it."""
|
|
with _active_procs_lock:
|
|
_active_procs.setdefault(job_id, []).append(proc)
|
|
|
|
|
|
def unregister_proc(job_id: str, proc) -> None:
|
|
with _active_procs_lock:
|
|
lst = _active_procs.get(job_id)
|
|
if lst and proc in lst:
|
|
lst.remove(proc)
|
|
if lst is not None and not lst:
|
|
_active_procs.pop(job_id, None)
|
|
|
|
|
|
def kill_job_procs(job_id: str) -> None:
|
|
"""Kill every subprocess still running under a given job id. Idempotent."""
|
|
with _active_procs_lock:
|
|
procs = list(_active_procs.get(job_id, []))
|
|
for proc in procs:
|
|
try:
|
|
if proc.returncode is None:
|
|
proc.kill()
|
|
except ProcessLookupError:
|
|
pass
|
|
except Exception as e:
|
|
logger.warning(
|
|
"Failed to kill subprocess for %s: %s",
|
|
job_id.replace("\n", " ").replace("\r", " "), e,
|
|
)
|
|
with _active_procs_lock:
|
|
_active_procs.pop(job_id, None)
|
|
|
|
|
|
def has_active_procs(job_id: str) -> bool:
|
|
with _active_procs_lock:
|
|
return bool(_active_procs.get(job_id))
|