Files
VoiceStudio/tests/backend/services/test_token_resolver.py
T
Palash Debnath 4a6b978df9 Phase 1 Wave 1: HF token persistence + redactor (closes #35) (#91)
* feat(01-01): encrypted settings store + alembic migration (AUTH-02, T-01-01)

Adds the SQLite-backed encrypted settings store that Phase 1 token resolver
will read from. Closes the at-rest plaintext risk for HF tokens (T-01-01).

- backend/services/settings_store.py: get_hf_token / set_hf_token /
  clear_hf_token using Fernet symmetric AEAD. Stored value column never
  contains the literal "hf_" substring.
- backend/services/_secret_key.py: per-install Fernet key derived via
  scrypt(machine-id + 16-byte random salt). machine-id resolution covers
  macOS (ioreg IOPlatformUUID), Linux (/etc/machine-id and dbus fallback),
  Windows (HKLM Cryptography MachineGuid via winreg). Final fallback to
  hostname+user with a warn log.
- backend/migrations/versions/0001_phase1_settings_table.py: alembic
  migration adding `settings(key, value, updated_at)`. Idempotent — checks
  for an existing table so fresh installs (where _BASE_SCHEMA already
  created it) and v0.2.7 upgrades both succeed.
- backend/core/db.py: _BASE_SCHEMA grows the settings table for fresh
  installs; init_db() now runs `alembic upgrade head` after the CREATE.
- backend/migrations/env.py: honours an externally-set sqlalchemy.url so
  tests can point alembic at a fixture DB; falls back to core.config
  DB_PATH for production.
- pyproject.toml: cryptography>=41 added explicitly (RESEARCH.md
  Assumption A1 was checked at execute-time and proved false; the dep was
  not present transitively, so the install would fail without this).

Tests (10 cases, all green):
- Round-trip encryption + plaintext-leakage check (T-01-01 invariant)
- Salt persistence across clear/set cycles
- InvalidToken decrypt path returns None (Open Question #5 resolution)
- Concurrent reads consistent under sqlite WAL
- Alembic upgrade on a hand-built v0.2.7 fixture DB preserves all
  existing tables + seeded rows (CLAUDE.md backward-compat constraint)
- Alembic downgrade -1 drops only the settings table

Refs #35.

* feat(01-01): 3-source HF token resolver + log redactor + 5 read sites patched

Closes the #35 bug class (bare os.environ.get('HF_TOKEN') reads) by routing
every backend HF-token consumer through one resolver, and mitigates
T-01-02 (info disclosure via logs) by stripping `hf_[A-Za-z0-9]{30,}`
substrings from every log record at the root logger.

backend/services/token_resolver.py:
  - resolve(skip)   — 3-source cascade (App → Env → HF-CLI), each source
    validated via huggingface_hub.whoami(); first valid wins.
  - on_401(active) — invalidate cache and re-resolve skipping the source
    that just 401'd (AUTH-06).
  - state()        — three SourceState rows for the Settings UI: set,
    masked preview (hf_…<last 3>), whoami_user, whoami_ok.
  - save_app_token / clear_app_token — wraps settings_store + calls
    huggingface_hub.login(add_to_git_credential=False) per Pitfall #2.
  - 300-second whoami cache so repeated Settings-page renders don't hit
    the HF API.

backend/core/logging_filter.py:
  - HFTokenRedactor(logging.Filter) — regex `hf_[A-Za-z0-9]{30,}` so real
    tokens are masked but `hf_hub` / `hf_token` literals survive.
  - install_redaction_filter() — idempotent attach to root + every handler.

backend/main.py: install the redactor at startup, BEFORE the file
handler is added. Re-installed after the file handler attaches so the
handler-attached filter list includes it too.

Read-side call sites patched (per Pitfall #1 — every HF token read must
flow through token_resolver.resolve()):
  - backend/api/routers/dub_core.py:540  (the original #35 site)
  - backend/api/routers/system.py:38     (_has_hf_token notification)
  - backend/services/model_manager.py:480 (diarization pipeline auth)
  - backend/services/sonitranslate.py:143 (Popen env for SoniTranslate child)
  - backend/services/sonitranslate.py:217 (gradio_client predict call)

New endpoint:
  - GET /system/hf-token/state — returns the 3-source cascade state with
    masked tokens for the Wave 2 Settings UI panel.

Grep gate confirmed clean: zero `os.environ.get("HF_TOKEN")` reads remain
outside token_resolver.py.

Tests (17 new cases, all green):
  - tests/backend/services/test_token_resolver.py: priority cascade, 401
    skip mid-resolve, on_401 fallback, state() shape, save+login
    invariant (add_to_git_credential=False), HUGGING_FACE_HUB_TOKEN
    alias acceptance.
  - tests/backend/core/test_logging_filter.py: msg + args redaction,
    multi-token redaction, non-string args pass-through, short-token
    literals preserved, install_redaction_filter idempotence.

Refs #35.

* feat(01-01): Settings hf-token API endpoints + subprocess env injection (AUTH-03/04)

Backend half of the Wave 2 Settings → API Keys UI plus the AUTH-04
subprocess env-injection invariant.

backend/api/routers/settings.py:
  - POST /api/settings/hf-token       — body {token: str} → save_app_token
  - DELETE /api/settings/hf-token     — also_clear_hf_cli query → clear_app_token
  - GET /api/settings/hf-token/state  — same shape as token_resolver.state()
  All three are gated by `Depends(require_loopback)` at the router level
  (threat T-01-03 mitigation; non-loopback Host → 403).

backend/main.py: router mounted alongside existing API routers.

Subprocess env injection (AUTH-04, threat T-01-04 disposition=accept):
  - backend/services/sonitranslate.py already updated in Task 2 to read
    via token_resolver.resolve() and inject HF_TOKEN + YOUR_HF_TOKEN into
    the SoniTranslate child env block.
  - backend/services/gpu_sandbox.py: NOT patched — the GPU sandbox runs
    in-process TTS generation that uses the parent's already-loaded HF
    state. Adding env injection there is a no-op (parent and child share
    state via multiprocessing.Pipe before any HF API call).
  - backend/services/model_manager.py:480 (Task 2): resolves in-process,
    no subprocess crosses here.
  - backend/api/routers/exports.py: subprocess.Popen calls only spawn
    `open` / `explorer` / `xdg-open` — file-manager launchers with no
    HF needs. Skipped per Task 3 conservative-patching rule.

So the canonical AUTH-04 site for this milestone is sonitranslate.py.
Future SubprocessBackend work in Phase 2 will inherit the same pattern.

Tests (8 new cases, all green):
  - tests/backend/test_engine_spawn_token.py
    * POST /hf-token loopback → 200 + state.active == "app"
    * POST /hf-token non-loopback → 403 ("loopback origin required")
    * DELETE /hf-token clears settings_store + state.active == None
    * GET /hf-token/state returns 3 source rows in priority order
    * GET /hf-token/state non-loopback → 403
    * env block contains HF_TOKEN + YOUR_HF_TOKEN when resolver returns one
    * env block does NOT contain an injected empty HF_TOKEN when resolver
      returns None
    * source-level check that backend/services/sonitranslate.py still
      reads via token_resolver.resolve() (regression guard against
      silent reverts of the AUTH-04 wiring)

Full Wave 1 test suite: 35/35 green. Phase 0 smoke tests still green.

Refs #35.

* docs(01-01): SUMMARY + STATE update for Phase 1 Wave 1 completion

Records execution outcome of the 3-task plan: 10 files created, 9 modified,
35 new test cases, 5 read sites patched, grep gate clean. Documents the
two Rule-3/Rule-2 deviations applied (cryptography dep, env.py URL
override), the subprocess-launcher inventory for Phase 2, and the
known stray edit to the main repo's pyproject.toml that needs a one-
line user action to revert.

Updates STATE.md current-position table, progress bar, and open TODOs to
point at Wave 2 (Plan 01-02) and Wave 3 (Plan 01-03) as the next steps.
2026-05-20 05:10:37 +05:30

255 lines
8.7 KiB
Python

"""Tests for backend/services/token_resolver.py — AUTH-01 / AUTH-03 / AUTH-06.
Covers the 3-source cascade (App → Env → HF-CLI), the on_401 fallback,
the state() shape consumed by the Settings UI, and the save/clear API the
React panel will call.
"""
import sys
from unittest.mock import MagicMock
import pytest
APP_TOKEN = "hf_appapp00000000000000000000000000000000abc"
ENV_TOKEN = "hf_envenv00000000000000000000000000000000def"
CLI_TOKEN = "hf_clicli00000000000000000000000000000000ghi"
@pytest.fixture
def fresh_resolver(monkeypatch, tmp_path):
"""Fresh import of services.token_resolver + a tmp DB-backed settings
store. Also clears HF env vars so the env source is empty by default."""
monkeypatch.setenv("OMNIVOICE_DATA_DIR", str(tmp_path))
monkeypatch.delenv("HF_TOKEN", raising=False)
monkeypatch.delenv("HUGGING_FACE_HUB_TOKEN", raising=False)
# Aggressive purge — see note in test_settings_store.py.
for mod in list(sys.modules):
if (
mod == "core" or mod.startswith("core.")
or mod == "services" or mod.startswith("services.")
):
del sys.modules[mod]
from core import db as _db
_db.init_db()
from services import token_resolver
token_resolver.invalidate_cache()
return token_resolver
def _mock_whoami(monkeypatch, mapping):
"""Patch huggingface_hub.whoami so that a known {token: result} mapping
drives validity. `mapping[token]` may be a dict like {"name": "alice"}
or an Exception instance to raise."""
import huggingface_hub
def _fake_whoami(token=None, **kwargs):
if token in mapping:
outcome = mapping[token]
if isinstance(outcome, Exception):
raise outcome
return outcome
raise RuntimeError(f"unexpected token in whoami: {token!r}")
monkeypatch.setattr(huggingface_hub, "whoami", _fake_whoami)
def _mock_get_token(monkeypatch, value):
import huggingface_hub
monkeypatch.setattr(huggingface_hub, "get_token", lambda: value)
def test_resolve_priority_app_wins(fresh_resolver, monkeypatch):
"""All three sources set + valid → App wins."""
tr = fresh_resolver
from services import settings_store
settings_store.set_hf_token(APP_TOKEN)
monkeypatch.setenv("HF_TOKEN", ENV_TOKEN)
_mock_get_token(monkeypatch, CLI_TOKEN)
_mock_whoami(monkeypatch, {
APP_TOKEN: {"name": "alice"},
ENV_TOKEN: {"name": "bob"},
CLI_TOKEN: {"name": "carol"},
})
tr.invalidate_cache()
result = tr.resolve()
assert result is not None
assert result.token == APP_TOKEN
assert result.source == "app"
assert result.username == "alice"
def test_resolve_skips_empty_falls_to_env(fresh_resolver, monkeypatch):
"""No app token, env set, no CLI → Env wins."""
tr = fresh_resolver
monkeypatch.setenv("HF_TOKEN", ENV_TOKEN)
_mock_get_token(monkeypatch, None)
_mock_whoami(monkeypatch, {ENV_TOKEN: {"name": "bob"}})
tr.invalidate_cache()
result = tr.resolve()
assert result is not None
assert result.token == ENV_TOKEN
assert result.source == "env"
def test_resolve_returns_none_when_all_empty(fresh_resolver, monkeypatch):
tr = fresh_resolver
_mock_get_token(monkeypatch, None)
_mock_whoami(monkeypatch, {})
tr.invalidate_cache()
assert tr.resolve() is None
def test_resolve_skips_401_app_falls_to_env(fresh_resolver, monkeypatch):
"""App set but 401, env set and valid → resolver returns Env, not App.
AUTH-06 mid-resolve fallback."""
tr = fresh_resolver
from services import settings_store
from huggingface_hub.errors import HfHubHTTPError
settings_store.set_hf_token(APP_TOKEN)
monkeypatch.setenv("HF_TOKEN", ENV_TOKEN)
_mock_get_token(monkeypatch, None)
err = HfHubHTTPError("401 Unauthorized", response=MagicMock(status_code=401))
_mock_whoami(monkeypatch, {
APP_TOKEN: err,
ENV_TOKEN: {"name": "bob"},
})
tr.invalidate_cache()
result = tr.resolve()
assert result is not None
assert result.source == "env"
assert result.token == ENV_TOKEN
def test_on_401_cascade(fresh_resolver, monkeypatch):
"""Active source = 'app', call on_401('app') → next valid source."""
tr = fresh_resolver
from services import settings_store
settings_store.set_hf_token(APP_TOKEN)
monkeypatch.setenv("HF_TOKEN", ENV_TOKEN)
_mock_get_token(monkeypatch, None)
_mock_whoami(monkeypatch, {
APP_TOKEN: {"name": "alice"},
ENV_TOKEN: {"name": "bob"},
})
tr.invalidate_cache()
# Initial resolve picks app.
assert tr.resolve().source == "app"
# When the consumer reports a 401 from "app", fall back.
fallback = tr.on_401("app")
assert fallback is not None
assert fallback.source == "env"
def test_state_returns_three_rows(fresh_resolver, monkeypatch):
"""state() returns one row per source in priority order, with masked
token + whoami fields populated. The Settings UI consumes this."""
tr = fresh_resolver
from services import settings_store
settings_store.set_hf_token(APP_TOKEN)
monkeypatch.setenv("HF_TOKEN", ENV_TOKEN)
_mock_get_token(monkeypatch, None)
_mock_whoami(monkeypatch, {
APP_TOKEN: {"name": "alice"},
ENV_TOKEN: {"name": "bob"},
})
tr.invalidate_cache()
s = tr.state()
assert "sources" in s
assert "active" in s
assert [r.source for r in s["sources"]] == ["app", "env", "hf-cli"]
assert s["active"] == "app"
# masked format: hf_…<last 3 chars of the token>
app_row = s["sources"][0]
assert app_row.set is True
assert app_row.masked is not None and app_row.masked.startswith("hf_")
assert app_row.masked.endswith(APP_TOKEN[-3:])
assert app_row.whoami_user == "alice"
assert app_row.whoami_ok is True
# HF-CLI row: unset
cli_row = s["sources"][2]
assert cli_row.set is False
assert cli_row.masked is None
def test_save_writes_both_store_and_login(fresh_resolver, monkeypatch):
"""save_app_token() must update settings_store AND call
huggingface_hub.login(token=..., add_to_git_credential=False) so the
canonical HF file (~/.cache/huggingface/token) is in sync."""
tr = fresh_resolver
import huggingface_hub
login_calls = []
monkeypatch.setattr(
huggingface_hub,
"login",
lambda **kw: login_calls.append(kw),
)
tr.save_app_token(APP_TOKEN)
# Settings store now has it.
from services import settings_store
assert settings_store.get_hf_token() == APP_TOKEN
# huggingface_hub.login was called with the right kwargs.
assert len(login_calls) == 1
call = login_calls[0]
assert call.get("token") == APP_TOKEN
# Pitfall #2: must always pass add_to_git_credential=False.
assert call.get("add_to_git_credential") is False
def test_save_uses_add_to_git_credential_false(fresh_resolver, monkeypatch):
"""Standalone check for Pitfall #2 (explicit so reviewers see the
invariant being enforced)."""
tr = fresh_resolver
import huggingface_hub
captured = {}
def _fake_login(**kwargs):
captured.update(kwargs)
monkeypatch.setattr(huggingface_hub, "login", _fake_login)
tr.save_app_token(APP_TOKEN)
assert captured.get("add_to_git_credential") is False
def test_clear_app_token_removes_from_store(fresh_resolver, monkeypatch):
tr = fresh_resolver
import huggingface_hub
monkeypatch.setattr(huggingface_hub, "login", lambda **kw: None)
monkeypatch.setattr(huggingface_hub, "logout", lambda: None)
tr.save_app_token(APP_TOKEN)
from services import settings_store
assert settings_store.get_hf_token() == APP_TOKEN
tr.clear_app_token(also_clear_hf_cli=False)
assert settings_store.get_hf_token() is None
def test_clear_app_token_also_logs_out_when_requested(fresh_resolver, monkeypatch):
tr = fresh_resolver
import huggingface_hub
monkeypatch.setattr(huggingface_hub, "login", lambda **kw: None)
logout_calls = []
monkeypatch.setattr(huggingface_hub, "logout", lambda: logout_calls.append(True))
tr.save_app_token(APP_TOKEN)
tr.clear_app_token(also_clear_hf_cli=True)
assert logout_calls == [True]
def test_resolve_accepts_hugging_face_hub_token_alias(fresh_resolver, monkeypatch):
"""HF docs list HUGGING_FACE_HUB_TOKEN as the legacy env-var name; the
resolver must accept either."""
tr = fresh_resolver
monkeypatch.delenv("HF_TOKEN", raising=False)
monkeypatch.setenv("HUGGING_FACE_HUB_TOKEN", ENV_TOKEN)
_mock_get_token(monkeypatch, None)
_mock_whoami(monkeypatch, {ENV_TOKEN: {"name": "bob"}})
tr.invalidate_cache()
result = tr.resolve()
assert result is not None
assert result.source == "env"
assert result.token == ENV_TOKEN