* feat(network): automatic Hugging Face endpoint selection — probe, pick, remember
Restricted-network first-runs (the #984 class: huggingface.co unreachable,
user dead-ends before discovering the mirror setting) now self-heal by
default, while explicit endpoint choices are never second-guessed.
- New backend/services/endpoint_race.py: parallel HTTPS reachability +
latency probes of huggingface.co and the hf-mirror.com community mirror
(3s timeouts). Probes are the only signal — no geo-IP, no third-party
calls. Reachable beats unreachable; with both reachable the official
endpoint wins unless the mirror is decisively faster (anti-flap
hysteresis). The pick is cached in prefs and re-raced only on first run,
a network-classified download failure, staleness (>7 days), or an
explicit "Test again".
- Manual mode is sacred: HF_ENDPOINT env, an hf_endpoint pref, or any
explicit Settings pick disables auto-switching entirely;
OMNIVOICE_HF_ENDPOINT_MODE=manual is a hard opt-out.
- Wiring: the wizard preflight races endpoints when nothing is configured
(honest copy when the mirror wins; warn-not-block when nothing is
reachable); Model Store installs and the model-cache auto-repair resolve
their per-call endpoint= through the cached decision, and a
network-classified failure re-races once per repo per process and
retries on the new winner (same guard pattern as the cache-recovery
ladder).
- Settings → Models → Hugging Face mirror gains "Auto (recommended)":
shows the current pick, measured latency, last-checked time, and a
"Test again" button (POST /api/settings/hf-mirror/test). Existing
explicit configs surface as the matching manual mode. Panel notes that
hf_hub checksums every download regardless of endpoint.
- Tests: policy/cache/failover matrices in tests/test_endpoint_race.py,
preflight + settings + repair-failover integration with mocked probers,
HFMirrorPanel mode tests, and a suite-wide conftest guard that pins the
probers so no test can hit the real network.
- Docs: downloading-models.md and install/troubleshooting.md describe the
automatic default and both opt-outs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): Unreleased entry for automatic HF endpoint selection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(tests): endpoint-probe pin uses an isolated MonkeyPatch and clears the decision cache; dtype guard tolerates stubbed torch
The autouse probe pin requested the shared monkeypatch fixture, hoisting
its setup earlier for every test and reordering teardown against the fp16
guard — which then ran torch.get_default_dtype() on test_torch_compile_gate's
SimpleNamespace stub. The pin now uses its own MonkeyPatch context and also
clears the prefs-cached endpoint decision per test (one test's auto pick
leaked into other tests' preflight labels on CI ordering). The dtype guard
additionally skips non-module torch stubs outright.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(tests): endpoint env vars can no longer leak out of the mirror-settings suite
set_hf_mirror writes os.environ[HF_ENDPOINT] during the test, and
monkeypatch.delenv(raising=False) on an absent var records nothing to
undo — so the write leaked process-wide and flipped later suites'
preflight network checks into the explicit-endpoint branch (the CI-order
failures). Guaranteed save/restore autouse fixture at the source, plus
defensive env shedding in the preflight suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
snapshot_download's resume trusts an existing file by size, so a present-but-
corrupt blob is never re-fetched: the resume-repair 'succeeds' yet the reload
still raises the truncated-cache OSError, and the user was sent to a manual
delete-and-reinstall. Add a force=True path (force_download) and wire it as a
last resort — on the post-resume reload failure, force a full re-download once
(replacing corrupt blobs) and retry the load before falling back to the
actionable message. Force is reached only after a plain resume-repair didn't
fix it, so the common missing-file case still avoids re-downloading everything.
Tests: corrupt cache force-repairs on the 2nd failure (resume then force),
force_download is set only when force=True, and an unfixable cache still
surfaces the 'could not be auto-repaired' message.
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_repair_model_cache attempted snapshot_download exactly once; a single transient
failure (the very cause of an interrupted download) returned False and sent the
user back to a manual delete-and-reinstall. Wrap the re-fetch in a bounded retry
loop (3 attempts default, linear backoff) — snapshot_download resumes between
attempts so retries are cheap and idempotent. Counts/backoff are env-tunable
(OMNIVOICE_MODEL_REPAIR_RETRIES / _BACKOFF_S) for restricted networks and set to
zero-backoff in tests. Offline mode + the actionable fallback message are
unchanged.
Tests: retry-then-succeed self-heals, exhausted-retries returns False after N
attempts, single-attempt tunable, backoff disabled so the suite stays fast.
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An interrupted first download leaves the HF cache with config/tokenizer
files but no weight shard. transformers then raises an OSError ("does not
appear to have a file named pytorch_model.bin or model.safetensors") on
load, which model_manager translated into a 500 with a manual "delete the
model and install it again" instruction — a dead-end for the user.
Make the load path self-repair: on the truncated-cache OSError, re-fetch
just the missing files via snapshot_download (already-present blobs are
skipped, so a near-complete cache repairs fast and a healthy cache never
reaches this branch), then retry the load once. HF offline mode is
respected, and the actionable delete-and-reinstall message is preserved
as the fallback when repair can't fix it.
Adds tests/test_model_cache_repair.py covering completeness detection,
the fast path (no repair on healthy cache), self-repair + retry, the
offline guard, and the repair-failure fallback.
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
debpalash