Files

T

4cc55ab852 Fast model downloads: Xet fast path + accurate progress (FDL W0–W2 + W4) (#424 )

* feat(downloads): Xet fast path + accurate progress (FDL W0–W2)

Make model downloads fast and show accurate downloaded/remaining/speed.
Research confirmed hf-xet already implements the IDM/uGet technique
(content-defined chunking, parallel byte-range gets, dedup, resume), and
the spike found all 25 catalog repos are Xet-backed — so the win is
driving Xet well + accurate progress, not a custom downloader.

W1 — maximize + guarantee Xet:
- pin huggingface_hub>=1.7 + hf-xet>=1.1 (was transitive); no hf_transfer
- drive snapshot_download with explicit tqdm_class + max_workers + endpoint
- opt-in HF_XET_HIGH_PERFORMANCE / HDD sequential-write knobs (default off)
- /system/info reports fast_download {xet_enabled, xet_version, high_perf}

W2 — accurate progress:
- dry_run preflight -> install_plan event (exact total/cached/remaining)
- utils/download_aggregator.py: one overall bar; byte bars (by id) vs the
  "Fetching N files" count bar; windowed rate; emits one 'aggregate' event
- frontend overall bar (speed/remaining/ETA), cached-skip, ⚡ fast badge

Known limit (verified live): under Xet+hf_hub 1.7.2 per-file byte bars
never advance/close via tqdm, so mid-download the bar is file-granular and
bytes flush to the exact total on completion. Classic-LFS/mirror repos get
true byte progress (W4).

Drive-by: download.py used os.walk without importing os (latent NameError
in _validate_snapshot_has_weights on every install) — fixed.

Tests: tests/backend/setup/test_download_preflight.py (10). Spike + plan
under .planning/quick/260613-fdl-fast-model-downloads/.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(downloads): opt-in mirror + cancel + docs (FDL W4)

- mirror (FDL-10): snapshot_download(endpoint=) honours prefs hf_endpoint /
  env HF_ENDPOINT on preflight + download (per-call, no process-wide env).
  Documented as the classic-LFS path (no Xet) for restricted networks.
- cancel (FDL-11): POST /models/install/cancel {repo_id} stops further
  retries at the next boundary, emits install_cancelled, clears the cooldown
  (cancel is intent, not failure). Frontend treats it as a terminator.
- docs (FDL-12): docs/downloading-models.md (Xet fast path, progress
  semantics + byte-speed limitation, opt-in tuning, mirror, cancel,
  troubleshooting) + README pointer. Docs-sync rule satisfied.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(planning): model-management v2 cleanup plan (mm2)

GSD plan for cleaning the model-management subsystem: registry unload-on-
switch + per-engine unload() (fixes VRAM leak), model_lifecycle facade,
unified idle/timeout config, bounded cooldowns, sidecar VRAM self-report,
cache-fallback logging. Planning artifact only — no code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(downloads): reconcile with main's HF_HUB_DISABLE_XET; honest status

Rebasing onto main surfaced that main forces HF_HUB_DISABLE_XET=1 (classic
LFS) because Xet progress bypasses the tqdm hook — the same limitation found
here. Reconcile instead of fight:

- /system/info fast_download now reports runtime truth: xet_installed +
  xet_active (installed AND not HF_HUB_DISABLE_XET) + xet_enabled alias. The
  ⚡ badge only shows when Xet actually runs; startup log says
  "downloads: Xet disabled → legacy LFS".
- complete(): clear the rate window before the final flush so crediting the
  full size in one step can't emit an absurd instantaneous rate.
- docs/downloading-models.md rewritten: default is legacy LFS for accurate
  progress; Xet is opt-in via HF_HUB_DISABLE_XET=0. hf-xet pin stays (ready
  for a future Xet progress hook).

W2 (preflight total/remaining + aggregate bar + exact completion) is the
value on either path; W1's "maximize Xet" is dormant by main's design.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(downloads): opt-in segmented multi-connection accelerator (FDL W3)

Since main forces Xet off (HF_HUB_DISABLE_XET=1), the default path is
single-stream legacy LFS — so a segmented downloader is the way to get BOTH
parallel speed and live byte progress.

- services/segmented_download.py: async multi-connection Range downloader for
  one file — parallel byte-ranges, resume (.part + manifest), per-segment
  short-read truncation guard, optional sha256/etag verify, cancel, and a
  single-stream fallback when the server won't range. Auth-safe: the HF
  Authorization header is sent only to huggingface.co/hf.co and never
  forwarded to a CDN host on redirect (unit-tested).
- dispatch (download.py): opt-in via prefs segmented_downloader / env
  OMNIVOICE_SEGMENTED_DOWNLOAD (default off). When on and Xet inactive,
  fetches each file into the HF cache mirroring hf_hub_download (blobs +
  snapshot symlinks + refs/main), feeding real bytes to the aggregator. Any
  failure falls back to snapshot_download — never breaks a correct install.
- fix: complete() was adding a full total on top of accumulated segmented
  bytes (2x); now replaces byte bars so the sum is exactly total.

Verified live (accelerator on): real byte progress to ~16.6 MB/s, final
bytes==total, /models installed=True, delete frees correctly.

Tests: test_segmented_download.py (7) + aggregator double-count regression.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(downloads): relocate FDL tests to top-level; loop-isolate segmented test

CI runs the full suite, which exposed a pre-existing test-isolation leak:
several tests/backend/** fixtures purge core.*/services.* from sys.modules
under a temp OMNIVOICE_DATA_DIR and never restore, leaving core.config/core.db
bound to a dead temp dir. It only bites when collection order puts a purging
test ahead of a real-DB reader (test_longform_jobs). Adding tests under
tests/backend/setup/ reordered collection and tripped it.

Fix without touching the shared (fragile) fixtures or risking class-identity
breakage from a blanket sys.modules restore:
- move the two FDL test files to top-level tests/ (tests/test_fdl_*.py) so
  tests/backend/** collection order is identical to main — longform passes.
- rewrite the segmented test to run each case under asyncio.run() (fresh loop)
  instead of asyncio.get_event_loop(), which an earlier async test can leave
  closed in the full suite.

Full suite green locally: 1364 passed, 0 failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-13 20:15:15 +05:30

5.6 KiB

Raw Blame History

RESEARCH — Fast HuggingFace model downloads (2026)

Date: 2026-06-13 · For: 260613-fdl-PLAN.md

Bottom line

As of mid-2026 the fast path is hf-xet, on by default in modern huggingface_hub. Xet is itself a chunk-level, content-defined, massively-parallel downloader with adaptive concurrency — it is the "IDM/uGet-style segmented download," done for you and dedup-aware. hf_transfer is deprecated. Rolling your own segmented downloader or bridging to aria2 is not worth it as a default; the only thing we must build is (a) better driving + progress UI and (b) an opt-in segmented path for the legacy-LFS long tail (repos Xet doesn't back).

Installed in this repo: huggingface_hub 1.7.2, hf_xet present. snapshot_download here supports max_workers, tqdm_class, endpoint, dry_run (confirmed via inspect).

1. hf-xet — USE (default, no action needed beyond pinning)

Content-defined chunks grouped into blocks ("xorbs") in a content-addressable store; download = send file SHA256 → get reconstruction metadata + presigned URLs → fetch needed xorb ranges in parallel → reassemble; already-present chunks skipped (dedup). Auto-used by snapshot_download/hf_hub_download for Xet-backed repos since huggingface_hub 0.32. 2–3× over Git-LFS, up to ~1 GB/s. Knobs (defaults already tuned): HF_XET_NUM_CONCURRENT_RANGE_GETS (16), adaptive concurrency ON (max 64), HF_XET_DATA_MAX_CONCURRENT_FILE_DOWNLOADS (8), chunk cache disabled by default (better for pure download), HF_XET_HIGH_PERFORMANCE=1 (opt-in max throughput, needs RAM/bandwidth), HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 (HDD). 64-bit only.

2. hf_transfer — AVOID (deprecated)

HF_HUB_ENABLE_HF_TRANSFER flagged deprecated; Xet supersedes it. Historically broke tqdm progress / had no callbacks — directly conflicts with the accurate-progress goal. Successor for max throughput is HF_XET_HIGH_PERFORMANCE=1.

3. huggingface_hub native concurrency — USE defaults

snapshot_download(max_workers=...) = parallel FILES (default 8), orthogonal to Xet's intra-file chunk parallelism. For OmniVoice's 1–few-large-file models the win is mostly Xet's intra-file parallelism; don't crank max_workers (multiplies buffer pressure). Resume is automatic via cache + ETag (no resume_download flag to manage).

https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download

4. Custom IDM-style Range downloader — AVOID as default, BUILD as opt-in for LFS

/resolve/<rev>/<file> 302-redirects to CDN (Cloudfront) which honors Range + parallel byte-ranges. Catch: follow redirect, do NOT forward Authorization to the CDN host (presigned URL carries auth), verify ETag/sha256, auth on first hop only. Redundant vs Xet for Xet-backed repos (HF closed issue #3232 as "use Xet"), but genuinely helps non-Xet/legacy-LFS repos which get no intra-file parallelism. → our Wave 3 opt-in.

https://github.com/huggingface/huggingface_hub/issues/3232

5. aria2 — OPTIONAL, rejected for OmniVoice

aria2c -x16 -s16 -c --header="Authorization: Bearer <token>" is 3–5× on plain LFS, but: no dedup (worse than Xet for Xet repos), per-OS GPLv2 binary to package (parity burden — would have to be opt-in anyway), stdout/RPC progress scraping. The custom httpx path covers the same need with no binary. → not bundled.

https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f

6. Mirrors / HF_ENDPOINT — OPTIONAL, region-gated, breaks Xet

HF_ENDPOINT=https://hf-mirror.com redirects Hub traffic (standard for China). Xet CAS/presigned URLs point at HF infra → mirrors generally don't serve the Xet protocol → traffic falls back to classic LFS (no dedup, no Xet parallelism). So mirror and Xet fast-path are mutually exclusive; the realistic China stack is mirror + LFS + (our opt-in) segmented accelerator. → our Wave 4 opt-in, per-call endpoint= not process-wide.

7. Progress / speed — USE `tqdm_class` (xet-aware) + `dry_run` preflight

Unlike hf_transfer, Xet reports progress through the same tqdm interface; huggingface_hub aggregates per-file/thread bytes into a shared bar and feeds the tqdm_class you pass. So snapshot_download(tqdm_class=...) yields reliable aggregate bytes/total/rate/ETA even under parallel fetch. snapshot_download(dry_run=True) returns per-file sizes + cached flags → use for "will download X of Y, N GB" preflight. Speed sampling tunable via HF_XET_DATA_PROGRESS_UPDATE_INTERVAL (200ms).

Recommended architecture (→ plan)

Pin huggingface_hub>=1.7 + hf-xet; let Xet be the default (it IS the IDM technique). Drive snapshot_download(repo_id, tqdm_class=OmniVoiceProgress, max_workers=8, endpoint=<opt-in mirror>); dry_run=True first for total/remaining; aggregate bytes in a backend tracker → one overall bar (speed/remaining/ETA). Opt-in only: HF_XET_HIGH_PERFORMANCE (max speed), HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY (HDD), a custom httpx segmented downloader for legacy-LFS repos, and an HF_ENDPOINT mirror (classic-LFS fallback). Never enable hf_transfer; never bundle aria2; never make the segmented path the default.

5.6 KiB Raw Blame History Unescape Escape