# AnimateDiff (SD 1.5) AnimateDiff adds motion to a frozen Stable Diffusion 1.5 checkpoint by injecting a temporal-attention module at 20 UNet slots. The base SD 1.5 model, VAE, and text encoder are unchanged; only the motion module produces the temporal residual that turns a batch of independent frames into a coherent animation. Reference: Guo et al., "AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning" (https://arxiv.org/abs/2307.04725). ## Download weights - Motion module (v3, recommended) - fp16 safetensors: https://huggingface.co/conrevo/AnimateDiff-A1111/resolve/main/motion_module/mm_sd15_v3.safetensors - original checkpoint: https://huggingface.co/guoyww/animatediff/resolve/main/v3_sd15_mm.ckpt - SD 1.5 base model - any SD 1.5 checkpoint works. `realisticVisionV60B1` and `toonyou_beta3` are the ones used in guoyww's reference configs. - Domain Adapter LoRA (optional, v3 only, sharpens the base UNet's output toward the motion module's trained distribution) - ckpt: https://huggingface.co/guoyww/animatediff/resolve/main/v3_sd15_adapter.ckpt - place under your `--lora-model-dir` and reference in the prompt as ``. The motion module is `~836 MB` and loads alongside the SD 1.5 UNet via `--motion-module`. ## Motion module versions Per [animatediff.net/models](https://animatediff.net/models): | Module | Base | Native res | Character | |---------------------|------|------------|-----------| | `mm_sd_v14.ckpt` | 1.5 | 256x256 | earliest, more jittery | | `mm_sd_v15.ckpt` | 1.5 | 256x256 | improved stability over v1.4 | | `mm_sd_v15_v2.ckpt` | 1.5 | 384x384 | significantly better motion dynamics | | `v3_sd15_mm.ckpt` | 1.5 | 512x512 | smoothest, highest quality; pairs with a Domain Adapter LoRA | | `mm_sdxl_v10_beta` | SDXL | 512x512 | experimental, not yet supported here | Match your `-H -W` to the module's native resolution for best results. v3 is trained at 512x512 - going smaller (e.g. 384x384) still works but the motion character is closer to v2. ## Examples Generate an 8-frame animation at 512x512, seed 42, 20 steps. The sampler / scheduler / CFG values below match what mm_sd15_v3 was trained with; using SD 1.5 defaults (euler_a, low CFG) produces noise-like output. ``` .\bin\Release\sd-cli.exe -M vid_gen \ --model ..\models\checkpoints\realisticVisionV60B1.safetensors \ --motion-module ..\models\animatediff\mm_sd15_v3.safetensors \ --offload-to-cpu --diffusion-fa \ -p "a red apple on a wooden table" \ --cfg-scale 8.0 --sampling-method euler --scheduler discrete \ -H 512 -W 512 --video-frames 8 --fps 8 --steps 20 -s 42 \ -o out.avi ``` Generate at the motion module's native 16-frame context (recommended for best temporal quality). Needs more VRAM at 512x512, so drop to 384x384 or use layer streaming: ``` .\bin\Release\sd-cli.exe -M vid_gen \ --model ..\models\checkpoints\realisticVisionV60B1.safetensors \ --motion-module ..\models\animatediff\mm_sd15_v3.safetensors \ --offload-to-cpu --diffusion-fa \ -p "photo of coastline, rocks, storm weather, wind, waves, lightning" \ --cfg-scale 8.0 --sampling-method euler --scheduler discrete \ -H 384 -W 384 --video-frames 16 --fps 8 --steps 20 -s 42 \ -o out.avi ``` Low-VRAM streaming (verified with a 2 GiB cap on RTX 3060): ``` .\bin\Release\sd-cli.exe -M vid_gen \ --model ..\models\checkpoints\realisticVisionV60B1.safetensors \ --motion-module ..\models\animatediff\mm_sd15_v3.safetensors \ --max-vram 2.0 --stream-layers --diffusion-fa \ -p "photo of coastline, rocks, storm weather, wind, waves, lightning" \ --cfg-scale 8.0 --sampling-method euler --scheduler discrete \ -H 384 -W 384 --video-frames 8 --fps 8 --steps 20 -s 42 \ -o out.avi ``` ## Reference-quality reproduction Using guoyww's official reference configs on this impl (RealisticVision v6.0 base + `mm_sd15_v3` or `mm_sd_v15_v2` + native resolution + 16 frames + euler + 25 steps + CFG 8 + linear beta schedule) reproduces the reference AnimateDiff output style. ### v3 (512x512, `mm_sd15_v3`) | Prompt | Sample | |---------------------------------------|--------| | B&W man on stormy coastline | | | Close-up rabbit macro shot | | | Coastline, storm, waves, lightning | | | Old house, storm, forest, night | | ### v2 (384x384, `mm_sd_v15_v2.ckpt`) | Prompt | Sample | |---------------------------------------|--------| | B&W man on stormy coastline | | | Close-up rabbit macro shot | | | Coastline, storm, waves, lightning | | | Old house, storm, forest, night | | Motion is strong for scenes with motion cues in the prompt (storm/waves/wind) and subtle for static subjects (close-up macro), matching reference behavior. v2 has an additional motion module at the UNet middle block that v3 dropped; this impl auto-detects the topology from the checkpoint. ### v3 + Domain Adapter LoRA Attaching the `v3_sd15_adapter` LoRA sharpens the base UNet output toward the training distribution the motion module was fine-tuned against. Same prompt, seed, config as above: Individual fur strands, glowing inner-ear, and richer forest detail become visible compared to the no-LoRA rendering. ``` sd-cli -M vid_gen --model realisticVisionV60B1.safetensors \ --motion-module mm_sd15_v3.safetensors \ --lora-model-dir ./loras \ -p "close up photo of a rabbit ..." ... ``` ## Notes - The motion module was trained at `video_length=16`. Running with `--video-frames 16` gives the best coherence; F=8 works but shows a shorter motion arc. Frame counts up to 32 are supported by the positional encoding but exceed the trained regime and produce more static output. - At `--video-frames 1` the motion module is skipped entirely and the output is bit-identical to `-M img_gen`. This avoids the single-token temporal-attention degeneracy that would otherwise emit an untrained-magnitude residual on a single-frame sample. - The base UNet is frozen, so character identity and style follow the SD 1.5 checkpoint you pass to `--model`. LoRAs and prompt weighting attach to the base model in the usual way. - No mid_block motion module in v3. `mm_sdxl_v10_beta` (SDXL variant) is not supported yet. - Output is written as MJPEG AVI. Use `--fps` to set playback speed.