feat: add configurable conditioning cache for all models (#2034)

docs/caching.md

@@ -2,6 +2,16 @@
 
 Caching methods accelerate diffusion inference by reusing intermediate computations when changes between steps are small.
 
+### Conditioning Cache
+
+Conditioning results are cached per model context using an LRU cache. The default
+capacity is **0 (disabled) for `sd-cli`** and **4 entries for `sd-server` and the C
+API**. Set `--conditioning-cache-size N` to change the limit; `0` disables caching.
+For example, `sd-cli -m model.safetensors -p "a cat" --conditioning-cache-size 4`
+enables the cache in the CLI. The C API option is
+`sd_ctx_params_t::conditioning_cache_size`, initialized by `sd_ctx_params_init()`.
+This cache is independent of the diffusion-step `--cache-mode` options below.
+
 ### Cache Modes
 
 | Mode | Target | Description |

docs/edit.md

@@ -26,6 +26,9 @@ Stable-diffusion.spp also supports basic Unet-based editing models like instruct
 
 ## Configuring Reference Modes (`--ref-image-args`)
 
+For a one-time input transform before reference presets and model processing,
+including cropping, padding, and resizing algorithms, see [Image preprocessing](./image_preprocessing.md).
+
 Different DiT-based editing models require different configurations to process reference images correctly (e.g., whether to use a Vision Language Model (VLM) encoder or pass VAE-encoded images directly to the DiT).
 
 To simplify this, we provide **Presets**. By default, the system automatically selects the best preset based on the model architecture. However, you can override this using the `--ref-image-args` argument.

docs/esrgan.md

@@ -2,6 +2,8 @@
 
 You can use ESRGAN—such as the model [RealESRGAN_x4plus_anime_6B.pth](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth)—to upscale the generated images and improve their overall resolution and clarity.
 
+RGBA images, including Qwen Image 2.1 output, keep their alpha channel during model upscaling and hires fix. ESRGAN processes the RGB channels; the alpha channel is resized with bilinear interpolation and recombined with the upscaled image.
+
 - Specify the model path using the `--upscale-model PATH` parameter. example:
 
 ```bash

docs/image_preprocessing.md

@@ -0,0 +1,173 @@
+# Image preprocessing
+
+Use `--image-preprocess` to transform each image input once, before generation:
+
+```sh
+sd-cli ... \
+  --image-preprocess "target=init,mode=crop-resize,filter=lanczos,antialias=true" \
+  --image-preprocess "target=mask,filter=nearest-exact" \
+  --image-preprocess "target=ref,index=0,mode=fit-pad,width=768,height=768,filter=bicubic"
+```
+
+CLI and server image loaders decode at the original resolution. The generation
+entry point merges input defaults with user rules and prepares one transformed
+image per input. The original pipeline then consumes those images, including
+its mandatory canvas adaptation, reference resizing, and encoder preprocessing.
+
+```text
+native-resolution image
+  -> input defaults + user overrides
+  -> one input transform
+  -> original generation pipeline and model-specific processing
+```
+
+These rules do not override internal VAE, CLIP/VLM, ControlNet, or pixel-patch preprocessing.
+`--ref-image-args` retains its existing meaning and runs after this input transform.
+
+## Inputs and defaults
+
+| `target` | Input | Default geometry | Indexed? |
+| --- | --- | --- | --- |
+| `init` | img2img image or video first frame | Center crop to the generation aspect ratio, then resize | No |
+| `end` | Video last frame | Center crop, then resize | No |
+| `mask` | Inpainting mask | Inherit init geometry; otherwise center crop, then resize | No |
+| `control` | Control image | Center crop, then resize | No |
+| `ref` | Reference images | Preserve source dimensions | Yes |
+| `ip-adapter` | IP-Adapter image | Preserve source dimensions | No |
+| `id` | PhotoMaker identity images | Preserve source dimensions | Yes |
+| `control-frame` | Control video frames | Center crop, then resize | Yes |
+
+Canvas defaults use the aligned generation dimensions. Reference, IP-Adapter,
+and identity inputs use their original dimensions unless overridden. Default
+resampling is nearest for images and nearest-exact for masks.
+
+These defaults are shared by CLI, server, and C API. Moving geometry out of
+the loaders replaces the previous CLI/server BOX/sRGB resizing, so default
+pixels are not guaranteed to match earlier builds.
+
+Reference video and audio preprocessing are outside these image rules.
+Preprocessing options apply to `img_gen` and `vid_gen`, not standalone upscale
+or ADetailer mode. ADetailer clears the user's rules for its internal crops.
+
+## Rules
+
+Rules are comma-separated `key=value` lists. Repeat the CLI option or separate
+rules with semicolons. Every rule requires a `target` and at least one option.
+Rule syntax and input compatibility are checked when image/video generation
+starts. Unknown keys, invalid values, duplicate keys in a rule, missing images,
+and out-of-range indices cause generation to fail with an error log.
+
+Omit `index` to configure every image of that type; otherwise use a zero-based
+index. CLI directory inputs follow filename order. Indexed rules override
+type-wide rules field by field, regardless of order. At equal specificity,
+the last value for a field wins. `auto` selects the input preset.
+
+| `mode` | Input transform |
+| --- | --- |
+| `auto` | Use the input's default geometry |
+| `none` | Keep source dimensions without resizing, cropping, or padding |
+| `stretch` | Resize to the target dimensions |
+| `crop` | Crop a target-sized rectangle without resizing; fail if the source is too small |
+| `crop-resize` | Crop to the target aspect ratio, then resize |
+| `fit-pad` | Fit the entire image inside the target dimensions, preserving aspect ratio, then pad |
+
+`width` and `height` must be specified together as positive integers. They
+override the input transform's dimensions, not the generation or encoder size.
+For a native-size preset, specifying dimensions without a mode selects stretch.
+`mode=none` with explicit dimensions different from the source is contradictory
+and is rejected.
+
+`anchor=center|top|bottom|left|right` selects crop/padding placement.
+`pad_color=#RRGGBB` or `#RRGGBBAA` selects padding, defaulting to opaque black.
+A grayscale mask uses the first color component.
+
+`filter=auto|nearest|nearest-exact|bilinear|bicubic|lanczos` selects resampling.
+`antialias=auto|true|false` enables antialiasing automatically for filtered
+downscaling; explicit true requires bilinear, bicubic, or Lanczos.
+Filtered RGBA resizing uses premultiplied alpha.
+
+`canny=true|false` enables edge detection for any supported image target,
+defaulting to `false`. It runs once after geometry, before the original
+generation pipeline, including with `mode=none`. Grayscale, grayscale-alpha,
+RGB, and RGBA inputs are supported; alpha is preserved.
+
+Each input has its own Canny setting. Indexed rules can enable or disable it
+for individual references, identity images, or video control frames.
+
+```sh
+--image-preprocess "target=init,mode=fit-pad,canny=true"
+--image-preprocess "target=ref,index=0,mode=none,canny=true"
+--image-preprocess "target=control-frame,index=2,canny=true"
+```
+
+Init and mask sources must have the same dimensions. The mask inherits the
+init crop, resize, and padding coordinates, while retaining its own filter,
+padding value, and Canny setting. Conflicting mask geometry is rejected. An
+omitted mask remains absent until the original pipeline creates its default mask.
+
+## Downstream behavior
+
+`mode=none` only skips the input geometry transform. For example:
+
+```sh
+--image-preprocess "target=init,mode=none" \
+--image-preprocess "target=ref,mode=none"
+```
+
+The init image is still adapted to the generation canvas by the original
+pipeline. Reference images still follow `--ref-image-args` and model-specific
+resizing. CLIP retains its fixed input dimensions and normalization. HiDream-O1
+retains its original pixel-reference and visual preprocessing.
+
+Existing sharing between consumers is preserved: for example, Wan img2video
+uses the same adapted first frame for VAE conditioning and CLIP. High-resolution
+passes reuse the prepared images and apply their original size adaptation;
+they do not apply the user's crop a second time.
+
+To disable reference resizing before VAE encoding, use
+`--ref-image-args "resize_before_vae=false"` or the server field
+`"ref_image_args": "resize_before_vae=false"`. This is separate from
+`target=ref,mode=none`, which only skips input geometry. Model constraints
+still apply.
+
+## Server requests
+
+Native image/video requests and SDAPI accept `image_preprocess` as a string or
+an array of rule strings:
+
+```json
+{
+  "image_preprocess": [
+    "target=init,mode=fit-pad,filter=bicubic",
+    "target=mask,filter=nearest-exact",
+    "target=ref,index=0,mode=none"
+  ]
+}
+```
+
+OpenAI-compatible requests accept it through
+`<sd_cpp_extra_args>{...}</sd_cpp_extra_args>` in the prompt.
+Request rules replace server-default rules. Generation metadata records the
+user rules; image encodings and channel conventions are unchanged.
+
+## C API
+
+Set `image_preprocess` on the existing image/video generation parameters.
+The `generate_image()` and `generate_video()` signatures are unchanged:
+
+```c
+sd_img_gen_params_t params;
+sd_img_gen_params_init(&params);
+/* Set prompt, original-resolution input images, and generation options. */
+params.image_preprocess.rules = "target=init,mode=crop-resize,filter=lanczos;"
+                                "target=mask,filter=nearest-exact";
+bool ok = generate_image(ctx, &params, &images, &count);
+```
+
+Both generation parameter initializers set `image_preprocess.rules` to `NULL`,
+selecting input presets. Rule strings are borrowed for the synchronous call.
+The library owns temporary transformed pixels; caller images and arrays are
+not modified. Add `canny=true` to the desired target's rule in
+`image_preprocess.rules` to enable Canny.
+
+The parameter structs have grown; applications and bindings must be rebuilt.

examples/cli/README.md

@@ -14,6 +14,12 @@ equivalent to `--log-level verbose`. If repeated, the last logging option wins.
 For direct image repair or automatic post-generation YOLOv8 detection followed by cropped inpainting, see
 [ADetailer](../../docs/adetailer.md).
 
+Use repeatable `--image-preprocess` rules to select resizing, cropping, padding,
+and resampling separately for each image input. Add `canny=true` to any input
+rule for edge detection. See
+[Image preprocessing](../../docs/image_preprocessing.md) for input selectors,
+input defaults, downstream model processing, mask alignment, and examples.
+
 Metadata mode inspects PNG/JPEG container metadata without loading any model:
 
 ```bash

examples/cli/main.cpp

@@ -41,7 +41,6 @@ struct SDCliParams {
     std::string metadata_format = "text";
 
     sd_log_level_t log_level = SD_LOG_INFO;
-    bool canny_preprocess    = false;
     bool convert_name        = false;
 
     preview_t preview_method = PREVIEW_NONE;
@@ -107,10 +106,6 @@ struct SDCliParams {
         };
 
         options.bool_options = {
-            {"",
-             "--canny",
-             "apply canny preprocessor (edge detection)",
-             true, &canny_preprocess},
             {"",
              "--convert-name",
              "convert tensor name (for convert mode)",
@@ -268,7 +263,6 @@ struct SDCliParams {
             << "  metadata_format: \"" << metadata_format << "\",\n"
             << "  log_level: " << log_level_name(log_level) << ",\n"
             << "  color: " << (color ? "true" : "false") << ",\n"
-            << "  canny_preprocess: " << (canny_preprocess ? "true" : "false") << ",\n"
             << "  convert_name: " << (convert_name ? "true" : "false") << ",\n"
             << "  preview_method: " << previews_str[preview_method] << ",\n"
             << "  preview_interval: " << preview_interval << ",\n"
@@ -328,9 +322,7 @@ void sd_log_cb(enum sd_log_level_t level, const char* log, void* data) {
 
 bool load_images_from_dir(const std::string dir,
                           std::vector<SDImageOwner>& images,
-                          int expected_width  = 0,
-                          int expected_height = 0,
-                          int max_image_num   = 0) {
+                          int max_image_num = 0) {
     if (!fs::exists(dir) || !fs::is_directory(dir)) {
         LOG_ERROR("'%s' is not a valid directory\n", dir.c_str());
         return false;
@@ -358,7 +350,7 @@ bool load_images_from_dir(const std::string dir,
             int width             = 0;
             int height            = 0;
             int loaded_channel    = 0;
-            uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, loaded_channel, expected_width, expected_height);
+            uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, loaded_channel, 0, 0);
             if (image_buffer == nullptr) {
                 LOG_ERROR("load image from '%s' failed", path.c_str());
                 return false;
@@ -652,10 +644,11 @@ int main(int argc, const char* argv[]) {
 
     SDCliParams cli_params;
     SDContextParams ctx_params;
+    ctx_params.conditioning_cache_size = 0;
     SDGenerationParams gen_params;
 
-    parse_args(argc, argv, cli_params, ctx_params, gen_params);
     sd_set_log_callback(sd_log_cb, (void*)&cli_params);
+    parse_args(argc, argv, cli_params, ctx_params, gen_params);
 
     if (cli_params.mode == METADATA) {
         MetadataReadOptions options;
@@ -751,16 +744,8 @@ int main(int argc, const char* argv[]) {
 
     auto load_image_and_update_size = [&](const std::string& path,
                                           SDImageOwner& image,
-                                          bool resize_image    = true,
                                           int expected_channel = 3) -> bool {
-        int expected_width  = 0;
-        int expected_height = 0;
-        if (resize_image && gen_params.width_and_height_are_set()) {
-            expected_width  = gen_params.width;
-            expected_height = gen_params.height;
-        }
-
-        if (!load_sd_image_from_file(image.put(), path.c_str(), expected_width, expected_height, expected_channel)) {
+        if (!load_sd_image_from_file(image.put(), path.c_str(), 0, 0, expected_channel)) {
             LOG_ERROR("load image from '%s' failed", path.c_str());
             return false;
         }
@@ -783,7 +768,7 @@ int main(int argc, const char* argv[]) {
 
     if (gen_params.init_image_path.size() > 0) {
         const bool native_init = cli_params.mode == IMG_GEN || cli_params.mode == ADETAILER;
-        if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image, true, native_init ? 0 : 3)) {
+        if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image, native_init ? 0 : 3)) {
             return 1;
         }
     }
@@ -798,7 +783,7 @@ int main(int argc, const char* argv[]) {
         gen_params.ref_images.clear();
         for (auto& path : gen_params.ref_image_paths) {
             SDImageOwner ref_image({0, 0, 0, nullptr});
-            if (!load_image_and_update_size(path, ref_image, false, 0)) {
+            if (!load_image_and_update_size(path, ref_image, 0)) {
                 return 1;
             }
             gen_params.ref_images.push_back(std::move(ref_image));
@@ -839,41 +824,22 @@ int main(int argc, const char* argv[]) {
     if (gen_params.mask_image_path.size() > 0) {
         if (!load_sd_image_from_file(gen_params.mask_image.put(),
                                      gen_params.mask_image_path.c_str(),
-                                     gen_params.get_resolved_width(),
-                                     gen_params.get_resolved_height(),
+                                     0,
+                                     0,
                                      1)) {
             LOG_ERROR("load image from '%s' failed", gen_params.mask_image_path.c_str());
             return 1;
         }
-    } else {
-        sd_image_t generated_mask = {0, 0, 1, nullptr};
-        generated_mask.data       = (uint8_t*)malloc(gen_params.get_resolved_width() * gen_params.get_resolved_height());
-        if (generated_mask.data == nullptr) {
-            LOG_ERROR("malloc mask image failed");
-            return 1;
-        }
-        generated_mask.width  = gen_params.get_resolved_width();
-        generated_mask.height = gen_params.get_resolved_height();
-        memset(generated_mask.data, 255, gen_params.get_resolved_width() * gen_params.get_resolved_height());
-        gen_params.mask_image.reset(generated_mask);
     }
 
     if (gen_params.control_image_path.size() > 0) {
         if (!load_sd_image_from_file(gen_params.control_image.put(),
                                      gen_params.control_image_path.c_str(),
-                                     gen_params.get_resolved_width(),
-                                     gen_params.get_resolved_height())) {
+                                     0,
+                                     0)) {
             LOG_ERROR("load image from '%s' failed", gen_params.control_image_path.c_str());
             return 1;
         }
-        if (cli_params.canny_preprocess) {  // apply preprocessor
-            preprocess_canny(gen_params.control_image.get(),
-                             0.08f,
-                             0.08f,
-                             0.8f,
-                             1.0f,
-                             false);
-        }
     }
 
     if (gen_params.ip_adapter_image_path.size() > 0) {
@@ -890,8 +856,6 @@ int main(int argc, const char* argv[]) {
         gen_params.control_frames.clear();
         if (!load_images_from_dir(gen_params.control_video_path,
                                   gen_params.control_frames,
-                                  gen_params.get_resolved_width(),
-                                  gen_params.get_resolved_height(),
                                   gen_params.video_frames)) {
             return 1;
         }
@@ -900,10 +864,7 @@ int main(int argc, const char* argv[]) {
     if (!gen_params.pm_id_images_dir.empty()) {
         gen_params.pm_id_images.clear();
         if (!load_images_from_dir(gen_params.pm_id_images_dir,
-                                  gen_params.pm_id_images,
-                                  0,
-                                  0,
-                                  0)) {
+                                  gen_params.pm_id_images)) {
             return 1;
         }
     }

examples/common/common.cpp

@@ -571,6 +571,10 @@ ArgOptions SDContextParams::get_options() {
          "number of threads to use during computation (default: -1). "
          "If threads <= 0, then threads will be set to the number of CPU physical cores",
          &n_threads},
+        {"",
+         "--conditioning-cache-size",
+         "maximum number of conditioning results cached per model context (default: " + std::to_string(conditioning_cache_size) + ", 0 disables caching)",
+         &conditioning_cache_size},
     };
 
     options.bool_options = {
@@ -822,6 +826,10 @@ bool SDContextParams::resolve(SDMode mode) {
 }
 
 bool SDContextParams::validate(SDMode mode) {
+    if (conditioning_cache_size < 0) {
+        LOG_ERROR("error: conditioning-cache-size must be non-negative");
+        return false;
+    }
     if (mode == CONVERT) {
         const bool has_convert_input = model_path.length() != 0 ||
                                        clip_l_path.length() != 0 ||
@@ -898,6 +906,7 @@ std::string SDContextParams::to_string() const {
     std::ostringstream oss;
     oss << "SDContextParams {\n"
         << "  n_threads: " << n_threads << ",\n"
+        << "  conditioning_cache_size: " << conditioning_cache_size << ",\n"
         << "  model_path: \"" << model_path << "\",\n"
         << "  clip_l_path: \"" << clip_l_path << "\",\n"
         << "  clip_g_path: \"" << clip_g_path << "\",\n"
@@ -992,6 +1001,7 @@ sd_ctx_params_t SDContextParams::to_sd_ctx_params_t(bool taesd_preview) {
     sd_ctx_params.pulid_weights_path              = pulid_weights_path.c_str();
     sd_ctx_params.tensor_type_rules               = tensor_type_rules.c_str();
     sd_ctx_params.n_threads                       = n_threads;
+    sd_ctx_params.conditioning_cache_size         = conditioning_cache_size;
     sd_ctx_params.wtype                           = wtype;
     sd_ctx_params.rng_type                        = rng_type;
     sd_ctx_params.sampler_rng_type                = sampler_rng_type;
@@ -1128,6 +1138,9 @@ ArgOptions SDGenerationParams::get_options() {
          "Key-value list to set up the way the reference images are processed (empty = auto-detect from model weigths)",
          (int)',',
          &ref_image_args},
+        {"", "--image-preprocess",
+         "Image preprocessing rule: target=init|end|mask|control|ref|ip-adapter|id|control-frame,index=N,mode=auto|none|stretch|crop|crop-resize|fit-pad,filter=auto|nearest|nearest-exact|bilinear|bicubic|lanczos,antialias=auto|true|false,width=W,height=H,anchor=center|top|bottom|left|right,pad_color=#RRGGBB[AA],canny=true|false. Repeat for multiple rules.",
+         (int)';', &image_preprocess},
     };
 
     options.int_options = {
@@ -1308,11 +1321,6 @@ ArgOptions SDGenerationParams::get_options() {
          "automatically increase the indices of references images based on the order they are listed (starting with 1).",
          true,
          &increase_ref_index},
-        {"",
-         "--disable-auto-resize-ref-image",
-         "disable auto resize of ref images",
-         false,
-         &auto_resize_ref_image},
         {"",
          "--circular",
          "enable circular padding on both axes for tileable output",
@@ -1870,8 +1878,6 @@ bool decode_base64_image(const std::string& encoded_input,
 static bool parse_image_json_field(const json& parent,
                                    const char* key,
                                    int channels,
-                                   int expected_width,
-                                   int expected_height,
                                    SDImageOwner& out_image) {
     if (!parent.contains(key)) {
         return true;
@@ -1883,14 +1889,12 @@ static bool parse_image_json_field(const json& parent,
     if (!parent.at(key).is_string()) {
         return false;
     }
-    return decode_base64_image(parent.at(key).get<std::string>(), channels, expected_width, expected_height, out_image);
+    return decode_base64_image(parent.at(key).get<std::string>(), channels, 0, 0, out_image);
 }
 
 static bool parse_image_array_json_field(const json& parent,
                                          const char* key,
                                          int channels,
-                                         int expected_width,
-                                         int expected_height,
                                          std::vector<SDImageOwner>& out_images) {
     if (!parent.contains(key)) {
         return true;
@@ -1909,7 +1913,7 @@ static bool parse_image_array_json_field(const json& parent,
             return false;
         }
         SDImageOwner image;
-        if (!decode_base64_image(item.get<std::string>(), channels, expected_width, expected_height, image)) {
+        if (!decode_base64_image(item.get<std::string>(), channels, 0, 0, image)) {
             return false;
         }
         out_images.push_back(std::move(image));
@@ -2008,6 +2012,29 @@ static bool resolve_model_file_from_dir(const std::string& model_name,
     return false;
 }
 
+bool SDGenerationParams::parse_image_preprocess_json(const std::string& json_str) {
+    const auto value = json::parse(json_str, nullptr, false);
+    std::string rules;
+    if (value.is_string()) {
+        rules = value.get<std::string>();
+    } else if (value.is_array()) {
+        for (const auto& item : value) {
+            if (!item.is_string()) {
+                LOG_ERROR("image_preprocess must contain rule strings");
+                return false;
+            }
+            if (!rules.empty())
+                rules += ";";
+            rules += item.get<std::string>();
+        }
+    } else {
+        LOG_ERROR("image_preprocess must be a string or array of strings");
+        return false;
+    }
+    image_preprocess = std::move(rules);
+    return true;
+}
+
 bool SDGenerationParams::from_json_str(
     const std::string& json_str,
     const std::function<std::string(const std::string&)>& lora_path_resolver) {
@@ -2019,6 +2046,9 @@ bool SDGenerationParams::from_json_str(
         return false;
     }
 
+    if (j.contains("image_preprocess") && !parse_image_preprocess_json(j["image_preprocess"].dump()))
+        return false;
+
     auto load_if_exists = [&](const char* key, auto& out) {
         if (j.contains(key)) {
             using T = std::decay_t<decltype(out)>;
@@ -2056,6 +2086,7 @@ bool SDGenerationParams::from_json_str(
     load_if_exists("cache_mode", cache_mode);
     load_if_exists("cache_option", cache_option);
     load_if_exists("scm_mask", scm_mask);
+    load_if_exists("ref_image_args", ref_image_args);
 
     load_if_exists("clip_skip", clip_skip);
     load_if_exists("width", width);
@@ -2073,7 +2104,6 @@ bool SDGenerationParams::from_json_str(
     load_if_exists("moe_boundary", moe_boundary);
     load_if_exists("vace_strength", vace_strength);
 
-    load_if_exists("auto_resize_ref_image", auto_resize_ref_image);
     load_if_exists("increase_ref_index", increase_ref_index);
     load_if_exists("embed_image_metadata", embed_image_metadata);
 
@@ -2217,37 +2247,23 @@ bool SDGenerationParams::from_json_str(
         LOG_ERROR("invalid lora");
         return false;
     }
-    if (!parse_image_json_field(j, "init_image", 0, width, height, init_image)) {
-        LOG_ERROR("invalid init_image");
+    auto load_image = [&](const char* key, int channels, SDImageOwner& image) {
+        if (!parse_image_json_field(j, key, channels, image)) {
+            LOG_ERROR("invalid %s", key);
+            return false;
+        }
+        return true;
+    };
+    if (!load_image("init_image", 0, init_image) ||
+        !load_image("end_image", 3, end_image) ||
+        !load_image("mask_image", 1, mask_image) ||
+        !load_image("control_image", 3, control_image) ||
+        !load_image("ip_adapter_image", 3, ip_adapter_image)) {
         return false;
     }
-    if (!parse_image_json_field(j, "end_image", 3, width, height, end_image)) {
-        LOG_ERROR("invalid end_image");
-        return false;
-    }
-    if (!parse_image_array_json_field(j,
-                                      "ref_images",
-                                      0,
-                                      auto_resize_ref_image ? width : 0,
-                                      auto_resize_ref_image ? height : 0,
-                                      ref_images)) {
-        LOG_ERROR("invalid ref_images");
-        return false;
-    }
-    if (!parse_image_array_json_field(j, "control_frames", 3, width, height, control_frames)) {
-        LOG_ERROR("invalid control_frames");
-        return false;
-    }
-    if (!parse_image_json_field(j, "mask_image", 1, width, height, mask_image)) {
-        LOG_ERROR("invalid mask_image");
-        return false;
-    }
-    if (!parse_image_json_field(j, "control_image", 3, width, height, control_image)) {
-        LOG_ERROR("invalid control_image");
-        return false;
-    }
-    if (!parse_image_json_field(j, "ip_adapter_image", 3, width, height, ip_adapter_image)) {
-        LOG_ERROR("invalid ip_adapter_image");
+    if (!parse_image_array_json_field(j, "ref_images", 0, ref_images) ||
+        !parse_image_array_json_field(j, "control_frames", 3, control_frames)) {
+        LOG_ERROR("invalid input image array");
         return false;
     }
 
@@ -2491,6 +2507,10 @@ bool SDGenerationParams::resolve(const std::string& lora_model_dir, const std::s
 }
 
 bool SDGenerationParams::validate(SDMode mode) {
+    if (!image_preprocess.empty() && mode != IMG_GEN && mode != VID_GEN) {
+        LOG_ERROR("--image-preprocess requires img_gen or vid_gen mode");
+        return false;
+    }
     if (batch_count <= 0) {
         LOG_ERROR("error: batch_count must be greater than 0");
         return false;
@@ -2666,14 +2686,6 @@ sd_img_gen_params_t SDGenerationParams::to_sd_img_gen_params_t() {
         pulid_id_weight,
     };
 
-    if (!auto_resize_ref_image) {
-        if (!ref_image_args.empty()) {
-            ref_image_args += ",";
-        }
-        ref_image_args += "resize_before_vae=0";
-        LOG_WARN("Notice: --disable-auto-resize-ref-image is deprecated. Use --ref-image-args \"resize_before_vae=off\" instead.");
-    }
-
     if (increase_ref_index) {
         if (!ref_image_args.empty()) {
             ref_image_args += ",";
@@ -2721,6 +2733,7 @@ sd_img_gen_params_t SDGenerationParams::to_sd_img_gen_params_t() {
     params.hires.custom_sigmas_count = static_cast<int>(hires_custom_sigmas.size());
     params.circular_x                = circular || circular_x;
     params.circular_y                = circular || circular_y;
+    params.image_preprocess          = {image_preprocess.c_str()};
     return params;
 }
 
@@ -2823,6 +2836,7 @@ sd_vid_gen_params_t SDGenerationParams::to_sd_vid_gen_params_t() {
     params.hires.custom_sigmas_count = static_cast<int>(hires_custom_sigmas.size());
     params.circular_x                = circular || circular_x;
     params.circular_y                = circular || circular_y;
+    params.image_preprocess          = {image_preprocess.c_str()};
     return params;
 }
 
@@ -2879,7 +2893,8 @@ std::string SDGenerationParams::to_string() const {
         << "  ref_video_audio_paths: " << vec_str_to_string(ref_video_audio_paths) << ",\n"
         << "  ref_audio_paths: " << vec_str_to_string(ref_audio_paths) << ",\n"
         << "  control_video_path: \"" << control_video_path << "\",\n"
-        << "  auto_resize_ref_image: " << (auto_resize_ref_image ? "true" : "false") << ",\n"
+        << "  image_preprocess: " << image_preprocess << ",\n"
+        << "  ref_image_args: " << ref_image_args << ",\n"
         << "  increase_ref_index: " << (increase_ref_index ? "true" : "false") << ",\n"
         << "  pm_id_images_dir: \"" << pm_id_images_dir << "\",\n"
         << "  pm_id_embed_path: \"" << pm_id_embed_path << "\",\n"
@@ -3030,12 +3045,13 @@ std::string build_sdcpp_image_metadata_json(const SDContextParams& ctx_params,
     set_json_basename_if_not_empty(models, "control_net", ctx_params.control_net_path);
     root["models"] = std::move(models);
 
-    root["clip_skip"]             = gen_params.clip_skip;
-    root["strength"]              = gen_params.strength;
-    root["control_strength"]      = gen_params.control_strength;
-    root["ip_adapter_strength"]   = gen_params.ip_adapter_strength;
-    root["auto_resize_ref_image"] = gen_params.auto_resize_ref_image;
-    root["increase_ref_index"]    = gen_params.increase_ref_index;
+    root["clip_skip"]           = gen_params.clip_skip;
+    root["strength"]            = gen_params.strength;
+    root["control_strength"]    = gen_params.control_strength;
+    root["ip_adapter_strength"] = gen_params.ip_adapter_strength;
+    root["ref_image_args"]      = gen_params.ref_image_args;
+    root["image_preprocess"]    = gen_params.image_preprocess;
+    root["increase_ref_index"]  = gen_params.increase_ref_index;
     if (mode == VID_GEN) {
         root["video"] = {
             {"frame_count", gen_params.video_frames},

examples/common/common.h

@@ -116,7 +116,8 @@ bool decode_base64_image(const std::string& encoded_input,
                          SDImageOwner& out_image);
 
 struct SDContextParams {
-    int n_threads = -1;
+    int n_threads               = -1;
+    int conditioning_cache_size = 4;
     std::string model_path;
     std::string clip_l_path;
     std::string clip_g_path;
@@ -200,18 +201,17 @@ struct SDGenerationParams {
     std::string ad_prompt;
     std::string ad_negative_prompt;
     std::string extra_ad_args;
-    int clip_skip              = -1;  // <= 0 represents unspecified
-    int width                  = -1;
-    int height                 = -1;
-    int batch_count            = 1;
-    int qwen_image_layers      = 3;
-    int64_t seed               = 42;
-    float strength             = 0.75f;
-    float control_strength     = 0.9f;
-    float ip_adapter_strength  = 1.0f;
-    bool auto_resize_ref_image = true;
-    bool increase_ref_index    = false;
-    bool embed_image_metadata  = true;
+    int clip_skip             = -1;  // <= 0 represents unspecified
+    int width                 = -1;
+    int height                = -1;
+    int batch_count           = 1;
+    int qwen_image_layers     = 3;
+    int64_t seed              = 42;
+    float strength            = 0.75f;
+    float control_strength    = 0.9f;
+    float ip_adapter_strength = 1.0f;
+    bool increase_ref_index   = false;
+    bool embed_image_metadata = true;
 
     std::string init_image_path;
     std::string end_image_path;
@@ -247,6 +247,7 @@ struct SDGenerationParams {
     std::string extra_tiling_args;
 
     std::string ref_image_args;
+    std::string image_preprocess;
 
     std::string pm_id_images_dir;
     std::string pm_id_embed_path;
@@ -310,6 +311,7 @@ struct SDGenerationParams {
     ArgOptions get_options();
     bool from_json_str(const std::string& json_str,
                        const std::function<std::string(const std::string&)>& lora_path_resolver = {});
+    bool parse_image_preprocess_json(const std::string& json_str);
     bool initialize_cache_params();
     void extract_and_remove_lora(const std::string& lora_model_dir);
     bool width_and_height_are_set() const;

examples/server/api.md

@@ -148,17 +148,17 @@ Native extension fields:
 
 - any `sdcpp API` fields embedded through `sd_cpp_extra_args` inside `prompt`
 
-Reference image sizing follows `auto_resize_ref_image`, as in the native and SDAPI APIs.
-The server default is `true`; `--disable-auto-resize-ref-image` sets it to `false`.
-When enabled, uploaded references are center-cropped and resized to the request dimensions.
-If `size` is omitted, the first decoded image establishes those dimensions.
-When disabled, each reference retains its original dimensions.
-The init image and mask still use the request dimensions, independently of this option.
+Uploaded images are decoded at their original dimensions. The first decoded
+image establishes the generation dimensions if `size` is omitted. Input
+geometry follows `image_preprocess`: references preserve their dimensions by
+default, while init and mask use the generation canvas preset.
 
-To override the server default for one request, include this in `prompt`:
+Reference encoding then follows model presets and `ref_image_args`. To skip
+input geometry for references and disable resizing before VAE encoding, include
+this in `prompt`:
 
 ```text
-edit this image <sd_cpp_extra_args>{"auto_resize_ref_image":false}</sd_cpp_extra_args>
+edit this image <sd_cpp_extra_args>{"image_preprocess":"target=ref,mode=none","ref_image_args":"resize_before_vae=false"}</sd_cpp_extra_args>
 ```
 
 Response fields:
@@ -539,7 +539,7 @@ LTX and Wan preserve causal state between temporal tiles. Hunyuan Video and TAEH
 | Field | Type |
 | --- | --- |
 | `batch_count` | `integer` |
-| `auto_resize_ref_image` | `boolean` |
+| `ref_image_args` | `string` |
 | `increase_ref_index` | `boolean` |
 | `control_strength` | `number` |
 | `ip_adapter_strength` | `number` |
@@ -666,7 +666,7 @@ Example:
   "strength": 0.75,
   "seed": -1,
   "batch_count": 1,
-  "auto_resize_ref_image": true,
+  "ref_image_args": "",
   "increase_ref_index": false,
   "control_strength": 0.9,
   "ip_adapter_strength": 1.0,
@@ -741,6 +741,17 @@ Example:
 
 ### Image Encoding Rules
 
+Native image/video requests and SDAPI accept `image_preprocess` as a rule string
+or array of rule strings. OpenAI-compatible requests can supply it in
+`sd_cpp_extra_args`. See [Image preprocessing](../../docs/image_preprocessing.md)
+for one-time input geometry, native-resolution decoding, mask alignment, and
+`canny=true` for edge detection on any supported image input.
+
+Image generation also accepts `ref_image_args` as a string (for example,
+`"resize_before_vae=false"`) in native and SDAPI requests, or through
+`sd_cpp_extra_args` in OpenAI-compatible requests. It controls downstream
+reference encoding and is independent of input geometry rules.
+
 Any image field accepts:
 
 - a raw base64 string, or
@@ -776,7 +787,8 @@ Top-level scalar fields:
 | `strength` | `number` |
 | `seed` | `integer` |
 | `batch_count` | `integer` |
-| `auto_resize_ref_image` | `boolean` |
+| `ref_image_args` | `string` |
+| `image_preprocess` | `string \| array<string>` |
 | `increase_ref_index` | `boolean` |
 | `control_strength` | `number` |
 | `ip_adapter_strength` | `number` |

examples/server/main.cpp

@@ -76,9 +76,9 @@ int main(int argc, const char** argv) {
     SDSvrParams svr_params;
     SDContextParams ctx_params;
     SDGenerationParams default_gen_params;
-    parse_args(argc, argv, svr_params, ctx_params, default_gen_params);
 
     sd_set_log_callback(sd_log_cb, (void*)&svr_params);
+    parse_args(argc, argv, svr_params, ctx_params, default_gen_params);
 
     LOG_VERBOSE("version: %s", version_string().c_str());
     LOG_VERBOSE("%s", sd_get_system_info());

examples/server/routes_openai.cpp

@@ -158,18 +158,6 @@ static bool build_openai_edit_request(const httplib::Request& req,
     request.gen_params.batch_count = n;
 
     std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
-    if (!sd_cpp_extra_args_str.empty()) {
-        const json extra_args = json::parse(sd_cpp_extra_args_str, nullptr, false);
-        if (extra_args.is_discarded()) {
-            error_message = "invalid sd_cpp_extra_args";
-            return false;
-        }
-        // Resolve resizing before decoding while keeping embedded image overrides last.
-        if (extra_args.contains("auto_resize_ref_image") && extra_args["auto_resize_ref_image"].is_boolean()) {
-            request.gen_params.auto_resize_ref_image = extra_args["auto_resize_ref_image"].get<bool>();
-        }
-    }
-
     for (auto& bytes : images_bytes) {
         int img_w            = 0;
         int img_h            = 0;
@@ -178,12 +166,7 @@ static bool build_openai_edit_request(const httplib::Request& req,
              reinterpret_cast<const char*>(bytes.data()),
              static_cast<int>(bytes.size()),
              img_w, img_h, resolved_channel,
-            request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
-                 ? request.gen_params.width
-                 : 0,
-            request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
-                 ? request.gen_params.height
-                 : 0,
+             0, 0,
              0);
         if (raw_pixels == nullptr) {
             continue;
@@ -194,23 +177,10 @@ static bool build_openai_edit_request(const httplib::Request& req,
         request.gen_params.set_width_and_height_if_unset(image_owner.get().width, image_owner.get().height);
 
         if (is_first_ref_image) {
-            int init_w = 0;
-            int init_h = 0;
-            if (request.gen_params.width_and_height_are_set()) {
-                init_w = request.gen_params.width;
-                init_h = request.gen_params.height;
-            }
-
-            int init_img_w            = 0;
-            int init_img_h            = 0;
-            int init_resolved_channel = 0;
-            uint8_t* init_pixels      = load_image_from_memory(
-                     reinterpret_cast<const char*>(bytes.data()),
-                     static_cast<int>(bytes.size()),
-                     init_img_w, init_img_h, init_resolved_channel,
-                     init_w, init_h, 0);
-            if (init_pixels != nullptr) {
-                request.gen_params.init_image.reset({(uint32_t)init_img_w, (uint32_t)init_img_h, (uint32_t)init_resolved_channel, init_pixels});
+            request.gen_params.init_image = image_owner;
+            if (request.gen_params.init_image.get().data == nullptr) {
+                error_message = "could not allocate init image";
+                return false;
             }
         }
 
@@ -218,12 +188,6 @@ static bool build_openai_edit_request(const httplib::Request& req,
     }
 
     if (!mask_bytes.empty()) {
-        int expected_width  = 0;
-        int expected_height = 0;
-        if (request.gen_params.width_and_height_are_set()) {
-            expected_width  = request.gen_params.width;
-            expected_height = request.gen_params.height;
-        }
         int mask_w       = 0;
         int mask_h       = 0;
         int mask_channel = 0;
@@ -232,7 +196,7 @@ static bool build_openai_edit_request(const httplib::Request& req,
             reinterpret_cast<const char*>(mask_bytes.data()),
             static_cast<int>(mask_bytes.size()),
             mask_w, mask_h, mask_channel,
-            expected_width, expected_height, 1);
+            0, 0, 1);
         request.gen_params.mask_image.reset({(uint32_t)mask_w, (uint32_t)mask_h, 1, mask_raw});
         const sd_image_t& mask_image = request.gen_params.mask_image.get();
         request.gen_params.set_width_and_height_if_unset(mask_image.width, mask_image.height);

examples/server/routes_sdapi.cpp

@@ -80,17 +80,6 @@ static enum sample_method_t get_sdapi_sample_method(std::string name) {
     return it != hardcoded.end() ? it->second : SAMPLE_METHOD_COUNT;
 }
 
-static void assign_solid_mask(SDImageOwner& mask_owner, int width, int height) {
-    const size_t pixel_count = static_cast<size_t>(width) * static_cast<size_t>(height);
-    uint8_t* raw_mask        = static_cast<uint8_t*>(malloc(pixel_count));
-    if (raw_mask == nullptr) {
-        mask_owner.reset({0, 0, 1, nullptr});
-        return;
-    }
-    std::memset(raw_mask, 255, pixel_count);
-    mask_owner.reset({(uint32_t)width, (uint32_t)height, 1, raw_mask});
-}
-
 static bool build_sdapi_img_gen_request(const json& j,
                                         ServerRuntime& runtime,
                                         bool img2img,
@@ -193,15 +182,25 @@ static bool build_sdapi_img_gen_request(const json& j,
         }
     }
 
-    if (img2img) {
-        const int expected_width  = request.gen_params.width_and_height_are_set() ? request.gen_params.width : 0;
-        const int expected_height = request.gen_params.width_and_height_are_set() ? request.gen_params.height : 0;
+    if (j.contains("ref_image_args")) {
+        if (!j["ref_image_args"].is_string()) {
+            error_message = "ref_image_args must be a string";
+            return false;
+        }
+        request.gen_params.ref_image_args = j["ref_image_args"].get<std::string>();
+    }
 
+    if (j.contains("image_preprocess") && !request.gen_params.parse_image_preprocess_json(j["image_preprocess"].dump())) {
+        error_message = "invalid image_preprocess";
+        return false;
+    }
+
+    if (img2img) {
         if (j.contains("init_images") && j["init_images"].is_array() && !j["init_images"].empty()) {
             if (decode_base64_image(j["init_images"][0].get<std::string>(),
                                     0,
-                                    expected_width,
-                                    expected_height,
+                                    0,
+                                    0,
                                     request.gen_params.init_image)) {
                 const sd_image_t& image = request.gen_params.init_image.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);
@@ -211,8 +210,8 @@ static bool build_sdapi_img_gen_request(const json& j,
         if (j.contains("mask") && j["mask"].is_string()) {
             if (decode_base64_image(j["mask"].get<std::string>(),
                                     1,
-                                    expected_width,
-                                    expected_height,
+                                    0,
+                                    0,
                                     request.gen_params.mask_image)) {
                 const sd_image_t& image = request.gen_params.mask_image.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);
@@ -225,9 +224,7 @@ static bool build_sdapi_img_gen_request(const json& j,
                 }
             }
         } else {
-            const int resolved_width  = request.gen_params.get_resolved_width();
-            const int resolved_height = request.gen_params.get_resolved_height();
-            assign_solid_mask(request.gen_params.mask_image, resolved_width, resolved_height);
+            request.gen_params.mask_image.reset({0, 0, 1, nullptr});
         }
 
         float denoising_strength = j.value("denoising_strength", -1.f);
@@ -244,12 +241,7 @@ static bool build_sdapi_img_gen_request(const json& j,
             SDImageOwner image_owner;
             if (decode_base64_image(extra_image.get<std::string>(),
                                     0,
-                                    request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
-                                        ? request.gen_params.width
-                                        : 0,
-                                    request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
-                                        ? request.gen_params.height
-                                        : 0,
+                                    0, 0,
                                     image_owner)) {
                 const sd_image_t& image = image_owner.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);

examples/server/routes_sdcpp.cpp

@@ -127,7 +127,8 @@ static json make_img_gen_defaults_json(const SDGenerationParams& defaults, const
         {"seed", defaults.seed},
         {"batch_count", defaults.batch_count},
         {"qwen_image_layers", defaults.qwen_image_layers},
-        {"auto_resize_ref_image", defaults.auto_resize_ref_image},
+        {"ref_image_args", defaults.ref_image_args},
+        {"image_preprocess", defaults.image_preprocess},
         {"increase_ref_index", defaults.increase_ref_index},
         {"control_strength", defaults.control_strength},
         {"ip_adapter_strength", defaults.ip_adapter_strength},
@@ -153,6 +154,7 @@ static json make_vid_gen_defaults_json(const SDGenerationParams& defaults, const
         {"strength", defaults.strength},
         {"seed", defaults.seed},
         {"video_frames", defaults.video_frames},
+        {"image_preprocess", defaults.image_preprocess},
         {"fps", defaults.fps},
         {"moe_boundary", defaults.moe_boundary},
         {"vace_strength", defaults.vace_strength},

include/stable-diffusion.h

@@ -247,6 +247,7 @@ typedef struct {
     float attn_scale;                // Override flash-attention K/V scaling; 0 keeps the model default
     const char* tokenizer;           // tokenizer.json path or main=FILE,clip-l=FILE,clip-g=FILE assignments; required for PiD and Lens
     bool sage_attn;
+    int conditioning_cache_size;  // Maximum cached conditioning entries per context; 0 disables caching (default: 4)
 } sd_ctx_params_t;
 
 typedef struct {
@@ -263,6 +264,11 @@ typedef struct {
     uint8_t* data;
 } sd_image_t;
 
+typedef struct {
+    // Semicolon-separated target=...,key=value rules. NULL preserves defaults.
+    const char* rules;
+} sd_image_preprocess_params_t;
+
 typedef struct {
     sd_image_t* frames;
     int frame_count;
@@ -410,6 +416,7 @@ typedef struct {
     int qwen_image_layers;
     bool circular_x;
     bool circular_y;
+    sd_image_preprocess_params_t image_preprocess;
 } sd_img_gen_params_t;
 
 typedef struct {
@@ -443,6 +450,7 @@ typedef struct {
     sd_hires_params_t hires;
     bool circular_x;
     bool circular_y;
+    sd_image_preprocess_params_t image_preprocess;
 } sd_vid_gen_params_t;
 
 typedef struct sd_ctx_t sd_ctx_t;

src/conditioning/conditioner.hpp

@@ -3166,6 +3166,7 @@ struct LLMEmbedder : public Conditioner {
             int64_t tag_count    = static_cast<int64_t>(tags.size());
             result.c_token_types = sd::Tensor<int32_t>({tag_count}, std::move(tags));
         }
+
         return result;
     }
 };

src/conditioning/conditioning_cache.h

@@ -0,0 +1,107 @@
+#ifndef __SD_CONDITIONING_CONDITIONING_CACHE_H__
+#define __SD_CONDITIONING_CONDITIONING_CACHE_H__
+
+#include <algorithm>
+#include <list>
+#include <tuple>
+
+#include "conditioning/conditioner.hpp"
+
+class ConditioningCache {
+    struct Entry {
+        ConditionerParams params;
+        std::vector<sd::Tensor<float>> ref_images;
+        std::vector<MiniMaxH3PresentationItem> references;
+        SDCondition condition;
+
+        Entry(const ConditionerParams& input, const SDCondition& output)
+            : params(input), condition(output) {
+            // Request-owned reference pointers must not outlive the request.
+            if (input.ref_images != nullptr) {
+                ref_images        = *input.ref_images;
+                params.ref_images = &ref_images;
+            }
+            if (input.minimax_h3_references != nullptr) {
+                references                   = *input.minimax_h3_references;
+                params.minimax_h3_references = &references;
+            }
+        }
+
+        Entry(const Entry&)            = delete;
+        Entry& operator=(const Entry&) = delete;
+    };
+
+    size_t capacity_ = 4;
+    std::list<Entry> entries_;
+
+    static bool same_images(const std::vector<sd::Tensor<float>>& a,
+                            const std::vector<sd::Tensor<float>>& b) {
+        return std::equal(a.begin(), a.end(), b.begin(), b.end(),
+                          [](const sd::Tensor<float>& x, const sd::Tensor<float>& y) {
+                              return x.shape() == y.shape() && x.values() == y.values();
+                          });
+    }
+
+    static bool same_params(const ConditionerParams& a, const ConditionerParams& b) {
+        const auto fields = [](const ConditionerParams& p) {
+            const auto& r = p.ref_image_params;
+            return std::tie(p.text, p.clip_skip, p.width, p.height, p.zero_out_masked,
+                            r.pass_to_vlm, r.pass_to_dit, r.ref_index_mode,
+                            r.force_ref_timestep_zero, r.resize_before_vae, r.vae_input_max_pixels,
+                            r.vlm_resize_mode, r.vlm_min_size, r.vlm_max_size, r.resize_vae_to_target);
+        };
+        if (fields(a) != fields(b) ||
+            (a.ref_images == nullptr) != (b.ref_images == nullptr) ||
+            (a.minimax_h3_references == nullptr) != (b.minimax_h3_references == nullptr)) {
+            return false;
+        }
+        if (a.ref_images != nullptr && !same_images(*a.ref_images, *b.ref_images)) {
+            return false;
+        }
+        if (a.minimax_h3_references != nullptr &&
+            !std::equal(a.minimax_h3_references->begin(), a.minimax_h3_references->end(),
+                        b.minimax_h3_references->begin(), b.minimax_h3_references->end(),
+                        [](const MiniMaxH3PresentationItem& x, const MiniMaxH3PresentationItem& y) {
+                            return x.kind == y.kind && x.timestamps == y.timestamps && same_images(x.frames, y.frames);
+                        })) {
+            return false;
+        }
+        return true;
+    }
+
+public:
+    void set_capacity(size_t capacity) {
+        capacity_ = capacity;
+        while (entries_.size() > capacity_) {
+            entries_.pop_back();
+        }
+    }
+
+    void clear() {
+        entries_.clear();
+    }
+
+    SDCondition get(Conditioner& conditioner, int n_threads, const ConditionerParams& params) {
+        if (capacity_ == 0) {
+            return conditioner.get_learned_condition(n_threads, params);
+        }
+        for (auto it = entries_.begin(); it != entries_.end(); ++it) {
+            if (same_params(it->params, params)) {
+                entries_.splice(entries_.begin(), entries_, it);
+                LOG_INFO("conditioning cache hit");
+                return entries_.front().condition;
+            }
+        }
+        auto condition = conditioner.get_learned_condition(n_threads, params);
+        if (!condition.empty()) {
+            if (entries_.size() == capacity_) {
+                entries_.pop_back();
+            }
+            entries_.emplace_front(params, condition);
+            LOG_VERBOSE("conditioning cache stored (%zu/%zu)", entries_.size(), capacity_);
+        }
+        return condition;
+    }
+};
+
+#endif  // __SD_CONDITIONING_CONDITIONING_CACHE_H__

src/detailer.cpp

@@ -970,6 +970,7 @@ bool adetail_image(adetailer_ctx_t* context,
         generation.pm_params           = {};
         generation.pulid_params        = {};
         generation.hires.enabled       = false;
+        generation.image_preprocess    = {};
         if (params.steps > 0) {
             generation.sample_params.sample_steps        = params.steps;
             generation.sample_params.custom_sigmas       = nullptr;

src/pipeline/diffusion_engine.cpp

@@ -29,6 +29,7 @@
 #include "stable-diffusion.h"
 
 #include "conditioning/conditioner.hpp"
+#include "conditioning/conditioning_cache.h"
 #include "core/backend_fit.h"
 #include "extensions/generation_extension.h"
 #include "model/adapter/ip_adapter.hpp"
@@ -135,6 +136,7 @@ static_assert(std::atomic<sd_cancel_mode_t>::is_always_lock_free,
 
 StableDiffusionGGML::StableDiffusionGGML()
     : rng(std::make_shared<PhiloxRNG>()),
+      conditioning_cache_(std::make_unique<ConditioningCache>()),
       denoiser(std::make_shared<CompVisDenoiser>()) {}
 
 StableDiffusionGGML::~StableDiffusionGGML() = default;
@@ -204,6 +206,8 @@ void StableDiffusionGGML::end_runners() {
 }
 
 bool StableDiffusionGGML::reset_runners(const RunnerGroups& groups) {
+    conditioning_cache_->clear();
+    conditioning_loras_.clear();
     end_runners();
     clear_lora_adapters();
     runtime_lora_models.clear();
@@ -915,6 +919,11 @@ bool StableDiffusionGGML::init(const sd_ctx_params_t* sd_ctx_params) {
             return false;
         }
     }
+    if (sd_ctx_params->conditioning_cache_size < 0) {
+        LOG_ERROR("conditioning_cache_size must be non-negative");
+        return false;
+    }
+    conditioning_cache_->set_capacity(static_cast<size_t>(sd_ctx_params->conditioning_cache_size));
     auto configuration = std::make_unique<ModelConfig>(*sd_ctx_params);
     n_threads          = sd_ctx_params->n_threads;
     tensor_executor    = std::make_unique<sd::ParallelExecutor>(n_threads > 0 ? n_threads : sd_get_num_physical_cores());
@@ -1765,8 +1774,19 @@ bool StableDiffusionGGML::apply_loras(const sd_lora_t* loras, uint32_t lora_coun
     int64_t t0 = ggml_time_ms();
     end_runners();
     clear_lora_adapters();
-    if (!model_manager->prepare_lora_sources(all_loras))
+    if (!model_manager->prepare_lora_sources(all_loras)) {
+        conditioning_cache_->clear();
         return false;
+    }
+    if (!std::equal(all_loras.begin(), all_loras.end(),
+                    conditioning_loras_.begin(), conditioning_loras_.end(),
+                    [](const ModelManager::LoraSpec& a, const ModelManager::LoraSpec& b) {
+                        return a.file_id == b.file_id && a.file_revision == b.file_revision &&
+                               a.multiplier == b.multiplier && a.is_high_noise == b.is_high_noise &&
+                               a.tensor_name_prefix_filter == b.tensor_name_prefix_filter;
+                    })) {
+        conditioning_cache_->clear();
+    }
     runtime_lora_models.erase(std::remove_if(runtime_lora_models.begin(), runtime_lora_models.end(), [&](const RuntimeLora& entry) {
                                   return std::none_of(all_loras.begin(), all_loras.end(), [&](const ModelManager::LoraSpec& spec) {
                                       return entry.matches(spec);
@@ -1776,6 +1796,7 @@ bool StableDiffusionGGML::apply_loras(const sd_lora_t* loras, uint32_t lora_coun
     const bool success = apply_lora_immediately ? apply_loras_immediately(all_loras)
                                                 : apply_loras_at_runtime(all_loras);
     if (!success) {
+        conditioning_cache_->clear();
         clear_lora_adapters();
         runtime_lora_models.clear();
         return false;
@@ -1785,9 +1806,14 @@ bool StableDiffusionGGML::apply_loras(const sd_lora_t* loras, uint32_t lora_coun
     if (!all_loras.empty()) {
         LOG_INFO("apply_loras completed, taking %.2fs", (t1 - t0) * 1.0f / 1000);
     }
+    conditioning_loras_ = std::move(all_loras);
     return true;
 }
 
+SDCondition StableDiffusionGGML::get_learned_condition(const ConditionerParams& params) {
+    return conditioning_cache_->get(*cond_stage_model, n_threads, params);
+}
+
 void StableDiffusionGGML::reset_generation_extensions() {
     for (auto& extension : generation_extensions) {
         extension->reset_runtime_condition();
@@ -1972,6 +1998,8 @@ void StableDiffusionGGML::preview_image(int step,
         int patch_sz                     = 1;
         const float(*latent_rgb_proj)[3] = nullptr;
         float* latent_rgb_bias           = nullptr;
+        const float* latent_alpha_proj   = nullptr;
+        float latent_alpha_bias          = 1.f;
 
         if (channels == 128) {
             if (sd_version_uses_flux2_vae(version)) {
@@ -1985,6 +2013,16 @@ void StableDiffusionGGML::preview_image(int step,
                 LOG_WARN("No latent to RGB projection known for this model");
                 return;
             }
+        } else if (channels == 64) {
+            if (version == VERSION_QWEN_IMAGE_2_1) {
+                latent_rgb_proj   = qwen21_latent_rgb_proj;
+                latent_rgb_bias   = qwen21_latent_rgb_bias;
+                latent_alpha_proj = qwen21_latent_alpha_proj;
+                latent_alpha_bias = qwen21_latent_alpha_bias;
+            } else {
+                LOG_WARN("No latent to RGB projection known for this model");
+                return;
+            }
         } else if (channels == 48) {
             if (sd_version_is_wan(version)) {
                 latent_rgb_proj = wan_22_latent_rgb_proj;
@@ -2035,13 +2073,14 @@ void StableDiffusionGGML::preview_image(int step,
         uint32_t img_width  = static_cast<uint32_t>(_latents.shape()[0]) * patch_sz;
         uint32_t img_height = static_cast<uint32_t>(_latents.shape()[1]) * patch_sz;
 
-        uint8_t* data = (uint8_t*)malloc(frames * img_width * img_height * 3 * sizeof(uint8_t));
+        uint32_t img_channels = latent_alpha_proj != nullptr ? 4 : 3;
+        uint8_t* data         = (uint8_t*)malloc(frames * img_width * img_height * img_channels * sizeof(uint8_t));
         GGML_ASSERT(data != nullptr);
-        preview_latent_video(data, _latents, latent_rgb_proj, latent_rgb_bias, patch_sz);
+        preview_latent_video(data, _latents, latent_rgb_proj, latent_rgb_bias, patch_sz, latent_alpha_proj, latent_alpha_bias);
         sd_image_t* images = (sd_image_t*)malloc(frames * sizeof(sd_image_t));
         GGML_ASSERT(images != nullptr);
         for (uint32_t i = 0; i < frames; i++) {
-            images[i] = {img_width, img_height, 3, data + i * img_width * img_height * 3};
+            images[i] = {img_width, img_height, img_channels, data + i * img_width * img_height * img_channels};
         }
         step_callback(step, frames, images, is_noisy, step_callback_data);
         free(data);

src/pipeline/diffusion_engine.h

@@ -26,6 +26,7 @@ class RNG;
 struct Denoiser;
 struct LoraModel;
 struct ConditionerParams;
+class ConditioningCache;
 struct SDCondition;
 struct RefImageParams;
 namespace Wav2Vec2 {
@@ -178,6 +179,8 @@ public:
     std::recursive_mutex execution_mutex;
     std::unique_ptr<ModelConfig> config_;
     RunnerState runner_state_;
+    std::unique_ptr<ConditioningCache> conditioning_cache_;
+    std::vector<ModelManager::LoraSpec> conditioning_loras_;
     bool executing_ = false;
 
     std::shared_ptr<Denoiser> denoiser;
@@ -361,6 +364,8 @@ public:
 
     bool apply_loras(const sd_lora_t* loras, uint32_t lora_count);
 
+    SDCondition get_learned_condition(const ConditionerParams& params);
+
     void reset_generation_extensions();
 
     void prepare_generation_extensions(const sd_pm_params_t& pm_params,

src/pipeline/image.cpp

@@ -10,6 +10,7 @@
 #include "model/vae/vae.hpp"
 #include "request.h"
 #include "runtime/denoiser.hpp"
+#include "runtime/image_preprocess.h"
 #include "upscaler.h"
 
 namespace sd::pipeline {
@@ -440,8 +441,7 @@ namespace sd::pipeline {
         sd->compute_ip_adapter_tokens(sd_img_gen_params->ip_adapter_image, sd_img_gen_params->ip_adapter_strength);
         int64_t prepare_start_ms         = ggml_time_ms();
         condition_params.zero_out_masked = false;
-        auto cond                        = sd->cond_stage_model->get_learned_condition(sd->n_threads,
-                                                                                       condition_params);
+        auto cond                        = sd->get_learned_condition(condition_params);
         if (cond.empty()) {
             LOG_ERROR("failed to encode prompt");
             return std::nullopt;
@@ -479,8 +479,7 @@ namespace sd::pipeline {
                     // LLaDA-Image CFG keeps the source latent but drops its SigVQ features.
                     condition_params.ref_images = nullptr;
                 }
-                uncond = sd->cond_stage_model->get_learned_condition(sd->n_threads,
-                                                                     condition_params);
+                uncond = sd->get_learned_condition(condition_params);
                 if (uncond.empty()) {
                     LOG_ERROR("failed to encode negative prompt");
                     return std::nullopt;
@@ -508,8 +507,7 @@ namespace sd::pipeline {
                 if (use_ref_latent_img_cfg) {
                     condition_params.ref_images = &empty_ref_images;
                 }
-                img_uncond = sd->cond_stage_model->get_learned_condition(sd->n_threads,
-                                                                         condition_params);
+                img_uncond = sd->get_learned_condition(condition_params);
                 if (img_uncond.empty()) {
                     LOG_ERROR("failed to encode image guidance prompt");
                     return std::nullopt;
@@ -800,6 +798,12 @@ namespace sd::pipeline {
         int64_t t0            = ggml_time_ms();
         sd->vae_tiling_params = sd_img_gen_params->vae_tiling_params;
         GenerationRequest request(sd, sd_img_gen_params);
+        sd::ImagePreprocessor preprocessing(sd_img_gen_params->image_preprocess.rules);
+        sd_img_gen_params_t processed_params = *sd_img_gen_params;
+        if (!preprocessing.prepare_inputs(processed_params, request.width, request.height))
+            return false;
+        sd_img_gen_params = &processed_params;
+        request.pm_params = processed_params.pm_params;
         LOG_INFO("generate_image %dx%d", request.width, request.height);
 
         sd->rng->manual_seed(request.seed);

src/pipeline/video.cpp

@@ -15,6 +15,7 @@
 #include "model/vae/vae.hpp"
 #include "request.h"
 #include "runtime/denoiser.hpp"
+#include "runtime/image_preprocess.h"
 
 namespace sd::pipeline {
 
@@ -1161,8 +1162,7 @@ namespace sd::pipeline {
         }
 
         int64_t prepare_start_ms = ggml_time_ms();
-        embeds.cond              = sd->cond_stage_model->get_learned_condition(sd->n_threads,
-                                                                               condition_params);
+        embeds.cond              = sd->get_learned_condition(condition_params);
         if (embeds.cond.empty()) {
             LOG_ERROR("failed to encode video prompt");
             return std::nullopt;
@@ -1187,8 +1187,7 @@ namespace sd::pipeline {
         }
         if (request.use_uncond) {
             condition_params.text = request.negative_prompt;
-            embeds.uncond         = sd->cond_stage_model->get_learned_condition(sd->n_threads,
-                                                                                condition_params);
+            embeds.uncond         = sd->get_learned_condition(condition_params);
             if (embeds.uncond.empty()) {
                 LOG_ERROR("failed to encode negative video prompt");
                 return std::nullopt;
@@ -1526,6 +1525,7 @@ namespace sd::pipeline {
         img_gen_params.qwen_image_layers = 0;
         img_gen_params.circular_x        = sd_vid_gen_params->circular_x;
         img_gen_params.circular_y        = sd_vid_gen_params->circular_y;
+        img_gen_params.image_preprocess  = sd_vid_gen_params->image_preprocess;
 
         sd->animatediff_num_frames = n_frames;
         bool ok                    = generate_image(sd, &img_gen_params, frames_out, num_frames_out);
@@ -1556,6 +1556,11 @@ namespace sd::pipeline {
         sd->vae_tiling_params = sd_vid_gen_params->vae_tiling_params;
         sd->apply_circular_axes(sd_vid_gen_params->circular_x, sd_vid_gen_params->circular_y);
         GenerationRequest request(sd, sd_vid_gen_params);
+        sd::ImagePreprocessor preprocessing(sd_vid_gen_params->image_preprocess.rules);
+        sd_vid_gen_params_t processed_params = *sd_vid_gen_params;
+        if (!preprocessing.prepare_inputs(processed_params, request.width, request.height))
+            return false;
+        sd_vid_gen_params = &processed_params;
         if (fps_out != nullptr) {
             *fps_out = request.fps;
         }

src/runtime/image_preprocess.cpp

@@ -0,0 +1,447 @@
+#include "image_preprocess.h"
+#include "core/util.h"
+
+#include <climits>
+#include <set>
+
+namespace sd {
+
+    static constexpr std::pair<const char*, ImageTarget> image_targets[] = {
+        {"init", ImageTarget::Init},
+        {"end", ImageTarget::End},
+        {"mask", ImageTarget::Mask},
+        {"control", ImageTarget::Control},
+        {"ref", ImageTarget::Ref},
+        {"ip-adapter", ImageTarget::IPAdapter},
+        {"id", ImageTarget::ID},
+        {"control-frame", ImageTarget::ControlFrame},
+    };
+
+    static constexpr std::pair<const char*, ImageResizeMode> image_resize_modes[] = {
+        {"auto", ImageResizeMode::Auto},
+        {"none", ImageResizeMode::None},
+        {"stretch", ImageResizeMode::Stretch},
+        {"crop", ImageResizeMode::Crop},
+        {"crop-resize", ImageResizeMode::CropResize},
+        {"fit-pad", ImageResizeMode::FitPad},
+    };
+
+    template <typename T, size_t N>
+    static bool parse_enum(const std::string& text, const std::pair<const char*, T> (&names)[N], T& value) {
+        for (const auto& entry : names) {
+            if (text == entry.first) {
+                value = entry.second;
+                return true;
+            }
+        }
+        return false;
+    }
+
+    template <typename T, size_t N>
+    static const char* enum_name(T value, const std::pair<const char*, T> (&names)[N]) {
+        for (const auto& entry : names) {
+            if (value == entry.second)
+                return entry.first;
+        }
+        return "unknown";
+    }
+
+    template <typename T>
+    static bool one_of(T value, std::initializer_list<T> choices) {
+        return std::find(choices.begin(), choices.end(), value) != choices.end();
+    }
+
+    static bool one_of(const std::string& value, std::initializer_list<const char*> choices) {
+        for (const char* choice : choices) {
+            if (value == choice)
+                return true;
+        }
+        return false;
+    }
+
+    static ImageResizeMode resolve_mode(const std::map<std::string, std::string>& options, ImageResizeMode default_mode) {
+        auto it              = options.find("mode");
+        ImageResizeMode mode = ImageResizeMode::Auto;
+        if (it != options.end())
+            parse_enum(it->second, image_resize_modes, mode);
+        if (mode != ImageResizeMode::Auto)
+            return mode;
+        return options.count("width") && default_mode == ImageResizeMode::None ? ImageResizeMode::Stretch : default_mode;
+    }
+
+    bool ImagePreprocessor::fail(const std::string& message) const {
+        LOG_ERROR("image preprocessing: %s", message.c_str());
+        valid_ = false;
+        return false;
+    }
+
+    ImagePreprocessor::ImagePreprocessor(const char* text) {
+        if (text == nullptr || trim(text).empty())
+            return;
+        for (const auto& part : split_string(text, ';')) {
+            ImagePreprocessRule rule;
+            std::set<std::string> keys;
+            if (trim(part).empty()) {
+                fail("empty rule");
+                return;
+            }
+            for (const auto& entry : split_string(part, ',')) {
+                size_t equal = entry.find('=');
+                if (equal == std::string::npos) {
+                    fail("expected key=value: " + entry);
+                    return;
+                }
+                std::string key   = trim(entry.substr(0, equal));
+                std::string value = trim(entry.substr(equal + 1));
+                bool ok           = !value.empty() && keys.insert(key).second;
+                int number        = 0;
+                if (key == "target") {
+                    ok &= parse_enum(value, image_targets, rule.target);
+                } else if (key == "index") {
+                    ok &= parse_strict_int(value, rule.index) && rule.index >= 0;
+                } else {
+                    if (key == "mode") {
+                        ImageResizeMode mode;
+                        ok &= parse_enum(value, image_resize_modes, mode);
+                    } else if (key == "filter") {
+                        ok &= one_of(value, {"auto", "nearest", "nearest-exact", "bilinear", "bicubic", "lanczos"});
+                    } else if (key == "antialias") {
+                        ok &= one_of(value, {"auto", "true", "false"});
+                    } else if (key == "canny") {
+                        ok &= one_of(value, {"true", "false"});
+                    } else if (key == "anchor") {
+                        ok &= one_of(value, {"center", "top", "bottom", "left", "right"});
+                    } else if (key == "width" || key == "height") {
+                        ok &= parse_strict_int(value, number) && number > 0;
+                    } else if (key == "pad_color") {
+                        ok &= value.size() == 7 || value.size() == 9;
+                        ok &= !value.empty() && value[0] == '#';
+                        for (size_t i = 1; i < value.size(); ++i)
+                            ok &= std::isxdigit(static_cast<unsigned char>(value[i])) != 0;
+                    } else {
+                        ok = false;
+                    }
+                    rule.options[key] = value;
+                }
+                if (!ok) {
+                    fail("invalid or duplicate option: " + entry);
+                    return;
+                }
+            }
+            if (!keys.count("target") || rule.options.empty() ||
+                rule.options.count("width") != rule.options.count("height") ||
+                (rule.index >= 0 && !one_of(rule.target, {ImageTarget::Ref, ImageTarget::ID, ImageTarget::ControlFrame}))) {
+                fail("invalid target, index, or incomplete dimensions: " + part);
+                return;
+            }
+            rules_.push_back(std::move(rule));
+        }
+        for (const auto& rule : rules_) {
+            const auto options = resolve_options(rule.target, std::max(0, rule.index));
+            if (options.count("antialias") && options.at("antialias") == "true" && options.count("filter") &&
+                one_of(options.at("filter"), {"nearest", "nearest-exact"})) {
+                fail("antialias requires bilinear, bicubic, or lanczos");
+                return;
+            }
+        }
+    }
+
+    std::map<std::string, std::string> ImagePreprocessor::resolve_options(ImageTarget target, int index) const {
+        std::map<std::string, std::string> options;
+        for (int specificity = 0; specificity < 2; ++specificity) {
+            for (const auto& rule : rules_) {
+                if (rule.target == target &&
+                    rule.index == (specificity == 0 ? -1 : index)) {
+                    for (const auto& entry : rule.options)
+                        options[entry.first] = entry.second;
+                }
+            }
+        }
+        return options;
+    }
+
+    bool ImagePreprocessor::validate_inputs(const sd_img_gen_params_t& params) const {
+        const std::map<ImageTarget, int> counts = {
+            {ImageTarget::Init, params.init_image.data != nullptr},
+            {ImageTarget::Mask, params.mask_image.data != nullptr},
+            {ImageTarget::Control, params.control_image.data != nullptr},
+            {ImageTarget::IPAdapter, params.ip_adapter_image.data != nullptr},
+            {ImageTarget::Ref, params.ref_images != nullptr ? params.ref_images_count : 0},
+            {ImageTarget::ID, params.pm_params.id_images != nullptr ? params.pm_params.id_images_count : 0},
+        };
+        for (const auto& rule : rules_) {
+            auto it   = counts.find(rule.target);
+            int count = it == counts.end() ? 0 : it->second;
+            if (count <= 0 || rule.index >= count) {
+                return fail(std::string("rule targets an unavailable image: ") + enum_name(rule.target, image_targets));
+            }
+        }
+        return valid_;
+    }
+
+    bool ImagePreprocessor::validate_inputs(const sd_vid_gen_params_t& params) const {
+        const std::map<ImageTarget, int> counts = {
+            {ImageTarget::Init, params.init_image.data != nullptr},
+            {ImageTarget::End, params.end_image.data != nullptr},
+            {ImageTarget::Ref, params.ref_images != nullptr ? params.ref_images_count : 0},
+            {ImageTarget::ControlFrame, params.control_frames != nullptr ? params.control_frames_size : 0},
+        };
+        for (const auto& rule : rules_) {
+            auto it   = counts.find(rule.target);
+            int count = it == counts.end() ? 0 : it->second;
+            if (count <= 0 || rule.index >= count)
+                return fail(std::string("rule targets an unavailable video input: ") + enum_name(rule.target, image_targets));
+        }
+        return valid_;
+    }
+
+    static int anchor_offset(int remaining, const std::string& anchor, bool horizontal) {
+        if (anchor == (horizontal ? "left" : "top"))
+            return 0;
+        if (anchor == (horizontal ? "right" : "bottom"))
+            return remaining;
+        return remaining / 2;
+    }
+
+    Tensor<float> ImagePreprocessor::apply_transform(const Tensor<float>& image, const std::map<std::string, std::string>& options, ImageTransform p, const std::string& label, ops::InterpolateMode default_filter) const {
+        auto value = [&](const char* key, const char* fallback) {
+            auto it = options.find(key);
+            return it == options.end() ? std::string(fallback) : it->second;
+        };
+        std::string filter        = value("filter", "auto");
+        ops::InterpolateMode mode = default_filter;
+        if (filter == "nearest")
+            mode = ops::InterpolateMode::Nearest;
+        if (filter == "nearest-exact")
+            mode = ops::InterpolateMode::NearestExact;
+        if (filter == "bilinear")
+            mode = ops::InterpolateMode::Bilinear;
+        if (filter == "bicubic")
+            mode = ops::InterpolateMode::Bicubic;
+        if (filter == "lanczos")
+            mode = ops::InterpolateMode::Lanczos;
+        bool filtered  = ops::is_2d_filter_interpolate_mode(mode);
+        bool antialias = value("antialias", "auto") == "true" ||
+                         (value("antialias", "auto") == "auto" && filtered &&
+                          (p.resize_width < p.crop_width || p.resize_height < p.crop_height));
+        if (antialias && !filtered) {
+            fail(label + ": antialias requires bilinear, bicubic, or lanczos");
+            return {};
+        }
+        auto cropped = ops::slice(ops::slice(image, 0, p.x, p.x + p.crop_width), 1, p.y, p.y + p.crop_height);
+        int channels = static_cast<int>(image.shape()[2]);
+        bool resize  = p.resize_width != p.crop_width || p.resize_height != p.crop_height;
+        if (resize && channels == 4 && filtered) {
+            for (int64_t i = 0, pixels = cropped.shape()[0] * cropped.shape()[1]; i < pixels; ++i) {
+                for (int c = 0; c < 3; ++c)
+                    cropped[i + c * pixels] *= cropped[i + 3 * pixels];
+            }
+        }
+        auto resized = ops::interpolate(cropped, {p.resize_width, p.resize_height, channels, 1}, mode, false, antialias);
+        if (resize && channels == 4 && filtered) {
+            for (int64_t i = 0, pixels = resized.shape()[0] * resized.shape()[1]; i < pixels; ++i) {
+                float alpha = std::clamp(resized[i + 3 * pixels], 0.f, 1.f);
+                for (int c = 0; c < 3; ++c)
+                    resized[i + c * pixels] = alpha > 1e-6f ? resized[i + c * pixels] / alpha : 0.f;
+            }
+        }
+        resized = ops::clamp(resized, 0.f, 1.f);
+        Tensor<float> output({p.width, p.height, channels, 1});
+        std::string color = value("pad_color", "#000000ff");
+        if (color.size() == 7)
+            color += "ff";
+        uint8_t rgba[4];
+        for (int c = 0; c < 4; ++c)
+            rgba[c] = static_cast<uint8_t>(std::strtoul(color.substr(1 + c * 2, 2).c_str(), nullptr, 16));
+        for (int c = 0; c < channels; ++c) {
+            float fill = rgba[channels == 1 ? 0 : c] / 255.f;
+            for (int y = 0; y < p.height; ++y) {
+                for (int x = 0; x < p.width; ++x) {
+                    output.index(x, y, c, 0) = x >= p.pad_x && x < p.pad_x + p.resize_width && y >= p.pad_y && y < p.pad_y + p.resize_height
+                                                   ? resized.index(x - p.pad_x, y - p.pad_y, c, 0)
+                                                   : fill;
+                }
+            }
+        }
+        LOG_INFO("preprocess %s: %dx%d crop=(%d,%d,%d,%d) resize=%dx%d pad=(%d,%d) output=%dx%d filter=%s(%d) antialias=%s",
+                 label.c_str(), p.source_width, p.source_height, p.x, p.y, p.crop_width, p.crop_height,
+                 p.resize_width, p.resize_height, p.pad_x, p.pad_y, p.width, p.height, filter.c_str(), static_cast<int>(mode), BOOL_STR(antialias));
+        return output;
+    }
+
+    Tensor<float> ImagePreprocessor::apply_geometry(const Tensor<float>& image, ImageTarget target, int index, int width, int height, ImageResizeMode default_mode, ops::InterpolateMode default_filter, ImageTransform* plan_out) const {
+        if (!valid_ || image.empty())
+            return {};
+        const std::string label = std::string(enum_name(target, image_targets)) + "[" + std::to_string(index) + "]";
+        auto options            = resolve_options(target, index);
+        if (image.dim() != 4 || image.shape()[3] != 1 || image.shape()[2] < 1 || image.shape()[2] > 4) {
+            fail(label + ": expected one image with 1 to 4 channels");
+            return {};
+        }
+        ImageTransform p;
+        p.source_width = p.crop_width = static_cast<int>(image.shape()[0]);
+        p.source_height = p.crop_height = static_cast<int>(image.shape()[1]);
+        int target_width                = width > 0 ? width : p.source_width;
+        int target_height               = height > 0 ? height : p.source_height;
+        if (options.count("width")) {
+            parse_strict_int(options.at("width"), target_width);
+            parse_strict_int(options.at("height"), target_height);
+        }
+        ImageResizeMode mode = resolve_mode(options, default_mode);
+        std::string anchor   = options.count("anchor") ? options.at("anchor") : "center";
+        p.width = p.resize_width = target_width;
+        p.height = p.resize_height = target_height;
+        if (mode == ImageResizeMode::None) {
+            if (options.count("width") && (target_width != p.source_width || target_height != p.source_height)) {
+                fail(label + ": mode=none conflicts with requested dimensions");
+                return {};
+            }
+            p.width = p.resize_width = p.source_width;
+            p.height = p.resize_height = p.source_height;
+        } else if (mode == ImageResizeMode::Crop || mode == ImageResizeMode::CropResize) {
+            if (mode == ImageResizeMode::Crop) {
+                p.crop_width  = target_width;
+                p.crop_height = target_height;
+            } else if (int64_t(p.source_width) * target_height > int64_t(p.source_height) * target_width) {
+                p.crop_width = std::max(1, static_cast<int>(int64_t(p.source_height) * target_width / target_height));
+            } else {
+                p.crop_height = std::max(1, static_cast<int>(int64_t(p.source_width) * target_height / target_width));
+            }
+            if (p.crop_width > p.source_width || p.crop_height > p.source_height) {
+                fail(label + ": crop exceeds source dimensions");
+                return {};
+            }
+            p.x = anchor_offset(p.source_width - p.crop_width, anchor, true);
+            p.y = anchor_offset(p.source_height - p.crop_height, anchor, false);
+        } else if (mode == ImageResizeMode::FitPad) {
+            double scale    = std::min(double(target_width) / p.source_width, double(target_height) / p.source_height);
+            p.resize_width  = std::max(1, std::min(target_width, static_cast<int>(std::round(p.source_width * scale))));
+            p.resize_height = std::max(1, std::min(target_height, static_cast<int>(std::round(p.source_height * scale))));
+            p.pad_x         = anchor_offset(target_width - p.resize_width, anchor, true);
+            p.pad_y         = anchor_offset(target_height - p.resize_height, anchor, false);
+        }
+        if (p.width <= 0 || p.height <= 0) {
+            fail(label + ": invalid output dimensions");
+            return {};
+        }
+        uint64_t max_pixels = std::min<uint64_t>(INT64_MAX, SIZE_MAX / sizeof(float)) / static_cast<uint64_t>(image.shape()[2]);
+        if (uint64_t(p.width) * p.height > max_pixels || uint64_t(p.resize_width) * p.resize_height > max_pixels) {
+            fail(label + ": image allocation size overflows");
+            return {};
+        }
+        if (plan_out != nullptr)
+            *plan_out = p;
+        return apply_transform(image, options, p, label, default_filter);
+    }
+
+    Tensor<float> ImagePreprocessor::preprocess_input(sd_image_t image, ImageTarget target, int index, int width, int height) {
+        if (image.data == nullptr || image.width == 0 || image.height == 0 || image.width > INT_MAX || image.height > INT_MAX || image.channel < 1 || image.channel > 4) {
+            fail(std::string(enum_name(target, image_targets)) + ": invalid input image");
+            return {};
+        }
+        auto tensor = sd_image_to_tensor(image);
+        if (target == ImageTarget::Mask && has_init_transform_) {
+            auto options = resolve_options(target, index);
+            if (image.width != init_transform_.source_width || image.height != init_transform_.source_height) {
+                fail("mask and init source dimensions must match");
+                return {};
+            }
+            bool geometry_override = options.count("width") || options.count("anchor") ||
+                                     (options.count("mode") && options.at("mode") != "auto");
+            if (geometry_override) {
+                ImageTransform p;
+                auto init_options            = resolve_options(ImageTarget::Init, 0);
+                ImageResizeMode default_mode = resolve_mode(init_options, ImageResizeMode::CropResize);
+                auto result                  = apply_geometry(tensor, target, index, init_transform_.width, init_transform_.height, default_mode, ops::InterpolateMode::NearestExact, &p);
+                if (result.empty())
+                    return {};
+                const auto& q = init_transform_;
+                if (p.x != q.x || p.y != q.y || p.crop_width != q.crop_width || p.crop_height != q.crop_height ||
+                    p.resize_width != q.resize_width || p.resize_height != q.resize_height || p.pad_x != q.pad_x || p.pad_y != q.pad_y || p.width != q.width || p.height != q.height) {
+                    fail("mask geometry conflicts with init; configure geometry on init and filter on mask");
+                    return {};
+                }
+                return result;
+            }
+            return apply_transform(tensor, options, init_transform_, "mask[0]", ops::InterpolateMode::NearestExact);
+        }
+        auto result = apply_geometry(tensor, target, index, width, height, width > 0 ? ImageResizeMode::CropResize : ImageResizeMode::None,
+                                     target == ImageTarget::Mask ? ops::InterpolateMode::NearestExact : ops::InterpolateMode::Nearest,
+                                     target == ImageTarget::Init ? &init_transform_ : nullptr);
+        if (target == ImageTarget::Init)
+            has_init_transform_ = !result.empty();
+        return result;
+    }
+
+    ImagePreprocessor::~ImagePreprocessor() {
+        for (const auto& image : owned_images_)
+            std::free(image.data);
+    }
+
+    bool ImagePreprocessor::prepare_image(sd_image_t& image, ImageTarget target, int index, int width, int height) {
+        if (image.data == nullptr)
+            return true;
+        auto options = resolve_options(target, index);
+        bool canny   = options.count("canny") && options.at("canny") == "true";
+        auto tensor  = preprocess_input(image, target, index, width, height);
+        if (tensor.empty())
+            return false;
+        auto output = tensor_to_sd_image(tensor);
+        if (output.data == nullptr)
+            return fail("could not allocate input preprocessing buffer");
+        owned_images_.push_back(output);
+        if (canny && !preprocess_canny(output, 0.08f, 0.08f, 0.8f, 1.f, false))
+            return fail("Canny preprocessing failed");
+        image = output;
+        return true;
+    }
+
+    bool ImagePreprocessor::prepare_array(sd_image_t*& images, int count, ImageTarget target, std::vector<sd_image_t>& storage, int width, int height) {
+        if (count < 0 || (count > 0 && images == nullptr))
+            return fail(std::string("invalid image array: ") + enum_name(target, image_targets));
+        if (count == 0)
+            return true;
+        storage.assign(images, images + count);
+        for (int i = 0; i < count; ++i) {
+            if (storage[i].data == nullptr)
+                return fail(std::string("empty image in array: ") + enum_name(target, image_targets));
+            if (!prepare_image(storage[i], target, i, width, height))
+                return false;
+        }
+        images = storage.data();
+        return true;
+    }
+
+    bool ImagePreprocessor::prepare_inputs(sd_img_gen_params_t& params, int width, int height) {
+        if (prepared_)
+            return fail("inputs have already been prepared");
+        prepared_ = true;
+        if (!valid_ || !validate_inputs(params))
+            return false;
+        if (!prepare_image(params.init_image, ImageTarget::Init, 0, width, height) ||
+            !prepare_image(params.mask_image, ImageTarget::Mask, 0, width, height) ||
+            !prepare_image(params.control_image, ImageTarget::Control, 0, width, height) ||
+            !prepare_image(params.ip_adapter_image, ImageTarget::IPAdapter, 0, -1, -1) ||
+            !prepare_array(params.ref_images, params.ref_images_count, ImageTarget::Ref, ref_images_) ||
+            !prepare_array(params.pm_params.id_images, params.pm_params.id_images_count, ImageTarget::ID, id_images_))
+            return false;
+        params.image_preprocess = {};
+        return true;
+    }
+
+    bool ImagePreprocessor::prepare_inputs(sd_vid_gen_params_t& params, int width, int height) {
+        if (prepared_)
+            return fail("inputs have already been prepared");
+        prepared_ = true;
+        if (!valid_ || !validate_inputs(params))
+            return false;
+        if (!prepare_image(params.init_image, ImageTarget::Init, 0, width, height) ||
+            !prepare_image(params.end_image, ImageTarget::End, 0, width, height) ||
+            !prepare_array(params.ref_images, params.ref_images_count, ImageTarget::Ref, ref_images_) ||
+            !prepare_array(params.control_frames, params.control_frames_size, ImageTarget::ControlFrame, control_frames_, width, height))
+            return false;
+        params.image_preprocess = {};
+        return true;
+    }
+
+}  // namespace sd

src/runtime/image_preprocess.h

@@ -0,0 +1,88 @@
+#ifndef __SD_RUNTIME_IMAGE_PREPROCESS_H__
+#define __SD_RUNTIME_IMAGE_PREPROCESS_H__
+
+#include <map>
+#include <string>
+#include <vector>
+
+#include "core/tensor.hpp"
+#include "stable-diffusion.h"
+
+namespace sd {
+
+    enum class ImageTarget {
+        Init,
+        End,
+        Mask,
+        Control,
+        Ref,
+        IPAdapter,
+        ID,
+        ControlFrame,
+    };
+
+    enum class ImageResizeMode {
+        Auto,
+        None,
+        Stretch,
+        Crop,
+        CropResize,
+        FitPad,
+    };
+
+    struct ImageTransform {
+        int source_width  = 0;
+        int source_height = 0;
+        int x             = 0;
+        int y             = 0;
+        int crop_width    = 0;
+        int crop_height   = 0;
+        int resize_width  = 0;
+        int resize_height = 0;
+        int width         = 0;
+        int height        = 0;
+        int pad_x         = 0;
+        int pad_y         = 0;
+    };
+
+    struct ImagePreprocessRule {
+        ImageTarget target = ImageTarget::Init;
+        int index          = -1;
+        std::map<std::string, std::string> options;
+    };
+
+    class ImagePreprocessor {
+        std::vector<ImagePreprocessRule> rules_;
+        mutable bool valid_ = true;
+        ImageTransform init_transform_;
+        bool has_init_transform_ = false;
+        bool prepared_           = false;
+        std::vector<sd_image_t> owned_images_;
+        std::vector<sd_image_t> ref_images_;
+        std::vector<sd_image_t> id_images_;
+        std::vector<sd_image_t> control_frames_;
+
+        bool fail(const std::string& message) const;
+        std::map<std::string, std::string> resolve_options(ImageTarget target, int index) const;
+        Tensor<float> apply_transform(const Tensor<float>& image, const std::map<std::string, std::string>& options, ImageTransform plan, const std::string& label, ops::InterpolateMode default_filter) const;
+
+        bool prepare_image(sd_image_t& image, ImageTarget target, int index, int width, int height);
+        bool prepare_array(sd_image_t*& images, int count, ImageTarget target, std::vector<sd_image_t>& storage, int width = -1, int height = -1);
+
+    public:
+        explicit ImagePreprocessor(const char* rules = nullptr);
+        ~ImagePreprocessor();
+        ImagePreprocessor(const ImagePreprocessor&)            = delete;
+        ImagePreprocessor& operator=(const ImagePreprocessor&) = delete;
+        bool prepare_inputs(sd_img_gen_params_t& params, int width, int height);
+        bool prepare_inputs(sd_vid_gen_params_t& params, int width, int height);
+        bool is_valid() const { return valid_; }
+        bool validate_inputs(const sd_img_gen_params_t& params) const;
+        bool validate_inputs(const sd_vid_gen_params_t& params) const;
+        Tensor<float> apply_geometry(const Tensor<float>& image, ImageTarget target, int index, int width, int height, ImageResizeMode default_mode = ImageResizeMode::Stretch, ops::InterpolateMode default_filter = ops::InterpolateMode::Nearest, ImageTransform* plan_out = nullptr) const;
+        Tensor<float> preprocess_input(sd_image_t image, ImageTarget target, int index = 0, int width = -1, int height = -1);
+    };
+
+}  // namespace sd
+
+#endif  // __SD_RUNTIME_IMAGE_PREPROCESS_H__

src/runtime/latent-preview.h

@@ -4,6 +4,86 @@
 #include "core/tensor.hpp"
 #include "ggml.h"
 
+// RGB is projected to [-1, 1]; alpha is projected directly to [0, 1].
+const float qwen21_latent_rgb_proj[64][3] = {
+    {0.00860495522f, 0.01219501462f, -0.00321337196f},
+    {0.01889233090f, 0.01246581216f, 0.01074959482f},
+    {0.1255941446f, 0.1176879344f, -0.0332212352f},
+    {0.0418238528f, 0.1043427754f, 0.0121666316f},
+    {0.02025338f, 0.01453670296f, -0.000224336044f},
+    {-0.01896720702f, -0.0206099030f, -0.0322728584f},
+    {0.00438984796f, -0.01374969766f, 0.02849196f},
+    {-0.0374495856f, -0.0286777126f, -0.0693192810f},
+    {0.01511914734f, 0.0242979386f, 0.0553878870f},
+    {-0.1138629518f, -0.020391466f, 0.001550520522f},
+    {-0.0233650696f, -0.0417292018f, -0.0362361182f},
+    {-0.0351603342f, -0.0243595924f, -0.00216261038f},
+    {0.01093355288f, -0.0373466924f, 0.00241315350f},
+    {0.01778704744f, -0.00401984678f, -0.0343259192f},
+    {0.0486059334f, 0.0253144f, 0.0672564966f},
+    {0.0309463558f, 0.0277963166f, 0.0520869622f},
+    {0.0374485008f, 0.0551753676f, 0.0225853902f},
+    {-0.0090809962f, -0.004756176f, 0.00636443612f},
+    {-0.0270455652f, -0.0384966954f, -0.00905908082f},
+    {-0.00553493756f, 0.01484553684f, -0.0211502468f},
+    {0.01319502562f, 0.00948005666f, 0.0483789212f},
+    {-0.00931847104f, -0.00276452734f, -0.01011985302f},
+    {0.0180478258f, 0.01614954356f, -0.0209424690f},
+    {-0.0214530434f, -0.00272961176f, 0.0217887476f},
+    {-0.0636772304f, -0.0208893548f, 0.0479167742f},
+    {-0.0250321236f, -0.0286715676f, 0.0530110146f},
+    {-0.01853078078f, 0.01647272818f, -0.00207747588f},
+    {0.0023101082f, 0.01228800748f, 0.01303505006f},
+    {-0.01243671408f, -0.0258638728f, -0.0379116264f},
+    {0.00598934710f, 0.00642563550f, -0.01234514304f},
+    {-0.0296733996f, -0.0234698050f, 0.00060018212f},
+    {-0.0322019498f, -0.0529200462f, -0.00344987414f},
+    {-0.00205026458f, -0.00846599446f, 0.00455971038f},
+    {-0.01082227064f, 0.0315661948f, -0.0677753362f},
+    {0.0645553474f, 0.1109666998f, 0.0674744864f},
+    {0.01036801108f, -0.00484841210f, -0.001529168474f},
+    {0.01264353566f, 0.01548126338f, -0.00966374324f},
+    {-0.0223892408f, -0.00871751526f, -0.000306421670f},
+    {0.0271322742f, 0.03496524f, -0.0089692858f},
+    {0.0512178672f, 0.0173080034f, 0.00804227746f},
+    {0.01210987192f, 0.00758025926f, -0.00281712586f},
+    {0.1897278390f, 0.1210261828f, 0.062603892f},
+    {0.0208058822f, 0.00547548182f, 0.01262955638f},
+    {0.00813332858f, 0.01015930914f, 0.01301771290f},
+    {-0.000927236014f, -0.00152540594f, -0.00599213302f},
+    {0.01663314616f, -0.00582789626f, 0.0163958132f},
+    {-0.0252546342f, -0.0604193732f, -0.1606919922f},
+    {-0.091722686f, -0.0409201224f, -0.0959576198f},
+    {0.0282963112f, -0.01387223872f, -0.01648814464f},
+    {0.0552316818f, 0.0967547788f, 0.0413586632f},
+    {0.00922849292f, 0.00451467542f, -0.0529172378f},
+    {0.0558600768f, 0.0122988308f, -0.01445942422f},
+    {0.000210660902f, -0.01295958782f, -0.01804761764f},
+    {0.0358136250f, -0.0472505970f, -0.1156405142f},
+    {-0.0506390696f, -0.0471914842f, 0.0349791468f},
+    {-0.0480143168f, 0.00628389868f, -0.0545163826f},
+    {0.0315499582f, 0.0564846606f, -0.0430850488f},
+    {-0.0362330316f, -0.01267788554f, 0.0061024772f},
+    {0.0038627542f, 0.00911055916f, -0.00758526008f},
+    {-0.0447103298f, -0.00835411408f, 0.01545872328f},
+    {-0.015006738f, 0.00270612302f, -0.00784361356f},
+    {-0.0221755048f, -0.0513344748f, -0.0475317424f},
+    {-0.01036656294f, -0.00422146068f, -0.0213499052f},
+    {0.01788952706f, 0.01191944190f, 0.0397205238f},
+};
+float qwen21_latent_rgb_bias[3] = {-0.043293118f, -0.02695978f, -0.11986706f};
+
+const float qwen21_latent_alpha_proj[64] = {
+    -0.0416241114f, -0.00678954612f, -0.0169095515f, -0.0230551401f, 0.0100882595f, 0.00655586802f, 0.0401166874f, -0.0055510216f,
+    0.0224234441f, -0.0389640963f, -0.0114492163f, -0.00721128977f, -0.0029064082f, 0.0150300547f, -0.00321615308f, -0.0498856338f,
+    -0.0215251401f, 0.0240220482f, 0.0117338008f, -0.0460420624f, 0.0387872889f, 0.0131517207f, 0.0147100836f, 0.0266985286f,
+    0.0153097324f, -0.0418119757f, 0.0421013917f, 0.0401724499f, 0.00972515915f, 0.011718495f, 0.0117622291f, 0.0136505134f,
+    -0.0350017363f, -0.0100692606f, -0.0131582529f, -0.00660639315f, 0.00253611396f, -0.0195736368f, -0.04240184f, 0.0321299262f,
+    0.0106089414f, -0.0179845306f, -0.00806212708f, 0.0135889057f, -0.0157393098f, -0.0267791344f, 0.0109068534f, 0.0283931966f,
+    -0.0435370078f, 0.00187883536f, -0.0108995378f, -0.0450757676f, -0.0699481501f, 0.0123562106f, -0.0222592249f, 0.0216155907f,
+    0.0563755424f, -0.0073379912f, 0.0160012921f, 0.0411637742f, 0.0189607258f, -0.024025029f, -0.0161487905f, -0.016913203f};
+const float qwen21_latent_alpha_bias = 0.871268134f;
+
 const float minimax_latent_rgb_proj[24][3] = {
     {0.19819857f, 0.11584999f, 0.07929777f},
     {-0.16047224f, -0.10601170f, -0.15996324f},
@@ -324,7 +404,7 @@ const float sd_latent_rgb_proj[4][3] = {
     {-0.178022f, -0.200862f, -0.678514f}};
 float sd_latent_rgb_bias[3] = {-0.017478f, -0.055834f, -0.105825f};
 
-void preview_latent_video(uint8_t* buffer, ggml_tensor* latents, const float (*latent_rgb_proj)[3], const float latent_rgb_bias[3], int patch_size) {
+void preview_latent_video(uint8_t* buffer, ggml_tensor* latents, const float (*latent_rgb_proj)[3], const float latent_rgb_bias[3], int patch_size, const float* latent_alpha_proj = nullptr, float latent_alpha_bias = 1.f) {
     size_t buffer_head = 0;
 
     uint32_t latent_width  = static_cast<uint32_t>(latents->ne[0]);
@@ -338,7 +418,8 @@ void preview_latent_video(uint8_t* buffer, ggml_tensor* latents, const float (*l
     uint32_t rgb_width  = latent_width * patch_size;
     uint32_t rgb_height = latent_height * patch_size;
 
-    uint32_t unpatched_dim = dim / (patch_size * patch_size);
+    uint32_t unpatched_dim         = dim / (patch_size * patch_size);
+    const uint32_t output_channels = latent_alpha_proj != nullptr ? 4 : 3;
 
     for (uint32_t k = 0; k < frames; k++) {
         for (uint32_t rgb_x = 0; rgb_x < rgb_width; rgb_x++) {
@@ -356,13 +437,16 @@ void preview_latent_video(uint8_t* buffer, ggml_tensor* latents, const float (*l
                 // should be incremented by 1 for each pixel
                 size_t pixel_id = k * rgb_width * rgb_height + rgb_y * rgb_width + rgb_x;
 
-                float r = 0, g = 0, b = 0;
+                float r = 0, g = 0, b = 0, a = 0;
                 if (latent_rgb_proj != nullptr) {
                     for (uint32_t d = 0; d < unpatched_dim; d++) {
                         float value = *(float*)((char*)latents->data + latent_id + (d * patch_size * patch_size + channel_offset) * latents->nb[ggml_n_dims(latents) - 1]);
                         r += value * latent_rgb_proj[d][0];
                         g += value * latent_rgb_proj[d][1];
                         b += value * latent_rgb_proj[d][2];
+                        if (latent_alpha_proj != nullptr) {
+                            a += value * latent_alpha_proj[d];
+                        }
                     }
                 } else {
                     // interpret first 3 channels as RGB
@@ -386,9 +470,13 @@ void preview_latent_video(uint8_t* buffer, ggml_tensor* latents, const float (*l
                 g = g >= 0 ? g <= 1 ? g : 1 : 0;
                 b = b >= 0 ? b <= 1 ? b : 1 : 0;
 
-                buffer[pixel_id * 3 + 0] = (uint8_t)(r * 255);
-                buffer[pixel_id * 3 + 1] = (uint8_t)(g * 255);
-                buffer[pixel_id * 3 + 2] = (uint8_t)(b * 255);
+                buffer[pixel_id * output_channels + 0] = (uint8_t)(r * 255);
+                buffer[pixel_id * output_channels + 1] = (uint8_t)(g * 255);
+                buffer[pixel_id * output_channels + 2] = (uint8_t)(b * 255);
+                if (latent_alpha_proj != nullptr) {
+                    a                                      = std::min(1.0f, std::max(0.0f, a + latent_alpha_bias));
+                    buffer[pixel_id * output_channels + 3] = (uint8_t)(a * 255);
+                }
             }
         }
     }
@@ -398,16 +486,17 @@ static inline bool preview_latent_tensor_is_video(const sd::Tensor<float>& laten
     return latents.dim() == 5;
 }
 
-void preview_latent_video(uint8_t* buffer, const sd::Tensor<float>& latents, const float (*latent_rgb_proj)[3], const float latent_rgb_bias[3], int patch_size) {
+void preview_latent_video(uint8_t* buffer, const sd::Tensor<float>& latents, const float (*latent_rgb_proj)[3], const float latent_rgb_bias[3], int patch_size, const float* latent_alpha_proj = nullptr, float latent_alpha_bias = 1.f) {
     uint32_t latent_width  = static_cast<uint32_t>(latents.shape()[0]);
     uint32_t latent_height = static_cast<uint32_t>(latents.shape()[1]);
     bool is_video          = preview_latent_tensor_is_video(latents);
     uint32_t frames        = is_video ? static_cast<uint32_t>(latents.shape()[2]) : 1;
     uint32_t dim           = is_video ? static_cast<uint32_t>(latents.shape()[3]) : static_cast<uint32_t>(latents.shape()[2]);
 
-    uint32_t rgb_width     = latent_width * patch_size;
-    uint32_t rgb_height    = latent_height * patch_size;
-    uint32_t unpatched_dim = dim / (patch_size * patch_size);
+    uint32_t rgb_width             = latent_width * patch_size;
+    uint32_t rgb_height            = latent_height * patch_size;
+    uint32_t unpatched_dim         = dim / (patch_size * patch_size);
+    const uint32_t output_channels = latent_alpha_proj != nullptr ? 4 : 3;
 
     for (uint32_t k = 0; k < frames; k++) {
         for (uint32_t rgb_x = 0; rgb_x < rgb_width; rgb_x++) {
@@ -427,7 +516,7 @@ void preview_latent_video(uint8_t* buffer, const sd::Tensor<float>& latents, con
                                : latents.values()[latent_x + latent_width * (latent_y + latent_height * latent_channel)];
                 };
 
-                float r = 0.f, g = 0.f, b = 0.f;
+                float r = 0.f, g = 0.f, b = 0.f, a = 0.f;
                 if (latent_rgb_proj != nullptr) {
                     for (uint32_t d = 0; d < unpatched_dim; d++) {
                         uint32_t latent_channel = d * patch_size * patch_size + channel_offset;
@@ -435,6 +524,9 @@ void preview_latent_video(uint8_t* buffer, const sd::Tensor<float>& latents, con
                         r += value * latent_rgb_proj[d][0];
                         g += value * latent_rgb_proj[d][1];
                         b += value * latent_rgb_proj[d][2];
+                        if (latent_alpha_proj != nullptr) {
+                            a += value * latent_alpha_proj[d];
+                        }
                     }
                 } else {
                     r = latent_value(0);
@@ -450,9 +542,13 @@ void preview_latent_video(uint8_t* buffer, const sd::Tensor<float>& latents, con
                 g = std::min(1.0f, std::max(0.0f, g * .5f + .5f));
                 b = std::min(1.0f, std::max(0.0f, b * .5f + .5f));
 
-                buffer[pixel_id * 3 + 0] = (uint8_t)(r * 255);
-                buffer[pixel_id * 3 + 1] = (uint8_t)(g * 255);
-                buffer[pixel_id * 3 + 2] = (uint8_t)(b * 255);
+                buffer[pixel_id * output_channels + 0] = (uint8_t)(r * 255);
+                buffer[pixel_id * output_channels + 1] = (uint8_t)(g * 255);
+                buffer[pixel_id * output_channels + 2] = (uint8_t)(b * 255);
+                if (latent_alpha_proj != nullptr) {
+                    a                                      = std::min(1.0f, std::max(0.0f, a + latent_alpha_bias));
+                    buffer[pixel_id * output_channels + 3] = (uint8_t)(a * 255);
+                }
             }
         }
     }

src/runtime/preprocessing.hpp

@@ -165,16 +165,18 @@ static inline sd::Tensor<float> convolve_tensor(const sd::Tensor<float>& input,
     return output;
 }
 
-static inline sd::Tensor<float> grayscale_tensor(const sd::Tensor<float>& rgb_img) {
-    GGML_ASSERT(rgb_img.dim() == 4);
-    GGML_ASSERT(rgb_img.shape()[2] >= 3);
-    sd::Tensor<float> grayscale({rgb_img.shape()[0], rgb_img.shape()[1], 1, rgb_img.shape()[3]});
-    for (int64_t iy = 0; iy < rgb_img.shape()[1]; ++iy) {
-        for (int64_t ix = 0; ix < rgb_img.shape()[0]; ++ix) {
-            float r    = preprocessing_get_4d(rgb_img, ix, iy, 0, 0);
-            float g    = preprocessing_get_4d(rgb_img, ix, iy, 1, 0);
-            float b    = preprocessing_get_4d(rgb_img, ix, iy, 2, 0);
-            float gray = 0.2989f * r + 0.5870f * g + 0.1140f * b;
+static inline sd::Tensor<float> grayscale_tensor(const sd::Tensor<float>& image) {
+    GGML_ASSERT(image.dim() == 4);
+    GGML_ASSERT(image.shape()[2] >= 1);
+    sd::Tensor<float> grayscale({image.shape()[0], image.shape()[1], 1, image.shape()[3]});
+    for (int64_t iy = 0; iy < image.shape()[1]; ++iy) {
+        for (int64_t ix = 0; ix < image.shape()[0]; ++ix) {
+            float gray = preprocessing_get_4d(image, ix, iy, 0, 0);
+            if (image.shape()[2] >= 3) {
+                float g = preprocessing_get_4d(image, ix, iy, 1, 0);
+                float b = preprocessing_get_4d(image, ix, iy, 2, 0);
+                gray    = 0.2989f * gray + 0.5870f * g + 0.1140f * b;
+            }
             preprocessing_set_4d(grayscale, gray, ix, iy, 0, 0);
         }
     }
@@ -317,11 +319,12 @@ bool preprocess_canny(sd_image_t img, float high_threshold, float low_threshold,
     image_gray              = non_max_supression(G, theta);
     threshold_hystersis(&image_gray, high_threshold, low_threshold, weak, strong);
 
+    const uint32_t color_channels = img.channel == 2 || img.channel == 4 ? img.channel - 1 : img.channel;
     for (uint32_t iy = 0; iy < img.height; ++iy) {
         for (uint32_t ix = 0; ix < img.width; ++ix) {
             float gray = preprocessing_get_4d(image_gray, ix, iy, 0, 0);
             gray       = inverse ? 1.0f - gray : gray;
-            for (uint32_t c = 0; c < img.channel; ++c) {
+            for (uint32_t c = 0; c < color_channels; ++c) {
                 preprocessing_set_4d(image, gray, ix, iy, c, 0);
             }
         }

src/stable-diffusion.cpp

@@ -326,6 +326,7 @@ void sd_hires_params_init(sd_hires_params_t* hires_params) {
 void sd_ctx_params_init(sd_ctx_params_t* sd_ctx_params) {
     *sd_ctx_params                           = {};
     sd_ctx_params->n_threads                 = sd_get_num_physical_cores();
+    sd_ctx_params->conditioning_cache_size   = 4;
     sd_ctx_params->wtype                     = SD_TYPE_COUNT;
     sd_ctx_params->rng_type                  = CUDA_RNG;
     sd_ctx_params->sampler_rng_type          = RNG_TYPE_COUNT;
@@ -378,6 +379,7 @@ char* sd_ctx_params_to_str(const sd_ctx_params_t* sd_ctx_params) {
              "pulid_weights_path: %s\n"
              "tensor_type_rules: %s\n"
              "n_threads: %d\n"
+             "conditioning_cache_size: %d\n"
              "wtype: %s\n"
              "rng_type: %s\n"
              "sampler_rng_type: %s\n"
@@ -418,6 +420,7 @@ char* sd_ctx_params_to_str(const sd_ctx_params_t* sd_ctx_params) {
              SAFE_STR(sd_ctx_params->pulid_weights_path),
              SAFE_STR(sd_ctx_params->tensor_type_rules),
              sd_ctx_params->n_threads,
+             sd_ctx_params->conditioning_cache_size,
              sd_type_name(sd_ctx_params->wtype),
              sd_rng_type_name(sd_ctx_params->rng_type),
              sd_rng_type_name(sd_ctx_params->sampler_rng_type),
@@ -751,28 +754,20 @@ SD_API bool generate_video(sd_ctx_t* sd_ctx,
                            int* num_frames_out,
                            sd_audio_t** audio_out,
                            int* fps_out) {
-    if (sd_ctx == nullptr || sd_ctx->sd == nullptr || sd_vid_gen_params == nullptr) {
-        if (fps_out != nullptr) {
-            *fps_out = 0;
-        }
-        return false;
-    }
-
-    if (frames_out != nullptr) {
+    if (frames_out != nullptr)
         *frames_out = nullptr;
-    }
-    if (audio_out != nullptr) {
+    if (audio_out != nullptr)
         *audio_out = nullptr;
-    }
-    if (num_frames_out != nullptr) {
+    if (num_frames_out != nullptr)
         *num_frames_out = 0;
+    if (fps_out != nullptr)
+        *fps_out = 0;
+    if (sd_ctx == nullptr || sd_ctx->sd == nullptr || sd_vid_gen_params == nullptr) {
+        return false;
     }
 
     StableDiffusionGGML::ExecutionScope execution(*sd_ctx->sd);
     if (!execution.ready) {
-        if (fps_out != nullptr) {
-            *fps_out = 0;
-        }
         return false;
     }
 

src/upscaler.cpp

@@ -111,10 +111,22 @@ bool UpscalerGGML::load_from_file(const std::string& esrgan_path,
 
 sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_tensor) {
     sd::ParallelScope tensor_scope(&tensor_executor);
+    if (input_tensor.empty() || input_tensor.dim() != 4 ||
+        (input_tensor.shape()[2] != 3 && input_tensor.shape()[2] != 4)) {
+        LOG_ERROR("esrgan expects a 4D RGB or RGBA image tensor");
+        return {};
+    }
+
+    const bool has_alpha = input_tensor.shape()[2] == 4;
+    sd::Tensor<float> rgb;
+    if (has_alpha) {
+        rgb = sd::ops::slice(input_tensor, 2, 0, 3);
+    }
+    const sd::Tensor<float>& model_input = has_alpha ? rgb : input_tensor;
     sd::Tensor<float> upscaled;
     const int scale = esrgan_upscaler->config.scale;
     if (tile_size <= 0 || (input_tensor.shape()[0] <= tile_size && input_tensor.shape()[1] <= tile_size)) {
-        upscaled = esrgan_upscaler->compute(n_threads, input_tensor);
+        upscaled = esrgan_upscaler->compute(n_threads, model_input);
     } else {
         auto on_processing = [&](const sd::Tensor<float>& input_tile) -> sd::Tensor<float> {
             auto output_tile = esrgan_upscaler->compute(n_threads, input_tile);
@@ -125,7 +137,7 @@ sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_te
             return output_tile;
         };
 
-        upscaled = process_tiles_2d(input_tensor,
+        upscaled = process_tiles_2d(model_input,
                                     static_cast<int>(input_tensor.shape()[0] * scale),
                                     static_cast<int>(input_tensor.shape()[1] * scale),
                                     scale,
@@ -141,6 +153,14 @@ sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_te
         LOG_ERROR("esrgan compute failed");
         return {};
     }
+    if (has_alpha) {
+        auto alpha       = sd::ops::slice(input_tensor, 2, 3, 4);
+        auto alpha_shape = alpha.shape();
+        alpha_shape[0]   = upscaled.shape()[0];
+        alpha_shape[1]   = upscaled.shape()[1];
+        alpha            = sd::ops::interpolate(alpha, alpha_shape, sd::ops::InterpolateMode::Bilinear);
+        upscaled         = sd::ops::concat(upscaled, alpha, 2);
+    }
     return upscaled;
 }