fix: preserve alpha when upscaling RGBA images with ESRGAN (#2029)

docs/edit.md

@@ -26,6 +26,9 @@ Stable-diffusion.spp also supports basic Unet-based editing models like instruct
 
 ## Configuring Reference Modes (`--ref-image-args`)
 
+For a one-time input transform before reference presets and model processing,
+including cropping, padding, and resizing algorithms, see [Image preprocessing](./image_preprocessing.md).
+
 Different DiT-based editing models require different configurations to process reference images correctly (e.g., whether to use a Vision Language Model (VLM) encoder or pass VAE-encoded images directly to the DiT).
 
 To simplify this, we provide **Presets**. By default, the system automatically selects the best preset based on the model architecture. However, you can override this using the `--ref-image-args` argument.

docs/esrgan.md

@@ -2,6 +2,8 @@
 
 You can use ESRGAN—such as the model [RealESRGAN_x4plus_anime_6B.pth](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth)—to upscale the generated images and improve their overall resolution and clarity.
 
+RGBA images, including Qwen Image 2.1 output, keep their alpha channel during model upscaling and hires fix. ESRGAN processes the RGB channels; the alpha channel is resized with bilinear interpolation and recombined with the upscaled image.
+
 - Specify the model path using the `--upscale-model PATH` parameter. example:
 
 ```bash

docs/image_preprocessing.md

@@ -0,0 +1,173 @@
+# Image preprocessing
+
+Use `--image-preprocess` to transform each image input once, before generation:
+
+```sh
+sd-cli ... \
+  --image-preprocess "target=init,mode=crop-resize,filter=lanczos,antialias=true" \
+  --image-preprocess "target=mask,filter=nearest-exact" \
+  --image-preprocess "target=ref,index=0,mode=fit-pad,width=768,height=768,filter=bicubic"
+```
+
+CLI and server image loaders decode at the original resolution. The generation
+entry point merges input defaults with user rules and prepares one transformed
+image per input. The original pipeline then consumes those images, including
+its mandatory canvas adaptation, reference resizing, and encoder preprocessing.
+
+```text
+native-resolution image
+  -> input defaults + user overrides
+  -> one input transform
+  -> original generation pipeline and model-specific processing
+```
+
+These rules do not override internal VAE, CLIP/VLM, ControlNet, or pixel-patch preprocessing.
+`--ref-image-args` retains its existing meaning and runs after this input transform.
+
+## Inputs and defaults
+
+| `target` | Input | Default geometry | Indexed? |
+| --- | --- | --- | --- |
+| `init` | img2img image or video first frame | Center crop to the generation aspect ratio, then resize | No |
+| `end` | Video last frame | Center crop, then resize | No |
+| `mask` | Inpainting mask | Inherit init geometry; otherwise center crop, then resize | No |
+| `control` | Control image | Center crop, then resize | No |
+| `ref` | Reference images | Preserve source dimensions | Yes |
+| `ip-adapter` | IP-Adapter image | Preserve source dimensions | No |
+| `id` | PhotoMaker identity images | Preserve source dimensions | Yes |
+| `control-frame` | Control video frames | Center crop, then resize | Yes |
+
+Canvas defaults use the aligned generation dimensions. Reference, IP-Adapter,
+and identity inputs use their original dimensions unless overridden. Default
+resampling is nearest for images and nearest-exact for masks.
+
+These defaults are shared by CLI, server, and C API. Moving geometry out of
+the loaders replaces the previous CLI/server BOX/sRGB resizing, so default
+pixels are not guaranteed to match earlier builds.
+
+Reference video and audio preprocessing are outside these image rules.
+Preprocessing options apply to `img_gen` and `vid_gen`, not standalone upscale
+or ADetailer mode. ADetailer clears the user's rules for its internal crops.
+
+## Rules
+
+Rules are comma-separated `key=value` lists. Repeat the CLI option or separate
+rules with semicolons. Every rule requires a `target` and at least one option.
+Rule syntax and input compatibility are checked when image/video generation
+starts. Unknown keys, invalid values, duplicate keys in a rule, missing images,
+and out-of-range indices cause generation to fail with an error log.
+
+Omit `index` to configure every image of that type; otherwise use a zero-based
+index. CLI directory inputs follow filename order. Indexed rules override
+type-wide rules field by field, regardless of order. At equal specificity,
+the last value for a field wins. `auto` selects the input preset.
+
+| `mode` | Input transform |
+| --- | --- |
+| `auto` | Use the input's default geometry |
+| `none` | Keep source dimensions without resizing, cropping, or padding |
+| `stretch` | Resize to the target dimensions |
+| `crop` | Crop a target-sized rectangle without resizing; fail if the source is too small |
+| `crop-resize` | Crop to the target aspect ratio, then resize |
+| `fit-pad` | Fit the entire image inside the target dimensions, preserving aspect ratio, then pad |
+
+`width` and `height` must be specified together as positive integers. They
+override the input transform's dimensions, not the generation or encoder size.
+For a native-size preset, specifying dimensions without a mode selects stretch.
+`mode=none` with explicit dimensions different from the source is contradictory
+and is rejected.
+
+`anchor=center|top|bottom|left|right` selects crop/padding placement.
+`pad_color=#RRGGBB` or `#RRGGBBAA` selects padding, defaulting to opaque black.
+A grayscale mask uses the first color component.
+
+`filter=auto|nearest|nearest-exact|bilinear|bicubic|lanczos` selects resampling.
+`antialias=auto|true|false` enables antialiasing automatically for filtered
+downscaling; explicit true requires bilinear, bicubic, or Lanczos.
+Filtered RGBA resizing uses premultiplied alpha.
+
+`canny=true|false` enables edge detection for any supported image target,
+defaulting to `false`. It runs once after geometry, before the original
+generation pipeline, including with `mode=none`. Grayscale, grayscale-alpha,
+RGB, and RGBA inputs are supported; alpha is preserved.
+
+Each input has its own Canny setting. Indexed rules can enable or disable it
+for individual references, identity images, or video control frames.
+
+```sh
+--image-preprocess "target=init,mode=fit-pad,canny=true"
+--image-preprocess "target=ref,index=0,mode=none,canny=true"
+--image-preprocess "target=control-frame,index=2,canny=true"
+```
+
+Init and mask sources must have the same dimensions. The mask inherits the
+init crop, resize, and padding coordinates, while retaining its own filter,
+padding value, and Canny setting. Conflicting mask geometry is rejected. An
+omitted mask remains absent until the original pipeline creates its default mask.
+
+## Downstream behavior
+
+`mode=none` only skips the input geometry transform. For example:
+
+```sh
+--image-preprocess "target=init,mode=none" \
+--image-preprocess "target=ref,mode=none"
+```
+
+The init image is still adapted to the generation canvas by the original
+pipeline. Reference images still follow `--ref-image-args` and model-specific
+resizing. CLIP retains its fixed input dimensions and normalization. HiDream-O1
+retains its original pixel-reference and visual preprocessing.
+
+Existing sharing between consumers is preserved: for example, Wan img2video
+uses the same adapted first frame for VAE conditioning and CLIP. High-resolution
+passes reuse the prepared images and apply their original size adaptation;
+they do not apply the user's crop a second time.
+
+To disable reference resizing before VAE encoding, use
+`--ref-image-args "resize_before_vae=false"` or the server field
+`"ref_image_args": "resize_before_vae=false"`. This is separate from
+`target=ref,mode=none`, which only skips input geometry. Model constraints
+still apply.
+
+## Server requests
+
+Native image/video requests and SDAPI accept `image_preprocess` as a string or
+an array of rule strings:
+
+```json
+{
+  "image_preprocess": [
+    "target=init,mode=fit-pad,filter=bicubic",
+    "target=mask,filter=nearest-exact",
+    "target=ref,index=0,mode=none"
+  ]
+}
+```
+
+OpenAI-compatible requests accept it through
+`<sd_cpp_extra_args>{...}</sd_cpp_extra_args>` in the prompt.
+Request rules replace server-default rules. Generation metadata records the
+user rules; image encodings and channel conventions are unchanged.
+
+## C API
+
+Set `image_preprocess` on the existing image/video generation parameters.
+The `generate_image()` and `generate_video()` signatures are unchanged:
+
+```c
+sd_img_gen_params_t params;
+sd_img_gen_params_init(&params);
+/* Set prompt, original-resolution input images, and generation options. */
+params.image_preprocess.rules = "target=init,mode=crop-resize,filter=lanczos;"
+                                "target=mask,filter=nearest-exact";
+bool ok = generate_image(ctx, &params, &images, &count);
+```
+
+Both generation parameter initializers set `image_preprocess.rules` to `NULL`,
+selecting input presets. Rule strings are borrowed for the synchronous call.
+The library owns temporary transformed pixels; caller images and arrays are
+not modified. Add `canny=true` to the desired target's rule in
+`image_preprocess.rules` to enable Canny.
+
+The parameter structs have grown; applications and bindings must be rebuilt.

docs/qwen_image_2.1.md

@@ -39,3 +39,21 @@ Pass the reference image with `-r` and describe the edit in `-p`. Vision weights
 ```
 
 For multiple reference images, repeat `-r` in the desired order, for example `-r first.png -r second.png`.
+
+### Alpha channel
+
+This model supports alpha channel output. As the model determines whether to output a regular image or with transparency through the prompt, according to [official recommendation](https://github.com/QwenLM/Qwen-Image-2.1#transparent-image-generation-rgba), use the following prompt format for better results:
+
+> `This is an RGBA image with transparency. <your description>. The image has alpha channel and the background is transparent.`
+
+Since transparency is decided by the prompt rather than by the input or an explicit switch, the same format applies equally to editing, whether or not the reference image itself has an alpha channel. Note that alpha is kept only in `.png` and `.webp` outputs; saving as `.jpg` drops the transparency.
+
+Here are some examples ran with Q6_K quantization:
+| Input | Prompt | Output |
+| --- | --- | --- |
+| ![Qwen Image 2.1 alpha input example 1](../assets/qwen/qwen-image-2.1-alpha-in1.png) | This is an RGBA image with transparency. Replace the text "BLOOM" with "Qwen Image 2.1", keeping the same font of the original text. The image has alpha channel and the background is transparent. | ![Qwen Image 2.1 alpha output example 1](../assets/qwen/qwen-image-2.1-alpha-out1.png) |
+| ![Qwen Image 2.1 alpha input example 2](../assets/logo.png) | This is an RGBA image with transparency. Remove the background of the image, keeping only the text and cat. The image has alpha channel and the background is transparent. | ![Qwen Image 2.1 alpha output example 2](../assets/qwen/qwen-image-2.1-alpha-out2.png) |
+
+### Other features
+
+Other features of the model could be found on the [model card from QwenLM/Qwen-Image-2.1 repo](https://github.com/QwenLM/Qwen-Image-2.1), including 2 finetuned prompt rewriting Qwen3.5-9B model.

examples/cli/README.md

@@ -14,6 +14,12 @@ equivalent to `--log-level verbose`. If repeated, the last logging option wins.
 For direct image repair or automatic post-generation YOLOv8 detection followed by cropped inpainting, see
 [ADetailer](../../docs/adetailer.md).
 
+Use repeatable `--image-preprocess` rules to select resizing, cropping, padding,
+and resampling separately for each image input. Add `canny=true` to any input
+rule for edge detection. See
+[Image preprocessing](../../docs/image_preprocessing.md) for input selectors,
+input defaults, downstream model processing, mask alignment, and examples.
+
 Metadata mode inspects PNG/JPEG container metadata without loading any model:
 
 ```bash

examples/cli/main.cpp

@@ -41,7 +41,6 @@ struct SDCliParams {
     std::string metadata_format = "text";
 
     sd_log_level_t log_level = SD_LOG_INFO;
-    bool canny_preprocess    = false;
     bool convert_name        = false;
 
     preview_t preview_method = PREVIEW_NONE;
@@ -107,10 +106,6 @@ struct SDCliParams {
         };
 
         options.bool_options = {
-            {"",
-             "--canny",
-             "apply canny preprocessor (edge detection)",
-             true, &canny_preprocess},
             {"",
              "--convert-name",
              "convert tensor name (for convert mode)",
@@ -268,7 +263,6 @@ struct SDCliParams {
             << "  metadata_format: \"" << metadata_format << "\",\n"
             << "  log_level: " << log_level_name(log_level) << ",\n"
             << "  color: " << (color ? "true" : "false") << ",\n"
-            << "  canny_preprocess: " << (canny_preprocess ? "true" : "false") << ",\n"
             << "  convert_name: " << (convert_name ? "true" : "false") << ",\n"
             << "  preview_method: " << previews_str[preview_method] << ",\n"
             << "  preview_interval: " << preview_interval << ",\n"
@@ -328,9 +322,7 @@ void sd_log_cb(enum sd_log_level_t level, const char* log, void* data) {
 
 bool load_images_from_dir(const std::string dir,
                           std::vector<SDImageOwner>& images,
-                          int expected_width  = 0,
-                          int expected_height = 0,
-                          int max_image_num   = 0) {
+                          int max_image_num = 0) {
     if (!fs::exists(dir) || !fs::is_directory(dir)) {
         LOG_ERROR("'%s' is not a valid directory\n", dir.c_str());
         return false;
@@ -357,7 +349,8 @@ bool load_images_from_dir(const std::string dir,
             LOG_VERBOSE("load image %zu from '%s'", images.size(), path.c_str());
             int width             = 0;
             int height            = 0;
-            uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, expected_width, expected_height);
+            int loaded_channel    = 0;
+            uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, loaded_channel, 0, 0);
             if (image_buffer == nullptr) {
                 LOG_ERROR("load image from '%s' failed", path.c_str());
                 return false;
@@ -365,7 +358,7 @@ bool load_images_from_dir(const std::string dir,
 
             images.emplace_back(sd_image_t{(uint32_t)width,
                                            (uint32_t)height,
-                                           3,
+                                           (uint32_t)loaded_channel,
                                            image_buffer});
 
             if (max_image_num > 0 && static_cast<int>(images.size()) >= max_image_num) {
@@ -653,8 +646,8 @@ int main(int argc, const char* argv[]) {
     SDContextParams ctx_params;
     SDGenerationParams gen_params;
 
-    parse_args(argc, argv, cli_params, ctx_params, gen_params);
     sd_set_log_callback(sd_log_cb, (void*)&cli_params);
+    parse_args(argc, argv, cli_params, ctx_params, gen_params);
 
     if (cli_params.mode == METADATA) {
         MetadataReadOptions options;
@@ -750,16 +743,8 @@ int main(int argc, const char* argv[]) {
 
     auto load_image_and_update_size = [&](const std::string& path,
                                           SDImageOwner& image,
-                                          bool resize_image    = true,
                                           int expected_channel = 3) -> bool {
-        int expected_width  = 0;
-        int expected_height = 0;
-        if (resize_image && gen_params.width_and_height_are_set()) {
-            expected_width  = gen_params.width;
-            expected_height = gen_params.height;
-        }
-
-        if (!load_sd_image_from_file(image.put(), path.c_str(), expected_width, expected_height, expected_channel)) {
+        if (!load_sd_image_from_file(image.put(), path.c_str(), 0, 0, expected_channel)) {
             LOG_ERROR("load image from '%s' failed", path.c_str());
             return false;
         }
@@ -781,7 +766,8 @@ int main(int argc, const char* argv[]) {
     };
 
     if (gen_params.init_image_path.size() > 0) {
-        if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image)) {
+        const bool native_init = cli_params.mode == IMG_GEN || cli_params.mode == ADETAILER;
+        if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image, native_init ? 0 : 3)) {
             return 1;
         }
     }
@@ -795,8 +781,8 @@ int main(int argc, const char* argv[]) {
     if (gen_params.ref_image_paths.size() > 0) {
         gen_params.ref_images.clear();
         for (auto& path : gen_params.ref_image_paths) {
-            SDImageOwner ref_image({0, 0, 3, nullptr});
-            if (!load_image_and_update_size(path, ref_image, false)) {
+            SDImageOwner ref_image({0, 0, 0, nullptr});
+            if (!load_image_and_update_size(path, ref_image, 0)) {
                 return 1;
             }
             gen_params.ref_images.push_back(std::move(ref_image));
@@ -837,41 +823,22 @@ int main(int argc, const char* argv[]) {
     if (gen_params.mask_image_path.size() > 0) {
         if (!load_sd_image_from_file(gen_params.mask_image.put(),
                                      gen_params.mask_image_path.c_str(),
-                                     gen_params.get_resolved_width(),
-                                     gen_params.get_resolved_height(),
+                                     0,
+                                     0,
                                      1)) {
             LOG_ERROR("load image from '%s' failed", gen_params.mask_image_path.c_str());
             return 1;
         }
-    } else {
-        sd_image_t generated_mask = {0, 0, 1, nullptr};
-        generated_mask.data       = (uint8_t*)malloc(gen_params.get_resolved_width() * gen_params.get_resolved_height());
-        if (generated_mask.data == nullptr) {
-            LOG_ERROR("malloc mask image failed");
-            return 1;
-        }
-        generated_mask.width  = gen_params.get_resolved_width();
-        generated_mask.height = gen_params.get_resolved_height();
-        memset(generated_mask.data, 255, gen_params.get_resolved_width() * gen_params.get_resolved_height());
-        gen_params.mask_image.reset(generated_mask);
     }
 
     if (gen_params.control_image_path.size() > 0) {
         if (!load_sd_image_from_file(gen_params.control_image.put(),
                                      gen_params.control_image_path.c_str(),
-                                     gen_params.get_resolved_width(),
-                                     gen_params.get_resolved_height())) {
+                                     0,
+                                     0)) {
             LOG_ERROR("load image from '%s' failed", gen_params.control_image_path.c_str());
             return 1;
         }
-        if (cli_params.canny_preprocess) {  // apply preprocessor
-            preprocess_canny(gen_params.control_image.get(),
-                             0.08f,
-                             0.08f,
-                             0.8f,
-                             1.0f,
-                             false);
-        }
     }
 
     if (gen_params.ip_adapter_image_path.size() > 0) {
@@ -888,8 +855,6 @@ int main(int argc, const char* argv[]) {
         gen_params.control_frames.clear();
         if (!load_images_from_dir(gen_params.control_video_path,
                                   gen_params.control_frames,
-                                  gen_params.get_resolved_width(),
-                                  gen_params.get_resolved_height(),
                                   gen_params.video_frames)) {
             return 1;
         }
@@ -898,10 +863,7 @@ int main(int argc, const char* argv[]) {
     if (!gen_params.pm_id_images_dir.empty()) {
         gen_params.pm_id_images.clear();
         if (!load_images_from_dir(gen_params.pm_id_images_dir,
-                                  gen_params.pm_id_images,
-                                  0,
-                                  0,
-                                  0)) {
+                                  gen_params.pm_id_images)) {
             return 1;
         }
     }

examples/common/common.cpp

@@ -1128,6 +1128,9 @@ ArgOptions SDGenerationParams::get_options() {
          "Key-value list to set up the way the reference images are processed (empty = auto-detect from model weigths)",
          (int)',',
          &ref_image_args},
+        {"", "--image-preprocess",
+         "Image preprocessing rule: target=init|end|mask|control|ref|ip-adapter|id|control-frame,index=N,mode=auto|none|stretch|crop|crop-resize|fit-pad,filter=auto|nearest|nearest-exact|bilinear|bicubic|lanczos,antialias=auto|true|false,width=W,height=H,anchor=center|top|bottom|left|right,pad_color=#RRGGBB[AA],canny=true|false. Repeat for multiple rules.",
+         (int)';', &image_preprocess},
     };
 
     options.int_options = {
@@ -1308,11 +1311,6 @@ ArgOptions SDGenerationParams::get_options() {
          "automatically increase the indices of references images based on the order they are listed (starting with 1).",
          true,
          &increase_ref_index},
-        {"",
-         "--disable-auto-resize-ref-image",
-         "disable auto resize of ref images",
-         false,
-         &auto_resize_ref_image},
         {"",
          "--circular",
          "enable circular padding on both axes for tileable output",
@@ -1848,28 +1846,28 @@ bool decode_base64_image(const std::string& encoded_input,
         return false;
     }
 
-    int decoded_width  = 0;
-    int decoded_height = 0;
-    uint8_t* raw_data  = load_image_from_memory(reinterpret_cast<const char*>(image_bytes.data()),
-                                                static_cast<int>(image_bytes.size()),
-                                                decoded_width,
-                                                decoded_height,
-                                                expected_width,
-                                                expected_height,
-                                                target_channels);
+    int decoded_width    = 0;
+    int decoded_height   = 0;
+    int resolved_channel = target_channels;
+    uint8_t* raw_data    = load_image_from_memory(reinterpret_cast<const char*>(image_bytes.data()),
+                                                  static_cast<int>(image_bytes.size()),
+                                                  decoded_width,
+                                                  decoded_height,
+                                                  resolved_channel,
+                                                  expected_width,
+                                                  expected_height,
+                                                  target_channels);
     if (raw_data == nullptr) {
         return false;
     }
 
-    out_image.reset({(uint32_t)decoded_width, (uint32_t)decoded_height, (uint32_t)target_channels, raw_data});
+    out_image.reset({(uint32_t)decoded_width, (uint32_t)decoded_height, (uint32_t)resolved_channel, raw_data});
     return true;
 }
 
 static bool parse_image_json_field(const json& parent,
                                    const char* key,
                                    int channels,
-                                   int expected_width,
-                                   int expected_height,
                                    SDImageOwner& out_image) {
     if (!parent.contains(key)) {
         return true;
@@ -1881,14 +1879,12 @@ static bool parse_image_json_field(const json& parent,
     if (!parent.at(key).is_string()) {
         return false;
     }
-    return decode_base64_image(parent.at(key).get<std::string>(), channels, expected_width, expected_height, out_image);
+    return decode_base64_image(parent.at(key).get<std::string>(), channels, 0, 0, out_image);
 }
 
 static bool parse_image_array_json_field(const json& parent,
                                          const char* key,
                                          int channels,
-                                         int expected_width,
-                                         int expected_height,
                                          std::vector<SDImageOwner>& out_images) {
     if (!parent.contains(key)) {
         return true;
@@ -1907,7 +1903,7 @@ static bool parse_image_array_json_field(const json& parent,
             return false;
         }
         SDImageOwner image;
-        if (!decode_base64_image(item.get<std::string>(), channels, expected_width, expected_height, image)) {
+        if (!decode_base64_image(item.get<std::string>(), channels, 0, 0, image)) {
             return false;
         }
         out_images.push_back(std::move(image));
@@ -2006,6 +2002,29 @@ static bool resolve_model_file_from_dir(const std::string& model_name,
     return false;
 }
 
+bool SDGenerationParams::parse_image_preprocess_json(const std::string& json_str) {
+    const auto value = json::parse(json_str, nullptr, false);
+    std::string rules;
+    if (value.is_string()) {
+        rules = value.get<std::string>();
+    } else if (value.is_array()) {
+        for (const auto& item : value) {
+            if (!item.is_string()) {
+                LOG_ERROR("image_preprocess must contain rule strings");
+                return false;
+            }
+            if (!rules.empty())
+                rules += ";";
+            rules += item.get<std::string>();
+        }
+    } else {
+        LOG_ERROR("image_preprocess must be a string or array of strings");
+        return false;
+    }
+    image_preprocess = std::move(rules);
+    return true;
+}
+
 bool SDGenerationParams::from_json_str(
     const std::string& json_str,
     const std::function<std::string(const std::string&)>& lora_path_resolver) {
@@ -2017,6 +2036,9 @@ bool SDGenerationParams::from_json_str(
         return false;
     }
 
+    if (j.contains("image_preprocess") && !parse_image_preprocess_json(j["image_preprocess"].dump()))
+        return false;
+
     auto load_if_exists = [&](const char* key, auto& out) {
         if (j.contains(key)) {
             using T = std::decay_t<decltype(out)>;
@@ -2054,6 +2076,7 @@ bool SDGenerationParams::from_json_str(
     load_if_exists("cache_mode", cache_mode);
     load_if_exists("cache_option", cache_option);
     load_if_exists("scm_mask", scm_mask);
+    load_if_exists("ref_image_args", ref_image_args);
 
     load_if_exists("clip_skip", clip_skip);
     load_if_exists("width", width);
@@ -2071,7 +2094,6 @@ bool SDGenerationParams::from_json_str(
     load_if_exists("moe_boundary", moe_boundary);
     load_if_exists("vace_strength", vace_strength);
 
-    load_if_exists("auto_resize_ref_image", auto_resize_ref_image);
     load_if_exists("increase_ref_index", increase_ref_index);
     load_if_exists("embed_image_metadata", embed_image_metadata);
 
@@ -2215,37 +2237,23 @@ bool SDGenerationParams::from_json_str(
         LOG_ERROR("invalid lora");
         return false;
     }
-    if (!parse_image_json_field(j, "init_image", 3, width, height, init_image)) {
-        LOG_ERROR("invalid init_image");
+    auto load_image = [&](const char* key, int channels, SDImageOwner& image) {
+        if (!parse_image_json_field(j, key, channels, image)) {
+            LOG_ERROR("invalid %s", key);
+            return false;
+        }
+        return true;
+    };
+    if (!load_image("init_image", 0, init_image) ||
+        !load_image("end_image", 3, end_image) ||
+        !load_image("mask_image", 1, mask_image) ||
+        !load_image("control_image", 3, control_image) ||
+        !load_image("ip_adapter_image", 3, ip_adapter_image)) {
         return false;
     }
-    if (!parse_image_json_field(j, "end_image", 3, width, height, end_image)) {
-        LOG_ERROR("invalid end_image");
-        return false;
-    }
-    if (!parse_image_array_json_field(j,
-                                      "ref_images",
-                                      3,
-                                      auto_resize_ref_image ? width : 0,
-                                      auto_resize_ref_image ? height : 0,
-                                      ref_images)) {
-        LOG_ERROR("invalid ref_images");
-        return false;
-    }
-    if (!parse_image_array_json_field(j, "control_frames", 3, width, height, control_frames)) {
-        LOG_ERROR("invalid control_frames");
-        return false;
-    }
-    if (!parse_image_json_field(j, "mask_image", 1, width, height, mask_image)) {
-        LOG_ERROR("invalid mask_image");
-        return false;
-    }
-    if (!parse_image_json_field(j, "control_image", 3, width, height, control_image)) {
-        LOG_ERROR("invalid control_image");
-        return false;
-    }
-    if (!parse_image_json_field(j, "ip_adapter_image", 3, width, height, ip_adapter_image)) {
-        LOG_ERROR("invalid ip_adapter_image");
+    if (!parse_image_array_json_field(j, "ref_images", 0, ref_images) ||
+        !parse_image_array_json_field(j, "control_frames", 3, control_frames)) {
+        LOG_ERROR("invalid input image array");
         return false;
     }
 
@@ -2489,6 +2497,10 @@ bool SDGenerationParams::resolve(const std::string& lora_model_dir, const std::s
 }
 
 bool SDGenerationParams::validate(SDMode mode) {
+    if (!image_preprocess.empty() && mode != IMG_GEN && mode != VID_GEN) {
+        LOG_ERROR("--image-preprocess requires img_gen or vid_gen mode");
+        return false;
+    }
     if (batch_count <= 0) {
         LOG_ERROR("error: batch_count must be greater than 0");
         return false;
@@ -2664,14 +2676,6 @@ sd_img_gen_params_t SDGenerationParams::to_sd_img_gen_params_t() {
         pulid_id_weight,
     };
 
-    if (!auto_resize_ref_image) {
-        if (!ref_image_args.empty()) {
-            ref_image_args += ",";
-        }
-        ref_image_args += "resize_before_vae=0";
-        LOG_WARN("Notice: --disable-auto-resize-ref-image is deprecated. Use --ref-image-args \"resize_before_vae=off\" instead.");
-    }
-
     if (increase_ref_index) {
         if (!ref_image_args.empty()) {
             ref_image_args += ",";
@@ -2719,6 +2723,7 @@ sd_img_gen_params_t SDGenerationParams::to_sd_img_gen_params_t() {
     params.hires.custom_sigmas_count = static_cast<int>(hires_custom_sigmas.size());
     params.circular_x                = circular || circular_x;
     params.circular_y                = circular || circular_y;
+    params.image_preprocess          = {image_preprocess.c_str()};
     return params;
 }
 
@@ -2821,6 +2826,7 @@ sd_vid_gen_params_t SDGenerationParams::to_sd_vid_gen_params_t() {
     params.hires.custom_sigmas_count = static_cast<int>(hires_custom_sigmas.size());
     params.circular_x                = circular || circular_x;
     params.circular_y                = circular || circular_y;
+    params.image_preprocess          = {image_preprocess.c_str()};
     return params;
 }
 
@@ -2877,7 +2883,8 @@ std::string SDGenerationParams::to_string() const {
         << "  ref_video_audio_paths: " << vec_str_to_string(ref_video_audio_paths) << ",\n"
         << "  ref_audio_paths: " << vec_str_to_string(ref_audio_paths) << ",\n"
         << "  control_video_path: \"" << control_video_path << "\",\n"
-        << "  auto_resize_ref_image: " << (auto_resize_ref_image ? "true" : "false") << ",\n"
+        << "  image_preprocess: " << image_preprocess << ",\n"
+        << "  ref_image_args: " << ref_image_args << ",\n"
         << "  increase_ref_index: " << (increase_ref_index ? "true" : "false") << ",\n"
         << "  pm_id_images_dir: \"" << pm_id_images_dir << "\",\n"
         << "  pm_id_embed_path: \"" << pm_id_embed_path << "\",\n"
@@ -3028,12 +3035,13 @@ std::string build_sdcpp_image_metadata_json(const SDContextParams& ctx_params,
     set_json_basename_if_not_empty(models, "control_net", ctx_params.control_net_path);
     root["models"] = std::move(models);
 
-    root["clip_skip"]             = gen_params.clip_skip;
-    root["strength"]              = gen_params.strength;
-    root["control_strength"]      = gen_params.control_strength;
-    root["ip_adapter_strength"]   = gen_params.ip_adapter_strength;
-    root["auto_resize_ref_image"] = gen_params.auto_resize_ref_image;
-    root["increase_ref_index"]    = gen_params.increase_ref_index;
+    root["clip_skip"]           = gen_params.clip_skip;
+    root["strength"]            = gen_params.strength;
+    root["control_strength"]    = gen_params.control_strength;
+    root["ip_adapter_strength"] = gen_params.ip_adapter_strength;
+    root["ref_image_args"]      = gen_params.ref_image_args;
+    root["image_preprocess"]    = gen_params.image_preprocess;
+    root["increase_ref_index"]  = gen_params.increase_ref_index;
     if (mode == VID_GEN) {
         root["video"] = {
             {"frame_count", gen_params.video_frames},

examples/common/common.h

@@ -200,18 +200,17 @@ struct SDGenerationParams {
     std::string ad_prompt;
     std::string ad_negative_prompt;
     std::string extra_ad_args;
-    int clip_skip              = -1;  // <= 0 represents unspecified
-    int width                  = -1;
-    int height                 = -1;
-    int batch_count            = 1;
-    int qwen_image_layers      = 3;
-    int64_t seed               = 42;
-    float strength             = 0.75f;
-    float control_strength     = 0.9f;
-    float ip_adapter_strength  = 1.0f;
-    bool auto_resize_ref_image = true;
-    bool increase_ref_index    = false;
-    bool embed_image_metadata  = true;
+    int clip_skip             = -1;  // <= 0 represents unspecified
+    int width                 = -1;
+    int height                = -1;
+    int batch_count           = 1;
+    int qwen_image_layers     = 3;
+    int64_t seed              = 42;
+    float strength            = 0.75f;
+    float control_strength    = 0.9f;
+    float ip_adapter_strength = 1.0f;
+    bool increase_ref_index   = false;
+    bool embed_image_metadata = true;
 
     std::string init_image_path;
     std::string end_image_path;
@@ -247,6 +246,7 @@ struct SDGenerationParams {
     std::string extra_tiling_args;
 
     std::string ref_image_args;
+    std::string image_preprocess;
 
     std::string pm_id_images_dir;
     std::string pm_id_embed_path;
@@ -310,6 +310,7 @@ struct SDGenerationParams {
     ArgOptions get_options();
     bool from_json_str(const std::string& json_str,
                        const std::function<std::string(const std::string&)>& lora_path_resolver = {});
+    bool parse_image_preprocess_json(const std::string& json_str);
     bool initialize_cache_params();
     void extract_and_remove_lora(const std::string& lora_model_dir);
     bool width_and_height_are_set() const;

examples/common/media_io.cpp

@@ -261,6 +261,10 @@ uint8_t* decode_webp_image_to_buffer(const uint8_t* data,
     height               = features.height;
     source_channel_count = features.has_alpha ? 4 : 3;
 
+    if (expected_channel == 0) {
+        expected_channel = source_channel_count;
+    }
+
     const size_t pixel_count = static_cast<size_t>(width) * static_cast<size_t>(height);
 
     if (expected_channel == 1) {
@@ -481,7 +485,8 @@ uint8_t* load_image_common(bool from_memory,
                            int& height,
                            int expected_width,
                            int expected_height,
-                           int expected_channel) {
+                           int expected_channel,
+                           int& out_channel) {
     const char* image_path;
     FreeUniquePtr<uint8_t> image_buffer;
     int source_channel_count = 0;
@@ -538,6 +543,32 @@ uint8_t* load_image_common(bool from_memory,
         LOG_ERROR("load image from '%s' failed", image_path);
         return nullptr;
     }
+    if (expected_channel == 0) {
+        expected_channel = source_channel_count == 2 ? 4 : (source_channel_count == 1 ? 3 : source_channel_count);
+        if (expected_channel != source_channel_count) {
+            FreeUniquePtr<uint8_t> promoted((uint8_t*)malloc((size_t)width * height * expected_channel));
+            if (promoted == nullptr) {
+                LOG_ERROR("error: allocate memory for channel promotion, image_path = %s", image_path);
+                return nullptr;
+            }
+            const size_t pixel_count = (size_t)width * (size_t)height;
+            for (size_t i = 0; i < pixel_count; ++i) {
+                if (source_channel_count == 1) {
+                    promoted.get()[i * 3 + 0] = image_buffer.get()[i];
+                    promoted.get()[i * 3 + 1] = image_buffer.get()[i];
+                    promoted.get()[i * 3 + 2] = image_buffer.get()[i];
+                } else {
+                    promoted.get()[i * 4 + 0] = image_buffer.get()[i * 2];
+                    promoted.get()[i * 4 + 1] = image_buffer.get()[i * 2];
+                    promoted.get()[i * 4 + 2] = image_buffer.get()[i * 2];
+                    promoted.get()[i * 4 + 3] = image_buffer.get()[i * 2 + 1];
+                }
+            }
+            image_buffer         = std::move(promoted);
+            source_channel_count = expected_channel;
+        }
+    }
+    // stb reports the source channel count even when it converts the output.
     if (source_channel_count < expected_channel) {
         fprintf(stderr,
                 "the number of channels for the input image must be >= %d,"
@@ -597,7 +628,7 @@ uint8_t* load_image_common(bool from_memory,
         }
         stbir_resize(image_buffer.get(), width, height, 0,
                      resized_image_buffer.get(), expected_width, expected_height, 0, STBIR_TYPE_UINT8,
-                     expected_channel, STBIR_ALPHA_CHANNEL_NONE, 0,
+                     expected_channel, expected_channel == 4 ? 3 : STBIR_ALPHA_CHANNEL_NONE, 0,
                      STBIR_EDGE_CLAMP, STBIR_EDGE_CLAMP,
                      STBIR_FILTER_BOX, STBIR_FILTER_BOX,
                      STBIR_COLORSPACE_SRGB, nullptr);
@@ -605,6 +636,7 @@ uint8_t* load_image_common(bool from_memory,
         height       = expected_height;
         image_buffer = std::move(resized_image_buffer);
     }
+    out_channel = expected_channel;
     return image_buffer.release();
 }
 
@@ -777,10 +809,11 @@ bool write_image_to_file(const std::string& path,
 uint8_t* load_image_from_file(const char* image_path,
                               int& width,
                               int& height,
+                              int& out_channel,
                               int expected_width,
                               int expected_height,
                               int expected_channel) {
-    return load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel);
+    return load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel, out_channel);
 }
 
 bool load_sd_image_from_file(sd_image_t* image,
@@ -790,13 +823,14 @@ bool load_sd_image_from_file(sd_image_t* image,
                              int expected_channel) {
     int width;
     int height;
-    image->data = load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel);
+    int resolved_channel = expected_channel;
+    image->data          = load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel, resolved_channel);
     if (image->data == nullptr) {
         return false;
     }
     image->width   = width;
     image->height  = height;
-    image->channel = expected_channel;
+    image->channel = resolved_channel;
     return true;
 }
 
@@ -804,10 +838,11 @@ uint8_t* load_image_from_memory(const char* image_bytes,
                                 int len,
                                 int& width,
                                 int& height,
+                                int& out_channel,
                                 int expected_width,
                                 int expected_height,
                                 int expected_channel) {
-    return load_image_common(true, image_bytes, len, width, height, expected_width, expected_height, expected_channel);
+    return load_image_common(true, image_bytes, len, width, height, expected_width, expected_height, expected_channel, out_channel);
 }
 
 static void append_avi_metadata(std::vector<uint8_t>& data, const std::string& parameters) {

examples/common/media_io.h

@@ -32,9 +32,12 @@ bool write_image_to_file(const std::string& path,
                          const std::string& parameters = "",
                          int quality                   = 90);
 
+// expected_channel == 0 preserves native channels (grayscale -> RGB, gray+alpha -> RGBA).
+// out_channel receives the output channel count.
 uint8_t* load_image_from_file(const char* image_path,
                               int& width,
                               int& height,
+                              int& out_channel,
                               int expected_width   = 0,
                               int expected_height  = 0,
                               int expected_channel = 3);
@@ -49,6 +52,7 @@ uint8_t* load_image_from_memory(const char* image_bytes,
                                 int len,
                                 int& width,
                                 int& height,
+                                int& out_channel,
                                 int expected_width   = 0,
                                 int expected_height  = 0,
                                 int expected_channel = 3);

examples/server/api.md

@@ -148,6 +148,19 @@ Native extension fields:
 
 - any `sdcpp API` fields embedded through `sd_cpp_extra_args` inside `prompt`
 
+Uploaded images are decoded at their original dimensions. The first decoded
+image establishes the generation dimensions if `size` is omitted. Input
+geometry follows `image_preprocess`: references preserve their dimensions by
+default, while init and mask use the generation canvas preset.
+
+Reference encoding then follows model presets and `ref_image_args`. To skip
+input geometry for references and disable resizing before VAE encoding, include
+this in `prompt`:
+
+```text
+edit this image <sd_cpp_extra_args>{"image_preprocess":"target=ref,mode=none","ref_image_args":"resize_before_vae=false"}</sd_cpp_extra_args>
+```
+
 Response fields:
 
 | Field | Type | Notes |
@@ -526,7 +539,7 @@ LTX and Wan preserve causal state between temporal tiles. Hunyuan Video and TAEH
 | Field | Type |
 | --- | --- |
 | `batch_count` | `integer` |
-| `auto_resize_ref_image` | `boolean` |
+| `ref_image_args` | `string` |
 | `increase_ref_index` | `boolean` |
 | `control_strength` | `number` |
 | `ip_adapter_strength` | `number` |
@@ -653,7 +666,7 @@ Example:
   "strength": 0.75,
   "seed": -1,
   "batch_count": 1,
-  "auto_resize_ref_image": true,
+  "ref_image_args": "",
   "increase_ref_index": false,
   "control_strength": 0.9,
   "ip_adapter_strength": 1.0,
@@ -728,6 +741,17 @@ Example:
 
 ### Image Encoding Rules
 
+Native image/video requests and SDAPI accept `image_preprocess` as a rule string
+or array of rule strings. OpenAI-compatible requests can supply it in
+`sd_cpp_extra_args`. See [Image preprocessing](../../docs/image_preprocessing.md)
+for one-time input geometry, native-resolution decoding, mask alignment, and
+`canny=true` for edge detection on any supported image input.
+
+Image generation also accepts `ref_image_args` as a string (for example,
+`"resize_before_vae=false"`) in native and SDAPI requests, or through
+`sd_cpp_extra_args` in OpenAI-compatible requests. It controls downstream
+reference encoding and is independent of input geometry rules.
+
 Any image field accepts:
 
 - a raw base64 string, or
@@ -735,12 +759,15 @@ Any image field accepts:
 
 Channel expectations:
 
-- `init_image`: 3 channels
-- `ref_images[]`: 3 channels
+- `init_image`: native channels (3 or 4); alpha is preserved and applied per model
+- `ref_images[]`: native channels (3 or 4); alpha is preserved and applied per model
 - `control_image`: 3 channels
 - `ip_adapter_image`: 3 channels
 - `mask_image`: 1 channel
 
+Models that support RGBA (e.g. Qwen-Image 2.1) use the alpha channel of `init_image`
+and `ref_images[]`. RGB-only models drop it, so sending RGBA is safe for every model.
+
 If omitted or null:
 
 - single-image fields map to an empty `sd_image_t`
@@ -760,7 +787,8 @@ Top-level scalar fields:
 | `strength` | `number` |
 | `seed` | `integer` |
 | `batch_count` | `integer` |
-| `auto_resize_ref_image` | `boolean` |
+| `ref_image_args` | `string` |
+| `image_preprocess` | `string \| array<string>` |
 | `increase_ref_index` | `boolean` |
 | `control_strength` | `number` |
 | `ip_adapter_strength` | `number` |

examples/server/main.cpp

@@ -76,9 +76,9 @@ int main(int argc, const char** argv) {
     SDSvrParams svr_params;
     SDContextParams ctx_params;
     SDGenerationParams default_gen_params;
-    parse_args(argc, argv, svr_params, ctx_params, default_gen_params);
 
     sd_set_log_callback(sd_log_cb, (void*)&svr_params);
+    parse_args(argc, argv, svr_params, ctx_params, default_gen_params);
 
     LOG_VERBOSE("version: %s", version_string().c_str());
     LOG_VERBOSE("%s", sd_get_system_info());

examples/server/routes_openai.cpp

@@ -157,39 +157,30 @@ static bool build_openai_edit_request(const httplib::Request& req,
     request.gen_params.height      = height;
     request.gen_params.batch_count = n;
 
+    std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
     for (auto& bytes : images_bytes) {
-        int img_w           = 0;
-        int img_h           = 0;
-        uint8_t* raw_pixels = load_image_from_memory(
-            reinterpret_cast<const char*>(bytes.data()),
-            static_cast<int>(bytes.size()),
-            img_w, img_h,
-            0, 0, 3);
+        int img_w            = 0;
+        int img_h            = 0;
+        int resolved_channel = 0;
+        uint8_t* raw_pixels  = load_image_from_memory(
+             reinterpret_cast<const char*>(bytes.data()),
+             static_cast<int>(bytes.size()),
+             img_w, img_h, resolved_channel,
+             0, 0,
+             0);
         if (raw_pixels == nullptr) {
             continue;
         }
 
         const bool is_first_ref_image = request.gen_params.ref_images.empty();
-        SDImageOwner image_owner({(uint32_t)img_w, (uint32_t)img_h, 3, raw_pixels});
+        SDImageOwner image_owner({(uint32_t)img_w, (uint32_t)img_h, (uint32_t)resolved_channel, raw_pixels});
         request.gen_params.set_width_and_height_if_unset(image_owner.get().width, image_owner.get().height);
 
         if (is_first_ref_image) {
-            int init_w = 0;
-            int init_h = 0;
-            if (request.gen_params.width_and_height_are_set()) {
-                init_w = request.gen_params.width;
-                init_h = request.gen_params.height;
-            }
-
-            int init_img_w       = 0;
-            int init_img_h       = 0;
-            uint8_t* init_pixels = load_image_from_memory(
-                reinterpret_cast<const char*>(bytes.data()),
-                static_cast<int>(bytes.size()),
-                init_img_w, init_img_h,
-                init_w, init_h, 3);
-            if (init_pixels != nullptr) {
-                request.gen_params.init_image.reset({(uint32_t)init_img_w, (uint32_t)init_img_h, 3, init_pixels});
+            request.gen_params.init_image = image_owner;
+            if (request.gen_params.init_image.get().data == nullptr) {
+                error_message = "could not allocate init image";
+                return false;
             }
         }
 
@@ -197,20 +188,15 @@ static bool build_openai_edit_request(const httplib::Request& req,
     }
 
     if (!mask_bytes.empty()) {
-        int expected_width  = 0;
-        int expected_height = 0;
-        if (request.gen_params.width_and_height_are_set()) {
-            expected_width  = request.gen_params.width;
-            expected_height = request.gen_params.height;
-        }
-        int mask_w = 0;
-        int mask_h = 0;
+        int mask_w       = 0;
+        int mask_h       = 0;
+        int mask_channel = 0;
 
         uint8_t* mask_raw = load_image_from_memory(
             reinterpret_cast<const char*>(mask_bytes.data()),
             static_cast<int>(mask_bytes.size()),
-            mask_w, mask_h,
-            expected_width, expected_height, 1);
+            mask_w, mask_h, mask_channel,
+            0, 0, 1);
         request.gen_params.mask_image.reset({(uint32_t)mask_w, (uint32_t)mask_h, 1, mask_raw});
         const sd_image_t& mask_image = request.gen_params.mask_image.get();
         request.gen_params.set_width_and_height_if_unset(mask_image.width, mask_image.height);
@@ -223,7 +209,6 @@ static bool build_openai_edit_request(const httplib::Request& req,
         });
     }
 
-    std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
     if (!sd_cpp_extra_args_str.empty() && !request.gen_params.from_json_str(sd_cpp_extra_args_str)) {
         error_message = "invalid sd_cpp_extra_args";
         return false;

examples/server/routes_sdapi.cpp

@@ -80,17 +80,6 @@ static enum sample_method_t get_sdapi_sample_method(std::string name) {
     return it != hardcoded.end() ? it->second : SAMPLE_METHOD_COUNT;
 }
 
-static void assign_solid_mask(SDImageOwner& mask_owner, int width, int height) {
-    const size_t pixel_count = static_cast<size_t>(width) * static_cast<size_t>(height);
-    uint8_t* raw_mask        = static_cast<uint8_t*>(malloc(pixel_count));
-    if (raw_mask == nullptr) {
-        mask_owner.reset({0, 0, 1, nullptr});
-        return;
-    }
-    std::memset(raw_mask, 255, pixel_count);
-    mask_owner.reset({(uint32_t)width, (uint32_t)height, 1, raw_mask});
-}
-
 static bool build_sdapi_img_gen_request(const json& j,
                                         ServerRuntime& runtime,
                                         bool img2img,
@@ -193,15 +182,25 @@ static bool build_sdapi_img_gen_request(const json& j,
         }
     }
 
-    if (img2img) {
-        const int expected_width  = request.gen_params.width_and_height_are_set() ? request.gen_params.width : 0;
-        const int expected_height = request.gen_params.width_and_height_are_set() ? request.gen_params.height : 0;
+    if (j.contains("ref_image_args")) {
+        if (!j["ref_image_args"].is_string()) {
+            error_message = "ref_image_args must be a string";
+            return false;
+        }
+        request.gen_params.ref_image_args = j["ref_image_args"].get<std::string>();
+    }
 
+    if (j.contains("image_preprocess") && !request.gen_params.parse_image_preprocess_json(j["image_preprocess"].dump())) {
+        error_message = "invalid image_preprocess";
+        return false;
+    }
+
+    if (img2img) {
         if (j.contains("init_images") && j["init_images"].is_array() && !j["init_images"].empty()) {
             if (decode_base64_image(j["init_images"][0].get<std::string>(),
-                                    3,
-                                    expected_width,
-                                    expected_height,
+                                    0,
+                                    0,
+                                    0,
                                     request.gen_params.init_image)) {
                 const sd_image_t& image = request.gen_params.init_image.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);
@@ -211,8 +210,8 @@ static bool build_sdapi_img_gen_request(const json& j,
         if (j.contains("mask") && j["mask"].is_string()) {
             if (decode_base64_image(j["mask"].get<std::string>(),
                                     1,
-                                    expected_width,
-                                    expected_height,
+                                    0,
+                                    0,
                                     request.gen_params.mask_image)) {
                 const sd_image_t& image = request.gen_params.mask_image.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);
@@ -225,9 +224,7 @@ static bool build_sdapi_img_gen_request(const json& j,
                 }
             }
         } else {
-            const int resolved_width  = request.gen_params.get_resolved_width();
-            const int resolved_height = request.gen_params.get_resolved_height();
-            assign_solid_mask(request.gen_params.mask_image, resolved_width, resolved_height);
+            request.gen_params.mask_image.reset({0, 0, 1, nullptr});
         }
 
         float denoising_strength = j.value("denoising_strength", -1.f);
@@ -243,13 +240,8 @@ static bool build_sdapi_img_gen_request(const json& j,
             }
             SDImageOwner image_owner;
             if (decode_base64_image(extra_image.get<std::string>(),
-                                    3,
-                                    request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
-                                        ? request.gen_params.width
-                                        : 0,
-                                    request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
-                                        ? request.gen_params.height
-                                        : 0,
+                                    0,
+                                    0, 0,
                                     image_owner)) {
                 const sd_image_t& image = image_owner.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);

examples/server/routes_sdcpp.cpp

@@ -127,7 +127,8 @@ static json make_img_gen_defaults_json(const SDGenerationParams& defaults, const
         {"seed", defaults.seed},
         {"batch_count", defaults.batch_count},
         {"qwen_image_layers", defaults.qwen_image_layers},
-        {"auto_resize_ref_image", defaults.auto_resize_ref_image},
+        {"ref_image_args", defaults.ref_image_args},
+        {"image_preprocess", defaults.image_preprocess},
         {"increase_ref_index", defaults.increase_ref_index},
         {"control_strength", defaults.control_strength},
         {"ip_adapter_strength", defaults.ip_adapter_strength},
@@ -153,6 +154,7 @@ static json make_vid_gen_defaults_json(const SDGenerationParams& defaults, const
         {"strength", defaults.strength},
         {"seed", defaults.seed},
         {"video_frames", defaults.video_frames},
+        {"image_preprocess", defaults.image_preprocess},
         {"fps", defaults.fps},
         {"moe_boundary", defaults.moe_boundary},
         {"vace_strength", defaults.vace_strength},

include/stable-diffusion.h

@@ -263,6 +263,11 @@ typedef struct {
     uint8_t* data;
 } sd_image_t;
 
+typedef struct {
+    // Semicolon-separated target=...,key=value rules. NULL preserves defaults.
+    const char* rules;
+} sd_image_preprocess_params_t;
+
 typedef struct {
     sd_image_t* frames;
     int frame_count;
@@ -410,6 +415,7 @@ typedef struct {
     int qwen_image_layers;
     bool circular_x;
     bool circular_y;
+    sd_image_preprocess_params_t image_preprocess;
 } sd_img_gen_params_t;
 
 typedef struct {
@@ -443,6 +449,7 @@ typedef struct {
     sd_hires_params_t hires;
     bool circular_x;
     bool circular_y;
+    sd_image_preprocess_params_t image_preprocess;
 } sd_vid_gen_params_t;
 
 typedef struct sd_ctx_t sd_ctx_t;

src/conditioning/conditioner.hpp

@@ -137,6 +137,7 @@ struct ConditionerParams {
     const std::vector<sd::Tensor<float>>* ref_images                    = nullptr;  // for qwen image edit
     const std::vector<MiniMaxH3PresentationItem>* minimax_h3_references = nullptr;
     RefImageParams ref_image_params;
+    bool allow_cache = false;
 };
 
 struct Conditioner {
@@ -1954,6 +1955,10 @@ struct LLMEmbedder : public Conditioner {
     std::shared_ptr<LLM::LLMRunner> llm;
     std::shared_ptr<T5Runner> byt5;
 
+    bool h3_text_cache_valid = false;
+    std::string h3_text_cache_text;
+    SDCondition h3_text_cache;
+
     LLMEmbedder(ggml_backend_t backend,
                 const String2TensorStorage& tensor_storage_map      = {},
                 SDVersion version                                   = VERSION_QWEN_IMAGE,
@@ -2298,6 +2303,25 @@ struct LLMEmbedder : public Conditioner {
 
     SDCondition get_learned_condition(int n_threads,
                                       const ConditionerParams& conditioner_params) override {
+        const bool h3_text_cacheable =
+            sd_version_is_minimax_h3(version) &&
+            conditioner_params.allow_cache &&
+            (conditioner_params.minimax_h3_references == nullptr ||
+             conditioner_params.minimax_h3_references->empty()) &&
+            (conditioner_params.ref_images == nullptr ||
+             conditioner_params.ref_images->empty());
+
+        if (sd_version_is_minimax_h3(version) && !h3_text_cacheable) {
+            h3_text_cache_valid = false;
+        }
+
+        if (h3_text_cacheable &&
+            h3_text_cache_valid &&
+            h3_text_cache_text == conditioner_params.text) {
+            LOG_INFO("H3 conditioning cache hit");
+            return h3_text_cache;
+        }
+
         std::string prompt;
         std::pair<int, int> prompt_attn_range;
         std::vector<std::string> extra_prompts;
@@ -3166,6 +3190,14 @@ struct LLMEmbedder : public Conditioner {
             int64_t tag_count    = static_cast<int64_t>(tags.size());
             result.c_token_types = sd::Tensor<int32_t>({tag_count}, std::move(tags));
         }
+
+        if (h3_text_cacheable) {
+            h3_text_cache_text  = conditioner_params.text;
+            h3_text_cache       = result;
+            h3_text_cache_valid = true;
+            LOG_INFO("H3 conditioning cache stored");
+        }
+
         return result;
     }
 };

src/detailer.cpp

@@ -970,6 +970,7 @@ bool adetail_image(adetailer_ctx_t* context,
         generation.pm_params           = {};
         generation.pulid_params        = {};
         generation.hires.enabled       = false;
+        generation.image_preprocess    = {};
         if (params.steps > 0) {
             generation.sample_params.sample_steps        = params.steps;
             generation.sample_params.custom_sigmas       = nullptr;

src/pipeline/diffusion_engine.cpp

@@ -1762,6 +1762,8 @@ bool StableDiffusionGGML::apply_loras(const sd_lora_t* loras, uint32_t lora_coun
         extension->collect_loras(all_loras);
     }
 
+    conditioning_cache_allowed_ = all_loras.empty();
+
     int64_t t0 = ggml_time_ms();
     end_runners();
     clear_lora_adapters();

src/pipeline/diffusion_engine.h

@@ -178,6 +178,7 @@ public:
     std::recursive_mutex execution_mutex;
     std::unique_ptr<ModelConfig> config_;
     RunnerState runner_state_;
+    bool conditioning_cache_allowed_ = false;
     bool executing_ = false;
 
     std::shared_ptr<Denoiser> denoiser;

src/pipeline/image.cpp

@@ -10,6 +10,7 @@
 #include "model/vae/vae.hpp"
 #include "request.h"
 #include "runtime/denoiser.hpp"
+#include "runtime/image_preprocess.h"
 #include "upscaler.h"
 
 namespace sd::pipeline {
@@ -800,6 +801,12 @@ namespace sd::pipeline {
         int64_t t0            = ggml_time_ms();
         sd->vae_tiling_params = sd_img_gen_params->vae_tiling_params;
         GenerationRequest request(sd, sd_img_gen_params);
+        sd::ImagePreprocessor preprocessing(sd_img_gen_params->image_preprocess.rules);
+        sd_img_gen_params_t processed_params = *sd_img_gen_params;
+        if (!preprocessing.prepare_inputs(processed_params, request.width, request.height))
+            return false;
+        sd_img_gen_params = &processed_params;
+        request.pm_params = processed_params.pm_params;
         LOG_INFO("generate_image %dx%d", request.width, request.height);
 
         sd->rng->manual_seed(request.seed);

src/pipeline/video.cpp

@@ -15,6 +15,7 @@
 #include "model/vae/vae.hpp"
 #include "request.h"
 #include "runtime/denoiser.hpp"
+#include "runtime/image_preprocess.h"
 
 namespace sd::pipeline {
 
@@ -470,11 +471,15 @@ namespace sd::pipeline {
         sd::Tensor<float> end_image;
 
         if (sd_vid_gen_params->init_image.data) {
-            start_image = sd_image_to_tensor(sd_vid_gen_params->init_image, request->width, request->height);
+            start_image = ensure_image_tensor_channels(
+                sd_image_to_tensor(sd_vid_gen_params->init_image, request->width, request->height),
+                sd->get_image_channels());
         }
 
         if (sd_vid_gen_params->end_image.data) {
-            end_image = sd_image_to_tensor(sd_vid_gen_params->end_image, request->width, request->height);
+            end_image = ensure_image_tensor_channels(
+                sd_image_to_tensor(sd_vid_gen_params->end_image, request->width, request->height),
+                sd->get_image_channels());
         }
 
         if (sd_version_is_minimax_h3(sd->version)) {
@@ -1152,6 +1157,10 @@ namespace sd::pipeline {
         condition_params.zero_out_masked       = true;
         condition_params.ref_images            = &latents.ref_images;
         condition_params.minimax_h3_references = &latents.minimax_presentation_refs;
+        condition_params.allow_cache =
+            sd_version_is_minimax_h3(sd->version) &&
+            sd->conditioning_cache_allowed_ &&
+            !request.use_uncond;
         if (sd_version_is_lingbot_video(sd->version) || sd_version_is_minimax_h3(sd->version)) {
             condition_params.ref_image_params.vlm_resize_mode = RefImageResizeMode::AREA;
         }
@@ -1416,7 +1425,9 @@ namespace sd::pipeline {
         sd::Tensor<float> video_mask = make_ltxav_video_denoise_mask(video_latent, 1.f);
 
         if (sd_vid_gen_params->init_image.data != nullptr) {
-            sd::Tensor<float> start_image = sd_image_to_tensor(sd_vid_gen_params->init_image, image_width, image_height);
+            sd::Tensor<float> start_image = ensure_image_tensor_channels(
+                sd_image_to_tensor(sd_vid_gen_params->init_image, image_width, image_height),
+                sd->get_image_channels());
             if (!apply_ltxav_condition_image_by_latent_index(sd,
                                                              start_image,
                                                              &video_latent,
@@ -1429,7 +1440,9 @@ namespace sd::pipeline {
         }
 
         if (sd_vid_gen_params->end_image.data != nullptr) {
-            sd::Tensor<float> end_image        = sd_image_to_tensor(sd_vid_gen_params->end_image, image_width, image_height);
+            sd::Tensor<float> end_image = ensure_image_tensor_channels(
+                sd_image_to_tensor(sd_vid_gen_params->end_image, image_width, image_height),
+                sd->get_image_channels());
             sd::Tensor<float> end_image_latent = encode_ltxav_condition_image(sd, end_image, "end");
             if (end_image_latent.empty()) {
                 return false;
@@ -1518,6 +1531,7 @@ namespace sd::pipeline {
         img_gen_params.qwen_image_layers = 0;
         img_gen_params.circular_x        = sd_vid_gen_params->circular_x;
         img_gen_params.circular_y        = sd_vid_gen_params->circular_y;
+        img_gen_params.image_preprocess  = sd_vid_gen_params->image_preprocess;
 
         sd->animatediff_num_frames = n_frames;
         bool ok                    = generate_image(sd, &img_gen_params, frames_out, num_frames_out);
@@ -1548,6 +1562,11 @@ namespace sd::pipeline {
         sd->vae_tiling_params = sd_vid_gen_params->vae_tiling_params;
         sd->apply_circular_axes(sd_vid_gen_params->circular_x, sd_vid_gen_params->circular_y);
         GenerationRequest request(sd, sd_vid_gen_params);
+        sd::ImagePreprocessor preprocessing(sd_vid_gen_params->image_preprocess.rules);
+        sd_vid_gen_params_t processed_params = *sd_vid_gen_params;
+        if (!preprocessing.prepare_inputs(processed_params, request.width, request.height))
+            return false;
+        sd_vid_gen_params = &processed_params;
         if (fps_out != nullptr) {
             *fps_out = request.fps;
         }

src/runtime/image_preprocess.cpp

@@ -0,0 +1,447 @@
+#include "image_preprocess.h"
+#include "core/util.h"
+
+#include <climits>
+#include <set>
+
+namespace sd {
+
+    static constexpr std::pair<const char*, ImageTarget> image_targets[] = {
+        {"init", ImageTarget::Init},
+        {"end", ImageTarget::End},
+        {"mask", ImageTarget::Mask},
+        {"control", ImageTarget::Control},
+        {"ref", ImageTarget::Ref},
+        {"ip-adapter", ImageTarget::IPAdapter},
+        {"id", ImageTarget::ID},
+        {"control-frame", ImageTarget::ControlFrame},
+    };
+
+    static constexpr std::pair<const char*, ImageResizeMode> image_resize_modes[] = {
+        {"auto", ImageResizeMode::Auto},
+        {"none", ImageResizeMode::None},
+        {"stretch", ImageResizeMode::Stretch},
+        {"crop", ImageResizeMode::Crop},
+        {"crop-resize", ImageResizeMode::CropResize},
+        {"fit-pad", ImageResizeMode::FitPad},
+    };
+
+    template <typename T, size_t N>
+    static bool parse_enum(const std::string& text, const std::pair<const char*, T> (&names)[N], T& value) {
+        for (const auto& entry : names) {
+            if (text == entry.first) {
+                value = entry.second;
+                return true;
+            }
+        }
+        return false;
+    }
+
+    template <typename T, size_t N>
+    static const char* enum_name(T value, const std::pair<const char*, T> (&names)[N]) {
+        for (const auto& entry : names) {
+            if (value == entry.second)
+                return entry.first;
+        }
+        return "unknown";
+    }
+
+    template <typename T>
+    static bool one_of(T value, std::initializer_list<T> choices) {
+        return std::find(choices.begin(), choices.end(), value) != choices.end();
+    }
+
+    static bool one_of(const std::string& value, std::initializer_list<const char*> choices) {
+        for (const char* choice : choices) {
+            if (value == choice)
+                return true;
+        }
+        return false;
+    }
+
+    static ImageResizeMode resolve_mode(const std::map<std::string, std::string>& options, ImageResizeMode default_mode) {
+        auto it              = options.find("mode");
+        ImageResizeMode mode = ImageResizeMode::Auto;
+        if (it != options.end())
+            parse_enum(it->second, image_resize_modes, mode);
+        if (mode != ImageResizeMode::Auto)
+            return mode;
+        return options.count("width") && default_mode == ImageResizeMode::None ? ImageResizeMode::Stretch : default_mode;
+    }
+
+    bool ImagePreprocessor::fail(const std::string& message) const {
+        LOG_ERROR("image preprocessing: %s", message.c_str());
+        valid_ = false;
+        return false;
+    }
+
+    ImagePreprocessor::ImagePreprocessor(const char* text) {
+        if (text == nullptr || trim(text).empty())
+            return;
+        for (const auto& part : split_string(text, ';')) {
+            ImagePreprocessRule rule;
+            std::set<std::string> keys;
+            if (trim(part).empty()) {
+                fail("empty rule");
+                return;
+            }
+            for (const auto& entry : split_string(part, ',')) {
+                size_t equal = entry.find('=');
+                if (equal == std::string::npos) {
+                    fail("expected key=value: " + entry);
+                    return;
+                }
+                std::string key   = trim(entry.substr(0, equal));
+                std::string value = trim(entry.substr(equal + 1));
+                bool ok           = !value.empty() && keys.insert(key).second;
+                int number        = 0;
+                if (key == "target") {
+                    ok &= parse_enum(value, image_targets, rule.target);
+                } else if (key == "index") {
+                    ok &= parse_strict_int(value, rule.index) && rule.index >= 0;
+                } else {
+                    if (key == "mode") {
+                        ImageResizeMode mode;
+                        ok &= parse_enum(value, image_resize_modes, mode);
+                    } else if (key == "filter") {
+                        ok &= one_of(value, {"auto", "nearest", "nearest-exact", "bilinear", "bicubic", "lanczos"});
+                    } else if (key == "antialias") {
+                        ok &= one_of(value, {"auto", "true", "false"});
+                    } else if (key == "canny") {
+                        ok &= one_of(value, {"true", "false"});
+                    } else if (key == "anchor") {
+                        ok &= one_of(value, {"center", "top", "bottom", "left", "right"});
+                    } else if (key == "width" || key == "height") {
+                        ok &= parse_strict_int(value, number) && number > 0;
+                    } else if (key == "pad_color") {
+                        ok &= value.size() == 7 || value.size() == 9;
+                        ok &= !value.empty() && value[0] == '#';
+                        for (size_t i = 1; i < value.size(); ++i)
+                            ok &= std::isxdigit(static_cast<unsigned char>(value[i])) != 0;
+                    } else {
+                        ok = false;
+                    }
+                    rule.options[key] = value;
+                }
+                if (!ok) {
+                    fail("invalid or duplicate option: " + entry);
+                    return;
+                }
+            }
+            if (!keys.count("target") || rule.options.empty() ||
+                rule.options.count("width") != rule.options.count("height") ||
+                (rule.index >= 0 && !one_of(rule.target, {ImageTarget::Ref, ImageTarget::ID, ImageTarget::ControlFrame}))) {
+                fail("invalid target, index, or incomplete dimensions: " + part);
+                return;
+            }
+            rules_.push_back(std::move(rule));
+        }
+        for (const auto& rule : rules_) {
+            const auto options = resolve_options(rule.target, std::max(0, rule.index));
+            if (options.count("antialias") && options.at("antialias") == "true" && options.count("filter") &&
+                one_of(options.at("filter"), {"nearest", "nearest-exact"})) {
+                fail("antialias requires bilinear, bicubic, or lanczos");
+                return;
+            }
+        }
+    }
+
+    std::map<std::string, std::string> ImagePreprocessor::resolve_options(ImageTarget target, int index) const {
+        std::map<std::string, std::string> options;
+        for (int specificity = 0; specificity < 2; ++specificity) {
+            for (const auto& rule : rules_) {
+                if (rule.target == target &&
+                    rule.index == (specificity == 0 ? -1 : index)) {
+                    for (const auto& entry : rule.options)
+                        options[entry.first] = entry.second;
+                }
+            }
+        }
+        return options;
+    }
+
+    bool ImagePreprocessor::validate_inputs(const sd_img_gen_params_t& params) const {
+        const std::map<ImageTarget, int> counts = {
+            {ImageTarget::Init, params.init_image.data != nullptr},
+            {ImageTarget::Mask, params.mask_image.data != nullptr},
+            {ImageTarget::Control, params.control_image.data != nullptr},
+            {ImageTarget::IPAdapter, params.ip_adapter_image.data != nullptr},
+            {ImageTarget::Ref, params.ref_images != nullptr ? params.ref_images_count : 0},
+            {ImageTarget::ID, params.pm_params.id_images != nullptr ? params.pm_params.id_images_count : 0},
+        };
+        for (const auto& rule : rules_) {
+            auto it   = counts.find(rule.target);
+            int count = it == counts.end() ? 0 : it->second;
+            if (count <= 0 || rule.index >= count) {
+                return fail(std::string("rule targets an unavailable image: ") + enum_name(rule.target, image_targets));
+            }
+        }
+        return valid_;
+    }
+
+    bool ImagePreprocessor::validate_inputs(const sd_vid_gen_params_t& params) const {
+        const std::map<ImageTarget, int> counts = {
+            {ImageTarget::Init, params.init_image.data != nullptr},
+            {ImageTarget::End, params.end_image.data != nullptr},
+            {ImageTarget::Ref, params.ref_images != nullptr ? params.ref_images_count : 0},
+            {ImageTarget::ControlFrame, params.control_frames != nullptr ? params.control_frames_size : 0},
+        };
+        for (const auto& rule : rules_) {
+            auto it   = counts.find(rule.target);
+            int count = it == counts.end() ? 0 : it->second;
+            if (count <= 0 || rule.index >= count)
+                return fail(std::string("rule targets an unavailable video input: ") + enum_name(rule.target, image_targets));
+        }
+        return valid_;
+    }
+
+    static int anchor_offset(int remaining, const std::string& anchor, bool horizontal) {
+        if (anchor == (horizontal ? "left" : "top"))
+            return 0;
+        if (anchor == (horizontal ? "right" : "bottom"))
+            return remaining;
+        return remaining / 2;
+    }
+
+    Tensor<float> ImagePreprocessor::apply_transform(const Tensor<float>& image, const std::map<std::string, std::string>& options, ImageTransform p, const std::string& label, ops::InterpolateMode default_filter) const {
+        auto value = [&](const char* key, const char* fallback) {
+            auto it = options.find(key);
+            return it == options.end() ? std::string(fallback) : it->second;
+        };
+        std::string filter        = value("filter", "auto");
+        ops::InterpolateMode mode = default_filter;
+        if (filter == "nearest")
+            mode = ops::InterpolateMode::Nearest;
+        if (filter == "nearest-exact")
+            mode = ops::InterpolateMode::NearestExact;
+        if (filter == "bilinear")
+            mode = ops::InterpolateMode::Bilinear;
+        if (filter == "bicubic")
+            mode = ops::InterpolateMode::Bicubic;
+        if (filter == "lanczos")
+            mode = ops::InterpolateMode::Lanczos;
+        bool filtered  = ops::is_2d_filter_interpolate_mode(mode);
+        bool antialias = value("antialias", "auto") == "true" ||
+                         (value("antialias", "auto") == "auto" && filtered &&
+                          (p.resize_width < p.crop_width || p.resize_height < p.crop_height));
+        if (antialias && !filtered) {
+            fail(label + ": antialias requires bilinear, bicubic, or lanczos");
+            return {};
+        }
+        auto cropped = ops::slice(ops::slice(image, 0, p.x, p.x + p.crop_width), 1, p.y, p.y + p.crop_height);
+        int channels = static_cast<int>(image.shape()[2]);
+        bool resize  = p.resize_width != p.crop_width || p.resize_height != p.crop_height;
+        if (resize && channels == 4 && filtered) {
+            for (int64_t i = 0, pixels = cropped.shape()[0] * cropped.shape()[1]; i < pixels; ++i) {
+                for (int c = 0; c < 3; ++c)
+                    cropped[i + c * pixels] *= cropped[i + 3 * pixels];
+            }
+        }
+        auto resized = ops::interpolate(cropped, {p.resize_width, p.resize_height, channels, 1}, mode, false, antialias);
+        if (resize && channels == 4 && filtered) {
+            for (int64_t i = 0, pixels = resized.shape()[0] * resized.shape()[1]; i < pixels; ++i) {
+                float alpha = std::clamp(resized[i + 3 * pixels], 0.f, 1.f);
+                for (int c = 0; c < 3; ++c)
+                    resized[i + c * pixels] = alpha > 1e-6f ? resized[i + c * pixels] / alpha : 0.f;
+            }
+        }
+        resized = ops::clamp(resized, 0.f, 1.f);
+        Tensor<float> output({p.width, p.height, channels, 1});
+        std::string color = value("pad_color", "#000000ff");
+        if (color.size() == 7)
+            color += "ff";
+        uint8_t rgba[4];
+        for (int c = 0; c < 4; ++c)
+            rgba[c] = static_cast<uint8_t>(std::strtoul(color.substr(1 + c * 2, 2).c_str(), nullptr, 16));
+        for (int c = 0; c < channels; ++c) {
+            float fill = rgba[channels == 1 ? 0 : c] / 255.f;
+            for (int y = 0; y < p.height; ++y) {
+                for (int x = 0; x < p.width; ++x) {
+                    output.index(x, y, c, 0) = x >= p.pad_x && x < p.pad_x + p.resize_width && y >= p.pad_y && y < p.pad_y + p.resize_height
+                                                   ? resized.index(x - p.pad_x, y - p.pad_y, c, 0)
+                                                   : fill;
+                }
+            }
+        }
+        LOG_INFO("preprocess %s: %dx%d crop=(%d,%d,%d,%d) resize=%dx%d pad=(%d,%d) output=%dx%d filter=%s(%d) antialias=%s",
+                 label.c_str(), p.source_width, p.source_height, p.x, p.y, p.crop_width, p.crop_height,
+                 p.resize_width, p.resize_height, p.pad_x, p.pad_y, p.width, p.height, filter.c_str(), static_cast<int>(mode), BOOL_STR(antialias));
+        return output;
+    }
+
+    Tensor<float> ImagePreprocessor::apply_geometry(const Tensor<float>& image, ImageTarget target, int index, int width, int height, ImageResizeMode default_mode, ops::InterpolateMode default_filter, ImageTransform* plan_out) const {
+        if (!valid_ || image.empty())
+            return {};
+        const std::string label = std::string(enum_name(target, image_targets)) + "[" + std::to_string(index) + "]";
+        auto options            = resolve_options(target, index);
+        if (image.dim() != 4 || image.shape()[3] != 1 || image.shape()[2] < 1 || image.shape()[2] > 4) {
+            fail(label + ": expected one image with 1 to 4 channels");
+            return {};
+        }
+        ImageTransform p;
+        p.source_width = p.crop_width = static_cast<int>(image.shape()[0]);
+        p.source_height = p.crop_height = static_cast<int>(image.shape()[1]);
+        int target_width                = width > 0 ? width : p.source_width;
+        int target_height               = height > 0 ? height : p.source_height;
+        if (options.count("width")) {
+            parse_strict_int(options.at("width"), target_width);
+            parse_strict_int(options.at("height"), target_height);
+        }
+        ImageResizeMode mode = resolve_mode(options, default_mode);
+        std::string anchor   = options.count("anchor") ? options.at("anchor") : "center";
+        p.width = p.resize_width = target_width;
+        p.height = p.resize_height = target_height;
+        if (mode == ImageResizeMode::None) {
+            if (options.count("width") && (target_width != p.source_width || target_height != p.source_height)) {
+                fail(label + ": mode=none conflicts with requested dimensions");
+                return {};
+            }
+            p.width = p.resize_width = p.source_width;
+            p.height = p.resize_height = p.source_height;
+        } else if (mode == ImageResizeMode::Crop || mode == ImageResizeMode::CropResize) {
+            if (mode == ImageResizeMode::Crop) {
+                p.crop_width  = target_width;
+                p.crop_height = target_height;
+            } else if (int64_t(p.source_width) * target_height > int64_t(p.source_height) * target_width) {
+                p.crop_width = std::max(1, static_cast<int>(int64_t(p.source_height) * target_width / target_height));
+            } else {
+                p.crop_height = std::max(1, static_cast<int>(int64_t(p.source_width) * target_height / target_width));
+            }
+            if (p.crop_width > p.source_width || p.crop_height > p.source_height) {
+                fail(label + ": crop exceeds source dimensions");
+                return {};
+            }
+            p.x = anchor_offset(p.source_width - p.crop_width, anchor, true);
+            p.y = anchor_offset(p.source_height - p.crop_height, anchor, false);
+        } else if (mode == ImageResizeMode::FitPad) {
+            double scale    = std::min(double(target_width) / p.source_width, double(target_height) / p.source_height);
+            p.resize_width  = std::max(1, std::min(target_width, static_cast<int>(std::round(p.source_width * scale))));
+            p.resize_height = std::max(1, std::min(target_height, static_cast<int>(std::round(p.source_height * scale))));
+            p.pad_x         = anchor_offset(target_width - p.resize_width, anchor, true);
+            p.pad_y         = anchor_offset(target_height - p.resize_height, anchor, false);
+        }
+        if (p.width <= 0 || p.height <= 0) {
+            fail(label + ": invalid output dimensions");
+            return {};
+        }
+        uint64_t max_pixels = std::min<uint64_t>(INT64_MAX, SIZE_MAX / sizeof(float)) / static_cast<uint64_t>(image.shape()[2]);
+        if (uint64_t(p.width) * p.height > max_pixels || uint64_t(p.resize_width) * p.resize_height > max_pixels) {
+            fail(label + ": image allocation size overflows");
+            return {};
+        }
+        if (plan_out != nullptr)
+            *plan_out = p;
+        return apply_transform(image, options, p, label, default_filter);
+    }
+
+    Tensor<float> ImagePreprocessor::preprocess_input(sd_image_t image, ImageTarget target, int index, int width, int height) {
+        if (image.data == nullptr || image.width == 0 || image.height == 0 || image.width > INT_MAX || image.height > INT_MAX || image.channel < 1 || image.channel > 4) {
+            fail(std::string(enum_name(target, image_targets)) + ": invalid input image");
+            return {};
+        }
+        auto tensor = sd_image_to_tensor(image);
+        if (target == ImageTarget::Mask && has_init_transform_) {
+            auto options = resolve_options(target, index);
+            if (image.width != init_transform_.source_width || image.height != init_transform_.source_height) {
+                fail("mask and init source dimensions must match");
+                return {};
+            }
+            bool geometry_override = options.count("width") || options.count("anchor") ||
+                                     (options.count("mode") && options.at("mode") != "auto");
+            if (geometry_override) {
+                ImageTransform p;
+                auto init_options            = resolve_options(ImageTarget::Init, 0);
+                ImageResizeMode default_mode = resolve_mode(init_options, ImageResizeMode::CropResize);
+                auto result                  = apply_geometry(tensor, target, index, init_transform_.width, init_transform_.height, default_mode, ops::InterpolateMode::NearestExact, &p);
+                if (result.empty())
+                    return {};
+                const auto& q = init_transform_;
+                if (p.x != q.x || p.y != q.y || p.crop_width != q.crop_width || p.crop_height != q.crop_height ||
+                    p.resize_width != q.resize_width || p.resize_height != q.resize_height || p.pad_x != q.pad_x || p.pad_y != q.pad_y || p.width != q.width || p.height != q.height) {
+                    fail("mask geometry conflicts with init; configure geometry on init and filter on mask");
+                    return {};
+                }
+                return result;
+            }
+            return apply_transform(tensor, options, init_transform_, "mask[0]", ops::InterpolateMode::NearestExact);
+        }
+        auto result = apply_geometry(tensor, target, index, width, height, width > 0 ? ImageResizeMode::CropResize : ImageResizeMode::None,
+                                     target == ImageTarget::Mask ? ops::InterpolateMode::NearestExact : ops::InterpolateMode::Nearest,
+                                     target == ImageTarget::Init ? &init_transform_ : nullptr);
+        if (target == ImageTarget::Init)
+            has_init_transform_ = !result.empty();
+        return result;
+    }
+
+    ImagePreprocessor::~ImagePreprocessor() {
+        for (const auto& image : owned_images_)
+            std::free(image.data);
+    }
+
+    bool ImagePreprocessor::prepare_image(sd_image_t& image, ImageTarget target, int index, int width, int height) {
+        if (image.data == nullptr)
+            return true;
+        auto options = resolve_options(target, index);
+        bool canny   = options.count("canny") && options.at("canny") == "true";
+        auto tensor  = preprocess_input(image, target, index, width, height);
+        if (tensor.empty())
+            return false;
+        auto output = tensor_to_sd_image(tensor);
+        if (output.data == nullptr)
+            return fail("could not allocate input preprocessing buffer");
+        owned_images_.push_back(output);
+        if (canny && !preprocess_canny(output, 0.08f, 0.08f, 0.8f, 1.f, false))
+            return fail("Canny preprocessing failed");
+        image = output;
+        return true;
+    }
+
+    bool ImagePreprocessor::prepare_array(sd_image_t*& images, int count, ImageTarget target, std::vector<sd_image_t>& storage, int width, int height) {
+        if (count < 0 || (count > 0 && images == nullptr))
+            return fail(std::string("invalid image array: ") + enum_name(target, image_targets));
+        if (count == 0)
+            return true;
+        storage.assign(images, images + count);
+        for (int i = 0; i < count; ++i) {
+            if (storage[i].data == nullptr)
+                return fail(std::string("empty image in array: ") + enum_name(target, image_targets));
+            if (!prepare_image(storage[i], target, i, width, height))
+                return false;
+        }
+        images = storage.data();
+        return true;
+    }
+
+    bool ImagePreprocessor::prepare_inputs(sd_img_gen_params_t& params, int width, int height) {
+        if (prepared_)
+            return fail("inputs have already been prepared");
+        prepared_ = true;
+        if (!valid_ || !validate_inputs(params))
+            return false;
+        if (!prepare_image(params.init_image, ImageTarget::Init, 0, width, height) ||
+            !prepare_image(params.mask_image, ImageTarget::Mask, 0, width, height) ||
+            !prepare_image(params.control_image, ImageTarget::Control, 0, width, height) ||
+            !prepare_image(params.ip_adapter_image, ImageTarget::IPAdapter, 0, -1, -1) ||
+            !prepare_array(params.ref_images, params.ref_images_count, ImageTarget::Ref, ref_images_) ||
+            !prepare_array(params.pm_params.id_images, params.pm_params.id_images_count, ImageTarget::ID, id_images_))
+            return false;
+        params.image_preprocess = {};
+        return true;
+    }
+
+    bool ImagePreprocessor::prepare_inputs(sd_vid_gen_params_t& params, int width, int height) {
+        if (prepared_)
+            return fail("inputs have already been prepared");
+        prepared_ = true;
+        if (!valid_ || !validate_inputs(params))
+            return false;
+        if (!prepare_image(params.init_image, ImageTarget::Init, 0, width, height) ||
+            !prepare_image(params.end_image, ImageTarget::End, 0, width, height) ||
+            !prepare_array(params.ref_images, params.ref_images_count, ImageTarget::Ref, ref_images_) ||
+            !prepare_array(params.control_frames, params.control_frames_size, ImageTarget::ControlFrame, control_frames_, width, height))
+            return false;
+        params.image_preprocess = {};
+        return true;
+    }
+
+}  // namespace sd

src/runtime/image_preprocess.h

@@ -0,0 +1,88 @@
+#ifndef __SD_RUNTIME_IMAGE_PREPROCESS_H__
+#define __SD_RUNTIME_IMAGE_PREPROCESS_H__
+
+#include <map>
+#include <string>
+#include <vector>
+
+#include "core/tensor.hpp"
+#include "stable-diffusion.h"
+
+namespace sd {
+
+    enum class ImageTarget {
+        Init,
+        End,
+        Mask,
+        Control,
+        Ref,
+        IPAdapter,
+        ID,
+        ControlFrame,
+    };
+
+    enum class ImageResizeMode {
+        Auto,
+        None,
+        Stretch,
+        Crop,
+        CropResize,
+        FitPad,
+    };
+
+    struct ImageTransform {
+        int source_width  = 0;
+        int source_height = 0;
+        int x             = 0;
+        int y             = 0;
+        int crop_width    = 0;
+        int crop_height   = 0;
+        int resize_width  = 0;
+        int resize_height = 0;
+        int width         = 0;
+        int height        = 0;
+        int pad_x         = 0;
+        int pad_y         = 0;
+    };
+
+    struct ImagePreprocessRule {
+        ImageTarget target = ImageTarget::Init;
+        int index          = -1;
+        std::map<std::string, std::string> options;
+    };
+
+    class ImagePreprocessor {
+        std::vector<ImagePreprocessRule> rules_;
+        mutable bool valid_ = true;
+        ImageTransform init_transform_;
+        bool has_init_transform_ = false;
+        bool prepared_           = false;
+        std::vector<sd_image_t> owned_images_;
+        std::vector<sd_image_t> ref_images_;
+        std::vector<sd_image_t> id_images_;
+        std::vector<sd_image_t> control_frames_;
+
+        bool fail(const std::string& message) const;
+        std::map<std::string, std::string> resolve_options(ImageTarget target, int index) const;
+        Tensor<float> apply_transform(const Tensor<float>& image, const std::map<std::string, std::string>& options, ImageTransform plan, const std::string& label, ops::InterpolateMode default_filter) const;
+
+        bool prepare_image(sd_image_t& image, ImageTarget target, int index, int width, int height);
+        bool prepare_array(sd_image_t*& images, int count, ImageTarget target, std::vector<sd_image_t>& storage, int width = -1, int height = -1);
+
+    public:
+        explicit ImagePreprocessor(const char* rules = nullptr);
+        ~ImagePreprocessor();
+        ImagePreprocessor(const ImagePreprocessor&)            = delete;
+        ImagePreprocessor& operator=(const ImagePreprocessor&) = delete;
+        bool prepare_inputs(sd_img_gen_params_t& params, int width, int height);
+        bool prepare_inputs(sd_vid_gen_params_t& params, int width, int height);
+        bool is_valid() const { return valid_; }
+        bool validate_inputs(const sd_img_gen_params_t& params) const;
+        bool validate_inputs(const sd_vid_gen_params_t& params) const;
+        Tensor<float> apply_geometry(const Tensor<float>& image, ImageTarget target, int index, int width, int height, ImageResizeMode default_mode = ImageResizeMode::Stretch, ops::InterpolateMode default_filter = ops::InterpolateMode::Nearest, ImageTransform* plan_out = nullptr) const;
+        Tensor<float> preprocess_input(sd_image_t image, ImageTarget target, int index = 0, int width = -1, int height = -1);
+    };
+
+}  // namespace sd
+
+#endif  // __SD_RUNTIME_IMAGE_PREPROCESS_H__

src/runtime/preprocessing.hpp

@@ -165,16 +165,18 @@ static inline sd::Tensor<float> convolve_tensor(const sd::Tensor<float>& input,
     return output;
 }
 
-static inline sd::Tensor<float> grayscale_tensor(const sd::Tensor<float>& rgb_img) {
-    GGML_ASSERT(rgb_img.dim() == 4);
-    GGML_ASSERT(rgb_img.shape()[2] >= 3);
-    sd::Tensor<float> grayscale({rgb_img.shape()[0], rgb_img.shape()[1], 1, rgb_img.shape()[3]});
-    for (int64_t iy = 0; iy < rgb_img.shape()[1]; ++iy) {
-        for (int64_t ix = 0; ix < rgb_img.shape()[0]; ++ix) {
-            float r    = preprocessing_get_4d(rgb_img, ix, iy, 0, 0);
-            float g    = preprocessing_get_4d(rgb_img, ix, iy, 1, 0);
-            float b    = preprocessing_get_4d(rgb_img, ix, iy, 2, 0);
-            float gray = 0.2989f * r + 0.5870f * g + 0.1140f * b;
+static inline sd::Tensor<float> grayscale_tensor(const sd::Tensor<float>& image) {
+    GGML_ASSERT(image.dim() == 4);
+    GGML_ASSERT(image.shape()[2] >= 1);
+    sd::Tensor<float> grayscale({image.shape()[0], image.shape()[1], 1, image.shape()[3]});
+    for (int64_t iy = 0; iy < image.shape()[1]; ++iy) {
+        for (int64_t ix = 0; ix < image.shape()[0]; ++ix) {
+            float gray = preprocessing_get_4d(image, ix, iy, 0, 0);
+            if (image.shape()[2] >= 3) {
+                float g = preprocessing_get_4d(image, ix, iy, 1, 0);
+                float b = preprocessing_get_4d(image, ix, iy, 2, 0);
+                gray    = 0.2989f * gray + 0.5870f * g + 0.1140f * b;
+            }
             preprocessing_set_4d(grayscale, gray, ix, iy, 0, 0);
         }
     }
@@ -317,11 +319,12 @@ bool preprocess_canny(sd_image_t img, float high_threshold, float low_threshold,
     image_gray              = non_max_supression(G, theta);
     threshold_hystersis(&image_gray, high_threshold, low_threshold, weak, strong);
 
+    const uint32_t color_channels = img.channel == 2 || img.channel == 4 ? img.channel - 1 : img.channel;
     for (uint32_t iy = 0; iy < img.height; ++iy) {
         for (uint32_t ix = 0; ix < img.width; ++ix) {
             float gray = preprocessing_get_4d(image_gray, ix, iy, 0, 0);
             gray       = inverse ? 1.0f - gray : gray;
-            for (uint32_t c = 0; c < img.channel; ++c) {
+            for (uint32_t c = 0; c < color_channels; ++c) {
                 preprocessing_set_4d(image, gray, ix, iy, c, 0);
             }
         }

src/stable-diffusion.cpp

@@ -751,28 +751,20 @@ SD_API bool generate_video(sd_ctx_t* sd_ctx,
                            int* num_frames_out,
                            sd_audio_t** audio_out,
                            int* fps_out) {
-    if (sd_ctx == nullptr || sd_ctx->sd == nullptr || sd_vid_gen_params == nullptr) {
-        if (fps_out != nullptr) {
-            *fps_out = 0;
-        }
-        return false;
-    }
-
-    if (frames_out != nullptr) {
+    if (frames_out != nullptr)
         *frames_out = nullptr;
-    }
-    if (audio_out != nullptr) {
+    if (audio_out != nullptr)
         *audio_out = nullptr;
-    }
-    if (num_frames_out != nullptr) {
+    if (num_frames_out != nullptr)
         *num_frames_out = 0;
+    if (fps_out != nullptr)
+        *fps_out = 0;
+    if (sd_ctx == nullptr || sd_ctx->sd == nullptr || sd_vid_gen_params == nullptr) {
+        return false;
     }
 
     StableDiffusionGGML::ExecutionScope execution(*sd_ctx->sd);
     if (!execution.ready) {
-        if (fps_out != nullptr) {
-            *fps_out = 0;
-        }
         return false;
     }
 

src/upscaler.cpp

@@ -111,10 +111,22 @@ bool UpscalerGGML::load_from_file(const std::string& esrgan_path,
 
 sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_tensor) {
     sd::ParallelScope tensor_scope(&tensor_executor);
+    if (input_tensor.empty() || input_tensor.dim() != 4 ||
+        (input_tensor.shape()[2] != 3 && input_tensor.shape()[2] != 4)) {
+        LOG_ERROR("esrgan expects a 4D RGB or RGBA image tensor");
+        return {};
+    }
+
+    const bool has_alpha = input_tensor.shape()[2] == 4;
+    sd::Tensor<float> rgb;
+    if (has_alpha) {
+        rgb = sd::ops::slice(input_tensor, 2, 0, 3);
+    }
+    const sd::Tensor<float>& model_input = has_alpha ? rgb : input_tensor;
     sd::Tensor<float> upscaled;
     const int scale = esrgan_upscaler->config.scale;
     if (tile_size <= 0 || (input_tensor.shape()[0] <= tile_size && input_tensor.shape()[1] <= tile_size)) {
-        upscaled = esrgan_upscaler->compute(n_threads, input_tensor);
+        upscaled = esrgan_upscaler->compute(n_threads, model_input);
     } else {
         auto on_processing = [&](const sd::Tensor<float>& input_tile) -> sd::Tensor<float> {
             auto output_tile = esrgan_upscaler->compute(n_threads, input_tile);
@@ -125,7 +137,7 @@ sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_te
             return output_tile;
         };
 
-        upscaled = process_tiles_2d(input_tensor,
+        upscaled = process_tiles_2d(model_input,
                                     static_cast<int>(input_tensor.shape()[0] * scale),
                                     static_cast<int>(input_tensor.shape()[1] * scale),
                                     scale,
@@ -141,6 +153,14 @@ sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_te
         LOG_ERROR("esrgan compute failed");
         return {};
     }
+    if (has_alpha) {
+        auto alpha       = sd::ops::slice(input_tensor, 2, 3, 4);
+        auto alpha_shape = alpha.shape();
+        alpha_shape[0]   = upscaled.shape()[0];
+        alpha_shape[1]   = upscaled.shape()[1];
+        alpha            = sd::ops::interpolate(alpha, alpha_shape, sd::ops::InterpolateMode::Bilinear);
+        upscaled         = sd::ops::concat(upscaled, alpha, 2);
+    }
     return upscaled;
 }